Data fusion method and device, terminal equipment and storage medium

Abstract

The invention is suitable for the technical field of data processing, and provides a data fusion method and device, terminal equipment and a storage medium, and the method comprises the steps: obtaining initial point cloud data collected by a radar which is disposed in a tunnel and has a detection direction consistent with a driving direction, initial image data collected by a camera device which is arranged in the tunnel and has the shooting direction consistent with the driving direction is obtained; and performing data fusion processing on the initial point cloud data and the initial image data according to a timestamp to obtain target data. According to the method, the detection direction of the radar and the shooting direction of the camera device are consistent with the driving direction, so that pollutants brought by the vehicle in the driving process when the detection direction and the shooting direction are opposite to the driving direction are avoided, namely, pollution brought by the coming vehicle is avoided; the quality of the obtained initial point cloud data and initial image data is improved, so that the quality of data fusion is improved, and the detection accuracy of the target object in the data is improved.

Description

Technical Field

[0001] This application belongs to the technical field of data processing, and particularly relates to a data fusion method, apparatus, terminal device, and storage medium. Background Art

[0002] The final twin display effect of the intelligent tunnel completely depends on the perception effect of the roadside base station. Therefore, all-weather, long-time stable operation and detection requirements are put forward for roadside perception. Currently, roadside perception usually fuses the point cloud data collected by the radar and the image data collected by the camera to improve the perception accuracy of the target object.

[0003] Existing data fusion methods usually only simply preprocess the point cloud data collected by the radar and preprocess the image data collected by the camera. After that, the processed point cloud data and image data are fused. The consideration is not comprehensive enough, and the influence of many dust and other pollutants in the tunnel on the perception devices (such as radar and camera) is not taken into account, making it difficult for the existing technology to meet the actual needs, reducing the quality of data fusion, and further reducing the detection success rate of the target object in the data. Summary of the Invention

[0004] The embodiments of this application provide a data fusion method, apparatus, terminal device, and storage medium, which improve the quality of data fusion, and further improve the detection success rate of the target object in the data.

[0005] In a first aspect, the embodiments of this application provide a data fusion method, including:

[0006] Obtaining initial point cloud data collected by a radar disposed in the tunnel and having a detection direction consistent with the driving direction, and obtaining initial image data collected by an imaging device disposed in the tunnel and having a shooting direction consistent with the driving direction;

[0007] Performing data fusion processing on the initial point cloud data and the initial image data according to timestamps to obtain target data.

[0008] Optionally, before obtaining the initial point cloud data collected by the radar disposed in the tunnel and having a detection direction consistent with the driving direction, and obtaining the initial image data collected by the imaging device disposed in the tunnel and having a shooting direction consistent with the driving direction, it further includes:

[0009] Determining the environmental information in the tunnel;

[0010] Adjusting the detection angle of the radar and the shooting angle of the imaging device respectively according to the environmental information.

[0011] Optionally, the data fusion process of the initial point cloud data and the initial image data according to the time stamp to obtain target data includes:

[0012] Obtain background point cloud data in the same scene as the initial point cloud data, and background image data in the same scene as the initial image data;

[0013] Process the initial point cloud data according to the background point cloud data to obtain target point cloud data;

[0014] Process the initial image data according to the background image data to obtain target image data;

[0015] Perform fusion processing on the target point cloud data and the target image data according to the time stamp to obtain the target data.

[0016] Optionally, the number of the radars is multiple, there is an overlapping part between the detection areas of any two adjacent radars, and each radar corresponds to one of the camera devices.

[0017] Optionally, the camera device corresponding to each radar includes a short-focus camera and a long-focus camera.

[0018] Optionally, there is an overlapping part between the shooting ranges of the short-focus camera and the long-focus camera corresponding to each radar.

[0019] Optionally, there is an overlapping part between the shooting range of the long-focus camera corresponding to each radar and the shooting range of the short-focus camera corresponding to the adjacent radar.

[0020] In a second aspect, an embodiment of the present application provides a data fusion device, including:

[0021] A first acquisition unit, configured to acquire initial point cloud data collected by a radar disposed in a tunnel and having a detection direction consistent with the driving direction, and acquire initial image data collected by a camera device disposed in the tunnel and having a shooting direction consistent with the driving direction;

[0022] A first processing unit, configured to perform data fusion processing on the initial point cloud data and the initial image data according to the time stamp to obtain target data.

[0023] In a third aspect, an embodiment of the present application provides a terminal device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the data fusion method described in any one of the first aspects above is implemented.

[0024] Fourthly, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which when executed by a processor, implements the data fusion method as described in any one of the first aspects above.

[0025] Fifthly, an embodiment of the present application provides a computer program product, which when running on a terminal device, enables the terminal device to execute the data fusion method as described in any one of the first aspects above.

[0026] The beneficial effects of the embodiments of the present application compared with the prior art are as follows:

[0027] A data fusion method provided by an embodiment of the present application includes obtaining initial point cloud data collected by a radar disposed in a tunnel and having a detection direction consistent with the driving direction, and obtaining initial image data collected by an imaging device disposed in the tunnel and having a shooting direction consistent with the driving direction; performing data fusion processing on the initial point cloud data and the initial image data according to timestamps to obtain target data. In practical applications, due to the enclosed and narrow structure in the tunnel, when the detection direction of the radar and the shooting direction of the imaging device are opposite to the driving direction, pollutants brought by the vehicle during driving are likely to affect the sensing effects of the radar and the imaging device. However, in this method, both the detection direction of the radar and the shooting direction of the imaging device are consistent with the driving direction, thereby avoiding the pollutants brought by the vehicle during driving when the detection direction and the shooting direction are opposite to the driving direction, that is, avoiding the pollution brought by oncoming vehicles, improving the sensing effects of the radar and the imaging device, improving the quality of the obtained initial point cloud data and initial image data, thus improving the quality of data fusion, and further improving the detection accuracy of target objects in the data. Description of the Drawings

[0028] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present application, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0029] Figure 1 is a schematic layout diagram of a radar and an imaging device in a tunnel provided by an embodiment of the present application;

[0030] Figure 2 is a flowchart of the implementation of the data fusion method provided by an embodiment of the present application;

[0031] Figure 3 is a flowchart of the implementation of the data fusion method provided by another embodiment of the present application;

[0032] Figure 4 is the implementation flowchart of the data fusion method provided by another embodiment of the present application;

[0033] Figure 5 is the structural schematic diagram of the data fusion device provided by an embodiment of the present application;

[0034] Figure 6 is the structural schematic diagram of the terminal device provided by an embodiment of the present application. Detailed implementation manners

[0035] In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0036] It should be understood that when used in the specification of the present application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0037] It should also be understood that the term " / and" as used in the specification of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0038] As used in the specification of the present application and the appended claims, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" according to the context. Similarly, the phrase "if determined" or "if detecting [the described condition or event]" can be interpreted as meaning "once determined", "in response to determining", "once detecting [the described condition or event]", or "in response to detecting [the described condition or event]" according to the context.

[0039] In addition, in the description of the specification of the present application and the appended claims, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0040] References to "one embodiment" or "some embodiments" or the like described in the specification of this application mean that specific features, structures, or characteristics described in connection with that embodiment are included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0041] In practical applications, the final twin display effect of the intelligent tunnel completely depends on the perception effect of the roadside base station. Therefore, all-weather, long-term stable operation and detection requirements are put forward for roadside perception. Currently, roadside perception usually fuses the point cloud data collected by the radar and the image data collected by the camera to improve the perception accuracy of the target object. The existing layout schemes of the radar and camera devices in the tunnel usually make the detection direction of the radar opposite to the driving direction to obtain the point cloud data of the oncoming vehicle, and make the shooting direction of the camera device opposite to the driving direction to obtain the image data of the oncoming vehicle.

[0042] However, due to the closed and narrow structure of the tunnel, a large amount of pollutants such as dust are easily accumulated. At the same time, the pollutants brought by vehicles during driving will affect the perception effects of the radar with the detection direction opposite to the driving direction and the camera device with the shooting direction opposite to the driving direction. Therefore, in order to avoid the pollution brought by oncoming vehicles, improve the quality of the obtained initial point cloud data and initial image data, thereby improving the quality of data fusion, and further improving the detection accuracy of the target object in the data, the embodiments of this application provide a data fusion method, which is shown in the corresponding embodiments in the following figure and will not be elaborated here.

[0043] Please refer to Figure 1 , Figure 1 which is a schematic layout diagram of the radar and camera devices in the tunnel provided by an embodiment of this application. As Figure 1 shown, A is the radar (only two are shown in the figure), B is the camera device (only two are shown in the figure), the detection direction of the radar is consistent with the driving direction, and the shooting direction of the camera device is consistent with the driving direction.

[0044] It should be noted that one radar corresponds to a group of camera devices, and a group of camera devices includes a short-focus camera and a long-focus camera.

[0045] In the embodiments of the present application, since the detection direction of the radar is consistent with the driving direction, and the shooting direction of the camera device is consistent with the driving direction, the pollution caused by oncoming vehicles is avoided, that is, the pollution to the radar and the camera device is reduced, and the sensing effect of the radar and the camera device is improved.

[0046] Please refer to Figure 2 , Figure 2 which is a flowchart of the implementation of a data fusion method provided by an embodiment of the present application. In the embodiments of the present application, the execution subject of the data fusion method is a terminal device. Among them, the terminal device includes, but is not limited to, devices such as desktop computers, laptops, and computers.

[0047] As Figure 2 shown, the data fusion method provided by an embodiment of the present application may include S101 to S102, which are described in detail as follows:

[0048] In S101, obtain the initial point cloud data collected by a radar disposed in the tunnel and having a detection direction consistent with the driving direction, and obtain the initial image data collected by a camera device disposed in the tunnel and having a shooting direction consistent with the driving direction.

[0049] In the embodiments of the present application, the radar disposed in the tunnel and having a detection direction consistent with the driving direction senses the target object within the detection range in real time, and the camera device disposed in the tunnel and having a shooting direction consistent with the driving direction shoots the target object within the shooting range in real time.

[0050] Based on this, when the terminal device needs to perform data fusion on the radar and the camera device, it can obtain the initial point cloud data collected by the radar disposed in the tunnel and having a detection direction consistent with the driving direction, and obtain the initial image data collected by the camera device disposed in the tunnel and having a shooting direction consistent with the driving direction.

[0051] In an embodiment of the present application, since the length of the tunnel is usually greater than the maximum detection range that a single radar can detect, and the length of the tunnel is also usually greater than the maximum shooting range that a single camera device can shoot, therefore, in order to avoid blind spots that cannot be detected by the radar in the lanes for vehicle driving in the tunnel, the number of radars in the tunnel is multiple, and there is an overlapping part between the detection areas of any two adjacent radars. Each radar corresponds to a camera device, that is to say, the number of camera devices is also multiple.

[0052] Among them, the first distance between any two adjacent radars may be the same or different.

[0053] The above distance can be set according to actual needs and is not limited here.

[0054] It should be noted that the detection area of each radar coincides with the shooting range of the corresponding imaging device.

[0055] In another embodiment of the present application, in order to avoid blind spots where the imaging devices corresponding to each radar in the tunnel cannot take pictures, resulting in the failure to successfully fuse the initial point cloud data collected by each radar with the initial image data collected by its corresponding imaging device, the imaging device may include a short-focus camera for shooting the proximal scene and a long-focus camera for shooting the distal scene.

[0056] In still another embodiment of the present application, in order to avoid blind spots where the short-focus camera and the long-focus camera corresponding to each radar in the tunnel cannot take pictures, there is an overlapping part between the shooting ranges of the short-focus camera and the long-focus camera corresponding to each radar.

[0057] In yet another embodiment of the present application, in order to avoid blind spots where the imaging devices in the tunnel cannot take pictures, there is an overlapping part between the shooting range of the long-focus camera corresponding to each radar and the shooting range of the short-focus camera corresponding to the adjacent radar.

[0058] In yet another embodiment of the present application, in order to avoid the pollution of each radar and each imaging device by pollutants such as vehicle exhaust brought by oncoming vehicles in the tunnel, the positions of each radar and its corresponding imaging device can be set according to the height of the tunnel and the vehicle height restricted by the tunnel.

[0059] In S102, the initial point cloud data and the initial image data are subjected to data fusion processing according to the time stamp to obtain target data.

[0060] In the embodiment of the present application, after the terminal device obtains the above initial point cloud data and the above initial image data, in order to improve the quality of data fusion, the terminal device can perform data fusion processing according to the time stamp to obtain target data.

[0061] In an embodiment of the present application, in order to further improve the quality of data fusion, the terminal device can perform data preprocessing on the initial point cloud data and image preprocessing on the initial image data. Then, the terminal device can perform fusion processing on the initial point cloud data after data preprocessing and the initial image data after image preprocessing at the same moment according to the time stamp to obtain the fused data, that is, target data.

[0062] In practical applications, the methods for performing data fusion processing on the initial point cloud data and the initial image data at the same moment include, but are not limited to: raw data-level fusion, feature-level fusion, and target-level fusion, etc.

[0063] Among them, the raw data - level fusion specifically refers to projecting the initial point - cloud data at the same moment based on the coordinates of each point - cloud point onto the pixel points at the same coordinates in the initial image data, realizing the joint calibration and matching of the initial point - cloud data and the pixel points of the initial image data, that is, realizing data fusion.

[0064] The feature - level fusion specifically refers to projecting the target point - cloud points of the initial point - cloud data at the same moment onto the pixel points at the same coordinates as the target point - cloud points in the initial image data, generating an area of interest around this pixel point, and only searching within this area of interest. After the search, it is matched with the target point - cloud points, thereby realizing data fusion.

[0065] The target - level fusion specifically refers to performing data fusion on the pixel points corresponding to the target objects included in the initial image data and the point - cloud points corresponding to the target objects included in the initial point - cloud data.

[0066] As can be seen from the above, a data - fusion method provided by an embodiment of the present application obtains the initial point - cloud data collected by a radar disposed in the tunnel with a detection direction consistent with the driving direction, and obtains the initial image data collected by an imaging device disposed in the tunnel with a shooting direction consistent with the driving direction; performs data - fusion processing on the initial point - cloud data and the initial image data according to the time stamp to obtain target data. In practical applications, due to the closed and narrow structure in the tunnel, when the detection direction of the radar and the shooting direction of the imaging device are opposite to the driving direction, the pollutants brought by the vehicle during driving are likely to affect the perception effect of the radar and the imaging device. However, in this method, both the detection direction of the radar and the shooting direction of the imaging device are consistent with the driving direction, thus avoiding the pollutants brought by the vehicle during driving when the detection direction and the shooting direction are opposite to the driving direction, that is, avoiding the pollution brought by oncoming vehicles, which also improves the perception effect of the radar and the imaging device, improves the quality of the obtained initial point - cloud data and initial image data, thereby improving the quality of data fusion, and further improving the detection accuracy of the target objects in the data.

[0067] Please refer to Figure 3 , Figure 3 which is the implementation flowchart of the data - fusion method provided by another embodiment of the present application. Compared with Figure 2 the corresponding embodiment, before S101, this embodiment may further include S201 - S202, which are described in detail as follows:

[0068] In S201, determine the environmental information in the tunnel.

[0069] In this embodiment, the environmental information includes but is not limited to: humidity and traffic conditions. Among them, the traffic conditions are used to describe whether the traffic in the tunnel is congested.

[0070] Traffic conditions include, but are not limited to: the first condition, the second condition, and the third condition. Among them, the first condition is used to describe that the traffic in the tunnel is not congested, the second condition is used to describe that the traffic in the tunnel is generally congested, and the third condition is used to describe that the traffic in the tunnel is highly congested.

[0071] Based on this, in one implementation of this embodiment, the terminal device can obtain the humidity in the tunnel in real time through a humidity sensor disposed in the tunnel and wirelessly connected to it.

[0072] In another implementation of this embodiment, the terminal device can determine the traffic conditions in the tunnel in real time through a tunnel monitoring platform wirelessly connected to it.

[0073] In S202, according to the environmental information, the detection angle of the radar and the shooting angle of the imaging device are adjusted respectively.

[0074] In this embodiment, the terminal device can adjust the detection angle of the radar and the shooting angle of the imaging device according to the environmental information to avoid the pollution of the radar and the imaging device by vehicle exhaust gas generated by vehicles in the driving direction.

[0075] Specifically, in combination with S201, when the terminal device detects that the humidity is greater than the set threshold and the traffic condition is the first condition, it means that although the humidity in the tunnel is relatively high at this time, the traffic in the tunnel is not congested. That is to say, the number of vehicles in the tunnel is small, and the pollution brought by the vehicle driving process is small. Therefore, the terminal device does not need to adjust the detection angle of the radar and the shooting angle of the imaging device. Among them, the set threshold can be determined according to actual needs and is not limited here.

[0076] When the terminal device detects that the humidity is greater than the set threshold and the traffic condition is the second condition, it means that the humidity in the tunnel is relatively high at this time, and the traffic in the tunnel is generally congested. That is to say, the number of vehicles in the tunnel is large, and the pollution brought by the vehicle driving process is large. Therefore, the terminal device needs to adjust the detection angle of the radar and the shooting angle of the imaging device.

[0077] When the terminal device detects that the humidity is greater than the set threshold and the traffic condition is the third condition, it means that the humidity in the tunnel is relatively high at this time, and the traffic in the tunnel is highly congested. That is to say, the number of vehicles in the tunnel is very large, and the pollution brought by the vehicle driving process is very large. Therefore, the terminal device needs to adjust the detection angle of the radar and the shooting angle of the imaging device.

[0078] As can be seen from the above, for the data fusion method provided in this embodiment, before obtaining the initial point cloud data collected by the radar disposed in the tunnel and with the detection direction consistent with the driving direction, and the initial image data collected by the imaging device disposed in the tunnel and with the shooting direction consistent with the driving direction, the environmental information in the tunnel can be determined; and according to the environmental information, the detection angle of the radar and the shooting angle of the imaging device are adjusted respectively. The method provided in this embodiment avoids the pollution of the radar and the imaging device by vehicle exhaust gas generated by vehicles in the driving direction, and improves the sensing effect of the radar and the imaging device.

[0079] Please refer to Figure 4 , Figure 4 which is the implementation flowchart of the data fusion method provided in another embodiment of this application. Compared with Figure 2 the corresponding embodiment, in this embodiment, step S102 specifically includes S301 to S304, which are described in detail as follows:

[0080] In S301, obtain the background point cloud data in the same scene as the initial point cloud data, and the background image data in the same scene as the initial image data.

[0081] In an implementation manner of this embodiment, the terminal device can obtain in real time, through a server wirelessly connected to it, the background point cloud data in the same scene as the initial point cloud data, and the background image data in the same scene as the initial image data. Among them, the server can be an electronic device such as a desktop computer or a computer, or a cloud server.

[0082] In some possible embodiments, the terminal device can obtain the historical point cloud data of the first preset number of frames before the initial point cloud data, and process the historical point cloud data of the first preset number of frames to obtain the background point cloud data in the same scene as the initial point cloud data. Among them, the first preset number of frames can be determined according to actual needs and is not limited here. Exemplarily, the first preset number of frames can be 5 frames.

[0083] Specifically, the terminal device can use the Gaussian mixture background modeling algorithm to process the historical point cloud data of the first preset number of frames, so as to obtain the background point cloud data.

[0084] It should be noted that the Gaussian mixture background modeling algorithm can be an existing background modeling based on the Gaussian mixture model.

[0085] In some other possible embodiments, the terminal device may acquire historical image data of a second preset number of frames before the initial image data, and process the historical image data of the second preset number of frames to obtain background image data in the same scene as the initial image data. The second preset number of frames may be determined according to actual needs and is not limited herein. Exemplarily, the second preset number of frames may be 10 frames.

[0086] Specifically, the terminal device may use the Gaussian mixture background modeling algorithm to process the historical image data of the second preset number of frames, thereby obtaining the background image data.

[0087] In S302, the initial point cloud data is processed according to the background point cloud data to obtain target point cloud data.

[0088] In S303, the initial image data is processed according to the background image data to obtain target image data.

[0089] In S304, the target point cloud data and the target image data are fused according to timestamps to obtain the target data.

[0090] In this embodiment, the terminal device may perform background difference processing on the initial point cloud data according to the background point cloud data to obtain a first difference image, that is, the target point cloud data.

[0091] The terminal device may perform background difference processing on the initial image data according to the background image data to obtain a second difference image, that is, the target image data.

[0092] In practical applications, background difference processing specifically refers to extracting the changing area from the sequence data (such as the initial point cloud data and the initial image data) from the background data (such as the background point cloud data and the background image data).

[0093] It should be noted that since the background data in the initial point cloud data and the background data in the initial image data at the same moment are the same, the coordinates of each point cloud point in the target point cloud data correspond one-to-one with the coordinates of each pixel point in the target image data. Therefore, the terminal device can directly fuse the target point cloud data and the target image data to obtain the target data.

[0094] In an embodiment of the present application, in order to improve the data quality of the target data, the terminal device may perform data fusion on the background point cloud data, the background image data, and the target data to obtain the final target data.

[0095] From the above, it can be seen that the data fusion method provided in this embodiment obtains background point cloud data in the same scene as the initial point cloud data, and background image data in the same scene as the initial image data; processes the initial point cloud data according to the background point cloud data to obtain target point cloud data; processes the initial image data according to the background image data to obtain target image data; and fuses the target point cloud data and the target image data according to the timestamp to obtain the target data. The method provided in this embodiment improves the success rate and accuracy of data fusion.

[0096] In one embodiment of the present application, when the tunnel is a bidirectional tunnel, that is, one side of the tunnel (such as the left side or the right side) is used for oncoming vehicles to pass through, and the other side of the tunnel (such as the right side or the left side) is used for outgoing vehicles to pass through, at this time, in order to avoid pollution caused by oncoming vehicles while obtaining point cloud data and image data on either side, radars and camera devices are installed on both sides of the tunnel, and the detection direction of the radar on either side is consistent with the driving direction of that side, and the shooting direction of the camera on either side is consistent with the driving direction of that side.

[0097] In this embodiment, for any side of the tunnel, in order to avoid contamination of the radar and camera device on any side by oncoming vehicles relative to that side (i.e., vehicles on the other side) during driving, the detection angle of the radar on that side and the shooting angle of the camera device can be adjusted so that the detection area of ​​the radar on that side includes as much as possible only the lane area on that side, and the shooting area of ​​the camera device on that side includes as much as possible only the lane area on that side.

[0098] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0099] Corresponding to a data fusion method described in the above embodiment, Figure 5 The structure diagram of a data fusion device provided by an embodiment of the present application is shown. For the convenience of explanation, only the part related to the embodiment of the present application is shown. Figure 5 The data fusion device 500 includes: a first acquisition unit 51 and a first processing unit 52. Wherein:

[0100] The first acquisition unit 51 is used to acquire initial point cloud data collected by a radar installed in the tunnel and having a detection direction consistent with the driving direction, and to acquire initial image data collected by a camera installed in the tunnel and having a shooting direction consistent with the driving direction.

[0101] The first processing unit 52 is configured to perform data fusion processing on the initial point cloud data and the initial image data according to time stamps to obtain target data.

[0102] In an embodiment of the present application, the data fusion device 500 further includes: a determination unit and an adjustment unit. Wherein:

[0103] The determination unit is configured to determine the environmental information in the tunnel.

[0104] The adjustment unit is configured to adjust the detection angle of the radar and the shooting angle of the imaging device respectively according to the environmental information.

[0105] In an embodiment of the present application, the first processing unit 52 specifically includes: a second acquisition unit, a second processing unit, a third processing unit, and a fourth processing unit. Wherein:

[0106] The second acquisition unit is configured to acquire background point cloud data in the same scene as the initial point cloud data, and background image data in the same scene as the initial image data.

[0107] The second processing unit is configured to process the initial point cloud data according to the background point cloud data to obtain target point cloud data.

[0108] The third processing unit is configured to process the initial image data according to the background image data to obtain target image data.

[0109] The fourth processing unit is configured to perform fusion processing on the target point cloud data and the target image data according to time stamps to obtain the target data.

[0110] In an embodiment of the present application, the number of radars is multiple, and there is an overlapping part between the detection areas of any two adjacent radars, and each radar corresponds to one imaging device.

[0111] In an embodiment of the present application, the imaging device corresponding to each radar includes a short-focus camera and a long-focus camera.

[0112] In an embodiment of the present application, there is an overlapping part between the shooting ranges of the short-focus camera and the long-focus camera corresponding to each radar.

[0113] In an embodiment of the present application, there is an overlapping part between the shooting range of the long-focus camera corresponding to each radar and the shooting range of the short-focus camera corresponding to an adjacent radar.

[0114] It should be noted that for the information interaction, execution process, etc. between the above-mentioned devices / units, since they are based on the same concept as the method embodiments of the present application, for their specific functions and the technical effects brought about, reference can be specifically made to the method embodiment part, and details will not be elaborated here.

[0115] Those skilled in the art can clearly understand that for the convenience and conciseness of description, only the above-mentioned division of each functional unit and module is used as an example for illustration. In actual applications, the above-mentioned functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments, and details will not be elaborated here.

[0116] Figure 6 This is a schematic structural diagram of a terminal device provided by an embodiment of the present application. As Figure 6 shown, the terminal device 6 in this embodiment includes: at least one processor 60 ( Figure 6 only one is shown in the figure), a processor, a memory 61, and a computer program 62 stored in the memory 61 and executable on the at least one processor 60. When the processor 60 executes the computer program 62, the steps in any of the above-mentioned data fusion method embodiments are implemented.

[0117] The terminal device may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art can understand that Figure 6 this is only an example of the terminal device 6 and does not constitute a limitation on the terminal device 6. It may include more or fewer components than shown in the figure, or combine some components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0118] The so-called processor 60 may be a Central Processing Unit (CPU), and the processor 60 may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0119] In some embodiments, the memory 61 may be an internal storage unit of the terminal device 6, such as the memory of the terminal device 6. In other embodiments, the memory 61 may also be an external storage device of the terminal device 6, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the terminal device 6. Further, the memory 61 may also include both the internal storage unit and the external storage device of the terminal device 6. The memory 61 is used to store an operating system, application programs, a BootLoader, data, and other programs, such as the program code of the computer program, etc. The memory 61 may also be used to temporarily store data that has been output or is to be output.

[0120] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above various method embodiments can be implemented.

[0121] The embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above various method embodiments when executed.

[0122] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-described embodiment methods of this application, a computer program can be used to instruct relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, an executable file, or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to a terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk, or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0123] In the above embodiments, the descriptions of the various embodiments each have their own emphases. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0124] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A data fusion method, characterized in that, Including: Obtaining initial point cloud data collected by a radar disposed in the tunnel and having a detection direction consistent with the driving direction, and obtaining initial image data collected by an imaging device disposed in the tunnel and having a shooting direction consistent with the driving direction; Performing data fusion processing on the initial point cloud data and the initial image data according to timestamps to obtain target data.

2. The data fusion method according to claim 1, characterized in that Before obtaining the initial point cloud data collected by the radar disposed in the tunnel and having a detection direction consistent with the driving direction, and obtaining the initial image data collected by the imaging device disposed in the tunnel and having a shooting direction consistent with the driving direction, it further includes: Determining the environmental information in the tunnel; Adjusting the detection angle of the radar and the shooting angle of the imaging device respectively according to the environmental information.

3. The data fusion method according to claim 1, wherein The performing data fusion processing on the initial point cloud data and the initial image data according to timestamps to obtain target data includes: Obtaining background point cloud data in the same scene as the initial point cloud data, and background image data in the same scene as the initial image data; Processing the initial point cloud data according to the background point cloud data to obtain target point cloud data; Processing the initial image data according to the background image data to obtain target image data; Performing fusion processing on the target point cloud data and the target image data according to timestamps to obtain the target data.

4. The data fusion method according to any one of claims 1-3, characterized in that The number of the radars is multiple, there is an overlapping part between the detection areas of any two adjacent radars, and each radar corresponds to one imaging device.

5. The data fusion method according to claim 4, wherein The imaging device corresponding to each radar includes a short-focus camera and a long-focus camera.

6. The data fusion method according to claim 5, wherein There is an overlapping part between the shooting range of the short-focus camera corresponding to each radar and the shooting range of the long-focus camera.

7. The data fusion method according to claim 5, wherein There is an overlapping part between the shooting range of the long-focus camera corresponding to each radar and the shooting range of the short-focus camera corresponding to an adjacent radar.

8. A data fusion device, characterized in that, Including: A first obtaining unit, configured to obtain initial point cloud data collected by a radar disposed in the tunnel and having a detection direction consistent with the driving direction, and obtain initial image data collected by an imaging device disposed in the tunnel and having a shooting direction consistent with the driving direction; A first processing unit, configured to perform data fusion processing on the initial point cloud data and the initial image data according to timestamps to obtain target data.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data fusion method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the data fusion method according to any one of claims 1 to 7.

Images