Data fusion method and device, electronic equipment and storage medium

Abstract

The invention provides a data fusion method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining first sensor data and a plurality of second sensor data, the first sensor data comprising position data of three-dimensional space coordinate points, and the second sensor data being discrete data; constructing a space map according to the first sensor data, and projecting the space map on a horizontal plane to obtain a horizontal map; projecting two-dimensional data corresponding to each piece of second sensor data onto the horizontal map to obtain a plurality of horizontal data maps; and splicing the first feature maps corresponding to the horizontal data maps to generate a horizontal comprehensive feature map, and converting the sensor data containing discrete data into horizontal plane data for data fusion, thereby realizing more comprehensive and higher-precision data fusion.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data fusion method, device, electronic equipment and storage medium. Background Art

[0002] In the automatic control scenario, multi-sensor data fusion is a key technology. It obtains new data by fusing the data collected by different sensors. The new data can include richer information, thereby improving control accuracy.

[0003] For example, reconnaissance robots used in closed scenes such as factories are usually equipped with multiple sensors, such as infrared thermal imagers, visible light cameras, gas sensors, lidars, and various environmental monitoring equipment to obtain key data such as temperature distribution, smoke concentration, fire source location, obstacle information, and topography. By integrating the data of multiple sensors on the robot, it can help remote workers to grasp the on-site situation in a timely manner and improve the robot's perception ability and decision-making accuracy in complex and dangerous fire environments.

[0004] However, the fusion method of multi-sensor data in the prior art is still to fuse or filter at the decision layer, which often separates more interacting detailed information and affects the final detection result. Summary of the invention

[0005] The present invention provides a data fusion method, device, electronic equipment and storage medium to achieve more comprehensive and higher precision data fusion.

[0006] The present invention provides a data fusion method, comprising the following steps: acquiring first sensor data and multiple second sensor data, wherein the first sensor data includes position data of three-dimensional spatial coordinate points, and the second sensor data is discrete data; constructing a spatial map based on the first sensor data, and projecting the spatial map onto a horizontal plane to obtain a horizontal map; projecting the two-dimensional data corresponding to each of the second sensor data onto the horizontal map to obtain multiple horizontal data maps; and splicing the first feature map corresponding to each of the horizontal data maps to generate a horizontal comprehensive feature map.

[0007] According to a data fusion method provided by the present invention, the second sensor data includes gas data, and the two-dimensional data corresponding to each of the second sensor data is projected onto the horizontal map to obtain multiple horizontal data maps, including: updating the gas diffusion model according to the gas data to obtain multiple gas concentration data, and the gas concentration data is the two-dimensional data; each of the gas concentration data is projected onto the horizontal map to obtain multiple gas concentration maps, and the gas concentration map is the horizontal data map.

[0008] According to a data fusion method provided by the present invention, the first feature map corresponding to each of the horizontal data maps is spliced ​​to generate a horizontal comprehensive feature map, including: inputting each of the horizontal data maps into a time series neural network to obtain multiple horizontal feature maps; based on the weight of the horizontal feature map, the horizontal feature map is weightedly fused with the historical feature map to obtain multiple first feature maps, wherein the historical feature map is a feature map of the sensor corresponding to the horizontal feature map at the previous moment; and the multiple first feature maps are spliced ​​to obtain the horizontal comprehensive feature map.

[0009] According to a data fusion method provided by the present invention, the method further includes: setting the weight of each of the horizontal feature maps according to user input information.

[0010] According to a data fusion method provided by the present invention, the method also includes: acquiring at least two third sensor data, the third sensor data including two-dimensional image data; projecting the spatial map onto a vertical plane to obtain a vertical map; performing feature extraction on each of the third sensor data to obtain at least two second feature maps; performing weighted fusion on the second feature maps at the same position on the vertical map and the third feature map corresponding to the first sensor data to obtain a vertical comprehensive feature map.

[0011] According to a data fusion method provided by the present invention, the second feature maps at the same position on the vertical map and the third feature map corresponding to the first sensor are weightedly fused to obtain a vertical comprehensive feature map, including: calculating the weight of each pixel in each third sensor data to obtain a weight set of each third sensor data; using the weight set of each third sensor data as the weight set of the second feature map corresponding to the third sensor data to obtain a weight set of each second feature map; performing feature extraction on the first sensor data to obtain a third feature map; calculating the weight of each pixel in the first sensor data to obtain a weight set of the first sensor data; using the weight set of the first sensor data as the weight set of the third feature map; and weighted fusion of the second feature maps at the same position on the vertical map and the third feature map based on the weight sets of all second feature maps and the weight set of the third feature map to obtain the vertical comprehensive feature map.

[0012] The present invention also provides a data fusion device, comprising the following modules: an acquisition module, used to acquire first sensor data and multiple second sensor data, the first sensor data including position data of three-dimensional spatial coordinate points, and the second sensor data being discrete data; a generation module, used to construct a spatial map based on the first sensor data, and project the spatial map onto a horizontal plane to obtain a horizontal map; a projection module, used to project the two-dimensional data corresponding to each of the second sensor data onto the horizontal map to obtain multiple horizontal data maps; a splicing module, used to splice the first feature map corresponding to each of the horizontal data maps to generate a horizontal comprehensive feature map.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned data fusion methods when executing the computer program.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the data fusion method described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned data fusion methods.

[0016] The data fusion method, device, electronic device and storage medium provided by the present invention project the second sensor data into the first sensor data to achieve fusion of the two, thereby converting discrete sensor data into data parallel to the ground, which can better extract the characteristics of discrete data and facilitate matching of different discrete data to achieve fusion of different discrete sensor data. Therefore, the data fusion method of the present application achieves more comprehensive and higher-precision data fusion by converting sensor data containing discrete data into horizontal plane data for data fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of a data fusion method provided by the present invention.

[0019] Figure 2 It is a flow chart of another data fusion method provided by the present invention.

[0020] Figure 3 It is a schematic diagram of the overall flow of a data fusion method provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the data fusion device provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Multi-sensor data fusion is widely used in various scenarios. For example, in relatively closed scenarios such as factories, in order to better detect the safety status of the factory and improve the accuracy of autonomous detection, it is necessary to collect data from multiple sensors such as visible cameras, infrared cameras, lidar, gas sensors, etc., and fuse multi-source heterogeneous data to facilitate the subsequent use of multi-source data for comprehensive reconnaissance of the factory environment.

[0025] The data fusion methods in the prior art mainly include the following two methods: One method is to fuse or filter at the decision layer, but this method often separates more interacting details, which affects the final detection results.

[0026] Another method is to extract features from the data collected by each sensor, map the feature maps of other sensors to the corresponding area based on the feature map of one type of sensor, and assign weights to different feature maps to finally obtain a fused feature map. However, when this method maps feature maps of different sensors to the same space, it is necessary to ensure the accuracy and consistency of the mapping, especially when there are large differences in viewing angles, resolution mismatches, or inconsistent perception ranges between sensors, which may cause information distortion or loss.

[0027] In view of this, an embodiment of the present invention provides a data fusion method, which projects the second sensor data into the first sensor data to achieve fusion of the two, thereby converting discrete sensor data into data parallel to the ground, which can better extract the characteristics of discrete data and facilitate matching different discrete data to achieve fusion of different discrete sensor data. Therefore, the data fusion method of the present application achieves more comprehensive and higher-precision data fusion by converting sensor data containing discrete data into horizontal plane data for data fusion.

[0028] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0029] Figure 1 The present invention provides a flow chart of a data fusion method, which can be applied to electronic devices, which can be various types of devices with information processing capabilities during implementation. For example, the electronic device can include a personal computer, a laptop, a PDA or a server, etc.; the electronic device can also be a mobile terminal, for example, the mobile terminal can include a mobile phone, a car computer, a tablet computer or a projector, etc.

[0030] like Figure 1 As shown, the method may include the following steps 110 to 160: Step 110: Acquire first sensor data and a plurality of second sensor data, wherein the first sensor data includes position data of three-dimensional space coordinate points, and the second sensor data is discrete data.

[0031] The data measured by each sensor for the detection space is obtained to obtain the data of multiple sensors. The detection space refers to the space measured by the sensor at the current moment. For example, a target device is pre-set, and the target device can be a sweeping robot, a detection robot, etc. The target device is provided with multiple sensors, such as various types of gas sensors, lidar, visible light cameras, infrared cameras, etc. The target device will continue to move along the planned path until it completes the scanning of the space within the user-specified range to achieve three-dimensional map construction. The target device will use its own sensors to measure at the measurement points on the planned path to generate sensor data at the current moment. At this time, the detection space is the space within the measurement range of the target device. Obtain these sensor data to obtain multiple sensor data.

[0032] In other examples of the present application, the detection space may also be a designated space, for example, the detection space is a room, and a plurality of sensors are arranged in the room. Under preset conditions such as a designated time, the sensors perform measurements to obtain a plurality of sensor data.

[0033] The acquired data includes multiple first sensor data and at least one second sensor data. The first sensor data refers to the data measured by the sensor, including the position data of the three-dimensional space coordinate points, such as the point cloud data measured by the laser radar. The second sensor data refers to the data measured by the sensor is discrete, such as the temperature data measured by the temperature sensor, the gas sensor data, etc. Discrete means that the data is independent and limited. For example, the gas sensor usually only performs one measurement at the current position to obtain one data point. Even if the gas is measured multiple times, the data obtained is limited and independent.

[0034] Step 120: construct a spatial map based on the first sensor data, and project the spatial map onto a horizontal plane to obtain a horizontal map.

[0035] The map data of the current space can be constructed through the data of the first sensor to obtain the three-dimensional space map data. For example, the sensor includes lidar data, and the point cloud data collected by the lidar is SLAM (simultaneous positioning and mapping) to synchronously locate and generate maps.

[0036] The spatial map of the detection space is projected onto a horizontal plane to generate a horizontal map.

[0037] Step 130: Project the two-dimensional data corresponding to each of the second sensor data onto the horizontal map to obtain a plurality of horizontal data maps.

[0038] For each first data, the corresponding two-dimensional data is mapped onto the horizontal map to generate a corresponding horizontal data map.

[0039] This application maps discrete sensors such as gas sensors, temperature sensors, motion sensors, proximity sensors, etc. to a horizontal map, thereby spreading them into two-dimensional data parallel to the ground. For example, the data of the temperature sensor is mapped to the position of the temperature sensor on the horizontal map to obtain the temperature data of the position point, and an algorithm such as interpolation is used to calculate the temperature data of other positions based on the existing temperature data, thereby obtaining a temperature distribution map, which is a horizontal data map.

[0040] Converting discrete sensor data into data parallel to the ground can not only better extract the features of these data, but also obtain the correspondence between different discrete sensor data, thereby realizing data fusion.

[0041] In one embodiment of the present application, the second sensor data includes gas data, and the two-dimensional data corresponding to each second sensor data is projected onto the horizontal map to obtain multiple horizontal data maps, including: updating the gas diffusion model according to the gas data to obtain multiple gas concentration data, and the gas concentration data is the two-dimensional data; each of the gas concentration data is projected onto the horizontal map to obtain multiple gas concentration maps, and the gas concentration map is the horizontal data map.

[0042] The second sensor data may be data of a gas sensor, and one second sensor data represents data of a gas sensor. For example, the plurality of second sensor data include: data of a carbon dioxide sensor, data of a dust concentration sensor, and data of a nitric oxide sensor.

[0043] When the second sensor data is gas sensor data, it can be expanded into two-dimensional data through the gas diffusion model: For each gas data, the gas diffusion model is updated for multiple groups of gas data collected by the gas sensor based on the generated map to obtain gas concentration data, which is two-dimensional data of the gas data. Among them, the gas diffusion model can be constructed using methods such as the Gaussian model, and the present application does not limit the method for constructing the gas diffusion model. For example, a gas diffusion model is pre-modeled based on gas properties, wind force changes, etc., and the gas data at the current moment is brought into the model for calculation to realize the update of the gas diffusion model.

[0044] For each gas concentration, it is mapped onto the horizontal map to generate a gas concentration map, which is the horizontal data map.

[0045] In the above embodiment, a method for rapidly and accurately expanding the data of the gas sensor into two-dimensional data parallel to the ground is provided.

[0046] Step 140: splicing the first feature maps corresponding to each of the horizontal data maps to generate a horizontal comprehensive feature map.

[0047] For each horizontal data map, its feature map is obtained to obtain a first feature map of the horizontal data map.

[0048] Each first feature map is mapped onto the horizontal map to obtain the corresponding relationship between each first feature map and the horizontal map. Based on the corresponding relationship between each first feature map and the horizontal map, each first feature map corresponding to each area on the horizontal map is obtained. For each area on the horizontal map, all first feature maps corresponding to the area are spliced ​​together to obtain a horizontal comprehensive feature map of the area. The horizontal comprehensive feature maps corresponding to all areas on the horizontal map constitute the final output horizontal comprehensive feature map.

[0049] The present application does not limit the splicing of the first feature maps, which can be splicing in the depth dimension, that is, each first feature map is used as a channel of the horizontal comprehensive feature map, or it can be splicing in the width dimension, that is, each first feature map is connected in width.

[0050] The above-mentioned data fusion method projects the second sensor data into the first sensor data to achieve the fusion of the two, thereby converting the discrete sensor data into data parallel to the ground, which can better extract the characteristics of the discrete data and facilitate the matching of different discrete data to achieve the fusion of different discrete sensor data. Therefore, the data fusion method of the present application realizes more comprehensive and higher-precision data fusion by converting the sensor data containing discrete data into horizontal plane data for data fusion.

[0051] In one embodiment of the present application, the first feature maps corresponding to each of the horizontal data maps are spliced ​​to generate a horizontal comprehensive feature map, including: inputting each of the horizontal data maps into a time series neural network to obtain multiple horizontal feature maps; based on the weight of the horizontal feature map, weightedly fusing the horizontal feature map with the historical feature map to obtain multiple first feature maps, wherein the historical feature map is a feature map of the sensor corresponding to the horizontal feature map at the previous moment; and splicing the multiple first feature maps to obtain the horizontal comprehensive feature map.

[0052] Since data parallel to the ground often have a more obvious relationship of change over time than data perpendicular to the ground, time-series-related neural networks can be used to extract features, thereby extracting richer features.

[0053] Therefore, for each horizontal data map, a first neural network is used to extract features thereof, thereby generating a plurality of first feature maps. The first neural network is a time-series-related neural network such as a recurrent neural network or a long short-term memory network.

[0054] For example, Figure 2 As shown, Figure 2 It is a flowchart of another data fusion method. Figure 2 For the data of the gas sensor, the gas diffusion model is updated using the data at the current moment. The updated gas diffusion model is mapped to the horizontal map of the real-time map to generate a horizontal data map.

[0055] For each horizontal data map, it is input into the time-series related neural network, and the neural network will extract the feature map of the horizontal data map.

[0056] Get the weight of the horizontal feature map, Figure 2 For each horizontal data map, in addition to the time-series-related neural network, it is also input into the weight algorithm. The weight algorithm will calculate the weight of each pixel.

[0057] In one embodiment of the present application, the method further includes: setting the weight of each of the horizontal feature maps according to user input information.

[0058] After the horizontal feature map is generated, the user input is requested to obtain the weight value set by the user for the horizontal feature map in the user input information, and the weight value is used as the weight of the horizontal feature map.

[0059] For each horizontal feature map, obtain the historical feature map corresponding to the feature map. The historical feature map is a feature map generated by the sensor corresponding to the horizontal feature map when acquiring data at the previous moment. For example, the historical feature map is a horizontal feature map obtained by updating the gas diffusion model based on the gas data at the previous moment to generate a horizontal data map.

[0060] Or in another example, a horizontal feature map for CO2 sensor data using Figure 2 The characteristic graph generated by the process shown in the figure is based on the data obtained by the carbon dioxide sensor at the last moment, and uses Figure 2 The feature map generated by the process shown in the figure. Similarly, the weighted collection of the historical feature map is also obtained by the weight algorithm of the horizontal data map corresponding to the historical feature map.

[0061] For each horizontal feature map, weighted fusion is performed according to the weight corresponding to the feature map, the historical feature map corresponding to the feature map, and the weight of the historical feature map to obtain a first feature map.

[0062] Preferably, mapping can be performed based on the horizontal data map corresponding to the horizontal feature map and the horizontal data map corresponding to the historical feature map to obtain the same area in the two maps. For each same area, the weight of the horizontal feature map, the horizontal feature map, the weight of the historical feature map and the historical feature map are used for weighted fusion to generate a first feature map.

[0063] By concatenating multiple first feature maps, a horizontal comprehensive feature map is obtained.

[0064] In the above embodiment, the characteristics of the data parallel to the ground can be better obtained through the temporal neural network, and the weight of each pixel during fusion is obtained through the image segmentation algorithm, thereby achieving pixel-level fusion, and the generated fusion data is more comprehensive and accurate.

[0065] In other embodiments of the present application, a weight algorithm calculates the weight of each horizontal data map, including: using an image segmentation algorithm to calculate the segmentation confidence of each pixel in each horizontal data map, and the segmentation confidence is the weight.

[0066] The weight algorithm is an image segmentation algorithm such as a region growing image segmentation algorithm and a fast Fourier image segmentation algorithm. These algorithms calculate the confidence that each pixel in the image belongs to the segmented region, and use the confidence of the pixel as the pixel weight. After the horizontal data map is processed using the weight algorithm, the weight of each pixel in the horizontal data map will be obtained. In this application, the weight of each pixel in the horizontal data map is referred to as the weight set of the horizontal data map.

[0067] For each set of weights of the horizontal data map, it is used as the set of weights of the horizontal feature map corresponding to the map. For example, if the size of the horizontal data map and the horizontal feature map are the same, the pixels of the two maps are matched one by one according to the pixel coordinates, and the weights of the pixels in the horizontal data map are used as the weights of the corresponding pixels in the horizontal feature map. In other embodiments of the present application, if the size of the horizontal data map and the horizontal feature map are different, the horizontal data map can be upsampled or downsampled, and the weights are generated or fused accordingly. The sampled horizontal data map and the horizontal feature map are matched one by one according to the pixel coordinates, and the weights of the pixels in the horizontal data map are used as the weights of the corresponding pixels in the horizontal feature map.

[0068] In one embodiment of the present application, the method also includes: obtaining at least two third sensor data, the third sensor data including two-dimensional image data; projecting the spatial map onto a vertical plane to obtain a vertical map; performing feature extraction on each of the third sensor data to obtain at least two second feature maps; weighted fusion of the second feature maps at the same position on the vertical map and the third feature map corresponding to the first sensor data to obtain a vertical comprehensive feature map.

[0069] A plurality of third sensor data are acquired, where data of one third sensor refers to data that is two-dimensional image data, such as data from a visible light camera, data from an infrared imager, and the like.

[0070] For each third sensor data, it is input into the second neural network for feature extraction to obtain the second feature. The second neural network can be various types of feature extraction networks, such as convolutional neural networks, fully connected neural networks, etc.

[0071] A vertical map generated by projecting a three-dimensional spatial map onto a vertical plane.

[0072] For each second feature map, the feature map is mapped to the vertical map to obtain a mapping relationship between the feature map and the vertical map. For example, the second feature map can be mapped to the vertical map based on a feature matching algorithm. The second data is a measurement of an object or space, and the second feature map extracted from it also includes the features of the object or space. The vertical map also includes the same space or object. Therefore, the same object has different representations in the second feature map and the vertical map, and this feature can be used to perform feature matching between the two.

[0073] A third feature map corresponding to the first sensor data is obtained, and the feature map is mapped onto the vertical map.

[0074] For vertical Figure 1 For each position, all second feature maps corresponding to the position and the third feature map corresponding to the first sensor data can be obtained. These feature maps are weighted fused to obtain the vertical comprehensive feature map of the position. After weighted fusion of all positions on the vertical map, the final output vertical comprehensive feature map is obtained.

[0075] In the above embodiment, the feature maps are matched based on the vertical map, which avoids the situation where different sensor data cannot be matched due to different viewing angles or positions, and achieves data fusion with higher accuracy.

[0076] like Figure 2As shown, the sensor data includes laser radar data. For laser radar data, the point cloud data measured is a set of vectors in a three-dimensional coordinate system. The data is recorded in the form of points, and each point contains three-dimensional coordinates. Point cloud data can be used as the first sensor data. At the same time, point cloud data can also be converted into a two-dimensional image and participate in the generation of vertical feature maps.

[0077] In one embodiment of the present application, the weighted fusion of the second feature maps at the same position on the vertical map to obtain a vertical comprehensive feature map includes: calculating the weight of each pixel in each of the third sensor data to obtain a weight set of each of the third sensor data; using the weight set of each of the third sensor data as the weight set of the second feature map corresponding to the third sensor data to obtain a weight set of each second feature map; performing feature extraction on the first sensor data to obtain a third feature map; calculating the weight of each pixel of the first sensor data to obtain a weight set of the first sensor data; using the weight set of the first sensor data as the weight set of the third feature map; and weighted fusion of the second feature maps and the third feature map at the same position on the vertical map based on the weight sets of all the second feature maps and the weight set of the third feature map to obtain the vertical comprehensive feature map.

[0078] For the LiDAR data, it is input into the convolutional neural network for feature extraction to obtain a feature map. The feature map is used for image segmentation. During the segmentation process, the confidence that each pixel belongs to the segmented area is calculated, and this confidence is used as the weight of the pixel. After all the pixels on the LiDAR data are processed, the weight set of the LiDAR data is obtained. The weight set of the LiDAR data is used as the weight set of the LiDAR feature map.

[0079] The weight algorithm in the above embodiment includes two steps of feature extraction and segmentation, that is, the feature extraction algorithm performs feature extraction and weight calculation. In other embodiments of the present application, the laser radar data distribution can also be input into the feature extraction algorithm and the weight algorithm to obtain the feature map and the weight set respectively.

[0080] The data from the visible light camera and the infrared imager are respectively input into the convolutional neural network for feature extraction to obtain two feature maps. The feature maps are then used for image segmentation. During the segmentation process, the confidence that each pixel belongs to the segmented area is calculated and used as the weight of the pixel.

[0081] For each feature map, feature mapping is performed and mapped to a vertical map. Figure 1A position can be obtained by obtaining all the second feature maps corresponding to the position. The second feature maps are weighted fused according to the weight set of these second feature maps to obtain the vertical comprehensive feature map of the position. After weighted fusion of all positions on the vertical map, the final output vertical comprehensive feature map is obtained.

[0082] Figure 2 In the process, the feature map of the laser radar can also be used for target recognition, thereby realizing the classification of the target. The target area of ​​the specified target is obtained from it according to the classification result. Similarly, the part of each feature map belonging to the specified target is mapped to the specified target in the vertical map.

[0083] The following describes an exemplary application of an embodiment of the present invention in a practical application scenario.

[0084] Example robot configuration: Install visible light camera, gas sensor and lidar. Real-time map generation: First, use the point cloud data collected by the lidar to perform slam, and simultaneously locate and generate maps.

[0085] Figure 3 FIG. 1 is a schematic diagram of the overall flow of a data fusion method provided by the present invention. Figure 3 As shown, the method includes the following steps.

[0086] For radar point cloud data: Step 3031: Obtain radar point cloud data.

[0087] Step 3012: synchronously locate and map the radar point cloud data to obtain a real-time map; Step 3013: extracting features from the radar point cloud data to obtain three-dimensional target data; Step 3014: Map the three-dimensional target data to a visible light image to obtain vertical point cloud features.

[0088] For gas concentration data: Step 3021: Obtain gas concentration data; Step 3022: Generate a gas diffusion model, process the gas concentration, and obtain a three-dimensional gas concentration matrix; Step 3023: Map the three-dimensional gas concentration matrix to the real-time map to obtain a horizontal gas concentration map; Step 3024: extracting time series features from the horizontal gas concentration graph to obtain a first feature graph; Step 3025: Map the first feature map to a real-time map to obtain a horizontal comprehensive feature map.

[0089] For visible light images: Step 3031: Acquire visible light image data; Step 3032: extracting features from the visible light image data to obtain a second feature map; Step 3033: performing feature mapping on the second feature map to obtain a vertical image feature map; Step 3034: Combine and splice the vertical image feature map with the vertical point cloud feature map to obtain a vertical comprehensive feature map.

[0090] Specifically, the steps for obtaining the horizontal comprehensive feature map include: simultaneously modeling and updating the gas diffusion model for multiple sets of gas data collected by the gas sensor based on the generated map, projecting the inferred gas concentration onto the map to generate a concentration map, sending it to a time-series related neural network (such as a recurrent neural network) to obtain a horizontal feature map, and generating a horizontal comprehensive feature map based on the confidence calculated by the model.

[0091] Specifically, the steps of obtaining the vertical comprehensive feature map include: extracting features from the visible light camera data to generate a feature map. Performing three-dimensional target detection after extracting features from the lidar point cloud data, and then mapping it to the coordinate space corresponding to the visible light camera feature map. Based on the confidence of the feature map calculated by the model, the two sets of feature maps are combined to generate a vertical comprehensive feature map.

[0092] The data fusion method provided by the present invention divides data based on the sensor perspective, and expands the discrete sensor data that needs to be processed, such as gas sensors, into two-dimensional data parallel to the ground using a gas diffusion model. Since the data parallel to the ground often has a more obvious relationship of change over time than the data perpendicular to the ground, a time-related neural network can be used to extract features. However, the data perpendicular to the ground often cannot match each other well due to the movement of the robot and the change of perspective, so data fusion is performed based on the three-dimensional relationship generated by the laser radar. The method achieves more comprehensive and higher-precision data fusion.

[0093] Based on the foregoing embodiments, an embodiment of the present invention provides a data fusion device, and each module included in the device and each unit included in each module can be implemented by a processor; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0094] The data fusion device provided by the present invention is described below. The data fusion device described below and the data fusion method described above can be referred to each other.

[0095] Figure 4 Schematic diagram of the structure of the data fusion device provided by the embodiment of the present invention. Figure 3As shown, the device 400 includes an acquisition module 410, a generation module 420, a projection module 430 and a splicing module 440, wherein: An acquisition module 410 is used to acquire first sensor data and a plurality of second sensor data, wherein the first sensor data includes position data of a three-dimensional space coordinate point, and the second sensor data is discrete data; A generating module 420, configured to construct a spatial map according to the first sensor data, and project the spatial map onto a horizontal plane to obtain a horizontal map; A projection module 430, configured to project the two-dimensional data corresponding to each of the second sensor data onto the horizontal map to obtain a plurality of horizontal data maps; A splicing module 440 is used for the splicing module, and is used to splice the first feature map corresponding to each of the horizontal data maps to generate a horizontal comprehensive feature map; In one embodiment of the present application, the second sensor data includes gas data, and the projection 430 is specifically used to update the gas diffusion model according to the gas data to obtain a plurality of gas concentration data, and the gas concentration data is the two-dimensional data; each of the gas concentration data is projected onto the horizontal map to obtain a plurality of gas concentration maps, and the gas concentration map is the horizontal data map.

[0096] In one embodiment of the present application, the splicing module 440 is specifically used to input each horizontal data map into a time series neural network to obtain multiple horizontal feature maps; based on the weight of the horizontal feature map, the horizontal feature map is weightedly fused with the historical feature map to obtain multiple first feature maps, and the historical feature map is the feature map of the sensor corresponding to the horizontal feature map at the previous moment; the multiple first feature maps are spliced ​​to obtain the horizontal comprehensive feature map.

[0097] In one embodiment of the present application, the stitching module 440 is further configured to set the weight of each of the horizontal feature maps according to user input information.

[0098] In one embodiment of the present application, the device also includes: a vertical feature map generation module, the vertical feature map generation module is used to obtain at least two third sensor data, the third sensor data includes two-dimensional image data; project the spatial map onto a vertical plane to obtain a vertical map; perform feature extraction on each of the third sensor data to obtain at least two second feature maps; perform weighted fusion on the second feature maps at the same position on the vertical map and the third feature map corresponding to the first sensor data to obtain a vertical comprehensive feature map.

[0099] In one embodiment of the present application, the vertical feature map generation module is specifically used to calculate the weight of each pixel in each of the third sensor data to obtain a weight set of each of the third sensor data; use the weight set of each of the third sensor data as the weight set of the second feature map corresponding to the third sensor data to obtain a weight set of each second feature map; perform feature extraction on the first sensor data to obtain a third feature map; calculate the weight of each pixel of the first sensor data to obtain a weight set of the first sensor data; use the weight set of the first sensor data as the weight set of the third feature map; based on the weight sets of all the second feature maps and the weight set of the third feature map, perform weighted fusion on the second feature maps and the third feature maps at the same position on the vertical map to obtain the vertical comprehensive feature map.

[0100] Figure 5 A schematic diagram of the physical structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the above data fusion method.

[0101] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0102] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data fusion method provided by the above methods.

[0103] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0104] In yet another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the data fusion method provided by the above methods.

[0105] The above-mentioned computer-readable storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0106] Computer-readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0107] The program code embodied on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.

[0108] Computer program code for performing the operations of the present specification may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0109] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0110] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data fusion method, characterized in that: include: Acquire first sensor data and a plurality of second sensor data, wherein the first sensor data includes position data of a three-dimensional space coordinate point, and the second sensor data is discrete data; constructing a spatial map according to the first sensor data, and projecting the spatial map onto a horizontal plane to obtain a horizontal map; Projecting the two-dimensional data corresponding to each of the second sensor data onto the horizontal map to obtain a plurality of horizontal data maps; The first feature maps corresponding to each of the horizontal data maps are spliced ​​to generate a horizontal comprehensive feature map.

2. The method according to claim 1, characterized in that The second sensor data includes gas data, and the two-dimensional data corresponding to each of the second sensor data is projected onto the horizontal map to obtain a plurality of horizontal data maps, including: Update the gas diffusion model according to the gas data to obtain a plurality of gas concentration data, wherein the gas concentration data is the two-dimensional data; Each of the gas concentration data is projected onto the horizontal map to obtain a plurality of gas concentration maps, where the gas concentration map is the horizontal data map.

3. The method according to claim 2, characterized in that The step of splicing the first feature maps corresponding to each of the horizontal data maps to generate a horizontal comprehensive feature map includes: Input each of the horizontal data maps into a temporal neural network to obtain a plurality of horizontal feature maps; According to the weight of the horizontal feature map, the horizontal feature map and the historical feature map are weightedly fused to obtain a plurality of first feature maps, wherein the historical feature map is a feature map of the sensor corresponding to the horizontal feature map at a previous moment; The multiple first feature maps are concatenated to obtain the horizontal comprehensive feature map.

4. The method according to claim 1, characterized in that: The method further comprises: The weight of each of the horizontal feature maps is set according to user input information.

5. The method according to claim 1, characterized in that The method further comprises: acquiring at least two third sensor data, wherein the third sensor data comprises two-dimensional image data; Projecting the spatial map onto a vertical plane to obtain a vertical map; Performing feature extraction on each of the third sensor data to obtain at least two second feature maps; The second feature maps at the same position on the vertical map and the third feature maps corresponding to the first sensor data are weightedly fused to obtain a vertical comprehensive feature map.

6. The method according to claim 5, characterized in that The step of weightedly fusing the second feature maps at the same position on the vertical map and the third feature map corresponding to the first sensor to obtain a vertical comprehensive feature map includes: Calculating the weight of each pixel in each of the third sensor data to obtain a weight set of each of the third sensor data; Using the weight set of each of the third sensor data as the weight set of the second feature map corresponding to the third sensor data, to obtain the weight set of each second feature map; Performing feature extraction on the first sensor data to obtain a third feature map; Calculating the weight of each pixel of the first sensor data to obtain a weight set of the first sensor data; Using the weight set of the first sensor data as the weight set of the third feature map; According to the weight set of all the second feature maps and the weight set of the third feature map, the second feature maps and the third feature maps at the same position on the vertical map are weightedly fused to obtain the vertical comprehensive feature map.

7. A data fusion device, characterized in that: include: An acquisition module, used to acquire first sensor data and a plurality of second sensor data, wherein the first sensor data includes position data of a three-dimensional space coordinate point, and the second sensor data is discrete data; A generating module, configured to construct a spatial map according to the first sensor data, and project the spatial map onto a horizontal plane to obtain a horizontal map; A projection module, used for projecting the two-dimensional data corresponding to each of the second sensor data onto the horizontal map to obtain a plurality of horizontal data maps; The splicing module is used to splice the first feature maps corresponding to each of the horizontal data maps to generate a horizontal comprehensive feature map.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the data fusion method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data fusion method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data fusion method according to any one of claims 1 to 6 is implemented.

Images