Data fusion method and device and vehicle

Abstract

The invention relates to a data fusion method and device and a vehicle, and relates to the technical field of vehicle electronics, and the method comprises the steps: obtaining the detection data of a plurality of sensors, the detection data comprises the data obtained by a positioning difference module, the data obtained by a camera, and the data obtained by at least one radar; performing preprocessing corresponding to the sensor type on each piece of data in the detection data to obtain preprocessed target data; and performing data fusion on the preprocessed target data to obtain a fused sensing result. According to the method, data obtained by various sensors in the vehicle can be fused, and a more accurate sensing result can be provided for the vehicle by combining the advantages of different sensors.

Description

Technical Field

[0001] The present disclosure relates to the field of vehicle electronic technology, and in particular, to a data fusion method, device and vehicle. Background Art

[0002] With the maturity and development of artificial intelligence technology, intelligent driving vehicles have gradually entered people's field of vision. However, intelligent driving vehicles are not only related to the convenience of transportation, but also closely related to the life safety of passengers. The safe driving of intelligent driving vehicles is closely related to the environmental perception capabilities of various sensors installed on them. Since each sensor has its own limitations and characteristics, intelligent driving vehicle systems usually integrate multiple sensors to obtain more accurate and comprehensive environmental information.

[0003] When integrating sensor data, the more sensors the better. We must also consider the complementary functions and processing performance of each sensor, in order to obtain the best environmental perception capability and achieve real-time processing. Most of the existing multi-sensor data fusion methods focus on the fusion of multiple sensor data of the same type, or only perform data fusion on two sensors. The fusion method does not reflect the unique advantages of each type of sensor, but simply filters the data detected by all sensors. The final result cannot fully reflect the advantages of each sensor. Therefore, providing a processing method for the fusion of multiple sensor data for intelligent driving vehicles is an urgent problem to be solved. Summary of the invention

[0004] To overcome the problems existing in the related art, the present disclosure provides a data fusion method, device and vehicle.

[0005] According to a first aspect of an embodiment of the present disclosure, a data fusion method is provided, including: Acquire detection data from a plurality of sensors, wherein the detection data includes: data acquired by a positioning differential module, data acquired by a camera, and data acquired by at least one radar; Preprocessing each item of the detection data according to the sensor type to obtain preprocessed target data; The preprocessed target data is subjected to data fusion to obtain a fused perception result.

[0006] Optionally, the at least one radar includes: a millimeter wave radar and a laser radar, and the preprocessing of each item of the detection data corresponding to the sensor type to obtain the preprocessed target data includes: Performing a first preprocessing on the second data acquired by the millimeter wave radar through a filtering algorithm and the first data acquired by the positioning difference module to obtain first target data; Performing a second preprocessing on the third data acquired by the laser radar by using a deep learning algorithm and the first data to obtain second target data; Performing a third preprocessing on the third data by using a clustering algorithm and the first data to obtain third target data; The fourth data acquired by the camera is subjected to fourth preprocessing by using an image recognition algorithm and the first data to obtain fourth target data.

[0007] Optionally, the first data obtained by the filtering algorithm and the positioning difference module performs a first preprocessing on the second data obtained by the millimeter wave radar to obtain the first target data, including: Performing filtering and tracking processing on the second data by using a filtering algorithm; Adding a UTC time mark to the processed second data according to the first data; The second data to which the UTC time mark is added is encapsulated to obtain the first target data.

[0008] Optionally, performing second preprocessing on the third data acquired by the laser radar using the deep learning algorithm and the first data to obtain second target data includes: Performing obstacle recognition on the third data through a deep learning algorithm to obtain obstacle information of the obstacle; adding a UTC time mark to the obstacle information according to the first data; The obstacle information after adding the UTC time mark is encapsulated to obtain the second target data.

[0009] Optionally, performing a third preprocessing on the third data by using a clustering algorithm and the first data to obtain third target data includes: Performing clustering processing on the third data by using a clustering algorithm to determine point cloud data corresponding to the obstacle from the third data; Performing boundary fitting on the point cloud data corresponding to the obstacle to obtain obstacle information of the obstacle; adding a UTC time mark to the obstacle information according to the first data; The obstacle information after adding the UTC time mark is encapsulated to obtain the third target data.

[0010] Optionally, performing fourth preprocessing on fourth data acquired by the camera using an image recognition algorithm and the first data to obtain fourth target data includes: Performing image recognition processing on the fourth data by using an image recognition algorithm to obtain image recognition data; Adding a UTC time stamp to the image recognition data according to the first data; The image recognition data after adding the UTC time mark is encapsulated to obtain the fourth target data.

[0011] Optionally, performing data fusion on the preprocessed target data to obtain a fused perception result includes: Performing a first data fusion on the second target data and the third target data to obtain a first fusion result; Performing second data fusion on the first fusion result and the first target data to obtain a second fusion result; Performing a third data fusion on the second fusion result and the fourth target data to obtain a third fusion result; The third fusion result is processed by unscented Kalman filtering to determine the perception result.

[0012] Optionally, performing first data fusion on the second target data and the third target data to obtain a first fusion result includes: Performing time synchronization on the second target data and the third target data; Performing coordinate conversion on the second target data and the third target data after the time synchronization, wherein the coordinate conversion is used to convert the coordinates of the second target data and the third target data after the time synchronization into geodetic coordinates respectively; Performing association matching on the second target data and the third target data after the coordinate transformation to obtain an association matching result, wherein the association matching is used to determine matching pair data of the same obstacle in the second target data and the third target data after the coordinate transformation; The associated matching result is updated with data, and the data update is used to fuse the matching pair data of the same obstacle in the associated matching result to obtain the first fusion result.

[0013] Optionally, performing second data fusion on the first fusion result and the first target data to obtain a second fusion result includes: Performing time synchronization on the first fusion result and the first target data; Performing coordinate conversion on the first fusion result and the first target data after the time synchronization, wherein the coordinate conversion is used to convert the coordinates of the first fusion result and the first target data after the time synchronization into geodetic coordinates respectively; Performing association matching on the first fusion result after the coordinate transformation and the first target data to obtain an association matching result, wherein the association matching is used to determine matching pair data of the same obstacle in the first fusion result after the coordinate transformation and the first target data; The associated matching result is updated with data, and the data update is used to fuse the matching pair data of the same obstacle in the associated matching result to obtain the second fusion result.

[0014] Optionally, performing a third data fusion on the second fusion result and the fourth target data to obtain a third fusion result includes: Performing time synchronization on the second fusion result and the fourth target data; Performing coordinate transformation on the second fusion result and the fourth target data after the time synchronization, wherein the coordinate transformation is used to transform the coordinates of the second fusion result and the fourth target data after the time synchronization into a pixel coordinate system respectively; Performing association matching on the second fusion result after the coordinate transformation and the fourth target data to obtain an association matching result, wherein the association matching is used to determine matching pair data of the same obstacle in the first fusion result after the coordinate transformation and the first target data; The associated matching result is updated with data, and the data update is used to fuse the matching pair data of the same obstacle in the associated matching result to obtain the third fusion result.

[0015] According to a second aspect of an embodiment of the present disclosure, there is provided a data fusion device, comprising: An acquisition module, used to acquire detection data of multiple sensors, wherein the detection data includes: data acquired by the positioning difference module, data acquired by the camera, and data acquired by the at least one radar; A preprocessing module, used to perform preprocessing on each item of the detection data corresponding to the sensor type to obtain preprocessed target data; The fusion module is used to perform data fusion on the preprocessed target data to obtain a fused perception result.

[0016] According to a third aspect of an embodiment of the present disclosure, a vehicle is provided, wherein the vehicle includes the data fusion device provided by the second aspect of the present disclosure.

[0017] Through the above technical solution, the detection data of multiple sensors are obtained, and the detection data includes: data obtained by the positioning differential module, data obtained by the camera, and data obtained by the at least one radar; each item in the detection data is preprocessed according to the sensor type to obtain the preprocessed target data; the preprocessed target data is data fused to obtain the fused perception result. This method can perform data fusion on the data obtained by multiple sensors in the vehicle, and can combine the advantages of different sensors to provide more accurate perception results for the vehicle.

[0018] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings: Figure 1 The figure is a flow chart of a data fusion method according to an exemplary embodiment.

[0020] Figure 2 The figure is a flow chart of a data fusion method according to an exemplary embodiment.

[0021] Figure 3 The figure is a flow chart of a data fusion method according to an exemplary embodiment.

[0022] Figure 4 The figure is a flow chart of a data fusion method according to an exemplary embodiment.

[0023] Figure 5 The figure is a flow chart of a data fusion method according to an exemplary embodiment.

[0024] Figure 6 The figure is a flow chart of a data fusion method according to an exemplary embodiment.

[0025] Figure 7 The figure is a flow chart of a data fusion method according to an exemplary embodiment.

[0026] Figure 8 The figure is a flow chart of a data fusion method according to an exemplary embodiment.

[0027] Fig. 9 The figure is a flow chart of a data fusion method according to an exemplary embodiment.

[0028] Fig.10 The figure is a flow chart of a data fusion method according to an exemplary embodiment.

[0029] Fig.11 The figure is a flow chart of a data fusion method according to an exemplary embodiment.

[0030] Fig.12 It is a block diagram of a data fusion device 1200 according to an exemplary embodiment.

[0031] Fig.13 is a block diagram of an electronic device 1300 according to an exemplary embodiment. DETAILED DESCRIPTION

[0032] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0033] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the device is located and with the authorization given by the owner of the corresponding device.

[0034] Figure 1 FIG. 1 is a flow chart of a data fusion method according to an exemplary embodiment. Figure 1 As shown, the method comprises the following steps: In step S11, detection data of multiple sensors are acquired, and the detection data includes: data acquired by a positioning differential module, data acquired by a camera, and data acquired by the at least one radar.

[0035] For example, smart driving vehicles usually need to face complex and changing road environments. A single sensor may not be able to provide comprehensive and accurate information due to environmental limitations (such as weather and lighting conditions) or its own performance limitations. Therefore, by acquiring and fusing data obtained by different types of sensors, the advantages of different types of sensors can be comprehensively utilized to obtain more accurate and complete environmental information during vehicle driving. For example, cameras can capture images of the vehicle's surroundings and use image processing algorithms to identify objects such as roads, vehicles, and pedestrians. However, cameras are sensitive to lighting conditions and may have difficulty identifying objects in strong light, low light, or backlight conditions. Millimeter-wave radars can detect obstacles around the vehicle while it is driving by emitting and receiving millimeter-wave signals. They can provide a longer detection distance and better anti-interference capabilities, but millimeter-wave radars often have difficulty obtaining accurate information on the shape and type of objects. LiDARs can use laser beams to scan the distance and shape of surrounding objects while the vehicle is driving, but the detection distance is relatively short and may be interfered with in rainy and snowy weather. Ultrasonic radars can use ultrasonic beams to detect close-range obstacles or assist in parking, but their detection capabilities for longer-distance obstacles are limited. The high-precision positioning information and time information that the positioning differential module can provide can be used as a reference for other sensor data, helping to achieve temporal and spatial synchronization and calibration of multi-sensor data.

[0036] In step S12, each item of the detection data is preprocessed according to the sensor type to obtain preprocessed target data.

[0037] Exemplarily, the detection data acquired by each sensor needs to be preprocessed before data fusion, and the preprocessing methods for the detection data acquired by different sensors may be different.

[0038] For example, the preprocessing of the detection data obtained by the positioning difference module may include eliminating obviously abnormal data points in the detection data (such as location information beyond a reasonable range or data with a wrong timestamp), performing error correction on the detection data, etc.; The preprocessing of the detection data obtained by the millimeter wave radar may include eliminating obvious abnormal points in the detection data or data without actual physical meaning (such as empty targets and invalid targets caused by noise and interference), removing high-frequency and low-frequency noise of the detection data through filtering, and improving the signal-to-noise ratio of the data; The preprocessing of the detection data acquired by the laser radar may include removing invalid points (such as points beyond the measurement range, points with extremely low reflection intensity, outliers, etc.) generated in the point cloud data acquired by the laser radar due to equipment failure, occlusion or other reasons, and removing noise points in the point cloud data; The preprocessing of the detection data acquired by the camera may include removing noise from the image acquired by the camera, improving the signal-to-noise ratio of the image, improving the clarity, contrast and brightness of the image, correcting geometric distortion and perspective deformation in the image, etc.

[0039] In step S13, the preprocessed target data is fused to obtain a fused perception result.

[0040] Exemplarily, data fusion is performed on the preprocessed target data, and the data fusion method may include: Kalman filtering method, neural network, fuzzy logic, weighted average method, etc.

[0041] Through the above technical solution, the detection data of multiple sensors are obtained, and the detection data includes: data obtained by the positioning differential module, data obtained by the camera, and data obtained by the at least one radar; each item in the detection data is preprocessed according to the sensor type to obtain the preprocessed target data; the preprocessed target data is data fused to obtain the fused perception result. This method can perform data fusion on the data obtained by multiple sensors in the vehicle, and can combine the advantages of different sensors to provide more accurate perception results for the vehicle.

[0042] Figure 2 FIG. 1 is a flow chart of a data fusion method according to an exemplary embodiment. Figure 2 As shown, the at least one radar includes: a millimeter wave radar and a laser radar, and step S12 includes the following steps: In step S121, a first preprocessing is performed on the second data acquired by the millimeter wave radar through the filtering algorithm and the first data acquired by the positioning difference module to obtain the first target data.

[0043] Exemplarily, the at least one radar may include a millimeter wave radar and a laser radar; the millimeter wave radar can detect and track multiple target objects at the same time, so the second data acquired by the millimeter wave radar may include information such as the distance, speed and angle of each target object. The noise and interference in the second data can be filtered out by a filtering algorithm, and the target information of the useful target object can be extracted. The first data acquired by the positioning differential module can be time synchronized and calibrated for the data in the target information, and the time synchronized and calibrated target information can be used as the first target data; in addition, the present disclosure does not limit the filtering algorithm, and the filtering algorithm can be a static clutter filtering algorithm, a Kalman filtering algorithm, a particle filter, etc.

[0044] In step S122, the third data acquired by the laser radar is subjected to a second preprocessing by using a deep learning algorithm and the first data to obtain second target data.

[0045] Exemplarily, the third data (point cloud data) acquired by the lidar can be segmented through a deep learning algorithm, that is, the points corresponding to different target objects, the boundaries and positions of different target objects, and the types and quantities of identified objects can be detected from the point cloud data. The first data acquired by the positioning differential module can be used to time synchronize and calibrate the third data processed by the deep learning algorithm, and the third data after time synchronization and calibration can be used as the second target data.

[0046] In step S123, a third preprocessing is performed on the third data by using a clustering algorithm and the first data to obtain third target data.

[0047] For example, the clustering algorithm can divide the discrete points in the point cloud data acquired by the laser radar into multiple groups or clusters according to certain characteristics or rules, so as to achieve effective analysis and processing of the data. Therefore, the points of different target objects in the third data (point cloud data) acquired by the laser radar can be clustered by the clustering algorithm to obtain the target information of different target objects, and then the clustered third data can be synchronized and calibrated in time by the first data acquired by the positioning difference module, and the third data after time synchronization and calibration can be used as the third target data.

[0048] In step S124, fourth preprocessing is performed on the fourth data acquired by the camera using an image recognition algorithm and the first data to obtain fourth target data.

[0049] Exemplarily, the image recognition algorithm can perform image recognition on the fourth data (image data) acquired by the camera to identify targets such as obstacles, lane lines, traffic lights and traffic signs in the fourth data, and then perform time synchronization and calibration on the image recognition result of the fourth data based on the first data acquired by the positioning differential module. The third data after time synchronization and calibration can be used as the fourth target data.

[0050] Figure 3 FIG. 1 is a flow chart of a data fusion method according to an exemplary embodiment. Figure 3 As shown, step S121 includes the following steps: In step S1211, the second data is subjected to filtering and tracking processing by means of a filtering algorithm.

[0051] For example, the second data acquired by the millimeter-wave radar may have problems such as instability and high noise. The unstable and high-noise second data can be filtered and tracked through a filtering algorithm to remove obvious erroneous data and outliers in the second data, and the second data can be smoothed using simple sliding average or median filtering methods to reduce random noise.

[0052] In step S1212, a UTC time stamp is added to the processed second data according to the first data.

[0053] In step S1213, the second data to which the UTC time stamp is added is encapsulated to obtain the first target data.

[0054] For example, as an environmental perception sensor, the data of millimeter-wave radar often needs to be integrated with the data of other sensors (such as cameras, lidar, GPS, etc.). In order to achieve accurate integration of multi-sensor data, it is necessary to ensure that all data are in the same time frame; and the high-precision UTC time (Coordinated Universal Time) output by the positioning differential module can provide a globally unified and accurate time standard, which helps to achieve time synchronization between different sensor data. Therefore, the first data (UTC time) obtained by the positioning differential module can be used to add a UTC time mark to each frame of the processed second data, and the second data with the UTC time mark added can be packaged as a single frame of data to obtain the first target data.

[0055] Figure 4 FIG. 1 is a flow chart of a data fusion method according to an exemplary embodiment. Figure 4 As shown, step S122 includes the following steps: In step S1221, obstacle identification is performed on the third data using a deep learning method to obtain obstacle information of the obstacle.

[0056] In step S1222, a UTC time mark is added to the obstacle information according to the first data.

[0057] In step S1223, the obstacle information after adding the UTC time mark is encapsulated to obtain the second target data.

[0058] Exemplarily, the deep learning method can identify obstacles in the point cloud data acquired by the lidar through a trained data model (such as PointNet, PointNet++, VoxelNet, etc.), and obtain the identified obstacle information; therefore, the third data can be used to identify obstacles through a deep learning algorithm to obtain the obstacle information of the obstacle, and the first data (UTC time) can be used to add a UTC time stamp to each frame of data in the obstacle information, and the obstacle information with the UTC time stamp added can be packaged as a single frame of data to obtain the second target data.

[0059] Figure 5 FIG. 1 is a flow chart of a data fusion method according to an exemplary embodiment. Figure 5 As shown, step S123 includes the following steps: In step S1231, clustering processing is performed on the third data to determine point cloud data corresponding to the obstacle from the third data.

[0060] Exemplarily, the clustering algorithm can perform clustering processing on the third data (point cloud data) acquired by the laser radar, separate obstacles from the point cloud data, and determine the points corresponding to each obstacle; the clustering processing includes removing noise points in the point cloud data (such as abnormal points caused by the sensor itself or the environment), and grouping the points in the point cloud data after removing the noise points into different clusters, each cluster can represent a potential obstacle, and the point cloud data corresponding to each obstacle can be determined. In addition, the present disclosure does not limit the type of the clustering algorithm, for example, the clustering algorithm can be DBSCAN (Density-Based Spatial Clustering of Applications with Noise), K-means clustering, Euclidean Cluster Extraction.

[0061] In step S1232, boundary fitting and filter tracking are performed on the point cloud data corresponding to the obstacle to obtain obstacle information of the obstacle.

[0062] For example, the point cloud data corresponding to each obstacle can use boundary fitting algorithms (such as least squares plane fitting, convex hull algorithm, Alpha Shape, etc.) to estimate the three-dimensional contour of the obstacle, calculate the obstacle's center point (such as centroid), direction (such as main direction), size (such as the size of the bounding box) and other position information, and use filtering algorithms (such as Kalman filtering, particle filtering, etc.) to smooth the obstacle information after boundary fitting and predict the movement trajectory of the obstacle, while eliminating the noise in the perception data to obtain the obstacle information of the obstacle. Through this step, according to the obstacle information in continuous frames, cross-frame obstacle tracking can be performed to maintain continuous cognition of the obstacle identity.

[0063] In step S1233, a UTC time mark is added to the obstacle information according to the first data.

[0064] In step S1234, the obstacle information with the added UTC time mark is encapsulated to obtain the third target data.

[0065] Exemplarily, a UTC time mark can be added to each frame of data in the obstacle information through the first data (UTC time), and the obstacle information with the added UTC time mark is encapsulated as a single frame of data to obtain the third target data.

[0066] Figure 6 FIG. 1 is a flow chart of a data fusion method according to an exemplary embodiment. Figure 6 As shown, step S124 includes the following steps: In step S1241, image recognition processing is performed on the fourth data using an image recognition algorithm to obtain image recognition data.

[0067] In step S1242, a UTC time stamp is added to the image recognition data according to the first data.

[0068] In step S1243, the image recognition data to which the UTC time stamp is added is encapsulated to obtain the fourth target data.

[0069] Exemplarily, the image recognition algorithm can be used to identify image recognition data of obstacles, lane lines, traffic lights, traffic signs, etc. in the fourth data (image information) acquired by the camera, and a UTC time stamp is added to each frame of data in the image recognition data through the first data (UTC time). The image recognition data with the added UTC time stamp is packaged as a single frame of data to obtain the fourth target data.

[0070] Figure 7 FIG. 1 is a flow chart of a data fusion method according to an exemplary embodiment. Figure 7As shown, step S13 includes the following steps: In step S131, a first data fusion is performed on the second target data and the third target data to obtain a first fusion result.

[0071] For example, when the deep learning method identifies obstacle information in the third data (point cloud data), it is usually affected by the quality of the input point cloud data. If the point cloud data has noise, occlusion or missing problems, it may lead to a decrease in model performance and low recognition efficiency for objects with complex shapes and structures (such as parking fences, temporary traffic lights, etc.). The clustering method has lower requirements for the quality of point cloud data and has a higher recognition efficiency for objects with complex shapes and structures in the environment. Therefore, the second target data obtained by deep learning and the third target data obtained by the clustering algorithm are fused for the first data. The first fusion result can contain more comprehensive obstacle information in the vehicle situation environment, and the size prediction and category classification of the target can be more accurate.

[0072] In step S132, a second data fusion is performed on the first fusion result and the first target data to obtain a second fusion result.

[0073] Exemplarily, the first target data includes obstacle data acquired by a millimeter-wave radar, such as information such as the distance, speed, and angle of the obstacle from the vehicle. The second data fusion is performed on the first fusion result and the first target data. Information such as distance, speed, and angle can be added to the obstacle data in the first fusion result based on the first target data to obtain a second fusion result.

[0074] In step S133, a third data fusion is performed on the second fusion result and the fourth target data to obtain a third fusion result.

[0075] Exemplarily, the fourth target data includes obstacle data acquired by the camera, such as data on the vehicle's lane, lane lines, road driving signs, traffic lights, traffic light colors and other obstacles. The second fusion result and the fourth target data are subjected to a third data fusion. The third fusion result can be obtained by adding information such as color and road location to the obstacle in the second fusion result based on the fourth target data.

[0076] In step S134, the third fusion result is processed by unscented Kalman filtering to determine the perception result.

[0077] For example, the unscented Kalman filter adds unscented transformation to the Kalman filter, so that it can handle the state estimation problem of nonlinear systems. The unscented Kalman filter mainly includes three parts: state prediction, data association and data update; State prediction predicts the state of the next moment based on the dynamic model of the system. In the state prediction stage, the Kalman filter uses the state estimate of the previous moment (including mean and covariance) and the control input of the system (if any), and predicts the state of the current moment through the state transfer equation.

[0078] Data association is the process of associating observation data from different sensors or at different time points to corresponding targets or system states.

[0079] Prediction updates use current observations to correct state predictions and obtain more accurate state estimates.

[0080] Therefore, the unscented Kalman filter can perform state prediction, data association, and prediction update on the third fusion result to determine the perception result.

[0081] Figure 8 FIG. 1 is a flow chart of a data fusion method according to an exemplary embodiment. Figure 8 As shown, step S131 includes the following steps: In step S1311, the second target data and the third target data are time synchronized.

[0082] Exemplarily, when the first data fusion is performed on the second target data and the third target data, the first data fusion includes four steps: time synchronization, coordinate conversion, association matching, and data update; the data of the unsynchronized sensors can be filtered through the time synchronization method, which can reduce the error caused by the unsynchronized sensor data, increase the accuracy of data association matching, and thus ensure the accuracy of multi-sensor data fusion. Among them, the time synchronization includes: saving the second target data and the third target data into corresponding double-ended queues; comparing the timestamps of the second target data and the third target data in the latest frame, selecting a data path with a smaller timestamp as a benchmark, and then traversing the data in the other queue, looking for a group of data with a smaller difference in timestamp from the benchmark data and within a preset threshold for subsequent matching and updating, and clearing all the data before this timestamp to ensure that the time is continued backward.

[0083] In step S1312, coordinate transformation is performed on the time-synchronized second target data and the third target data, and the coordinate transformation is used to convert the coordinates of the time-synchronized second target data and the third target data into geodetic coordinates, respectively.

[0084] For example, the coordinate transformation is to transform the time-synchronized second target data and the third target data into the georeferenced system, so as to achieve obstacle matching and tracking. The coordinate transformation usually involves rotation and translation transformation, which can be achieved by the external parameter matrix obtained by sensor calibration.

[0085] In step S1313, association matching is performed on the second target data and the third target data after the coordinate transformation to obtain an association matching result, wherein the association matching is used to determine matching pair data of the same obstacle in the second target data and the third target data after the coordinate transformation.

[0086] Exemplarily, association matching is used to determine whether the two data of the second target data and the third target data are the same target, establish an association matrix for the two data after coordinate transformation, calculate the positional relationship between the two data to find the optimal match for each data, and obtain a matching pair.

[0087] For example, an association matrix is ​​established for the second target data and the third target data after coordinate transformation, and the IOU (intersection over union) value between each pair of possible corresponding data in the association matrix is ​​calculated. When the IOU value is greater than a specified IOU threshold, it is considered that the possible corresponding data are from the same object. Among them, IOU is an indicator to measure the degree of overlap between two bounding boxes, which can be used to determine the matching relationship in target detection and target tracking.

[0088] In step S1314, data of the associated matching result is updated, and the data update is used to fuse the matching pair data of the same obstacle in the associated matching result to obtain the first fusion result.

[0089] Exemplarily, the association matching refers to updating the data of the matching pairs obtained by association matching, fusing the data of the same target in the two data channels, and outputting the original information of other data that are not associated matched to obtain the first fusion result.

[0090] Fig. 9 FIG. 1 is a flow chart of a data fusion method according to an exemplary embodiment. Fig. 9 As shown, step S132 includes the following steps: In step S1321, the first fusion result and the first target data are time synchronized.

[0091] In step S1322, coordinate transformation is performed on the first fusion result and the first target data after the time synchronization, and the coordinate transformation is used to convert the coordinates of the first fusion result and the first target data after the time synchronization into geodetic coordinates respectively.

[0092] In step S1323, the first fusion result after the coordinate transformation and the first target data are associated and matched to obtain an associated matching result, and the associated matching is used to determine matching pair data of the same obstacle in the first fusion result after the coordinate transformation and the first target data.

[0093] In step S1324, data of the associated matching result is updated, and the data update is used to fuse the matching pair data of the same obstacle in the associated matching result to obtain the second fusion result.

[0094] Exemplarily, when the second data fusion is performed on the first fusion result and the first target data, the second data fusion may also include four steps: time synchronization, coordinate conversion, association matching and data updating. The time synchronization, coordinate conversion, association matching and data updating of the second data fusion may refer to Figure 8 The time synchronization, coordinate conversion, association matching and data update steps in the first data fusion in the embodiment are not repeated here; in addition, the first fusion result is the fusion result of the data acquired by the lidar, and the first target data is the data acquired by the millimeter wave radar. Therefore, considering the advantages of the two types of sensors in detecting object characteristics and ranges, when updating data, data combination substitution can be adopted to update the data, and then data encapsulation is performed.

[0095] Fig.10 FIG. 1 is a flow chart of a data fusion method according to an exemplary embodiment. Fig.10 As shown, step S133 includes the following steps: In step S1331, the second fusion result and the fourth target data are time synchronized.

[0096] In step S1332, coordinate transformation is performed on the second fusion result and the fourth target data after the time synchronization, and the coordinate transformation is used to convert the coordinates of the second fusion result and the fourth target data after the time synchronization into a pixel coordinate system respectively.

[0097] In step S1333, the second fusion result after the coordinate transformation and the fourth target data are associated and matched to obtain an associated matching result, and the associated matching is used to determine matching pair data of the same obstacle in the first fusion result after the coordinate transformation and the first target data.

[0098] In step S1334, data of the associated matching result is updated, and the data update is used to fuse the matching pair data of the same obstacle in the associated matching result to obtain the third fusion result.

[0099] Exemplarily, when the third data fusion is performed on the second fusion result and the fourth target data, the third data fusion may also include four steps: time synchronization, coordinate conversion, association matching and data updating. The time synchronization in the third data fusion may be achieved by performing time synchronization from the second fusion data to the fourth target data. Coordinate conversion is to calculate the corner point information of the second fusion data after time synchronization by using the conversion relationship between the vehicle body coordinate system and the image pixel coordinate system to obtain the rectangular border information of the obstacle in the second fusion data in the image pixel coordinate system. Association matching may use the obstacle pixel information and category information to establish an association matrix, and obtain matching pairs based on the idea of ​​global optimal matching. Data updating may use the type in the fourth target data to correct the target type in the second fusion data according to the matching pair, and also use the perception information in the fourth target data to eliminate targets with low confidence in the second fusion data, and finally perform data encapsulation to obtain the final third fusion result.

[0100] To sum up, when updating the data of the matching pair, the motion information of the millimeter-wave radar in the first target data is fused with the position information and appearance feature information of the lidar in the first fusion result, and then the classification information of the target is corrected using the precise classification of the camera in the fourth target data. This can better bring into play the advantages of the three types of sensors to determine the shape, type, motion information, etc. of the obstacle.

[0101] Through the above technical solution, the detection data of multiple sensors are obtained, and the detection data includes: data obtained by the positioning differential module, data obtained by the camera, and data obtained by the at least one radar; each item in the detection data is preprocessed according to the sensor type to obtain the preprocessed target data; the preprocessed target data is data fused to obtain the fused perception result. This method can perform data fusion on the data obtained by multiple sensors in the vehicle, and can combine the advantages of different sensors to provide more accurate perception results for the vehicle.

[0102] Fig.11 is a flow chart of a data fusion method according to an exemplary embodiment. Fig.11As shown, the data obtained by the positioning difference module and the millimeter wave radar are preprocessed 1 to obtain the first target data; the data obtained by the positioning difference module and the laser radar are preprocessed 2 to obtain the second target data (laser radar deep learning target level data) and the third target data (laser radar clustering algorithm target level data); the data obtained by the positioning difference module and the camera are preprocessed 3 to obtain the fourth target data (camera target level data); then the second target data and the third target data are subjected to the first layer fusion processing, and then the first target data and the result of the first layer fusion processing are subjected to the second layer fusion processing; then the fourth target data and the result of the second layer fusion processing are subjected to the third layer fusion processing, and finally the result of the third layer fusion processing is subjected to multi-target tracking processing to obtain the perception result.

[0103] Fig.12 is a block diagram of a data fusion device 1200 according to an exemplary embodiment. Fig.12 As shown, the device includes: an acquisition module 1210, a preprocessing module 1220 and a fusion module 1230; The acquisition module 1210 is used to acquire detection data of multiple sensors, where the detection data includes: data acquired by the positioning difference module, data acquired by the camera, and data acquired by the at least one radar; The preprocessing module 1220 is used to preprocess each item of the detection data according to the sensor type to obtain preprocessed target data; The fusion module 1230 is used to perform data fusion on the preprocessed target data to obtain a fused perception result.

[0104] Optionally, the at least one radar includes: a millimeter wave radar and a laser radar; the preprocessing module 1220 includes: a first preprocessing submodule, a second preprocessing submodule, a third preprocessing submodule and a fourth preprocessing submodule; The first preprocessing submodule is used to perform first preprocessing on the second data acquired by the millimeter wave radar through the filtering algorithm and the first data acquired by the positioning difference module to obtain first target data; The second preprocessing submodule is used to perform a second preprocessing on the third data acquired by the laser radar through a deep learning algorithm and the first data to obtain second target data; The third preprocessing submodule is used to perform a third preprocessing on the third data by using a clustering algorithm and the first data to obtain third target data; The fourth preprocessing submodule is used to perform fourth preprocessing on fourth data acquired by the camera through an image recognition algorithm and the first data to obtain fourth target data.

[0105] Optionally, the first preprocessing submodule is further used for: Performing filtering and tracking processing on the second data by using a filtering algorithm; Adding a UTC time mark to the processed second data according to the first data; The second data to which the UTC time mark is added is encapsulated to obtain the first target data.

[0106] Optionally, the second preprocessing submodule is further used for: Performing obstacle recognition on the third data through a deep learning algorithm to obtain obstacle information of the obstacle; Adding a UTC time mark to the obstacle information according to the first data; The obstacle information after adding the UTC time mark is encapsulated to obtain the second target data.

[0107] Optionally, the third preprocessing submodule is further used for: Performing clustering processing on the third data by using a clustering algorithm to determine point cloud data corresponding to the obstacle from the third data; Perform boundary fitting on the point cloud data corresponding to the obstacle to obtain obstacle information of the obstacle; Adding a UTC time mark to the obstacle information according to the first data; The obstacle information after adding the UTC time mark is encapsulated to obtain the third target data.

[0108] Optionally, the fourth preprocessing submodule is further used for: Performing image recognition processing on the fourth data by using an image recognition algorithm to obtain image recognition data; Adding a UTC time stamp to the image recognition data according to the first data; The image recognition data after the UTC time mark is added is encapsulated to obtain the fourth target data.

[0109] Optionally, the fusion module 1230 includes: a first fusion submodule, a second fusion submodule, a third fusion submodule and a perception recognition submodule; The first fusion submodule is used to perform a first data fusion on the second target data and the third target data to obtain a first fusion result; The second fusion submodule is used to perform a second data fusion on the first fusion result and the first target data to obtain a second fusion result; The third fusion submodule is used in the first fusion submodule to perform a third data fusion on the second fusion result and the fourth target data to obtain a third fusion result; The perception recognition submodule is used to process the third fusion result through unscented Kalman filtering to determine the perception result.

[0110] Optionally, the first fusion submodule is further used for: Performing time synchronization on the second target data and the third target data; Performing coordinate conversion on the second target data and the third target data after the time synchronization, wherein the coordinate conversion is used to convert the coordinates of the second target data and the third target data after the time synchronization into geodetic coordinates respectively; Performing association matching on the second target data and the third target data after the coordinate transformation to obtain an association matching result, wherein the association matching is used to determine matching pair data of the same obstacle in the second target data and the third target data after the coordinate transformation; The associated matching result is updated with data, and the data update is used to fuse the matching pair data of the same obstacle in the associated matching result to obtain the first fusion result.

[0111] Optionally, the second fusion submodule is further used for: Performing time synchronization on the first fusion result and the first target data; Performing coordinate transformation on the first fusion result and the first target data after the time synchronization, wherein the coordinate transformation is used to transform the coordinates of the first fusion result and the first target data after the time synchronization into geodetic coordinates respectively; Performing association matching on the first fusion result after the coordinate transformation and the first target data to obtain an association matching result, wherein the association matching is used to determine matching pair data of the same obstacle in the first fusion result after the coordinate transformation and the first target data; The associated matching result is updated with data, and the data update is used to fuse the matching pair data of the same obstacle in the associated matching result to obtain the second fusion result.

[0112] Optionally, the third fusion submodule is further used for: Performing time synchronization on the second fusion result and the fourth target data; Performing coordinate transformation on the second fusion result and the fourth target data after the time synchronization, wherein the coordinate transformation is used to transform the coordinates of the second fusion result and the fourth target data after the time synchronization into a pixel coordinate system respectively; Performing association matching on the second fusion result after the coordinate transformation and the fourth target data to obtain an association matching result, wherein the association matching is used to determine matching pair data of the same obstacle in the first fusion result after the coordinate transformation and the first target data; The associated matching result is updated with data, and the data update is used to fuse the matching pair data of the same obstacle in the associated matching result to obtain the third fusion result.

[0113] Through the above technical solution, the detection data of multiple sensors are obtained, and the detection data includes: data obtained by the positioning differential module, data obtained by the camera, and data obtained by the at least one radar; each item in the detection data is preprocessed according to the sensor type to obtain the preprocessed target data; the preprocessed target data is data fused to obtain the fused perception result. This method can perform data fusion on the data obtained by multiple sensors in the vehicle, and can combine the advantages of different sensors to provide more accurate perception results for the vehicle.

[0114] In another exemplary embodiment, a vehicle is also provided, the vehicle comprising the above Fig.12 The data fusion device in.

[0115] Fig.13 FIG. 1 is a block diagram of an electronic device 1300 according to an exemplary embodiment. Fig.13 As shown, the electronic device 1300 may include: a processor 1301 , a memory 1302 . The electronic device 1300 may also include one or more of a multimedia component 1303 , an input / output (I / O) interface 1304 , and a communication component 1305 .

[0116] The processor 1301 is used to control the overall operation of the electronic device 1300 to complete all or part of the steps in the above-mentioned data fusion method. The memory 1302 is used to store various types of data to support the operation of the electronic device 1300. For example, these data may include instructions for any application or method used to operate on the electronic device 1300, and application-related data, such as contact data, sent and received messages, pictures, audio, video, etc. The memory 1302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, magnetic disk or optical disk. The multimedia component 1303 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone, which is used to receive external audio signals. The received audio signal may be further stored in the memory 1302 or sent through the communication component 1305. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 1304 provides an interface between the processor 1301 and other interface modules, and the above-mentioned other interface modules may be keyboards, mice, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 1305 is used for wired or wireless communication between the electronic device 1300 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more of them, is not limited here. Therefore, the corresponding communication component 1305 may include: Wi-Fi module, Bluetooth module, NFC module, etc.

[0117] In an exemplary embodiment, the electronic device 1300 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned data fusion method.

[0118] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned data fusion method are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 1302 including program instructions, and the above-mentioned program instructions can be executed by the processor 1301 of the electronic device 1300 to complete the above-mentioned data fusion method.

[0119] In another exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program executable by a programmable device. The computer program has a code portion for executing the above data fusion method when executed by the programmable device.

[0120] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0121] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0122] In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A data fusion method, characterized in that: include: Acquire detection data from a plurality of sensors, wherein the detection data includes: data acquired by a positioning differential module, data acquired by a camera, and data acquired by at least one radar; Preprocessing each item of the detection data according to the sensor type to obtain preprocessed target data; The preprocessed target data is subjected to data fusion to obtain a fused perception result.

2. The method according to claim 1, characterized in that The at least one radar includes: a millimeter wave radar and a laser radar, and the preprocessing of each item of the detection data corresponding to the sensor type to obtain the preprocessed target data includes: Performing a first preprocessing on the second data acquired by the millimeter wave radar through a filtering algorithm and the first data acquired by the positioning difference module to obtain first target data; Performing a second preprocessing on the third data acquired by the laser radar by using a deep learning algorithm and the first data to obtain second target data; Performing a third preprocessing on the third data by using a clustering algorithm and the first data to obtain third target data; The fourth data acquired by the camera is subjected to fourth preprocessing by using an image recognition algorithm and the first data to obtain fourth target data.

3. The method according to claim 2, characterized in that The first data obtained by the filtering algorithm and the positioning differential module performs a first preprocessing on the second data obtained by the millimeter wave radar to obtain the first target data, including: Performing filtering and tracking processing on the second data by using a filtering algorithm; Adding a UTC time mark to the processed second data according to the first data; The second data to which the UTC time mark is added is encapsulated to obtain the first target data.

4. The method according to claim 2, characterized in that: The second preprocessing of the third data acquired by the laser radar using the deep learning algorithm and the first data to obtain second target data includes: Performing obstacle recognition on the third data through a deep learning algorithm to obtain obstacle information of the obstacle; adding a UTC time mark to the obstacle information according to the first data; The obstacle information after adding the UTC time mark is encapsulated to obtain the second target data.

5. The method according to claim 2, characterized in that: The performing a third preprocessing on the third data by using a clustering algorithm and the first data to obtain third target data includes: Performing clustering processing on the third data by using a clustering algorithm to determine point cloud data corresponding to the obstacle from the third data; Performing boundary fitting on the point cloud data corresponding to the obstacle to obtain obstacle information of the obstacle; adding a UTC time mark to the obstacle information according to the first data; The obstacle information after adding the UTC time mark is encapsulated to obtain the third target data.

6. The method according to claim 2, characterized in that The fourth data acquired by the camera is subjected to fourth preprocessing by using the image recognition algorithm and the first data to obtain fourth target data, including: Performing image recognition processing on the fourth data by using an image recognition algorithm to obtain image recognition data; Adding a UTC time stamp to the image recognition data according to the first data; The image recognition data after adding the UTC time mark is encapsulated to obtain the fourth target data.

7. The method according to claim 2, characterized in that The step of fusing the preprocessed target data to obtain a fused perception result includes: Performing a first data fusion on the second target data and the third target data to obtain a first fusion result; Performing second data fusion on the first fusion result and the first target data to obtain a second fusion result; Performing a third data fusion on the second fusion result and the fourth target data to obtain a third fusion result; The third fusion result is processed by unscented Kalman filtering to determine the perception result.

8. The method according to claim 7, characterized in that The performing first data fusion on the second target data and the third target data to obtain a first fusion result includes: Performing time synchronization on the second target data and the third target data; Performing coordinate conversion on the second target data and the third target data after the time synchronization, wherein the coordinate conversion is used to convert the coordinates of the second target data and the third target data after the time synchronization into geodetic coordinates respectively; Performing association matching on the second target data and the third target data after the coordinate transformation to obtain an association matching result, wherein the association matching is used to determine matching pair data of the same obstacle in the second target data and the third target data after the coordinate transformation; The associated matching result is updated with data, and the data update is used to fuse the matching pair data of the same obstacle in the associated matching result to obtain the first fusion result.

9. The method according to claim 7, characterized in that: The performing second data fusion on the first fusion result and the first target data to obtain a second fusion result includes: Performing time synchronization on the first fusion result and the first target data; Performing coordinate conversion on the first fusion result and the first target data after the time synchronization, wherein the coordinate conversion is used to convert the coordinates of the first fusion result and the first target data after the time synchronization into geodetic coordinates respectively; Performing association matching on the first fusion result after the coordinate transformation and the first target data to obtain an association matching result, wherein the association matching is used to determine matching pair data of the same obstacle in the first fusion result after the coordinate transformation and the first target data; The associated matching result is updated with data, and the data update is used to fuse the matching pair data of the same obstacle in the associated matching result to obtain the second fusion result.

10. The method according to claim 7, characterized in that The performing third data fusion on the second fusion result and the fourth target data to obtain a third fusion result includes: Performing time synchronization on the second fusion result and the fourth target data; Performing coordinate transformation on the second fusion result and the fourth target data after the time synchronization, wherein the coordinate transformation is used to transform the coordinates of the second fusion result and the fourth target data after the time synchronization into a pixel coordinate system respectively; Performing association matching on the second fusion result after the coordinate transformation and the fourth target data to obtain an association matching result, wherein the association matching is used to determine matching pair data of the same obstacle in the first fusion result after the coordinate transformation and the first target data; The associated matching result is updated with data, and the data update is used to fuse the matching pair data of the same obstacle in the associated matching result to obtain the third fusion result.

11. A data fusion device, characterized in that: include: An acquisition module, used to acquire detection data of multiple sensors, wherein the detection data includes: data acquired by the positioning difference module, data acquired by the camera, and data acquired by the at least one radar; A preprocessing module, used to perform preprocessing on each item of the detection data corresponding to the sensor type to obtain preprocessed target data; The fusion module is used to perform data fusion on the preprocessed target data to obtain a fused perception result.

12. A vehicle, characterized in that: The vehicle comprises the data fusion device as claimed in claim 11.

Images