Multi-source data fusion method and driving decision system

Abstract

The invention provides a multi-source data fusion method and a driving decision system. The multi-source data fusion method comprises the following steps: sequentially performing time synchronization calibration and space synchronization calibration on initial visual data and initial sensor data to obtain second visual data and second sensor data; performing multi-source data feature extraction according to the second visual data and the second sensor data, and executing feature association matching according to a feature extraction result; and further correcting the classification probability based on the feature association matching result so as to output a target classification result according to the corrected classification probability and the feature extraction result, and outputting target fusion data based on the feature association matching result. According to the method, the advantages of the visual data in target classification precision and the advantages of the self-vehicle sensing data in motion state measurement are fully played, the problems of data limitation of a single sensor and low precision of simple superposition fusion are solved, and the precision and stability of data mining in a complex scene are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data fusion, in particular to a multi-source data fusion method and a driving decision system. BACKGROUND

[0002] For target detection, tracking and state analysis in a traffic scene, a single sensor data or simple data superposition is usually relied on, which is difficult to play the complementary advantages of different sensor data, resulting in low data precision and poor stability in subsequent driving decision. SUMMARY

[0003] The present application provides a multi-source data fusion method and a driving decision system to solve the above technical problems.

[0004] Specifically, the present application provides a multi-source data fusion method, comprising the following steps: taking a preset timestamp as a reference, time-synchronously calibrating initial visual data and initial sensor data to obtain first visual data and first sensor data, and space-synchronously calibrating the first visual data and the first sensor data based on the same spatial coordinate system to obtain second visual data and second sensor data; performing multi-source data feature extraction according to the second visual data and the second sensor data, and performing feature correlation matching according to the feature extraction result; and correcting classification probability based on the feature correlation matching result, outputting target classification result according to the corrected classification probability and the feature extraction result, and outputting target fusion data based on the feature correlation matching result.

[0005] In the above technical solution, the advantages of visual data in target classification accuracy and the advantages of self-sensor data in motion state measurement are fully utilized, the limitations of single sensor data and the low fusion accuracy of simple superposition are solved, and the accuracy and stability of data mining in complex scenes are improved.

[0006] Further, the time-synchronous calibration of the initial visual data and the initial sensor data comprises: obtaining a real physical time based on the original timestamp of a setting device and the transmission delay, and judging whether the real physical time is aligned with the preset timestamp, if not, performing time interpolation on the initial visual data or the initial sensor data output by the setting device based on the preset timestamp; otherwise, directly performing space-synchronous calibration.

[0007] In the above technical solution, the uniformity of all sensor data in the time dimension can be ensured, and the use of visual data at t time to match sensor data at t+0.1s time is avoided.

[0008] Further, the spatial synchronization calibration of the first visual data and the first sensor data based on the same spatial coordinate system comprises: converting the first visual data to a vehicle coordinate system based on camera internal parameters and camera external parameters to obtain second visual data; and compensating for installation deviation of the first sensor data based on an installation yaw angle error of the sensor device to obtain second sensor data.

[0009] In the above technical solution, the visual and sensor data are aligned in the same spatial coordinate system, avoiding the failure of associating the visual target frame and the sensor output data with a deviation of 10-20 cm.

[0010] Further, the multi-source data feature extraction comprises: extracting visual features of the second visual data using a convolutional neural network to obtain visual classification features and visual position features; and extracting sensor features of the second sensor data using a convolutional neural network to obtain first radar features and second radar features.

[0011] In the above technical solution, by extracting deep features of visual and self-sensor data, the problem of insufficient depth of multi-source data fusion is solved, and the target "classification + motion state" is accurately mined.

[0012] Further, the feature association matching according to the feature extraction result comprises: calculating a spatial distance similarity based on the visual position features and the second radar features, and calculating a feature consistency similarity based on the visual classification features and the first radar features; calculating an association similarity based on the spatial distance similarity, the feature consistency similarity, and a preset weight coefficient; determining whether the association similarity is greater than an association threshold, and whether the association similarity is the maximum association similarity of the visual target and the radar target with each other, if both conditions are met, determining that the visual target and the radar target are the same target to obtain a feature association matching result; otherwise, determining that they are independent targets, and taking the one with higher confidence in the visual position features and the second radar features as target fusion data, and taking the one with higher confidence in the visual classification features and the first radar features as target classification result.

[0013] In the above technical solution, the association matching can confirm whether the visual detection target and the self-sensor detection target are the same physical target, providing a basis for subsequent fusion.

[0014] Further, the output target classification result comprises: taking the visual classification features in the feature association matching result as target classification features; combining the target classification features and the second radar features to correct the classification probability, so as to obtain the target classification result according to the corrected classification probability and the target classification features.

[0015] In the technical solution, the classification probability is corrected by combining the target classification feature and the second radar feature, so that the information of multi-source data can be comprehensively utilized, the classification result is more in line with the actual situation, and the influence of the classification error of a single sensor is reduced.

[0016] Further, the target fusion data is output based on the feature association matching result, including: fusing the visual position feature and the first radar feature in the feature association matching result based on Kalman filtering to obtain the target fusion data.

[0017] In the technical solution, the Kalman filtering fusion error is only 0.3-0.45 m / s in the occlusion scene; the size measurement error of a long-distance target is reduced from the existing 15%-20% to below 8%, which provides accurate data for automatic driving decision-making.

[0018] Further, after obtaining the target classification result and the target fusion data, the target classification result and the target fusion data are further subjected to abnormality detection, if there is abnormality, the data is supplemented after executing the data supplement mechanism, and then the target classification result and the target fusion data after data supplement are output; otherwise, the target classification result and the target fusion data are directly output.

[0019] In the technical solution, the abnormality of sensor data is identified and processed through abnormality detection and redundant data supplement, the problem of low reliability of data mining result is solved, and the safety of automatic driving decision-making in subsequent application is ensured.

[0020] Further, it further includes: identifying the driving scene of the current target vehicle, and adjusting the preset time stamp, the preset weight coefficient and the association threshold based on the driving scene.

[0021] In the technical solution, the traffic scene prior information library is introduced, the scene-adaptive data mining strategy is established, the data processing range and algorithm complexity are dynamically adjusted according to different scenes (such as urban intersection, highway and congested road section), and the problems of poor scene adaptability and low calculation efficiency of the prior art are solved.

[0022] Based on the same concept, the application further provides a driving decision-making system, including: a data acquisition unit for acquiring the target classification result and the target fusion data by using the multi-source data fusion method; and a driving decision-making unit for performing driving decision-making work based on the target classification result and the target fusion data.

[0023] In the above technical solution, through effective fusion and processing of multi-source data, the driving decision system can better cope with complex traffic scenes; for example, in adverse weather conditions, a single sensor may be greatly affected, while multi-source data fusion can make up for the deficiency by utilizing the advantages of other sensors, so that the system can maintain good performance in various situations, improving the reliability and stability of the system.

[0024] Compared with the prior art, the application has the beneficial effects that: The application fully utilizes the advantages of visual data in target classification accuracy and the advantages of self-sensor data in motion state measurement, solves the problems of single sensor data limitations and low fusion accuracy of simple superposition, and improves the accuracy and stability of data mining in complex scenes. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 A flowchart of the multi-source data fusion method described in the application.

[0026] Figure 2 A framework diagram of the driving decision system described in the application. DETAILED DESCRIPTION

[0027] The multi-source data fusion method and driving decision system of the application will be further described in detail below in combination with specific embodiments and drawings.

[0028] Please refer to Figure 1 The application provides a multi-source data fusion method, which includes the following steps S100-S300.

[0029] In a feasible implementation, through time stamp compensation and spatial coordinate conversion optimization, the space-time deviation of the initial visual data and the initial sensor data is eliminated, and at this time the second visual data and the second sensor data are obtained; then through extraction of deep features of visual and self-sensor data and fusion, accurate mining of classification and motion state is realized, which involves using a convolutional neural network to extract features of the space-time synchronized visual data and sensor data, and determining whether the target detected by visual detection and the target detected by radar (i.e. self-sensor) are the same physical target, and then further fusing the data of feature association and matching; the fusion includes correcting the classification probability by combining the visual classification features and laser radar reflection intensity features obtained by feature extraction, and then outputting the target classification result based on the corrected classification probability and visual classification features, and using Kalman filtering and combining millimeter wave radar speed and visual optical flow speed to obtain target fusion data (such as target speed).

[0030] Next, the specific implementation of the steps S100-S300 will be described in detail.

[0031] Step S100: Based on a preset timestamp, perform time synchronization calibration on the initial visual data and initial sensor data to obtain first visual data and first sensor data, and perform spatial synchronization calibration on the first visual data and first sensor data based on the same spatial coordinate system to obtain second visual data and second sensor data.

[0032] The step of performing time synchronization calibration on the initial visual data and initial sensor data includes: obtaining the real physical time based on the original timestamp and transmission delay of the set device, and determining whether the real physical time is aligned with the preset timestamp. If they are not aligned, time interpolation is performed on the initial visual data or initial sensor data output by the set device based on the preset timestamp; otherwise, spatial synchronization calibration is performed directly.

[0033] In some embodiments, initial visual data is obtained by a visual device (such as a camera), and initial sensor data is obtained by millimeter-wave radar and lidar; wherein, the visual data acquisition frequency... (15-30 frames / second, i.e.) s), millimeter-wave radar frequency f_r (10-20Hz, s), lidar frequency (5-10Hz, s) There are differences, and data transmission delay (50-100ms) (10-20ms) The difference of 20-30ms can cause time misalignment in multi-source data at the same "physical moment", thus requiring time synchronization calibration.

[0034] The time synchronization calibration includes: using high-frequency timestamps from the vehicle's IMU (Inertial Measurement Unit). (100Hz, Based on s), timestamp mapping and interpolation compensation are performed on visual and radar data; it should be noted that those skilled in the art can also use the timestamps of other sensors as a reference for time synchronization calibration.

[0035] Specifically, let the original timestamp of a certain sensor be... (representing vision, radar, and lidar respectively), transmission delay is (Measured in advance through sensor calibration experiments, unit: seconds), then the "real physical moment" corresponding to this sensor data (representing the initial visual data or initial sensor data) is... for: ;in, The original timestamp of the sensor output data (such as the acquisition time of a visual image). The fixed delay between the acquisition and transmission of sensor data to the processing unit is pre-calibrated using a "synchronization signal trigger + time difference measurement" method. For example, all sensors are triggered using a GPS synchronization signal, the time difference between the arrival of data from each sensor to the processing unit is recorded, and the average of multiple measurements is taken as the data. .

[0036] Furthermore, if Not matched with IMU base timestamp (k is the IMU data frame number) Fully aligned, time interpolation of the sensor data is required to generate a result that is consistent with... Aligned data ( Represents sensor data, such as visual target coordinates and radar target velocity; specifically, let's set... (where i is the sensor data frame number), then: ; in, (The actual time interval between two adjacent frames of data from the sensor) These are the raw data from the sensor in frames i and i+1, respectively. For interpolation and Aligned sensor data.

[0037] The above technical solution can ensure the consistency of all sensor data in the time dimension, avoiding the use of visual data at time t to match sensor data at time t+0.1s.

[0038] Furthermore, the spatial synchronization calibration of the first visual data and the first sensor data based on the same spatial coordinate system includes: transforming the first visual data to the vehicle coordinate system based on camera intrinsic and extrinsic parameters to obtain second visual data; and compensating the first sensor data for installation offset based on the installation yaw angle error of the sensor device to obtain second sensor data.

[0039] In some embodiments, the vehicle coordinate system has the vehicle's center of mass as its origin, the vehicle's forward direction as forward, the X-axis forward, the Y-axis left, and the Z-axis upward; the visual detection outputs the pixel coordinates of the target. (u is the horizontal pixel column number, v is the vertical pixel row number), which needs to be converted into three-dimensional coordinates in the vehicle coordinate system through camera intrinsic and extrinsic parameters. ;wherein, pixel coordinates are transformed into camera coordinates (camera intrinsic matrix K): ; in, For camera internal parameters ( The focal length of the camera in the x and y directions, in pixels; The camera principal point coordinates, i.e., image center pixel coordinates, are obtained through a camera calibration experiment, such as Zhang Zhengyou calibration method. The target coordinates in the camera coordinate system are obtained. The target-to-camera distance needs to be obtained through monocular vision depth estimation or laser radar assistance.

[0040] Further, the camera coordinate system is converted into the vehicle coordinate system (external parameter matrix T, including rotation matrix R and translation vector t): Second vision data: ; Wherein, R is a 3x3 rotation matrix (describing the pose of the camera coordinate system relative to the vehicle coordinate system, such as the rotation matrix product corresponding to the pitch angle, roll angle, and yaw angle), is a 3x1 translation vector (describing the position offset of the camera origin relative to the vehicle center of mass, unit: m), and R and t are obtained through a sensor installation calibration experiment (such as fixing a calibration board at a known position in the vehicle coordinate system and shooting multiple images to calculate the external parameters).

[0041] Further, the raw data of the millimeter wave radar and the laser radar have been output in the vehicle coordinate system based on their installation positions, but the installation error needs to be compensated (such as the deviation of the actual installation angle of the radar from the designed angle ), Specifically, let the raw output target coordinates of the radar be , the installation yaw angle error be (rotation error around the vehicle Z axis), and the compensated coordinates be : Second sensor data: ; Wherein, It is obtained through a field calibration (such as aligning the radar to a fixed target with a known distance, comparing the deviation of the theoretical coordinates from the actual output coordinates, and inversely calculating the angle error).

[0042] In the above technical solution, the vision and sensor data are aligned in the same spatial coordinate system, avoiding the failure of the association of the vision target frame and the sensor output data offset by 10-20 cm.

[0043] Step S200: performing multi-source data feature extraction according to the second vision data and the second sensor data, and performing feature association matching according to the feature extraction result.

[0044] Wherein, the multi-source data feature extraction includes: using a convolutional neural network to extract vision features from the second vision data to obtain vision classification features and vision position features; and using a convolutional neural network to extract sensor features from the second sensor data to obtain first radar features and second radar features.

[0045] In some embodiments, two types of features are extracted from the second visual image obtained after spatio-temporal calibration using a convolutional neural network (such as YOLOv8), including: visual classification features a vector (C is the number of target categories, such as vehicles, pedestrians, non-motor vehicles, representing the probability that the target belongs to the i-th category, satisfying ); and visual position features a vector (the coordinates of the target bounding box in the vehicle coordinate system , i.e., the minimum and maximum values of the bounding box in the X and Y directions).

[0046] Further, based on the second sensor data obtained after spatio-temporal calibration, two types of features can also be extracted using a convolutional neural network, including: first radar features a vector (motion state parameters of the target is the radial velocity of the target relative to the ego vehicle, is the radial acceleration, is the speed measurement variance, reflecting the data reliability); and second radar features a vector (three-dimensional contour and reflection features of the target is the length, width, and height of the target, is the average reflection intensity, is the reflection intensity variance).

[0047] In the above technical solution, by extracting deep features of visual and ego vehicle sensor data, the problem of insufficient depth of multi-source data fusion is solved, and precise mining of target "classification + motion state" is achieved.

[0048] Further, the feature association matching based on the feature extraction results includes: calculating the spatial distance similarity based on the visual position features and the second radar features, and calculating the feature consistency similarity based on the visual classification features and the first radar features; calculating the association similarity based on the spatial distance similarity, the feature consistency similarity, and a preset weight coefficient; determining whether the association similarity is greater than an association threshold and whether the association similarity is the maximum association similarity between the visual target and the radar target, if both conditions are met, determining that the visual target and the radar target are the same target to obtain the feature association matching result; otherwise, determining that they are independent targets, and taking the one with higher confidence in the visual position features and the second radar features as the target fusion data, and taking the one with higher confidence in the visual classification features and the first radar features as the target classification result.

[0049] In some embodiments, first, the center of the visual target bounding box in the vehicle coordinate system (i.e., the visual position features) and the center of the radar target Euclidean distance calculation of (i.e. the second radar feature) and (i.e. the visual feature): Spatial distance similarity:

[0050] wherein, is the distance between the visual target center and the radar target center; is the distance attenuation coefficient (e.g. 0.5 m, the closer the distance, the closer to 1, the higher the similarity).

[0051] and the matching degree calculation based on the visual classification result (i.e. the visual classification feature) and the radar motion feature (i.e. the first radar feature) (e.g. when the visual classification is "pedestrian", the radar speed should be less than 5 m / s, and if it is satisfied, the feature consistency is high): Feature consistency similarity: wherein, is the radial velocity of the jth radar target (unit: m / s), is the typical speed corresponding to the visual classification (e.g. pedestrian , car ), is the speed tolerance coefficient (e.g. 2 m / s).

[0052] Further, the correlation similarity is obtained based on the spatial distance similarity, the feature consistency similarity and a preset weight coefficient ; wherein, (the preset weight coefficient is determined by experimental verification, e.g. ).

[0053] Further, it is judged whether is greater than a correlation threshold (e.g. 0.7), if , and i and j are the maximum similarity match with each other (i.e. ), it is determined that the visual target i and the radar target j are the same target, and enters the feature fusion link; otherwise, it is determined as an independent target, and the high confidence data (e.g. visual classification confidence > 0.8 or radar speed measurement variance ) is kept separately.

[0054] In the above technical solution, through the correlation matching, it can be confirmed whether the target detected by the visual detection and the target detected by the self-vehicle sensor are the same physical target, which provides a basis for subsequent fusion.

[0055] ​Step S300: correcting the classification probability based on the feature correlation matching result, so as to output a target classification result according to the corrected classification probability and the feature extraction result, and output target fusion data based on the feature correlation matching result.

[0056] The output target classification result includes: taking the visual classification feature in the feature correlation matching result as a target classification feature; combining the target classification feature and the second radar feature to correct the classification probability, so as to obtain a target classification result according to the corrected classification probability and the target classification feature.

[0057] In some embodiments, the visual classification feature is combined with the second radar feature to correct the classification probability. The laser radar reflection intensity feature is combined with the visual classification feature to correct the classification probability. For example, the reflection intensity of a pedestrian is low, and the reflection intensity of a vehicle is high. Wherein, k is the target class number, is the typical reflection intensity of the kth target (for example, the reflection intensity of a pedestrian , a small car , obtained by sample statistics), is a correction coefficient (for example, 0.01), and Z is a normalization coefficient (to ensure ); and the final classification result (i.e. the target classification result) is .

[0058] It should be noted that in the prior art, the classification accuracy of single vision in bad weather is only 50%-60%, the similar category confusion rate of single radar is up to 35%, and the accuracy of the preliminary fusion technology is up to 85%; but through classification probability correction, the target classification accuracy in bad weather is stably above 92%, and the "pedestrian-non-motor vehicle" confusion rate is reduced to below 8%.

[0059] In the above technical solution, by combining the target classification feature and the second radar feature to correct the classification probability, the information of multi-source data can be comprehensively utilized, so that the classification result is more in line with the actual situation, and the influence of single sensor classification error is reduced.

[0060] Further, the output target fusion data based on the feature correlation matching result includes: fusing the visual position feature in the feature correlation matching result and the first radar feature based on Kalman filtering to obtain target fusion data.

[0061] In some embodiments, the millimeter wave radar speed is fused with the visual optical flow speed (the target motion speed calculated by adjacent visual images), and the Kalman filtering is updated: Prediction step: is the fusion speed at the last time, For the last time acceleration).

[0062] Update step: .

[0063] In the above technical solution, the Kalman filter fusion error is only 0.3-0.45m / s in the occlusion scene; the target size measurement error is reduced from the existing 15%-20% to below 8%, providing accurate data for automatic driving decision.

[0064] Further, after obtaining the target classification result and target fusion data, it further includes: performing abnormality detection on the target classification result and target fusion data respectively, if there is an abnormality, then executing a data completion mechanism, and then outputting the target classification result and target fusion data after data completion; otherwise, directly output the target classification result and target fusion data.

[0065] In the above technical solution, through abnormality detection and redundant data completion, the problem of identifying and processing sensor data anomalies is solved, and the problem of low reliability of data mining results is solved. Fault injection test shows that the reliability of the application result is 96%, which is much higher than the existing technology of 72%, ensuring the safety of automatic driving decision in subsequent applications, so that the radar anomaly completion error only increases 0.1-0.2m / s, the visual anomaly classification accuracy is still above 85%, and the multi-sensor anomaly is within 3 frames. The completion error is less than or equal to 10%.

[0066] Further, it further includes: identifying the driving scene of the current target vehicle, and adjusting the preset timestamp, preset weight coefficient and correlation threshold based on the driving scene.

[0067] In the above technical solution, the traffic scene prior information library is introduced, the scene-adaptive data mining strategy is established, the data processing range and algorithm complexity are dynamically adjusted according to different scenes (such as urban intersection, highway, congested road section), and the problems of poor scene adaptability and low calculation efficiency of the prior art are solved.

[0068] It should be noted that through the scene optimization parameter, the intersection tracking loss rate is reduced to below 9%, the highway scene tracking accuracy is improved by 10%-12%, the target identity confusion rate in the congested road section is reduced from 18% to below 6%, and the cross-scene comprehensive performance fluctuation is only 5%-8%, which is much lower than the existing technology of 15%-20%.

[0069] In addition, the prior art full-scene unified processing takes more than 150 ms for a single frame in a target dense scene, which cannot meet the real-time requirement; the scene adaptive adjustment processing range of the application reduces the target processing amount to 30-40 in a high-speed scene, and the time consumption is 60-80 ms; the processing amount is 20-30 in a congested road section, and the time consumption is 50-70 ms; the time consumption of an intersection scene is also controlled within 90-100 ms, and the calculation resource occupancy rate is reduced by 40%-60% compared with the prior art.

[0070] In summary, the application fully utilizes the advantages of visual data in target classification accuracy and the advantages of self-vehicle sensing data in motion state measurement, solves the problems of single sensor data limitation and low fusion accuracy of simple superposition, and improves the accuracy and stability of data mining in complex scenes.

[0071] Based on the same concept, please refer to Figure 2 The application also provides a driving decision system, which comprises: a data acquisition unit configured to acquire the target classification result and target fusion data by using the multi-source data fusion method; and a driving decision unit configured to perform driving decision work based on the target classification result and target fusion data.

[0072] The data acquisition unit acquires the target classification result (such as pedestrians, vehicles, traffic signs, and lights) and target fusion data (such as speed) by using the specific embodiments of the above multi-source data fusion method.

[0073] The driving decision unit uses a decision algorithm combining rules (such as traffic rules, safety principles, etc.) and machine learning to control the driving state of the vehicle based on the target classification result and target fusion data; for example, in a normal driving scene, when the target classification result shows that the road ahead is clear and there are no obstacles and other vehicles, the driving decision unit controls the vehicle to maintain a stable driving speed and drive according to the lane line based on the vehicle speed and position information in the target fusion data; in a pedestrian scene, if the target classification result detects that a pedestrian is crossing the road ahead, the driving decision unit judges whether the pedestrian will enter the vehicle driving path based on the position, speed, and motion direction of the pedestrian in the target fusion data, and if it is judged that the pedestrian will enter the path, the system will immediately issue a deceleration signal to control the vehicle to slow down and stop, and then continue driving after the pedestrian safely passes; in a front vehicle deceleration scene, when the target classification result identifies that the front vehicle is decelerating, the driving decision unit calculates a safe following distance based on the speed and distance information of the front vehicle in the target fusion data, and if the current distance is less than the safe distance, the system will control the vehicle to brake and slow down to maintain a safe following distance.

[0074] In the above technical solution, through effective fusion and processing of multi-source data, the driving decision system can better cope with complex traffic scenes; for example, in adverse weather conditions, a single sensor may be greatly affected, and multi-source data fusion can make up for the deficiency by utilizing the advantages of other sensors, so that the system can maintain good performance in various situations, improving the reliability and stability of the system.

[0075] Although the example embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the example embodiments are only exemplary and are not intended to limit the scope of the present application. Those of ordinary skill in the art can make various changes and modifications without departing from the scope and spirit of the present application. All such changes and modifications are intended to be included within the scope of the present application as claimed in the appended claims.

[0076] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0077] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the above-described device embodiments are only illustrative, for example, the division of the units is only a logical functional division, and actual implementation can have another division, for example, multiple units or components can be combined or integrated into another device, or some features can be omitted or not executed.

[0078] Embodiments of various components of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those of ordinary skill in the art will understand that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some modules according to embodiments of the present application. The present application can also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such program implementing the present application can be stored on a computer readable medium or can have one or more signals in the form. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0079] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting; it is not intended to exclude myriad other embodiments of the present application that other inventors can develop based on the description provided herein. It must be stressed, however, that any combination of the components or features taught according to any aspect of the present application can be important sufficient to confer "patentable" subject matter status on a single, stand-alone claim. It is therefore intended to embrace all such alternatives, modifications and variations that fall within the spirit and broad scope of the appended claims. One skilled in the art will appreciate that the application can be practiced with embodiments other than those described. The disclosed embodiments are illustrative only and not restrictive.

[0080] While the application has been described in connection with specific embodiments thereof, it will be understood that it is capable of further modifications and this application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended to cover and embrace all adaptations or modifications of the application.

Claims

1. A multi-source data fusion method, characterized in that, Includes the following steps: Based on a preset timestamp, the initial visual data and initial sensor data are time-synchronized and calibrated to obtain the first visual data and the first sensor data. The first visual data and the first sensor data are then spatially synchronized and calibrated based on the same spatial coordinate system to obtain the second visual data and the second sensor data. Multi-source data feature extraction is performed based on the second visual data and the second sensor data, and feature association matching is performed based on the feature extraction results; Furthermore, the classification probability is corrected based on the feature association matching result, so as to output the target classification result according to the corrected classification probability and the feature extraction result, and at the same time, the target fusion data is output based on the feature association matching result.

2. The multi-source data fusion method according to claim 1, characterized in that, The time synchronization calibration of the initial visual data and initial sensor data includes: The real physical time is obtained based on the original timestamp and transmission delay of the set device, and it is determined whether the real physical time is aligned with the preset timestamp. If they are not aligned, the initial visual data or initial sensor data output by the set device is time-interpolated based on the preset timestamp; otherwise, spatial synchronization calibration is performed directly.

3. The multi-source data fusion method according to claim 2, characterized in that, The spatial synchronization calibration of the first visual data and the first sensor data based on the same spatial coordinate system includes: The first visual data is transformed into the vehicle coordinate system based on the camera intrinsic and extrinsic parameters to obtain the second visual data. Additionally, the installation offset compensation is applied to the first sensor data based on the installation yaw angle error of the sensor device to obtain the second sensor data.

4. The multi-source data fusion method according to claim 3, characterized in that, The multi-source data feature extraction includes: A convolutional neural network is used to extract visual features from the second visual data to obtain visual classification features and visual location features. And a convolutional neural network is used to extract sensor features from the second sensor data to obtain first radar features and second radar features.

5. The multi-source data fusion method according to claim 4, characterized in that, The step of performing feature association matching based on the feature extraction results includes: Spatial distance similarity is calculated based on visual location features and second radar features, and feature consistency similarity is calculated based on visual classification features and first radar features. The association similarity is calculated based on the spatial distance similarity, feature consistency similarity, and preset weight coefficients; If the correlation similarity is greater than the correlation threshold and if the correlation similarity is the maximum correlation similarity between the visual target and the radar target, then the visual target and the radar target are determined to be the same target to obtain the feature correlation matching result; otherwise, they are determined to be independent targets, and the visual position feature and the second radar feature with higher confidence are taken as the target fusion data, and the visual classification feature and the first radar feature with higher confidence are taken as the target classification result.

6. The multi-source data fusion method according to claim 5, characterized in that, The output target classification result includes: The visual classification features in the feature association matching results are used as the target classification features; The classification probability is corrected by combining the target classification features and the second radar features, so as to obtain the target classification result based on the corrected classification probability and the target classification features.

7. The multi-source data fusion method according to claim 5, characterized in that, The output of target fusion data based on the feature association matching result includes: The target fusion data is obtained by fusing the visual position features and the first radar features in the feature association matching results based on Kalman filtering.

8. The multi-source data fusion method according to any one of claims 6 or 7, characterized in that, After obtaining the target classification results and target fusion data, the following is also included: Anomaly detection is performed on the target classification result and the target fusion data respectively. If anomalies are found, a data completion mechanism is executed before the target classification result and the target fusion data are output after data completion; otherwise, the target classification result and the target fusion data are output directly.

9. The multi-source data fusion method according to claim 5, characterized in that, Also includes: Identify the current driving scenario of the target vehicle, and adjust the preset timestamp, preset weight coefficient, and association threshold based on the driving scenario.

10. A driving decision-making system, characterized in that, include: The data acquisition unit is used to acquire the target classification result and target fusion data using the multi-source data fusion method as described in any one of claims 1-9; The driving decision unit is used to perform driving decision-making based on the target classification results and target fusion data.

Images