Multi-source information sensor fusion target detection method

Through sensor space-time synchronization, data association and multi-objective tracking algorithms, the problem of insufficient real-time and reliability of multi-sensor fusion target detection in the prior art is solved, and the accuracy of autonomous driving environment perception and system fault tolerance are improved.

CN119006785APending Publication Date: 2024-11-22ANHUI DAIZE INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411089639.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing multiple sensor fusion target detection methods have shortcomings in real-time and reliability, especially in dynamic and changing scenarios, which are difficult to meet the requirements of high-level autonomous driving, and have caused information dimension explosion and vehicle-side computing power pressure.

Method used

Spatial-time synchronization, data association and multi-target tracking are performed through multiple sensors (camera, lidar and millimeter wave radar). The frequency down processing, coordinate system conversion, data association algorithm and random finite set tracking algorithm are used to eliminate data ambiguity and Improve the accuracy and accuracy of target recognition, and improve system redundancy and fault tolerance.

Benefits of technology

It improves the accuracy and accuracy of target recognition, alleviates the explosion of information dimensions and the pressure of computing power on the vehicle, improves the real-time and reliability of the system, and realizes highly reliable multi-source sensor fusion target detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006785A_ABST
    Figure CN119006785A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-source information sensor fusion target detection method, which is mainly applied to the field of self-driving automobile environment perception and aims at improving the redundancy and fault tolerance of a perception system and ensuring the real-time performance and reliability of multi-source heterogeneous perception data fusion. The method specifically comprises the following steps: step 1, acquiring image information and point cloud information after a detection target is processed through a plurality of sensors on an automobile; step 2, performing frequency reduction processing and coordinate system conversion on the data of each sensor according to the difference between the sampling period and the spatial position of the plurality of sensors so as to realize time-space synchronization of the plurality of sensors; step 3, aiming at the problem that association of the target data of the multiple sensors is easy to be disordered, a data association algorithm of the multiple sensors is adopted to eliminate data ambiguity; and step 4, tracking multiple targets in the dynamic scene, associating the same moving object in different frames in time sequence to form continuous understanding of the multiple targets, and outputting and sensing information such as target positions, speeds and orientations in real time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving vehicle environment perception, and in particular relates to a multi-source information sensor fusion target detection method. Background Art

[0002] With the development of intelligent network technology, it is difficult to meet the requirements of high-level autonomous driving in dynamic and changeable scenarios by relying on single-mode information perception technology. Therefore, the detection of the vehicle's external environment requires the use of multiple sensors (such as cameras, lidars, and millimeter-wave radars, etc.) for synchronous detection and data fusion. At present, there are various fusion schemes, including camera + millimeter-wave radar, camera + lidar, camera + millimeter-wave radar + lidar, etc., which fuse the data collected from multiple sensors such as cameras, millimeter-wave radars, and lidars. The fusion of multiple sensor information is mainly achieved through three steps: spatiotemporal synchronization, data association, and target tracking. Spatiotemporal synchronization is to align the data collected by each sensor in the time dimension and space dimension respectively. With the increase in the number of sensors and the increasing amount of data, the processing speed requirements of the spatiotemporal synchronization algorithm are getting higher and higher. Data association is to match the targets detected by different sensors and determine the corresponding relationship between the targets and the measurement information received by the sensors. Its robustness needs to be improved. Dynamic scene multi-target tracking mainly associates the time sequence of the same moving object in different frames to form a continuous understanding of multiple targets and output the position, speed and orientation information of the perceived objects.

[0003] At the same time, equipping a car with multiple sensors also brings about problems such as dimensionality explosion and excessive computing pressure on the vehicle side, and puts high demands on the real-time and reliability of multi-source heterogeneous perception data fusion. Therefore, multimodal, highly reliable collaborative perception and information fusion technology is an issue that needs to be solved urgently. It is necessary to design a highly reliable multi-source sensor fusion target detection method. Summary of the invention

[0004] Aiming at the poor real-time performance and reliability of existing multiple sensor fusion target detection methods, the present invention provides a multi-source information sensor fusion target detection method.

[0005] To achieve the above purpose, the technical solution provided by the present invention is as follows:

[0006] A multi-source information sensor fusion target detection method, characterized in that it includes the following steps:

[0007] Step 1, obtaining processed image information and point cloud information of the detection target through multiple sensors on the car;

[0008] Step 2: based on the differences in sampling period and spatial position of multiple sensors, the data of each sensor is down-converted and the coordinate system is converted to achieve spatiotemporal synchronization of multiple sensors;

[0009] Step 3: To solve the problem of confusion in the association of target data from multiple sensors, a multiple sensor data association algorithm is used to eliminate data ambiguity.

[0010] Step 4: Track multiple targets in a dynamic scene, associate the time sequence of the same moving object in different frames, form a continuous understanding of multiple targets, and output information such as the perceived target position, speed, and direction in real time.

[0011] Furthermore, the multiple sensors in step 1 include: 1 camera, 1 laser radar and 1 millimeter wave radar;

[0012] Furthermore, the spatiotemporal synchronization of multiple sensors in step 2 includes the following steps:

[0013] Step A: Down-convert the high-frequency LiDAR and millimeter-wave radar by setting parameters to keep them consistent with the camera sampling frequency.

[0014] Step B, transmitting the camera, lidar and millimeter wave radar measurement data to the computing unit via CAN communication and Ethernet, using a time matching algorithm to directly eliminate the time difference of multiple sensor measurement data before fusion, and centralizing the multi-source information and realizing time synchronization using a time synchronization algorithm;

[0015] In step C, the camera, lidar and millimeter-wave radar are spatially calibrated to obtain the internal and external parameters of the camera and the conversion relationship between each coordinate system; through a series of coordinate transformations, the targets detected by the lidar and millimeter-wave radar are projected onto the camera image to obtain the target frame detected by the radar, thereby realizing the spatial fusion of multiple sensors.

[0016] Furthermore, eliminating data ambiguity in step 3 includes the following steps:

[0017] Step A: Threshold the measurement data obtained by the camera, lidar and millimeter wave radar, and use prior knowledge to roughly filter the data outside the threshold, reduce some impossible measurement and track pairing relationships, and obtain feasible measurement-track pairing;

[0018] Step B, forming a correlation matrix of the key point positions in the target rectangular frame detected by the camera, laser radar and millimeter wave radar to measure the matching degree of each point track with its corresponding track;

[0019] Step C: Comprehensively analyze all the tracks that fall into the correlation matrix, assign corresponding tracks to each track point according to the confidence level and assignment strategy to eliminate ambiguity.

[0020] Furthermore, the multi-target tracking in step 4 includes the following steps:

[0021] Step A, a multi-target tracking algorithm based on a random finite set, using a random finite set to represent the motion state of multiple targets and the measurement data of the sensor at each observation time, to estimate the number of multiple targets and the state of multiple targets;

[0022] Step B: Based on the actual needs of autonomous driving, the output variables of the observed tracking target are modeled and divided into state variables that participate in state updates and other non-state attributes that do not participate in updates;

[0023] In step C, based on the detection results of three different sensors, the same random finite set tracker is input and the tracking output results are fused with high confidence.

[0024] The present invention provides a multi-source information sensor fusion target detection method, which can avoid the limitations of a single sensor by fusing multiple sensors, improve the accuracy and precision of target recognition, alleviate the explosion of information dimensions and the pressure of vehicle-side computing power, and improve the redundancy and fault tolerance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is an overall flow chart of a multi-source information sensor fusion target detection method of the present invention;

[0026] Figure 2 This is the specific process of camera target detection in the present invention;

[0027] Figure 3 This is the specific process of laser radar target detection in the present invention;

[0028] Figure 4 This is the specific process of millimeter wave radar target detection in the present invention;

[0029] Figure 5 It is the specific process of spatiotemporal synchronization of multiple sensors in the present invention;

[0030] Figure 6 It is a specific process of synchronizing multiple sensor spaces in the present invention;

[0031] Figure 7 This is the technical route for multi-target tracking in dynamic scenes in the present invention. DETAILED DESCRIPTION

[0032] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0033] The present application provides a multi-source information sensor fusion target detection method, which can be applied to the environmental perception of autonomous driving vehicles, such as Figure 1The overall flow chart shown, the method comprises the following steps:

[0034] The processed image information and point cloud information of the detection target are obtained through multiple sensors on the car, that is, the detection information of the detection target is obtained through the on-board camera, lidar and millimeter-wave radar.

[0035] The specific process of camera target detection is as follows Figure 2 As shown in the figure, the specific steps are as follows: First, use the generative adversarial network (GAN) and some daytime scene pictures in the open source autonomous driving dataset to produce sufficient night scene pictures to increase the diversity of training data. Secondly, design a spatial transformation network based on the attention mechanism to spatially transform the spatial domain information in the camera pictures, retain the areas that need to be paid attention to, and maintain the rotation and scaling invariance of the target; finally, choose to input the lightweight version of YOLOv3 for forward reasoning, obtain target detection, semantic segmentation and other results, and reversely map them back to the original image.

[0036] The specific process of LiDAR target detection is as follows: Figure 3 As shown in the figure, the specific steps are as follows: first, based on the Graph modeling method, an undirected graph G={N,E} is established directly according to the three-dimensional point cloud, where N represents the nodes in the graph and E represents the edges between the nodes; secondly, the normal vector is used as the node feature for judgment. If the normal vectors of two adjacent nodes are similar, it means that the local plane where the two nodes are located is relatively smooth, and the two nodes should belong to the same object; finally, the feature similarity between adjacent nodes is used for clustering, and the ground segmentation method is used to segment obstacles such as vehicles, pedestrians and roadblocks.

[0037] The specific process of millimeter wave radar target detection is as follows Figure 4 As shown in the figure, the specific steps are as follows: preliminarily select valid targets, that is, screen the radar data obtained in this cycle, eliminate unreasonable or interfering data, and identify targets such as vehicles and pedestrians in the environment; secondly, according to the continuity characteristics of object motion, use the effective life cycle verification method to make continuity judgments on the motion parameters of the screened radar data in adjacent cycles and track multiple frames of data. Radar targets that pass the effective life cycle test and can be continuously tracked are valid targets.

[0038] In view of the differences in sampling period and spatial position of multiple sensors, the data of each sensor is down-converted and the coordinate system is converted, and the data collected by each sensor is aligned one by one in the time dimension and space dimension respectively, so as to achieve the time and space synchronization of multiple sensors. Figure 5As shown, it is mainly divided into time synchronization and space synchronization. Time synchronization is to unify each sensor to a reference standard time. If the time of each sensor cannot be synchronized, the subsequent information fusion cannot be carried out accurately. Space synchronization is to establish a transformation relationship from the different coordinate systems used by each sensor to a unified reference coordinate system, so that the data of each sensor can be effectively fused in a unified coordinate system. The specific steps are as follows:

[0039] Step 1: Down-sample the laser radar and millimeter-wave radar with higher frequencies by setting parameters, so that the radar and camera sampling frequencies are consistent;

[0040] Step 2: The camera, lidar and millimeter-wave radar measurement data are transmitted to the computing unit via CAN communication and Ethernet, and the time difference of the above sensor measurement data is directly eliminated by the time matching algorithm before fusion, and the multi-source information is concentrated in the ROS to achieve time synchronization using the time synchronization algorithm;

[0041] Step 3: Perform spatial calibration on the camera, LiDAR, and millimeter-wave radar to obtain the internal and external parameters of the camera and the conversion relationship between the coordinate systems. The specific process is as follows: Figure 6 As shown. Since the radar is installed on the vehicle body, the radar to vehicle body coordinate system is converted to the translation matrix L and the rotation matrix F, where the translation matrix L is the position of the origin of the radar coordinate system in the vehicle body coordinate system, and the rotation matrix is ​​determined by three Euler angles (ψ, θ, φ). The conversion relationship is shown in formula (1). Then, the vehicle coordinate system can be converted to the camera coordinate system by reverse calculation.

[0042]

[0043] Where: (X b ,Y b ,Z b ) is the coordinate in the vehicle body coordinate system; (X l ,Y l ,Z l ) is the coordinate in the radar coordinate system; (l x ,l y ,l z ) is the position of the origin of the radar coordinate system in the vehicle coordinate system; F(Ψ), F(θ), and F(φ) are the Euler angle rotation matrices.

[0044] The camera completes the conversion from the three-dimensional coordinate system to the two-dimensional image coordinate system. Although the distance information is lost in the conversion process, there is only one pixel corresponding to a point in the three-dimensional space, that is, for any laser radar data, there is also only one pixel corresponding to it in the image. Therefore, the laser radar and millimeter wave radar detection targets can be projected onto the camera image to obtain the radar detection target rectangular frame, which can realize the spatial fusion of the camera, laser radar and millimeter wave radar. The conversion relationship is shown in formula (2).

[0045]

[0046] Where: λ is the scaling factor, (u0,v 0, 1) represents the offset of the camera's principal point; α and β represent the horizontal and vertical focal lengths, which are determined by the camera's internal parameters; dx and dy represent the unit lengths of the projection point on the x-axis and y-axis of the image coordinate system; (u, v, 1) represents the coordinates of the projection point of the camera coordinate system on the image plane; R and t represent the rotation matrix and translation matrix, respectively, which are determined by the camera's external parameters; M is the projection matrix; (X s ,Y s ,Z s ,1) is the camera coordinate system.

[0047] Aiming at the problem that the association of multiple sensor and multi-target data is easy to be confused, the multiple sensor data association algorithm proposed in the present invention is adopted to eliminate data ambiguity, thereby making the target matching more accurate. The specific steps are as follows:

[0048] Step 1: In actual perception, the sensor measurement data will randomly generate some wild points with a small probability. These wild points seriously deviate from the change trend of the target data. Therefore, the measurement data of the camera, lidar and millimeter wave radar need to be thresholded. The screening formulas are shown in formulas (3) and (4). Make full use of prior knowledge to roughly filter the data outside the threshold, reduce some impossible measurement and track pairing relationships, and obtain feasible measurement-track pairing;

[0049] In the formula, Z i (k+1) is the interest value of the measurement point; Z i (k+1 / k) is the measured value Z i (k+1) the predicted value at the next moment based on the original change trend; r i (k+1) is Z i (k+1) The innovation value of the measurement point; H i (k+1) is the measurement matrix of the system at time k+1; H T i (k+1) is H i(k+1) transposed matrix; C is a constant, which is determined according to the system requirements. The smaller the value, the lower the threshold for discrimination; P i (k+1 / k) is the prediction covariance matrix of the system; R i (k+1) is the measurement noise covariance matrix; if formula (3) is established, then Z i (k+1) is the correct observation, otherwise it is an outlier. i (k+1 / k) instead.

[0050] Step 2, forming a correlation matrix of the key point positions in the target rectangular frame detected by the camera, laser radar and millimeter wave radar to measure the matching degree of each point track with its corresponding track;

[0051] Step 3: Comprehensively integrate all the tracks that fall into the correlation matrix, assign corresponding tracks to each track according to the confidence level and the assignment strategy to eliminate ambiguity. Assume that the sensor has established N tracks before the k+1th scan, and the k+1th measurement data is Z i (k+1), i=1,2…N, in the association matrix of the i-th track, the corresponding track is matched by the following formula (5):

[0052] Where: S ij -1 (k+1) is the system in R j (k+1) to get the covariance matrix; Obtained by formula (4); It is a metric for judging whether a point track is a target track. If it is greater than 1, it is a target track.

[0053] For multi-target tracking in dynamic scenes, the time sequence of the same moving object in different frames is correlated to form a continuous understanding of multiple targets, which is actually the system in R j (k+1) to get the covariance matrix; Obtained by formula (4); d ij 2 It is a metric to determine whether a point track is a target track. If it is greater than 1, it is a target track.

[0054] For multi-target tracking in dynamic scenes, the time sequence of the same moving object in different frames is associated to form a continuous understanding of multiple targets, and the position, speed, and orientation of the perceived targets are output in real time. The specific steps include:

[0055] Step 1: The multi-target tracking algorithm based on random finite sets regards the observation values ​​obtained by each sensor as a whole, and uses random finite sets to represent the motion state of multiple targets and the measurement data of sensors at each moment, so as to estimate the number and state of multiple targets. The core mechanism of random finite sets is the target hypothesis probability maintenance mechanism and the probability hypothesis label maintenance mechanism: the target hypothesis probability maintenance mechanism couples the target corresponding label and confidence, establishes possible states for all observations, and merges states according to the label hypothesis. States above the confidence threshold are filtered and output as a known target. The probability hypothesis label maintenance mechanism assigns a new label to each new state, models multiple hypotheses on the probability that the label is the same as the label of the existing target, and uses the Murty fast algorithm to obtain the most likely label.

[0056] Step 2: Model the output variables of the observed and tracked targets according to the actual needs of autonomous driving. The only observed variable detected by the lidar is the plane position, while the observed variables of the millimeter-wave radar include the plane position and speed. The modeling of the tracked target is divided into state variables that participate in state updates and other non-state attributes that do not participate in updates. State variables include plane position, speed, yaw angle, yaw angular velocity, and confidence. Other non-state variables include length, width, height, category, color, and name. Most of them do not change between different frames, but categories, etc. will also give different results with detection, so a continuous understanding is made in the time domain to prevent jumps between frames.

[0057] Step 3: Based on the detection results of three different sensors, the same random finite set tracker is input, and the tracking output results are fused based on the generalized covariance intersection algorithm (GCI) under the random finite set framework. This is a distributed fusion, and finally a fusion tracking result with higher confidence is obtained. The technical route is as follows: Figure 7 As shown in Figure 2, the most important variable in the fusion process is confidence. After detection, tracking, and fusion, a confidence evaluation index will be given, and the information of the observed target will be output based on the confidence.

[0058] The above is only a preferred specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field within the technical scope disclosed by the present invention, who makes equivalent replacement or change according to the technical solution and inventive concept of the present invention, shall be covered within the protection scope of the present invention. Outputting information such as the position, speed and direction of the perceived target at the same time specifically includes the following steps:

[0059] Step 1: The multi-target tracking algorithm based on random finite sets regards the observation values ​​obtained by each sensor as a whole, and uses random finite sets to represent the motion state of multiple targets and the measurement data of sensors at each moment, so as to estimate the number and state of multiple targets. The core mechanism of random finite sets is the target hypothesis probability maintenance mechanism and the probability hypothesis label maintenance mechanism: the target hypothesis probability maintenance mechanism couples the target corresponding label and confidence, establishes possible states for all observations, and merges states according to the label hypothesis. States above the confidence threshold are filtered and output as a known target. The probability hypothesis label maintenance mechanism assigns a new label to each new state, models multiple hypotheses on the probability that the label is the same as the label of the existing target, and uses the Murty fast algorithm to obtain the most likely label.

[0060] Step 2: Model the output variables of the observed and tracked targets according to the actual needs of autonomous driving. The only observed variable detected by the lidar is the plane position, while the observed variables of the millimeter-wave radar include the plane position and speed. The modeling of the tracked target is divided into state variables that participate in state updates and other non-state attributes that do not participate in updates. State variables include plane position, speed, yaw angle, yaw angular velocity, and confidence. Other non-state variables include length, width, height, category, color, and name. Most of them do not change between different frames, but categories, etc. will also give different results with detection, so a continuous understanding is made in the time domain to prevent jumps between frames.

[0061] Step 3: Based on the detection results of three different sensors, the same random finite set tracker is input, and the tracking output results are fused based on the generalized covariance intersection algorithm (GCI) under the random finite set framework. This is a distributed fusion, and finally a fusion tracking result with higher confidence is obtained. The technical route is as follows: Figure 7 As shown in Figure 2, the most important variable in the fusion process is confidence. After detection, tracking, and fusion, a confidence evaluation index will be given, and the information of the observed target will be output based on the confidence.

[0062] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A multi-source information sensor fusion target detection method, characterized in that: The following steps are involved: Step 1, obtaining processed image information and point cloud information of the detection target through multiple sensors on the car; Step 2: based on the differences in sampling period and spatial position of multiple sensors, the data of each sensor is down-converted and the coordinate system is converted to achieve spatiotemporal synchronization of multiple sensors; Step 3: To solve the problem of confusion in the association of target data from multiple sensors, a multiple sensor data association algorithm is used to eliminate data ambiguity. Step 4: Track multiple targets in a dynamic scene, associate the time sequence of the same moving object in different frames, form a continuous understanding of multiple targets, and output the perceived target position, speed and orientation information in real time.

2. A multi-source information sensor fusion target detection method according to claim 1, characterized in that: The multiple sensors in step 1 include: 1 camera, 1 laser radar and 1 millimeter wave radar.

3. A multi-source information sensor fusion target detection method according to claim 1, characterized in that: The spatiotemporal synchronization of multiple sensors in step 2 includes the following steps: Step A: Down-clock the laser radar and millimeter-wave radar through parameter setting to keep consistent with the camera sampling frequency; Step B, transmitting the camera, lidar and millimeter wave radar measurement data to the computing unit via CAN communication and Ethernet, using a time matching algorithm to directly eliminate the time difference of multiple sensor measurement data before fusion, and centralizing the multi-source information and realizing time synchronization using a time synchronization algorithm; In step C, the camera, lidar and millimeter-wave radar are spatially calibrated to obtain the internal and external parameters of the camera and the conversion relationship between each coordinate system; through a series of coordinate transformations, the targets detected by the lidar and millimeter-wave radar are projected onto the camera image to obtain the target frame detected by the radar, thereby realizing the spatial fusion of multiple sensors.

4. A multi-source information sensor fusion target detection method according to claim 1, characterized in that: Eliminating data ambiguity in step 3 includes the following steps: Step A: Threshold the measurement data obtained by the camera, lidar and millimeter wave radar, and roughly filter the data outside the threshold using prior knowledge to reduce some impossible measurement and track pairing relationships and obtain feasible measurement and track pairing; Step B, forming a correlation matrix of the key point positions in the target rectangular frame detected by the camera, laser radar and millimeter wave radar to measure the matching degree of each point track with its corresponding track; Step C: Comprehensively analyze all the tracks that fall into the correlation matrix, assign corresponding tracks to each track point according to the confidence level and assignment strategy to eliminate ambiguity.

5. A multi-source information sensor fusion target detection method according to claim 1, characterized in that: The multi-target tracking in step 4 includes the following steps: Step A, a multi-target tracking algorithm based on a random finite set, using a random finite set to represent the motion state of multiple targets and the measurement data of the sensor at each observation time, to estimate the number of multiple targets and the state of multiple targets; Step B: Based on the actual needs of autonomous driving, the output variables of the observed tracking target are modeled and divided into state variables that participate in state updates and other non-state attributes that do not participate in updates; In step C, based on the detection results of three different sensors, the same random finite set tracker is input and the tracking output results are fused with high confidence.

Citation Information

Cited By

  • Multi-mode automatic driving three-dimensional target tracking method based on space-time fusion

    CN119274167A

  • Universe vehicle track detection method based on multi-source data fusion

    CN120009876A