A data fusion method and a perception fusion system

Abstract

The specification discloses a data fusion method and a perception fusion system, the method comprising: receiving current frame target information acquired by a laser radar, a millimeter wave radar and a camera from an environment, converting target information data of the millimeter wave radar and the camera to a coordinate system of the laser radar, and storing the current frame target information through a linked list; associating and matching the current frame target information with previous frame target information, the association and matching comprising matching of adjacent two frame data of a same sensor and matching of adjacent two frame data between different sensors; in a case where a matching result of the adjacent two frame data is a same target, performing data fusion of current frame target information and previous frame target information of the target according to a Kalman filtering algorithm; updating target information of each target in the linked list according to the fused data, converting the target information in the linked list into a preset format, and sending the target information in the preset format to a server.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology for automobiles and roadside perception equipment, and more specifically, to a data fusion method and a perception fusion system. Background Technology

[0002] With the continuous development of intelligent driving technology, increasingly higher demands are being placed on environmental perception. Currently, intelligent driving technology primarily uses visual cameras, millimeter-wave radar, and lidar for environmental perception. Visual cameras are generally used for target identification and detection, and also for positioning, but their positioning accuracy is low and easily affected by lighting conditions. Millimeter-wave radar obtains the target's position and velocity by sending electromagnetic waves to the target and receiving the echoes; it can operate in all weather conditions, but its accuracy is lower than lidar. Lidar obtains the target's position, velocity, and heading angle through laser scanning; it has high accuracy and good stability, but is greatly affected by weather changes and is costly. It can be seen that each of the three types of sensors has its advantages and disadvantages; using only a single sensor for environmental perception results in low accuracy and poor robustness. Therefore, how to achieve accurate environmental perception through the fusion of information from multiple sensors is a primary problem to be solved in the field of intelligent driving. Currently, research on multi-sensor perception fusion mainly focuses on the theoretical level, and commonly used fusion algorithms include: Kalman filtering, DS evidence reasoning, multi-Bayes estimation, and neural networks. Although fusion algorithms are mature, there are few systems that implement multi-sensor fusion in intelligent driving engineering, and no multi-sensor fusion system has yet been applied to roadside equipment. Existing multi-sensor fusion solutions generally can only fuse data from a limited number of sensors, and the data association methods are relatively simple. At the engineering level, there is currently no complete multi-sensor fusion system applied to roadside equipment.

[0003] Therefore, there is an urgent need to study a multi-sensor fusion method and system to achieve the fusion of data from multiple sensors and improve the accuracy of perception. Summary of the Invention

[0004] This specification provides a data fusion method and a perception fusion system to overcome at least one technical problem existing in related technologies.

[0005] According to a first aspect of the embodiments of this specification, a data fusion method is provided, comprising: receiving current frame target information acquired from the environment by a lidar, a millimeter-wave radar, and a camera; converting the target information data of the millimeter-wave radar and the camera to the coordinate system of the lidar; storing the current frame target information through a linked list, the current frame target information including the target's category, position information, velocity information, and current timestamp; associating and matching the current frame target information with the previous frame target information, the association matching including matching adjacent frames of data from the same sensor and matching adjacent frames of data from different sensors; when the matching result of adjacent frames of data is the same target, fusing the current frame target information of the target with the previous frame target information according to a Kalman filter algorithm, the data fusion including data fusion of the target's position information and data fusion of its velocity information; updating the target information of each target in the linked list according to the fused data; converting the target information in the linked list into a preset format; and sending the target information to a server in the preset format.

[0006] Optionally, the step of storing the target information of the current frame through a linked list includes: identifying the target based on the target information of the current frame, creating a track for each target, wherein the track is used to store the target's category, location information, speed information and target timestamp; and storing all tracks in a linked list.

[0007] Optionally, the step of associating and matching the target information of the current frame with the target information of the previous frame includes the steps of matching adjacent frames of data from the same sensor and matching adjacent frames of data from different sensors. This includes: associating and matching the detected targets between two consecutive frames of data from the LiDAR, associating and matching the detected targets of the LiDAR with the detected targets of the millimeter-wave radar in two consecutive frames of data, and associating and matching the detected targets of the millimeter-wave radar between two consecutive frames of data. The goal is to determine whether the targets are the same by judging the relationship between the distances between the targets identified in adjacent frames of data and a preset first threshold range; for the camera... The association matching between the detected targets in two frames of data is based on the bounding boxes of each detected target identified by the camera model in each frame of data. The boundary boxes of each target identified in the two frames of data are compared to determine whether they are the same target. For the association matching between the detected targets of the LiDAR and the detected targets of the camera in two frames of data, the detection model of the LiDAR identifies the three-dimensional bounding boxes of each detected target in each frame of data and the camera model identifies the bounding boxes of each detected target in each frame of data. The range of the three-dimensional bounding boxes and the bounding boxes of each target identified in the two frames of data is compared to determine whether they are the same target.

[0008] Optionally, the step of determining whether a target is the same target by judging the relationship between the distance between targets identified in two consecutive frames of LiDAR data and the detection targets of millimeter-wave radar in two consecutive frames of data, and the correlation matching between targets detected in two consecutive frames of millimeter-wave radar data, includes:

[0009] Based on the pose data of each detected target in the previous frame, the pose data of each detected target in the next frame is predicted. The pose data includes position, velocity, and direction of travel. For each detected target, the position of the detected target in the previous frame and the predicted position of the target in the next frame are used as the foci of an ellipse. The associated region is delineated according to the set eccentricity of the ellipse, as shown in Equation (1.1):

[0010] (1.1);

[0011] in, Indicates eccentricity. Indicates the speed of the target;

[0012] Compare the positions of each detected target in the current frame data with the elliptical ranges corresponding to each detected target in the previous frame. If the position of a detected target in the current frame data is within the elliptical range of a detected target in the previous frame, determine that the detected targets corresponding to the two frames of data are the same target.

[0013] Optionally, the step of determining whether the detected targets in two consecutive frames of camera data are the same target by comparing the bounding boxes of the targets identified in each frame of data using the camera model, includes:

[0014] Based on the first bounding box of each detected target detected by the camera model in the previous frame and the second bounding box of each detected target detected by the current frame, when the first bounding box and the second bounding box intersect, the intersection and union of the first bounding box and the second bounding box are calculated, and the area ratio of the intersection and the union is calculated.

[0015] If the area ratio meets the preset second threshold range, it is determined that the detection target corresponding to the first bounding box and the detection target corresponding to the second bounding box are the same target in the two frames of data.

[0016] Optionally, the step of determining whether a target is the same as another target by comparing the ranges of the 3D bounding boxes and the bounding boxes of the targets identified by the LiDAR and the camera in the two consecutive frames of data, based on the LiDAR detection model identifying the 3D bounding boxes of each target in each frame of data and the camera model identifying the bounding boxes of each target in each frame of data, includes:

[0017] Based on the 3D bounding boxes of each detected target identified by the lidar and the 2D bounding boxes of each detected target identified by the camera, the corner coordinates of the 3D bounding boxes are projected onto the 2D image according to a mapping relationship. The mapping relationship is obtained through joint calibration of the camera and lidar, as shown in Equation (1.2):

[0018] (1.2);

[0019] Where K is the intrinsic parameter matrix of the camera. The extrinsic parameter matrix of the camera is obtained through joint calibration; , The focal length of the camera is measured in pixels along the x and y directions; , The pixel coordinates of the image center; This represents the coordinates of the point along the Z-axis in the camera coordinate system. Let be the coordinates of a point in the lidar coordinate system. These are the coordinates of the point in the pixel coordinate system.

[0020] Calculate the number of corner points of the three-dimensional bounding box of the lidar contained in the two-dimensional bounding box of each detected target identified by the camera;

[0021] If the number of corner points within the two-dimensional bounding box is greater than a preset third threshold, the detection target corresponding to the two-dimensional bounding box and the detection target corresponding to the three-dimensional bounding box are determined to be the same target.

[0022] Optionally, when the matching result of two adjacent frames is the same target, the target information of the current frame is fused with the target information of the previous frame according to the Kalman filter algorithm. The data fusion includes the steps of data fusion of the target information's position information and data fusion of its velocity information, including:

[0023] The prior state estimation matrix and covariance matrix of the current frame are calculated using the optimal state estimation of the previous frame, as shown in equations (2.1) and (2.2):

[0024] (2.1);

[0025] (2.2);

[0026] in, This is used to describe the target's position and velocity; This is the state transition matrix for the objective. The control matrix represents the control input. The effect on the current state; P is the covariance matrix, representing the uncertainty of the target state at different times; Q is the process noise covariance, representing the noise of the model itself;

[0027] The Kalman coefficients are calculated as shown in equation (2.3):

[0028] (2.3);

[0029] in, , is the observation matrix; The covariance matrix of the measured values;

[0030] A posterior state estimation equation is established, and data fusion is performed, as shown in equation (2.4):

[0031] (2.4);

[0032] in, This is the current measurement value;

[0033] The optimal covariance matrix is ​​then updated, as shown in equation (2.5):

[0034] (2.5);

[0035] Optionally, the step of updating the target information of each target in the linked list based on the fused data, converting the target information in the linked list into a preset format, and sending the target information to the server in the preset format includes: when updating the target information in the linked list based on the fused data, the target category is based on the camera, the target position information is based on the position data of the lidar, the target velocity information is based on the velocity data fused from the lidar and millimeter-wave radar, and the target timestamp is consistent with the timestamp of the most recent frame of data that updated the target information; if the difference between the target timestamp and the current timestamp is greater than a set fourth threshold, the target is deleted from the linked list; and the linked list is... Each target number in the list is used to obtain a unique ID for each target. The location information of each target in the linked list is converted into coordinates in the GPS coordinate system. The length, width, and height of each target are based on the length, width, and height obtained by the lidar, and the driving direction of each target is based on the driving direction obtained by the lidar. The ID, location information, speed information, category, length, width, height, and driving direction of each target are sent to the server. Before sending the ID, location information, speed information, category, length, width, height, and driving direction of each target to the server, the difference between the target's timestamp and the current timestamp is verified again to see if it is greater than the set fourth threshold. If it is greater than the set fourth threshold, the target is deleted.

[0036] According to a second aspect of the embodiments of this specification, a perception fusion system is provided, comprising: a lidar, a camera, a millimeter-wave radar, a development board, a PC host, and a server, wherein the lidar and millimeter-wave radar are directly connected to the PC host via network cables; the camera is connected to the development board, and the development board is connected to the PC host via a network cable; the PC host is connected to the server via optical fiber; the lidar, millimeter-wave radar, and camera acquire target information in the environment and transmit the target information to the PC host, wherein the lidar and millimeter-wave radar transmit the target information to the PC host via Ethernet, the camera transmits the target information to the development board, the development board encodes the target information from the camera and transmits it to the PC host via Ethernet, the targets include vehicles and pedestrians, and the target information includes the target type, position, speed, and direction of movement; the PC host executes the data fusion method as described above; and the server receives and stores the data from the PC host.

[0037] Optionally, the number of lidars is one, the number of cameras is three, including a first camera, a second camera and a third camera, the number of millimeter-wave radars is three, and the number of development boards is two, including a first development board and a second development board. The first camera and the second camera are connected to the first development board, and the third camera is connected to the second development board. The cameras transmit target information to the development boards in a serialized stream.

[0038] The beneficial effects of the embodiments in this specification are as follows:

[0039] This specification provides a data fusion method and a perception fusion system. The method correlates and matches consecutive frames of target information acquired from the environment by LiDAR, millimeter-wave radar, and cameras. It achieves matching of adjacent frames from the same sensor and adjacent frames from different sensors. When the matching result of adjacent frames indicates the same target, the target information from consecutive frames is fused using a Kalman filter algorithm. The stored target information for each target is then updated based on the fused data. The system includes a LiDAR, camera, millimeter-wave radar, development board, PC, and server. It acquires multi-sensor data through LiDAR, camera, and millimeter-wave radar. The PC processes the target information using the data fusion method and finally sends it to the server. This specification embodiment implements the acquisition, fusion, and transmission of multi-sensor data on V2X roadside equipment. Compared to the single data association method in existing multi-sensor fusion schemes, this method better integrates the performance advantages of each sensor, considers the characteristics of different sensors, and uses multiple data association methods to correlate and match data from different sensors. Different association algorithms are used between different sensors, thereby obtaining more accurate target information. In addition, compared to existing technologies that can only fuse data from a limited number of sensors, this method can fuse data from up to seven sensors, increasing the sensing range and improving the accuracy of sensing. It also achieves complementarity and unification of information from different types of sensors, providing reliable support and effective protection for the driving safety of intelligent vehicles.

[0040] The technical effects of the embodiments in this specification include at least the following:

[0041] 1. In this specification, the system includes: a LiDAR, a camera, a millimeter-wave radar, a development board, a PC host, and a server. The LiDAR, millimeter-wave radar, and camera acquire target information in the environment and transmit the target information to the PC host. The PC host fuses the target information using a fusion algorithm to obtain fused data, and then sends the fused data to the server. This system architecture is one of the technical effects of the embodiments in this specification.

[0042] 2. In this specification, the data fusion method employs different association algorithms based on sensor type, fully leveraging the performance advantages of each sensor and ensuring the accuracy of data fusion. This is one of the technical effects of the embodiments in this specification.

[0043] 3. In this specification, the data fusion method uses the camera's recognition result as the standard because the target category identified by the camera's algorithm is more accurate than that of other sensors. Since the lidar can obtain highly accurate position and velocity information, and the millimeter-wave radar can obtain relatively accurate velocity information, when updating the fused target state, the position is based on the lidar, and the velocity is based on both the lidar and millimeter-wave radar. This ensures the accuracy of the data fusion from the seven sensors, not only increasing the sensing range but also improving the accuracy of perception. This is one of the technical effects of the embodiments in this specification. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments or related technologies of this specification, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A schematic flowchart illustrating a data fusion method provided in one embodiment of this specification;

[0046] Figure 2 This is a schematic diagram illustrating the calculation of IoU as provided in one embodiment of this specification;

[0047] Figure 3 A schematic diagram of corner points and bounding boxes provided for one embodiment of this specification;

[0048] Figure 4 This is a schematic diagram of the structure of a perception fusion system provided in one embodiment of this specification. Detailed Implementation

[0049] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and drawings of this specification are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0051] This specification discloses a data fusion method and a perception fusion system, which will be described in detail below.

[0052] Figure 1 This is a flowchart illustrating a data fusion method provided in one embodiment of this specification. Figure 1 As shown, a data fusion method includes:

[0053] S110: Receive the current frame target information obtained from the environment by the lidar, millimeter-wave radar, and camera; convert the target information data of the millimeter-wave radar and camera to the coordinate system of the lidar; store the current frame target information through a linked list; the current frame target information includes the target's category, position information, velocity information, and current timestamp.

[0054] Before data fusion, it is necessary to first clarify the type of data detected by each sensor. The sensors used in the embodiments of this specification can detect the type, quantity, location, and velocity of the target, while lidar and millimeter-wave radar can also obtain information such as the target's acceleration and direction of travel. Therefore, after receiving the transmitted data, the terminal needs to preprocess the target data collected by each sensor. This preprocessing includes decoding the target information, coordinate system transformation, and time synchronization.

[0055] In this system, the world coordinate system coincides with the lidar coordinate system.

[0056] Specifically, the step of storing the current frame target information through a linked list includes:

[0057] S1102. Identify targets based on the target information in the current frame, and create a track for each target. The track is used to store the target's category, location information, speed information, and target timestamp.

[0058] S1104. Store all tracks in a linked list.

[0059] During preprocessing, relevant data from the millimeter-wave radar and camera, such as position, velocity, acceleration, and direction of travel, need to be converted to the lidar coordinate system. Finally, the processed data from each sensor is stored in a unified format (struct) in a linked list obj_m (the first linked list).

[0060] Real-time data fusion is performed on the collected data. First, a tracking module is created to track each target in the scene. The algorithm creates a track for each target object in the scene (one target to one track), and each track stores historical information about the corresponding target detected by each sensor. All tracks are stored in a linked list `track_list` (the second linked list). Then, a corresponding data association algorithm is used to compare the targets in `obj_m` (the first linked list) with the targets in `track_list` (the second linked list). When both represent the same object, the fusion module uses a Kalman fusion algorithm to perform the fusion.

[0061] The multi-sensor data fusion algorithm in the embodiments of this specification includes: a data association algorithm and a data fusion algorithm. The specific process of the data association algorithm is as shown in step S120.

[0062] S120. The target information of the current frame is associated and matched with the target information of the previous frame. The association and matching includes the matching of adjacent two frames of data from the same sensor and the matching of adjacent two frames of data from different sensors.

[0063] The data association algorithm achieves target data matching between different frames, with different association algorithms used between different sensors. The association algorithms in this scheme are mainly divided into two categories: distance-based association algorithms and IoU (Intersection over Union)-based association algorithms. Distance-based association algorithms are used for association of LiDAR and millimeter-wave radar data, while IoU-based association algorithms are used for association of LiDAR and camera data. Intersection over Union (IoU) is used in the field of target detection to evaluate the degree of overlap between the model's predicted bounding box and the ground truth bounding box. This specification extends this concept to determine the matching degree of detection results between two adjacent frames.

[0064] In a specific embodiment, step S120, which involves associating and matching the current frame target information with the previous frame target information, includes the steps of matching adjacent frames of data from the same sensor and matching adjacent frames of data from different sensors, and includes:

[0065] S122. For the correlation matching between each detected target in the two consecutive frames of data of the lidar, the correlation matching between each detected target of the lidar and each detected target of the millimeter-wave radar in the two consecutive frames of data, and the correlation matching between each detected target of the millimeter-wave radar in the two consecutive frames of data, the relationship between the distance between each target identified in two adjacent frames of data and the preset first threshold range is used to determine whether they are the same target.

[0066] Distance-based association algorithms determine whether a target matches by comparing the distance between targets identified in two adjacent frames with a set threshold. If the distance between two targets in two frames is less than the set threshold, the two targets are considered a match.

[0067] In one specific implementation, step S122, which involves determining whether targets identified in two consecutive frames of LiDAR data are the same target by judging the relationship between the distance between targets identified in adjacent frames of data and a preset first threshold range, includes:

[0068] S1222. Based on the pose data of each detected target in the previous frame, predict the pose data of each detected target in the next frame. The pose data includes position, velocity, and direction of travel. For each detected target, the position of the detected target in the previous frame and the predicted position of the target in the next frame are used as the foci of an ellipse. The associated region is delineated according to the set eccentricity of the ellipse, as shown in Equation (1.1):

[0069] (1.1);

[0070] in, Indicates eccentricity. This indicates the speed of the target.

[0071] The association region of this algorithm is elliptical. When the target's velocity is less than 20 m / s, the eccentricity of the ellipse is 0, and a distance-based circular association algorithm is used. When the target's velocity is greater than or equal to 20 m / s, the eccentricity of the ellipse increases with the velocity. The greater the target's velocity, the less likely the target is to deviate from its velocity direction in the next moment; therefore, the eccentricity is appropriately increased to flatten the ellipse.

[0072] S1224. Compare the position of each detected target in the current frame data with the elliptical range corresponding to each detected target in the previous frame. If the position of a detected target in the current frame data is within the elliptical range of a detected target in the previous frame, determine that the detected targets corresponding to the two frames of data are the same target.

[0073] In this specification, distance-based association algorithms are used for matching LiDAR-detected targets between two consecutive frames, matching LiDAR-detected targets with millimeter-wave radar-detected targets, and matching millimeter-wave radar-detected targets with each other.

[0074] S124. For the association matching between each detected target in two consecutive frames of camera data, based on the bounding boxes of each detected target identified by the camera model in each frame of data, the bounding boxes of each target identified in the two consecutive frames of data are compared to determine whether they are the same target.

[0075] Figure 2 This is a schematic diagram illustrating the calculation of IoU according to an embodiment of this specification. Figure 2 As shown, the calculation of IoU involves the intersection and union of the first bounding box 201 and the second bounding box 202. When the first bounding box 201 and the second bounding box 202 intersect, the IoU defined in this embodiment can be used as an indicator to measure the degree of overlap between the first and second bounding boxes. Since the detection results from the camera model are bounding boxes, the matching of camera-detected targets between two frames is performed by directly calculating the IoU to determine whether the two targets match. If the IoU between two targets between two frames is less than a set threshold, then the two targets are considered to match.

[0076] In one specific implementation, step S124, the association matching between detected targets in two consecutive frames of camera data, based on the bounding boxes of each detected target identified by the camera model in each frame of data, and determining whether they are the same target by comparing the bounding boxes of each target identified in the two consecutive frames of data, includes:

[0077] S1242. Based on the first bounding box of each detected target detected by the camera model in the previous frame and the second bounding box of each detected target detected by the current frame, when the first bounding box and the second bounding box intersect, calculate the intersection and union of the first bounding box and the second bounding box, and calculate the area ratio of the intersection and the union.

[0078] S1244. If the area ratio meets the preset second threshold range, determine that the detection target corresponding to the first bounding box and the detection target corresponding to the second bounding box are the same target in the two frames of data.

[0079] S126. For the association matching of each detected target of the LiDAR and each detected target of the camera in two consecutive frames of data, the detection model based on the LiDAR identifies the three-dimensional bounding box of each detected target based on each frame of data and the bounding box of each detected target identified by the camera model based on each frame of data, and determines whether they are the same target by comparing the range of the three-dimensional bounding box and the bounding box of each target identified in the two consecutive frames of data.

[0080] The detection results of the LiDAR-based model are 3D bounding boxes, while the detection results of the camera-based model are bounding boxes. The correlation between the two includes the mapping between 2D and 3D and the data association after mapping. For the correlation, firstly, the corner coordinates of the LiDAR bounding box are mapped onto the image, and the mapping relationship is obtained through joint calibration of the camera and LiDAR; then, the number of corner points of the LiDAR bounding box within the bounding box detected by the camera is calculated. If the number of corner points within the bounding box is greater than a set threshold, the two targets are considered to match.

[0081] In one specific implementation, step S126, the association matching between each detected target of the LiDAR and each detected target of the camera in two consecutive frames of data, and the step of determining whether they are the same target by comparing the range of the 3D bounding boxes and the bounding boxes of each detected target identified by the LiDAR detection model based on each frame of data and the bounding boxes of each detected target identified by the camera model based on each frame of data, includes:

[0082] S1262. Based on the three-dimensional bounding boxes of each detected target identified by the lidar and the two-dimensional bounding boxes of each detected target identified by the camera, the corner coordinates of the three-dimensional bounding boxes are projected onto the two-dimensional image according to a mapping relationship. The mapping relationship is obtained through joint calibration of the camera and lidar, as shown in Equation (1.2):

[0083] (1.2);

[0084] Where K is the intrinsic parameter matrix of the camera. The extrinsic parameter matrix of the camera is obtained through joint calibration; , The focal length of the camera is measured in pixels along the x and y directions; , The pixel coordinates of the image center; This represents the coordinates of the point along the Z-axis in the camera coordinate system. Let be the coordinates of a point in the lidar coordinate system. This represents the coordinates of the point in the pixel coordinate system.

[0085] S1264. Calculate the number of corner points of the three-dimensional bounding box of the lidar included in the two-dimensional bounding box of each detected target identified by the camera.

[0086] The eight corner points of the 3D bounding box of the LiDAR recognition result are transformed to the pixel coordinate system, and then the number of the eight corner points inside the rectangular bounding box recognized by the camera is calculated.

[0087] Figure 3This is a schematic diagram of corner points and bounding boxes provided for one embodiment of this specification. For example... Figure 3 As shown, the corner points of the two-dimensional bounding box 310 are ABCD, and E is the two-dimensional coordinate of a corner point of the three-dimensional bounding box 320 of the LiDAR. To determine whether point E is inside rectangle ABCD, it is only necessary to determine...

[0088] ;

[0089] S1266. When the number of corner points within the two-dimensional bounding box is greater than a preset third threshold, the detection target corresponding to the two-dimensional bounding box and the detection target corresponding to the three-dimensional bounding box are determined to be the same target.

[0090] For example, if 6 out of 8 corner points are within the rectangle recognized by the camera, then the LiDAR data is considered to match the camera data.

[0091] Since the information collected by a multi-sensor system includes redundant and complementary information from different sensors, a fusion algorithm is designed and implemented to obtain consistent and comprehensive information. The specific implementation of the data fusion algorithm in the embodiments of this specification is as follows: step S130.

[0092] S130. When the matching result of two adjacent frames is the same target, the target information of the current frame of the target is fused with the target information of the previous frame according to the Kalman filter algorithm. The data fusion includes the data fusion of the position information and the data fusion of the velocity information of the target information.

[0093] In a specific embodiment, step S130, when the matching result of two adjacent frames of data is the same target, performs data fusion between the target information of the current frame and the target information of the previous frame according to the Kalman filter algorithm. The data fusion includes the steps of data fusion of the target information's position information and data fusion of its velocity information, including:

[0094] S132. Calculate the prior state estimation matrix and covariance matrix of the current frame using the optimal state estimation of the previous frame, as shown in equations (2.1) and (2.2):

[0095] (2.1);

[0096] (2.2);

[0097] in, This is used to describe the target's position and velocity; This is the state transition matrix for the objective. The control matrix represents the control input. The effect on the current state; P is the covariance matrix, representing the uncertainty of the target state at different times; Q is the process noise covariance, representing the noise of the model itself;

[0098] The Kalman coefficients are calculated as shown in equation (2.3):

[0099] (2.3);

[0100] in, , is the observation matrix; The covariance matrix of the measured values;

[0101] S134. Establish the posterior state estimation equation and perform data fusion, as shown in equation (2.4):

[0102] (2.4);

[0103] in, This is the current measurement value;

[0104] The optimal covariance matrix is ​​then updated, as shown in equation (2.5):

[0105] (2.5);

[0106] The above formula can be used to fuse the target's position and velocity. After data fusion, it is necessary to design update strategies for the fused target position and velocity based on different situations, and then store the fused information.

[0107] S140. Update the target information of each target in the linked list according to the data obtained by fusion, convert the target information in the linked list into a preset format, and send the target information to the server in the preset format.

[0108] In a specific embodiment, step S140, updating the target information of each target in the linked list according to the fused data, converting the target information in the linked list into a preset format, and sending the target information to the server in the preset format, includes:

[0109] S142. When updating the target information in the linked list based on the fused data, the target category is based on the camera, the target position information is based on the position data of the lidar, the target velocity information is based on the velocity data fused from the lidar and millimeter-wave radar, and the target timestamp is consistent with the timestamp of the most recent frame of data that updated the target information.

[0110] Because lidar can acquire high-precision position and velocity information, and millimeter-wave radar can acquire relatively accurate velocity information, when updating the fused target state, the position is based on lidar data, and the velocity is based on both lidar and millimeter-wave radar data. When both the current and historical values ​​of the target are relatively accurate, or neither is accurate, the information of the fused target is the information obtained by fusing the two; when one of the current and historical values ​​of the target is relatively accurate, the fused target adopts the more accurate value. The update logic of the linked list can be summarized in Table 1.

[0111] ;

[0112] The first row of the table indicates the sensor type of the historical target information stored in the track list (second linked list), and the first column indicates the sensor type of the current target information stored in obj_m (first linked list). Together, they determine the position and velocity information adopted by the fused target. In the table, pos represents position; vel represents velocity; fused_pos represents the fused position after fusing the current position and historical position corresponding to the current row and column; fused_vel represents the fused velocity after fusing the current velocity and historical velocity corresponding to the current row and column; lidar_c, radar_c, and camera_c represent the current data from the lidar, millimeter-wave radar, and camera, respectively; lidar_h and radar_h represent the historical data from the lidar and millimeter-wave radar, respectively. Taking pos=lidar_h.pos as an example, this equation means using the historical lidar position as the position of the fused target. pos=fused_vel means using the fused velocity as the velocity of the fused target.

[0113] The second linked list only stores historical information from LiDAR and millimeter-wave radar because cameras, unlike other sensors, do not have range or velocity measurement capabilities. The location information of targets identified by the camera can only be obtained through estimation algorithms or by mapping to LiDAR data. The location information obtained in both methods is not very accurate, and fusing it with the more accurate locations from other sensors will actually worsen the results. Camera data serves two main purposes: first, the target categories identified by camera-based algorithms are more accurate than those from other sensors, so the final target category information is based on the camera's data; second, the camera has higher resolution, enabling it to identify more targets. When neither LiDAR nor millimeter-wave radar identifies a particular target, camera data plays a crucial role.

[0114] S144. If the difference between the target's timestamp and the current timestamp is greater than the set fourth threshold, remove the target from the linked list.

[0115] The fusion module also has a timeout removal mechanism, which means that when the difference between the current timestamp and the timestamp of the track target is greater than a set threshold, the track is removed from the track_list to ensure the timeliness of tracking.

[0116] S146. Obtain the unique ID of each target in the linked list by assigning it a number. Convert the position information of each target in the linked list into coordinates in the GPS coordinate system. The length, width, and height of each target are based on the length, width, and height obtained by the lidar, and the driving direction of each target is based on the driving direction obtained by the lidar. Send the ID, position information, speed information, category, length, width, height, and driving direction of each target to the server. After processing and integration, the final target information sent to the server is shown in Table 2.

[0117] ;

[0118] S148. Before sending the ID, location information, speed information, category, length, width, height and driving direction of each target to the server, verify again whether the difference between the target's timestamp and the current timestamp is greater than the set fourth threshold. If it is greater than the set fourth threshold, delete the target.

[0119] Before sending the fused data to the server, the data needs to be processed and integrated in the final stages, such as converting the location information of the fused target into GPS coordinates, assigning a number to the target, and re-verifying whether the tracking target disappearance time has expired.

[0120] Figure 4 This is a schematic diagram of the structure of a sensory fusion system provided in one embodiment of this specification. Figure 4 As shown, a perception fusion system includes: a lidar, a camera, a millimeter-wave radar, a development board, a PC host 119, and a server 121.

[0121] In a specific embodiment, there is one lidar 101, three cameras including a first camera 103, a second camera 105, and a third camera 107, three millimeter-wave radars including a first millimeter-wave radar 109, a second millimeter-wave radar 111, and a third millimeter-wave radar 113, and two development boards including a first development board 115 and a second development board 117. The first camera 103 and the second camera 105 are connected to the first development board 115, and the third camera 107 is connected to the second development board 117. The cameras transmit target information to the development boards in a serialized stream.

[0122] The lidar and millimeter-wave radar are directly connected to the PC host 119 via network cables; the camera is connected to the development board, which is connected to the PC host 119 via a network cable; the PC host 119 is connected to the server 121 via optical fiber.

[0123] The lidar, millimeter-wave radar, and camera acquire target information in the environment and transmit the target information to the PC host 119. The targets include vehicles and pedestrians, and the target information includes the target type, location, speed, and direction of movement. The lidar and millimeter-wave radar transmit the target information to the PC host 119 via Ethernet, and the camera transmits the target information to the development board. The development board encodes the target information from the camera and transmits it to the PC host 119 via Ethernet.

[0124] PC host 119 executes the data fusion method described above.

[0125] The server receives and stores data from the PC host.

[0126] The development board in question is the A6 development board.

[0127] In summary, the embodiments of this specification provide a data fusion method and a perception fusion system, which realizes the acquisition, fusion, and transmission of multi-sensor data on V2X roadside equipment. Compared with the single data association method in existing multi-sensor fusion schemes, the data fusion method of the embodiments of this specification better integrates the performance advantages of each sensor, considers the characteristics of different sensors, and uses multiple data association methods to associate and match the data of different sensors. Different association algorithms are used between different sensors to obtain more accurate target information, realize the complementarity and unification of information from different types of sensors, and achieve the fusion of data from up to at least 7 sensors, thereby increasing the perception range, improving the accuracy of perception, and providing reliable support and effective protection for the driving safety of intelligent vehicles.

[0128] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0129] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data fusion method characterized by, The method comprises the following steps: receiving current frame target information acquired by a laser radar, a millimeter wave radar and a camera from an environment, converting target information data of the millimeter wave radar and the camera to a coordinate system of the laser radar, and storing the current frame target information through a linked list, wherein the current frame target information comprises a category, position information, speed information and a current timestamp of a target; performing associated matching on the current frame target information and previous frame target information, wherein the associated matching comprises matching of adjacent two frame data of a same sensor and matching of adjacent two frame data between different sensors, wherein the matching of adjacent two frame data between different sensors comprises matching of each detected target in adjacent two frame data of the laser radar and the millimeter wave radar, matching of each detected target in adjacent two frame data of the laser radar and the camera; in a case where the matching result of the adjacent two frame data is a same target, performing data fusion on the current frame target information and the previous frame target information of the target according to a Kalman filtering algorithm, wherein the data fusion comprises data fusion of position information and data fusion of speed information of the target information; updating target information of each target in the linked list according to the fused data, converting the target information in the linked list into a preset format, and sending the target information in the preset format to a server; the step of performing associated matching on the current frame target information and the previous frame target information, wherein the associated matching comprises matching of adjacent two frame data of a same sensor and matching of adjacent two frame data between different sensors, comprises the following steps: for associated matching between each detected target in adjacent two frame data of the laser radar, associated matching between each detected target of the laser radar and each detected target of the millimeter wave radar in adjacent two frame data, and associated matching between each detected target in adjacent two frame data of the millimeter wave radar, whether the targets are same targets is determined by judging a relationship between distances between the targets identified by adjacent two frame data and a preset first threshold range; for associated matching between each detected target in adjacent two frame data of the camera, whether the targets are same targets is determined by comparing bounding boxes of the targets identified by adjacent two frame data based on a camera model; for associated matching between each detected target of the laser radar and each detected target of the camera in adjacent two frame data, whether the targets are same targets is determined by comparing ranges of three-dimensional bounding boxes of the targets identified by adjacent two frame data based on a detection model of the laser radar and ranges of bounding boxes of the targets identified by adjacent two frame data based on a camera model; the step of determining whether the targets are same targets by judging a relationship between distances between the targets identified by adjacent two frame data and a preset first threshold range for associated matching between each detected target in adjacent two frame data of the laser radar, associated matching between each detected target of the laser radar and each detected target of the millimeter wave radar in adjacent two frame data, and associated matching between each detected target in adjacent two frame data of the millimeter wave radar, comprises the following steps: According to the pose data of each detection target in the previous frame, pose data of each detection target in the next frame is predicted, the pose data including position, speed and running direction, for each detection target, the position of the detection target in the previous frame and the predicted position of the target in the next frame are taken as the foci of an ellipse, and a correlation region is drawn according to a set eccentricity of the ellipse, as shown in formula (1.1): (1.1) ； wherein, denotes the eccentricity, denotes the velocity of the target; The position of each detection target in the current frame data is compared with the range of the ellipse of each detection target in the previous frame, and in the case that the position of a detection target in the current frame data is located in the ellipse range of a detection target in the previous frame, it is determined that the detection targets corresponding to the two frames of data are the same target.

2. The method of claim 1, wherein, The step of storing the current frame target information through a linked list comprises: According to the current frame target information, a target is identified, and a track is created for each target, the track being used to store the category, position information, speed information and target timestamp of the target; All tracks are stored in a linked list.

3. The method of claim 1, wherein, The association matching between each detection target of the camera in the two frames of data is based on the bounding box of each detection target identified according to each frame of data, and whether it is the same target is determined by comparing the bounding box of each target identified in the two frames of data, comprising: Based on the first bounding box of each detection target detected in the previous frame of data and the second bounding box of each detection target detected in the current frame of data, when the first bounding box and the second bounding box have an intersection, the intersection and the union of the first bounding box and the second bounding box are calculated, and the area ratio of the intersection to the union is calculated; In the case that the area ratio meets the preset second threshold range, it is determined that the detection target corresponding to the first bounding box and the detection target corresponding to the second bounding box are the same target in the two frames of data.

4. The method of claim 1, wherein, The association matching between each detection target of the laser radar and each detection target of the camera in the two frames of data is based on the three-dimensional bounding box of each detection target identified according to the detection model of the laser radar and the bounding box of each detection target identified according to the camera model according to each frame of data, and whether it is the same target is determined by comparing the range of the three-dimensional bounding box and the bounding box of each target identified in the two frames of data, comprising: Based on the three-dimensional bounding box of each detection target identified by the laser radar and the two-dimensional bounding box of each detection target identified by the camera, the corner point coordinates of the three-dimensional bounding box are projected onto a two-dimensional image according to a mapping relationship, the mapping relationship being obtained through joint calibration of the camera and the laser radar, as shown in formula (1.2): （1.2）； wherein K is an intrinsic matrix of the camera, is an extrinsic matrix of the camera, obtained by joint calibration, wherein the symbol represents a rotation matrix from the camera coordinate system to the LiDAR coordinate system, the symbol represents a translation vector from the camera coordinate system to the LiDAR coordinate system; , is the focal length of the camera in x, y directions in the metric of pixels; , is the pixel coordinate of the image center; is the coordinate of the corner point of the three-dimensional bounding box of the LiDAR in the Z-axis direction in the camera coordinate system; is the coordinate of a point in the LiDAR coordinate system, is the coordinate of the point in the pixel coordinate system; The number of corner points contained in the two-dimensional bounding box of each detection target identified by the camera is calculated; In the case that the number of corner points in the two-dimensional bounding box is greater than a preset third threshold, it is determined that the detection target corresponding to the two-dimensional bounding box and the detection target corresponding to the three-dimensional bounding box are the same target.

5. The method of claim 4, wherein, In the case that the matching results of the two adjacent frames of data are the same target, the current frame target information of the target and the previous frame target information are fused according to the Kalman filtering algorithm, the data fusion includes the steps of data fusion of position information of the target information and data fusion of speed information, comprising: The prior state estimation matrix and the covariance matrix of the current frame are calculated through the optimal state estimation of the previous frame, as shown in formula (2.1) and (2.2): （2.1）； （2.2）； wherein, , for describing the position and velocity of the target, the symbol represents the position component of the target, the symbol represents the velocity component of the target; is the state transition matrix of the target; is the control matrix, representing the control amount on the current state; P is the covariance matrix, representing the uncertainty of the target state at different times; the symbol represents the time interval of adjacent two frames of data, Q is the process noise covariance, representing the noise of the model itself; the symbol represents the prior state estimation matrix of the current frame; the symbol represents the covariance matrix of the current frame; The Kalman coefficient is calculated, as shown in formula (2.3): （2.3）； wherein, is an observation matrix; is a covariance matrix of the measurements; The posterior state estimation equation is established, and the data is fused, as shown in formula (2.4): （2.4）； wherein is the current measurement value; And the optimal covariance matrix is updated, as shown in formula (2.5): （2.5）。 6. The method of claim 5, wherein, The steps of updating the target information of each target in the linked list according to the fused data, converting the target information in the linked list into a preset format, and sending the target information to the server in the preset format, comprising: When updating the target information in the linked list according to the fused data, the category of the target is based on the camera, the position information of the target is based on the position data of the laser radar, the speed information of the target is based on the speed data fused by the laser radar and the millimeter wave radar, and the timestamp of the target is consistent with the timestamp of the latest frame of data for updating the target information; In the case that the difference between the timestamp of the target and the current timestamp is greater than the fourth threshold value, the target is deleted from the linked list; Each target is numbered to obtain a unique ID of each target, the position information of each target in the linked list is converted into coordinates in the GPS coordinate system, the length, width and height of each target are based on the length, width and height obtained by the laser radar, the driving direction of each target is based on the driving direction obtained by the laser radar, and the ID, position information, speed information, category, length, width, height and driving direction of each target are sent to the server; Before sending the ID, position information, speed information, category, length, width, height and driving direction of each target to the server, it is verified again whether the difference between the timestamp of the target and the current timestamp is greater than the fourth threshold value, and when it is greater than the fourth threshold value, the target is deleted.

7. A perception fusion system, comprising: Comprising: The laser radar, the camera, the millimeter wave radar, the development board, the PC host and the server, wherein, The laser radar and the millimeter wave radar are directly connected with the PC host through a network cable; the camera is connected with the development board, and the development board is connected with the PC host through a network cable; the PC host is connected with the server through an optical fiber; The laser radar, the millimeter wave radar and the camera obtain target information in the environment, and transmit the target information to the PC host, wherein the laser radar and the millimeter wave radar transmit the target information to the PC host through Ethernet, the camera transmits the target information to the development board, the development board encodes the target information of the camera and then transmits the target information to the PC host through Ethernet, the target includes a vehicle and a pedestrian, and the target information includes the type, position, speed and motion direction of the target; The PC host executes the data fusion method according to any one of claims 1-6; The server receives the data of the PC host and stores it.

8. The system of claim 7, wherein, The number of the laser radars is one, the number of the cameras is three, including a first camera, a second camera and a third camera, the number of the millimeter wave radars is three, the number of the development boards is two, including a first development board and a second development board, wherein the first camera and the second camera are connected with the first development board, the third camera is connected with the second development board, and the cameras transmit target information to the development boards in the form of a serialized stream.

Images