Multi-sensor information fusion method for autonomous driving system

Abstract

The present invention relates to the field of unmanned vehicles, and in particular to a multi-sensor information fusion method for an automatic driving system, comprising the following steps: 1) obtaining an original point cloud data set; 2) separating non-ground point cloud data; 3) obtaining a plurality of obstacle point cloud data; 4) obtaining a three-dimensional bounding box corresponding to each obstacle point cloud data; 5) using an RGB image as a pixel plane, and using the product of an intrinsic and extrinsic parameter matrix of a laser radar as a projection transformation matrix; 6) using the projection transformation matrix to project to a camera coordinate system respectively, and projecting each three-dimensional bounding box in the camera coordinate system to the pixel plane respectively, to obtain a corresponding obstacle bounding box; 7) obtaining corresponding obstacle ROI regions; 8) using a greedy algorithm to fuse each obstacle ROI region to obtain a final ROI region; 9) using a D-S evidence theory fusion algorithm synthesis rule based on an improved Yager synthesis rule to calculate the confidence of the obstacle category detection after fusion and the confidence of other unknown category obstacles.

Description

Technical Field

[0001] The present invention relates to the field of unmanned vehicles, and in particular to a multi-sensor information fusion method for an automatic driving system. Background Art

[0002] In machine vision and image processing, the area to be processed is outlined in the form of a box, circle, ellipse, irregular polygon, etc. from the image being processed, which is called the region of interest (ROI). Various operators and functions are often used in machine vision software such as Halcon, OpenCV, and Matlab to obtain the ROI of the region of interest and perform the next step of image processing.

[0003] With the development and breakthrough of technologies in sensor fusion technology, artificial intelligence, big data processing and analysis, computer vision and other fields, automobile intelligence has become an important research direction for the development of the automobile industry. Autonomous driving vehicles, also known as driverless cars or wheeled mobile robots, are mainly composed of three parts: perception, decision-making and control. As an automated vehicle, an autonomous driving vehicle must have the ability to sense the surrounding environment without human operation, that is, intelligent environmental perception. This capability requires the autonomous driving system to be able to fuse the data information of each sensor, obtain the surrounding environment of the unmanned vehicle and its own status information in real time, and then analyze and process the sensor perception data through the decision-making planning layer. Comprehensive and accurate perception of the vehicle's surrounding environment information is the guarantee of driving safety and intelligence. The redundant perception system based on sensor fusion technology uses a variety of sensors with complementary characteristics to enhance perception capabilities and achieve all-weather environmental perception. It has broad development prospects in the field of autonomous driving and is the development trend of intelligent environmental perception technology.

[0004] Target detection using sensor fusion detection is a basic and key task in the perception stage. It can provide autonomous vehicles with category information, positioning information, and parameter information such as position, posture, and physical size of pedestrians, vehicles and other targets in road scenes. The quality of the detection effect directly affects the subsequent processing of the decision-making and planning system.

[0005] However, the existing sensor fusion detection method has a high missed detection rate, low detection accuracy for overlapping objects, and poor real-time performance, which directly affects the subsequent processing effect of the decision-making planning system. Summary of the invention

[0006] The purpose of the present invention is to provide a multi-sensor information fusion method for an autonomous driving system to address the corresponding deficiencies in the prior art, quickly generate ROI areas for various obstacles in a complex traffic environment, combine the advantages of two sensors, and make fast, accurate and highly robust judgments on obstacle information.

[0007] The object of the present invention is to adopt the following scheme to achieve: a multi-sensor information fusion method for an autonomous driving system, comprising the following steps:

[0008] 1) Use LiDAR to obtain the spatial three-dimensional information of the vehicle's surrounding environment in real time to form an original point cloud data set;

[0009] 2) Separating the ground point cloud data from the non-ground point cloud data in the original point cloud data set;

[0010] 3) Set the vehicle passage area according to the reflectivity information of the ground point cloud data, and filter the non-ground point cloud data to obtain some obstacle point cloud data;

[0011] 4) Using clustering algorithm to process each obstacle point cloud data separately, and obtain the three-dimensional bounding box corresponding to each obstacle point cloud data;

[0012] 5) Obtain an RGB image of the front environment information through the camera, and use the RGB image as the pixel plane; obtain the intrinsic and extrinsic parameter matrices of the laser radar offline, and use the product of the intrinsic and extrinsic parameter matrices of the laser radar as the projection transformation matrix;

[0013] 6) The three-dimensional bounding boxes corresponding to the obstacle point cloud data are projected to the camera coordinate system using the projection transformation matrix, and the three-dimensional bounding boxes in the camera coordinate system are enlarged using the dynamic threshold algorithm. Then, the camera intrinsic parameter matrix is ​​obtained offline, and each three-dimensional bounding box in the camera coordinate system is projected to the pixel plane to obtain the corresponding obstacle bounding box;

[0014] 7) According to the obstacle bounding boxes in the pixel plane, the corresponding obstacle ROI areas are obtained respectively, and the category information and confidence of each obstacle are obtained;

[0015] 8) Use the greedy algorithm to fuse the ROI areas of each obstacle to obtain the final ROI area, and obtain the category information and confidence of each obstacle in the final ROI area;

[0016] 9) Using the DS evidence theory fusion algorithm synthesis rule based on the improved Yager synthesis rule, the following formula is used to calculate the confidence of the obstacle category detection after fusion and the confidence of other unknown categories of obstacles:

[0017]

[0018] Where m(A) is the confidence of obstacle category detection after fusion, m(X) is the confidence of other unknown category obstacles, and A nis the category information of the nth obstacle, n is the total number of obstacle categories, k represents the conflict degree of obstacle categories detected by different sensors in the final ROI area, ε is the credibility of the detection result, and m n (A n ) is the nth type of sensor that detects A in the final ROI area n Confidence of class obstacles, m i (A) is the confidence of the output of the i-th sensor for the type A obstacle, m i (X) is the confidence of the output of the i-th sensor for other unknown types of obstacles, It is the confidence that the obstacle category output after fusion is an empty set, which is always zero.

[0019] Preferably, the category information and confidence level of each obstacle in the final ROI area are acquired using a 2D image detection algorithm.

[0020] Preferably, the category information and confidence level of each obstacle ROI area are acquired using a 3D point cloud detection algorithm.

[0021] Preferably, the greedy algorithm is used to fuse the obstacle ROI regions, and the specific steps of obtaining the final ROI region are as follows:

[0022] ① Obtain the horizontal and vertical coordinate extreme values ​​of the four vertices of each obstacle ROI area in the pixel plane respectively;

[0023] ② Using the horizontal and vertical coordinate extreme values ​​of the four vertices of each obstacle ROI area in the pixel plane, all obstacle ROI areas in the current pixel plane are converted into corresponding obstacle matrices to form an obstacle matrix set, which is used to represent the ROI area corresponding to each obstacle on the pixel plane at the current moment;

[0024] ③ Establish a final ROI area obstacle set to store the coordinate information of each obstacle in the final ROI area;

[0025] ④ Arrange the obstacle matrices in the obstacle matrix set in ascending order according to the left boundary;

[0026] ⑤ Add the first obstacle matrix in the obstacle matrix set after being arranged in ascending order according to the left boundary to the final ROI area obstacle set;

[0027] ⑥ Add the obstacle matrices in the obstacle matrix set to the final ROI area obstacle set in the following manner:

[0028] (1) If the minimum horizontal coordinate value of the currently selected obstacle matrix is ​​less than the maximum horizontal coordinate value of the last obstacle matrix in the final ROI area obstacle set, the currently selected obstacle matrix is ​​added to the final ROI area obstacle set and arranged in ascending order with other obstacle matrices in the final ROI area obstacle set;

[0029] (2) If the minimum value of the horizontal coordinate of the currently selected obstacle matrix is ​​≥ the maximum value of the horizontal coordinate of the last obstacle matrix in the final ROI area obstacle set, then compare the maximum value of the horizontal coordinate of the last obstacle matrix in the final ROI area obstacle set with the maximum value of the horizontal coordinate of the currently selected obstacle matrix:

[0030] If the maximum value of the abscissa of the last obstacle matrix in the final ROI area obstacle set is ≤ the maximum value of the abscissa of the currently selected obstacle matrix, the maximum value of the abscissa of the last obstacle matrix in the final ROI area obstacle set is modified to the maximum value of the abscissa of the currently selected obstacle matrix;

[0031] If the maximum abscissa value of the last obstacle matrix in the final ROI area obstacle set is greater than the maximum abscissa value of the currently selected obstacle matrix, the maximum abscissa value of the last obstacle matrix in the final ROI area obstacle set remains unchanged.

[0032] Preferably, the method of filtering the non-ground point cloud data to obtain a number of obstacle point cloud data is through filtering or voxel downsampling filtering to reduce the number of point clouds and avoid increasing the computational burden.

[0033] Preferably, the clustering algorithm is a three-dimensional space clustering algorithm. The specific steps of using the clustering algorithm to process each obstacle point cloud data respectively to obtain a three-dimensional bounding box corresponding to each obstacle point cloud data are as follows:

[0034] ① Obtain several independent point cloud clusters corresponding to the obstacle point cloud data through a three-dimensional spatial clustering algorithm;

[0035] ② Use the oriented bounding box (OBB) to fit the independent point cloud cluster described in step ① to obtain a three-dimensional bounding box (3DBoudingBox).

[0036] Preferably, the clustering algorithm is a two-dimensional plane clustering algorithm. The specific steps of using the clustering algorithm to process each obstacle point cloud data respectively to obtain a three-dimensional bounding box corresponding to each obstacle point cloud data are as follows:

[0037] ① Use KD-Tree to store obstacle point cloud data for efficient indexing;

[0038] ② Perform dimensionality reduction on the two-dimensional plane of the point cloud to obtain several independent point cloud clusters corresponding to the obstacle point cloud data;

[0039] ③ Fitting the several independent point cloud clusters described in step ② to obtain the corresponding two-dimensional bounding boxes;

[0040] ④According to the KD-Tree index, the upper boundary coordinate value and the lower boundary coordinate value (i.e., the number of standard deviations from the mean) of the independent point cloud cluster in step ② on the Z axis in the three-dimensional coordinate system are obtained, and the difference between the upper boundary coordinate value and the lower boundary coordinate value is taken as the height difference;

[0041] ⑤Convert the 2D bounding box into a 3D bounding box based on the height difference.

[0042] Preferably, the two-dimensional bounding box corresponding to the obstacle point cloud data is obtained by fitting a number of independent point cloud clusters corresponding to the obstacle point cloud data using a minimum area circumscribed rectangle method (ie, MER) or an LshapeFIT method.

[0043] Preferably, the Z axis is perpendicular to the ground axis in the three-dimensional coordinate system.

[0044] The advantages of the present invention are the following three points:

[0045] ① The invention generates the region of interest at a relatively fast speed;

[0046] ② Combining the advantages of the two sensors, LiDAR and camera, the camera is provided with an ROI area while the LiDAR outputs the detection results, so that one sensor can have multiple uses and has strong robustness;

[0047] ③ The present invention greatly reduces the missed detection rate, enhances the real-time performance of detection, improves the detection accuracy of overlapping obstacles and the subsequent processing effect of the decision-making and planning system. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0049] like Figure 1 As shown, a multi-sensor information fusion method for an autonomous driving system includes the following steps:

[0050] 1) Use LiDAR to obtain the spatial three-dimensional information of the vehicle's surrounding environment in real time to form an original point cloud data set;

[0051] 2) Separating the ground point cloud data from the non-ground point cloud data in the original point cloud data set;

[0052] 3) Setting the vehicle passage area according to the reflectivity information of the ground point cloud data, and filtering the non-ground point cloud data by straight-through filtering or voxel downsampling filtering to obtain some obstacle point cloud data;

[0053] 4) Using clustering algorithm to process each obstacle point cloud data separately, and obtain the three-dimensional bounding box corresponding to each obstacle point cloud data;

[0054] In this embodiment, the clustering algorithm is a three-dimensional space clustering algorithm. The specific steps of using the clustering algorithm to process each obstacle point cloud data respectively and obtain the three-dimensional bounding box corresponding to each obstacle point cloud data are as follows:

[0055] ① Obtain several independent point cloud clusters corresponding to the obstacle point cloud data through a three-dimensional spatial clustering algorithm;

[0056] ② Use the oriented bounding box (OBB) to fit the independent point cloud cluster described in step ① to obtain a three-dimensional bounding box (3DBoudingBox).

[0057] If the clustering algorithm is a two-dimensional plane clustering algorithm, the specific steps of using the clustering algorithm to process each obstacle point cloud data respectively to obtain the three-dimensional bounding box corresponding to each obstacle point cloud data are as follows:

[0058] ① Use KD-Tree to store obstacle point cloud data for efficient indexing;

[0059] ② Perform dimensionality reduction on the two-dimensional plane of the point cloud to obtain several independent point cloud clusters corresponding to the obstacle point cloud data;

[0060] ③ Fitting the several independent point cloud clusters described in step ② to obtain a two-dimensional bounding box corresponding to the obstacle point cloud data. The two-dimensional bounding box corresponding to the obstacle point cloud data is usually obtained by fitting several independent point cloud clusters corresponding to the obstacle point cloud data using the minimum area circumscribed rectangle method (ie, MER) or the LshapeFIT method.

[0061] ④According to the KD-Tree index, the upper boundary coordinate value and the lower boundary coordinate value (i.e., the number of standard deviations from the mean, where the Z axis is perpendicular to the ground axis in the three-dimensional coordinate system) of the independent point cloud cluster in step ② are obtained, and the difference between the upper boundary coordinate value and the lower boundary coordinate value is taken as the height difference;

[0062] ⑤Convert the 2D bounding box into a 3D bounding box based on the height difference.

[0063] 5) Obtain an RGB image of the front environment information through the camera, and use the RGB image as the pixel plane; obtain the intrinsic and extrinsic parameter matrices of the laser radar offline, and use the product of the intrinsic and extrinsic parameter matrices of the laser radar as the projection transformation matrix;

[0064] 6) The three-dimensional bounding boxes corresponding to the obstacle point cloud data are projected to the camera coordinate system using the projection transformation matrix, and the three-dimensional bounding boxes in the camera coordinate system are enlarged using the dynamic threshold algorithm. Then, the camera intrinsic parameter matrix is ​​obtained offline, and each three-dimensional bounding box in the camera coordinate system is projected to the pixel plane to obtain the corresponding obstacle bounding box;

[0065] 7) According to the obstacle bounding boxes in the pixel plane, the corresponding obstacle ROI areas are obtained respectively, and the category information and confidence of each obstacle are obtained by using the 3D point cloud detection algorithm;

[0066] 8) A greedy algorithm is used to fuse the ROI regions of each obstacle to obtain the final ROI region, and a 2D image detection algorithm is used to obtain the category information and confidence of each obstacle in the final ROI region;

[0067] In this embodiment, the greedy algorithm is used to fuse the obstacle ROI regions, and the specific steps of obtaining the final ROI region are as follows:

[0068] ① Obtain the horizontal and vertical coordinate extreme values ​​of the four vertices of each obstacle ROI area in the pixel plane respectively;

[0069] ② Using the horizontal and vertical coordinate extreme values ​​of the four vertices of each obstacle ROI area in the pixel plane, all obstacle ROI areas in the current pixel plane are converted into corresponding obstacle matrices to form an obstacle matrix set, which is used to represent the ROI area corresponding to each obstacle on the pixel plane at the current moment;

[0070] The obstacle matrix corresponding to the nth obstacle ROI area is as follows:

[0071]

[0072] In the formula, is the obstacle matrix corresponding to the nth obstacle ROI area, is the minimum horizontal coordinate value of the upper left vertex of the nth obstacle ROI area, is the maximum horizontal coordinate of the upper left vertex of the nth obstacle ROI area, is the minimum ordinate value of the upper left vertex of the nth obstacle ROI area, is the maximum vertical coordinate of the upper left vertex of the nth obstacle ROI area, is the minimum horizontal coordinate value of the lower left vertex of the nth obstacle ROI area, is the maximum horizontal coordinate of the lower left vertex of the nth obstacle ROI area, is the minimum ordinate value of the lower left vertex of the nth obstacle ROI area, is the maximum vertical coordinate of the lower left vertex of the nth obstacle ROI area, is the minimum horizontal coordinate value of the upper right vertex of the nth obstacle ROI area, is the maximum horizontal coordinate of the upper right vertex of the nth obstacle ROI area, is the minimum ordinate value of the upper right vertex of the nth obstacle ROI area, is the maximum vertical coordinate of the upper right vertex of the nth obstacle ROI area, is the minimum value of the horizontal coordinate of the lower right vertex of the nth obstacle ROI area, is the maximum horizontal coordinate of the lower right vertex of the nth obstacle ROI area, is the minimum ordinate value of the lower right vertex of the nth obstacle ROI area, The maximum vertical coordinate of the lower right vertex of the nth obstacle ROI area;

[0073] ③ Establish a final ROI area obstacle set to store the coordinate information of each obstacle in the final ROI area;

[0074] ④ Arrange the obstacle matrices in the obstacle matrix set in ascending order according to the left boundary;

[0075] ⑤ Add the first obstacle matrix in the obstacle matrix set after being arranged in ascending order according to the left boundary to the final ROI area obstacle set;

[0076] ⑥ Add the obstacle matrices in the obstacle matrix set to the final ROI area obstacle set in the following manner:

[0077] (1) If the minimum value of the horizontal coordinates of the four vertices of the currently selected obstacle matrix is ​​less than the maximum value of the horizontal coordinates of the four vertices of the last obstacle matrix in the final ROI area obstacle set, the currently selected obstacle matrix is ​​added to the final ROI area obstacle set and arranged in ascending order with other obstacle matrices in the final ROI area obstacle set;

[0078] (2) If the minimum value of the horizontal coordinates of the four vertices of the currently selected obstacle matrix is ​​≥ the maximum value of the horizontal coordinates of the four vertices of the last obstacle matrix in the final ROI area obstacle set, then compare the maximum value of the horizontal coordinates of the four vertices of the last obstacle matrix in the final ROI area obstacle set with the maximum value of the horizontal coordinates of the four vertices of the currently selected obstacle matrix:

[0079] If the maximum value of the horizontal coordinates of the four vertices of the last obstacle matrix in the final ROI area obstacle set is ≤ the maximum value of the horizontal coordinates of the four vertices of the currently selected obstacle matrix, the maximum value of the horizontal coordinates of the four vertices of the last obstacle matrix in the final ROI area obstacle set is modified to the maximum value of the horizontal coordinates of the four vertices of the currently selected obstacle matrix;

[0080] If the maximum abscissa value of the four vertices of the last obstacle matrix in the final ROI area obstacle set is greater than the maximum abscissa value of the four vertices of the currently selected obstacle matrix, the maximum abscissa value of the four vertices of the last obstacle matrix in the final ROI area obstacle set remains unchanged.

[0081] 9) Using the DS evidence theory fusion algorithm synthesis rule based on the improved Yager synthesis rule, the following formula is used to calculate the confidence of the obstacle category detection after fusion and the confidence of other unknown categories of obstacles:

[0082]

[0083] Where m(A) is the confidence of obstacle category detection after fusion, m(X) is the confidence of other unknown category obstacles, and A n is the category information of the nth obstacle, n is the total number of obstacle categories, k is the conflict degree of obstacle categories detected by different sensors in the final ROI area, ε is the credibility of the detection result, and m n (A n ) is the nth type of sensor that detects A in the final ROI area n Confidence of class obstacles, m i (A) is the confidence of the output of the i-th sensor for the type A obstacle, m i (X) is the confidence of the output of the i-th sensor for other unknown types of obstacles, It is the confidence that the obstacle category output after fusion is an empty set, which is always zero.

[0084] In this embodiment, let Θ be a recognition framework, under Θ, there are two groups of detection results obtained by the laser radar and the camera, namely E1 and E2, let the obstacle confidence function of E1 be m1, and the obstacle category information of E1 be A1, A2, ...A r , the obstacle confidence function of E2 is m2, and the obstacle category information of E1 is B1, B2, …B r .

[0085] The synthesis rules of DS evidence theory are as follows:

[0086]

[0087]

[0088] In the formula, k is the conflict degree of obstacle categories detected by different sensors in the final ROI area, which reflects the conflict degree between the detection results of two different types of sensors, lidar and camera, in the same ROI area. The larger k is, the greater the conflict between the two.

[0089] When k = 1, it indicates that there is a strong conflict between the evidences and synthesis cannot be performed;

[0090] When k≠1, a new confidence allocation function can be obtained by the above formula.

[0091] The synthesis rule assigns the confidence of the empty set to 0, and the normalization factor 11-k ensures that the total confidence is 1 after removing the empty set.

[0092] Since the DS evidence theory has limitations when facing high-conflict problems, it will produce erroneous results. The present invention proposes an improved method based on evidence credibility, removes the normalization factor 11-k, introduces an unknown type obstacle X, and assigns all conflicts to the unknown type obstacle X, avoiding the choice of conflict;

[0093] In order to solve the problem that if one of the detection results of the laser radar and camera sensors denies the Class A obstacle, the synthesis result also denies the Class A obstacle, evidence credibility is introduced to improve the fusion result. The specific formula is as follows:

[0094] Under Θ, there are two sets of detection results obtained by the laser radar and the camera, E1 and E2 respectively. The obstacle confidence function of E1 is m1, the obstacle confidence function of E2 is m2, and the conflict between the detection result sets E1 and E2 is k ij , the reliability of the detection result is ε, and the formulas are as follows:

[0095]

[0096] Where e is a natural constant.

[0097] Finally, by using the DS evidence theory fusion algorithm synthesis rule based on the improved Yager synthesis rule, the confidence of the obstacle category detection after fusion and the confidence of other unknown category obstacles can be obtained as follows:

[0098]

[0099] Where m(A) is the confidence of obstacle category detection after fusion, m(X) is the confidence of other unknown category obstacles, and A n is the category information of the nth obstacle, n is the total number of obstacle categories, k is the conflict degree of obstacle categories detected by different sensors in the final ROI area, ε is the credibility of the detection result, and m n (A n ) is the nth type of sensor that detects A in the final ROI area n Confidence of class obstacles, m i(A) is the confidence of the output of the i-th sensor for the type A obstacle, m i (X) is the confidence of the output of the i-th sensor for other unknown types of obstacles, It is the confidence that the obstacle category output after fusion is an empty set, which is always zero.

[0100] The confidence of the fused obstacle category detection and the confidence of other unknown obstacle categories are output to the decision-making planning system for further processing. Combining the advantages of the two sensors, lidar and camera, an ROI area is provided to the camera while the lidar outputs the detection results, which greatly reduces the missed detection rate, enhances the real-time detection, and improves the detection accuracy of overlapping obstacles and the subsequent processing effect of the decision-making planning system.

[0101] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification made to the present invention by those skilled in the art without departing from the spirit of the present invention shall fall within the protection scope of the present invention.

Claims

1. A multi-sensor information fusion method for an autonomous driving system, characterized in that: The following steps are involved: 1) Use LiDAR to obtain the spatial three-dimensional information of the vehicle's surrounding environment in real time to form an original point cloud data set; 2) Separating the ground point cloud data from the non-ground point cloud data in the original point cloud data set; 3) Set the vehicle passage area according to the reflectivity information of the ground point cloud data, and filter the non-ground point cloud data to obtain some obstacle point cloud data; 4) Using clustering algorithm to process each obstacle point cloud data separately, and obtain the three-dimensional bounding box corresponding to each obstacle point cloud data; 5) Obtain an RGB image of the front environment information through the camera, and use the RGB image as the pixel plane; obtain the intrinsic and extrinsic parameter matrices of the laser radar offline, and use the product of the intrinsic and extrinsic parameter matrices of the laser radar as the projection transformation matrix; 6) The three-dimensional bounding boxes corresponding to the obstacle point cloud data are projected to the camera coordinate system using the projection transformation matrix, and the three-dimensional bounding boxes in the camera coordinate system are enlarged using the dynamic threshold algorithm. Then, the camera intrinsic parameter matrix is ​​obtained offline, and each three-dimensional bounding box in the camera coordinate system is projected to the pixel plane to obtain the corresponding obstacle bounding box; 7) According to the obstacle bounding boxes in the pixel plane, the corresponding obstacle ROI areas are obtained respectively, and the category information and confidence of each obstacle are obtained; 8) Use the greedy algorithm to fuse the ROI areas of each obstacle to obtain the final ROI area, and obtain the category information and confidence of each obstacle in the final ROI area; 9) Using the DS evidence theory fusion algorithm synthesis rule based on the improved Yager synthesis rule, the following formula is used to calculate the confidence of the obstacle category detection after fusion and the confidence of other unknown categories of obstacles: Where m(A) is the confidence of obstacle category detection after fusion, m(X) is the confidence of other unknown category obstacles, and A n is the category information of the nth obstacle, n is the total number of obstacle categories, k represents the conflict degree of obstacle categories detected by different sensors in the final ROI area, ε is the credibility of the detection result, and m n (A n ) is the nth type of sensor that detects A in the final ROI area n Confidence of class obstacles, m i (A) is the confidence of the output of the i-th sensor for the type A obstacle, m i (X) is the confidence of the output of the i-th sensor for other unknown types of obstacles, It is the confidence that the obstacle category output after fusion is an empty set, which is always zero.

2. The multi-sensor information fusion method of the automatic driving system according to claim 1, characterized in that: The category information and confidence of each obstacle in the final ROI area are obtained using a 2D image detection algorithm.

3. The multi-sensor information fusion method of the automatic driving system according to claim 1, characterized in that: The category information and confidence of each obstacle ROI area are obtained by using a 3D point cloud detection algorithm.

4. The multi-sensor information fusion method of the automatic driving system according to claim 1, characterized in that: The specific steps of using the greedy algorithm to fuse the ROI areas of each obstacle and obtain the final ROI area are as follows: ① Obtain the horizontal and vertical coordinate extreme values ​​of the four vertices of each obstacle ROI area in the pixel plane respectively; ② Using the horizontal and vertical coordinate extreme values ​​of the four vertices of each obstacle ROI area in the pixel plane, all obstacle ROI areas in the current pixel plane are converted into corresponding obstacle matrices to form an obstacle matrix set, which is used to represent the ROI area corresponding to each obstacle on the pixel plane at the current moment; ③ Establish a final ROI area obstacle set to store the coordinate information of each obstacle in the final ROI area; ④ Arrange the obstacle matrices in the obstacle matrix set in ascending order according to the left boundary; ⑤ Add the first obstacle matrix in the obstacle matrix set after being arranged in ascending order according to the left boundary to the final ROI area obstacle set; ⑥ Add the obstacle matrices in the obstacle matrix set to the final ROI area obstacle set in the following manner: (1) If the minimum horizontal coordinate value of the currently selected obstacle matrix is ​​less than the maximum horizontal coordinate value of the last obstacle matrix in the final ROI area obstacle set, the currently selected obstacle matrix is ​​added to the final ROI area obstacle set and arranged in ascending order with other obstacle matrices in the final ROI area obstacle set; (2) If the minimum value of the horizontal coordinate of the currently selected obstacle matrix is ​​≥ the maximum value of the horizontal coordinate of the last obstacle matrix in the final ROI area obstacle set, then compare the maximum value of the horizontal coordinate of the last obstacle matrix in the final ROI area obstacle set with the maximum value of the horizontal coordinate of the currently selected obstacle matrix: If the maximum value of the abscissa of the last obstacle matrix in the final ROI area obstacle set is ≤ the maximum value of the abscissa of the currently selected obstacle matrix, the maximum value of the abscissa of the last obstacle matrix in the final ROI area obstacle set is modified to the maximum value of the abscissa of the currently selected obstacle matrix; If the maximum abscissa value of the last obstacle matrix in the final ROI area obstacle set is greater than the maximum abscissa value of the currently selected obstacle matrix, the maximum abscissa value of the last obstacle matrix in the final ROI area obstacle set remains unchanged.

5. The multi-sensor information fusion method of the automatic driving system according to claim 1, characterized in that: The non-ground point cloud data are filtered to obtain a number of obstacle point cloud data by straight-through filtering or voxel downsampling filtering.

6. The multi-sensor information fusion method of the automatic driving system according to claim 1, characterized in that: The clustering algorithm is a three-dimensional space clustering algorithm. The specific steps of using the clustering algorithm to process each obstacle point cloud data respectively and obtain the three-dimensional bounding box corresponding to each obstacle point cloud data are as follows: ① Obtain several independent point cloud clusters corresponding to the obstacle point cloud data through a three-dimensional spatial clustering algorithm; ② Use the oriented bounding box to fit the independent point cloud cluster described in step ① to obtain a three-dimensional bounding box.

7. The multi-sensor information fusion method of the automatic driving system according to claim 1, characterized in that: The clustering algorithm is a two-dimensional plane clustering algorithm. The specific steps of using the clustering algorithm to process each obstacle point cloud data respectively and obtain the three-dimensional bounding box corresponding to each obstacle point cloud data are as follows: ① Use KD-Tree storage to achieve efficient indexing of obstacle point cloud data; ② Perform dimensionality reduction on the two-dimensional plane of the point cloud to obtain several independent point cloud clusters corresponding to the obstacle point cloud data; ③ Fitting the several independent point cloud clusters described in step ② to obtain the corresponding two-dimensional bounding boxes; ④According to the KD-Tree index, the upper boundary coordinate value and the lower boundary coordinate value of the independent point cloud cluster in step ② on the Z axis in the three-dimensional coordinate system are obtained, and the difference between the upper boundary coordinate value and the lower boundary coordinate value is taken as the height difference; ⑤Convert the 2D bounding box into a 3D bounding box based on the height difference.

8. The multi-sensor information fusion method of the automatic driving system according to claim 7, characterized in that: The two-dimensional bounding box corresponding to the obstacle point cloud data is obtained by fitting a number of independent point cloud clusters corresponding to the obstacle point cloud data using the minimum area circumscribed rectangle method or the LshapeFIT method.

9. The multi-sensor information fusion method of the automatic driving system according to claim 7, characterized in that: The Z axis is perpendicular to the ground axis in the three-dimensional coordinate system.

Images