Multi-sensor fusion sensing method based on high-precision map

Abstract

The invention discloses a multi-sensor fusion sensing method based on a high-precision map. The method comprises the following steps: sensing and acquiring information of static obstacles based on the high-precision map; performing original data fusion of dynamic obstacle information based on multi-sensor fusion, and summarizing data of all sensors to obtain a super sensor; obtaining the symmetry of the geometric structure of the multi-dimensional body according to the obtained super sensor, and extracting data information of a target obstacle based on edge detection and an obstacle IOU contour; and performing sensing fusion on a high-precision map based on stereo matching and the sensors, and performing shielding detection on inconsistent areas by adopting multi-source consistency check to complete final fusion of the multiple sensors. According to the invention, the possible road condition in front of the current position can be known in advance, and the function of the sensor module is dynamically adjusted, so that the performance requirement on the sensor is reduced, and vehicle-mounted computing resources are reasonably allocated.

Description

Technical Field

[0001] The present invention relates to the field of sensor fusion perception technology, and in particular to a multi-sensor fusion perception method based on high-precision maps. Background Art

[0002] In the rapidly developing field of unmanned driving technology, open-pit mines, as complex and changeable working environments, place extremely high demands on the perception capabilities of unmanned mining vehicles. Currently, the perception system based on high-precision maps and multi-sensor fusion has become an advanced solution widely used in this field, aiming to provide environmental prior knowledge through high-precision maps, combined with real-time data collected by multiple sensors such as lidar, cameras, millimeter-wave radars, etc., to achieve comprehensive and accurate perception of the environment around the mining vehicle.

[0003] The shortcomings of existing technologies, although the technology has shown significant advantages, still have shortcomings such as reduced perception accuracy and data fusion processing delays when facing the complex terrain, adverse weather conditions (such as sandstorms and dense fog) and dynamically changing operating scenes unique to open-pit mines. Real-time updates of high-precision maps are difficult, and sensors are easily affected by environmental interference, resulting in misjudgments or missed detections, which affects the safety and operating efficiency of unmanned mining vehicles. Therefore, continuously optimizing perception algorithms, improving sensor anti-interference capabilities, and strengthening real-time fusion processing of map and sensor data have become key issues that need to be solved urgently. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art. To achieve the above purpose, a multi-sensor fusion perception method based on high-precision maps is adopted to solve the problems raised in the above background technology.

[0005] A multi-sensor fusion perception method based on high-precision maps includes the following steps:

[0006] Step S1: Sense and obtain information about static obstacles based on a high-precision map;

[0007] Step S2: fusing the original data of dynamic obstacle information based on multi-sensor fusion, and aggregating the data of all sensors to obtain a super sensor;

[0008] Step S3, obtaining the symmetry of the multidimensional geometric structure according to the obtained super sensor, and extracting the data information of the target obstacle based on edge detection and obstacle IOU contour;

[0009] Step S4: Based on the fusion of high-precision maps and sensor perception based on stereo matching, multi-source consistency check is used to perform occlusion detection on inconsistent areas to complete the final fusion of multiple sensors.

[0010] As a further solution of the present invention: the specific steps in step S1 include:

[0011] Provide beyond-horizon prior information based on high-precision maps;

[0012] At the planning level, obtain data information on lane lines, ground prints, traffic signs, guardrails, and traffic lights recorded in high-precision maps; and

[0013] Obtain high-precision map information of the unmanned mine car, as well as the initial position and destination position of the unmanned mine car.

[0014] As a further solution of the present invention: the specific steps in step S2 include:

[0015] Based on the front fusion of LiDAR and camera, the data of all sensors are aggregated to obtain the super-sensor for detection;

[0016] Project the 3D point cloud onto the 2D image and perform front fusion on the raw data from the sensor.

[0017] As a further solution of the present invention: the specific steps in step S3 include:

[0018] Step S31, determining the main axis direction of the target obstacle;

[0019] In the binary data of the multi-dimensional sensor, the main axis direction is obtained through the IOU moment; the IOU moment is used to describe the characteristics of the obstacle IOU size and direction;

[0020] Step S32, correcting the inclination of the target obstacle;

[0021] The target obstacle is rotated according to the main axis direction angle; when the angle is greater than zero, it is rotated counterclockwise; when the angle is less than zero, it is rotated clockwise;

[0022] Step S33: target obstacle edge detection and contour extraction;

[0023] Adopt the neighborhood idea to extract the contour and eliminate the internal points of the target obstacle area. The internal points need to be judged according to the pixel neighborhood of the current integrated point cloud and image information to obtain the complete contour of the obstacle;

[0024] Step S34, determination of the symmetry axis and extraction of the generatrix;

[0025] Perform pixel search on the entire contour line, find the target pixel point, locate the column where the rotation axis is located, and determine the upper and lower boundaries of the contour line in turn;

[0026] After locating the upper and lower boundary rows, the busbar area is intercepted in combination with the rotation axis; the left busbar and the right busbar are extracted and stored as an array.

[0027] As a further solution of the present invention: the specific steps in step S4 include:

[0028] Multi-source consistency check is used to detect occlusion and inconsistent areas, and two perception maps are obtained through high-precision maps and sensors.

[0029] Find the corresponding matching points in the LiDAR point cloud and the camera image through the high-precision map, and determine whether the positions of the two points are equal. If they are not equal, mark the corresponding points in the LiDAR point cloud or the camera image as occlusion points;

[0030] Then replace the occluded point with the minimum disparity value of the nearest unoccluded pixel on the same scan line;

[0031] Bilateral filtering is used as a weighted median filter to eliminate artifacts and retain edge information. The weight is expressed as:

[0032]

[0033] In the formula, are spatial scaling and contour similarity, k i is the regularization factor, |ij| 2 and|i i -j j | 2 Used to measure the spatial similarity between the center and the neighborhood;

[0034] Select Window R i When the size is (2r+1)×(2r+1), the number of pixels it contains is n, and the calculation window R i A pair of sequences {I(i),w ij}’s pixel values ​​and weights;

[0035] The weights are then sorted until the cumulative weight is greater than half of the weighted value, and the corresponding I(i) is used as the fused pixel value of the window center point.

[0036] Compared with the prior art, the present invention has the following technical effects:

[0037] By adopting the above technical solution, the mutual constraints between the static information of the high-precision map and the dynamic information of the sensor are comprehensively considered to solve the problems of untimely updating of the high-precision map or rapid divergence of the perception system error caused by sensor failure. At the same time, the high-precision map framework can also support "plug and play" of multiple sensors, providing a general and flexibly configurable framework for the fusion of static obstacle information / dynamic obstacle information.

[0038] The stereo matching algorithm is used to replace the conventional Kalman filter algorithm to complete the information fusion of static obstacle information and dynamic obstacle information. In the process of gradual iteration, all perception results from sensors are used as unknown quantities, and all position information from high-precision maps is used as state constraints. The optimal solution for the obstacle position is obtained through the stereo matching algorithm. Compared with Kalman filtering, the multi-sensor fusion perception method based on high-precision maps can make full use of the historical trajectory information existing in high-precision maps and the mutual connection between the current state obtained by sensors, so as to improve the IOU position perception accuracy of obstacles.

[0039] The perception system integrated with high-precision maps can break through the performance boundaries of sensors and provide beyond-visual-range perception in all scenarios and climates. Unmanned vehicles can understand the possible road conditions ahead of their current location in advance and dynamically adjust the functions of sensor modules, thereby reducing the performance requirements for sensors and reasonably allocating on-board computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings:

[0041] Figure 1 A schematic diagram of the steps of the fusion perception method according to the embodiment disclosed in this application;

[0042] Figure 2 This is a flowchart of the fusion perception method of the embodiment disclosed in this application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Please refer to Figure 1 and Figure 2 In an embodiment of the present invention, a multi-sensor fusion perception method based on a high-precision map includes the following steps:

[0045] Step S1: sensing and obtaining information about static obstacles based on a high-precision map. The specific steps include:

[0046] Provide beyond-horizon prior information based on high-precision maps;

[0047] At the planning level, obtain data information on lane lines, ground prints, traffic signs, guardrails, and traffic lights recorded in high-precision maps; and

[0048] Obtain high-precision map information of the unmanned mining vehicle, as well as the initial and destination locations of the unmanned mining vehicle;

[0049] In this embodiment, due to the complex road conditions in the open-pit mine scene, there are a large number of irregular obstacles such as retaining walls, vehicles, and people. Therefore, how to achieve obstacle recognition and then plan a driving route to avoid obstacles becomes the key to autonomous driving. The fusion perception system on a single vehicle is limited by the physical characteristics of the sensor and the influence of bad weather. It can only achieve what is visible, and cannot obtain beyond-visual-range perception, and it is impossible to grasp the characteristics of the open-pit mine road and traffic conditions in real time.

[0050] The main function of high-precision maps is to provide a priori information beyond visual range, similar to the memory function of the human brain, mainly for the lane lines, ground prints, traffic signs, guardrails, and signal lights recorded in the high-precision maps at the planning level. Specifically, it includes obtaining the high-precision map information where the unmanned mining car is located, as well as the initial position and destination position of the unmanned mining car.

[0051] Step S2: fusing the original data of dynamic obstacle information based on multi-sensor fusion, and aggregating the data of all sensors to obtain a super sensor. The specific steps include:

[0052] Based on the front fusion of LiDAR and camera, the data of all sensors are aggregated to obtain the super-sensor for detection;

[0053] Project the 3D point cloud onto the 2D image and perform front fusion on the raw data of the sensor;

[0054] Open3D is an open source library for processing 3D data, including reading, processing, and visualization of point cloud data. The operation of converting a 3D point cloud image to a 2D plane can be achieved through the projection of the point cloud data.

[0055] Step S3: Acquire the symmetry of the geometric structure of the multidimensional body according to the obtained super sensor, and extract the data information of the target obstacle based on edge detection and obstacle IOU contour. The specific steps include:

[0056] In this embodiment, the symmetry of the multidimensional geometric structure obtained by the super-sensor is used to finally determine the IOU rotation axis and generatrix of the target obstacle. The generatrix and axis of the rotating body are quickly and effectively obtained to reduce subsequent perception errors and provide data for the fusion of high-precision maps and sensor perception.

[0057] Step S31, determining the main axis direction of the target obstacle;

[0058] In the binary data of multi-dimensional sensors, the direction of obstacles is very important. To determine the geometric direction of the target, the main axis direction is obtained through the IOU moment. The IOU moment is used to describe the characteristics of the obstacle IOU size and direction.

[0059] Step S32, correcting the inclination of the target obstacle;

[0060] The target obstacle is rotated according to the main axis direction angle; when the angle is greater than zero, it is rotated counterclockwise; when the angle is less than zero, it is rotated clockwise;

[0061] Step S33: target obstacle edge detection and contour extraction;

[0062] Adopt the neighborhood idea to extract the contour and eliminate the internal points in the target obstacle area. The internal points need to be judged based on the pixel neighborhood of the current integrated point cloud and image information to obtain the complete outline of the obstacle;

[0063] In this embodiment, a point is determined to be an internal point if the following two conditions are met.

[0064] 1) Assume P(x, y) is the target pixel, set the target pixel to white, recorded as 1, and the background pixel to black, recorded as 0, then P(x, y) = 1;

[0065] 2) The neighborhood pixels of P(x, y) are all target pixels 1.

[0066] The complete outline of the obstacle can be obtained by deleting the internal points that meet the above conditions and replacing them with background point 0.

[0067] Step S34, determination of the symmetry axis and extraction of the generatrix;

[0068] Perform pixel search on the entire contour line, find the target pixel point, locate the column where the rotation axis is located, and determine the upper and lower boundaries of the contour line in turn;

[0069] After locating the upper and lower boundary rows, the busbar area is intercepted in combination with the rotation axis; the left busbar and the right busbar are extracted and stored as an array;

[0070] In this embodiment, the specific implementation steps are:

[0071] 1) Perform pixel search on the entire contour line to find the target pixel point (pixel value is 1).

[0072] 2) Locate the column where the rotation axis is located.

[0073] 3) Determine the upper and lower boundaries of the contour line in turn, and the row where the upper boundary is located is recorded as i max , the row where the lower boundary is located is recorded as i min .

[0074] 4) After locating the upper and lower boundary rows, intercept the busbar area in conjunction with the rotation axis.

[0075] 5) The extracted left busbar is recorded as L, and the extracted right busbar is recorded as R, and they are stored in array form.

[0076] Step S4: Based on stereo matching, the high-precision map is integrated with sensor perception, and multi-source consistency check is used to perform occlusion detection on inconsistent areas to complete the final fusion of multiple sensors. The specific steps include:

[0077] In this embodiment, due to the occlusion and inconsistency between the high-precision map and the sensor perception, the fusion effect of the area is not satisfactory.

[0078] Multi-source consistency check is used to detect occlusion and inconsistent areas, and two perception maps are obtained through high-precision maps and sensors.

[0079] Find the corresponding matching points in the LiDAR point cloud and the camera image through the high-precision map, and determine whether the positions of the two points are equal. If they are not equal, mark the corresponding points in the LiDAR point cloud or the camera image as occlusion points;

[0080] Then replace the occluded point with the minimum disparity value of the nearest unoccluded pixel on the same scan line;

[0081] Bilateral filtering is used as a weighted median filter to eliminate artifacts and retain edge information. The weight is expressed as:

[0082]

[0083] In the formula, are spatial scaling and contour similarity, k i is the regularization factor, |ij| 2 and|i i -j j | 2 Used to measure the spatial similarity between the center and the neighborhood;

[0084] Select Window R i When the size is (2r+1)×(2r+1), the number of pixels it contains is n, and the calculation window R i A pair of sequences {I(i),w ij}’s pixel values ​​and weights;

[0085] The weights are then sorted until the cumulative weight is greater than half of the weighted value, and the corresponding I(i) is used as the fused pixel value of the window center point.

[0086] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents, and all should be included in the scope of protection of the present invention.

Claims

1. A multi-sensor fusion perception method based on high-precision maps, characterized in that: The following steps are involved: Step S1: Sense and obtain information about static obstacles based on a high-precision map; Step S2: fusing the original data of dynamic obstacle information based on multi-sensor fusion, and aggregating the data of all sensors to obtain a super sensor; Step S3, obtaining the symmetry of the multidimensional geometric structure according to the obtained super sensor, and extracting the data information of the target obstacle based on edge detection and obstacle IOU contour; Step S4: Based on the fusion of high-precision maps and sensor perception based on stereo matching, multi-source consistency check is used to perform occlusion detection on inconsistent areas to complete the final fusion of multiple sensors.

2. According to claim 1, a multi-sensor fusion perception method based on high-precision maps is characterized in that: The specific steps in step S1 include: Provide beyond-horizon prior information based on high-precision maps; At the planning level, obtain data information on lane lines, ground prints, traffic signs, guardrails, and traffic lights recorded in high-precision maps; and Obtain high-precision map information of the unmanned mine car, as well as the initial position and destination position of the unmanned mine car.

3. According to claim 1, a multi-sensor fusion perception method based on high-precision maps is characterized in that: The specific steps in step S2 include: Based on the front fusion of LiDAR and camera, the data of all sensors are aggregated to obtain the super-sensor for detection; Project the 3D point cloud onto the 2D image and perform front fusion on the raw data from the sensor.

4. According to claim 1, a multi-sensor fusion perception method based on high-precision maps is characterized in that: The specific steps in step S3 include: Step S31, determining the main axis direction of the target obstacle; In the binary data of the multi-dimensional sensor, the main axis direction is obtained through the IOU moment; the IOU moment is used to describe the characteristics of the obstacle IOU size and direction; Step S32, correcting the inclination of the target obstacle; The target obstacle is rotated according to the main axis direction angle; when the angle is greater than zero, it is rotated counterclockwise; when the angle is less than zero, it is rotated clockwise; Step S33: target obstacle edge detection and contour extraction; Adopt the neighborhood idea to extract the contour and eliminate the internal points of the target obstacle area. The internal points need to be judged according to the pixel neighborhood of the current integrated point cloud and image information to obtain the complete contour of the obstacle; Step S34, determination of the symmetry axis and extraction of the generatrix; Perform pixel search on the entire contour line, find the target pixel point, locate the column where the rotation axis is located, and determine the upper and lower boundaries of the contour line in turn; After locating the upper and lower boundary rows, the busbar area is intercepted in combination with the rotation axis; the left busbar and the right busbar are extracted and stored as an array.

5. According to claim 1, a multi-sensor fusion perception method based on high-precision maps is characterized in that: The specific steps in step S4 include: Multi-source consistency check is used to detect occlusion and inconsistent areas, and two perception maps are obtained through high-precision maps and sensors. Find the corresponding matching points in the LiDAR point cloud and the camera image through the high-precision map, and determine whether the positions of the two points are equal. If they are not equal, mark the corresponding points in the LiDAR point cloud or the camera image as occlusion points; Then replace the occluded point with the minimum disparity value of the nearest unoccluded pixel on the same scan line; Bilateral filtering is used as a weighted median filter to eliminate artifacts and retain edge information. The weight is expressed as: In the formula, are spatial scaling and contour similarity, k i is the regularization factor, |ij| 2 and|i i -j j | 2 Used to measure the spatial similarity between the center and the neighborhood; Select Window R i When the size is (2r+1)×(2r+1), the number of pixels it contains is n, and the calculation window R i A pair of sequences {I(i),w ij }’s pixel values ​​and weights; The weights are then sorted until the cumulative weight is greater than half of the weighted value, and the corresponding I(i) is used as the fused pixel value of the window center point.

Images