Perception fusion method of unmanned driving system

Abstract

The invention discloses a perception fusion method of an unmanned driving system, and relates to the field of unmanned driving, and the method comprises the steps: obtaining a laser point cloud, image data and millimeter wave radar data; a Zhang's calibration method is adopted to obtain internal parameters of a camera, and a first pose transformation relation of a camera coordinate system relative to a laser radar coordinate system is solved; a second pose transformation relation of the millimeter wave radar relative to the laser radar is obtained by using the millimeter wave reflector; a Point RCNN method is adopted to obtain first target information; obtaining target information II by adopting a YOLOv8 method; obtaining target information III by adopting a clustering algorithm; and carrying out coordinate unification on the target information by utilizing a pose transformation relation, fusing the multi-sensor target information under a unified coordinate system by utilizing an unscented Kalman filtering algorithm to obtain target state estimation, and generating an unmanned driving control instruction. The target sensing precision is improved aiming at the low target sensing precision of multiple sensors under the unmanned driving scene of the well industry and mining in the prior art.

Description

Technical Field

[0001] The present application relates to the field of unmanned driving, and in particular to a perception fusion method for an unmanned driving system. Background Art

[0002] With the rapid development of technologies such as artificial intelligence, big data, and 5G, autonomous driving has attracted great attention from the industry and academia as a trend of change in the automotive industry and an important development direction for future travel. As one of the core technologies for realizing autonomous driving, environmental perception provides autonomous vehicles with information about the surrounding environment, which is the basis for autonomous vehicles to realize functions such as positioning, obstacle avoidance, and path planning. However, the current autonomous driving perception technology is still immature and faces many challenges, which has become a bottleneck for the industrialization and application of autonomous driving.

[0003] The perception module of the unmanned driving system usually uses sensors such as laser radar, camera, and millimeter wave radar to collect environmental data. Among them, laser radar obtains the spatial position information of the target by actively emitting laser beams, with high ranging accuracy, but high cost and difficulty in identifying the target category; the camera uses the imaging principle to obtain the visual image information of the environment, which can capture the texture and color characteristics of the target, but is easily affected by light and weather; millimeter wave radar detects the target through the reflection characteristics of electromagnetic waves, with strong penetration, but low spatial resolution. It can be seen that different types of sensors have their own advantages and disadvantages in terms of environmental adaptability, detection distance, information dimension, etc. A single sensor is difficult to meet the needs of unmanned vehicles for comprehensive, accurate and reliable perception in complex and harsh environments. Therefore, it is urgent to study multi-sensor fusion perception technology to give full play to the complementary advantages of various types of sensors and improve the environmental understanding and traffic safety level of unmanned driving systems.

[0004] At present, multi-sensor fusion perception technology is still in its infancy. On the one hand, domestic sensor suppliers each provide single sensor products and supporting algorithms, lacking multi-sensor system integration and collaborative optimization; on the other hand, most of the existing fusion methods adopt a multi-level fusion strategy, that is, first independently process the data of each sensor, and then combine them at the feature layer or decision layer. This type of loosely coupled fusion mode does not fully exploit the spatiotemporal correlation and complementarity between sensors, and the performance improvement is limited. In addition, as one of the key application scenarios of unmanned driving, underground mines have complex and changeable terrain and a wide variety of obstacles, which puts higher requirements on the perception system of unmanned vehicles. However, there is currently little research on perception fusion for underground mining unmanned vehicles. Summary of the invention

[0005] In response to the problem of low multi-sensor target perception accuracy in unmanned driving scenarios in underground mines that exists in the prior art, the present application provides a perception fusion method for an unmanned driving system, which improves target perception accuracy by calibrating the posture relationship between sensors and performing unscented Kalman filtering fusion on the target detection results of each sensor in a unified coordinate system.

[0006] The purpose of this application is achieved through the following technical solutions.

[0007] The present application provides a perception fusion method for an unmanned driving system, which is used in underground mining scenarios, including: obtaining laser point cloud data through a vehicle-mounted laser radar, obtaining image data through a camera, and obtaining millimeter-wave radar data through a millimeter-wave radar; taking the vehicle-mounted laser radar as a reference, using Zhang's calibration method to obtain the camera internal parameters, and solving the position and posture transformation relationship of the camera coordinate system relative to the laser radar coordinate system by extracting edge information from the laser point cloud data and the image data and matching the corresponding edge points; using a millimeter-wave reflector, using the gradient descent method to solve the optimal heading angle of the millimeter-wave radar relative to the laser radar, and obtaining the position and posture transformation relationship of the millimeter-wave radar relative to the laser radar; using the PointRCNN method to perform target detection on the laser point cloud data, and obtaining to target information one; use the YOLOv8 method to detect the target on the image data to obtain target information two; use the clustering algorithm to cluster the millimeter-wave radar data to obtain target information three; among which, the targets include pedestrians, bicycles and cars; according to the posture transformation relationship one, transform the target information two from the camera coordinate system to the lidar coordinate system; according to the posture transformation relationship two, transform the target information three from the millimeter-wave radar coordinate system to the lidar coordinate system; unify the target information one, the converted target information two and the target information three in the lidar coordinate system; use the unscented Kalman filter algorithm to fuse the multi-sensor target information in the unified coordinate system to obtain the target state estimation; generate the unmanned driving control command according to the target state estimation.

[0008] Among them, the camera coordinate system is a three-dimensional rectangular coordinate system established with the optical center of the camera as the origin, the main optical axis of the camera as the Z axis, the horizontal direction of the image as the X axis, and the vertical direction of the image as the Y axis. The camera imaging principle conforms to the pinhole imaging model, and the three-dimensional world coordinates are mapped to the two-dimensional image coordinates through perspective projection. The origin of the camera coordinate system is located at the optical center of the camera, its Z axis coincides with the optical axis of the camera and points to the scene, the X axis is parallel to the row direction of the image, and the Y axis is parallel to the column direction of the image. The camera coordinate system and the image coordinate system can be converted through the intrinsic parameter matrix (focal length, principal point coordinates, etc.) and the distortion coefficient. In this application, the camera coordinate system is usually used to represent the spatial position and posture information of the image target detection results. The two-dimensional bounding box in the target information 2 is represented in the camera coordinate system.

[0009] The LiDAR coordinate system is a three-dimensional rectangular coordinate system with the LiDAR sensor as the origin, the front of the sensor as the X-axis, the left side as the Y-axis, and the top as the Z-axis. LiDAR acquires three-dimensional point cloud data of the surrounding environment by emitting and receiving laser beams, and the coordinates of each point are referenced to the LiDAR coordinate system. The origin of the LiDAR coordinate system is located at the rotation center of the sensor, with the X-axis pointing to the front of the sensor, the Y-axis pointing to the left side of the sensor, and the Z-axis pointing upward perpendicular to the XY plane. There is a fixed posture transformation relationship between the LiDAR coordinate system and the vehicle coordinate system, which is usually determined by manual measurement or calibration methods.

[0010] The pose transformation relationship describes the relative position and direction relationship between two coordinate systems, and is usually represented by a transformation matrix or transformation function. The pose transformation relationship includes a translation vector (representing the origin position difference) and a rotation matrix (representing the coordinate axis direction difference). The conversion of points and vectors in different coordinate systems can be achieved through pose transformation. In this application, pose transformation relationship 1 describes the pose relationship of the camera coordinate system relative to the lidar coordinate system, and pose transformation relationship 2 describes the pose relationship of the millimeter wave radar coordinate system relative to the lidar coordinate system. These two pose transformation relationships are solved by Zhang's calibration method and gradient descent method respectively. Through the pose transformation relationship, the target information detected by different sensors can be converted to the same reference coordinate system (such as the lidar coordinate system) to achieve spatial alignment and unified representation of the target information, providing a basis for subsequent multi-sensor fusion.

[0011] The yaw angle refers to the angle between a certain axis of the sensor coordinate system (usually the X-axis) and the corresponding axis of the carrier coordinate system. In an unmanned driving system, the yaw angle describes the angular deviation between the sensor installation position and direction and the vehicle's driving direction. For a vehicle-mounted millimeter-wave radar, the angle between the X-axis of its coordinate system and the longitudinal axis of the vehicle is the yaw angle of the millimeter-wave radar. Due to installation errors and measurement errors, the actual yaw angle of the millimeter-wave radar may deviate from the nominal value, and it is necessary to estimate the optimal yaw angle value through calibration or optimization methods. In this application, the gradient descent method is used to estimate the yaw angle of the millimeter-wave radar relative to the lidar. By minimizing the position error between the millimeter-wave radar target and the lidar target, the optimal yaw angle estimate is iteratively optimized, thereby establishing an accurate posture transformation relationship between the two sensor coordinate systems.

[0012] Target information 1 refers to the target information obtained by detecting the target on the laser point cloud data. Specifically, the PointRCNN (point cloud regional convolutional neural network) method is used to process the three-dimensional point cloud data collected by the vehicle-mounted laser radar to realize the positioning, segmentation and classification of the targets in the point cloud. PointRCNN divides the original point cloud into a series of small spatial regions, and uses the regional convolutional neural network to extract the features of each region. Through the steps of region candidate, fine segmentation and category judgment, the three-dimensional bounding box (such as length, width, height), center position (such as x, y, z coordinates) and semantic category (such as pedestrians, bicycles, cars) of each target in the point cloud are obtained. Therefore, target information 1 can be represented as a set of target attributes: ;in is the 3D bounding box of the jth target, is the target center position, is the target category, is the total number of targets detected by the laser radar. Target information 1 reflects the environmental perception result from the perspective of the laser point cloud.

[0013] Target information 2 refers to target information obtained by performing target detection on image data. Specifically, the YOLOv8 (You Only Look Once v8) method is used to process the images collected by the vehicle-mounted camera to locate and classify the targets in the image. YOLOv8 divides the input image into S×S grids, and each grid is responsible for detecting targets that fall within its area. Image features are extracted through a convolutional neural network, and the position, size, target probability, and category probability of B anchor boxes are predicted in each grid. After non-maximum suppression (NMS) processing, the two-dimensional bounding box (such as the coordinates of the upper left corner and the lower right corner) and the corresponding category label of each target in the image are obtained. Therefore, target information 2 can be expressed as a set of target attributes: ;in is the 2D bounding box of the i-th target, is the target category, The total number of objects detected by the camera. Usually used In the form of is the coordinate of the upper left corner of the bounding box, is the coordinate of the lower right corner. Target information 2 reflects the environmental perception result from the image perspective.

[0014] Target information three refers to the target information obtained by clustering the millimeter-wave radar data. Specifically, a clustering algorithm (such as DBSCAN) is used to cluster the reflection points measured by the millimeter-wave radar, and each clustering result corresponds to a potential target. The millimeter-wave radar measures the distance, speed, angle and other information of the target by emitting and receiving electromagnetic waves. For each detected reflection point, the millimeter-wave radar records its attributes such as distance, angle and Doppler speed. The clustering algorithm divides the reflection points belonging to the same target into a cluster based on the spatial distribution and speed differences of the reflection points, and filters out noise and false alarms. The centroid position and average speed of each cluster can be used as the state estimate of the target. Therefore, target information three can be expressed as a set of target attributes: ;in is the center of mass position of the kth target (such as distance and angle), is the average speed of the target, is the target category (e.g., determined by the reflection intensity and velocity distribution of the cluster), is the total number of targets detected by the millimeter-wave radar. Target information 3 reflects the environmental perception result from the perspective of the millimeter-wave radar.

[0015] Furthermore, solving the pose transformation relationship of the camera coordinate system relative to the laser radar coordinate system includes: setting a calibration plate, using Zhang's calibration method to calibrate the camera internal parameters, and obtaining the camera internal parameters; moving the calibration plate, obtaining laser point cloud data through the vehicle-mounted laser radar, and obtaining image data through the camera; calculating the normal vector of the point cloud through a point cloud processing algorithm based on the laser point cloud data, and when the angle between the normal vectors of adjacent points is greater than a threshold, taking the corresponding point as the edge point of the laser point cloud; according to the image data, using an edge detection algorithm, obtaining the image edge point by calculating the gradient amplitude and direction of the image; using feature descriptors and matching algorithms to match the laser point cloud edge points and the image edge points to obtain matched edge points; according to the matched edge points, establishing a transformation relationship matrix from the camera coordinate system to the laser radar coordinate system; using the least squares method to solve, obtaining the pose transformation relationship of the camera coordinate system relative to the laser radar coordinate system.

[0016] Among them, the calibration plate is a special pattern plate used for camera calibration and coordinate system calibration, usually composed of a black and white chessboard or dot array. The materials used to make the calibration plate include flat plates (such as acrylic plates, aluminum plates) and printing paper, etc., on which high-contrast calibration patterns are printed. The size and pattern layout of the calibration plate are pre-designed, and the physical coordinates of the intersection of the chessboard or the center position of the dots on the board are known. By shooting the image of the calibration plate with a camera, extracting the pixel coordinates of the calibration points in the image, and establishing a corresponding relationship between the pixel coordinates and the physical coordinates, the intrinsic calibration and extrinsic parameter estimation of the camera can be achieved. In this application, the calibration plate has two functions: one is to estimate the intrinsic parameters of the camera through the Zhang calibration method, and the other is to provide a common observation target for the laser radar and the camera by moving the calibration plate to different positions, and establish a corresponding relationship between the coordinate systems of the two sensors. Common calibration plates include chessboard calibration plates (such as 8x6, 9x7 chessboards) and dot array calibration plates (such as 11x9, 12x10 dot arrays).

[0017] Specifically, in this application, by calculating the angle between the normal vectors of adjacent points, when the angle is greater than a certain threshold, it is considered that the two points belong to different planes, and the connection point is the edge of the point cloud. The selection of the threshold depends on the density of the point cloud, the noise level and the geometric characteristics of the target object. The general value range is 10°~90°. The larger the threshold, the sparser the extracted edge points, and the smaller the threshold, the denser the extracted edge points. By calculating the amplitude and direction of the image gradient, when the gradient amplitude is greater than a certain threshold, the pixel is considered to belong to the edge. The selection of the threshold depends on the contrast of the image, the noise level and the strength of the edge. The general value range is 10~100 (grayscale), and commonly used values ​​are 30, 50, 70, etc. The larger the threshold, the sparser the extracted edge, and the smaller the threshold, the denser the extracted edge.

[0018] Furthermore, feature descriptors and matching algorithms are used to match laser point cloud edge points and image edge points, including: calculating geometric features of point clouds within a preset range of laser point cloud edge points to obtain geometric feature descriptors of laser point cloud edge points; calculating SIFT features or SURF features of image edge points to obtain feature descriptors of image edge points; using a nearest neighbor matching algorithm based on the geometric feature descriptors of laser point cloud edge points and the feature descriptors of image edge points to obtain feature descriptor pairs with the smallest Euclidean distance between laser point cloud edge points and image edge points as potential matching edge point pairs; using a RANSAC algorithm to construct a transformation matrix based on the potential matching edge point pairs, and calculating the number of inliers in the transformation matrix, selecting the transformation matrix with the largest number of inliers as the optimal matrix, and using the inliers of the optimal matrix as matching edge point pairs; wherein the inliers represent matching point pairs that satisfy the transformation matrix.

[0019] Among them, SIFT (Scale-Invariant Feature Transform) feature is an algorithm for extracting and describing local features of images. In this application, SIFT features are extracted from image edge points to obtain feature descriptors of edge points, which are used to match the geometric features of laser point cloud edge points. SURF (Speeded Up Robust Features) feature is an algorithm for extracting and describing local features of images based on SIFT. In this application, SURF features are extracted from image edge points to replace SIFT features to obtain more efficient feature descriptors. Nearest Neighbor Matching is a commonly used feature matching algorithm. By calculating the distance metric between two feature descriptors, the closest feature pair is found as a matching pair. By calculating the Euclidean distance between the geometric feature descriptor of the laser point cloud edge point and the SIFT / SURF feature descriptor of the image edge point, the nearest neighbor matching is used to find potential matching edge point pairs, providing candidate matching pairs for subsequent RANSAC estimation. RANSAC (Random Sample Consensus) is a robust model estimation algorithm that can estimate the optimal model parameters in data with noise and mismatches. The RANSAC algorithm finds a model that satisfies the most inliers through repeated random sampling and evaluation, eliminating the influence of false matches and outliers. RANSAC is used to estimate the transformation matrix (such as rigid body transformation, affine transformation, etc.) between the laser point cloud and the edge points of the image, and an optimal transformation matrix is ​​found through the maximum number of inliers criterion, eliminating false matching point pairs, and obtaining stable and reliable matching results.

[0020] Further, the position transformation relationship of the camera coordinate system relative to the laser radar coordinate system is obtained, including: according to each pair of matched edge points obtained and , establish an equation describing the transformation relationship from the camera coordinate system to the lidar coordinate system , where R is a 3x3 rotation matrix and t is a 3x1 translation vector. The rotation matrix R and the translation vector t are combined into a 4x4 homogeneous transformation matrix T; the equation for the transformation relationship established for all matching edge point pairs is As the error equation, an overdetermined system of equations AX=B is constructed, where A is an M×12 matrix, M is the number of matching edge point pairs, X is the unknown parameter vector of the 12×1 transformation matrix T, and B is the M×1 observation value vector; the least squares method is used to solve the overdetermined system of equations AX=B, and the error equation is minimized. The square sum of , get an optimal transformation matrix T*, the transformation matrix T* represents the pose transformation relationship of the camera coordinate system relative to the lidar coordinate system; decompose the solved transformation matrix T* into a 3x3 rotation matrix R* and a 3x1 translation vector t*, as the pose transformation relationship of the camera coordinate system relative to the lidar coordinate system.

[0021] Further, obtaining the second posture transformation relationship of the millimeter-wave radar relative to the laser radar includes: receiving laser point cloud data and millimeter-wave radar data; wherein the laser point cloud data includes the first coordinate of the reflector position detected by the laser radar, and the millimeter-wave radar data includes the second coordinate of the reflector position detected by the millimeter-wave radar; setting the objective function , where θ is the yaw angle of the millimeter wave radar relative to the laser radar, is the coordinate transformation matrix corresponding to θ, ;Σ represents the sum of all reflector positions; Initialize yaw angle is a random value; calculate the objective function At yaw angle The gradient at ,gradient: ; According to the gradient Update yaw angle , where α is the learning rate; calculate the objective function After the updated yaw angle The value at Is it less than the preset threshold ε or the number of iterations reaches the preset value K? If so, the current yaw angle As the optimal yaw angle θ*, otherwise As new Continue iterating; substitute the optimal yaw angle θ into the transformation matrix , and obtain the second posture transformation relationship of the millimeter wave radar relative to the laser radar.

[0022] Among them, the reflector is a device for reflecting electromagnetic wave signals, usually made of metal or other highly reflective materials. In radar systems, reflectors can provide stable and reliable signal reflection targets for positioning, tracking, calibration and other purposes. Common radar reflectors include: Corner reflector: composed of three mutually perpendicular planes, it can reflect the incident wave back along the original path and provide a strong echo signal. It is often used in scenes such as radar calibration and target enhancement. Spherical reflector: composed of a metal sphere, it can provide consistent scattering characteristics in all directions, and is often used in scenes such as radar cover calibration and positioning. Luneburg lens: an artificially designed spherical dielectric lens that can converge the incident wave to a point on the back of the sphere, providing the scattering characteristics of a point target, and is often used in scenes such as radar cover calibration and positioning. In this application, by placing several reflectors with known positions in the scene, the millimeter wave radar and lidar can observe these reflectors at the same time and obtain the position coordinates of the reflectors in the two sensor coordinate systems. Using the reflector position coordinates observed by the two sensors, the correspondence between the millimeter-wave radar coordinate system and the lidar coordinate system can be established, and then the pose transformation matrix between the two sensors can be estimated.

[0023] Furthermore, the unscented Kalman filter algorithm is used to fuse the multi-sensor target information in the unified coordinate system to obtain the target state estimation, including: establishing the state equation , and the measurement equation ;in, is the state vector at time k, , Contains the lateral position of the target at time k , vertical position , lateral speed , longitudinal speed , lateral acceleration , longitudinal acceleration , target width and target length ; is the state transfer matrix; is the process noise, H is the measurement matrix, is the measurement noise, k is the current moment, is the system observation vector; the Sigma point in the unscented Kalman filter algorithm is calculated by the following formula: ; ; ;in, for The mean of for The variance of ; λ represents the scaling parameter, , to ensure the semi-positive definiteness of the variance matrix, preferably , ɑ represents the scaling parameter, which indicates the degree to which the test point deviates from the expected value, and is usually a very small positive number; i represents a positive integer; represents the scaling parameter; calculates the weight of each Sigma point; and uses the unscented Kalman filter algorithm to perform time update and observation update on the target information detected by each sensor according to the obtained state equation, Sigma point and weight to obtain the target state estimation value.

[0024] Furthermore, the weight of each Sigma point is calculated by the following formula: ; ;in, is the weight of the mean, is the weight of the variance; m represents the mean; c represents the variance.

[0025] Furthermore, the unscented Kalman filter algorithm is used to update the target information detected by each sensor in time, including: initializing the mean and covariance of the state vector: ; ; Calculate the Sigma point at time k according to the Sigma point selection formula : ; Update each Sigma point in time to get the predicted Sigma point : ; According to the predicted Sigma point , weighted calculation of state prediction mean : ; According to the predicted Sigma point and the state prediction mean , weighted calculation of state prediction covariance: .

[0026] Furthermore, the unscented Kalman filter algorithm is used to observe and update the target information detected by each sensor, including: according to the state equation, the predicted Sigma point Observe and obtain the observed Sigma point : ; According to the observed Sigma point , weighted calculation of the observed predicted mean : ; According to the observed Sigma point and the observed predicted mean , calculate the prediction covariance matrix and the cross-correlation matrix between state space and prediction space : ; ; According to the prediction covariance matrix and the cross-correlation matrix , calculate the Kalman gain : ; Using Kalman gain , update the state estimate and the state covariance matrix : ; ; According to the state estimate and the state covariance matrix , and obtain the target state estimate at time k+1.

[0027] Furthermore, it also includes: judging whether there are new observation targets, if so, initializing the corresponding state vector, and adding the state vector to the tracking list; judging whether each tracking target at time k matches the observation data; if yes, using the observation data to update the corresponding state; if not, using the state prediction mean Update the corresponding status; for the tracking target that has not matched the observation data in consecutive T frames, it is determined to be lost and deleted.

[0028] Compared with the prior art, the advantages of this application are:

[0029] The multi-modal sensor data is obtained through the vehicle-mounted laser radar, camera and millimeter-wave radar, the camera internal parameters are calibrated using Zhang's calibration method, the external parameters of the camera and laser radar are solved by edge extraction and matching, and the external parameters of the millimeter-wave radar and laser radar are solved using millimeter-wave reflectors and gradient descent methods, realizing the spatiotemporal registration and coordinate system of multi-sensor data. Heterogeneous sensor data with different perspectives and different physical quantities are mapped to the same reference system, laying the foundation for subsequent fusion processing.

[0030] PointRCNN, YOLOv8 and clustering algorithm are used to detect targets on the data of three sensors, and the target detection results of laser radar, camera and millimeter wave radar are obtained. PointRCNN can capture the spatial structural characteristics of point cloud and has good robustness to occlusion and truncation; YOLOv8 makes full use of the texture, color and other information of the image to accurately estimate the category and position of the target; the clustering algorithm aggregates the millimeter wave radar points into targets according to the spatial distance, and can detect hidden targets that are difficult to detect with vision and laser. The complementary cooperation of the three heterogeneous sensors comprehensively improves the detection rate and positioning accuracy of multi-category targets in complex environments.

[0031] In a unified coordinate system, the unscented Kalman filter algorithm is used to fuse the multi-sensor target detection results. The filtering algorithm approximates the nonlinear system through Sigma points, predicts the state and observation value of each Sigma point, and then calculates the weighted state prediction mean and covariance. When new observation data arrives, the Kalman gain is updated according to the cross-correlation matrix of the state and observation, and the state is further corrected to obtain the optimal state estimate. Compared with the extended Kalman filter based on linearization, the unscented filter approximates nonlinearity through deterministic sampling, has higher estimation accuracy, and is insensitive to initial values. Combining the observation information of multiple sensors to dynamically track the target, it effectively improves the accuracy and continuity of target state estimation in complex scenes of underground mining.

[0032] In the framework of unscented Kalman filter, mechanisms such as new target initialization, observation matching, and target deletion are designed. The threshold strategy is used to determine whether the newly added observation is a new target, and the state vector of the new target is adaptively initialized. The observation is associated with the existing target using the nearest neighbor and threshold criteria. The state of the target that matches the observation is updated with the observed value, and the state of the target that does not match is updated with the predicted value. Targets that are not observed for multiple consecutive frames are considered to be lost and deleted. A reasonable multi-target management mechanism helps to improve the adaptability and robustness of the perception system to dynamic environmental changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present application will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0034] Figure 1 is an exemplary flow chart of a perception fusion method of an unmanned driving system according to some embodiments of the present application;

[0035] Figure 2 is an exemplary flow chart of target detection and recognition according to some embodiments of the present application;

[0036] Figure 3 It is an exemplary flowchart of the fusion processing of target data shown in some embodiments of the present application. DETAILED DESCRIPTION

[0037] The method and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0038] Figure 1This is an exemplary flow chart of a perception fusion method for an unmanned driving system shown in some embodiments of the present application, a perception fusion method for an unmanned driving system, used in underground mining scenarios, including: obtaining laser point cloud data through a vehicle-mounted laser radar, obtaining image data through a camera, and obtaining millimeter-wave radar data through a millimeter-wave radar; taking the vehicle-mounted laser radar as a reference, using Zhang's calibration method to obtain the camera internal parameters, and solving the position and posture transformation relationship of the camera coordinate system relative to the laser radar coordinate system by extracting edge information from the laser point cloud data and the image data and matching the corresponding edge points; using a millimeter-wave reflector, using the gradient descent method to solve the optimal heading angle of the millimeter-wave radar relative to the laser radar, and obtaining the position and posture transformation relationship of the millimeter-wave radar relative to the laser radar; using the PointRCNN method The laser point cloud data is detected by the YOLOv8 method to obtain target information one; the image data is detected by the YOLOv8 method to obtain target information two; the millimeter-wave radar data is clustered by the clustering algorithm to obtain target information three; among which, the targets include pedestrians, bicycles and cars; according to the posture transformation relationship one, the target information two is converted from the camera coordinate system to the lidar coordinate system; according to the posture transformation relationship two, the target information three is converted from the millimeter-wave radar coordinate system to the lidar coordinate system; the target information one, the converted target information two and the target information three are unified in the lidar coordinate system; the unscented Kalman filter algorithm is used to fuse the multi-sensor target information in the unified coordinate system to obtain the target state estimation; according to the target state estimation, the unmanned driving control command is generated.

[0039] Obtain multi-source heterogeneous data through on-board sensors. First, use the 16-line laser radar installed on the top of the unmanned vehicle to scan the surrounding environment at a frequency of 10Hz, and obtain about 100,000 laser points per frame. Each point contains the three-dimensional spatial coordinates (x, y, z) and the reflection intensity i to form a point cloud data set. , where N is the number of points in the point cloud. At the same time, the high-definition camera behind the front windshield captures the road scene ahead at a rate of 30 frames per second to generate an RGB image sequence with a resolution of 1920×1080 , M is the number of image frames. Each frame image is a three-channel matrix with a size of 1080×1920×3, which represents the height, width and RGB color channels of the image respectively. Finally, three 77GHz millimeter-wave radars arranged in the front bumper of the vehicle are used to transmit and receive the reflected electromagnetic wave signals at a frequency of 20Hz. Multiple radar reflection points are obtained each time, and each point contains information such as distance d, angle a, speed v, etc., which constitute the millimeter-wave radar data. , K is the number of reflection points detected. Through the on-board laser radar, camera and millimeter wave radar, the original multimodal perception data {P, I, R} within a certain time range is obtained. The data collected by different sensors have different data structures, physical meanings and spatiotemporal resolutions. Data calibration, feature extraction, target detection and fusion processing are required in subsequent steps to finally output high-precision three-dimensional target information of the environment around the unmanned vehicle for decision-making and control of the unmanned driving system.

[0040] Taking the vehicle-mounted laser radar as the benchmark, the Zhang calibration method is used to obtain the camera internal parameters. By extracting the edge information in the laser point cloud data and image data and matching the corresponding edge points, the pose transformation relationship between the camera coordinate system and the laser radar coordinate system is solved. First, a checkerboard calibration plate containing M×N black and white squares is set, and the side length of each square is L mm. The calibration plate is placed in different poses in the camera field of view, and K images are collected. For each image , extract the coordinates of the chessboard corner points , establish the mapping relationship from pixel coordinate system to world coordinate system: ; where s is the scale factor, A is the camera internal parameter matrix, including the focal length and main point , and is the rotation matrix and translation vector corresponding to the kth image, is the coordinate of the corner point of the chessboard in the world coordinate system. The camera intrinsic parameter matrix A is obtained by solving the above equations through Zhang's calibration method.

[0041] LiDAR-Camera extrinsic calibration: Move the checkerboard calibration plate to different positions in front of the vehicle and collect laser point cloud data simultaneously and image data . For point cloud Perform preprocessing and calculate each point The normal vector When the adjacent points and The normal vector angle When it is greater than the threshold value θ, and Mark as edge points and get the point cloud edge point set . For images Perform Canny edge detection and calculate the gradient magnitude M(u, v) and direction θ(u, v) at the pixel (u, v). Mark pixels with gradient magnitude greater than the threshold T as edge points to obtain the image edge point set. .

[0042] Establish the matching relationship between the edge points of the point cloud and the edge points of the image: for each point cloud edge point , extract its neighborhood point set ,calculate The geometric features (such as curvature, normal vector, etc.) are obtained Geometric feature descriptor For each image edge point , calculate its SIFT feature descriptor For each ,exist Find the feature descriptor with the closest Euclidean distance ,like ,but is a potential matching point pair, denoted by .

[0043] Use the RANSAC algorithm to estimate the optimal transformation matrix: randomly sample m matching point pairs , calculate the transformation matrix ,make For all potential matching point pairs, calculate the internal point set , where the interior points satisfy Repeat N times and select the transformation matrix with the most internal points. As the optimal estimate, the corresponding interior point set This is the final set of matching edge points. , use the least squares method to solve the rotation matrix from the laser radar to the camera and translation vectors , get the laser radar-camera external parameters At this point, through chessboard calibration and edge matching, the camera intrinsic parameter A and the laser radar-camera extrinsic parameter , realizing the unification of the laser radar and camera coordinate systems. The two sensor data are unified into the same coordinate system for data fusion processing.

[0044] Establish the coordinate transformation relationship from camera to lidar: According to the matching edge point set S^* obtained in the previous step, for each pair of matching points ,in is a 3D point in the camera coordinate system, For the three-dimensional points in the laser radar coordinate system, the following transformation relationship equation is established: ; Where R is a 3×3 rotation matrix and t is a 3×1 translation vector. R and t are combined into a 4×4 homogeneous transformation matrix T: ; The transformation matrix T describes the rigid body transformation relationship from the camera coordinate system to the lidar coordinate system.

[0045] Least squares solution to the transformation matrix: Substitute all M pairs of matching edge points into the transformation relationship equation to obtain M error equations: ; ;......; ; Arrange the above equations into a matrix form AX=B, where: , ; ; Where A is the M×12 coefficient matrix, X is the 12×1 transformation parameter vector, and B is the M×1 observation vector. The least squares method is used to solve the overdetermined equation system AX=B, that is, to find an X such that the error sum of squares Minimum. The solution formula is: ; After calculating the optimal transformation parameter vector X, rearrange it into a 4×4 matrix form to obtain the optimal transformation matrix T: , extract the rotation matrix and translation vector: extract the upper left 3×3 submatrix from the optimal transformation matrix T as the optimal rotation matrix R, and extract the first three elements of the last column as the optimal translation vector t*: , At this point, the optimal position transformation relationship between the camera coordinate system and the lidar coordinate system is obtained , achieving the purpose of converting the 3D point P in the camera coordinate system to the 3D point Q in the lidar coordinate system: .

[0046] Establishing the position transformation relationship between the millimeter-wave radar and the laser radar includes: arranging several millimeter-wave reflectors on the unmanned vehicle, and using the vehicle-mounted laser radar and the millimeter-wave radar to collect data synchronously. The laser radar identifies the reflector position by the reflection intensity. , millimeter wave radar identifies the reflector position through the reflected signal , where n is the number of reflectors, and represent the three-dimensional coordinates of the i-th reflector in the two sensor coordinate systems respectively.

[0047] Construct the optimization objective function: define the yaw angle of the millimeter wave radar relative to the laser radar as θ, then the corresponding two-dimensional rotation transformation matrix is: , construct the least squares error objective function: ; where ||·|| represents the Euclidean distance and ∑ represents the sum of all reflector positions. This function represents the sum of squared errors in coordinate transformation between the laser radar and millimeter wave radar coordinate systems of the reflector at a given yaw angle θ. The optimization goal is to find the optimal yaw angle , so that the objective function Minimum. Gradient descent solves the optimal yaw angle: Initialize the yaw angle is a random value, setting the learning rate α and the upper limit of the number of iterations K, and the error threshold ε. For the tth iteration: calculate the objective function In the current The gradient at : ,in, Update the yaw angle according to the gradient descent formula: ; Calculate the updated objective function value , determine whether the iteration termination condition is met: or, ;but, If the termination condition is not met, return to continue iteration. Calculate the optimal transformation matrix: convert the optimal yaw angle Substitute the rotation matrix to obtain the optimal transformation matrix from millimeter wave radar to lidar : At this point, by minimizing the reflector coordinate conversion error, the yaw angle of the millimeter-wave radar relative to the lidar is solved using gradient descent optimization. , and thus the optimal transformation relationship between the two sensor coordinate systems is obtained This transformation matrix can be used to subsequently align the millimeter-wave radar data to the lidar coordinate system to achieve data fusion positioning.

[0048] Figure 2 This is an exemplary flow chart of target detection and recognition according to some embodiments of the present application, laser point cloud target detection: 3D target detection is performed on laser point cloud data P using the PointRCNN model. PointRCNN consists of two parts: a segmentation branch and a detection branch. The segmentation branch divides the point cloud P into a foreground point set F and a background point set B. Specifically, PointNet++ is used to extract point cloud features, and then the segmentation score of each point is generated through the UNet structure, and the points with a score greater than the threshold τ are classified as foreground F. The detection branch predicts the 3D target box in the foreground point set F. First, a series of seed points S are sampled, and K adjacent points are sampled for each seed point as the center to extract local features. Then the RPN network is used to generate a 3D candidate box and perform non-maximum suppression to obtain Ro1 preliminary boxes. Then the point cloud in each preliminary box is sampled, grouped, and feature extracted, and sent to the refinement network to predict the precise 3D target box, category, and confidence. Finally, non-maximum suppression is performed on the Ro1 precise boxes to obtain Ro2 final detection results. ,in are the target frame parameters (center coordinates, length, width, height, and orientation angle), For categories, is the confidence score. These detection results are recorded as target information .

[0049] Image object detection: Use the YOLOv8 model to perform 2D object detection on image data I. YOLOv8 divides the input image into S×S grids and predicts B bounding boxes for each grid. For each bounding box b, the model predicts 5 parameters , respectively represent the center coordinates and width and height offsets of the box, as well as the target confidence. At the same time, predict C conditional category probabilities The bounding box offset is added to the grid coordinates and multiplied by the image size to get the absolute coordinates and dimensions (x, y, w, h) of the bounding box in the image. The object confidence is multiplied by the class probability to get the confidence score for each class. The prediction boxes are filtered by threshold and suppressed by non-maximum value to obtain the final detection result of Ro3. ,in is the bounding box parameter, For categories, is the confidence score. The image detection result is recorded as target information 2 .

[0050] Millimeter-wave radar target detection: For millimeter-wave radar data R, the DBSCAN clustering algorithm is used for target detection. Represented as a two-dimensional feature vector: ;in and are the distance and velocity measurements of the i-th reflection point, respectively. is the total number of reflection points. Perform DBSCAN clustering: Set the distance threshold ε and density threshold MinPts. , calculate the number of points in its ε-neighborhood .like ,but For each core point , find all its density reachable points and form a cluster . Mark all noise points (neither core points nor boundary points) as -1. Filter out clusters with less than N points and get Ro4 valid reflection point clusters. .

[0051] Target feature extraction: For each reflection point cluster , calculate its geometric features: number of points: ; Distance to average: ; Distance standard deviation: ; Average speed: ; Speed ​​standard deviation: ; The feature vector With predefined target templates Match and get clusters Corresponding target category : ; Where L is the total number of target categories, is the feature template of the lth type of target.

[0052] Target motion estimation: For each reflection point cluster , according to its average speed Estimate the target's velocity vector: ,in Cluster The azimuth average value of the velocity vector The size and direction of the target , determine the target's motion state :if ,but If and ,but Go straight. If and ,but Turn. is the speed threshold, Cluster In the azimuth of the previous frame, is the steering angle threshold. The information of the millimeter-wave radar target is organized as follows: : ,in is the reflection point cluster of the kth target, is the target category, The target motion state.

[0053] The target information It is passed to the subsequent data fusion and tracking modules to be associated and fused with the detection results of other sensors. The target detection process of the millimeter-wave radar converts the original reflection point data into a high-level target information representation through DBSCAN clustering, feature extraction and template matching. By analyzing and judging the motion state of the target, it provides dynamic environmental perception capabilities for the unmanned driving system. Due to its distance and speed detection capabilities, millimeter-wave radar has significant advantages in complex weather conditions and is one of the important sensors for environmental perception. In summary, through methods such as PointRCNN, YOLOv8 and DBSCAN, pedestrians, bicycles, cars and other targets are detected from laser point clouds, images and millimeter-wave radar data, and the target information of the three sensors is obtained. , , , used for subsequent multi-sensor fusion perception.

[0054] Figure 3 is an exemplary flow chart of the fusion processing of target data shown in some embodiments of the present application, according to the posture transformation relationship 1, the target information 2 is converted from the camera coordinate system to the laser radar coordinate system; for the target information detected by the camera , according to the transformation matrix Convert it from the camera coordinate system to the lidar coordinate system: ;in for The coordinates of the i-th target under the camera, To convert to laser radar coordinates, is the number of camera targets. The converted camera target information is For millimeter wave radar target information , using the transformation matrix To perform coordinate transformation: ;in for The coordinates of the jth target under millimeter wave, To convert to laser radar coordinates, is the number of millimeter wave targets. The converted millimeter wave target information is .

[0055] The target information of three sensors, lidar, camera and millimeter wave radar, is , , Unify in the laser radar coordinate system and record it as ,in is the total number of targets detected by all sensors. For the target set D in the unified coordinate system, the unscented Kalman filter (UKF) algorithm is used to fuse and estimate the state of each target. For the kth target , whose state vector is , containing the target's position, velocity, acceleration, and size information.

[0056] The unscented Kalman filter approximates the state distribution through Sigma points, avoiding the complex calculation of nonlinear function derivatives, including: establishing the state equation , and the measurement equation ;in, is the state vector at time k, , Contains the lateral position of the target at time k , vertical position , lateral speed , longitudinal speed , lateral acceleration , longitudinal acceleration , target width and target length ; is the state transfer matrix; is the process noise, H is the measurement matrix, is the measurement noise, k is the current moment, is the system observation vector; the Sigma point in the unscented Kalman filter algorithm is calculated by the following formula: ; ; ;in, for The mean of for The variance of ; λ represents the scaling parameter, , to ensure the semi-positive definiteness of the variance matrix, preferably , ɑ represents the scaling parameter, which indicates the degree to which the test point deviates from the expected value, and is usually a very small positive number; i represents a positive integer; Represents the scaling parameter.

[0057] Calculate the weight of each Sigma point; , ;in, is the weight of the mean, is the weight of the variance, which is used to represent the distribution information of X; where m represents the mean; c represents the variance; based on the obtained state equation, Sigma point and weight, the unscented Kalman filter algorithm is used to perform time update and observation update on the target information detected by each sensor to obtain the target state estimate.

[0058] The unscented Kalman filter algorithm is used to update the target information detected by each sensor in time, including: initializing the mean and covariance of the state vector: ; ; Calculate the Sigma point at time k according to the Sigma point selection formula : ; Update each Sigma point in time to get the predicted Sigma point : ; According to the predicted Sigma point , weighted calculation of state prediction mean : ; According to the predicted Sigma point and the state prediction mean , weighted calculation of state prediction covariance: .

[0059] The unscented Kalman filter algorithm is used to observe and update the target information detected by each sensor, including: predicting the Sigma point according to the state equation Observe and obtain the observed Sigma point : ; According to the observed Sigma point , weighted calculation of the observed predicted mean : ; According to the observed Sigma point and the observed predicted mean , calculate the prediction covariance matrix and the cross-correlation matrix between state space and prediction space : ; ; According to the prediction covariance matrix and the cross-correlation matrix , calculate the Kalman gain : ; Using Kalman gain , update the state estimate and the state covariance matrix : ; , according to the state estimation and the state covariance matrix , and obtain the target state estimate at time k+1.

[0060] Determine whether there are new observation targets. If so, initialize the corresponding state vector and add the state vector to the tracking list; at each time k, compare the observation target set and tracking target set .for Each observation target in , traverse All tracking targets in :if With any Match (if the Euclidean distance is less than the threshold δ), then it is considered Corresponding to an existing tracking target, no initialization is required. With all If none of them match, then For a new target, its state vector needs to be initialized. , according to its observed value Initialize the state vector and the covariance matrix : ; , where the position and size Taken directly from observations, the speed and acceleration Initialized to 0. Covariance matrix The diagonal elements of reflect the initial uncertainty of each state component. Add to Watchlist , forming an augmented tracking set .

[0061] Update of target status: For Each tracking target in , in the observation set Search for matching observation targets in If there is a matching , then use its observed value right The status is updated: ,in represents the observation update step of the unscented Kalman filter. If there is no matching observation target, the predicted state mean is used to update The status is updated: ,in Represents the time update (prediction) step of the unscented Kalman filter. Record each tracking target The number of consecutive unmatched observation frames .if If the threshold value T is exceeded (such as T=5), the target Track lost, remove it from the tracking list If When the observation is matched in the current frame, Reset to 0. So far, the updated tracking target set is obtained , as the prior information for the next moment.

[0062] Generate unmanned driving control instructions based on target state estimation. Each tracking target in , extract its state estimate: Position: ;speed: ; Acceleration: ;size: . Determine the type of target based on the target state attributes (pedestrians, bicycles, cars, etc.). Combined with the target state, type and the unmanned vehicle's own state (such as position, speed, heading angle, etc.), the control instructions of the unmanned vehicle are generated: If the target ahead is a stationary obstacle (speed is 0) and the distance is less than the safety threshold, a deceleration or stop instruction is generated. If the target ahead is a moving obstacle (speed is not 0) and the estimated collision time is less than the safety threshold, a deceleration, lane change or detour instruction is generated. If the target ahead is a pedestrian or bicycle, a deceleration instruction is generated to ensure the safety of traffic participants. If there is no target on the road ahead, a resume cruise or acceleration instruction is generated.

[0063] The control instructions are sent to the planning and control module of the unmanned vehicle to realize the automatic driving control of the vehicle. By initializing new targets, updating the status of existing targets, and deleting targets that have been lost for a long time, continuous tracking of multiple targets in a dynamic environment is achieved. And according to the state estimation of the tracked targets, the corresponding unmanned vehicle control instructions are generated to ensure the safety and smoothness of the automatic driving process. The combination of multi-sensor fusion and target tracking technology enables the unmanned vehicle to accurately perceive and understand complex traffic scenes, providing reliable environmental information for its autonomous decision-making.

Claims

1. A perception fusion method for an unmanned driving system, used in underground mining scenarios, characterized in that: include: Obtain laser point cloud data through vehicle-mounted laser radar, obtain image data through camera, and obtain millimeter-wave radar data through millimeter-wave radar; Taking the vehicle-mounted laser radar as the benchmark, the Zhang calibration method is used to obtain the camera internal parameters. By extracting the edge information in the laser point cloud data and image data and matching the corresponding edge points, the position transformation relationship between the camera coordinate system and the laser radar coordinate system is solved. Using the millimeter-wave reflector, the gradient descent method is used to solve the optimal heading angle of the millimeter-wave radar relative to the laser radar, and the position transformation relationship of the millimeter-wave radar relative to the laser radar is obtained; The PointRCNN method is used to detect the target on the laser point cloud data to obtain the target information 1; The YOLOv8 method is used to detect the target on the image data to obtain target information 2; Clustering algorithm is used to cluster the millimeter wave radar data to obtain target information three; Among them, targets include pedestrians, bicycles, and cars; According to the posture transformation relationship 1, the target information 2 is converted from the camera coordinate system to the laser radar coordinate system; According to the posture transformation relationship 2, the target information 3 is converted from the millimeter wave radar coordinate system to the laser radar coordinate system; Unify the target information 1, the converted target information 2 and the target information 3 in the laser radar coordinate system; The unscented Kalman filter algorithm is used to fuse the multi-sensor target information in a unified coordinate system to obtain the target state estimation; Generate unmanned driving control instructions based on target state estimation.

2. The perception fusion method of the unmanned driving system according to claim 1, characterized in that: Solve the position transformation relationship of the camera coordinate system relative to the lidar coordinate system, including: Set up a calibration board and use Zhang's calibration method to calibrate the camera's intrinsic parameters to obtain the camera's intrinsic parameters; Move the calibration plate to obtain laser point cloud data through the vehicle-mounted laser radar and obtain image data through the camera; According to the laser point cloud data, the normal vector of the point cloud is calculated by the point cloud processing algorithm. When the angle between the normal vectors of adjacent points is greater than the threshold, the corresponding point is regarded as the edge point of the laser point cloud. According to the image data, the edge detection algorithm is used to obtain the image edge points by calculating the gradient amplitude and direction of the image; Using feature descriptors and matching algorithms, edge points of laser point clouds and image edges are matched to obtain matched edge points. According to the matched edge points, a transformation relationship matrix 1 from the camera coordinate system to the laser radar coordinate system is established; The least square method is used to solve and obtain the position transformation relationship of the camera coordinate system relative to the lidar coordinate system.

3. The perception fusion method of the unmanned driving system according to claim 2, characterized in that: Matching laser point cloud edge points and image edge points, including: Calculate the geometric features of the point cloud within a preset range of the edge points of the laser point cloud to obtain the geometric feature descriptors of the edge points of the laser point cloud; Calculate the SIFT features or SURF features of the image edge points to obtain the feature descriptors of the image edge points; According to the geometric feature descriptors of the edge points of the laser point cloud and the feature descriptors of the edge points of the image, a nearest neighbor matching algorithm is used to obtain the feature descriptor pair with the smallest Euclidean distance between the edge points of the laser point cloud and the edge points of the image as a potential matching edge point pair; According to the potential matching edge point pairs, the RANSAC algorithm is used to construct the transformation matrix, and the number of inliers in the transformation matrix is ​​calculated. The transformation matrix with the largest number of inliers is selected as the optimal matrix, and the inliers of the optimal matrix are used as the matching edge point pairs; wherein, the inliers represent the matching point pairs that satisfy the transformation matrix.

4. The perception fusion method of the unmanned driving system according to claim 3, characterized in that: The pose transformation relationship of the camera coordinate system relative to the lidar coordinate system is obtained, including: According to each pair of matching edge points and , establish an equation describing the transformation relationship from the camera coordinate system to the lidar coordinate system , where R is a 3x3 rotation matrix and t is a 3x1 translation vector. The rotation matrix R and the translation vector t are combined into a 4x4 homogeneous transformation matrix T; The equation of the transformation relationship established for all matching edge point pairs As the error equation, an overdetermined system of equations AX=B is constructed, where A is an M×12 matrix, M is the number of matching edge point pairs, X is the unknown parameter vector of the 12×1 transformation matrix T, and B is the M×1 observation value vector; The least squares method is used to solve the overdetermined system of equations AX=B by minimizing the error equation The square sum of , we get an optimal transformation matrix T*, which represents the position transformation relationship between the camera coordinate system and the lidar coordinate system; The solved transformation matrix T* is decomposed into a 3x3 rotation matrix R* and a 3x1 translation vector t* as the position transformation relationship of the camera coordinate system relative to the lidar coordinate system.

5. The perception fusion method of the unmanned driving system according to claim 2, characterized in that: Obtaining the second position transformation relationship between the millimeter wave radar and the laser radar, including: Receiving laser point cloud data and millimeter wave radar data; wherein the laser point cloud data includes reflector position coordinate 1 detected by the laser radar, and the millimeter wave radar data includes reflector position coordinate 2 detected by the millimeter wave radar; Setting the objective function , where θ is the yaw angle of the millimeter wave radar relative to the laser radar, is the coordinate transformation matrix corresponding to θ, ;Σ represents the sum of all reflector positions; Initialize yaw angle is a random value; Calculate the objective function At yaw angle The gradient at ,gradient ; According to the gradient Update yaw angle , where α is the learning rate; Calculate the objective function After the updated yaw angle The value at Is it less than the preset threshold ε or the number of iterations reaches the preset value K? If so, the current yaw angle As the optimal yaw angle θ*, otherwise As new Continue to iterate; Substitute the optimal yaw angle θ into the transformation matrix , and obtain the second posture transformation relationship of the millimeter wave radar relative to the laser radar.

6. The perception fusion method of an unmanned driving system according to any one of claims 2 to 5, characterized in that: The unscented Kalman filter algorithm is used to fuse the multi-sensor target information in a unified coordinate system to obtain the target state estimation, including: Establishing the equation of state , and the measurement equation ; in, is the state vector at time k, , Contains the lateral position of the target at time k , vertical position , lateral speed , longitudinal speed , lateral acceleration , longitudinal acceleration , target width and target length ; is the state transfer matrix; is the process noise, H is the measurement matrix, is the measurement noise, k is the current moment, is the system observation vector; The Sigma point in the unscented Kalman filter algorithm is calculated using the following formula: ； ； ； in, for The mean of for The variance of ; λ represents the scaling parameter; ɑ represents the scaling parameter; i represents a positive integer; represents the scaling parameter; Calculate the weight of each Sigma point; According to the obtained state equation, Sigma point and weight, the unscented Kalman filter algorithm is used to perform time update and observation update on the target information detected by each sensor to obtain the target state estimation value.

7. The perception fusion method of the unmanned driving system according to claim 6, characterized in that: The weight of each Sigma point is calculated by the following formula: ； ； in, is the weight of the mean, is the weight of the variance; m represents the mean; c represents the variance.

8. The perception fusion method of the unmanned driving system according to claim 7, characterized in that: The unscented Kalman filter algorithm is used to update the target information detected by each sensor in time, including: Initialize the mean and covariance of the state vector: ； ； Calculate the Sigma point at time k according to the Sigma point selection formula : ; Update each Sigma point in time to get the predicted Sigma point : ; According to the predicted Sigma point , weighted calculation of state prediction mean : ; According to the predicted Sigma point and the state prediction mean , weighted calculation of state prediction covariance: 。 9. The perception fusion method of the unmanned driving system according to claim 8, characterized in that: The unscented Kalman filter algorithm is used to observe and update the target information detected by each sensor, including: According to the state equation, the predicted Sigma point Observe and obtain the observed Sigma point : ； According to the observed Sigma point , weighted calculation of the observed predicted mean : ； According to the observed Sigma point and the observed predicted mean , calculate the prediction covariance matrix and the cross-correlation matrix between state space and prediction space : ； ； According to the prediction covariance matrix and the cross-correlation matrix , calculate the Kalman gain : ； Using Kalman Gain , update the state estimate and the state covariance matrix : ； ； According to the state estimate and the state covariance matrix , and obtain the target state estimate at time k+1.

10. The perception fusion method of the unmanned driving system according to claim 9, characterized in that: Also includes: Determine whether there are new observation targets. If so, initialize the corresponding state vector and add the state vector to the tracking list; Determine whether each tracking target matches the observation data at time k; If so, use the observed data to update the corresponding state; If no, use the state-predicted mean Update the corresponding status; For tracking targets that are not matched to observation data in consecutive T frames, they are considered untracked and lost and deleted.

Images