Radar-vision fusion target perception method, system, device, processor and computer-readable storage medium based on hybrid strategy

Abstract

The present invention discloses a method for target perception based on a hybrid strategy of radar-visual fusion, including: setting a camera and a 4D millimeter-wave radar sensor to be triggered synchronously; obtaining the category of the target in the camera image data and 12 key points-2D Box results based on an end-to-end NMS-free 2D target key point detection network model; using the homography matrix H obtained by the IPM method to project the ground point in the key point to the road bev perspective; using the external parameter calibration result to map the point cloud to the two-dimensional image data; filtering the 4D millimeter-wave radar point cloud in the two-dimensional image space according to the target 12 key points-2D Box result information in the two-dimensional image; and performing data post-processing on the filtered point cloud in the three-dimensional space. The present invention also relates to a corresponding system, device, processor and storage medium thereof. The method, system, device, processor and storage medium thereof of the present invention can provide a radar-visual fusion target perception result with high accuracy, high robustness and high redundancy.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to the field of vehicle target fusion perception, and specifically refers to a radar-vision fusion target perception method, system, device, processor and computer-readable storage medium thereof based on a hybrid strategy. Background Art

[0002] In the field of autonomous driving and vehicle-road collaboration, it is usually necessary to perceive vehicle targets on the road. In order to ensure the accuracy of the perception results and make up for the insufficient performance of a single sensor, it is usually necessary to fuse data from multiple sensors for vehicle target perception.

[0003] Taking the patent with application number 202310723468.2 as an example, it mainly uses lidar and camera for radar-vision fusion perception. Although lidar has higher point cloud accuracy and density than traditional 3D millimeter-wave radar (which can only provide target distance, angle, and speed information), lidar currently has common shortcomings such as high cost and susceptibility to rain and snow, and its perception robustness is poorer than that of millimeter-wave radar. At the same time, this method uses deep learning networks for target detection for both image data and three-dimensional lidar point cloud data, which will bring about the problem of large computing power consumption, increasing system costs while also being unfavorable for the deployment of the method on edge computing platforms.

[0004] Taking patents with application numbers 202310109166.6 and 202310932152.4 as examples, they mainly adopt the idea of ​​traditional 3D millimeter-wave radar for radar-vision fusion, obtain target trajectory data of radar equipment and camera equipment respectively, and then fuse and optimize them at the target trajectory level, and finally obtain the fused target result. Relying on such methods cannot fully utilize the advantages of 4D millimeter-wave radar (which provides target distance, angle, height, and speed information at the same time) with high point cloud density and high-dimensional information, and only performing data association at the target trajectory level is prone to target matching errors, which can easily lead to loss of target perception accuracy.

[0005] Based on this, in view of the various defects in the existing technology, it is necessary to propose a data fusion solution that combines radar and camera data in a two-dimensional + three-dimensional hybrid strategy to make up for the shortcomings of the traditional 3D millimeter-wave radar and radar-vision fusion perception method, such as poor target positioning accuracy and error-prone target association matching. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a radar-vision fusion target perception method, system, device, processor and computer-readable storage medium thereof based on a hybrid strategy.

[0007] In order to achieve the above-mentioned object, the radar-visual fusion target perception method, system, device, processor and computer-readable storage medium thereof based on hybrid strategy of the present invention are as follows:

[0008] The main feature of the radar-visual fusion target perception method based on the hybrid strategy is that the method comprises the following steps:

[0009] (1) Setting the camera and the 4D millimeter-wave radar sensor to be triggered synchronously at a specific time interval to obtain camera image data and 4D millimeter-wave radar point cloud data in a time-aligned state;

[0010] (2) Based on the end-to-end NMS-free 2D target key point detection network model, the target key point detection results in the camera image data are obtained: target category, 12 key points and 2D Bo x result;

[0011] (3) Using the pixels and spatial coordinates of the corner points of the open road to perform IPM to solve the homography matrix H, the target road grounding point in the key point detection result is converted to the road bev perspective using the homography matrix H to obtain the heading angle of the target in space;

[0012] (4) Use the external parameter calibration results to map the 4D millimeter wave radar point cloud into two-dimensional image data;

[0013] (5) filtering the 4D millimeter wave radar point cloud in the two-dimensional image space according to the target 12 boundary point result information in the two-dimensional image;

[0014] (6) The filtered 4D millimeter-wave radar point cloud is post-processed in three-dimensional space to fully restore the target’s 3D position, speed, category, and orientation angle information.

[0015] Preferably, the step (2) specifically comprises the following steps:

[0016] (2.1) Image data preprocessing: Dedistorting the camera image data using the camera intrinsic parameter matrix and distortion coefficients, and converting the image size and image storage mode into a format suitable for input of the end-to-end NMS-free 2D target key point detection network model;

[0017] (2.2) Network model reasoning: The data after data pre-processing is sent to the network model reasoning to obtain the target detection data results;

[0018] (2.3) Image data post-processing: Convert the format of the network model output data to output the two-dimensional visual key point detection results of the corresponding target.

[0019] Preferably, the step (2.1) specifically includes the following steps:

[0020] (2.1.1) First, use the OpenCV camera calibration tool to complete the camera calibration process and obtain the camera intrinsic parameter matrix And the distortion coefficient D = [k1, k2, p1, p2, k3], where f x is the horizontal focal length, f y is the vertical focal length, s is the coordinate axis tilt factor (skewfactor), c x is the y coordinate of the optical center in the image coordinate system, c y is the y coordinate of the optical center in the image coordinate system, k1 to k3 are the first-order, second-order, and third-order radial distortion coefficients, respectively, and p1 and p2 are the tangential distortion coefficients;

[0021] (2.1.2) For any pixel point P in the camera image uv1 =[u,v,1] T , using the internal parameter matrix K to obtain the corresponding normalized camera coordinate system coordinate P = K -1 P uv1 , whose coordinates are [x, y, 1] T , the distance between point P and the origin of the coordinate system is r, and the corresponding camera image is dedistorted using the following formula:

[0022]

[0023] Get the normalized camera coordinates P after dedistortion distorted =[x distorted ,y distorted , 1] T , and then get the pixel position P after dedistortion uv_distorted =KP distorted , where T represents vector transpose;

[0024] (2.1.3) Resize the dedistorted camera RGB image to 640×640 while maintaining its aspect ratio. Fill the short side blanks with 114 pixels, and place the original image in the center of the resized image.

[0025] (2.1.4) Convert the original data storage format of the camera image from RGB three-channel data of (640, 640, 3) dimension to array data of (1, 3, 640, 640) dimension;

[0026] (2.1.5) This completes the pre-processing of the camera image.

[0027] Preferably, the end-to-end NMS-free 2D target key point detection network model in step (2.2) specifically includes the following steps:

[0028] (2.2.1) Model Backbon e CSPDarknet is used, Neck uses PAFPN, and three scale feature maps (20×20, 40×40, 80×80) are used to meet the requirements of multi-scale target feature extraction;

[0029] (2.2.2) The network sets up three detection head branches: classification branch, box detection branch and points detection branch. The three branches perform concat operations at each scale, and then concat the feature map results of each scale to output. In the classification branch, the output dimension is set to the number of categories; in the box detection branch, the output dimension is set to 64 (4×16); in the key point branch, the key point output dimension is set to 36 to achieve the detection of 12 key points (x, y, score) of the target;

[0030] (2.2.3) During the training process, two positive and negative sample allocation strategies are set: topk10 and topk1. Topk10 is used to enrich the model's positive samples and make the model converge quickly; topk1 is used for subsequent model export, inference, and reduction of model output;

[0031] (2.2.4) In the model inference and export stage, the feature map of the topk1 branch in the above training stage is split according to the number of categories, 64, and 36, and is divided into category decoding branch, box decoding branch, and key point decoding branch in turn. The classification branch directly performs sigmoid decoding; the box branch first reduces the 64 dimensions to 4 dimensions through dfl regression, and uses anchor points to predict the upper left point (x, y) and lower right point (x, y) of the box box respectively; the point branch directly uses anchor points to predict 12 key points (x, y, score), and the score of each point is directly generated by sigmoid;

[0032] (2.2.5) After the decoded results are concat, channel transformed, and split, the top 30 targets are selected by category using topk1. The selected results are gathered with the boxes and points, and finally 30 target results are output.

[0033] Preferably, the step (2.3) specifically comprises the following steps:

[0034] (2.3.1) Slice the obtained one-dimensional array data according to the model output dimension, that is, the number of categories + 4 + 36, to obtain 30 output targets;

[0035] (2.3.2) Traverse each target result, obtain the target type according to the category corresponding to the maximum value in the category score, and compare the category score with the output threshold confidence. If it is greater than the threshold, it is the detected target. The final target result includes the target type, target bo x Results and objectives12 key point results.

[0036] Preferably, the step (3) specifically comprises the following steps:

[0037] (3.1) Select an empty road scene with clear lane lines, place the vehicle in the middle of the road, with the front direction of the vehicle parallel to the lane line direction, select the four corner points of a pair of parallel lane lines, obtain the pixel values ​​of the four corner points on the image and the (x, y) values ​​of the vehicle coordinate system in space to establish point pairs, and use the four matching point pairs to solve the IPM inverse transformation homography matrix H from the image perspective to the road bev space;

[0038] (3.2) According to the homography matrix H, the target ground point in the image key point detection is converted to the road be v Perspective, solve the angle between the target head and tail vector and the vehicle's driving direction, and obtain the target heading angle.

[0039] Preferably, the step (4) specifically comprises the following steps:

[0040] (4.1) Obtain the ego-camera extrinsic matrix H c , this matrix is ​​used to transform the vehicle coordinate system to the camera pixel coordinate system;

[0041] (4.2) Obtain radar-vehicle external parameter matrix H r , this matrix is ​​used to transform the radar coordinate system to the vehicle coordinate system;

[0042] (4.3) Obtain the radar-camera extrinsic matrix H, which is used to transform the radar coordinate system to the camera pixel coordinate system;

[0043] (4.4) In the coordinate mapping process, the corresponding camera coordinate system coordinates are obtained according to the radar-camera extrinsic matrix H, the camera coordinate system coordinates are normalized, and the pixel coordinates are obtained using the camera intrinsic parameter matrix K, so as to obtain the complete data of the 4D millimeter wave point cloud.

[0044] Preferably, the step (4.1) specifically comprises the following steps:

[0045] (4.1.1) According to the internal parameter matrix K and external parameter matrix H c Available In, Z c is the z-axis coordinate in the camera coordinate system, H c is a 3×4 dimensional external parameter matrix;

[0046] (4.1.2) Construct a nonlinear least squares problem: In this way, the vehicle-camera external parameter matrix H is obtained. c .

[0047] Preferably, the step (4.2) specifically includes the following steps:

[0048] (4.2.1) According to the coordinates P = (x, y, z) of the center points of several small corner reflectors in the vehicle calibration room relative to the vehicle coordinate system T , use the millimeter wave radar calibration tool to obtain the coordinates P of the center of the corner reflector in the radar coordinate system r =(x r ,y r , z r ) T , where P = R r P r +t r , R r is the rotation matrix from the radar coordinate system to the vehicle coordinate system, t r =[t x t y t z ] T is the translation vector;

[0049] (4.2.2) Construct the linear least squares problem: In this way, the radar-vehicle external parameter matrix H is obtained. r .

[0050] Preferably, the step (4.3) is specifically as follows:

[0051] According to the radar-vehicle external parameter matrix H r and the ego-camera extrinsic matrix H c , obtain the radar-camera extrinsic matrix H as follows:

[0052]

[0053] Among them, R is the rotation matrix in the radar-camera extrinsic matrix, t is the translation vector in the radar-camera extrinsic matrix, and R r is the rotation matrix in the radar-vehicle extrinsic matrix, t r is the translation vector in the radar-ego vehicle extrinsic matrix, R c is the rotation matrix in the ego-camera extrinsic matrix, t c is the translation vector in the ego-camera extrinsic matrix.

[0054] Preferably, the step (5) specifically comprises the following steps:

[0055] (5.1) Using the 12 key points of the target obtained by visual two-dimensional detection - 2DBox information, construct the geometric outer bounding box of the target in the two-dimensional image;

[0056] (5.2) According to the pixel coordinates (u, v) in the 4D millimeter-wave radar point cloud data, determine whether it is located within the geometric outer enclosure of the target in the two-dimensional image. If so, proceed to step (5.2), otherwise proceed to step (5.3);

[0057] (5.3) Based on Z-Scor e The method detects speed outliers in the same detection frame, specifically:

[0058] For all the radar point cloud speed values ​​in each detection frame, the mean speed μ is calculated speed and the speed standard deviation σ speed ; Calculate the speed Z-Score value of each point cloud: If the Z value is greater than the empirical threshold, the point is considered to be a velocity outlier and the corresponding radar point cloud is directly deleted; otherwise, the point is a non-velocity outlier and proceeds to step (5.4);

[0059] (5.4) Further determine whether the distance of the corresponding radar point cloud relative to the vehicle exceeds the trust distance threshold of visual perception, and directly delete the point cloud that does not exceed the threshold, otherwise, go to step (5.5);

[0060] (5.5) The corresponding point cloud is retained for subsequent clustering and tracking processing, and the complete data of all radar point clouds after point cloud filtering is obtained.

[0061] Preferably, the step (6) specifically comprises the following steps:

[0062] (6.1) Use the uniformly accelerated linear motion model to predict the state of the target in the previous frame;

[0063] (6.2) Using the density-based DBSCAN clustering algorithm to perform three-dimensional point cloud clustering on the radar point cloud filtered by the current frame;

[0064] (6.3) Use the Kalman filter to filter the target prediction state of the previous frame and the target state after clustering in the current frame to obtain stable target 3D position and velocity information;

[0065] (6.4) combining the filtered target speed and the target three-dimensional heading angle obtained by visual detection in step (2), updating the target heading angle information, specifically: if the absolute value of the filtered target speed is greater than a preset threshold, the heading angle obtained after filtering is used as the final heading angle of the target; if the absolute value of the filtered target speed is not greater than the preset threshold, the target three-dimensional heading angle obtained by visual detection is used as the final heading angle of the target;

[0066] (6.5) Combined with the target category obtained in step (2.3), the 3D position, speed, category and orientation angle information of the target can be fully obtained;

[0067] (6.6) Repeat the above steps to achieve the continuous clustering tracking process of the target.

[0068] Preferably, the step (6.1) is specifically as follows:

[0069] Assume that the motion state of the target in the previous frame is X = [x, y, v x , v y , a x , a y ] T , where x, y, v x , v y , a x , a y They are the x-axis coordinate, y-axis coordinate, x-axis speed, y-axis speed, x-axis acceleration, and y-axis acceleration of the target in the previous frame.

[0070] Correspondingly, the predicted target motion state is X′=[x′, y′, v′ x , v′ y , a′ x , a′ y ] T , then X′=FX, F is the state transfer matrix corresponding to the uniformly accelerated linear motion model: Where Δt represents the time difference between two frames.

[0071] Preferably, the step (6.3) specifically comprises the following steps:

[0072] (6.3.1) Based on the nearest neighbor data association principle, the predicted target of the previous frame and the clustered target of the current frame are associated according to the location of the target;

[0073] (6.3.2) Calculating Kalman Gain Using Kalman Filter Among them, H is the observation matrix, P is the prediction error covariance matrix, and R is the observation noise covariance matrix;

[0074] (6.3.3) Calculate the final target output: in, is the final target state output, Z is the target prediction state of the previous frame, X is the target state after clustering in the current frame, and the h function is the conversion function from the target state to the target prediction state.

[0075] The system for realizing radar-visual fusion target perception processing based on a hybrid strategy using the above-mentioned method is mainly characterized in that the system comprises:

[0076] The sensor synchronization trigger acquisition module is used to trigger the camera and 4D millimeter wave radar sensor to start a frame of data acquisition at a specific time interval to obtain data from the two sensors at the same time;

[0077] An image key point detection module is connected to the sensor synchronization trigger acquisition module for processing image sensor data and outputting target detection categories, 2D Boxes and 12 key point results using an end-to-end NMS-free 2D target key point detection network model;

[0078] The IPM module for solving the heading angle is connected to the image key point detection module, and converts the ground point in the image key point to the road bev perspective through the pre-selected open road pixel value and the spatial coordinate point through the homography matrix H solved by the IPM, and solves the target heading angle;

[0079] A radar point cloud coordinate conversion module is connected to the image key point detection module and is used to convert the collected radar point cloud from a three-dimensional coordinate system to a two-dimensional image coordinate system according to the sensor external parameter calibration parameters, and simultaneously obtain the two-dimensional and three-dimensional coordinate information of the radar point cloud;

[0080] A point cloud filtering module is connected to the radar point cloud coordinate conversion module and the IPM heading angle solution module, and is used to filter the 4D radar point cloud in the two-dimensional image space according to the geometric outer enclosure result information constructed by the 12 key points of the target in the image, so as to eliminate the false targets that may exist in the point cloud;

[0081] The point cloud post-processing module is communicated with the point cloud filtering module and is used to perform data post-processing on the filtered 4D millimeter wave radar point cloud in three-dimensional space to completely restore the target's 3D position, speed, category and orientation angle information.

[0082] The main feature of the hybrid strategy-based radar-visual fusion target perception device is that the device comprises:

[0083] a processor configured to execute computer executable instructions;

[0084] A memory stores one or more computer executable instructions, and when the computer executable instructions are executed by the processor, the steps of the above-mentioned radar-visual fusion target perception method based on the hybrid strategy are implemented.

[0085] The main feature of the radar-visual fusion target perception processor based on a hybrid strategy is that the processor is configured to execute computer-executable instructions, and when the computer-executable instructions are executed by the processor, the steps of the radar-visual fusion target perception method based on a hybrid strategy are implemented.

[0086] The main feature of the computer-readable storage medium is that a computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the above-mentioned radar-visual fusion target perception method based on a hybrid strategy.

[0087] The hybrid strategy-based radar-vision fusion target perception method, system, device, processor and computer-readable storage medium thereof of the present invention are adopted, and a two-dimensional + three-dimensional hybrid strategy is innovatively adopted for radar-vision data fusion, and data fusion is performed in a two-dimensional image coordinate system and a three-dimensional space coordinate system at the same time, so as to improve the accuracy and precision of the target perception results; the advantages of 4D millimeter-wave radar, such as strong robustness, high point cloud density and provision of high-dimensional information, are fully utilized to provide high-accuracy, high-robustness and high-redundancy radar-vision fusion target perception results, and make up for the shortcomings of poor target positioning accuracy and error-prone target association matching in traditional 3D millimeter-wave radar radar-vision fusion perception methods; compared with the radar-vision fusion solution of lidar + camera, only the deep learning network model is used for the camera data, and the dependence on the computing power of the computing platform is relatively low, and it is easy to deploy on the edge computing platform, with lower system cost and stronger robustness, which is conducive to large-scale mass production and application in the fields of autonomous driving and vehicle-road collaboration. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 The present invention is a processing flow chart of the radar-vision fusion target perception method based on a hybrid strategy.

[0089] Figure 2 The present invention is a flow chart of point cloud filtering processing based on the hybrid strategy of the radar-vision fusion target perception method of the present invention.

[0090] Figure 3 The present invention provides a flow chart for solving the heading angle using the radar-vision fusion target perception method based on a hybrid strategy.

[0091] Figure 4 Road bev graph during the process of solving the homography matrix H using the IPM solution method.

[0092] Figure 5The figure is a schematic diagram of the two-dimensional key point detection result and radar point cloud in the camera image according to the present invention.

[0093] Figure 6 It is a structural schematic diagram of the system for realizing radar-vision fusion target perception processing based on a hybrid strategy of the present invention. DETAILED DESCRIPTION

[0094] In order to more clearly describe the technical content of the present invention, further description is given below in conjunction with specific embodiments.

[0095] Before describing in detail embodiments according to the present invention, it should be noted that, hereinafter, the terms "comprises", "includes" or any other variations are intended to cover non-exclusive inclusion, whereby a process, method, article or apparatus comprising a series of elements includes not only these elements, but also other elements not explicitly listed or inherent to such process, method, article or apparatus.

[0096] See also Figure 1 As shown, the radar-visual fusion target perception method based on the hybrid strategy, wherein the method comprises the following steps:

[0097] (1) Setting the camera and the 4D millimeter-wave radar sensor to be triggered synchronously at a specific time interval to obtain camera image data and 4D millimeter-wave radar point cloud data in a time-aligned state;

[0098] (2) Obtain the target key point detection results in the camera image data based on the end-to-end 2D target key point detection network model without NMS (non-maximum suppression): target category, 12 key points and 2D Box results;

[0099] (3) Using the corner pixels and spatial coordinates of the open road, the IPM is used to solve the homography matrix H. The target road grounding point in the key point detection result is converted to the road bev perspective using the above H to obtain the heading angle of the target in space;

[0100] (4) Use the external parameter calibration results to map the 4D millimeter wave radar point cloud into two-dimensional image data;

[0101] (5) filtering the 4D millimeter wave radar point cloud in the two-dimensional image space according to the target 12 boundary point result information in the two-dimensional image;

[0102] (6) The filtered 4D millimeter-wave radar point cloud is post-processed in three-dimensional space to restore the target’s 3D position, category, and orientation angle information.

[0103] As a preferred embodiment of the present invention, the step (2) specifically comprises the following steps:

[0104] (2.1) Image data preprocessing: Dedistorting the camera image data using the camera intrinsic parameter matrix and distortion coefficients, and converting the image size and image storage mode into a format suitable for input of the end-to-end NMS-free 2D target key point detection network model;

[0105] (2.2) Network model reasoning: The data after data pre-processing is sent to the network model reasoning to obtain the target detection data results:

[0106] (2.3) Image data post-processing: Convert the format of the network model output data to output the two-dimensional visual key point detection results of the corresponding target.

[0107] As a preferred embodiment of the present invention, the step (2.1) specifically includes the following steps:

[0108] (2.1.1) First, use the OpenCV camera calibration tool to complete the camera calibration process and obtain the camera intrinsic parameter matrix And the distortion coefficient D = [k1, k2, p1, p2, k3], where f x is the horizontal focal length, f y is the vertical focal length, s is the coordinate axis tilt factor (skewfactor), c x is the y coordinate of the optical center in the image coordinate system, c y is the y coordinate of the optical center in the image coordinate system, k1 to k3 are the first-order, second-order, and third-order radial distortion coefficients, respectively, and p1 and p2 are the tangential distortion coefficients;

[0109] (2.1.2) For any pixel point P in the camera image uv1 =[u,v,1] T , using the internal parameter matrix K to obtain the corresponding normalized camera coordinate system coordinate P = K -1 P uv1 , whose coordinates are [x, y, 1] T , the distance between point P and the origin of the coordinate system is r, and the corresponding camera image is dedistorted using the following formula:

[0110]

[0111] Get the normalized camera coordinates P after dedistortion distorted =[x distorted ,y distorred , 1] T , and then get the pixel position P after dedistortion uv_distorred =KP distorted, where k1 to k3 are the first-order, second-order, and third-order radial distortion coefficients, p1 and p2 are two tangential distortion coefficients, and T represents vector transposition;

[0112] (2.1.3) Resize the dedistorted camera RGB image to 640×640 while maintaining its aspect ratio. Fill the short side blanks with 114 pixels, and place the original image in the center of the resized image.

[0113] (2.1.4) Convert the original data storage format of the camera image from RGB three-channel data of (640, 640, 3) dimension to array data of (1, 3, 640, 640) dimension;

[0114] (2.1.5) This completes the pre-processing of the camera image.

[0115] As a preferred embodiment of the present invention, the end-to-end NMS-free 2D target key point detection network model in step (2.2) specifically includes the following steps:

[0116] (2.2.1) The backbone network model Backbone adopts the CSPDarknet network, and the neck network Neck adopts the PAN+FPN structure. Three scale feature maps (20×20, 40×40, 80×80) are used to meet the requirements of multi-scale target feature extraction;

[0117] (2.2.2) The network sets up three detection head branches: classification branch, box detection branch and points detection branch. The three branches perform concat (channel splicing) operations at each scale, and then concat the feature map results of each scale. In the classification branch, the output dimension is set to the number of categories; in the box detection branch, the output dimension is set to 64 (4×16); in the key point branch, the key point output dimension is set to 36 to achieve the detection of 12 key points (x, y, score) of the target;

[0118] (2.2.3) During the training process, two positive and negative sample allocation strategies are set: topk10 (elements ranked in the top 10 in scores) and topk1 (element ranked first in scores). Topk10 is used to enrich the model’s positive samples and make the model converge quickly; topk1 is used for subsequent model export, inference, and to reduce the model’s output.

[0119] (2.2.4) In the model inference and export stage, the feature map of the topk1 branch in the above training stage is split into channels according to the number of categories, 64, and 36, and is divided into category decoding branch, box decoding branch, and key point decoding branch in turn. The classification branch directly performs sigmoid decoding; the box branch first reduces the 64-dimensionality to 4 dimensions through dfl (distributed focus loss) regression, and uses anchor points to predict the upper left point (x, y) and lower right point (x, y) of the box box respectively; the point branch directly uses anchor points to predict 12 key points (x, y, score), and the score of each point is directly generated by sigmoid;

[0120] (2.2.5) After the decoded results are concat, channel transformed, and split, the top 30 targets are selected by category using topk1. The selected results are gathered with the box and points to extract the results at the specified position, and finally 30 target results are output.

[0121] As a preferred embodiment of the present invention, the step (2.3) specifically includes the following steps:

[0122] (2.3.1) Slice the obtained one-dimensional array data according to the model output dimension, that is, the number of categories + 4 + 36, to obtain 30 output targets;

[0123] (2.3.2) Traverse each target result, obtain the target category according to the category corresponding to the maximum value in the category score, and compare the category score with the output threshold confidence. If it is greater than the threshold, it is the detected target. The final target result includes the target category, target box result and the target 12 key point results.

[0124] As a preferred embodiment of the present invention, the step (3) specifically comprises the following steps:

[0125] (3.1) Select an empty road scene with clear lane lines, place the vehicle in the middle of the road, with the front direction of the vehicle parallel to the lane line direction, select the four corner points of a pair of parallel lane lines, obtain the pixel values ​​of the four corner points on the image and the (x, y) values ​​of the vehicle coordinate system in space to establish point pairs, and use the four matching point pairs to solve the IPM inverse transformation homography matrix H from the image perspective to the road bev space;

[0126] The two-dimensional road bev viewing space is established with the vehicle coordinate system (z=0). At this time, the camera pixel coordinates and the bev space have the following relationship:

[0127]

[0128] In the above formula, (x, y) is the point in bev space, (u, v) is the image pixel value, and H is the transformation matrix to be solved. The two formulas are combined and expanded to get:

[0129]

[0130] Under the scale invariance of the homogeneous coordinate system, the degree of freedom of H is 8. In the solution process, multiply by a non-zero factor so that h9 = 1 (when it takes a non-zero value). Rearrange it to:

[0131]

[0132] A set of pixels and be v Two constraints can be constructed from the spatial points, and the homography matrix H with 8 degrees of freedom can be calculated through 4 pairs of matching points, that is, solving the following linear equations:

[0133]

[0134] In the above formula, (u i , v i ) is the pixel coordinate of the i-th pairing point, (x i ,y i ) is the coordinate of the bev space point in the i-th group of paired points. Through the above calculation formula, the homography matrix H required for the IPM inverse transformation can be solved through 4 pairs of matching points.

[0135] (3.2) According to the homography matrix H, the target ground point in the image key point detection is converted to the road bev perspective, and the angle between the target head and tail vector and the vehicle's driving direction is solved to obtain the target heading angle.

[0136] Taking the vehicle target as an example, four tire contact point key points are selected from the key point detection output results. The center point of the front wheel is obtained using the two key points of the front wheel (taking the average), and the center point of the rear wheel is obtained using the two key points of the rear wheel. The above two center points are converted to the road bev space using the H matrix, and the vehicle direction vector is constructed from the center point of the rear of the vehicle to the center point of the front of the vehicle after the conversion. The angle between the direction vector and the unit vector directly in front of the vehicle (that is, the vector constructed from (0, 0) to (1, 0)) is solved as the heading angle of the target. The formula for solving the angle between vector v2 and vector v1 is:

[0137] θ=tan -1 (v 2_y , V 2_x )-tan -1 (v 1_y , V 1_x )

[0138] As a preferred embodiment of the present invention, the step (4) specifically comprises the following steps:

[0139] (4.1) Obtain the ego-camera extrinsic matrix H c , this matrix is ​​used to transform the vehicle coordinate system to the camera pixel coordinate system;

[0140] (4.2) Obtain radar-vehicle external parameter matrix H r , this matrix is ​​used to transform the radar coordinate system to the vehicle coordinate system;

[0141] (4.3) Obtain the radar-camera extrinsic matrix H, which is used to transform the radar coordinate system to the camera pixel coordinate system;

[0142] (4.4) In the coordinate mapping process, the corresponding camera coordinate system coordinates are obtained according to the radar-camera extrinsic matrix H, the camera coordinate system coordinates are normalized, and the pixel coordinates are obtained using the camera intrinsic parameter matrix K, so as to obtain the complete data of the 4D millimeter wave point cloud.

[0143] As a preferred embodiment of the present invention, the step (4.1) specifically includes the following steps:

[0144] (4.1.1) According to the internal parameter matrix K and external parameter matrix H c Available In, Z c is the z-axis coordinate in the camera coordinate system, H c is a 3×4 dimensional external parameter matrix;

[0145] (4.1.2) Construct a nonlinear least squares problem: In this way, the vehicle-camera external parameter matrix H is obtained. c .

[0146] As a preferred embodiment of the present invention, the step (4.2) specifically includes the following steps:

[0147] (4.2.1) According to the coordinates P = (x, y, z) of the center points of several small corner reflectors in the vehicle calibration room relative to the vehicle coordinate system T , use the millimeter wave radar calibration tool to obtain the coordinates P of the center of the corner reflector in the radar coordinate system r =(x r ,y r , z r ) T , where P = R r P r +t r , R r is the rotation matrix from the radar coordinate system to the vehicle coordinate system, t r =[t x t y t z ]T is the translation vector;

[0148] (4.2.2) Construct the linear least squares problem: In this way, the radar-vehicle external parameter matrix H is obtained. r .

[0149] As a preferred embodiment of the present invention, the step (4.3) is specifically as follows:

[0150] According to the radar-vehicle external parameter matrix H r and the ego-camera extrinsic matrix H c , obtain the radar-camera extrinsic matrix H as follows:

[0151]

[0152] Among them, R is the rotation matrix in the radar-camera extrinsic matrix, t is the translation vector in the radar-camera extrinsic matrix, and R r is the rotation matrix in the radar-vehicle extrinsic matrix, t r is the translation vector in the radar-ego vehicle extrinsic matrix, R c is the rotation matrix in the ego-camera extrinsic matrix, t c is the translation vector in the ego-camera extrinsic matrix.

[0153] As a preferred embodiment of the present invention, the step (5) specifically comprises the following steps:

[0154] (5.1) Using the 12 key points of the target obtained by visual two-dimensional detection - 2DBox information, construct the geometric outer bounding box of the target in the two-dimensional image;

[0155] (5.2) According to the pixel coordinates (u, v) in the 4D millimeter-wave radar point cloud data, determine whether it is located within the geometric outer enclosure of the target in the two-dimensional image. If so, proceed to step (5.2), otherwise proceed to step (5.3);

[0156] (5.3) Based on Z-Scor e The method detects speed outliers in the same detection frame, specifically:

[0157] For all the radar point cloud speed values ​​in each detection frame, the mean speed μ is calculated speed and the speed standard deviation σ speed ; Calculate the speed Z-Score value of each point cloud: If the Z value is greater than the empirical threshold, the point is considered to be a velocity outlier and the corresponding radar point cloud is directly deleted; otherwise, the point is a non-velocity outlier and proceeds to step (5.4);

[0158] (5.4) Further determine whether the distance of the corresponding radar point cloud relative to the vehicle exceeds the trust distance threshold of visual perception, and directly delete the point cloud that does not exceed the threshold, otherwise, go to step (5.5);

[0159] (5.5) The corresponding point cloud is retained for subsequent clustering and tracking processing, and the complete data of all radar point clouds after point cloud filtering is obtained.

[0160] As a preferred embodiment of the present invention, the step (6) specifically comprises the following steps:

[0161] (6.1) Use the uniformly accelerated linear motion model to predict the state of the target in the previous frame;

[0162] (6.2) Using the density-based DBSCAN clustering algorithm to perform three-dimensional point cloud clustering on the radar point cloud filtered by the current frame;

[0163] (6.3) Use the Kalman filter to filter the target prediction state of the previous frame and the target state after clustering in the current frame to obtain stable target 3D position and velocity information;

[0164] (6.4) combining the filtered target speed and the target three-dimensional heading angle obtained by visual detection in step (2), updating the target heading angle information, specifically: if the absolute value of the filtered target speed is greater than a preset threshold, the heading angle obtained after filtering is used as the final heading angle of the target; if the absolute value of the filtered target speed is not greater than the preset threshold, the target three-dimensional heading angle obtained by visual detection is used as the final heading angle of the target;

[0165] (6.5) Combined with the target category obtained in step (2.3), the 3D position, speed, category and orientation angle information of the target can be fully obtained;

[0166] (6.6) Repeat the above steps to achieve the continuous clustering tracking process of the target.

[0167] As a preferred embodiment of the present invention, the step (6.1) is specifically as follows:

[0168] Assume that the motion state of the target in the previous frame is X = [x, y, v x , v y , a x , a y ] T , where x, y, v x , v y ,a x , a y They are the x-axis coordinate, y-axis coordinate, x-axis speed, y-axis speed, x-axis acceleration, and y-axis acceleration of the target in the previous frame.

[0169] Correspondingly, the predicted target motion state is X′=[x′, y′, v′ x , v′ y , a′ x , a′ y ] T , then X′=FX, F is the state transfer matrix corresponding to the uniformly accelerated linear motion model: Where Δt represents the time difference between two frames.

[0170] As a preferred embodiment of the present invention, the step (6.3) specifically comprises the following steps:

[0171] (6.3.1) Based on the nearest neighbor data association principle, the predicted target of the previous frame and the clustered target of the current frame are associated according to the location of the target;

[0172] (6.3.2) Calculating Kalman Gain Using Kalman Filter Among them, H is the observation matrix, P is the prediction error covariance matrix, and R is the observation noise covariance matrix;

[0173] (6.3.3) Calculate the final target output: in, is the final target state output, Z is the target prediction state of the previous frame, X is the target state after clustering in the current frame, and h function is the conversion function from target state to target prediction state.

[0174] In practical applications, the radar-visual fusion target perception method based on a hybrid strategy of the technical solution specifically includes:

[0175] (1) Hardware synchronization triggering of camera and 4D millimeter-wave radar data. Through the software control program, the synchronization triggering of the camera and 4D millimeter-wave radar sensor is set at a specific time interval to ensure that the time stamp difference of each frame of data is as small as possible when the two sensors collect data.

[0176] (2) Obtaining time-aligned camera image data and radar point cloud data. Since the above hardware synchronization triggering step has been performed, the timestamp difference of the two sensor data obtained after the same trigger is as small as possible and can be regarded as a time-aligned state.

[0177] (3) Obtain the two-dimensional Box result of target detection in the image based on the YOLO network model. The YOLO series algorithm is one of the mainstream algorithms in the field of deep learning image detection. As a representative of the one-stage algorithm, it greatly improves the speed of target detection and can meet the balance between accuracy and real-time performance of target detection tasks. In order to pursue a balance between performance and speed and facilitate mass production, this solution specifically performs two-dimensional image detection based on the YOLOv6-S network model. In this process, this method is executed according to the basic process of the YOLO algorithm, which mainly includes image data pre-processing, network model reasoning and post-processing.

[0178] The pre-processing process is mainly responsible for removing distortion from the image data using the camera intrinsic matrix and distortion coefficients, and then converting the storage method into a format that is suitable for the network model input. Specifically, first use the OpenCV camera calibration tool to complete the camera calibration process to obtain the camera intrinsic matrix And distortion coefficient D = [k1, k2, p1, p2, k3], for any pixel point P in the image uv =[u,v,1] T The corresponding normalized camera coordinate system coordinate P = K can be obtained using the intrinsic parameter matrix -1 P uv , whose coordinates are [x, y, 1] T , the distance between point P and the origin of the coordinate system is r, according to the following dedistortion formula:

[0179]

[0180] The normalized camera coordinates P after dedistortion can be obtained distorted =[x distorted ,y distorted , 1] T , and then get the pixel position P after dedistortion uv_distorted =KP distorted , thus completing the image dedistortion process. Then, the dedistorted camera RGB image is resized to image data with a width of 640 pixels and a height of 640 pixels. Finally, the original data storage format of the RGB three-channel data with dimensions of (640, 640, 3) is converted into a one-dimensional array data with dimensions of (640×640×3, 1). This process is the so-called "encoding" process, which can be input to the network model for inference operations.

[0181] The network model inference process can send the pre-processed data into the network model for inference to obtain the target detection data results. In the YOLOv6-S network model, we mainly use the backbone network (Backbone), the neck network (Neck), and the prediction head network (Head). Among them, the backbone network is mainly composed of the convolution (Conv) structure, the efficient parameterized Rep-Block structure in the training stage, etc., which is mainly used to realize image feature extraction and representation in the model; the neck network is constructed based on the PAN topology by the Rep-Block structure, the Conv structure, and the upsampling (Upsample) structure to realize multi-scale feature integration; the prediction network is mainly constructed by the Conv structure, which is used to decouple the detection box information we need from the previous inference results. In the regression process of the detection box, we use the SIoU loss function commonly used in the industry to optimize the position and size of the bounding box predicted by the model relative to the real annotation box.

[0182] The post-processing process is responsible for format conversion of the model output data and outputting the two-dimensional visual detection box results. First, for the one-dimensional floating-point array obtained by the model inference of a frame of image, the two-dimensional detection box information of all targets in the current frame is parsed according to the format predicted by the model, mainly including the pixel coordinates of the upper left corner of the box, the pixel coordinates of the lower right corner of the box, the detection confidence, etc. This process is the so-called "decoding" process. In addition, since the YOLOv6-S network model inference will generate multiple prediction boxes for one target, NMS (non-maximum suppression) is used in the prediction stage to only retain the prediction box with the highest confidence as the two-dimensional detection box result of the target. The final detection box information is {u tl , v tl ,u dr , v dr , score}, where (u tl , v tl ) represents the pixel width and height coordinates of the upper left corner of the detection box, (u tl , v tl ) represents the pixel width and height coordinates of the lower right corner of the detection box, and score represents the confidence of the detection box.

[0183] (4) Use the external parameter calibration results to map the radar point cloud to the two-dimensional image data. This process mainly involves two parts: one is how to obtain the external parameters that map the three-dimensional coordinate system of the radar point cloud to the two-dimensional image pixel coordinate system, and the other is how to use the external parameters for coordinate mapping.

[0184] The acquisition of external parameters is explained by taking the sensor pre-calibration scenario in a vehicle mass production environment as an example. The final external parameters are indirectly obtained by converting the camera and millimeter-wave radar into the vehicle coordinate system respectively. The vehicle coordinate system is one of the commonly used rectangular coordinate systems in the field of intelligent driving, with the center of the vehicle's rear axle as the coordinate origin, the vehicle's forward direction as the x-axis, the left side as the y-axis, and the top as the z-axis.

[0185] Get the ego-camera extrinsic matrix H c , which is used to transform the ego vehicle coordinate system to the camera pixel coordinate system. When calibrating the vehicle sensor, the vehicle will be fixed at a specified position in a dedicated calibration room. At this time, there are several checkerboard calibration plates within the camera field of view. The coordinates of the corner points in these calibration plates relative to the ego vehicle coordinate system are P = (x, y, z, 1) T are all known, and the pixel coordinates P corresponding to the corner point at this time uv =(u, v, 1) T It can also be directly detected by the camera calibration tool, where u represents the width dimension coordinate of the image pixel and v represents the height dimension coordinate. c Features available: Where Zc is the z-axis coordinate in the camera coordinate system, Hc is the 3×4 dimensional external parameter matrix, which is composed of the rotation matrix R (3×3 matrix) and the translation vector t=[t x t y t z ] T It consists of two parts, namely H c =[R c t c ]. From this, the nonlinear least squares problem can be constructed: Solving this problem can obtain the ego-camera extrinsic matrix H c .

[0186] Get the radar-vehicle external parameter matrix H r , which is used to transform the radar coordinate system to the vehicle coordinate system. Also in the calibration room, there are several small corner reflectors with high millimeter wave reflectivity, and the coordinates of their center points relative to the vehicle coordinate system are P = (x, y, z) T The coordinates P of the centers of these corner reflectors in the radar coordinate system can be obtained using the millimeter wave radar calibration tool. r =(x r ,y r , z r ) T The relationship between the two is: P = R r P r +t r , where R r is the rotation matrix from the radar coordinate system to the vehicle coordinate system (3×3 matrix), tr =[t x t y t z ] T Then is the translation vector, from which the nonlinear least squares problem can be constructed: Solving this problem can obtain the radar-vehicle extrinsic matrix H r =[R r t r ].

[0187] Get the radar-camera extrinsic matrix H, which is used to transform the radar coordinate system to the camera pixel coordinate system. It can be obtained indirectly through the above two matrix calculations: H=[Rt]. It is worth noting that the two-dimensional images mentioned above are all images with distortion removed, and are in the same coordinate system as the two-dimensional detection box of the target detection. Therefore, the relevant conversion process does not need to consider the camera distortion removal operation.

[0188] In the coordinate mapping process, for any radar point cloud 3D coordinate point Pr = (x, y, z, 1) T , the corresponding camera coordinate system coordinates P can be obtained through the above radar-camera external parameter matrix H (3×4 matrix) c =(x c ,y c , z c ) T =H·P r , after normalization operation, we get P′ c =(x c / z c ,y c / z c , 1) T , and then use the camera intrinsic parameter matrix K to get the pixel coordinates P uv =(u, v, 1) T =K·P′ c , thus obtaining the complete data of the 4D millimeter wave point cloud {x, y, z, speed, u, v}, which includes the three-dimensional position coordinates (x, y, z), radial speed speed and two-dimensional pixel coordinates (u, v) in sequence.

[0189] (5) Based on the principle of millimeter wave as the main and vision as the auxiliary, the 4D radar point cloud is filtered in the two-dimensional image space according to the target two-dimensional Box result and other information in the image, so as to eliminate the possible false targets in the point cloud, enhance the robustness of radar target detection, and finally obtain the two-dimensional and three-dimensional coordinate results of the radar at the same time. The specific process includes the following (refer to Figure 2 ):

[0190] ①First, according to the pixel coordinates (u, v) in the 4D millimeter-wave radar point cloud data, determine whether it is within the visual two-dimensional detection frame {u tl , v tl ,u dr , v dr} within;

[0191] ② For radar point clouds that are not in the detection frame, further determine whether their distance to the vehicle exceeds the trust distance threshold of visual perception. The threshold is set according to the actual performance of the visual perception result. If the threshold is exceeded, the visual perception result is considered unreliable, and the radar result will be more trusted. Therefore, point clouds that exceed the threshold are retained, and those that do not exceed the threshold are deleted;

[0192] ③ For the radar point cloud existing in the detection frame, based on Z-Scor e Method: To detect speed outliers: First, for all radar point cloud speed values ​​(unit: m / s) in each detection frame, calculate the speed mean μ speed and the speed standard deviation σ speed , calculate the speed Z-Score value of each point cloud: If the Z value is greater than the empirical threshold, the point is considered to be a speed outlier. Otherwise, if the Z value is not greater than the empirical threshold, the point is considered to be a non-speed outlier.

[0193] ④ For speed outliers, the radar point cloud is deleted; for non-speed outliers, the point cloud is retained for subsequent cluster tracking process;

[0194] ⑤ After executing the above processing flow for all millimeter-wave radar point cloud data, the complete data of all radar point clouds after filtering (x, y, z, speed, u, v}) can be obtained, which contains the three-dimensional position coordinates (x, y, z), speed speed and two-dimensional pixel coordinates (u, v) in sequence.

[0195] (6) Clustering and tracking filtering are performed on the filtered 4D radar point cloud in three-dimensional space to finally obtain the target perception result after data fusion. The specific process includes the following:

[0196] ①Use the uniformly accelerated linear motion model to predict the state of the target in the previous frame. Assume that the motion state of the target in the previous frame is X = [x, y, v x , v y , a x , a y ] T , where the vector is x-axis coordinate, y-axis coordinate, x-axis velocity, y-axis velocity, x-axis acceleration, and y-axis acceleration. Correspondingly, the predicted target motion state is X′=[x′, y′, v′ x , v′y , a′ x , a′ y ] T , then X′=FX, F is the state transfer matrix corresponding to the uniformly accelerated linear motion model: Where Δt represents the time difference between two frames.

[0197] ②Since 4D millimeter-wave radar has the characteristic of dense point cloud, the density-based DBSCAN clustering algorithm is used to perform three-dimensional point cloud clustering on the radar point cloud after filtering the current frame to obtain the clustered current frame target position information. The following is the DBSCAN clustering algorithm process used in this case:

[0198] Initialization parameters: Set the neighborhood radius ε (epsilon) and the minimum number of neighborhood samples MinPts (minimum density threshold) as input parameters of the algorithm.

[0199] Select an unvisited data point: Select a data point from the dataset that is not marked as a core point, boundary point, or noise point as the current point.

[0200] Find neighborhood points: Calculate the distance from the current point to all other data points in the data set, and find all data points whose distance from the current point is within the range of ε to form a neighborhood.

[0201] Determine the core point: If the number of data points in the neighborhood is greater than or equal to MinPts, the current point is marked as a core point.

[0202] Expand clustering: If the current point is marked as a core point, add it to the current cluster and recursively process its neighboring points. For each point in the neighborhood, if it is unvisited, mark it as a boundary point and add it to the current cluster; if it is a core point, add the unvisited points in its neighborhood to the current cluster.

[0203] Termination condition: When all data points have been visited, the algorithm terminates. At this point, all core points, boundary points, and noise points are marked to form a set of clusters.

[0204] Post-processing: Mark the data points that are not assigned to any cluster as noise points, and calculate the center point position of a cluster of successfully clustered point clouds as the current frame position information of the target;

[0205] ③ Use Kalman filter (KF) to predict the state of the target in the previous frame Z = [xyv x v ya x a y ] T And the target state after clustering the current frame X = [xyv x v y ] T Filtering is performed to obtain stable target state information. First, based on the nearest neighbor data association principle, the predicted target of the previous frame and the clustered target of the current frame are associated according to the position of the target; then, the Kalman gain is calculated using the KF filter. Among them, H is the observation matrix, P is the prediction error covariance matrix, and R is the observation noise covariance matrix; finally, the final target output after filtering is calculated: in, is the final target state output, and the h function is the conversion function from the target state to the target predicted state.

[0206] By continuously executing the above three steps, the continuous clustering tracking process of the target can be realized, and finally a target perception and positioning result with high accuracy, high robustness and high stability can be obtained after radar and vision fusion.

[0207] In a specific embodiment of the present invention, the radar-visual fusion target perception method based on the hybrid strategy mainly includes the following steps:

[0208] (1) Hardware synchronization triggering of camera and 4D millimeter wave radar data: Through the software control program, the synchronization triggering of the camera and the 4D millimeter wave radar sensor is set at a specific time interval. In this embodiment, the time interval is 100ms.

[0209] (2) Obtain time-aligned camera image data and radar point cloud data.

[0210] (3) Obtaining the two-dimensional Box result of target detection in the image based on the YOLO network model. In this embodiment, the YOLOv6 network model is used. In this process, the method is executed according to the basic process of the YOLO algorithm, mainly including the pre-processing of image data, network model reasoning and post-processing. The pre-processing process uses the camera internal parameters to remove the distortion of the image data and then converts the storage mode into a format suitable for the network model input. The network model reasoning process can obtain the target detection data result by inputting the image data through the network model reasoning. The post-processing process can perform appropriate format conversion on the model output data and filter the targets with low confidence, and finally obtain a more accurate target two-dimensional Box result, which is located in the two-dimensional image coordinate system.

[0211] (4) Use the external parameter calibration results to map the radar point cloud to the two-dimensional image data. The two-dimensional images here are all images that have been distorted by using the camera intrinsic parameters, and are in the same coordinate system as the target detection two-dimensional box results. Therefore, this conversion process does not require additional camera distortion. In the mapping process, the external parameter calibration result usually represents the point P in the camera image coordinate system. 2d =(u, v, 1) T Mapped to point P in the three-dimensional coordinate system of the radar point cloud 3d =(x, y, z, 1) T The transformation matrix H, H represents a 3×4 transformation matrix, and the transformation formula is: 3d =H·P 2d , so the formula in this conversion process is: P 2d =H -1 ·P 3d In this embodiment, the visualization effect of the radar point cloud mapped to the image and the two-dimensional Box result output by YOLO is as follows: Figure 4 shown.

[0212] (5) Filter the 4D radar point cloud in the two-dimensional image space according to the target two-dimensional Box results and other information in the image, so as to remove the possible false targets in the point cloud, enhance the robustness of radar target detection, and finally obtain the two-dimensional and three-dimensional coordinate results of the radar at the same time. The specific process includes the following:

[0213] ① Delete the radar point cloud outside the target two-dimensional Box;

[0214] ② According to the speed value data of the radar point cloud in the Box, the speed outliers are deleted, that is, the outliers whose speed is inconsistent with most point clouds in the two-dimensional Box are filtered out;

[0215] ③ Obtain the two-dimensional and three-dimensional coordinate results of the filtered radar point cloud at the same time.

[0216] (6) Clustering and tracking filtering are performed on the filtered 4D radar point cloud in three-dimensional space to finally obtain the target perception result after data fusion. The specific process includes the following:

[0217] ① For the radar target tracking and filtering result of the previous frame, the motion state of the target in the previous frame (in this embodiment, specifically including three-dimensional position, velocity, acceleration, etc.) and the timestamp difference between the previous frame and the current frame are used to predict the state of the target in the previous frame using a uniform linear motion model, which is used as the predicted state input of the previous frame in the subsequent tracking and filtering process;

[0218] ② Performing 3D point cloud clustering for the radar point cloud after filtering of the current frame. In this embodiment, the DBSCAN clustering algorithm is used to implement density-based clustering of the 3D radar point cloud, and the clustered vehicle target position and posture information (in this embodiment, specifically including 3D position, speed, etc.) is obtained, which is used as the current observation state input of the subsequent tracking filtering process;

[0219] ③ Use the Kalman filter to take the above predicted state and current observed state as input, and output the current optimal motion state after filtering (specifically including three-dimensional position, motion speed, acceleration, etc.), to achieve the target continuous tracking process, and finally obtain the target perception and positioning results with high accuracy, high robustness and high redundancy.

[0220] In practical application, this technical solution has the following technical effects:

[0221] (1) The end-to-end 2D keypoint (12 keypoints) detection network without NMS (non-maximum suppression) post-processing in this technical solution removes the NMS operation in the current keypoint network post-processing, truly achieving end-to-end detection, reducing the time complexity of model post-processing, being more friendly to embedded devices, and facilitating mass production on the vehicle side;

[0222] (2) The network contains three detection heads: a classification head, a normal 2D box detection head, and a target key point head. The key point detection head can regress the coordinates and visibility of the 12 boundary points of the target, that is, 12×(x, y, score). There are two advantages of introducing key points: the outer boundary contour of the target is more accurate, which can limit and filter the radar point cloud more accurately than the original box; the road grounding point in the key point can be used to calculate the heading angle of the target in space. It solves the problem that the direction estimation deviation of the millimeter wave point cloud target tracking result is large at low speed, and the stationary target cannot determine the direction of the vehicle head.

[0223] (3) Ultimately, the target’s 3D position, speed, category, and more accurate orientation angle information can be output simultaneously, which makes the target’s state perception more accurate than traditional methods.

[0224] See also Figure 6 As shown, the system for realizing radar-visual fusion target perception processing based on a hybrid strategy, wherein the system comprises:

[0225] The sensor synchronization trigger acquisition module is used to trigger the camera and 4D millimeter wave radar sensor to start a frame of data acquisition at a specific time interval to obtain data from the two sensors at the same time;

[0226] An image key point detection module is connected to the sensor synchronization trigger acquisition module for processing image sensor data and obtaining the category and 12 key points of the target in the camera image data based on an end-to-end NMS-free 2D target key point detection network model;

[0227] The IPM module for solving the heading angle is connected to the image key point detection module, and uses the IPM method to obtain the homography matrix H of the camera plane corresponding to the road plane, and projects the ground point in the key point to the road bev perspective to obtain the heading angle of the target, and finally obtains the target category, 12 key points and 2D Box results by fusing the output of the image key point detection module;

[0228] A radar point cloud coordinate conversion module is connected to the image key point detection module and is used to convert the collected radar point cloud from a three-dimensional coordinate system to a two-dimensional image coordinate system according to the sensor external parameter calibration parameters, and simultaneously obtain the two-dimensional and three-dimensional coordinate information of the radar point cloud;

[0229] The point cloud filtering module is connected to the radar point cloud coordinate conversion module and the IPM heading angle solution module, and is used to filter the 4D radar point cloud in the two-dimensional image space according to the target 12 key points-2D Box result information in the image, so as to eliminate the false targets that may exist in the point cloud;

[0230] A point cloud clustering module is connected to the point cloud filtering module for performing three-dimensional point cloud clustering on the radar point cloud after filtering of the current frame to obtain the clustered vehicle target position and posture information;

[0231] The target tracking module is connected to the point cloud clustering module and is used to perform tracking and filtering processing on the target information that has finally been fused.

[0232] The radar-vision fusion target perception device based on a hybrid strategy, wherein the device comprises:

[0233] a processor configured to execute computer executable instructions;

[0234] A memory stores one or more computer executable instructions, and when the computer executable instructions are executed by the processor, the steps of the above-mentioned radar-visual fusion target perception method based on the hybrid strategy are implemented.

[0235] The radar-visual fusion target perception processor based on a hybrid strategy, wherein the processor is configured to execute computer-executable instructions, and when the computer-executable instructions are executed by the processor, the steps of the above-mentioned radar-visual fusion target perception method based on a hybrid strategy are implemented.

[0236] The computer-readable storage medium stores a computer program thereon, and the computer program can be executed by a processor to implement the steps of the above-mentioned radar-visual fusion target perception method based on a hybrid strategy.

[0237] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0238] It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device.

[0239] A person of ordinary skill in the art may understand that all or part of the steps of the method for implementing the above-mentioned embodiment may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0240] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0241] In the description of this specification, the description with reference to the terms "an embodiment", "some embodiments", "example", "specific example", or "embodiment" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0242] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

[0243] The hybrid strategy-based radar-vision fusion target perception method, system, device, processor and computer-readable storage medium thereof of the present invention are adopted, and a two-dimensional + three-dimensional hybrid strategy is innovatively adopted for radar-vision data fusion, and data fusion is performed in a two-dimensional image coordinate system and a three-dimensional space coordinate system at the same time, so as to improve the accuracy and precision of the target perception results; the advantages of 4D millimeter-wave radar, such as strong robustness, high point cloud density and provision of high-dimensional information, are fully utilized to provide high-accuracy, high-robustness and high-redundancy radar-vision fusion target perception results, and make up for the shortcomings of poor target positioning accuracy and error-prone target association matching in traditional 3D millimeter-wave radar radar-vision fusion perception methods; compared with the radar-vision fusion solution of lidar + camera, only the deep learning network model is used for the camera data, and the dependence on the computing power of the computing platform is relatively low, and it is easy to deploy on the edge computing platform, with lower system cost and stronger robustness, which is conducive to large-scale mass production and application in the fields of autonomous driving and vehicle-road collaboration.

[0244] In this specification, the present invention has been described with reference to specific embodiments thereof. However, it is apparent that various modifications and variations may be made without departing from the spirit and scope of the present invention. Therefore, the specification and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A radar-visual fusion target perception method based on a hybrid strategy, characterized in that: The method comprises the following steps: (1) Setting the camera and the 4D millimeter-wave radar sensor to be triggered synchronously at a specific time interval to obtain camera image data and 4D millimeter-wave radar point cloud data in a time-aligned state; (2) Obtain the target key point detection results in the camera image data based on the end-to-end NMS-free 2D target key point detection network model: target category, 12 key points and 2D Box results; (3) Using the pixels and spatial coordinates of the corner points of the open road to perform IPM to solve the homography matrix H, the target road grounding point in the key point detection result is converted to the road bev perspective using the homography matrix H to obtain the heading angle of the target in space; (4) Use the external parameter calibration results to map the 4D millimeter wave radar point cloud into two-dimensional image data; (5) filtering the 4D millimeter wave radar point cloud in the two-dimensional image space according to the target 12 boundary point result information in the two-dimensional image; (6) Perform data post-processing on the filtered 4D millimeter-wave radar point cloud in three-dimensional space to fully restore the target's 3D position, speed, category, and orientation angle information; The step (2) specifically comprises the following steps: (2.1) Image data preprocessing: Dedistorting the camera image data using the camera intrinsic parameter matrix and distortion coefficients, and converting the image size and image storage mode into a format suitable for input of the end-to-end NMS-free 2D target key point detection network model; (2.2) Network model reasoning: The data after data pre-processing is sent to the network model reasoning to obtain the target detection data results; (2.3) Image data post-processing: convert the format of the network model output data to output the two-dimensional visual key point detection results of the corresponding target; The step (2.1) specifically comprises the following steps: (2.1.1) First, use the OpenCV camera calibration tool to complete the camera calibration process and obtain the camera intrinsic parameter matrix And distortion coefficient D = [k1, k2, p1, p2, k3], where f x is the horizontal focal length, f y is the vertical focal length, s is the coordinate axis tilt factor, c x is the x coordinate of the optical center in the image coordinate system, c y is the y coordinate of the optical center in the image coordinate system, k1 to k3 are the first-order, second-order, and third-order radial distortion coefficients, respectively, and p1 and p2 are the tangential distortion coefficients; (2.1.2) For any pixel point P in the camera image uv1 =[u,v,1] T , using the internal parameter matrix K to obtain the corresponding normalized camera coordinate system coordinate P = K -1 P uv1 , whose coordinates are [x, y, 1] T , the distance relative to the origin of the camera coordinate system is r, and the corresponding camera image is dedistorted using the following formula: Get the normalized camera coordinates P after dedistortion distorted =[x distorted ,y distorted , 1] T , and then get the pixel position P after dedistortion uv_distorted =KP distorted , where T represents vector transpose; (2.1.3) Resize the dedistorted camera RGB image to 640×640 while maintaining its aspect ratio. Fill the short side blanks with 114 pixels, and place the original image in the center of the resized image. (2.1.4) Convert the original data storage format of the camera image from RGB three-channel data of (640, 640, 3) dimension to array data of (1, 3, 640, 640) dimension; (2.1.5) This completes the pre-processing of the camera image; The end-to-end NMS-free 2D target key point detection network model in step (2.2) specifically includes the following steps: (2.2.1) The backbone network model uses the CSPDarkne network, the neck network uses the PAFPN, and three scale feature maps (20×20, 40×40, 80×80) are used to meet the requirements of multi-scale target feature extraction; (2.2.2) The network sets up three detection head branches: classification branch, box detection branch and points detection branch. The three branches perform concat operations at each scale, and then concat the feature map results of each scale to output. In the classification branch, the output dimension is set to the number of categories; in the box detection branch, the output dimension is set to 64; in the key point branch, the key point output dimension is set to 36, so as to detect 12 key points (x, y, score) of the target; (2.2.3) During the training process, two positive and negative sample allocation strategies are set: topk10 and topk1. Topk10 is used to enrich the model's positive samples and make the model converge quickly; topk1 is used for subsequent model export, inference, and reduction of model output; (2.2.4) In the model inference and export stage, the feature map of the topk1 branch of the above training process is split according to the number of categories, 64, and 36, and is divided into category decoding branch, box decoding branch, and key point decoding branch in turn. The classification branch directly performs sigmoid decoding; the box branch first reduces the 64 dimensions to 4 dimensions through dfl regression, and uses anchor points to predict the upper left point (x, y) and lower right point (x, y) of the box box respectively; the point branch directly uses anchor points to predict 12 key points (x, y, score), and the score of each point is directly generated by sigmoid; (2.2.5) After the decoded results are concat, channel transformed, and split, the top 30 targets are selected by category using topk1, and the selected results are gathered with the boxes and points to finally output 30 target results; The step (2.3) specifically comprises the following steps: (2.3.1) Slice the obtained one-dimensional array data according to the model output dimension, that is, the number of categories + 4 + 36, to obtain 30 output targets; (2.3.2) Traverse each target result, obtain the target type according to the category corresponding to the maximum value in the category score, and compare the category score with the output threshold confidence. If it is greater than the threshold, it is a detected target. The final target result includes the target type, target box result and the target 12 key point results.

2. The target perception method based on hybrid strategy of radar and vision fusion according to claim 1 is characterized in that: The step (3) specifically comprises the following steps: (3.1) Select an empty road scene with clear lane lines, place the vehicle in the middle of the road, with the front direction of the vehicle parallel to the lane line direction, select the four corner points of a pair of parallel lane lines, obtain the pixel values ​​of the four corner points on the image and the (x, y) values ​​of the vehicle coordinate system in space to establish point pairs, and use the four matching point pairs to solve the IPM inverse transformation homography matrix H from the image perspective to the road bev space; (3.2) According to the homography matrix H, the target ground point in the image key point detection is converted to the road bev perspective, and the angle between the target head and tail vector and the vehicle's driving direction is solved to obtain the target heading angle.

3. The method for target perception based on hybrid strategy of radar and vision fusion according to claim 2 is characterized in that: The step (4) specifically comprises the following steps: (4.1) Obtain the ego-camera extrinsic matrix H c , this matrix is ​​used to transform the vehicle coordinate system to the camera pixel coordinate system; (4.2) Obtain radar-vehicle external parameter matrix H r , this matrix is ​​used to transform the radar coordinate system to the vehicle coordinate system; (4.3) Obtain the radar-camera extrinsic matrix H, which is used to transform the radar coordinate system to the camera pixel coordinate system; (4.4) In the coordinate mapping process, the corresponding camera coordinate system coordinates are obtained according to the radar-camera extrinsic matrix H, the camera coordinate system coordinates are normalized, and the pixel coordinates are obtained using the camera intrinsic parameter matrix K, so as to obtain the complete data of the 4D millimeter wave point cloud.

4. The target perception method based on hybrid strategy of radar and vision fusion according to claim 3 is characterized in that: The step (4.1) specifically comprises the following steps: (4.1.1) According to the internal parameter matrix K and external parameter matrix H c Available Among them, Z c is the z-axis coordinate in the camera coordinate system, H c is a 3×4 dimensional external parameter matrix; (4.1.2) Construct a nonlinear least squares problem: In this way, the vehicle-camera external parameter matrix H is obtained. c .

5. The method for target perception based on hybrid strategy of radar and visual fusion according to claim 4 is characterized in that: The step (4.2) specifically comprises the following steps: (4.2.1) According to the coordinates P2 = (x, y, z) of the center points of several small corner reflectors in the vehicle calibration room relative to the vehicle coordinate system T , use the millimeter wave radar calibration tool to obtain the coordinates P of the center of the corner reflector in the radar coordinate system r =(x r ,y r ,z r ) T , where P2 = R r P r +t r , R r is the rotation matrix from the radar coordinate system to the vehicle coordinate system, t r =[t x t y t z ] T is the translation vector; (4.2.2) Construct the linear least squares problem: In this way, the radar-vehicle external parameter matrix H is obtained. r .

6. The target perception method based on hybrid strategy of radar and vision fusion according to claim 4 is characterized in that: The step (4.3) is specifically: According to the radar-vehicle external parameter matrix H r and the ego-camera extrinsic matrix H c , obtain the radar-camera extrinsic matrix H as follows: Among them, R1 is the rotation matrix in the radar-camera extrinsic matrix, t is the translation vector in the radar-camera extrinsic matrix, and R r is the rotation matrix in the radar-ego vehicle extrinsic matrix, t r is the translation vector in the radar-ego vehicle extrinsic matrix, R c is the rotation matrix in the ego-camera extrinsic matrix, t c is the translation vector in the ego-camera extrinsic matrix.

7. The target perception method based on hybrid strategy of radar and vision fusion according to claim 1 is characterized in that: The step (5) specifically comprises the following steps: (5.1) Using the 12 key points of the target obtained by visual two-dimensional detection - 2D Box information, construct the geometric outer bounding box of the target in the two-dimensional image; (5.2) According to the pixel coordinates (u, v) in the 4D millimeter-wave radar point cloud data, determine whether it is located within the geometric outer enclosure of the target in the two-dimensional image. If so, proceed to step (5.2), otherwise proceed to step (5.3); (5.3) Based on the Z-Score method, the speed outliers in the same detection frame are detected, specifically: For all the radar point cloud speed values ​​in each detection frame, the mean speed μ is calculated speed and the speed standard deviation σ speed ; Calculate the speed Z-Score value of each point cloud: If the Z value is greater than the empirical threshold, the point is considered to be a velocity outlier and the corresponding radar point cloud is directly deleted; otherwise, the point is a non-velocity outlier and proceeds to step (5.4); (5.4) Further determine whether the distance of the corresponding radar point cloud relative to the vehicle exceeds the trust distance threshold of visual perception, and directly delete the point cloud that does not exceed the threshold, otherwise, go to step (5.5); (5.5) The corresponding point cloud is retained for subsequent clustering and tracking processing, and the complete data of all radar point clouds after point cloud filtering is obtained.

8. The method for target perception based on hybrid strategy of radar and vision fusion according to claim 7 is characterized in that: The step (6) specifically comprises the following steps: (6.1) Use the uniformly accelerated linear motion model to predict the state of the target in the previous frame; (6.2) Using the density-based DBSCAN clustering algorithm to perform three-dimensional point cloud clustering on the radar point cloud filtered by the current frame; (6.3) Use the Kalman filter to filter the target prediction state of the previous frame and the target state after clustering in the current frame to obtain stable target 3D position and velocity information; (6.4) combining the filtered target speed and the target three-dimensional heading angle obtained by visual detection in step (2), updating the target heading angle information, specifically: if the absolute value of the filtered target speed is greater than a preset threshold, the heading angle obtained after filtering is used as the final heading angle of the target; if the absolute value of the filtered target speed is not greater than the preset threshold, the target three-dimensional heading angle obtained by visual detection is used as the final heading angle of the target; (6.5) Combined with the target category obtained in step (2.3), the 3D position, speed, category and orientation angle information of the target can be fully obtained; (6.6) Repeat the above steps to achieve the continuous clustering tracking process of the target.

9. The method for target perception based on hybrid strategy of radar and vision fusion according to claim 8 is characterized in that: The step (6.1) is specifically: Assume that the motion state of the target in the previous frame is X = [x, y, v x ,v y ,a x ,a y ] T , where x,y,v x ,v y ,a x ,a y They are the x-axis coordinate, y-axis coordinate, x-axis speed, y-axis speed, x-axis acceleration, and y-axis acceleration of the target in the previous frame. Correspondingly, the predicted target motion state is X′=[x′,y′,v x ′,v y ′,a x ′,a y ′] T , then X′=FX, F is the state transfer matrix corresponding to the uniformly accelerated linear motion model: Where Δt represents the time difference between two frames; The step (6.3) specifically comprises the following steps: (6.3.1) Based on the nearest neighbor data association principle, the predicted target of the previous frame and the clustered target of the current frame are associated according to the location of the target; (6.3.2) Calculating Kalman Gain Using Kalman Filter Among them, H is the observation matrix, P is the prediction error covariance matrix, and R2 is the observation noise covariance matrix; (6.3.3) Calculate the final target output: in, is the final target state output, Z is the target prediction state of the previous frame, X is the target state after clustering in the current frame, and h function is the conversion function from target state to target prediction state.

10. A system for realizing radar-visual fusion target perception processing based on a hybrid strategy using the method of claim 1, characterized in that: The system comprises: The sensor synchronization trigger acquisition module is used to trigger the camera and 4D millimeter wave radar sensor to start a frame of data acquisition at a specific time interval to obtain data from the two sensors at the same time; An image key point detection module is connected to the sensor synchronization trigger acquisition module for processing image sensor data and outputting target detection categories, 2D Boxes and 12 key point results using an end-to-end NMS-free 2D target key point detection network model; The IPM module for solving the heading angle is connected to the image key point detection module, and converts the ground point in the image key point to the road bev perspective through the pre-selected open road pixel value and the spatial coordinate point through the homography matrix H solved by the IPM, and solves the target heading angle; A radar point cloud coordinate conversion module is connected to the image key point detection module and is used to convert the collected radar point cloud from a three-dimensional coordinate system to a two-dimensional image coordinate system according to the sensor external parameter calibration parameters, and simultaneously obtain the two-dimensional and three-dimensional coordinate information of the radar point cloud; A point cloud filtering module is connected to the radar point cloud coordinate conversion module and the IPM heading angle solution module, and is used to filter the 4D radar point cloud in the two-dimensional image space according to the geometric outer enclosure result information constructed by the 12 key points of the target in the image, so as to eliminate the false targets that may exist in the point cloud; The point cloud post-processing module is communicated with the point cloud filtering module and is used to perform data post-processing on the filtered 4D millimeter wave radar point cloud in three-dimensional space to completely restore the target's 3D position, speed, category and orientation angle information.

11. A radar-visual fusion target perception device based on a hybrid strategy, characterized in that: The device comprises: a processor configured to execute computer-executable instructions; A memory storing one or more computer executable instructions, wherein when the computer executable instructions are executed by the processor, the steps of the radar-visual fusion target perception method based on a hybrid strategy described in any one of claims 1 to 9 are implemented.

12. A radar-visual fusion target perception processor based on a hybrid strategy, characterized in that: The processor is configured to execute computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the radar-visual fusion target perception method based on a hybrid strategy described in any one of claims 1 to 9 are implemented.

13. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the radar-visual fusion target perception method based on a hybrid strategy as described in any one of claims 1 to 9.

Images