An observation feature extraction method based on the fusion of vision and millimeter-wave radar

Abstract

The technical solution of the present invention is to disclose an observation feature extraction method based on the fusion of vision and millimeter-wave radar. The present invention proposes an observation feature extraction method based on the fusion of visual targets and millimeter-wave radar point clouds. On the basis of the traditional target tracking system, an observation feature extraction module is added to the successfully associated observation target to provide accurate observation feature quantities for the subsequent track filtering update module. On the basis of multiple sensor target fusion schemes, the present invention proposes a new observation feature extraction method based on a fusion scheme, which uses 4D millimeter-wave radar point clouds and the predicted state of the tracked target to perform multiple mutual checks and corrections on multiple features of the observation target from multiple sensors. The method disclosed in the present invention can not only give full play to the advantages of different sensors, but also improve the accuracy and stability of the final perceived output target track by improving the accuracy of the observation features.

Description

Technical Field

[0001] The invention relates to a method for extracting observation features by fusing vision and millimeter-wave radar. Background Art

[0002] In order to make up for the disadvantage of insufficient information provided by a single sensor, the mainstream solution for environmental perception in the current intelligent driving industry is to use a variety of sensor fusion, among which the fusion of vision and millimeter-wave radar has attracted widespread attention due to its complementary advantages and economic characteristics. Both pure visual perception technology and pure radar perception technology are relatively mature, and the current radar-visual fusion method is mostly a post-fusion method, which directly fuses the visual perception target with the radar perception target at the target level, and there is limited room for performance improvement in the fusion part. Summary of the invention

[0003] The technical problem to be solved by the present invention is that most of the current radar-visual fusion methods are post-fusion methods, which directly fuse the visual perception target and the radar perception target at the target level, and there is limited room for performance improvement in the fusion part.

[0004] In order to solve the above technical problems, the technical solution of the present invention is to disclose an observation feature extraction method based on the fusion of vision and millimeter-wave radar, which is characterized in that it includes the following steps: Step 1, after obtaining the visual detection target after fusion with the millimeter-wave radar point cloud and the existing target track, the visual detection target is associated with the existing target track in the previous frame period, and the existing target track successfully associated with the visual detection target is defined as a tracking target, and the visual detection target after fusion with the millimeter-wave radar point cloud is defined as a detection target; Step 2, using the tracking target to perform alignment offset correction on the bbox frame position of the detection target; Step 3, using the bbox frame size and center position of the detection target target after correction and alignment directly as the observation feature quantity, and verifying after obtaining the observation quantity to determine whether it is an abnormal value: calculating the square of the Mahalanobis distance d k 2 : , where Z k is the observed value at the kth moment, is the observed value at the kth moment predicted by the k-1th moment, S k Z k The covariance of k 2 < 2 , then the current observation is considered to be credible, otherwise, it is an outlier and goes to step 4, where χ 2 is the critical chi-square value χ obtained according to the confidence setting value of the observation 2 ; Step 4, use the speed of the tracking target [v x ,v y] Remove abnormal points in the radar point cloud cluster associated with the detected target: Calculate , where θ is the azimuth of the point cloud; compare the radial velocity v of the point cloud r With v r θ If the difference exceeds the set threshold, the associated point cloud is removed. After the screening is completed, the remaining point cloud clusters are the final associated point cloud clusters, and the velocity component in the observation is updated to the radial velocity of the final associated point cloud cluster. The verification of step 3 is performed again. If the verification passes, the observation is considered to be credible, otherwise the observation is lost.

[0005] Preferably, the step 2 comprises the following steps: step 2.1, calculating the length difference ∆l and width difference ∆w between the detection target target box and the tracking target bbox box; step 2.2, calculating the unit vector of the long side direction of the tracking target bbox box and the broadside direction unit vector : , where (x 0 ,y 0 ) is the point with the closest Euclidean distance to the origin of the vehicle coordinates among all the vertices of the bbox box of the detection target, (x 1 ,y 1 ) is the nearest point (x) among all the vertices of the bbox box of the detection target 0 ,y 0 ) connected to the long side point, (x 2 ,y 2 ) is the nearest point (x) among all the vertices of the bbox box of the detection target 0 ,y 0 ) connected to the wide side points; Step 2.3, scale the bbox box of the detected target in the direction of the bbox box of the tracked target to compensate for the error caused by the size difference caused by the incomplete detection area, as shown in the following formula: , where (x' b ,y' b ) is the center position of the bbox frame of the detected target after correction and alignment, (x b ,y b ) is the center position of the bbox box of the detected target before correction and alignment.

[0006] Preferably, in step 3: if the tracking filter system is a linear Gaussian system, then: , where H k is the observation matrix, R k is the measurement noise covariance matrix, P k|k-1 is the prior estimated covariance matrix at time k, is the prior state estimate at the kth moment predicted at the k-1th moment; if the tracking filter system is a nonlinear Gaussian system, then: , where J H is the Jacobian matrix.

[0007] The present invention proposes an observation feature extraction method based on the fusion of visual targets and millimeter-wave radar point clouds. On the basis of the traditional target tracking system, an observation feature extraction module is added to the successfully associated observation target to provide accurate observation feature quantities for the subsequent track filtering update module.

[0008] Compared with the prior art solutions, the present invention has the following beneficial effects:

[0009] Based on the multiple sensor target fusion scheme, the present invention proposes a new observation feature extraction method based on the fusion scheme, which uses the 4D millimeter wave radar point cloud and the predicted state of the tracked target to perform multiple mutual verification and correction on multiple features of the observed target from multiple sensors. The method disclosed in the present invention can not only give full play to the advantages of different sensors, but also improve the accuracy and stability of the final perception output target track by improving the accuracy of the observation features. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a flow chart of an embodiment of the present invention;

[0011] Figure 2 The bbox of the detected target before correction is shown;

[0012] Figure 3 The corrected bbox of the detected object is shown. DETAILED DESCRIPTION

[0013] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0014] like Figure 1 As shown, an observation feature extraction method based on the fusion of visual target and millimeter wave radar point cloud disclosed in an embodiment of the present invention mainly includes the following steps:

[0015] Step 1: The upstream module obtains the visual detection target (hereinafter referred to as "detection target target" after fusion with the millimeter-wave radar point cloud, which has conventional visual target features: position, speed, bbox, type, etc., and also has an associated radar point cloud cluster) and the existing target track. The detection target is associated with the existing target track in the previous frame period. The association methods include but are not limited to: nearest neighbor association (NN), global nearest neighbor association (GNN), Hungarian matching, probabilistic data association (PDA), etc. Since it is not the main focus of the invention, it will not be elaborated on. The existing target track successfully associated with the detection target is defined as the "tracking target track".

[0016] like Figure 2 As shown in the figure, in this embodiment, the four vertices of the bbox of the tracking target track are (1,1), (1,2), (3,1), (3,2), the center point A coordinates (2,1.5), length length = 2m, width width = 1m, speed (v x ,v y ) is (6,0). The four vertices of the bbox of the detection target are (1,1), (1,1.5), (2,1), (2,1.5), the coordinates of the center point B are (1.5,1.25), the length is 1m, the width is 0.5m, and the three point cloud positions and velocities associated with it are (x, y, v r ) are (1.5,1.125,4.8), (1.2,0.9,4.8), and (2,1,0) respectively.

[0017] Step 2: Correct the position of the detection target:

[0018] Due to the problems of occlusion, detection angle, and visual detection error in the actual detection process, only the surface of the target closest to the vehicle can be detected, that is, the bbox frame of the actual output detection target target is often only a part of the real target frame. If the bbox frame size and center position of the current detection target target are directly used as the observation feature quantity, a large error will occur. Therefore, the existing tracking target track associated with the detection target target is used to align the bbox frame position offset correction, which specifically includes the following steps:

[0019] Step 2.1, calculate the length difference ∆l and width difference ∆w between the bbox box of the detection target target and the bbox box of the tracking target track:

[0020]

[0021] Step 2.2: Calculate the unit vector of the long side of the bbox box of the tracked target track and the broadside direction unit vector :

[0022]

[0023] Where: (x 0 ,y 0 ) is the point with the closest Euclidean distance to the origin of the vehicle coordinate system among all the vertices of the bbox of the detection target. In this embodiment, it is (1,1); (x 1 ,y 1 ) is the nearest point (x) among all the vertices of the bbox box of the detection target 0 ,y 0 ) is connected to the long side point, in this embodiment, it is (3,1); (x 2 ,y 2 ) is the nearest point (x) among all the vertices of the bbox box of the detection target 0 ,y 0 ) is connected to the broad side point, in this embodiment, it is (1,2). In this embodiment, the calculated long side direction unit vector =(1,0), the calculated unit vector in the broadside direction =(0,1).

[0024] Step 2.3: Scale the bbox of the detection target target in the direction of the bbox of the tracking target track to compensate for the error caused by the size difference caused by the incomplete detection area, as shown in the following formula:

[0025]

[0026] In the formula, (x' b ,y' b ) is the center position of the bbox box of the detection target after correction and alignment, (x b ,y b ) is the center position of the bbox box of the detection target target before correction alignment, and the graphical description is shown in Figure 2 as well as Figure 3 , that is, the center position of the bbox box of the detection target target after correction and alignment is (2,1.5).

[0027] Step 3: Use the bbox size and center position of the detected target after correction and alignment directly as the observed feature quantity, and verify it after obtaining the observed quantity to determine whether it is an outlier:

[0028] The observed quantity of the tracking filter system at the kth moment is set as the matrix Z k, the prior state estimate at the kth moment predicted by the k-1th moment is set as the matrix , where filtering refers to Bayesian filtering. Assuming that the system state estimation motion equation is linear, that is, the system is a linear Gaussian system, the linear Kalman filter (KF) is used, where the prior state estimation obeys Gaussian distribution, then:

[0029]

[0030] Where: H k is the observation matrix, which maps the state vector to the vector space where the measurement value is located; R k is the measurement noise covariance matrix; P k|k-1 is the prior estimated covariance matrix at time k; the observation Z k obey Gaussian distribution of is the observed value at the kth moment predicted by the k-1th moment.

[0031] Mahalanobis distance squared d k 2 is the observed quantity Z k With its covariance S k The square of the distance is used to determine whether the observed value deviates from the distribution and is an outlier:

[0032]

[0033] d k 2 ~x 2 (n) obeys the chi-square distribution with n degrees of freedom, where n is the number of observations in the observation matrix. We set p_CDF according to our confidence in the observations and query the chi-square distribution table to find the corresponding critical chi-square value χ 2 , when the calculated d k 2 < 2 , then the current observation is considered to be credible, otherwise it is an outlier.

[0034] If the system is a nonlinear Gaussian system, the extended Kalman filter (EKF) used by the filter needs to be processed as a local linear Gaussian system near the working point. The corresponding measurement matrix H k is the Jacobian matrix J H , which represents the derivative of the observation function with respect to the state variable.

[0035] In this embodiment, the initial observation matrix is ​​assumed to be for , The predicted state quantity is , Hk for , R k is a zero matrix. If we substitute it into the formula to calculate d k 2 = 20, and in the chi-square distribution, p_CDF=0.1 and 3 degrees of freedom are selected, and the corresponding critical chi-square value χ 2 =6.25, indicating that this set of observations is unreliable, and proceed to step 4.

[0036] Step 4: If the verification in step 3 fails, perform observation speed correction. Use the speed of the tracking target track [v x ,v y ] Remove the abnormal points in the radar point cloud cluster associated with the detection target target, and the radial velocity of the point cloud is v r , the azimuth angle is θ, then:

[0037]

[0038] Compare r With v r θ If the difference exceeds the set threshold, the associated point cloud is removed.

[0039] After the screening is completed, the remaining point cloud clusters are the final associated point cloud clusters, and the velocity component in the observation is updated to the radial velocity of the final associated point cloud cluster. The verification of step 3 is performed again. If the verification passes, the observation is considered to be credible, otherwise the observation is lost.

[0040] In this embodiment, the judgment threshold t=2m / s is set, and in the point cloud cluster (2,1,0), there is:

[0041]

[0042] ∆ = v r θ −v r = 5.36−0 = 5.36

[0043] The absolute value of the difference ∆>t, so the point is removed. The same method can be used to obtain other point clouds (1.5, 1.125, 4.8) and (1.2, 0.9, 4.8) which are not abnormal points. In order to detect the target target, the point cloud cluster is finally associated. Here, in order to simplify the calculation, the two point clouds v are directly used. r The mean of is taken as the observed velocity component, and the observation value matrix is ​​updated as , resubstitute into the formula to calculate d 2 < 2 , it is believed that this set of observations is credible and is the observation feature quantity finally extracted.

Claims

1. An observation feature extraction method based on the fusion of vision and millimeter wave radar, characterized in that: The following steps are involved: Step 1: After obtaining the visual detection target fused with the millimeter-wave radar point cloud and the existing target track, the visual detection target is associated with the existing target track in the previous frame period, and the existing target track successfully associated with the visual detection target is defined as the tracking target, and the visual detection target fused with the millimeter-wave radar point cloud is defined as the detection target; Step 2: Use the tracking target to align the offset of the bbox position of the detection target; Step 3: Use the bbox size and center position of the detection target target after correction and alignment directly as the observation feature quantity, and verify it after obtaining the observation quantity to determine whether it is an outlier: calculate the square of the Mahalanobis distance d k 2 : , where Z k is the observed value at the kth moment, is the observed value at the kth moment predicted by the k-1th moment, S k Z k The covariance of k 2 < 2 , then the current observation is considered to be credible, otherwise, it is an outlier and goes to step 4, where χ 2 is the critical chi-square value χ obtained according to the confidence setting value of the observation 2 ; Step 4, use the speed of the tracking target [v x ,v y ] Remove abnormal points in the radar point cloud cluster associated with the detected target: Calculate , where θ is the azimuth of the point cloud; compare the radial velocity v of the point cloud r With v r θ If the difference exceeds the set threshold, the associated point cloud is removed. After the screening is completed, the remaining point cloud clusters are the final associated point cloud clusters, and the velocity component in the observation is updated to the radial velocity of the final associated point cloud cluster. The verification of step 3 is performed again. If the verification passes, the observation is considered to be credible, otherwise the observation is lost.

2. The observation feature extraction method based on vision and millimeter wave radar fusion as claimed in claim 1, characterized in that: The step 2 comprises the following steps: step 2.1, calculating the length difference ∆l and width difference ∆w between the detection target target box and the tracking target bbox box; step 2.2, calculating the unit vector of the long side direction of the tracking target bbox box and the broadside direction unit vector : , where (x0, y0) is the point with the closest Euclidean distance to the origin of the vehicle coordinates among all the vertices of the bbox box of the detection target target, (x1, y1) is the long side point connected to the nearest point (x0, y0) among all the vertices of the bbox box of the detection target target, and (x2, y2) is the wide side point connected to the nearest point (x0, y0) among all the vertices of the bbox box of the detection target target; Step 2.3, scale the bbox box of the detection target in the direction of the bbox box of the tracking target to compensate for the error caused by the size difference caused by the incomplete detection area, as shown in the following formula: , where (x' b ,y' b ) is the center position of the bbox frame of the detected target after correction and alignment, (x b ,y b ) is the center position of the bbox box of the detected target before correction and alignment.

3. The observation feature extraction method based on vision and millimeter wave radar fusion as claimed in claim 1, characterized in that: In step 3: if the tracking filter system is a linear Gaussian system, then: , where H k is the observation matrix, R k is the measurement noise covariance matrix, P k|k-1 is the prior estimated covariance matrix at time k, is the prior state estimate at the kth moment predicted at the k-1th moment; if the tracking filter system is a nonlinear Gaussian system, then: , where J H is the Jacobian matrix.

Images