Track maturity management method based on 4D millimeter wave radar and vision fusion

Abstract

The present invention discloses a track maturity management method based on 4D millimeter wave radar and visual fusion, taking into account the frames of the radar point cloud and the visual network detection frame from the historical frame association clustering source, and designing the mature frame value according to the ratio of the network frame number to the total detection frame number. Compared with the single frame information, the use of historical frame number information is more reliable. And according to the specific scene and the target motion state, the mature frame number of different target tracks is designed, the basic mature frame number is set to 3 frames, and the additional mature frame number is increased according to different situations, so as to achieve the purpose of setting different track mature frames for different targets in different scenes, so as to create a more reliable track, and solve the problem that the prior art does not take into account that the confidence level of each target may be different, resulting in immature tracks.

Description

Technical Field

[0001] The invention relates to a track maturity management method based on 4D millimeter wave radar and vision fusion. Background Art

[0002] LiDAR uses optical sensing technology to achieve high-precision distance measurement. The fusion of LiDAR and camera data is often used for 3D target detection in the field of autonomous driving. However, it still has some shortcomings, such as the inability to directly measure speed, being easily affected by bad weather, and not being accurate enough for detecting distant targets. Millimeter-wave radar uses electromagnetic waves in the millimeter-wave frequency band to measure the distance, speed, and angle between the target object and the radar by transmitting and receiving echo signals. Millimeter-wave radar has high speed measurement accuracy, is unimpeded by wind and rain, is not afraid of bad weather, and has a longer detection distance.

[0003] LiDAR and vision fusion usually adopts post-fusion, which has obvious advantages, good flexibility and robustness, and can solve the impact of single sensor failure. However, the disadvantages of post-fusion are also obvious, such as insufficient information utilization, long system links, and increased maintenance costs. Since the feature information of millimeter-wave radar and camera data is relatively rich, the intermediate fusion method can be used to fuse the intermediate features to achieve accurate 3D target detection.

[0004] Track management includes processes such as track initiation, track update, and track extrapolation, and is an important component of multi-target tracking. A track needs to go through a certain number of frames from initiation to maturity to form a stable track. Therefore, it is necessary to manage the track maturity process and set and limit the track maturity frames for different scenarios so that the stable track does not contain random noise and the reliable track matures and stabilizes quickly. Summary of the invention

[0005] The problem to be solved by the present invention is: since the prior art does not take into account that the possible confidence of each target is different, for a motor vehicle target, its target detection requires a faster mature track, but for a noise target, its detection result credibility needs to be reduced, and the number of frames for track maturity is extended so that its track is not released. Therefore, it is unreasonable for the prior art to set the same track maturity frame number for each possible new target, resulting in an immature track.

[0006] In order to solve the above-mentioned prior art problems and achieve the purpose of "setting different track maturation frames for different targets in different scenarios to generate more reliable tracks", the technical solution of the present invention is to provide a track maturation management method based on 4D millimeter wave radar and vision fusion, including the following steps:

[0007] Get mature flag flg matureThe input target is false. The input target attributes include: target location information (x i ,y i ) where i = 1, 2, ..., N, where N is the total number of targets in the current frame and the number of network detection frames net_detect_cnt i 、Total number of detection frames detect_cnt i 、Network detection frames include point cloud frames net_pts_cnt i where net_pts_cnt i ≤net_detect_cnt i 、The number of suspected multipath frames ghost_cnt i , the speed of the vehicle ego_speed and the angular velocity ego_yawrate;

[0008] According to the distance between the target and the vehicle, the ratio of the network detection frame number to the total detection frame number, the position, speed, whether there is occlusion, and whether it is multipath of the new target outside the network detection range are clustered by DBSCAN, and the distribution map of the mature frame number factors of the track is obtained for partitioning, and the additional frames of the mature frame number are added, and the bottom line value of the basic mature frame number is preset as default cnt For 3 frames, calculate the number of mature frames of the target in each partition The sum of the basic mature frames and the additional mature frames;

[0009] When the target detection frame number satisfy When flg mature If the judgment is true, the target track is mature.

[0010] Preferably, the partitions include: area a is |x|≤20m and |y|≤10m, area b is |x|≤20m and |y|>10m, area c is |x|>20m and |x|≤max_netdet_dis and |y|≤10m, area d is |x|>20m and |x|≤max_netdet_dis and |y|>10m, and area e is |x|≤max_netdet_dis, wherein max_netdet_dis is the network detection range.

[0011] Preferably, the number of additional frames in the a region is as follows:

[0012]

[0013] Among them, the number of additional mature frames is pure_radar_cnt i Depends on the vehicle speed and angular velocity, multipath targets and target occlusion, optional_mature_cnt iThe number of frames detected by the network is net_detect_cnt i The total number of detection frames detect_cnt i The ratio of net_det_percent i The additional number of frames for the area that is different and introduced.

[0014] Preferably, the number of additional mature frames of the b area, the c area, the d area, and the e area is pure_radar_cnt i All are consistent with the a region.

[0015] Preferably, the number of mature frames of the target in the b region includes the additional number of mature frames introduced by the y-direction distance between the target and the vehicle. Therefore, the number of mature frames of the target in area b is:

[0016]

[0017] Preferably, the number of mature frames of the target in the c region includes the additional number of mature frames introduced by the x-direction distance between the target and the vehicle. Therefore, the number of mature frames of the target in area c is:

[0018]

[0019] Preferably, the number of mature frames of the target in the d area includes: the additional number of mature frames introduced by the x-direction distance between the target and the vehicle The additional mature frames introduced by the y distance between the target and the vehicle Therefore, the number of mature frames of the target in area d is:

[0020]

[0021] Preferably, the number of mature frames of the target in the e-area includes: the additional number of mature frames introduced by the x-direction distance between the target and the vehicle Additional mature frames introduced by the y-distance between the target and the vehicle And additional mature frame number pure_radar_cnt i , so the number of mature frames of the target in area e is:

[0022]

[0023] The technical solution of the present invention proposes a track maturity management method based on 4D millimeter wave radar and visual fusion, taking into account the frames of the historical frame association clustering source as radar point cloud and visual network detection frame, and designing the mature frame value according to the ratio of network frame number to total detection frame number. Compared with single frame information, the use of historical frame number information is more reliable. And according to the specific scene and target motion state, the number of frames for different target track maturity is designed, the basic mature frame number is set to 3 frames, and the additional mature frame number is increased according to different situations, mainly considering the following additional mature frame number situations: far away from the vehicle (x direction, y direction are considered separately); blocked by other targets; fewer detection frames of the visual network; located in the detection blind area; mirror multipath target; the speed of the vehicle is slow and the vehicle rotates, etc. The final total number of mature frames is the sum of the basic mature frame number and the additional mature frame number. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the distribution of factors of the number of mature frames of a track provided by an embodiment of the present invention;

[0025] Figure 2 A graph of additional mature frames introduced by target speed and rotation under pure radar detection provided by an embodiment of the present invention;

[0026] Figure 3 A graph of the number of additional mature frames introduced by multipath targets under pure radar detection provided by an embodiment of the present invention;

[0027] Figure 4 This is a graph of the additional mature frames introduced by the obstructed target under pure radar detection provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0029] The embodiment of the present invention provides a track maturity management method based on 4D millimeter wave radar and vision fusion, and the main process is as follows:

[0030] Enter all mature flags flg mature If the target is false, mature If true, the module will not be processed. The properties of the input target include: the location information of the target (x i ,y i )(i=1,2,...,N,N is the total number of targets in the current frame), network detection frame number net_detect_cnti , total number of detection frames detect_cnt i , the number of frames containing point clouds in the network detection frame number net_pts_cnt i (net_pts_cnt i ≤net_detect_cnt i ), the number of suspected multipath frames ghost_cnt i Etc. The ego vehicle's speed ego_speed and angular velocity ego_yawrate information should also be input into this module.

[0031] The bottom line value of the basic mature frame number is 3 frames (default cnt =3), and the remaining additional frames are considered in different scenarios. Figure 1 As shown, the number of mature additional frames for track management depends on the distance between the target and the vehicle, and the factors that need to be considered are different. Specifically, for closer targets, the additional mature frames due to the increase in the distance from the vehicle in the x direction are not considered, and for closer targets, only the increased number of mature frames due to the larger distance in the y direction is considered. In addition, the influence of the ratio of the number of network detection frames to the total number of detection frames on the number of additional mature frames needs to be considered. If the ratio of the number of network detection frames is large, the number of additional mature frames will be small. For targets farther away, the influence of the x- and y-direction distances and the ratio of the number of network detection frames on the number of additional mature frames of the target needs to be considered jointly; outside the network detection range max_netdet_dis, there is no network detection result, so all new targets are generated by DBSCAN clustering, and the number of mature additional frames for targets within this range depends on factors such as the position, speed, whether there is occlusion, and whether it is multipath. The specific number of partition additional frames is as follows:

[0032] When |x|≤20m and |y|≤10m, the number of additional frames in area a is as follows:

[0033]

[0034] In the formula, optional_mature_cnt i The number of frames detected by the network is net_detect_cnt i The total number of detection frames detect_cnt i The ratio of net_det_percent i The additional number of frames introduced in the area is different; when the ratio net_det_percent i If the ratio is greater than 0.7, the additional frames in the region are 0; when the ratio net_det_percent i When it is 0, the number of additional frames of the region optional_mature_cnt i With additional mature frame number pure_radar_cnti The values ​​are consistent.

[0035] Among them, pure_radar_cnt i depending on:

[0036] 1. Additional mature frames introduced by the ego vehicle speed and angular velocity, and the target vehicle speed and angular velocity: When the ego vehicle rotates (ego_yawrate>yawrate_cri), the additional mature frames are more than when the ego vehicle does not rotate. The target scenes are further classified according to the target speed into HIGHWAY, CITY, LOW_SPEED, and STOP scenes. The slower LOW_SPEED and STOP scenes have more additional mature frames. pure_radar_cnt i =spd_inc_cr.

[0037] 2. Additional mature frames introduced by multipath targets: If the target multipath frame number ghost_cnt i When it is greater than 0 and the vehicle speed is high, the number of additional mature frames introduced by multipath targets is considered, pure_radar_cnt i =mir_inc_cri.

[0038] 3. Additional mature frames introduced by target occlusion: When the target is determined to be occluded, the additional mature frames introduced by the occluded target are considered, pure_radar_cnt i =shade_inc_cri.

[0039] The specific settings of the additional mature frames corresponding to each factor under pure radar detection are as follows: Figure 2-Figure 4 As shown, Figure 2 This is the additional mature frame number diagram introduced by target speed and rotation under pure radar detection. Figure 3 This is the additional mature frame number diagram introduced by multipath targets under pure radar detection. Figure 4 is the additional mature frame number graph introduced by the occluded target under pure radar detection, that is, Figure 2-Figure 4 Corresponding to three additional mature frame settings 1 to 3 respectively.

[0040] Therefore, the number of mature frames of the target in area a is:

[0041]

[0042] When |x|≤20m and |y|>10m, the number of additional frames in area b is as follows:

[0043]

[0044] Among them, pure_radar_cnt iThe same as area a. Therefore, the number of mature frames of the target in area b is:

[0045]

[0046] in, Additional mature frames introduced for the y distance between the target and the ego vehicle.

[0047] When |x|>20m and |x|≤max_netdet_dis and |y|≤10m, the number of additional frames in region c is as follows:

[0048]

[0049] Among them, pure_radar_cnt i The same as area a. Therefore, the number of mature frames of the target in area c is:

[0050]

[0051] in, Additional mature frames introduced for the x-distance between the target and the ego vehicle.

[0052] When |x|>20m and |x|≤max_netdet_dis and |y|>10m, the number of additional frames in region d is as follows:

[0053]

[0054]

[0055] Among them, pure_radar_cnt i The same as area a. Therefore, the number of mature frames of the target in area d is:

[0056]

[0057] in, The additional mature frames introduced for the x-distance between the target and the ego vehicle; Additional mature frames introduced for the y distance between the target and the ego vehicle.

[0058] When |x|≤max_netdet_dis, the number of mature frames of the target in region e is:

[0059]

[0060] in, The additional mature frames introduced for the x-distance between the target and the ego vehicle; The additional mature frames introduced for the y-distance between the target and the vehicle, pure_radar_cnti Same as area a, default cnt This is the lower limit of the number of mature frames.

[0061] When the target detection frame number satisfy When flg mature If the judgment is true, the target track is mature.

[0062] The embodiment of the present invention proposes a track maturation management method based on 4D millimeter wave and vision fusion. According to the specific scene and target motion state, the frame number of track maturation of different targets is designed. The frame number of track maturation of the target can be adjusted according to the specific detection situation of the target, so that the track of the noise target matures later and the track of the target with higher credibility matures earlier, thereby improving the reliability and robustness of target maturation.

[0063] The embodiment of the present invention takes into account the historical frame association clustering source as radar point cloud and the number of frames from visual network detection frame when the track is mature and the track is newly generated, and designs the mature frame value according to the ratio of the network frame number to the total detection frame number. Compared with single frame information, the use of historical frame number information is more reliable.

[0064] The embodiment of the present invention adopts a 4D millimeter-wave radar and vision fusion strategy. Compared with the laser radar and vision fusion strategy, its speed measurement accuracy is improved, it can resist the impact of bad weather, and can detect targets at a farther distance. Within the visual network detection range, the accuracy of target detection by visual network detection and radar point cloud detection is considered at the same time, and the number of mature track frames is designed according to the accuracy. When it exceeds the network detection range, the number of mature track frames can be designed according to the characteristics of the radar point cloud, so that mature tracks in the distance are also reliable.

Claims

1. A track maturity management method based on 4D millimeter wave radar and vision fusion, characterized in that: The following steps are involved: Get mature flag flg mature The input target is false. The input target attributes include: target location information (x i ,y i ), where i = 1, 2, ..., N, where N is the total number of targets in the current frame and the number of network detection frames net_detect_cnt i 、Total number of detection frames detect_cnt i 、Network detection frames include point cloud frames net_pts_cnt i， where net_pts_cnt i ≤net_detect_cnt i 、The number of suspected multipath frames ghost_cnt i , the speed of the vehicle ego_speed and the angular velocity ego_yawrate; According to the distance between the target and the vehicle, the ratio of the network detection frame number to the total detection frame number, the position, speed, whether there is occlusion, and whether it is multipath of the new target outside the network detection range are clustered by DBSCAN, and the distribution map of the mature frame number factors of the track is obtained for partitioning, and the additional frames of the mature frame number are added, and the bottom line value of the basic mature frame number is preset as default cnt For 3 frames, calculate the number of mature frames of the target in each partition The sum of the basic mature frames and the additional mature frames; When the target detection frame number satisfy When flg mature If the judgment is true, the target track is mature.

2. The track maturity management method based on 4D millimeter wave radar and vision fusion as claimed in claim 1, characterized in that: The partitions include: area a is |x|≤20m and |y|≤10m, area b is |x|≤20m and |y|>10m, area c is |x|>20m and |x|≤max_netdet_dis and |y|≤10m, area d is |x|>20m and |x|≤max_netdet_dis and |y|>10m, and area e is |x|≤max_netdet_dis, where max_netdet_dis is the network detection range.

3. The track maturity management method based on 4D millimeter wave radar and vision fusion as claimed in claim 2 is characterized in that: The number of additional frames in the a region is as follows: Among them, the number of additional mature frames is pure_radar_cnt i Depends on the vehicle speed and angular velocity, multipath targets and target occlusion, optional_maturre_cnt i The number of frames detected by the network is net_detect_cnt i The total number of detection frames detect_cnt i The ratio of net_det_percent i The additional number of frames for the area that is different and introduced.

4. The track maturity management method based on 4D millimeter wave radar and vision fusion as claimed in claim 3 is characterized in that: The number of additional mature frames of the b area, the c area, the d area, and the e area is pure_radar_cnt i All are consistent with the a region.

5. The track maturity management method based on 4D millimeter wave radar and vision fusion as claimed in claim 4, characterized in that: The number of mature frames of the target in the b area includes the additional number of mature frames introduced by the y-distance between the target and the vehicle. Therefore, the number of mature frames of the target in area b is:

6. The track maturity management method based on 4D millimeter wave radar and vision fusion as claimed in claim 4, characterized in that: The number of mature frames of the target in the c area includes the additional number of mature frames introduced by the x-distance between the target and the vehicle. Therefore, the number of mature frames of the target in area c is:

7. The track maturity management method based on 4D millimeter wave radar and vision fusion as claimed in claim 4, characterized in that: The number of mature frames of the target in the d area includes: the additional number of mature frames introduced by the x-distance between the target and the vehicle The additional mature frames introduced by the y distance between the target and the vehicle Therefore, the number of mature frames of the target in area d is:

8. The track maturity management method based on 4D millimeter wave radar and vision fusion as claimed in claim 4, characterized in that: The number of mature frames of the target in the e area includes: the additional number of mature frames introduced by the x-distance between the target and the vehicle Additional mature frames introduced by the y-distance between the target and the vehicle And additional mature frame number pure_radar_cnt i , so the number of mature frames of the target in area e is:

Images