Tunnel target object perception method based on 4d millimeter wave radar and camera fusion

Abstract

The application discloses a tunnel target object sensing method based on 4D millimeter wave radar and camera fusion, comprising the following steps: judging whether a vehicle enters a tunnel or exits the tunnel; when judging that the vehicle enters the tunnel, tracking the target object until the vehicle exits the tunnel; collecting image data of the target object in a 2D space through a camera, converting the pixel depth information of the image data into a 3D space detection result and outputting; collecting point cloud information through a 4D millimeter wave radar, denoising and clustering the point cloud information to obtain a 3D target detection result and output; and integrating the detection result and the 3D target detection result to generate target object information. The application realizes redundant design of a sensing end, improves environmental sensing capability, and in the case that the misrecognition rate is low, can stably identify stationary vehicles or obstacles in the tunnel, improves the robustness and safety of an autonomous vehicle, and ensures passenger safety.

Description

Technical Field

[0001] This invention belongs to the field of target detection, specifically relating to a tunnel target perception method based on the fusion of 4D millimeter-wave radar and camera. Background Technology

[0002] With the increasing prevalence of intelligent vehicles, Level 2+ / Level 3- autonomous driving vehicles, as classified by SAE, are becoming more and more common in the market. Currently, mass-produced intelligent driving vehicles primarily rely on forward-facing millimeter-wave radar and forward-looking ADAS camera sensors, enabling functions such as AEB (Emergency Braking Assist), ACC (Adaptive Cruise Control), LKA (Lane Keeping Assist), ICA (Intelligent Cruise Assist), and TLC (Tempted Lane Change). However, due to the inherent disadvantages of millimeter-wave radar, it cannot accurately identify stationary objects on the road, and its performance is limited in tunnels, resulting in inaccurate environmental perception in tunnels. Furthermore, cameras are affected by lighting conditions, and in poor lighting conditions within tunnels, they cannot accurately identify objects. Summary of the Invention

[0003] The purpose of this invention is to provide a tunnel target object perception method based on the fusion of 4D millimeter-wave radar and camera. This method achieves redundant design at the perception end, improves environmental perception capability, and can reliably identify stationary vehicles or obstacles in the tunnel with a low false recognition rate. This improves the robustness and safety of autonomous vehicles and ensures passenger safety.

[0004] To solve the above-mentioned technical problems, the technical solution of the present invention is: a tunnel target perception method based on the fusion of 4D millimeter-wave radar and camera, comprising the following steps:

[0005] Determine whether a vehicle has entered or exited a tunnel;

[0006] Once it is determined that a vehicle has entered the tunnel, the target object is tracked until the vehicle exits the tunnel:

[0007] Image data of the target object in 2D space is acquired by a camera. Based on the pixel depth information of the image data, it is converted into detection results in 3D space and output. The detection results include at least the 3D detection box of the target object, the machine confidence of the target object type, the physical position of the target object in the image, the lateral velocity and the longitudinal velocity of the target object.

[0008] Point cloud information is acquired by 4D millimeter-wave radar. After noise reduction and clustering of the point cloud information, 3D target detection results are obtained and output. The 3D target detection results include at least the radial distance, velocity and acceleration of the target, and the height information of the target.

[0009] The detection results and 3D target detection results are combined to generate target object information.

[0010] The conditions for determining whether a vehicle has entered or exited a tunnel are:

[0011] When the number of target objects captured and converted by the camera decreases or there is a state jump, and a batch of stationary target objects appear in the point cloud information collected by the 4D millimeter-wave radar, and the height distribution of the stationary target objects in a certain longitudinal distance area shows a state of high in the middle and low on both sides, it is determined that the vehicle has entered the tunnel.

[0012] When the conditions for a vehicle to enter the tunnel are met, and the number of target objects captured and converted by the camera decreases or a state jump occurs, and the number of stationary target objects in the point cloud information collected by the 4D millimeter-wave radar decreases significantly, it is determined that the vehicle has exited the tunnel.

[0013] The tracking of target objects follows these rules:

[0014] If the camera and the target object it captures and converts are in a normal matching state, the camera will track the target object normally.

[0015] If the target object captured and converted by the camera disappears or jumps, entering a state where it cannot be matched, the 4D millimeter-wave radar will continue to track the target object that is already being tracked, continuing the prediction state.

[0016] The specific process of generating target object information is as follows:

[0017] If the target object captured and converted by the camera disappears or jumps, and enters a state where it cannot be matched, the judgment is made based on the historical matching status of the target object ID captured and converted by the camera; if the camera has matched the target object before, the original target object is matched and no new target object is generated; if the camera has not matched the target object before, the target object information is generated after filtering out the 3D target detection results output by the 4D millimeter-wave radar.

[0018] The filtering process for 3D target detection results output by 4D millimeter-wave radar is as follows: targets with speeds below a certain velocity are selected from the 3D target detection results; targets on the tunnel wall are excluded based on the radial distance and height of the target; and stationary clutter on the ground is excluded based on the height of the target.

[0019] After the filtering process is completed, target information is generated.

[0020] After the filtration process is completed, the following processing steps are also included:

[0021] When the target object captured and converted by the camera completely matches the 3D target detection result output by the filtered 4D millimeter-wave radar, the target object is placed in the first-level sequence and enters the target object information generation sequence for judgment.

[0022] If the target object captured and converted by the camera cannot match the 3D target detection result output by the filtered 4D millimeter-wave radar, the target object is placed in the secondary sequence and enters the target object information generation sequence for judgment.

[0023] If the target object captured and converted by the camera partially matches the 3D target detection result output by the filtered 4D millimeter-wave radar, the target object is placed in the third-level sequence and enters the target object information generation sequence for judgment.

[0024] In the target information generation sequence judgment, target information is generated for targets located in the first-level sequence; for targets located in the second-level sequence, the matching circle is expanded until the target acquired and converted by the camera completely matches the 3D target detection result output by the filtered 4D millimeter-wave radar, and then it enters the first-level sequence to generate target information; for targets located in the third-level sequence, stationary noise and non-stationary targets are excluded according to the target velocity. If the target is a non-stationary target, the matching circle is expanded until the target acquired and converted by the camera completely matches the 3D target detection result output by the filtered 4D millimeter-wave radar, and then it enters the first-level sequence to generate target information.

[0025] A tunnel target perception system based on the fusion of 4D millimeter-wave radar and camera is also provided, including:

[0026] The camera is used to acquire image data of the target object in 2D space, and converts the pixel depth information of the image data into detection results in 3D space and outputs them. The detection results include at least the 3D detection box of the target object, the machine confidence of the target object type, the physical position of the target object in the image, the lateral velocity and the longitudinal velocity of the target object.

[0027] 4D millimeter-wave radar is used to acquire point cloud information, and after denoising and clustering the point cloud information, 3D target detection results are obtained and output; the 3D target detection results include at least the radial distance, velocity and acceleration of the target, and the height information of the target.

[0028] The vehicle controller is used to determine whether a vehicle has entered or exited a tunnel; when a vehicle has entered a tunnel, it tracks the target object; and it integrates the detection results and 3D target detection results to generate target object information.

[0029] The conditions for determining whether a vehicle has entered or exited a tunnel are:

[0030] When the number of target objects captured and converted by the camera decreases or there is a state jump, and a batch of stationary target objects appear in the point cloud information collected by the 4D millimeter-wave radar, and the height distribution of the stationary target objects in a certain longitudinal distance area shows a state of high in the middle and low on both sides, it is determined that the vehicle has entered the tunnel.

[0031] When the conditions for a vehicle to enter the tunnel are met, and the number of target objects captured and converted by the camera decreases or a state jump occurs, and the number of stationary target objects in the point cloud information collected by the 4D millimeter-wave radar decreases significantly, it is determined that the vehicle has exited the tunnel.

[0032] The tracking of target objects follows these rules:

[0033] If the camera and the target object it captures and converts are in a normal matching state, the camera will track the target object normally.

[0034] If the target object captured and converted by the camera disappears or jumps, entering a state where it cannot be matched, the 4D millimeter-wave radar will continue to track the target object that is already being tracked, continuing the prediction state.

[0035] The specific process of generating target object information is as follows:

[0036] If the target object captured and converted by the camera disappears or jumps, and enters a state where it cannot be matched, the judgment is made based on the historical matching status of the target object ID captured and converted by the camera; if the camera has matched the target object before, the original target object is matched and no new target object is generated; if the camera has not matched the target object before, the target object information is generated after filtering out the 3D target detection results output by the 4D millimeter-wave radar.

[0037] The filtering process for 3D target detection results output by 4D millimeter-wave radar is as follows: targets with speeds below a certain velocity are selected from the 3D target detection results; targets on the tunnel wall are excluded based on the radial distance and height of the target; and stationary clutter on the ground is excluded based on the height of the target.

[0038] After the filtering process is completed, target information is generated.

[0039] After the filtration process is completed, the following processing steps are also included:

[0040] When the target object captured and converted by the camera completely matches the 3D target detection result output by the filtered 4D millimeter-wave radar, the target object is placed in the first-level sequence and enters the target object information generation sequence for judgment.

[0041] If the target object captured and converted by the camera cannot match the 3D target detection result output by the filtered 4D millimeter-wave radar, the target object is placed in the secondary sequence and enters the target object information generation sequence for judgment.

[0042] If the target object captured and converted by the camera partially matches the 3D target detection result output by the filtered 4D millimeter-wave radar, the target object is placed in the third-level sequence and enters the target object information generation sequence for judgment.

[0043] In the target information generation sequence judgment, target information is generated for targets located in the first-level sequence; for targets located in the second-level sequence, the matching circle is expanded until the target acquired and converted by the camera completely matches the 3D target detection result output by the filtered 4D millimeter-wave radar, and then it enters the first-level sequence to generate target information; for targets located in the third-level sequence, stationary noise and non-stationary targets are excluded according to the target velocity. If the target is a non-stationary target, the matching circle is expanded until the target acquired and converted by the camera completely matches the 3D target detection result output by the filtered 4D millimeter-wave radar, and then it enters the first-level sequence to generate target information.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] This invention uses 4D millimeter-wave radar and cameras as the perception system for autonomous vehicles, achieving redundant design at the perception end, improving environmental perception capabilities, and reliably identifying stationary vehicles or obstacles in tunnels with a low false recognition rate, thereby improving the robustness and safety of autonomous vehicles and ensuring passenger safety. Attached Figure Description

[0046] Figure 1 This is a framework diagram of the sensing system in an embodiment of the present invention;

[0047] Figure 2 This is a flowchart illustrating the 4D millimeter-wave radar detection process in an embodiment of the present invention.

[0048] Figure 3 This is a point cloud distribution map inside the tunnel in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0050] To achieve accurate identification of stationary targets inside a tunnel, this embodiment uses a stationary vehicle as an example.

[0051] Camera target detection: First, based on image recognition, data of the target in 2D space is obtained. Then, based on the depth information of the pixels, it is converted into detection results in 3D space. The output includes, but is not limited to, the 3D detection box of the target, the target type class and confidence score, the physical position in the image, and the horizontal and vertical velocity of the target.

[0052] 4D radar target detection: Based on point cloud information, 3D target detection results are obtained after denoising and clustering, including but not limited to the target's radial distance and velocity, acceleration, target height information, etc.

[0053] When entering and exiting a tunnel, the intensity of light changes significantly, which can cause the camera to be temporarily blinded. When driving in the tunnel, the reflection of millimeter waves by the tunnel walls can cause the millimeter wave radar to see mirror targets and a large number of stationary targets.

[0054] Upon entering the tunnel, the current scene can be determined by monitoring the current camera and radar targets: The number of camera targets decreases or a status jump occurs; a large number of stationary targets appear on the 4D radar, exhibiting the following characteristics: the height distribution of stationary targets within a certain longitudinal distance area shows a pattern of high in the middle and low on both sides. Upon exiting the tunnel, the number of camera targets decreases or a status jump occurs; the number of stationary targets on the 4D radar decreases significantly.

[0055] Target tracking: Continue tracking existing targets.

[0056] 1. Camera Target: If the camera target disappears or jumps, a mismatch may occur. If a mismatch is not possible, the existing tracking target will continue to match radar data using the method of continuing the prediction state. If a normal match is achieved, tracking will proceed normally.

[0057] 2. Radar targets: Moving radar targets are unaffected by entering or exiting tunnels; normal tracking of the target continues.

[0058] Target generation:

[0059] 1. Camera Targets: Due to camera target skipping, there may be situations where a match cannot be found, leading to the generation of a new target. It is necessary to determine this based on the historical matching status of the camera target ID. If a match has been found before, the original target will be matched, and no new target will be generated. If no match has been found before, the radar target will be matched.

[0060] 2. Radar Targets: Targets with speeds below a certain threshold are filtered out based on their speed. Then, targets that could be tunnel walls are eliminated based on their lateral distance and height. Finally, targets that could be stationary ground debris are eliminated based on their height. This initial filtering of radar targets is necessary because the initial radar target status is unstable, requiring multiple frames for evaluation.

[0061] a) If the camera target matches the radar target, the target is placed in the first-level sequence, and the next frame judgment process begins.

[0062] b) If no radar target is matched, the target is placed in the secondary sequence, and the next frame judgment process begins.

[0063] c) If there is an unmatched radar target, the target is placed in the third-level sequence to proceed with the next frame judgment process.

[0064] In the next frame, targets are matched according to sequence priority. Only first-level sequence targets can be generated after a certain number of frames. Second-level sequence targets expand the matching circle. After matching a radar target, they enter the first-level sequence. After a certain number of frames, third-level sequence targets are excluded as stationary clutter based on their velocity. If they are not stationary targets, the matching circle expands. After matching a camera target, they enter the first-level sequence.

[0065] The above describes a fusion perception system based on 4D millimeter-wave radar and cameras. The key is to utilize the target height information output by the 4D millimeter-wave radar to perform target-level fusion based on 3D detection frames. It leverages the advantages of radar speed and distance measurement, as well as the advantages of camera target type detection, to stably output perception results of the surrounding environment.

[0066] In common tunnel scenarios, if a stationary vehicle is present ahead, its visibility may be impaired by lighting conditions, preventing the camera from recognizing it. Simultaneously, traditional millimeter-wave radar may also fail to detect a stationary target. If the driver does not intervene in time, a collision could occur, posing a significant risk to this type of autonomous driving system. This system utilizes the fusion of 4D millimeter-wave radar and camera perception to effectively avoid this problem, greatly enhancing safety.

[0067] In a fusion sensing system, such as Figure 1 As shown.

[0068] Based on the principle of radar imaging, when entering a tunnel, electromagnetic waves are reflected inside the tunnel and can output a point cloud distributed on the tunnel wall, such as... Figure 3 As shown, since these points share similar characteristics, such as consistent height information, they can be simulated using a semi-cylindrical model. Based on the DBSCAN clustering algorithm, "useless" point clouds attached to the tunnel wall are removed. Then, using a conventional point cloud clustering algorithm, the radar detection results are output, including separation, target length, width, and height information.

[0069] After the point cloud signal of a stationary vehicle in the tunnel is processed as described above, the output pose signal is as follows: a stationary vehicle with height information. Since the ground point cloud was removed earlier, it is sufficient to determine that the obstacle is a stationary vehicle, rather than a false detection.

[0070] Similarly, other stationary obstacles in the tunnel can be stably identified through this fusion perception system, and then collision avoidance decisions can be made in the back-end control, greatly improving the safety of autonomous vehicles.

[0071] This invention uses cutting-edge 4D millimeter-wave radar and cameras as the perception system for autonomous vehicles, achieving redundant design at the perception end, improving environmental perception capabilities, and reliably identifying stationary vehicles or obstacles in tunnels with a low false recognition rate, thereby improving the robustness and safety of autonomous vehicles and ensuring passenger safety.

[0072] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A tunnel target perception method based on fusion of 4D millimeter-wave radar and camera, characterized in that, Includes the following steps: Determine whether a vehicle has entered or exited a tunnel; Once it is determined that a vehicle has entered the tunnel, the target object is tracked until the vehicle exits the tunnel: Image data of the target object in 2D space is acquired by a camera. Based on the pixel depth information of the image data, the detection results in 3D space are converted and output. The detection results include at least the 3D detection box of the target object, the machine confidence of the target object type, the physical position of the target object in the image, the lateral velocity and the longitudinal velocity of the target object. Point cloud information is acquired by 4D millimeter-wave radar. After denoising and clustering of the point cloud information, 3D target detection results are obtained and output. 3D target detection results should include at least the target's radial distance, velocity and acceleration, and target height information; The detection results and 3D target detection results are integrated to generate target object information; The specific process of generating target object information is as follows: If the target object captured and converted by the camera disappears or jumps, and enters a state where it cannot be matched, the judgment is made based on the historical matching status of the target object ID captured and converted by the camera; if the camera has matched the target object before, the original target object will be matched, and no new target object will be generated. If the camera has not matched the target, the target information is generated after filtering out the 3D target detection results output by the 4D millimeter-wave radar. After the filtration process is completed, the following processing steps are also included: When the target object captured and converted by the camera completely matches the 3D target detection result output by the filtered 4D millimeter-wave radar, the target object is placed in the first-level sequence and enters the target object information generation sequence for judgment. If the target object captured and converted by the camera cannot match the 3D target detection result output by the filtered 4D millimeter-wave radar, the target object is placed in the secondary sequence and enters the target object information generation sequence for judgment. If the target object captured and converted by the camera partially matches the 3D target detection result output by the filtered 4D millimeter-wave radar, the target object is placed in the third-level sequence and enters the target object information generation sequence for judgment. In the target information generation sequence determination, target information is generated for target objects located in the first-level sequence; For targets located in the secondary sequence, the matching circle is expanded until the target acquired and converted by the camera completely matches the 3D target detection result output by the filtered 4D millimeter-wave radar. Then, it enters the primary sequence to generate target information. For targets located in the third sequence, stationary clutter and non-stationary targets are excluded based on the target velocity. If the target is a non-stationary target, the matching circle is expanded until the target acquired and converted by the camera completely matches the 3D target detection result output by the filtered 4D millimeter-wave radar. Then, it enters the primary sequence to generate target information.

2. The tunnel target perception method based on 4D millimeter-wave radar and camera fusion according to claim 1, characterized in that, The conditions for determining whether a vehicle has entered or exited a tunnel are: When the number of target objects captured and converted by the camera decreases or there is a state jump, and a batch of stationary target objects appear in the point cloud information collected by the 4D millimeter-wave radar, and the height distribution of the stationary target objects in a certain longitudinal distance area shows a state of high in the middle and low on both sides, it is determined that the vehicle has entered the tunnel. When the conditions for a vehicle to enter the tunnel are met, and the number of target objects captured and converted by the camera decreases or a state jump occurs, and the number of stationary target objects in the point cloud information collected by the 4D millimeter-wave radar decreases significantly, it is determined that the vehicle has exited the tunnel.

3. The tunnel target perception method based on 4D millimeter-wave radar and camera fusion according to claim 1, characterized in that, The tracking of target objects follows these rules: If the camera and the target object it captures and converts are in a normal matching state, the camera will track the target object normally. If the target object captured and converted by the camera disappears or jumps, entering a state where it cannot be matched, the 4D millimeter-wave radar will continue to track the target object that is already being tracked, continuing the prediction state.

4. The tunnel target perception method based on 4D millimeter-wave radar and camera fusion according to claim 3, characterized in that, The filtering process for 3D target detection results output by 4D millimeter-wave radar is as follows: targets with speeds below a certain velocity are selected from the 3D target detection results; targets on the tunnel wall are excluded based on their radial distance and height; and stationary noise on the ground is excluded based on their height.

5. A system using the tunnel target perception method based on 4D millimeter-wave radar and camera fusion as described in claim 1, characterized in that, include: A camera is used to acquire image data of a target object in 2D space, and converts the pixel depth information of the image data into detection results in 3D space and outputs them. The detection results should include at least the 3D bounding box of the target object, the machine confidence score of the target object type, the physical location of the target object in the image, and the lateral and longitudinal velocities of the target object. 4D millimeter-wave radar is used to acquire point cloud information, and after denoising and clustering the point cloud information, 3D target detection results are obtained and output. 3D target detection results should include at least the target's radial distance, velocity and acceleration, and target height information; The vehicle controller is used to determine whether a vehicle has entered or exited a tunnel; when a vehicle has entered a tunnel, it tracks the target object; and it integrates the detection results and 3D target detection results to generate target object information. The specific process of generating target object information is as follows: If the target object captured and converted by the camera disappears or jumps, and enters a state where it cannot be matched, the judgment is made based on the historical matching status of the target object ID captured and converted by the camera; if the camera has matched the target object before, the original target object will be matched, and no new target object will be generated. If the camera has not matched the target, the target information is generated after filtering out the 3D target detection results output by the 4D millimeter-wave radar. After the filtration process is completed, the following processing steps are also included: When the target object captured and converted by the camera completely matches the 3D target detection result output by the filtered 4D millimeter-wave radar, the target object is placed in the first-level sequence and enters the target object information generation sequence for judgment. If the target object captured and converted by the camera cannot match the 3D target detection result output by the filtered 4D millimeter-wave radar, the target object is placed in the secondary sequence and enters the target object information generation sequence for judgment. If the target object captured and converted by the camera partially matches the 3D target detection result output by the filtered 4D millimeter-wave radar, the target object is placed in the third-level sequence and enters the target object information generation sequence for judgment. In the target information generation sequence determination, target information is generated for target objects located in the first-level sequence; For targets located in the secondary sequence, the matching circle is expanded until the target acquired and converted by the camera completely matches the 3D target detection result output by the filtered 4D millimeter-wave radar. Then, it enters the primary sequence to generate target information. For targets located in the third sequence, stationary clutter and non-stationary targets are excluded based on the target velocity. If the target is a non-stationary target, the matching circle is expanded until the target acquired and converted by the camera completely matches the 3D target detection result output by the filtered 4D millimeter-wave radar. Then, it enters the primary sequence to generate target information.

6. The system according to claim 5, characterized in that, The conditions for determining whether a vehicle has entered or exited a tunnel are: When the number of target objects captured and converted by the camera decreases or there is a state jump, and a batch of stationary target objects appear in the point cloud information collected by the 4D millimeter-wave radar, and the height distribution of the stationary target objects in a certain longitudinal distance area shows a state of high in the middle and low on both sides, it is determined that the vehicle has entered the tunnel. When the conditions for a vehicle to enter the tunnel are met, and the number of target objects captured and converted by the camera decreases or a state jump occurs, and the number of stationary target objects in the point cloud information collected by the 4D millimeter-wave radar decreases significantly, it is determined that the vehicle has exited the tunnel.

7. The system according to claim 5, characterized in that, The tracking of target objects follows these rules: If the camera and the target object it captures and converts are in a normal matching state, the camera will track the target object normally. If the target object captured and converted by the camera disappears or jumps, entering a state where it cannot be matched, the 4D millimeter-wave radar will continue to track the target object that is already being tracked, continuing the prediction state.

8. The system according to claim 7, characterized in that, The filtering process for 3D target detection results output by 4D millimeter-wave radar is as follows: targets with speeds below a certain velocity are selected from the 3D target detection results; targets on the tunnel wall are excluded based on their radial distance and height; and stationary noise on the ground is excluded based on their height.

Images