A method and system for automatic driving target recognition on unstructured roads

Abstract

This invention discloses a method and system for autonomous driving target recognition on unstructured roads. The system uses a visual sensor, a lidar, and a 4D millimeter-wave radar installed on an autonomous vehicle to sense the unstructured road ahead, extract and classify targets, and then fuse target points. The system ultimately outputs a structured road simulation image to achieve target identification and tracking. The system offers reliable detection results and enables multi-sensor target point fusion.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and system for automatic driving target recognition on unstructured roads. Background Art

[0002] Autonomous driving generally refers to the ability of an unmanned autonomous driving system to perform all driving operations within a defined road and environment. Therefore, in this scenario, autonomous driving systems place even more stringent demands on vehicle perception performance. A single sensor is no longer sufficient to meet the complex environmental information perception requirements of urban road scenarios. Consequently, multi-sensor fusion has become a common research focus. In recent years, multi-sensor fusion perception technology has been widely applied in the autonomous driving field, combining different sensors and data modalities for image semantic segmentation, object recognition, and object tracking, achieving remarkable results.

[0003] Object detection is a key component of autonomous driving technology. Autonomous vehicles require sensors to accurately perceive and detect objects in their surroundings. Commonly used sensors on vehicles include lidar, millimeter-wave radar, visual sensors, and ultrasonic sensors. Lidar offers the advantages of precise detection accuracy, high resolution, fast response speed, and strong anti-interference capabilities. It can accurately determine the dimensions of obstacles and, by tracking them, accurately determine their velocity and direction, serving as a basis for determining their outline, position, and velocity. Millimeter-wave radar, with its high resolution and strong anti-interference capabilities, can accurately measure radial velocity around obstacles. In multi-sensor fusion solutions, it can serve as the primary basis for obstacle speed measurement. Visual sensors, such as cameras, collect a wealth of information and can simultaneously detect roadways, vehicles, pedestrians, traffic signs, and traffic lights within their field of view. They can serve as the primary basis for obstacle type determination.

[0004] The advantages of existing 3D millimeter-wave radars are that they are weather-resistant and can provide information on the relative speed and distance of surrounding objects. However, their low resolution makes it difficult to clearly identify small objects, they are sensitive to metal, and they are weak in detecting static objects and have low accuracy. LiDAR has the advantages of high resolution and accuracy, capable of acquiring 3D point cloud images of surrounding objects. However, it is expensive, susceptible to environmental factors, and has limited installation locations. Vision sensors can measure color and light intensity, making them suitable for target classification and segmentation. They are low-cost and easy to install and deploy, but they are weak in distance perception and nighttime detection.

[0005] However, autonomous driving object detection in unstructured road scenarios remains problematic. Sensors are affected by external factors such as small objects, noise, and inclement weather. These factors can cause errors to accumulate, ultimately leading to unreliable object detection results. The main technical challenges in fusing millimeter-wave radar and vision sensors lie in matching and fusing target points between the two sensors, as well as correlating multiple targets.

[0006] Therefore, a new technical solution is needed to solve the above problems. Summary of the Invention

[0007] In order to solve the problems arising from the prior art, the present invention provides a method and system for automatic driving target recognition on unstructured roads, which has reliable detection results and realizes multi-sensor target point fusion.

[0008] To achieve the above objectives, the present invention is used in an automatic driving target recognition method for unstructured roads, which can adopt the following technical solutions:

[0009] A method for identifying targets on an unstructured road in an autonomous driving vehicle, wherein the unstructured road has multiple obstacles that obstruct each other. The autonomous driving vehicle is equipped with a radar and a camera, and a sensor is provided within the camera to identify obstacles in front of the vehicle. The method comprises the following steps:

[0010] 1) Sense the road conditions ahead of the vehicle, use sensors to detect obstacle categories, obtain multi-frame target classification maps, and use radar to detect obstacle status and obtain raw data;

[0011] 2) Preprocess the raw data to form a point cloud map, and calibrate the point cloud map with the target classification map to obtain the radar spatial fusion map and the camera spatial fusion map respectively;

[0012] 3) Extract feature points from the background image of the radar spatial fusion image of adjacent frames, process all feature points in the frame based on the optical flow vector length thresholding algorithm, filter static points and obtain a filtered image;

[0013] 4) The filter graph includes multiple classification samples, and the point cloud is clustered according to the local density and distance of each classification sample to obtain the target parameters;

[0014] 5) Based on the Kalman filter, a multimodal motion model is established to process target parameters for target tracking, and decisions are made based on the data list of the tracked target statistics to obtain radar point data;

[0015] 6) Project the radar point data onto the camera space fusion map, and obtain the target point after successful matching;

[0016] 7) Detect the target point based on the double judgment strategy, remove the noise points and false detection points, and obtain the correct target point.

[0017] Furthermore, the sensor is a visual sensor, and one is set; the radar is a 4D millimeter wave radar and a laser radar, and two 4D millimeter wave radars are set, and one laser radar is set; the laser radar is used when the millimeter wave radar or the visual sensor fails.

[0018] Furthermore, the raw data in step 1) includes the relative distance, relative speed, horizontal angle, and vertical angle of the obstacle.

[0019] Furthermore, in step 2), the preprocessing is to perform fast Fourier transform on the raw data to generate a point cloud map; calibration is completed through matrix operation, and when any at least one of the radar and visual sensors fails, no calibration is performed.

[0020] Furthermore, in step 3), the optical flow vector length thresholding algorithm is as follows: counting the number of optical flow vectors of all feature points in the frame, thresholding the length of the optical flow vector, adjusting the threshold according to the classification target, the threshold being the moving speed of the target, and accurately classifying the target to obtain multiple classification samples.

[0021] Furthermore, in step 4), multiple density peak points are determined according to the local density and distance of each classified sample, and clusters are divided according to the distance between the density peak points and the local density; the target parameters include target length, width, height, volume, projected area, orientation and speed.

[0022] Furthermore, in step 5), the multimodal motion model introduces an extended object tracker to process target parameters, and uses the target's historical trajectory data and predicted data to correct the target position; at the same time, the spatiotemporal dependency is introduced to determine the sampling period, and the largest period among the radar and sensor is selected as the sampling period.

[0023] Furthermore, in step 5), the decision-making process uses the data list to perform state information analysis and extraction to decompose measurement errors, trajectory prediction accuracy, and target dynamic behavior errors, update the model and its parameters, and determine accurate decisions for the target in different scenarios.

[0024] Furthermore, in step 6), a matching threshold is pre-set, and the radar point data is projected onto the camera space fusion map to obtain a target frame corresponding to the radar target; the ratio of the intersection and union of the radar target frame pixels and the camera target frame pixels is calculated as the matching ratio, and the calculated matching ratio is compared with the pre-set matching threshold. When the matching ratio is greater than the matching threshold, the match is successful.

[0025] To achieve the above objectives, the present invention provides an autonomous driving target recognition system for unstructured roads, which can adopt the following technical solutions:

[0026] An autonomous driving target recognition system for unstructured roads with multiple obstacles that obstruct each other. The autonomous driving vehicle is equipped with radar and a camera, and the camera is equipped with a sensor to identify obstacles in front of the vehicle. The system includes the following modules:

[0027] The data acquisition module is used to sense the road conditions ahead of the vehicle. The sensor detects the obstacle category and obtains a multi-frame target classification map. The radar senses the obstacle status and obtains the original data.

[0028] The data preprocessing module is used to preprocess the raw data to form a point cloud map, and calibrate the point cloud map with the target classification map to obtain the radar space fusion map and the camera space fusion map respectively;

[0029] The feature point processing module is used to extract feature points from the background image of the radar spatial fusion image of adjacent frames, process all feature points in the frame based on the optical flow vector length thresholding algorithm, filter static points and obtain a filtered image;

[0030] A target parameter processing module is used to perform point cloud clustering on multiple classification samples included in the filter graph according to the local density and distance of each classification sample to obtain the target parameter;

[0031] The radar point data processing module establishes a multimodal motion model based on Kalman filtering to process target parameters for target tracking, makes decisions based on the data list of the tracked target statistics, and obtains radar point data;

[0032] The target point acquisition module is used to project the radar point data onto the camera space fusion map and obtain the target point after successful matching;

[0033] The target point processing module is used to detect target points based on a dual judgment strategy, remove noise points and false detection points, and obtain the correct target points.

[0034] The present invention has the following beneficial effects:

[0035] 1. The system uses 4D millimeter-wave radar, lidar, and vision sensors, which have high resolution and can clearly identify smaller objects. It is also highly capable of sensing static objects and has high object penetration capabilities. At the same time, redundant lidar facilitates data collection and image processing when the 4D millimeter-wave radar or vision sensor is damaged, ensuring reliable detection results.

[0036] 2. The present invention spatially fuses the target classification map obtained by the sensor with the point cloud map obtained by the radar, performs temporal fusion based on the sampling time, and sets a threshold to achieve precise matching of the sensor's target frame with the radar's target frame, thus realizing multi-sensor target point fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of the target identification method of the present invention;

[0038] Figure 2 This is a scenario of target recognition on unstructured roads according to the present invention;

[0039] Figure 3 This is a flowchart for radar point cloud image processing;

[0040] Figure 4 This is the flow chart of the Kalman filter algorithm;

[0041] Figure 5 This is the perception status diagram of Example 1;

[0042] Figure 6 This is the road simulation diagram of Example 1;

[0043] Figure 7 This is the perception status diagram of Example 2;

[0044] Figure 8 This is a road simulation diagram of Example 2. DETAILED DESCRIPTION

[0045] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. After reading the present invention, modifications of various equivalent forms of the present invention made by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0046] See also Figures 1 to 3 As shown, the present invention discloses a method for automatic driving target recognition for unstructured roads. The unstructured roads have multiple obstacles that block each other. The automatic driving vehicle is equipped with a radar and a camera. The camera is provided with a sensor to identify obstacles in front of the vehicle. The method includes the following steps:

[0047] 1) Sense the road conditions ahead of the vehicle. Sensors detect obstacle categories and generate multi-frame target classification maps. Radar detects obstacle status and generates raw data, including the relative distance, relative speed, horizontal angle, and vertical angle of the obstacle. The sensor is a visual sensor, with one installed. The radars are two 4D millimeter-wave radars and one lidar. The lidar is used if the millimeter-wave radar or visual sensor fails.

[0048] 2) Preprocess the raw data to form a point cloud image. This preprocessing process includes performing a fast Fourier transform on the raw data. The point cloud image is then calibrated with the target classification map to produce a radar spatial fusion image and a camera spatial fusion image, respectively. In this embodiment, calibration is performed through matrix operations, and calibration is not performed if at least one of the radar and visual sensors fails.

[0049] 3) Extract feature points from the background image of the radar spatial fusion image of adjacent frames. All feature points within the frames are processed using an optical flow vector length thresholding algorithm. Static points (non-pedestrians and non-vehicles) are filtered out to obtain a filtered image. The optical flow vector length thresholding algorithm involves counting the optical flow vectors of all feature points within the frames, applying a threshold to the optical flow vector length, and adjusting the threshold based on the target's movement speed. This fine-tunes the target classification to obtain multiple classification samples.

[0050] In this embodiment, roadside obstacles, moving vehicles and pedestrians are distinguished and classified according to their moving speed; the car speed threshold is set to 80km / h, and the detected target speed exceeding this limit is considered to be a non-vehicle target and filtered; the pedestrian walking speed threshold is 1.8m / s, and the detected target speed less than this limit is considered to be a non-pedestrian target and filtered, ultimately achieving the filtering of static points.

[0051] 4) The filter graph includes multiple classification samples. Target parameters are obtained by clustering the point cloud based on the local density and distance of each classification sample. Specifically, multiple density peaks are determined based on the local density and distance of each classification sample. Clusters are then divided based on the distance between density peaks and the local density. The resulting target parameters include target length, width, height, volume, projected area, orientation, and speed.

[0052] 5) A multimodal motion model based on Kalman filtering is established to process target parameters for target tracking. Decisions are made based on the data list of the tracked target statistics to obtain radar point data. The multimodal motion model introduces an extended object tracker to process target parameters from multiple sensors and establish associations. The target's historical trajectory data and predicted data are used to correct the target position. Spatiotemporal dependencies are also introduced to determine the sampling period, with the largest period between the radar and the sensor selected as the sampling period. The decision-making process for obtaining radar point data involves analyzing and extracting state information using the data list to decompose measurement errors, trajectory prediction accuracy, and target dynamic behavior errors. The model and its parameters are then updated to determine accurate decisions about the target in different scenarios and obtain radar point data.

[0053] 6) Project the radar point data onto the camera spatial fusion map, and obtain the target point after successful matching. The specific process is as follows: pre-set the matching threshold, project the radar point data onto the camera spatial fusion map, and obtain the target frame corresponding to the radar target; calculate the ratio of the intersection and union of the radar target frame and camera target frame pixels as the matching ratio, and compare the calculated matching ratio with the pre-set matching threshold. When the matching ratio is greater than the matching threshold, the match is successful.

[0054] 7) Detect the target point based on the dual judgment strategy, remove noise points and false detection points, and obtain the correct target point. The target point is output in the form of an unstructured road simulation map.

[0055] The present invention also discloses an autonomous driving target recognition system for unstructured roads. The unstructured roads have multiple obstacles that block each other. The autonomous driving vehicle is equipped with a radar and a camera. The camera is equipped with a sensor to identify obstacles in front of the vehicle. The system includes the following modules:

[0056] The data acquisition module is used to sense the road conditions ahead of the vehicle. The sensor detects the obstacle category and obtains a multi-frame target classification map. The radar senses the obstacle status and obtains the original data.

[0057] The data preprocessing module is used to preprocess the raw data to form a point cloud map, and calibrate the point cloud map with the target classification map to obtain the radar space fusion map and the camera space fusion map respectively;

[0058] The feature point processing module is used to extract feature points from the background image of the radar spatial fusion image of adjacent frames, process all feature points in the frame based on the optical flow vector length thresholding algorithm, filter static points and obtain a filtered image;

[0059] A target parameter processing module is used to perform point cloud clustering on multiple classification samples included in the filter graph according to the local density and distance of each classification sample to obtain the target parameter;

[0060] The radar point data processing module establishes a multimodal motion model based on Kalman filtering to process target parameters for target tracking, makes decisions based on the data list of the tracked target statistics, and obtains radar point data;

[0061] The target point acquisition module is used to project the radar point data onto the camera space fusion map and obtain the target point after successful matching;

[0062] The target point processing module is used to detect target points based on a dual judgment strategy, remove noise points and false detection points, and obtain the correct target points.

[0063] In this embodiment, two types of unstructured roads are used for target recognition, namely, embodiment 1 and embodiment 2.

[0064] See also Figures 4 and 5 In Example 1, there are parked vehicles and pedestrians blocked by the vehicles on the unstructured road. In this example, target perception is performed on pedestrian A blocked by parked vehicle B. The method is as follows:

[0065] (1) Use 4D millimeter-wave radar and visual sensors to perceive the unstructured road ahead and obtain data on vehicles on the road, vehicles parked on the roadside, and pedestrians. The 4D millimeter-wave radar detects the relative distance and relative speed of dynamic vehicles by transmitting and receiving radar signals, obtaining raw data. The visual sensor mainly obtains dynamic vehicles, static vehicles, lane lines, and traffic signs. By perceiving the road environment ahead in real time, it preliminarily classifies the perceived objects and obtains a target classification map. LiDAR is mainly used for data perception and acquisition when the visual sensor and millimeter-wave radar are damaged.

[0066] (2) The raw data undergoes a fast Fourier transform to generate a point cloud. Using the RT matrix, the coordinates in the 4D millimeter wave coordinate system are transformed into the world coordinate system centered on the camera. The coordinates in the world coordinate system are then transformed into the camera coordinate system. Finally, the coordinates in the camera coordinate system are transformed into the coordinate system of the target classification map. Finally, the point cloud map and the target classification map are calibrated to complete the spatial fusion of the 4D millimeter wave radar and the camera, and the radar spatial fusion map and the camera spatial fusion map are obtained.

[0067] It should be noted that when any of the millimeter-wave radar and visual sensors fails, calibration will not be performed, and the system will automatically trigger the lidar, and the sensor and radar will work independently.

[0068] (3) The radar spatial fusion image is subjected to static point filtering, point cloud clustering, target tracking, and parameter extraction in sequence.

[0069] Among them, static points will affect the processing of subsequent data and the prediction of the vehicle's next driving state. The presence of static points can easily lead to misjudgment, so static points need to be filtered out. The specific process is: extract the feature points in the background image of the radar spatial fusion image of adjacent frames, and use the optical flow vector length thresholding algorithm to process the feature points in all frames. Based on the actual speed of the autonomous vehicle, the relative speed of the target is converted into an absolute speed, and the absolute speed is used as the threshold of the optical flow vector length. The specific operations and algorithms are as follows;

[0070] ① Obtain the relationship between the distance between target pedestrian A and the millimeter-wave radar, delay, and light speed (1-1), as well as the distance to the stationary target (1-2) as follows;

[0071] D = △t×c / 2; (1-1)

[0072] D=c×T×△f / 4△F; (1-2)

[0073] Where, the distance from the target pedestrian A to the 4D millimeter-wave radar is D, T is the 4D millimeter-wave radar period, the speed of light is c, the delay between the radar transmission signal and the reflected signal is △t, the frequency of the difference frequency signal after mixing is △f, and △F is the 4D millimeter-wave bandwidth.

[0074] ② According to the Doppler effect, compared with the stationary target signal, the moving target signal will produce a delay between the transmitted signal and the reflected signal, and will also produce a Doppler frequency shift. The relationship is as follows:

[0075] fb+=△f-fd;

[0076] fb-=△f+fd;

[0077] Combining the above two equations, we can get:

[0078] fd=(fb--fb+) / 2(2-1);

[0079] △f=(fb-+fb+) / 2(2-2);

[0080] Substituting (2-2) into (1-2), the formula for calculating the distance of the moving target is as follows;

[0081] D=c×T×(fb-+fb+) / 8△F;

[0082] At the same time, the difference frequency of the Doppler shift is proportional to the speed of the dynamic target, and the relationship is as follows;

[0083] fd=2×f0×v / c(2-3);

[0084] Combining equations (2-1) and (2-3), we can get the formula for calculating the absolute speed of the moving target, as follows:

[0085] v=c(fb--fb+) / 4f0;

[0086] Where fb+ is the frequency difference between the rising phase of the moving target's reflected signal and the transmitted signal, fb- is the frequency difference between the falling phase of the moving target's reflected signal and the transmitted signal, fd is the difference frequency caused by the Doppler shift, f0 is the frequency of the 4D millimeter-wave radar, and the speed of the target pedestrian A relative to the 4D millimeter-wave radar is v.

[0087] ③ The above formula can be used to determine the distance and absolute speed of pedestrian A, the target detected by the 4D millimeter-wave radar. After processing the data, the relative distance between parked vehicle B and the autonomous vehicle, as well as the relative distance and absolute speed of pedestrian A to the autonomous vehicle, can be determined. Roadside obstacles, moving vehicles, and pedestrians are classified based on their absolute speeds. The vehicle speed is set to 80 km / h, and targets exceeding this speed are considered non-vehicle targets and are filtered out. Pedestrians have a maximum walking speed of approximately 1.8 m / s, and targets below this speed are considered non-pedestrian targets and are filtered out, ultimately achieving static point filtering.

[0088] (4) The filter graph includes multiple classification samples. Point cloud clustering is performed based on the local density and distance of each classification sample to obtain the target parameters. To achieve target object clustering of the filter graph, the distance of the samples is calculated, and an appropriate distance threshold is selected. The density peak point is determined based on the local density and distance of each sample, and the clusters are divided based on the distance between the density peak points and the local density. The above operations are repeated continuously to obtain more accurate vehicle status data, i.e., the target parameters.

[0089] (5) Based on the Kalman filter, a multimodal motion model is established to process the target parameters and the target is tracked adaptively using a periodic method. The decision is made based on the data list of the tracked target statistics to obtain the radar point data.

[0090] The multimodal motion model learns the target's motion patterns and the influencing factors of the external environment, uses the target's historical trajectory data and predicted data to correct the target's position, and introduces spatiotemporal dependencies to improve target tracking accuracy. It should be noted that if either the millimeter-wave radar or the visual sensor fails, the target's position is predicted using the lidar data, while the other millimeter-wave radar or visual sensor performs standardized processing of target motion detection to output the target's historical trajectory data. If both sensors fail simultaneously, the lidar data is used to predict and correct the target's position.

[0091] Specifically, the adaptive sampling period temporal fusion method considers the sampling periods of both sensors during the fusion process. Based on factors such as the complexity of the real-time scene and the target's motion speed, the larger sampling period is intelligently selected as the fused sampling period. It should be noted that if either the millimeter-wave radar or the vision sensor fails, the system automatically triggers the lidar sensor, at which point the system's temporal fusion ceases. The lidar and vision sensor or millimeter-wave radar each perform target recognition independently and do not participate in data fusion, performing only data association analysis. Furthermore, the primary data detection task remains the operation of the surviving sensor, while the lidar continues to serve as a redundant detection reserve for data.

[0092] See also Figure 4 As shown in FIG, a multi-modal motion model based on Kalman filtering processes target parameters to realize multi-target tracking. The method flow is as follows, including:

[0093] ① Based on the target characteristics, the motion model and filtering algorithm are designed to establish a uniformly accelerated linear motion model and a Kalman filtering algorithm to process the target parameters.

[0094] ② According to the uniformly accelerated linear motion model and the Kalman filter algorithm, the state transfer matrix A, the measurement matrix H, and the initial noise covariance matrix Q, R, and P are determined.

[0095] ③ Based on the target parameters, the above matrices A, H, Q, R and P are processed to associate the target and calculate the time difference of the target movement.

[0096] ④ Update the noise covariance matrices Q, R, and P according to the associated targets and time differences to obtain Qk, Rk, and Pk.

[0097] ⑤ Establish the Kalman filter equation and calculate the gain matrix Kk based on the equation to obtain the state update Xk.

[0098] ⑥ Then use the trajectory-level fusion solution to fuse Qk, Rk, Pk, Xk, and output a data list of the tracked target, which represents the target tracking information.

[0099] It should be noted that a joint probabilistic data association tracker is set up for data association. When any of the millimeter-wave radar and visual sensors fails, the system automatically triggers the lidar sensor. At this time, the system automatically starts the joint probabilistic data association tracker to associate the lidar data with the data generated by the other non-failed sensor, continuously updating the target status and optimizing the target position to obtain accurate target tracking information.

[0100] The target object's data list is used to analyze and extract state information, decomposing measurement errors, trajectory prediction accuracy, and target dynamic behavior errors. Parameter and model update strategies are optimized to determine accurate target decisions in different scenarios and output radar point data. It should be noted that if either the millimeter-wave radar or the visual sensor fails, the system automatically triggers the lidar sensor. At this point, the optimized parameters must take into account the data detected by the lidar. If the millimeter-wave radar or the visual sensor is not failed, the lidar will not operate, and parameter status will not be optimized.

[0101] (6) Project the radar point data onto the camera spatial fusion map, and obtain the target point after successful matching. The specific process is as follows: estimate the position and size of the radar point data target in the camera spatial fusion map, pre-set the matching threshold, and project the radar point data onto the camera spatial fusion map to obtain the target frame corresponding to the radar target; quantify the degree of matching between the two, pay attention to whether the target frames overlap with each other, and consider the distribution of specific pixels to more accurately measure the radar and camera data, calculate the ratio of the intersection and union of the radar target frame and the camera target frame pixels as the matching ratio, and compare the calculated matching ratio with the pre-set matching threshold. When the matching ratio is greater than the matching threshold, the match is successful.

[0102] (7) Use the double judgment strategy to classify and distinguish the target points to solve the problems of noise points and false detection points in the target point database. Please refer to the output results. Figure 6 .

[0103] It should be noted that if either the millimeter-wave radar or the visual sensor fails, the system automatically triggers the lidar. At this point, the system automatically stops the dual-judgment strategy and adopts an independent judgment method to reduce the false positive rate. Furthermore, the lidar sensor data is also independently judged without triggering the dual-judgment mechanism. Multiple independent judgments of target points reduce the false positive rate and improve the robustness of the target recognition method in identifying target states and positions in unstructured road environments.

[0104] See also Figures 7 and 8 The difference between Example 2 and Example 1 is that the unstructured road environment is an intersection without traffic lights, including moving vehicles, stopped vehicles, and pedestrians crossing the road. A 4D millimeter-wave radar and camera are used to perceive the environment of the intersection ahead without traffic lights, and obtain general information data on the moving vehicles, stopped vehicles, and pedestrians crossing the road within the sensor range.

[0105] In summary, the present invention provides an automatic driving target recognition method and system for unstructured roads, which has the characteristics of reliable detection results and the realization of multi-sensor target point fusion.

Claims

1. A method for automatic driving target recognition on unstructured roads, characterized in that: There are multiple obstacles that block each other on the unstructured road. The autonomous vehicle is equipped with a radar and a camera, and a sensor is set in the camera to identify obstacles in front of the vehicle. The method includes the following steps: 1) Sense the road conditions ahead of the vehicle, use sensors to detect obstacle categories, obtain multi-frame target classification maps, and use radar to detect obstacle status and obtain raw data; 2) Preprocess the raw data to form a point cloud map, and calibrate the point cloud map with the target classification map to obtain the radar spatial fusion map and the camera spatial fusion map respectively; 3) Extract feature points from the background image of the radar spatial fusion image of adjacent frames, process all feature points in the frame based on the optical flow vector length thresholding algorithm, filter static points and obtain a filtered image; 4) The filter graph includes multiple classification samples, and the point cloud is clustered according to the local density and distance of each classification sample to obtain the target parameters; 5) Based on the Kalman filter, a multimodal motion model is established to process target parameters for target tracking, and decisions are made based on the data list of the tracked target statistics to obtain radar point data; 6) Project the radar point data onto the camera space fusion map, and obtain the target point after successful matching; 7) Detect the target point based on the double judgment strategy, remove the noise points and false detection points, and obtain the correct target point.

2. The method for automatic driving target recognition on unstructured roads according to claim 1, characterized in that: The sensor is a visual sensor, and one is provided; the radar is a 4D millimeter-wave radar and a laser radar, and two 4D millimeter-wave radars are provided, and one laser radar is provided; the laser radar is used when the millimeter-wave radar or the visual sensor fails.

3. The method for automatic driving target recognition on unstructured roads according to claim 2, characterized in that: The raw data in step 1) includes the relative distance, relative speed, horizontal angle, and vertical angle of the obstacle.

4. The method for automatic driving target recognition on unstructured roads according to claim 3, characterized in that: In step 2), the preprocessing is to perform fast Fourier transform on the raw data to generate a point cloud map; calibration is completed through matrix operation, and when at least one of the radar and visual sensors fails, no calibration is performed.

5. The method for automatic driving target recognition on unstructured roads according to claim 4, characterized in that: In step 3), the optical flow vector length thresholding algorithm is as follows: counting the number of optical flow vectors of all feature points in the frame, thresholding the length of the optical flow vector, adjusting the threshold according to the classification target, the threshold being the moving speed of the target, and accurately classifying the target to obtain multiple classification samples.

6. The method for automatic driving target recognition on unstructured roads according to claim 5, characterized in that: In step 4), multiple density peak points are determined according to the local density and distance of each classified sample, and clusters are divided according to the distance between the density peak points and the local density; the target parameters include target length, width, height, volume, projected area, direction and speed.

7. The method for automatic driving target recognition on unstructured roads according to claim 6, characterized in that: In step 5), the multimodal motion model introduces an extended object tracker to process target parameters, and uses the target's historical trajectory data and predicted data to correct the target position; at the same time, the spatiotemporal dependency relationship is introduced to determine the sampling period, and the largest period among the radar and sensor is selected as the sampling period.

8. The method for automatic driving target recognition on unstructured roads according to claim 7, characterized in that: In step 5), the decision-making process uses the data list to analyze and extract state information to decompose measurement errors, trajectory prediction accuracy, and target dynamic behavior errors, update the model and its parameters, and determine accurate decisions for the target in different scenarios.

9. The method for automatic driving target recognition on unstructured roads according to claim 8, characterized in that: In step 6), a matching threshold is pre-set, and the radar point data is projected onto the camera space fusion map to obtain a target frame corresponding to the radar target; The ratio of the intersection and union of the pixels of the radar target frame and the camera target frame is calculated as the matching ratio, and the calculated matching ratio is compared with the pre-set matching threshold. When the matching ratio is greater than the matching threshold, the match is successful.

10. An autonomous driving target recognition system for unstructured roads, characterized in that: There are multiple obstacles on the unstructured road that block each other. The autonomous vehicle is equipped with a radar and a camera. The camera is equipped with sensors to identify obstacles in front of the vehicle. The system includes the following modules: The data acquisition module is used to sense the road conditions ahead of the vehicle. The sensor detects the obstacle category and obtains a multi-frame target classification map. The radar senses the obstacle status and obtains the original data. The data preprocessing module is used to preprocess the raw data to form a point cloud map, and calibrate the point cloud map with the target classification map to obtain the radar space fusion map and the camera space fusion map respectively; The feature point processing module is used to extract feature points from the background image of the radar spatial fusion image of adjacent frames, process all feature points in the frame based on the optical flow vector length thresholding algorithm, filter static points and obtain a filtered image; A target parameter processing module is used to perform point cloud clustering on multiple classification samples included in the filter graph according to the local density and distance of each classification sample to obtain the target parameter; The radar point data processing module establishes a multimodal motion model based on Kalman filtering to process target parameters for target tracking, makes decisions based on the data list of the tracked target statistics, and obtains radar point data; The target point acquisition module is used to project the radar point data onto the camera space fusion map and obtain the target point after successful matching; The target point processing module is used to detect target points based on a dual judgment strategy, remove noise points and false detection points, and obtain the correct target points.

Images