A rapid vehicle target detection method based on radar-eye fusion

Abstract

This invention discloses a rapid vehicle target detection method based on radar-visual fusion. First, the preprocessed point cloud information of millimeter-wave radar is mapped onto the image pixel coordinate system to obtain the distribution of the target point cloud velocity information detected by the radar on the image. Second, the point cloud velocity information collected by the radar is used for region compensation to assign corresponding velocity information to all pixels in the image. Then, the RGB component values ​​of all pixels in the image and the coordinate position information of each pixel on the image are extracted. Next, the five-dimensional feature information of RGB component values, coordinate position, and velocity information are fused to form a set of feature vectors, determining the number of cluster centers K and K initial cluster center vectors. Then, the K-means algorithm is used to cluster the pixel point cloud, separating target pixels from background pixels, and outputting the classification results. Finally, vehicle target extraction is performed on the classification results to achieve rapid vehicle detection in a single frame.

Description

Technical Field

[0001] This invention relates to the field of cross-fusion technology of optical sensors and microwave sensors, and in particular to a rapid vehicle target detection method based on radar-visual fusion. Background Technology

[0002] In intelligent transportation systems, vehicle detection and tracking are key steps in making traffic event behavior decisions, providing fundamental support for subsequent high-level tasks such as vehicle decision-making, planning, and behavior control. Therefore, it is essential to use effective vehicle target detection algorithms to more accurately detect and extract moving targets, thereby laying a solid foundation for subsequent identification and tracking.

[0003] In their paper "Research on Moving Object Detection Based on Improved Gaussian Mixture Model," Huang Dongjun et al. proposed an improved Gaussian mixture model method. For the first N frames, they initialized each pixel using a combination of neighborhood characteristics and median filtering as the initial background image. For each pixel in subsequent frames, they performed a Gaussian mixture model update and matching. If a match was successful, the pixel was determined to be a background point; otherwise, it was determined to be a target point. Finally, using neighborhood characteristics, the final classification of the pixel was determined based on the probability of neighboring pixels, thus extracting the moving target. The drawback of this method is that if moving objects are present throughout the first N frames, there will be a deviation from the true background during background construction, affecting the pixel determination results. Secondly, since background modeling is based on color space modeling, matching the color of the moving target with the background color in a Gaussian background model can lead to areas with similar colors being considered background, causing some holes in the target area.

[0004] The most common method used for capturing moving vehicles in videos is background subtraction. However, the most crucial aspect of capturing moving vehicle targets using background subtraction is establishing a background model. In real traffic environments, the scenes are complex, the environment is variable, and there is a lot of interference. In particular, when the color of the vehicle target is similar to the background color, the target pixel may be identified as a background pixel. Therefore, it is difficult to estimate the true background image, which leads to the appearance of holes when extracting actual vehicle targets.

[0005] In summary, accurately identifying vehicle target outlines and resolving the void problem during vehicle target extraction, especially when the vehicle target color is similar to the background color, in complex and ever-changing scene environments, is a challenging problem to achieve rapid and accurate vehicle target detection. Summary of the Invention

[0006] The purpose of this invention is to propose a rapid vehicle target detection method based on radar-visual fusion, which solves the problem of voids in vehicle target extraction due to inaccurate background construction in complex and variable scene environments, thereby achieving rapid and accurate vehicle target detection and improving vehicle detection accuracy.

[0007] The rapid vehicle target detection method based on radar-eye fusion of the present invention includes the following steps:

[0008] S1. By transforming coordinates, the radar point cloud is projected onto the camera image to obtain the distribution of the radar-detected point cloud velocity information on the image.

[0009] S2. Use point cloud velocity information to perform region compensation, and assign velocity information to all pixels in the image.

[0010] S3. Extract the RGB color component features and pixel coordinate features of all pixels in the image;

[0011] S4. Cluster the pixels based on RGB component values, coordinate positions, and velocity information to determine the number of cluster centers K and the initial cluster center vector for each cluster center;

[0012] S5. The K-means algorithm is used to re-cluster the cluster centers, thereby achieving background and vehicle target classification;

[0013] S6. Use clustering results to extract vehicle targets.

[0014] Furthermore, the specific process of step S1 is as follows:

[0015] S11. Preprocess the raw point cloud information detected by radar to filter out unnecessary clutter and retain multiple radar reflection points of the target.

[0016] S12. Perform time matching on radar data and video data from the perspectives of time start point and data frame;

[0017] S13. Transform the matched point cloud from the radar coordinate system to the pixel coordinate system and project it onto the image.

[0018] Furthermore, the specific process of step S2 is as follows: taking each point in the point cloud as the center and an n*n window as the neighborhood, the velocity information of that point is assigned to other pixels within that window; while for the pixels of streetlights and the ground, the velocity information is compensated to 0, so that all pixels in the image are assigned velocity information.

[0019] Furthermore, the specific process of step S4 is as follows: Take pixel u from the image. i And set it as the center of the first class, then apply it to the other pixels in the image. sThe similarity of the feature vector convergence values ​​is determined according to the following formula:

[0020]

[0021]

[0022]

[0023] In the formula, Each pixel is u i The six elements of the normalized eigenvector; Each pixel is u s The 6 elements after normalization of the feature vector; m1, m2, and m3 are all thresholds that are greater than 0 and adjustable; These are respectively location distance bandwidth, velocity information bandwidth, and color distance bandwidth;

[0024] If the above formula is satisfied, then pixel u will be... s With pixel u i Merge them into one category; otherwise, merge pixel u. s As a new class center, continue to calculate the remaining sample pixels in this way until all pixel data has been processed, thus dividing the entire scene into K categories.

[0025] Furthermore, the specific process of step S5 is as follows:

[0026] S51. Take the mean vector u of all pixels at each cluster center. k As the cluster center vector; that is, each class has multiple pixels, and the mean vector of all pixels in each class is taken as the cluster center vector of that class, and there are a total of K cluster center vectors.

[0027] S52. Calculate u for each pixel in the cluster center. t With all mean vectors u k The Euclidean distance min(p) is given by pixel u. t If the value of u is closest to the mean vector of the p-th cluster center, then u is... t It belongs to the p-th cluster center;

[0028] min(p) = ||u k -u t ||

[0029] S53. Repeat S51-S52 until the cluster center vectors of adjacent rounds coincide.

[0030] S54. Display the clustering results.

[0031] Compared with existing technologies, this invention has the following advantages: 1) Compared with traditional methods using background subtraction for vehicle detection, it solves the problem of gaps in vehicle target extraction caused by the inability of the background construction to match the actual background when the environment is complex and varied and the color of the moving target is similar to the background color; 2) This invention uses radar-visual feature fusion to divide the scene into different categories, thereby extracting the moving vehicle target. Compared with traditional algorithms that rely solely on the grayscale values ​​of pixels for scene classification, it solves the problem of inaccurate separation of background and target when the grayscale values ​​of vehicle pixels are similar to those of background pixels; 3) This invention can achieve vehicle target detection using a single frame, which improves vehicle detection efficiency to a certain extent; 4) This invention utilizes information from different sensors for fusion and complementarity, which can comprehensively and multidimensionally describe the state information of each pixel, overcoming the one-sidedness of single-dimensional feature considerations and improving the accuracy of vehicle target extraction. Attached Figure Description

[0032] Figure 1 This is a flowchart of the rapid vehicle target detection method based on radar-visual fusion of the present invention;

[0033] Figure 2 These are the 25th and 50th frames captured by the camera in the real environment.

[0034] Figure 3 Images of the original point clouds of the target detected by the millimeter-wave radar corresponding to the camera in frames 25 and 50, after data preprocessing;

[0035] Figure 4 The images show the clustering results for frames 25 and 50 based on radar-visual fusion.

[0036] Figure 5 To compare the vehicle target detection results using the method of this invention with the vehicle detection results using Gaussian mixture background modeling for the 25th frame image;

[0037] Figure 6 To compare the vehicle target detection results using the method of this invention with the vehicle detection results using Gaussian mixture background modeling on the 50th frame image;

[0038] Figure 7 This is a comparison chart of the false negative rate and false positive rate of two different algorithms in the first 100 frames of images collected. Detailed Implementation

[0039] The embodiments and effects of the present invention will be described in further detail below with reference to the accompanying drawings.

[0040] like Figure 1As shown, this invention, from the perspective of radar-visual fusion, takes advantage of the high accuracy of millimeter-wave radar speed measurement, and achieves accurate separation of background and target by fusing radar data and optical data. Then, it extracts vehicle targets to achieve rapid detection of vehicle targets in a single frame.

[0041] S1. Obtain the distribution of the target point cloud velocity information detected by radar on the image.

[0042] S11. First, the raw point cloud information detected by radar is preprocessed to filter out unnecessary clutter and retain multiple radar reflection points of the target. Figure 3 The images are preprocessed versions of the original point clouds of targets detected by millimeter-wave radar in frames 25 and 50. Due to the sparsity and uncertainty of radar point clouds, the number of reflection points varies for different types of vehicle targets.

[0043] S12. Time Matching. Time matching between millimeter-wave radar and video mainly consists of two parts: alignment at the time start point and correspondence between data frames. During data acquisition, the start times of both the radar and video devices need to be marked and output as reference data for subsequent time start point alignment. After time start point alignment, the data frames of the two devices are matched according to their frame rates. The radar refresh rate is 20 frames / second, and the video frame rate is 60 frames / second. Since the video acquisition rate is greater than the radar acquisition rate and is an integer multiple thereof, video data is sampled every two frames at the timestamp of the radar data acquisition to achieve time frame matching.

[0044] S13, Spatial Matching. The radar detects the target's radial distance s and angle r.

[0045] (1) The target in the radar coordinate system x r y r The coordinates are represented as:

[0046] x r =s·sin(r),y r =s·cos(r);

[0047] (2) The target is transformed from the radar coordinate system to the world coordinate system x w y w z w The coordinates are represented as:

[0048] x w =x r ,y w =y r ,z w =0;

[0049] (3) The coordinate representation of the target uv in the pixel coordinate system after transformation from the world coordinate system is as follows:

[0050]

[0051] In the formula, R is the rotation coefficient matrix; T is the translation coefficient matrix; M1 is the camera's intrinsic parameter matrix, which can be obtained through Zhang Zhengyou's calibration method; M2 is the camera's extrinsic parameter matrix, which can be obtained through translation and rotation operations. From the following formula and data debugging, it can be seen that in the camera coordinate system x... c y c z c Below, since the radar and camera equipment are basically in the same position, the data is only offset by a slight angle. Here, ω is the rotation angle, and x' and y' are the offset amounts.

[0052] x c =x w cos(ω)-y w sin(ω)+x'

[0053] y c =x w cos(ω)+y w sin(ω)+y'

[0054] z c =z w

[0055] Through the above process, the preprocessed point cloud information from the radar can be displayed in the pixel coordinate system.

[0056] S2. Use the point cloud velocity information collected by radar to perform regional compensation, and assign corresponding velocity information to all pixels in the image.

[0057] The raw point cloud of the target vehicle detected by radar contains velocity information. This velocity information can be spatially compensated by assigning the velocity information of the center point to other pixels within an n*n window, centered on each point. For pixels in the background, such as streetlights and the ground, the velocity information can be compensated to 0, thus assigning the corresponding velocity information to all pixels in the image.

[0058] S3. Extract the RGB color component features and spatial location features of the image.

[0059] S31. In the RGB color space, the closer the points are, the more similar the colors they represent. By grouping close points in the RGB color space into one class and distant points into different classes, the resulting image segmentation shows regions with the same or similar colors grouped into the same class, while regions with significantly different colors are grouped into different classes. After processing the image, we can obtain the R, G, and B values ​​of each pixel.

[0060] S32. Images are composed of pixels, and pixel coordinates are the positions of pixels within an image. To determine pixel coordinates, we must first determine the image's coordinate system. Therefore, we establish a direct coordinate system uv, with the top-left corner of the image as the origin and pixels as the unit. The x-coordinate u and y-coordinate v of a pixel are its column number and row number in the image array, respectively. In this way, we can obtain the coordinate information of each pixel.

[0061] S4. Determine the number of cluster centers K and the K initial cluster center vectors.

[0062] The five dimensions of feature information—RGB component values, coordinate position, velocity information, etc.—are fused together to form a set of feature vectors. The feature vector of each pixel is represented as u. i =[x i ,y i ,v i ,R i G i B i The feature vector values ​​of each pixel are normalized to obtain... Then, we form a sample space {1, 2, 3...N} from all the pixels in the image. First, we take the first sample pixel u. i This serves as the center of the first class. Then, other sample pixels u in the sample space are considered. s The similarity of the feature vector convergence values ​​is determined according to the following formula:

[0063]

[0064]

[0065]

[0066] In the formula, For pixel u i Normalized elements; For pixel u s The normalized elements; m1, m2, and m3 are all thresholds that are greater than 0 and adjustable; These are the location distance bandwidth, velocity information bandwidth, and color distance bandwidth, respectively.

[0067] If the above formula is satisfied, then pixel u will be... s With pixel u i Merge them into one category; otherwise, merge pixel u. s This serves as a new class center. The remaining sample pixels are then calculated in this manner until all pixel data has been processed, thus dividing the entire scene into K categories.

[0068] S5. Use the K-means algorithm to classify the background and target.

[0069] S51. Take the mean vector u of all pixels at each cluster center. k As the cluster center vector; that is, each class has multiple pixels, and the mean vector of all pixels in each class is taken as the cluster center vector of that class, and there are a total of K cluster center vectors.

[0070] S52. Calculate u for each pixel in the cluster center. t With all mean vectors u k The Euclidean distance min(p) is given by pixel u. t If the value of u is closest to the mean vector of the p-th cluster center, then u is... t It belongs to the p-th cluster center;

[0071] min(p) = ||u k -u t ||

[0072] S53. Repeat S51-S52 until the cluster center vectors of adjacent rounds coincide.

[0073] S54. Assign the same value to pixels belonging to the same category from the above steps. This allows us to obtain the category of all pixels and display them. To make the effect easier to see, we use the same color to represent pixels of the same category. Figure 4 The same category is represented by the same color. It can be seen that grass, lane lines, and ground together constitute the background in this scene, while vehicle body and window together constitute the vehicle target.

[0074] S6. Use clustering results to extract vehicle targets.

[0075] Step S5 refines the scene into different categories, with the background further subdivided into grass, lane lines, and ground. These categories are then aggregated and represented by a single color to closely resemble the most realistic road background. Vehicle targets, consisting of vehicle bodies and windows, are extracted from the scene to achieve rapid vehicle target detection in a single frame.

[0076] like Figures 2 to 7 In the specific embodiment shown, Figure 2 Images from frames 25 and 50, captured in a real-world environment; Figure 3 The result of preprocessing the two frames of raw point cloud data detected by radar. Due to the sparsity and uncertainty of radar point clouds, the number of reflection points of different types of vehicle targets is different. Figure 4The image shows the clustering effect obtained through step S5 based on radar-video fusion. As can be seen from the image, the entire scene is divided into 5 categories, with the same category represented by the same color. The radar-video fusion effectively separates the target pixels from the background pixels, making it easier to extract vehicle targets in the future. Figure 5 The left image shows the vehicle target detection results using Gaussian mixture background modeling. It can be seen that because the background color and the vehicle body color are similar, areas with similar colors are considered background during background construction, resulting in some voids in the vehicle target area. Figure 5 The right side shows the vehicle target detection results using the method of this invention. It can be seen that the method of this invention can effectively distinguish between target pixels and background pixels, greatly reducing the hole phenomenon during vehicle target extraction. Similarly, Figure 6 The left image shows the vehicle target detection results using Gaussian mixture background modeling. It can be seen that numerous holes appear during vehicle target extraction, leading to missed detections. Figure 6 The right side shows the vehicle target detection result of the method of the present invention. It can be seen that the method of the present invention can effectively extract the vehicle target. Figure 7 This is a comparison chart of the false negative rate and false positive rate of two different algorithms in the first 100 frames of images collected.

[0077] This article calculates and compares the false detection rate R. FN And false detection rate R FP To evaluate the effectiveness of the proposed algorithm.

[0078]

[0079] In the formula, TP represents the number of correct target pixels; FP represents the number of falsely detected target pixels; TN represents the number of correct background pixels; and FN represents the number of falsely detected background pixels.

[0080] The accuracy of the algorithm was verified by analyzing the first 100 frames in a real-world environment. Data from both algorithms was plotted as a line graph for easy comparison; the comparison results are shown below. Figure 7 As shown, the vehicle detection method of the present invention has a lower average false negative rate and a lower average false positive rate than the target detection method of the Gaussian Mixture Model (GMM), which greatly reduces the void phenomenon generated during vehicle extraction and improves the vehicle detection accuracy.

[0081] The above are specific embodiments of the present invention, but the scope of protection of the present invention should not be limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A rapid vehicle target detection method based on radar-visual fusion, characterized in that, include: S1. By transforming coordinates, the radar point cloud is projected onto the camera image to obtain the distribution of the radar-detected point cloud velocity information on the image. S2. Use point cloud velocity information to perform region compensation, and assign velocity information to all pixels in the image. The specific process of step S2 is as follows: taking each point in the point cloud as the center and an n*n window as the neighborhood, the velocity information of the point is assigned to other pixels within the window; while for the pixels of streetlights and the ground, the velocity information is compensated to 0, so that all pixels in the image are assigned velocity information. S3. Extract the RGB color component features and pixel coordinate features of all pixels in the image; S4. Cluster the pixels based on RGB component values, coordinate positions, and velocity information to determine the number of cluster centers K and the initial cluster center vector for each cluster center; S5. The K-means algorithm is used to re-cluster the cluster centers, thereby achieving background and vehicle target classification; S6. Use clustering results to extract vehicle targets.

2. The rapid vehicle target detection method according to claim 1, characterized in that, The specific process of step S1 is as follows: S11. Preprocess the raw point cloud information detected by radar to filter out unnecessary clutter and retain multiple radar reflection points of the target. S12. Perform time matching on radar data and video data from the perspectives of time start point and data frame; S13. Transform the matched point cloud from the radar coordinate system to the pixel coordinate system and project it onto the image.

3. The rapid vehicle target detection method according to claim 1, characterized in that, The specific process of step S4 is as follows: Extract pixel u from the image. i And set it as the center of the first class, then apply it to the other pixels in the image. s The similarity of the feature vector convergence values ​​is determined according to the following formula: In the formula, Each pixel is u i The six elements of the normalized eigenvector; Each pixel is u s The six elements of the normalized eigenvector; m1; m2; m3 are all adjustable thresholds that are greater than 0; These are respectively location distance bandwidth, velocity information bandwidth, and color distance bandwidth; If the above formula is satisfied, then pixel u will be... s With pixel u i Merge them into one category; otherwise, merge pixel u. s As a new class center, continue to calculate the remaining sample pixels in this way until all pixel data has been processed, thus dividing the entire scene into K categories.

4. The rapid vehicle target detection method according to claim 1, characterized in that, The specific process of step S5 is as follows: S51. Take the mean vector u of all pixels at each cluster center. k As the cluster center vector; that is, each class has multiple pixels, and the mean vector of all pixels in each class is taken as the cluster center vector of that class, and there are a total of K cluster center vectors. S52. Calculate u for each pixel in the cluster center. t With all mean vectors u k The Euclidean distance min(p) is given by pixel u. t If the value of u is closest to the mean vector of the p-th cluster center, then u is... t It belongs to the p-th cluster center; min(p)=||u k -in t || S53. Repeat S51-S52 until the cluster center vectors of adjacent rounds coincide. S54. Display the clustering results.

Images