Identification method based on 4D millimeter wave radar fusion camera

Abstract

The invention belongs to the field of automobiles, and particularly relates to an identification method based on a 4D millimeter wave radar fusion camera. The 4D millimeter-wave radar can detect target distance, speed, azimuth angle and pitch angle (space coordinate and speed) information, realizes environment detection through reflection point cloud of emitted radar waves on an object, provides distance, height and other information of target objects around a vehicle, and meanwhile, calculates the distance by fusing an image detection function of a camera. And the vehicle sensing capability is greatly improved.

Description

Technical Field

[0001] The present invention belongs to the field of automobiles, and in particular relates to a recognition method based on a 4D millimeter wave radar fusion camera. Background Art

[0002] With the popularity of AVP functions, there will be toll poles before entering public parking lots. Vehicles need to be able to accurately identify the toll poles before entering the garage to complete parking. At present, 3D millimeter-wave radars can only obtain the distance, speed, and azimuth information of the target, and cannot reflect road traffic information in a real 3D space. However, 4D millimeter-wave radars can obtain the pitch angle (spatial coordinates and speed) of the target in addition to 3D information through a specific antenna array design. They operate well in different weather conditions and are particularly suitable for tasks of environmental perception in ADAS systems. At the same time, when 4D millimeter-wave radars work in conjunction with cameras, the system can provide a more realistic obstacle detection function. In future on-vehicle environmental monitoring systems, 4D millimeter-wave radars have the potential to replace lidars, greatly reducing the hardware cost of the system, and at the same time can make up for the shortcomings of other sensors such as cameras and ultrasonic radars in environmental monitoring during ADAS assisted driving. Summary of the invention

[0003] The present invention relates to a method for identifying a toll pole based on a 4D millimeter wave radar fused with a camera during parking. The 4D millimeter wave radar has the function of detecting target distance, speed, azimuth and pitch angle (spatial coordinates and speed), and realizes environmental detection through the reflection point cloud of the emitted radar wave on the object, providing information such as the distance and height of the target objects around the vehicle, and at the same time integrating the camera image detection function to calculate the distance, which will greatly improve the vehicle's perception ability. The specific technical solution is as follows:

[0004] A method for identifying a toll pole based on a 4D millimeter wave radar fusion camera during parking comprises the following steps:

[0005] Step 1: 4D millimeter wave radar data processing:

[0006] Step 1 1: Point cloud data acquisition, first obtain the point cloud sample data, train and classify the target.

[0007] Step 12: Point cloud data preprocessing, which completes the target point cloud segmentation and denoising, separates the ground point cloud and the surrounding point cloud parts, and the segmented point cloud is sent to the feature extraction module; reduces the interference to target recognition, echo modeling, distance-Doppler-time joint features, and extracts sample features for the next step.

[0008] Step 13: Extract feature of point cloud samples, extract multiple features of target attributes, such as toll poles and pillars, from the preprocessed point cloud samples. The objective basis of the features is the size invariance of the target during movement and the statistical characteristics of the target point cloud distribution.

[0009] Step 14: Target classification and recognition. Data preprocessing, feature extraction and feature transformation are performed to form several features required for target recognition, thereby determining the target attributes.

[0010] Step 2: Image recognition processing:

[0011] Step 21: Image acquisition: Use the forward-looking system to collect the target image, mark the target in the image, and freeze the weights of the backbone feature extraction network based on the weight file of the deep convolutional neural network target detection model.

[0012] Step 22: Post-processing of regional images. Use the weight file obtained in the previous step to detect targets in the images collected in real time by the forward-looking system and extract image information in the region. The front layers of the network extract general features of the image, while the back layers extract features related to specific tasks.

[0013] Step 23: Target area division. Filter, binarize, and open and close the extracted area to obtain the refined target area information. And filter out those that do not meet the target first, so as to divide the target area.

[0014] Step 24: Camera recognition results. Then, through image feature matching, the type of the detected target is matched from the sample feature database, and then the distance and relative position of the target to the vehicle are estimated based on the size of the target in the image;

[0015] Step 3: Time synchronization: The 4D millimeter-wave radar samples normally according to its own sampling cycle, and triggers the camera to sample each time the radar starts sampling a frame, and determines that the two data frames have a corresponding relationship.

[0016] Step 4: Target matching and output: Project the 4D millimeter wave radar point onto the image. With the radar point as the center, project the toll pole onto the image according to the projection principle to obtain the radar target frame. Calculate the ratio of the intersection and union of the radar recognition target frame and the camera recognition target frame pixels, and finally output the target object recognition.

[0017] Furthermore, the point cloud sample data of the 4D millimeter wave radar is obtained through the actual flight time, and a large amount of point cloud data is obtained by identifying the toll pole. The point cloud data is measurable, and the three-dimensional coordinate (x, y, z) information, shape, surface normal vector, etc. of the target can be directly obtained through the point cloud. If the acquisition performance is sufficient, the surface area and volume of the target can even be obtained.

[0018] The distance between two points and the three-dimensional (x, y, z) direction parameters are calculated using the following formula.

[0019]

[0020]

[0021] Where c is the speed of light in m / sec; kr is the distance index, FSAMP is the sampling frequency in Hz, S is the chirp slope in Hz / sec, NFFT is the 1D FFT size, W is the length of R projected onto the X&Y plane, Wx is the X-axis component of W, and Wy is the Y-axis component of W. is the angle between W and R, and θ is the angle between W and the Y axis.

[0022] The 4D millimeter wave radar of the present invention can accurately measure the straight-line distance between two points of the target, three-dimensional (x, y, z) spatial dimension information and speed information. The advantage of 4D millimeter wave radar height measurement is that it can accurately detect the target height information. At the same time, it integrates visual information to improve the confidence of detection. The camera has insufficient performance in bad weather environments such as night, rain and fog, and the strong environmental adaptability of 4D millimeter wave radar can make up for the shortcomings of the camera.

[0023] During the AVP cruise, there will be toll booth scenes. The perception solution of 4D millimeter-wave radar fusion camera can increase the parking usage scenario. Non-line-of-sight perception can be achieved to a certain extent, and the position and speed of the target blocked by the front vehicle can be detected. Since the 4D millimeter-wave radar can measure the vertical height of the target, it makes up for the shortcoming of the existing 3D millimeter-wave radar that cannot measure height. The perception solution of 4D millimeter-wave radar fusion camera is more cost-effective than the lidar solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Coordinate geometry schematic diagram of the present invention.

[0025] Figure 2 Schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The automatic target recognition process based on 4D millimeter wave radar fusion camera mainly includes the following steps:

[0028] Step 1: 4D millimeter wave radar data processing:

[0029] Step 11: Get point cloud data. First, get the point cloud sample data, train and classify the target. The point cloud sample data of 4D millimeter wave radar is mainly obtained through the actual flight time, and a large amount of point cloud data is obtained by identifying the toll pole. Point cloud data is measurable, and the three-dimensional coordinate (x, y, z) information, shape, surface normal vector, etc. of the target can be directly obtained through the point cloud. If the acquisition performance is sufficient, the surface area and volume of the target can even be obtained. Figure 1 The coordinate geometry and formula 1 shown can calculate the distance between two points and the three-dimensional (x, y, z) direction parameters.

[0030]

[0031] Where c is the speed of light (m / sec), kr is the distance exponent, and F SAMP is the sampling frequency (Hz), S is the chirp slope (Hz / sec), N FFT is the 1D FFT size, W is the length of R projected onto the X&Y plane, Wx is the X-axis component of W, and Wy is the Y-axis component of W. is the angle between W and R, and θ is the angle between W and the Y axis.

[0032] Step 12: Point cloud data preprocessing. This part is to complete the target point cloud segmentation, denoising, etc., separate the ground point cloud and the surrounding point cloud, and the segmented point cloud is sent to the feature extraction module. Reduce the interference to target recognition, echo modeling, range-Doppler-time joint features, and extract sample features for the next step.

[0033] Step 13: Extract feature of point cloud samples. Extract multiple features of target attributes, such as toll poles and pillars, from the preprocessed point cloud samples. The objective basis of the features is the size invariance of the target during movement and the statistical characteristics of the target point cloud distribution.

[0034] Step 14: Target classification and recognition. Data preprocessing, feature extraction and feature transformation are performed to form several features required for target recognition, thereby determining the target attributes.

[0035] Step 2: Image recognition processing:

[0036] Step 21: Image acquisition. The target image is acquired using the forward-looking system, the target is annotated in the image, and the weights of the backbone feature extraction network are frozen based on the weight file of the deep convolutional neural network target detection model.

[0037] Step 22: Post-processing of regional images. Use the weight file obtained in the previous step to detect targets in the images collected in real time by the forward-looking system and extract image information in the region. The front layers of the network extract general features of the image, while the back layers extract features related to specific tasks.

[0038] Step 23: Target area division. Filter, binarize, and open and close the extracted area to obtain the refined target area information. And filter out those that do not meet the target first, so as to divide the target area.

[0039] Step 24: Camera recognition results. Then, through image feature matching, the type of the detected target is matched from the sample feature database, and then the distance and relative position of the target to the vehicle are estimated based on the size of the target in the image;

[0040] Step 3: Time synchronization: The 4D millimeter-wave radar samples normally according to its own sampling cycle, and triggers the camera to sample each time the radar starts sampling a frame, and determines that the two data frames have a corresponding relationship.

[0041] Step 4: Target matching and output: Project the 4D millimeter wave radar point onto the image. With the radar point as the center, project the toll pole onto the image according to the projection principle to obtain the radar target frame. Calculate the ratio of the intersection and union of the radar recognition target frame and the camera recognition target frame pixels, and finally output the target object recognition. The above fusion algorithm process is as follows Figure 2 shown.

[0042] The above describes in detail the preferred implementation of this patent, but this patent is not limited to the above implementation. Various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of this patent.

Claims

1. A recognition method based on 4D millimeter wave radar fusion camera, characterized in that: The steps include: Step 1: 4D millimeter wave radar data processing: Step 11: Point cloud data acquisition: first obtain point cloud sample data, train and classify the target; Step 12: Point cloud data preprocessing, which completes the target point cloud segmentation, denoising, separation of ground point cloud and surrounding point cloud parts, and the segmented point cloud is sent to the feature extraction module; Reduce interference with target identification, echo modeling, range-Doppler-time joint features, and extract sample features for the next step; Step 13: Extracting point cloud sample features, extracting multiple features of target attributes from the preprocessed point cloud sample data; Step 14: Target classification and recognition: data preprocessing, feature extraction and feature transformation to form several features required for target recognition, thereby determining target attributes; Step 2: Image recognition processing: Step 21: Image acquisition: Use the forward-looking system to collect the target image, mark the target in the image, and freeze the weight of the backbone feature extraction network based on the weight file of the deep convolutional neural network target detection model; Step 22: Post-processing of regional images: using the weight file obtained in the previous step to detect targets in the images collected in real time by the forward-looking system, and extracting image information in the region; Step 23: Target area division: filtering, binarization, opening and closing operations are performed on the extracted area to obtain the refined target area information, and those that do not meet the target are filtered out first, thereby dividing the target area; Step 24: The camera recognizes the result, and then matches the type of the detected target from the sample feature database through image feature matching, and then estimates the distance and relative position information of the target to the vehicle based on the size of the target in the image; Step 3: Time synchronization: The 4D millimeter-wave radar samples normally according to its own sampling cycle, triggering the camera to sample each time the radar starts sampling a frame, and determining that the two data frames have a corresponding relationship; Step 4: Target matching and output: Project the 4D millimeter-wave radar point onto the image. With the radar point as the center, project the toll pole onto the image according to the projection principle to obtain the radar target frame. Calculate the ratio of the intersection and union of the radar recognition target frame and the camera recognition target frame pixels, and finally output the target object recognition.

2. The recognition method based on 4D millimeter wave radar fusion camera according to claim 1 is characterized in that: The point cloud sample data of 4D millimeter wave radar is obtained through actual flight time, and a large amount of point cloud data is obtained by identifying toll poles. Point cloud data is measurable, and the three-dimensional coordinate (x, y, z) information, shape, surface normal vector, etc. of the target can be directly obtained through the point cloud. If the acquisition performance is sufficient, the surface area and volume of the target can even be obtained. The distance between two points and the three-dimensional (x, y, z) direction parameters can be calculated by the following formula: Where c is the speed of light in m / sec; kr is the distance index, FSAMP is the sampling frequency in Hz, S is the chirp slope in Hz / sec, NFFT is the 1D FFT size, W is the length of R projected onto the X&Y plane, Wx is the X-axis component of W, and Wy is the Y-axis component of W. is the angle between W and R, and θ is the angle between W and the Y axis.

3. The recognition method based on 4D millimeter wave radar fusion camera according to claim 1, characterized in that: In Step 13, the feature is a toll pole or a pillar, and the objective basis of the feature is the size invariance of the target during movement and the statistical characteristics of the target point cloud distribution.

4. The recognition method based on 4D millimeter wave radar fusion camera according to claim 1 is characterized in that: In Step 22, the front layers of the network extract general features of the image, and the back layers extract features related to specific tasks.

Images