Front vehicle detection method based on deep learning ROI (Region of Interest) correlation thundersight fusion algorithm

Abstract

The invention aims to provide a novel thunder-vision fusion method, so as to solve the problem that thunder-vision fusion is relatively tedious in a traditional mathematical method, and further improve the safety of an automatic driving system. The invention provides a front vehicle detection method based on deep learning, and the method employs the data fusion of a millimeter wave radar and a camera, and achieves the vehicle detection (Deep-ROI, DR) through ROI correlation. Pictures shot under different conditions are cut into images with the same size to create a dep-fusion data set, a new method for manufacturing a radar data set is provided, multi-dimensional information of an effective target is projected to pixel points, and matrix format data with the same size as the images are generated. A backbone network of the deep learning network comprises two pipelines which are used for processing radar data and image data respectively, and two parameter association credibility (R parameter) and environment characteristics (E parameter) are added on the basis of a Resnet model for fusion. And the constructed deep learning network module is trained by using multiple frames of images containing different light and weather conditions and radar data.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection by using a radar-vision fusion, and in particular to a vehicle detection algorithm that utilizes a deep learning ROI detection method and associates a radar-vision fusion algorithm. Background Art

[0002] While motor vehicles have entered thousands of households, bringing great convenience to people, they also bring various traffic safety hazards. In order to protect people from hidden dangers and reduce the incidence of accidents, an efficient and accurate advanced auxiliary automatic driving system is needed, especially a radar data fusion algorithm based on millimeter-wave radar and camera.

[0003] The existing radar-vision fusion algorithm mainly emphasizes the complementarity of the detection results of multiple sensors, including detection intersection-over-union ratio and global nearest neighbor data association, ROI information association, etc. This method requires strict and complex mathematical calculations, which makes it inconvenient to directly apply to the driving system, and has poor real-time performance and accuracy in vehicle target detection. Summary of the invention

[0004] The technical problems to be solved by the present invention are:

[0005] In order to solve the problems of complex mathematical calculations of existing radar-vision fusion algorithms, which are not convenient for direct application in driving systems, and low real-time performance and accuracy when performing vehicle target recognition.

[0006] The present invention adopts the following technical solutions to solve the above technical problems:

[0007] The present invention provides a method for detecting a vehicle ahead based on deep learning, which utilizes the data fusion of millimeter wave radar and camera to realize vehicle detection through ROI (region of interest) association. The method includes the following steps:

[0008] Step 1: Data collection: Use millimeter-wave radar and camera to obtain raw data, where radar data contains the speed, direction, and distance information of the target, and image data contains visual information in the road scene;

[0009] Step 2: Data preprocessing: Filter and cluster the radar data, select and encode the valid targets into a matrix form, and then project them into the image coordinate system; the radar data is cropped and downsampled to align with the image data;

[0010] Step 3: Deep learning network construction: The deep learning model is divided into a backbone network and a head network. The backbone network processes radar and image data respectively through a dual-pipeline structure, uses residual connections to learn the deep features of the target, and connects each channel through associated credibility (R parameter) and environmental features (E parameter);

[0011] Step 4: Model training: Use multiple frames of images and radar data with different lighting and weather conditions for training to improve the robustness of the model in complex environments;

[0012] Step 5: Object detection: Input the processed data into the trained Deep-ROI deep learning network to detect and classify the vehicle ahead.

[0013] The data preprocessing process also includes the following steps:

[0014] Step 1: Decode the radar raw data and divide it into different types of data frames, which contain physical values ​​and raw values;

[0015] Step 2: Use the constant false alarm rate (CFAR) algorithm to screen possible collision targets and eliminate false targets and stationary targets;

[0016] Step 3: Perform coordinate transformation on the radar data after preliminary screening. The calculation formula is as follows:

[0017]

[0018] Step 4: Crop the three-dimensional core features of the data (probability, distance, speed) according to the image size to obtain 4128×2752×3 matrix format data.

[0019] Furthermore, the backbone network of the deep learning network includes two pipelines for processing radar data and image data respectively, and fusing them by associating credibility (R parameter) and environmental characteristics (E parameter).

[0020] Furthermore, the tail output of the backbone network is aligned with the size of the original input data through a downsampled convolutional layer, and a nonlinear activation function is added after the linear layer.

[0021] Furthermore, multi-frame training data includes extreme environments such as light changes and saturation changes, thereby enhancing the generalization ability of the model.

[0022] Furthermore, the deep-fusion dataset collects images under different combinations of aperture size, sensitivity and exposure time, and uses different types of cameras to shoot.

[0023] Compared with the existing technology, the technical solution provided in this application has at least the following technical effects or advantages:

[0024] First of all, this solution includes data sets shot under different light, aperture sizes, sensitivities, and camera types, as well as preprocessing methods for radar and image data. It can be widely used in the training of road scene target recognition, and can enable the model to provide high-quality results under adverse conditions, with good robustness and generalization performance.

[0025] Secondly, this scheme adopts a deep ROI (Deep_ROI) detection method, which can show better performance than traditional mathematical methods when the detection results of different sensors vary greatly, and show good recognition results at any time.

[0026] Finally, this scheme uses a more direct way to learn the correlation between millimeter-wave radar and camera data in road scenes, which does not require the calculation of strict and complex mathematical relationships and can be directly applied to driving systems.

[0027] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0029] Figure 1 A structural block diagram of a radar-visual fusion algorithm for ROI region association based on deep learning provided in an embodiment of the present application;

[0030] Figure 2 A flowchart of the data preprocessing process provided in the embodiment of the present application;

[0031] Figure 3 A block diagram of the deep learning network structure provided in the embodiment of the present application;

[0032] Explanation of the accompanying drawings: data acquisition module 11, data preprocessing module 12, deep learning network construction module 13, model training module 14, target detection module 15. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0034] The present invention discloses a method for radar-visual fusion based on ROI region association based on deep learning. The flow chart of the method is as follows: Figure 1 As shown, the following steps are included:

[0035] Step 1: construct a data acquisition module 11, and use a millimeter wave radar and a camera to obtain raw data, wherein the radar data contains the speed, direction, and distance information of the target, and the image data contains the visual information in the road scene;

[0036] Radar-visual fusion refers to the effective fusion of data obtained by millimeter-wave radar and camera sensors to improve the system's ability to perceive targets. Millimeter-wave radar has strong penetration and anti-interference capabilities, and can have high recognition capabilities, high robustness, speed and distance measurement capabilities in all weather and in bad weather; camera sensors have rich semantic information, target features and recognition capabilities, and their data are highly compatible and complementary with millimeter-wave radar. The target road is collected by millimeter-wave radar and camera, and the subsequent radar-visual fusion information is obtained, providing raw data for subsequent modules;

[0037] Step 2: Input the raw data into the data preprocessing module 12 to obtain processed radar data and image data with three-dimensional core features (probability, distance, speed);

[0038] like Figure 1 As shown, a data preprocessing module 12 is constructed to filter and cluster the radar data, screen and encode the valid targets into a matrix form, and then project them into the image coordinate system; the radar data is cropped and downsampled to align with the image data;

[0039] Construct a data preprocessing module 12. First, decode the radar raw data and divide it into different types of data frames, which contain physical values ​​and raw values. Secondly, use the constant false alarm algorithm (CFAR) to screen possible collision targets and eliminate false targets and stationary targets. Provide the following formula to transform the coordinates of the radar data after preliminary screening. Finally, crop the three-dimensional core features (probability, distance, speed) of the data according to the image size to obtain 4128×2752×3 matrix format data.

[0040] It lays the foundation for subsequent models;

[0041]

[0042] Step 3: Construct a deep learning network module 13, such as Figure 3As shown in the figure, the deep learning model is divided into a backbone network and a head network. The backbone network processes radar and image data respectively through a dual-pipeline structure, uses residual connections to learn the deep features of the target, and connects each channel through associated credibility (R parameter) and environmental features (E parameter). The tail output of the backbone network is aligned with the size of the original input data through a downsampled convolutional layer, and a nonlinear activation function is added after connecting the linear layer.

[0043] Step 4: construct a model training module 14, and train the constructed deep learning network module 13 using multiple frames of images and radar data containing different light and weather conditions, including using images collected under different aperture sizes, sensitivities and exposure time combinations, and using different types of cameras to obtain data to improve the robustness of the model in complex environments and enhance the generalization ability of the model;

[0044] Step 5: Construct a target detection module 15, and input the data processed by the data preprocessing module 12 into the model training module 14, i.e., the trained Deep-ROI deep learning network, so as to realize target detection and classification.

[0045] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0046] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A front vehicle detection method based on deep learning, which uses data fusion of millimeter wave radar and camera, Vehicle detection is achieved through ROI (Region of Interest) association, which is characterized by: The method comprises the following steps: (1) Data acquisition: Use millimeter-wave radar and cameras to obtain raw data. Radar data contains the speed, direction, and distance information of the target, and image data contains visual information in the road scene. (2) Data preprocessing: Filter and cluster the radar data, select and encode the valid targets into a matrix form, and then project them into the image coordinate system; the radar data is cropped and downsampled to align with the image data; (3) Deep learning network construction: The deep learning model is divided into a backbone network and a head network. The backbone network processes radar and image data respectively through a dual-pipeline structure, uses residual connections to learn the deep features of the target, and connects each channel through associated credibility (R parameter) and environmental features (E parameter); (4) Model training: Use multiple frames of images and radar data with different lighting and weather conditions for training to improve the robustness of the model in complex environments; (5) Object detection: The processed data is input into the trained Deep-ROI deep learning network to detect and classify the vehicle ahead.

2. The method for detecting a vehicle ahead based on deep learning according to claim 1, characterized in that: The data preprocessing process includes the following steps: (1) Decode the radar raw data and divide it into different types of data frames, which contain physical values ​​and raw values; (2) Use the constant false alarm rate (CFAR) algorithm to screen possible collision targets and eliminate false targets and stationary targets; (3) The radar data after preliminary screening is transformed into coordinates according to the following formula: (4) The three-dimensional core features of the data (probability, distance, speed) are cropped according to the image size to obtain a 4128×2752×3 matrix format data 3. The method for detecting a vehicle ahead based on deep learning according to claim 1 or 2, characterized in that: The backbone network of the deep learning network includes two pipelines for processing radar data and image data respectively, and fusing them by associating credibility (R parameter) and environmental characteristics (E parameter).

4. The method for detecting a vehicle ahead based on deep learning according to claim 3, characterized in that: The tail output of the backbone network is aligned with the size of the original input data through a downsampled convolutional layer, and a nonlinear activation function is added after connecting the linear layer.

5. The method for detecting a vehicle ahead based on deep learning according to any one of claims 1 to 4, characterized in that: The multi-frame training data includes extreme environments such as light changes and saturation changes to enhance the generalization ability of the model.

6. The method for detecting a vehicle ahead based on deep learning according to any one of claims 1 to 5, characterized in that: The deep-fusion dataset collects images under different combinations of aperture size, sensitivity and exposure time, and uses different types of cameras to shoot.

Images