A method and system for vehicle-mounted millimeter wave radar target detection

Abstract

The application provides a millimeter wave radar target detection method and system applied to a vehicle, relates to the technical field of millimeter wave radar target detection, and establishes a target detection model, performs self-supervised learning pre-training and supervised learning training on the target detection model based on a radar heat map obtained according to a target echo signal, and detects the category and position of a target according to the trained target detection model. Compared with the existing millimeter wave radar target detection technology, the application introduces a self-supervised learning method, uses output data obtained through self-supervised learning as input of supervised learning, still obtains higher detection effect under the support of less data, and improves the accuracy of target detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of millimeter wave radar target detection, in particular to a millimeter wave radar target detection method and system applied to a vehicle. BACKGROUND

[0002] The millimeter wave radar target detection task is an indispensable part of automatic driving technology, which can realize stable detection task in harsh environment.

[0003] However, the labeling work of the radar data set is difficult. Due to the ambiguous semantic information of the radar image, it is often difficult to achieve accurate labeling work by manual labeling. Although the automatic labeling by camera and radar combined semi-supervised method can solve this problem, there will still be a large number of mislabeling and missing labeling problems, which will lead to poor accuracy of the millimeter wave radar target detection model trained based on the training samples labeled.

[0004] Therefore, it is urgent to provide a millimeter wave radar target detection method based on self-supervised learning, which can still achieve higher detection effect with less data support, so as to detect the target more accurately. SUMMARY

[0005] The purpose of the present application is to provide a millimeter wave radar target detection method and system applied to a vehicle, which can still achieve higher detection effect with less data support by self-supervised learning of the target detection model, and improve the accuracy of target detection.

[0006] To achieve the above purpose, the present application provides the following scheme:

[0007] A millimeter wave radar target detection method applied to a vehicle, the target detection method comprising:

[0008] obtaining a radar heat map;

[0009] establishing a target detection model;

[0010] using a self-supervised learning pre-training method and a supervised learning training method to train the target detection model by taking the radar heat map as the input of the target detection model, and obtaining a trained target detection model;

[0011] detecting the category and position of the target based on the trained target detection model.

[0012] A millimeter wave radar target detection system applied to a vehicle, the target detection system comprising:

[0013] a radar heat map acquisition module for acquiring a radar heat map;

[0014] A model establishing module is configured to establish a target detection model.

[0015] A first training module is configured to take the radar heat map as an input of the target detection model, and train the target detection model by using a self-supervised learning pre-training method and a supervised learning training method to obtain a trained target detection model.

[0016] A detection module is configured to detect the category and position of a target based on the trained target detection model.

[0017] According to the embodiments of the present application, the following technical effects are provided.

[0018] The present application provides a millimeter wave radar target detection method and system applied to a vehicle, which establishes a target detection model, pre-trains and trains the target detection model based on a radar heat map obtained from a target echo signal by using a self-supervised learning method, and detects the category and position of a target based on the trained target detection model. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0020] Figure 1 A method flowchart of the millimeter wave radar target detection method applied to a vehicle provided in Embodiment 1 of the present application is shown.

[0021] Figure 2 A process diagram of obtaining a radar heat map by a vehicle radar provided in Embodiment 1 of the present application is shown.

[0022] Figure 3 A process diagram of outputting target category and position information by a target detection network provided in Embodiment 1 of the present application is shown.

[0023] Figure 4 A structure diagram of a pre-training network provided in Embodiment 1 of the present application is shown.

[0024] Figure 5 A system block diagram of the millimeter wave radar target detection system applied to a vehicle provided in Embodiment 2 of the present application is shown.

[0025] Figure 6 The structural schematic diagram of the vehicle-mounted millimeter wave radar target detection device provided in Embodiment 3 of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0027] The purpose of the present application is to provide a vehicle-mounted millimeter wave radar target detection method and system, which can improve the accuracy of target detection by performing self-supervised learning on the target detection model and achieving higher detection effect with less data support.

[0028] In order to make the above-mentioned purposes, characteristics and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0029] Embodiment 1:

[0030] Based on the problem that a large number of mislabeling and missing labeling still occur in the existing radar target detection method using the camera and radar combined semi-supervised manner, thereby affecting the accuracy of target detection, referring to Figure 1 The present embodiment provides a vehicle-mounted millimeter wave radar target detection method, which comprises the following steps:

[0031] S1: obtaining a radar heat map, specifically comprising:

[0032] obtaining the target echo signal;

[0033] performing fast Fourier transform on the target echo signal to obtain distance dimension information;

[0034] performing low-pass filtering processing on the target echo signal containing the distance dimension information to obtain a smoothed radar signal;

[0035] performing fast Fourier transform on the smoothed radar signal to obtain Doppler dimension information and angle dimension information, respectively;

[0036] obtaining a distance-Doppler heat map and a distance-angle heat map according to the distance dimension information, the Doppler dimension information and the angle dimension information, respectively, wherein the radar heat map comprises the distance-Doppler heat map and the distance-angle heat map.

[0037] As an optional implementation, a vehicle-mounted 77GHz FMCW (Frequency Modulated Continuous Wave Radar) is selected in this embodiment to detect the target echo signal.

[0038] Since the radar heat map only contains two-dimensional information of the object, i.e., distance-angle or distance-Doppler information, the information between multiple frames of radar heat maps is important for the target detection task. Although the millimeter wave radar point cloud information has more information, the point cloud is usually sparse and the model calculation is large, but the target detection method of the millimeter wave radar heat map has a low calculation amount. Therefore, the method of using multiple frames of millimeter wave radar heat maps for target detection has certain advantages and efficiency. For this purpose, the target echo signal is processed into a radar heat map by the vehicle-mounted radar, as shown in Figure 2 , which includes first performing FFT (Fast Fourier Transform) on the target echo signal to obtain distance dimension information; then performing LPF (Low-pass filter) on the obtained signal, and twice FFT to obtain Doppler and angle dimension information; finally, the RA (Range-Angle) and RD (Range, Doppler) two signals are displayed by using the heat map.

[0039] S2: establishing a target detection model; the target detection model includes a pre-training model and a target detection network;

[0040] S3: taking the radar heat map as the input of the target detection model, using a self-supervised learning pre-training method and a supervised learning training method to train the target detection model to obtain a trained target detection model, specifically including:

[0041] taking the radar heat map as the input of the pre-training model, using the MIM self-supervised learning method to train the pre-training model, outputting the pre-training weight and the heat map containing the target category and position information;

[0042] taking the heat map as the input of the target detection network, taking the pre-training weight as the training weight of the target detection network, and using supervised learning to train the target detection network.

[0043] As shown in Figure 3 , the pre-trained model after training loads its pre-training weight through a computer, and then uses supervised learning for the target detection network, i.e., supervised training using labeled data. The labeled data refers to the heat map containing the target category and position.

[0044] S4: detecting the target based on the trained target detection model, the target detection model after training can output the category and position information of the target more accurately, and in this way, the millimeter wave radar target detection technology can be realized.

[0045] The embodiment introduces a self-supervised learning method, uses the output data obtained through self-supervised learning as the input of supervised learning, and still obtains higher detection effect under less data support, thereby improving the accuracy of target detection.

[0046] Considering that the existing millimeter wave radar target detection algorithm uses 3D convolution as the main body for feature extraction, this method can effectively extract local features, but cannot handle long-distance dependencies. In order to realize the millimeter wave radar target detection method which has the ability to handle long-distance dependencies in addition to the ability to extract local features, so as to more accurately identify targets and improve the detection effect, the embodiment designs a masking method in the time and space dimensions and a pre-network, and uses the MIM self-supervised pre-training method as the main body, uses the MIM (Mask Image Modeling) method to pre-train the obtained RA and RD images by using an industrial computer, so as to obtain more rich prior feature information.

[0047] In this regard, as another optional implementation, the radar heat map is used as the input of the pre-training model, the pre-training model is trained by using the MIM self-supervised learning method, and specifically includes:

[0048] (1) performing a masking operation on the radar heat map to obtain a masked radar heat map, and specifically, performing a masking operation on the radar heat map in the spatial and temporal dimensions, that is, masking the same region in the internal radar heat map in the spatial dimension, and randomly selecting several frames of radar heat map to mask the whole heat map in the temporal dimension. The original radar heat map is used as the true value of pre-training, and the pre-training process realizes the image reconstruction proxy task.

[0049] (2) constructing a pre-training network based on the combination of the 3D convolution and the Transformer, and the pre-training network adopts an encoder-decoder structure, as shown in Figure 4 The encoder structure of the pre-training network includes an embedding feature module, a first feature extraction module, a first down-sampling module, a second feature extraction module, a second down-sampling module and a third feature extraction module connected in sequence.

[0050] The embedding feature module is used to divide the masked radar heat map into continuous three-dimensional feature blocks.

[0051] The first feature extraction module is configured to perform feature extraction on the three-dimensional feature block from local and global aspects respectively to obtain first scale coding features;

[0052] The first downsampling module is configured to perform downsampling operation on the first scale coding features to obtain first downsampled coding features;

[0053] The second feature extraction module is configured to perform feature extraction on the first downsampled coding features from local and global aspects respectively to obtain second scale coding features;

[0054] The second downsampling module is configured to perform downsampling operation on the second scale coding features to obtain second downsampled coding features;

[0055] The third feature extraction module is configured to perform feature extraction on the second downsampled coding features from local and global aspects respectively to obtain third scale coding features.

[0056] The embedding feature module mainly includes a 3D convolution layer for pre-processing the input image, dividing the masked radar heat map into continuous three-dimensional feature blocks; then a 3D convolution and a Transformer constitute a feature extraction module, which extracts features from local and global aspects respectively, making up for their respective shortcomings; finally, a 3D convolution is used to perform downsampling operation on the feature map to obtain feature information of different scales.

[0057] The decoder structure of the pre-training network comprises a first upsampling module, a first decoding module, a second upsampling module, a second decoding module and a detection head module connected in sequence;

[0058] The input of the first upsampling module is connected to the output of the third feature extraction module;

[0059] The first upsampling module is configured to perform upsampling operation on the third scale coding features to obtain first upsampled decoding features;

[0060] The first decoding module is configured to decode the first upsampled decoding features and the second scale coding features to obtain second scale decoding features;

[0061] The second upsampling module is configured to perform upsampling operation on the second scale decoding features to obtain second upsampled decoding features;

[0062] The second decoding module is configured to decode the second upsampled decoding features and the first scale coding features to obtain first scale decoding features;

[0063] The detection head module is configured to implement an image reconstruction proxy task on the first scale decoding features;

[0064] The first decoding module and the second decoding module each include a linear layer and a Transformer layer; an output of the linear layer is connected to an input of the Transformer layer.

[0065] The upsampling module uses 3D transpose convolution for upsampling, the decoding module composed of the linear layer and the Transformer decodes the encoded features, and the detection head module is mainly responsible for implementing different proxy tasks. In the pre-training process, the detection head uses a linear layer to implement the image reconstruction proxy task.

[0066] (3) The masked radar heat map is used as the input of the pre-training network, and the pre-training network is trained using a self-supervised learning method.

[0067] A large number of radar heat maps (RA and RD) are used for pre-training, and the pre-training weights are saved.

[0068] It should be noted that the pre-training network in the pre-training model mentioned in the embodiment and the target detection network are the same or two networks with the same structure.

[0069] The MIM-based self-supervised method designed in this embodiment can effectively improve the dependence of millimeter wave radar target detection on the data set, so that good detection effect can still be obtained with less data support. The MIM self-supervised method masks in time and space angles, so that the model can obtain more prior feature information, improve the learning of prior features of the model, and further improve the accuracy.

[0070] The pre-training model is constructed by using the Transformer combined with the 3D convolution, which improves the global and local modeling capabilities of the network and greatly improves the accuracy of the model.

[0071] Embodiment 2

[0072] Referring to Figure 5 , the embodiment provides a vehicle-mounted millimeter wave radar target detection system, which comprises:

[0073] A radar heat map acquisition module M1 is configured to acquire a radar heat map.

[0074] A model establishment module M2 is configured to establish a target detection model.

[0075] A first training module M3 is configured to use the radar heat map as the input of the target detection model, and train the target detection model using a self-supervised learning pre-training method and a supervised learning training method to obtain a trained target detection model.

[0076] A detection module M4 is configured to detect the category and position of the target based on the trained target detection model.

[0077] Embodiment 3

[0078] Referring to Figure 6 The embodiment provides a vehicle-mounted millimeter wave radar target detection device, which comprises a vehicle-mounted radar, an industrial computer and a target detection model. The vehicle-mounted radar is configured to detect a target echo signal and output a radar heat map after signal processing. The industrial computer is configured to obtain pre-training weights by performing MIM pre-training on the radar heat map, load the pre-training weights in subsequent supervised training, and then deploy the trained model on the industrial computer to realize millimeter wave radar target detection.

[0079] In the specification, each embodiment focuses on the difference from other embodiments, and the same or similar parts between embodiments can be understood by referring to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be understood by referring to the method part.

[0080] The principles and implementation manners of the present application are described by using specific examples in the present application. The above embodiment is only used to help understand the method of the present application and its core idea. Meanwhile, for the general skilled in the art, the specific implementation manners and application ranges can be changed according to the idea of the present application. In conclusion, the content of the specification should not be understood as the limitation of the present application.

Claims

1. A method for vehicle-mounted millimeter wave radar target detection, characterized in that, The target detection method comprises: acquiring a radar heat map; establishing a target detection model; training the target detection model by taking the radar heat map as input of the target detection model, using a self-supervised learning pre-training method and a supervised learning training method, to obtain a trained target detection model, specifically comprising: the target detection model comprises a pre-training model and a target detection network; taking the radar heat map as input of the pre-training model, training the pre-training model using a MIM self-supervised learning method, outputting pre-training weights and a heat map containing target categories and position information; taking the heat map as input of the target detection network and taking the pre-training weights as training weights of the target detection network, training the target detection network using supervised learning; detecting the categories and positions of targets based on the trained target detection network; detecting the categories and positions of targets based on the trained target detection model; the radar heat map is taken as input of the pre-training model, and the pre-training model is trained using a MIM self-supervised learning method, specifically comprising: masking the radar heat map to obtain a masked radar heat map, comprising: masking multiple frames of continuous radar heat maps in space and time to obtain the masked radar heat map; the space masking refers to masking part of the area of each radar heat map in multiple frames of continuous radar heat maps in the spatial dimension; the time masking refers to masking the entire area of any several frames of radar heat maps in multiple frames of continuous radar heat maps in the time dimension; a pre-training network based on a Transformer combined with a 3D convolution is constructed; the masked radar heat map is taken as input of the pre-training network, and the pre-training network is trained using a self-supervised learning method.

2. The object detection method of claim 1, wherein, The radar heat map is acquired, specifically comprising: acquiring the target echo signal; performing fast Fourier transform on the target echo signal to obtain distance dimension information; performing low-pass filtering processing on the target echo signal containing distance dimension information to obtain a smoothed radar signal; performing fast Fourier transform on the smoothed radar signal to obtain Doppler dimension information and angle dimension information, respectively; obtaining a range-Doppler heat map and a range-angle heat map according to the distance dimension information, the Doppler dimension information and the angle dimension information, wherein the radar heat map comprises the range-Doppler heat map and the range-angle heat map.

3. The object detection method of claim 1, wherein, The pre-training network adopts an encoder-decoder structure.

4. The object detection method of claim 3, wherein, The encoder structure of the pre-training network comprises an embedding feature module, a first feature extraction module, a first downsampling module, a second feature extraction module, a second downsampling module and a third feature extraction module connected in sequence; the embedding feature module is used for dividing the masked radar heat map into continuous three-dimensional feature blocks; the first feature extraction module is used for extracting features from local and global of the three-dimensional feature blocks respectively to obtain first scale encoding features; the first downsampling module is used for downsampling operation on the first scale encoding features to obtain first downsampled encoding features; The second feature extraction module is configured to perform feature extraction on the first down-sampled encoded features from local and global aspects respectively to obtain second scale encoded features; The second down-sampling module is configured to perform down-sampling operation on the second scale encoded features to obtain second down-sampled encoded features; The third feature extraction module is configured to perform feature extraction on the second down-sampled encoded features from local and global aspects respectively to obtain third scale encoded features; The first feature extraction module, the second feature extraction module and the third feature extraction module each include a 3D convolution layer and a Transformer layer; an output of the 3D convolution layer is connected to an input of the Transformer layer.

5. The object detection method of claim 4, wherein, The decoder structure of the pre-training network includes a first up-sampling module, a first decoding module, a second up-sampling module, a second decoding module and a detection head module connected in sequence; An input of the first up-sampling module is connected to an output of the third feature extraction module; The first up-sampling module is configured to perform up-sampling operation on the third scale encoded features to obtain first up-sampled decoded features; The first decoding module is configured to decode the first up-sampled decoded features and the second scale encoded features to obtain second scale decoded features; The second up-sampling module is configured to perform up-sampling operation on the second scale decoded features to obtain second up-sampled decoded features; The second decoding module is configured to decode the second up-sampled decoded features and the first scale encoded features to obtain first scale decoded features; The detection head module is configured to implement an image reconstruction proxy task on the first scale decoded features. The first decoding module and the second decoding module each include a linear layer and a Transformer layer; an output of the linear layer is connected to an input of the Transformer layer.

6. An application to a vehicle-mounted millimeter wave radar target detection system, characterized in that, The target detection system includes: A radar heat map acquisition module configured to acquire a radar heat map; A model establishment module configured to establish a target detection model; A first training module configured to take the radar heat map as an input of the target detection model, train the target detection model by using a self-supervised learning pre-training method and a supervised learning training method, and obtain a trained target detection model; The target detection model includes a pre-training model and a target detection network; the radar heat map is taken as an input of the pre-training model, the pre-training model is trained by using a MIM self-supervised learning method, pre-training weights and a heat map containing target categories and position information are output; the heat map is taken as an input of the target detection network, the pre-training weights are taken as training weights of the target detection network, the target detection network is trained by using a supervised learning method; and the categories and positions of targets are detected based on the trained target detection network. ​ The radar heat map is taken as an input of the pre-training model, and the pre-training model is trained by using a MIM self-supervised learning method, and specifically includes the following steps: The radar heat map is subjected to a masking operation to obtain a radar heat map after masking, including: respectively performing spatial masking and time masking on a plurality of frames of continuous radar heat maps to obtain the radar heat map after masking; the spatial masking refers to masking a partial region of each of the plurality of frames of continuous radar heat maps in a spatial dimension; and the time masking refers to masking all regions of any several frames of the radar heat maps in a time dimension. A pre-training network based on a Transformer combined with a 3D convolution is constructed. The radar heat map after masking is taken as an input of the pre-training network, and the pre-training network is trained by using a self-supervised learning method. A detection module detects the category and position of a target based on the trained target detection model.

Images