A radar point cloud prediction method and system

Abstract

The present invention relates to the field of autonomous driving technology, and discloses a radar point cloud prediction method and system, the method comprising: constructing an adversarial neural network GANs model, the adversarial neural network GANs model comprising a radar point cloud generator and a multipath discriminator; acquiring radar monitoring data of a millimeter wave radar, and training the adversarial neural network GANs model based on the radar monitoring data to obtain a radar point cloud prediction model; using the radar point cloud prediction model to predict and analyze the distance-Doppler tensor data obtained by radar monitoring, and obtaining a radar point cloud distribution map and a multipath situation distribution map. The present invention uses a deep learning model to infer the input radar tensor data, obtains a 0 / 1 distribution map of the radar point cloud and a 0 / 1 distribution map of the multipath situation, and improves the accuracy of point cloud generation and multipath judgment by training the adversarial network of the point cloud generator and the multipath discriminator; so as to generate a point cloud with higher resolution and higher precision.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a radar point cloud prediction method and system. Background Art

[0002] In the field of autonomous driving, traditional millimeter-wave radar point cloud prediction methods usually rely on constant false alarm rate (CFAR) processing to detect target point clouds. CFAR, as an adaptive threshold setting technology, aims to detect potential target signals from background noise. However, while filtering out noise, this method may also mistakenly remove some useful point cloud signals, resulting in a reduction in the number of output point clouds, thus affecting the radar's resolution and the integrity of point cloud detection.

[0003] In contrast, the 4D millimeter-wave radar point cloud prediction method based on deep learning can effectively overcome this limitation. By training deep neural networks to understand and simulate the complex patterns of radar signals, this method can more accurately distinguish between target points and noise, thereby retaining more point clouds that may contain useful information. This not only avoids the information loss in traditional CFAR methods, but also generates richer point cloud data, significantly improving the spatial resolution of the radar. In autonomous driving systems, this improved resolution means that the environment around the vehicle can be more carefully portrayed, including small objects and subtle road features, which is crucial for accurate navigation and decision-making in complex scenarios. For example, the system can better distinguish between closely spaced pedestrians and vehicles, or accurately identify traffic signs and lane lines in complex traffic environments. In addition, by generating more point clouds, deep learning methods also help improve the performance of target point detection under multipath effects. Under strong reflective roads, strong reflective fences or viaducts in cities, multipath effects may cause the performance of traditional radar systems to degrade, while deep learning models can better identify and process these complex signal paths to ensure the accuracy and robustness of radar detection.

[0004] However, there are still some challenges and problems in the deep learning-based 4D millimeter-wave radar point cloud generation method: 1) Information loss: Due to the insufficient or unbalanced quantity and quality of training data, the deep learning model may not fully capture all the characteristics of the radar signal, resulting in partial information loss during the point cloud generation process, reducing the accuracy and completeness of the point cloud; 2) False point cloud: Since the deep learning model may overfit the noise or specific situations in the training data during the learning process, it will generate false point clouds, that is, generate targets or structures that appear to be real but do not actually exist; 3) High misjudgment rate: Millimeter-wave radar signals are affected by complex environments and multiple interferences. Deep learning models may misjudge the multipath effect, that is, they may mistakenly identify real multipath signals as targets or mistakenly identify target signals as multipath.

[0005] Therefore, how to provide a radar point cloud prediction method and system with high resolution and high accuracy is a problem that needs to be solved urgently. Summary of the invention

[0006] The embodiments of the present invention provide a radar point cloud prediction method and system to solve the above technical problems in the prior art.

[0007] In order to have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended to be a general review, nor is it intended to identify key / important components or to delineate the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0008] According to a first aspect of an embodiment of the present invention, a radar point cloud prediction method is provided.

[0009] In one embodiment, the radar point cloud prediction method includes:

[0010] Constructing a GANs adversarial neural network model, wherein the GANs adversarial neural network model includes a radar point cloud generator and a multipath discriminator;

[0011] Acquire radar monitoring data of a millimeter-wave radar, and train the adversarial neural network GANs model based on the radar monitoring data to obtain a radar point cloud prediction model;

[0012] The radar point cloud prediction model is used to perform prediction analysis on the range-Doppler tensor data obtained by radar monitoring to obtain a radar point cloud distribution map and a multipath situation distribution map.

[0013] In one embodiment, constructing an adversarial neural network GANs model includes: obtaining a deep learning network U-Net model; and improving the deep learning network U-Net model to obtain an improved deep learning network U-Net model; using the improved deep learning network U-Net model as a radar point cloud generator; and using a multi-layer perceptron as a multipath discriminator to construct the adversarial neural network GANs model.

[0014] In one embodiment, the deep learning network U-Net model is improved to obtain an improved deep learning network U-Net model, including: adding two position feature extraction channels to the input end of the deep learning network U-Net model, and modifying the first layer network of the deep learning network U-Net model to a 3*3 convolutional layer to obtain an improved deep learning network U-Net model.

[0015] In one embodiment, the radar point cloud prediction method further includes: before training the adversarial neural network GANs model based on the radar monitoring data, preprocessing the radar monitoring data; wherein the preprocessing of the radar monitoring data includes:

[0016] Generate range-Doppler tensor data using a two-dimensional Fourier transform algorithm based on the ADC raw data of the millimeter-wave radar in the radar monitoring data;

[0017] According to the ADC raw data of the millimeter-wave radar in the radar monitoring data, an ordered statistical constant false alarm algorithm is used to generate a point cloud 0 / 1 binary classification fuzzy label set;

[0018] The point cloud 0 / 1 binary classification fuzzy label set is filtered by using a pre-configured laser radar to obtain a point cloud 0 / 1 binary classification precise label set; and a point cloud with a label of 0 is selected from the point cloud 0 / 1 binary classification precise label set, and multipath points are screened out from the point cloud with a label of 0, and the flag of the multipath point is set to 1, and the flag of the non-multipath point is set to 0, to obtain a multipath binary classification label set;

[0019] A frame of range-Doppler tensor data is selected from the range-Doppler tensor data and input into the radar point cloud generator to obtain a corresponding generator output, and the generator output is used as the pseudo point cloud data.

[0020] In one embodiment, the point cloud 0 / 1 binary classification fuzzy label set is filtered using a pre-configured laser radar to obtain a point cloud 0 / 1 binary classification accurate label set including:

[0021] configuring a laser radar synchronized with the millimeter-wave radar, and filtering the monitoring ranges of the laser radar and the millimeter-wave radar to obtain a monitoring area where the laser radar and the millimeter-wave radar overlap;

[0022] Obtain a millimeter-wave radar point with a label of 1 in the point cloud 0 / 1 binary classification fuzzy label set in the monitoring area; and use a three-dimensional Euclidean distance algorithm to calculate and analyze the millimeter-wave radar point to determine whether the millimeter-wave radar point exists in a three-dimensional Euclidean space.

[0023] If the judgment result is that there is a laser radar point for the millimeter-wave radar point in the three-dimensional Euclidean space, the label bit of the millimeter-wave radar point continues to be set to 1, otherwise the label bit is set to 0, and the point cloud 0 / 1 binary classification accurate label set is obtained.

[0024] In one embodiment, training the adversarial neural network GANs model based on the radar monitoring data includes: pre-training a radar point cloud generator of the adversarial neural network GANs model according to the radar monitoring data;

[0025] And pre-training the radar point cloud generator of the adversarial neural network GANs model according to the radar monitoring data includes: inputting the range-Doppler tensor data into the radar point cloud generator for forward propagation processing to obtain the generator output, and comparing and calculating the generator output with the point cloud 0 / 1 binary classification fuzzy label set to obtain the loss function value between the generator output and the point cloud 0 / 1 binary classification fuzzy label set; according to the loss function value, adjusting the parameters of the radar point cloud generator through the gradient back propagation algorithm to gradually reduce the loss function value; performing a predetermined number of iterations to complete the radar point cloud generator pre-training.

[0026] In one embodiment, training the adversarial neural network GANs model based on the radar monitoring data further includes: training a multipath discriminator of the adversarial neural network GANs model based on the radar monitoring data;

[0027] And the multipath discriminator training of the adversarial neural network GANs model based on the radar monitoring data includes: inputting the fake point cloud data into the multipath discriminator, and using the multipath discriminator to determine whether each point in the fake point cloud data is a multipath situation; if the judgment result is yes, setting the flag of the point to 1, otherwise setting the flag to 0, to obtain a 0 / 1 multipath map; according to the 0 / 1 multipath map and the point cloud 0 / 1 binary classification precise label set, for the multipath flag of each point, using the set similarity measurement function, calculate the loss value; according to the loss value, adjust the parameters of the multipath discriminator through the gradient back propagation algorithm to gradually reduce the loss function value; perform a predetermined number of iterations to complete the multipath discriminator training.

[0028] In one embodiment, when the adversarial neural network GANs model is trained based on the radar monitoring data, a radar point cloud generator of the adversarial neural network GANs model may also be trained secondary based on the radar monitoring data; and the secondary training of the radar point cloud generator of the adversarial neural network GANs model based on the radar monitoring data includes:

[0029] And based on the radar monitoring data, the radar point cloud generator of the adversarial neural network GANs model is trained twice, including: inputting the fake point cloud data into the multipath discriminator, and using the multipath discriminator to determine whether each point in the fake point cloud data is a multipath situation; if the judgment result is yes, the flag of the point is set to 1, otherwise the flag is set to 0 to obtain a 0 / 1 multipath map; according to the 0 / 1 multipath map and the point cloud 0 / 1 binary classification precise label set, for the multipath flag of each point, a set similarity measurement function is used to calculate the loss value; according to the loss value, the parameters of the radar point cloud generator are adjusted by the gradient back propagation algorithm to gradually reduce the loss function value; and a predetermined number of iterations are performed to complete the secondary training of the radar point cloud generator.

[0030] According to a second aspect of an embodiment of the present invention, a radar point cloud prediction system is provided.

[0031] In one embodiment, the radar point cloud prediction system comprises:

[0032] A model building module, used to build a GANs adversarial neural network model, wherein the GANs adversarial neural network model includes a radar point cloud generator and a multipath discriminator;

[0033] A model training module, used to obtain radar monitoring data of the millimeter wave radar, and train the adversarial neural network GANs model based on the radar monitoring data to obtain a radar point cloud prediction model;

[0034] The point cloud prediction module is used to use the radar point cloud prediction model to perform prediction analysis on the range-Doppler tensor data obtained by radar monitoring to obtain a radar point cloud distribution map and a multipath distribution map.

[0035] In one embodiment, when constructing the adversarial neural network GANs model, the model construction module obtains a deep learning network U-Net model; and improves the deep learning network U-Net model to obtain an improved deep learning network U-Net model; the improved deep learning network U-Net model is used as a radar point cloud generator; and a multi-layer perceptron is used as a multipath discriminator to construct the adversarial neural network GANs model.

[0036] In one embodiment, when the model building module improves the deep learning network U-Net model to obtain an improved deep learning network U-Net model, two position feature extraction channels are added to the input end of the deep learning network U-Net model, and the first layer network of the deep learning network U-Net model is modified to a 3*3 convolutional layer to obtain an improved deep learning network U-Net model.

[0037] In one embodiment, the radar point cloud prediction system further includes: a data processing module, used to pre-process the radar monitoring data before training the adversarial neural network GANs model based on the radar monitoring data.

[0038] The data processing module may include the following processing when preprocessing the radar monitoring data:

[0039] Generate range-Doppler tensor data using a two-dimensional Fourier transform algorithm based on the ADC raw data of the millimeter-wave radar in the radar monitoring data;

[0040] According to the ADC raw data of the millimeter-wave radar in the radar monitoring data, an ordered statistical constant false alarm algorithm is used to generate a point cloud 0 / 1 binary classification fuzzy label set;

[0041] The point cloud 0 / 1 binary classification fuzzy label set is filtered by using a pre-configured laser radar to obtain a point cloud 0 / 1 binary classification precise label set; and a point cloud with a label of 0 is selected from the point cloud 0 / 1 binary classification precise label set, and multipath points are screened out from the point cloud with a label of 0, and the flag of the multipath point is set to 1, and the flag of the non-multipath point is set to 0, to obtain a multipath binary classification label set;

[0042] A frame of range-Doppler tensor data is selected from the range-Doppler tensor data and input into the radar point cloud generator to obtain a corresponding generator output, and the generator output is used as the pseudo point cloud data.

[0043] In one embodiment, the point cloud 0 / 1 binary classification fuzzy label set is filtered using a pre-configured laser radar to obtain a point cloud 0 / 1 binary classification precise label set, including: configuring a laser radar synchronized with the millimeter-wave radar, and filtering the monitoring ranges of the laser radar and the millimeter-wave radar to obtain a monitoring area where the laser radar and the millimeter-wave radar overlap; obtaining millimeter-wave radar points with a label of 1 in the point cloud 0 / 1 binary classification fuzzy label set in the monitoring area; and using a three-dimensional Euclidean distance algorithm to calculate and analyze the millimeter-wave radar point to determine whether the millimeter-wave radar point exists in the three-dimensional Euclidean space. If the judgment result is that the millimeter-wave radar point exists in the three-dimensional Euclidean space, the label bit of the millimeter-wave radar point continues to be set to 1, otherwise the label bit is set to 0, to obtain a point cloud 0 / 1 binary classification precise label set.

[0044] In one embodiment, when the model training module trains the adversarial neural network GANs model based on the radar monitoring data, the model training module pre-trains the radar point cloud generator of the adversarial neural network GANs model according to the radar monitoring data;

[0045] When the radar point cloud generator of the adversarial neural network GANs model is pre-trained according to the radar monitoring data, the range-Doppler tensor data is input into the radar point cloud generator for forward propagation processing to obtain the generator output, and the generator output is compared and calculated with the point cloud 0 / 1 binary classification fuzzy label set to obtain the loss function value between the generator output and the point cloud 0 / 1 binary classification fuzzy label set; according to the loss function value, the parameters of the radar point cloud generator are adjusted through the gradient back propagation algorithm to gradually reduce the loss function value; and a predetermined number of iterations are performed to complete the pre-training of the radar point cloud generator.

[0046] In one embodiment, when the model training module trains the adversarial neural network GANs model based on the radar monitoring data, the model training module may train a multipath discriminator of the adversarial neural network GANs model based on the radar monitoring data;

[0047] When training the multipath discriminator of the adversarial neural network GANs model based on the radar monitoring data, the fake point cloud data is input into the multipath discriminator, and the multipath discriminator is used to determine whether each point in the fake point cloud data is a multipath situation; if the judgment result is yes, the flag of the point is set to 1, otherwise the flag is set to 0 to obtain a 0 / 1 multipath map; according to the 0 / 1 multipath map and the point cloud 0 / 1 binary classification precise label set, the multipath flag of each point is used to calculate the loss value by using the set similarity measurement function; according to the loss value, the parameters of the multipath discriminator are adjusted by the gradient back propagation algorithm to gradually reduce the loss function value; a predetermined number of iterations are performed to complete the multipath discriminator training.

[0048] In one embodiment, when the model training module trains the adversarial neural network GANs model based on the radar monitoring data, the model training module may also perform secondary training on the radar point cloud generator of the adversarial neural network GANs model based on the radar monitoring data;

[0049] When the radar point cloud generator of the adversarial neural network GANs model is trained for the second time based on the radar monitoring data, the fake point cloud data is input into the multipath discriminator, and the multipath discriminator is used to determine whether each point in the fake point cloud data is in a multipath situation; if the judgment result is yes, the flag of the point is set to 1, otherwise the flag is set to 0 to obtain a 0 / 1 multipath map; according to the 0 / 1 multipath map and the point cloud 0 / 1 binary classification precise label set, the multipath flag of each point is used to calculate the loss value by using the set similarity measurement function; according to the loss value, the parameters of the radar point cloud generator are adjusted by the gradient back propagation algorithm to gradually reduce the loss function value; a predetermined number of iterations are performed to complete the secondary training of the radar point cloud generator.

[0050] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0051] The present invention uses a deep learning model to infer the input radar tensor data to obtain the 0 / 1 distribution map of the radar point cloud and the 0 / 1 distribution map of the multipath situation. By training the adversarial network of the point cloud generator and the multipath discriminator, the accuracy of point cloud generation and multipath judgment is improved; the useful feature information in the radar source data is retained as much as possible to generate a point cloud with higher resolution and higher accuracy, so that an accurate multipath map can be predicted.

[0052] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0054] Figure 1 is a flow chart of a radar point cloud prediction method according to an exemplary embodiment;

[0055] Figure 2 is a structural block diagram of a radar point cloud prediction system according to an exemplary embodiment;

[0056] Figure 3 The figure is a schematic diagram showing the structure of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0057] The following description and accompanying drawings fully illustrate the specific embodiments of this article so that those skilled in the art can practice them. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of the embodiments of this article includes the entire scope of the claims, as well as all available equivalents of the claims. Herein, the terms "first", "second", etc. are only used to distinguish one element from another, without requiring or implying any actual relationship or order between these elements. In fact, the first element can also be called the second element, and vice versa. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the structure, device or equipment including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also include elements inherent to such structure, device or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the structure, device or equipment including the elements. Each embodiment is described in a progressive manner herein, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other.

[0058] The terms "longitudinal", "lateral", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. in this document indicate the orientation or position relationship based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing this document and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In the description of this document, unless otherwise specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a mechanical connection or an electrical connection, it can also be the internal communication of two elements, it can be a direct connection, or it can be an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0059] As used herein, the term "plurality" means two or more than two, unless otherwise specified.

[0060] In this document, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B.

[0061] In this article, the term "and / or" is a description of the association relationship between objects, indicating that three relationships may exist. For example, A and / or B means: A or B, or, A and B.

[0062] It should be understood that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0063] Each module in the device or system of the present application can be implemented in whole or in part by software, hardware, or a combination thereof. The above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to the above modules.

[0064] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0065] Figure 1 An embodiment of a radar point cloud prediction method of the present invention is shown.

[0066] In this optional embodiment, the radar point cloud prediction method includes:

[0067] Step S101, constructing a GANs adversarial neural network model, wherein the GANs adversarial neural network model includes a radar point cloud generator and a multipath discriminator;

[0068] Step S103, acquiring radar monitoring data of the millimeter wave radar, and training the adversarial neural network GANs model based on the radar monitoring data to obtain a radar point cloud prediction model;

[0069] Step S105, using the radar point cloud prediction model to perform prediction analysis on the range-Doppler tensor data obtained by radar monitoring, to obtain a radar point cloud distribution map and a multipath situation distribution map.

[0070] In this optional embodiment, when constructing an adversarial neural network GANs model, a deep learning network U-Net model (existing deep learning network U-Net model) can be first obtained; and two position feature extraction channels are added to the input end of the deep learning network U-Net model, and the first layer network of the deep learning network U-Net model is modified to a 3*3 convolutional layer to obtain an improved deep learning network U-Net model; then, the improved deep learning network U-Net model is used as a radar point cloud generator; and a multilayer perceptron is used as a multipath discriminator to form the adversarial neural network GANs model.

[0071] In this optional embodiment, when the adversarial neural network GANs model is trained based on the radar monitoring data, the radar point cloud generator of the adversarial neural network GANs model may be pre-trained according to the radar monitoring data; and when the radar point cloud generator of the adversarial neural network GANs model is pre-trained according to the radar monitoring data, the ADC raw data of the millimeter-wave radar in the radar monitoring data is used to generate range-Doppler tensor data using a two-dimensional Fourier transform algorithm; and the ADC raw data of the millimeter-wave radar in the radar monitoring data is used to generate range-Doppler tensor data using an ordered statistical constant. A false alarm algorithm is used to generate a point cloud 0 / 1 binary classification fuzzy label set; then, the range-Doppler tensor data is input into the radar point cloud generator for forward propagation processing to obtain a generator output, and the generator output is compared and calculated with the point cloud 0 / 1 binary classification fuzzy label set to obtain a loss function value between the generator output and the point cloud 0 / 1 binary classification fuzzy label set; according to the loss function value, the parameters of the radar point cloud generator are adjusted by a gradient back propagation algorithm to gradually reduce the loss function value; a predetermined number of iterations (such as 200 rounds) are performed to complete the radar point cloud generator pre-training.

[0072] It should be noted that for the range-Doppler tensor data and the cloud 0 / 1 binary classification fuzzy label set, they can be obtained by preprocessing the radar monitoring data before training. When calculating the loss function value, the loss function that can be used is the loss function based on the binary cross entropy, and its formula is as follows:

[0073] FocalLoss=-ylog(p)-(1-y)log(1-p)

[0074] Where FocalLoss refers to the loss function value; p is the probability prediction size, and y is the binary classification label (0 or 1).

[0075] In this optional embodiment, when the adversarial neural network GANs model is trained based on the radar monitoring data, the multipath discriminator of the adversarial neural network GANs model can also be trained based on the radar monitoring data; and when the multipath discriminator of the adversarial neural network GANs model is trained based on the radar monitoring data, the point cloud 0 / 1 binary classification fuzzy label set is filtered using a pre-configured laser radar to obtain a point cloud 0 / 1 binary classification precise label set; and a point cloud with a label of 0 is selected from the point cloud 0 / 1 binary classification precise label set, and multipath points are screened out from the point cloud with a label of 0, and the flag of the multipath point is set to 1, and the flag of the non-multipath point is set to 0, to obtain a multipath binary classification label set; and a frame of distance-Doppler tensor data is selected from the distance-Doppler tensor data and input into the radar point cloud generator to obtain the corresponding generator output, and the generator output is used as the fake point cloud data.

[0076] Among them, when using a pre-configured laser radar to filter the point cloud 0 / 1 binary classification fuzzy label set to obtain a point cloud 0 / 1 binary classification precise label set, first configure a laser radar synchronized with the millimeter wave radar, and filter the monitoring ranges of the laser radar and the millimeter wave radar to obtain a monitoring area where the laser radar and the millimeter wave radar overlap with each other; obtain the millimeter wave radar point with a label of 1 in the point cloud 0 / 1 binary classification fuzzy label set in the monitoring area; and use a three-dimensional Euclidean distance algorithm to calculate and analyze the millimeter wave radar point to determine whether the millimeter wave radar point exists in the three-dimensional Euclidean space. When the judgment result is that the millimeter wave radar point exists in the three-dimensional Euclidean space, the label bit of the millimeter wave radar point continues to be set to 1, otherwise the label bit is set to 0, and the point cloud 0 / 1 binary classification precise label set is obtained.

[0077] When the multipath discriminator is specifically trained, the fake point cloud data is input into the multipath discriminator, and the multipath discriminator is used to determine whether each point in the fake point cloud data is a multipath situation; if the judgment result is yes, the flag bit of the point is set to 1, otherwise the flag bit is set to 0, and a 0 / 1 multipath map is obtained; according to the 0 / 1 multipath map and the point cloud 0 / 1 binary classification precise label set, the multipath flag bit of each point is used to calculate the loss value by using the set similarity measurement function; according to the loss value, the parameters of the multipath discriminator are adjusted by the gradient back propagation algorithm to gradually reduce the loss function value; a predetermined number of iterations are performed to complete the multipath discriminator training.

[0078] It should be noted that the multipath binary classification label set and the fake point cloud data can be obtained by preprocessing the radar monitoring data before training. The formula of the set similarity measurement function is as follows:

[0079]

[0080] In the formula, DiceLoss refers to the function value of the set similarity measurement function; y i and Represents the predicted value (predicted 0 / 1 multipath map) and true value (point cloud 0 / 1 binary classification accurate label set) of pixel i, and N is the total number of pixels.

[0081] Similarly, in this optional embodiment, in order to achieve the effect of adversarial generation and enable the generator to generate more realistic data, the radar point cloud generator of the adversarial neural network GANs model can also be trained again based on the radar monitoring data.

[0082] When the radar point cloud generator of the adversarial neural network GANs model is trained for the second time based on the radar monitoring data, the fake point cloud data is input into the multipath discriminator, and the multipath discriminator is used to determine whether each point in the fake point cloud data is in a multipath situation; if the judgment result is yes, the flag of the point is set to 1, otherwise the flag is set to 0 to obtain a 0 / 1 multipath map; according to the 0 / 1 multipath map and the point cloud 0 / 1 binary classification precise label set, the multipath flag of each point is used to calculate the loss value by using the set similarity measurement function; according to the loss value, the parameters of the radar point cloud generator are adjusted by the gradient back propagation algorithm to gradually reduce the loss function value; a predetermined number of iterations are performed to complete the secondary training of the radar point cloud generator.

[0083] Similarly, the multipath binary classification label set and fake point cloud data can be obtained by preprocessing the radar monitoring data before training. The formula of the set similarity measurement function is as follows:

[0084]

[0085] In the formula, DiceLoss refers to the function value of the set similarity measurement function; y i and Represents the predicted value (predicted 0 / 1 multipath map) and true value (point cloud 0 / 1 binary classification accurate label set) of pixel i, and N is the total number of pixels.

[0086] Figure 2 An embodiment of a radar point cloud prediction system of the present invention is shown.

[0087] In this optional embodiment, the radar point cloud prediction system includes:

[0088] A model building module 201 is used to build a GANs adversarial neural network model, wherein the GANs adversarial neural network model includes a radar point cloud generator and a multipath discriminator;

[0089] The model training module 203 is used to obtain radar monitoring data of the millimeter wave radar, and train the adversarial neural network GANs model based on the radar monitoring data to obtain a radar point cloud prediction model;

[0090] The point cloud prediction module 205 is used to use the radar point cloud prediction model to perform prediction analysis on the range-Doppler tensor data obtained by radar monitoring to obtain a radar point cloud distribution map and a multipath distribution map.

[0091] In this optional embodiment, when constructing the adversarial neural network GANs model, the model construction module 201 obtains a deep learning network U-Net model (existing deep learning network U-Net model); and modifies the first layer network of the deep learning network U-Net model to a 3*3 convolutional layer to obtain an improved deep learning network U-Net model; then, the improved deep learning network U-Net model is used as a radar point cloud generator; and a multi-layer perceptron is used as a multipath discriminator to form the adversarial neural network GANs model.

[0092] In this optional embodiment, when the model training module 203 trains the adversarial neural network GANs model based on the radar monitoring data, the radar point cloud generator of the adversarial neural network GANs model may be pre-trained according to the radar monitoring data; and when the radar point cloud generator of the adversarial neural network GANs model is pre-trained according to the radar monitoring data, the ADC raw data of the millimeter-wave radar in the radar monitoring data is used to generate range-Doppler tensor data using a two-dimensional Fourier transform algorithm; and the ADC raw data of the millimeter-wave radar in the radar monitoring data is used to generate range-Doppler tensor data using a two-dimensional Fourier transform algorithm. Use the ordered statistical constant false alarm algorithm to generate a point cloud 0 / 1 binary classification fuzzy label set; then, input the range-Doppler tensor data into the radar point cloud generator for forward propagation processing to obtain the generator output, and compare and calculate the generator output with the point cloud 0 / 1 binary classification fuzzy label set to obtain the loss function value between the generator output and the point cloud 0 / 1 binary classification fuzzy label set; according to the loss function value, adjust the parameters of the radar point cloud generator through the gradient back propagation algorithm to gradually reduce the loss function value; perform a predetermined number of iterations (such as 200 rounds) to complete the radar point cloud generator pre-training.

[0093] It should be noted that for the range-Doppler tensor data and the cloud 0 / 1 binary classification fuzzy label set, they can be obtained by preprocessing the radar monitoring data before training (such as by a data preprocessing module (not shown in the figure)). When calculating the loss function value, the loss function that can be used is a loss function based on binary cross entropy, and its formula is as follows:

[0094] FocalLoss=-ylog(p)-(1-y)log(1-p)

[0095] Where FocalLoss refers to the loss function value; p is the probability prediction size, and y is the binary classification label (0 or 1).

[0096] In this optional embodiment, when the model training module 203 trains the adversarial neural network GANs model based on the radar monitoring data, it can also train the multipath discriminator of the adversarial neural network GANs model based on the radar monitoring data; and when the multipath discriminator of the adversarial neural network GANs model is trained based on the radar monitoring data, the pre-configured laser radar is used to filter the point cloud 0 / 1 binary classification fuzzy label set to obtain the point cloud 0 / 1 binary classification precise label set; and select the point cloud with label 0 from the point cloud 0 / 1 binary classification precise label set, filter out the multipath points from the point cloud with label 0, set the flag of the multipath points to 1, and set the flag of the non-multipath points to 0, to obtain the multipath binary classification label set; and select a frame of distance-Doppler tensor data from the distance-Doppler tensor data and input it into the radar point cloud generator to obtain the corresponding generator output, and use the generator output as the fake point cloud data.

[0097] Among them, when using a pre-configured laser radar to filter the point cloud 0 / 1 binary classification fuzzy label set to obtain a point cloud 0 / 1 binary classification precise label set, first configure a laser radar synchronized with the millimeter wave radar, and filter the monitoring ranges of the laser radar and the millimeter wave radar to obtain a monitoring area where the laser radar and the millimeter wave radar overlap with each other; obtain the millimeter wave radar point with a label of 1 in the point cloud 0 / 1 binary classification fuzzy label set in the monitoring area; and use a three-dimensional Euclidean distance algorithm to calculate and analyze the millimeter wave radar point to determine whether the millimeter wave radar point exists in the three-dimensional Euclidean space. When the judgment result is that the millimeter wave radar point exists in the three-dimensional Euclidean space, the label bit of the millimeter wave radar point continues to be set to 1, otherwise the label bit is set to 0, and the point cloud 0 / 1 binary classification precise label set is obtained.

[0098] When the multipath discriminator is specifically trained, the fake point cloud data is input into the multipath discriminator, and the multipath discriminator is used to determine whether each point in the fake point cloud data is a multipath situation; if the judgment result is yes, the flag bit of the point is set to 1, otherwise the flag bit is set to 0, and a 0 / 1 multipath map is obtained; according to the 0 / 1 multipath map and the point cloud 0 / 1 binary classification precise label set, the multipath flag bit of each point is used to calculate the loss value by using the set similarity measurement function; according to the loss value, the parameters of the multipath discriminator are adjusted by the gradient back propagation algorithm to gradually reduce the loss function value; a predetermined number of iterations are performed to complete the multipath discriminator training.

[0099] It should be noted that the multipath binary classification label set and the fake point cloud data can be obtained by preprocessing the radar monitoring data before training (such as by a data preprocessing module (not shown in the figure)). The formula of the set similarity measurement function is as follows:

[0100]

[0101] In the formula, DiceLoss refers to the function value of the set similarity measurement function; y i and Represents the predicted value (predicted 0 / 1 multipath map) and true value (point cloud 0 / 1 binary classification accurate label set) of pixel i, and N is the total number of pixels.

[0102] Similarly, in this optional embodiment, in order to achieve the effect of adversarial generation and enable the generator to generate more realistic data, the model training module 203 may also perform secondary training on the radar point cloud generator of the adversarial neural network GANs model based on the radar monitoring data when training the adversarial neural network GANs model based on the radar monitoring data;

[0103] When the radar point cloud generator of the adversarial neural network GANs model is trained for the second time based on the radar monitoring data, the fake point cloud data is input into the multipath discriminator, and the multipath discriminator is used to determine whether each point in the fake point cloud data is in a multipath situation; if the judgment result is yes, the flag of the point is set to 1, otherwise the flag is set to 0 to obtain a 0 / 1 multipath map; according to the 0 / 1 multipath map and the point cloud 0 / 1 binary classification precise label set, the multipath flag of each point is used to calculate the loss value by using the set similarity measurement function; according to the loss value, the parameters of the radar point cloud generator are adjusted by the gradient back propagation algorithm to gradually reduce the loss function value; a predetermined number of iterations are performed to complete the secondary training of the radar point cloud generator.

[0104] Similarly, the multipath binary classification label set and the fake point cloud data can be obtained by preprocessing the radar monitoring data before training (such as by a data preprocessing module (not shown in the figure)). The formula of the set similarity measurement function is as follows:

[0105]

[0106] In the formula, DiceLoss refers to the function value of the set similarity measurement function; y i and Represents the predicted value (predicted 0 / 1 multipath map) and true value (point cloud 0 / 1 binary classification accurate label set) of pixel i, and N is the total number of pixels.

[0107] In specific applications, during model training, the parameters of the generator and the discriminator are updated in each iteration to improve the quality of the point cloud generated by the generator and the ability of the discriminator to identify multipaths. This process is repeated until the specified number of iterations is reached or the loss values ​​of the generator and the discriminator converge to close to 0.

[0108] In order to facilitate understanding of the above technical solution of the present invention, the following takes a smart car as an example to describe the use process of the generator model and the discriminator model on the smart car, as follows:

[0109] 1. The use process of the generator model in smart cars:

[0110] 1) Use 4D millimeter wave radar to obtain 1 frame of Range-Doppler tensor data (range-Doppler tensor data);

[0111] 2) The radar data is transmitted to the radar's own processor or the smart car's central processing unit, and the generator code is run in the processor. The model will infer the 0 / 1 distribution of the point cloud and obtain a high-precision point cloud map for subsequent autonomous driving tasks such as target detection.

[0112] 2. The use of the discriminator model in smart cars:

[0113] 1) Use 4D millimeter wave radar to obtain 1 frame of point cloud data, or use the generator to generate 1 frame of point cloud data;

[0114] 2) Process point cloud data on the radar’s built-in processor or the smart car’s central processing unit: Run the discriminator model in the processor, and the model will infer the 0 / 1 distribution of multipath by itself to obtain a multipath map, which is used to reduce the radar’s misidentification of targets and reduce the impact of multipath on radar accuracy.

[0115] Figure 3An embodiment of a computer device of the present invention is shown. The computer device may be a server, and the computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store static information and dynamic information data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the steps in the above method embodiment are implemented.

[0116] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0117] In addition, the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiment when executing the computer program.

[0118] In addition, the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0119] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0120] The present invention is not limited to the structures which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A radar point cloud prediction method, characterized in that: include: Constructing a GANs adversarial neural network model, wherein the GANs adversarial neural network model includes a radar point cloud generator and a multipath discriminator; Acquire radar monitoring data of a millimeter-wave radar, preprocess the radar monitoring data, and train the adversarial neural network GANs model based on the preprocessed radar monitoring data to obtain a radar point cloud prediction model; The radar point cloud prediction model is used to predict and analyze the range-Doppler tensor data obtained by radar monitoring to obtain a radar point cloud distribution map and a multipath distribution map; Wherein, the radar monitoring data is preprocessed; Generate range-Doppler tensor data using a two-dimensional Fourier transform algorithm based on the ADC raw data of the millimeter-wave radar in the radar monitoring data; According to the ADC raw data of the millimeter-wave radar in the radar monitoring data, an ordered statistical constant false alarm algorithm is used to generate a point cloud 0 / 1 binary classification fuzzy label set; The point cloud 0 / 1 binary classification fuzzy label set is filtered by using a pre-configured laser radar to obtain a point cloud 0 / 1 binary classification precise label set; and a point cloud with a label of 0 is selected from the point cloud 0 / 1 binary classification precise label set, and multipath points are screened out from the point cloud with a label of 0, and the flag of the multipath point is set to 1, and the flag of the non-multipath point is set to 0, to obtain a multipath binary classification label set; A frame of range-Doppler tensor data is selected from the range-Doppler tensor data and input into the radar point cloud generator to obtain a corresponding generator output, and the generator output is used as the pseudo point cloud data.

2. The radar point cloud prediction method according to claim 1, characterized in that: Building an adversarial neural network GANs model includes: Obtaining a deep learning network U-Net model; and improving the deep learning network U-Net model to obtain an improved deep learning network U-Net model; The improved deep learning network U-Net model is used as a radar point cloud generator; and a multi-layer perceptron is used as a multipath discriminator to form the adversarial neural network GANs model.

3. The radar point cloud prediction method according to claim 2, characterized in that: The deep learning network U-Net model is improved, and the improved deep learning network U-Net model includes: Two position feature extraction channels are added to the input end of the deep learning network U-Net model, and the first layer network of the deep learning network U-Net model is modified to a 3*3 convolutional layer to obtain an improved deep learning network U-Net model.

4. The radar point cloud prediction method according to claim 3, characterized in that: The point cloud 0 / 1 binary classification fuzzy label set is filtered using a pre-configured laser radar to obtain a point cloud 0 / 1 binary classification accurate label set including: configuring a laser radar synchronized with the millimeter-wave radar, and filtering the monitoring ranges of the laser radar and the millimeter-wave radar to obtain a monitoring area where the laser radar and the millimeter-wave radar overlap; Obtain a millimeter-wave radar point with a label of 1 in the point cloud 0 / 1 binary classification fuzzy label set in the monitoring area; and use a three-dimensional Euclidean distance algorithm to calculate and analyze the millimeter-wave radar point to determine whether the millimeter-wave radar point exists in a three-dimensional Euclidean space. If the judgment result is that there is a laser radar point for the millimeter-wave radar point in the three-dimensional Euclidean space, the label bit of the millimeter-wave radar point continues to be set to 1, otherwise the label bit is set to 0, and the point cloud 0 / 1 binary classification accurate label set is obtained.

5. The radar point cloud prediction method according to claim 4, characterized in that: Training the adversarial neural network GANs model based on the radar monitoring data includes: Pre-training the radar point cloud generator of the adversarial neural network GANs model according to the radar monitoring data, and pre-training the radar point cloud generator of the adversarial neural network GANs model according to the radar monitoring data includes: Input the range-Doppler tensor data into the radar point cloud generator for forward propagation processing to obtain a generator output, and compare and calculate the generator output with the point cloud 0 / 1 binary classification fuzzy label set to obtain a loss function value between the generator output and the point cloud 0 / 1 binary classification fuzzy label set; According to the loss function value, adjusting the parameters of the radar point cloud generator through a gradient back propagation algorithm to gradually reduce the loss function value; Perform a predetermined number of iterations to complete the radar point cloud generator pre-training.

6. The radar point cloud prediction method according to claim 5, characterized in that: Training the adversarial neural network GANs model based on the radar monitoring data also includes: The multipath discriminator of the adversarial neural network GANs model is trained based on the radar monitoring data; and the multipath discriminator of the adversarial neural network GANs model is trained based on the radar monitoring data including: Inputting the fake point cloud data into the multipath discriminator, and using the multipath discriminator to determine whether each point in the fake point cloud data is a multipath situation; If the judgment result is yes, the flag bit of the point is set to 1, otherwise the flag bit is set to 0, and a 0 / 1 multipath map is obtained; According to the 0 / 1 multipath graph and the point cloud 0 / 1 binary classification precise label set, for the multipath flag of each point, a loss value is calculated using a set similarity measurement function; According to the loss value, adjusting the parameters of the multipath discriminator by a gradient back-propagation algorithm to gradually reduce the loss function value; Perform a predetermined number of iterations to complete the multipath discriminator training.

7. The radar point cloud prediction method according to claim 6, characterized in that: Training the adversarial neural network GANs model based on the radar monitoring data also includes: The radar point cloud generator of the adversarial neural network GANs model is trained for a second time based on the radar monitoring data; and the radar point cloud generator of the adversarial neural network GANs model is trained for a second time based on the radar monitoring data, including: Inputting the fake point cloud data into the multipath discriminator, and using the multipath discriminator to determine whether each point in the fake point cloud data is a multipath situation; If the judgment result is yes, the flag bit of the point is set to 1, otherwise the flag bit is set to 0, and a 0 / 1 multipath map is obtained; According to the 0 / 1 multipath graph and the point cloud 0 / 1 binary classification precise label set, for the multipath flag of each point, a loss value is calculated using a set similarity measurement function; According to the loss value, adjusting the parameters of the radar point cloud generator through a gradient back propagation algorithm to gradually reduce the loss function value; Perform a predetermined number of iterations to complete the secondary training of the radar point cloud generator.

8. A radar point cloud prediction system, characterized in that: include: A model building module, used to build a GANs adversarial neural network model, wherein the GANs adversarial neural network model includes a radar point cloud generator and a multipath discriminator; A model training module, used to obtain radar monitoring data of the millimeter wave radar, preprocess the radar monitoring data, and train the adversarial neural network GANs model based on the preprocessed radar monitoring data to obtain a radar point cloud prediction model; A point cloud prediction module is used to use the radar point cloud prediction model to perform prediction analysis on the range-Doppler tensor data obtained by radar monitoring to obtain a radar point cloud distribution map and a multipath distribution map; Wherein, the radar monitoring data is preprocessed; Generate range-Doppler tensor data using a two-dimensional Fourier transform algorithm based on the ADC raw data of the millimeter-wave radar in the radar monitoring data; According to the ADC raw data of the millimeter-wave radar in the radar monitoring data, an ordered statistical constant false alarm algorithm is used to generate a point cloud 0 / 1 binary classification fuzzy label set; The point cloud 0 / 1 binary classification fuzzy label set is filtered by using a pre-configured laser radar to obtain a point cloud 0 / 1 binary classification precise label set; and a point cloud with a label of 0 is selected from the point cloud 0 / 1 binary classification precise label set, and multipath points are screened out from the point cloud with a label of 0, and the flag of the multipath point is set to 1, and the flag of the non-multipath point is set to 0, to obtain a multipath binary classification label set; A frame of range-Doppler tensor data is selected from the range-Doppler tensor data and input into the radar point cloud generator to obtain a corresponding generator output, and the generator output is used as the pseudo point cloud data.

9. A radar point cloud prediction system according to claim 8, characterized in that: When constructing the adversarial neural network GANs model, the model construction module obtains a deep learning network U-Net model; improves the deep learning network U-Net model to obtain an improved deep learning network U-Net model; uses the improved deep learning network U-Net model as a radar point cloud generator; and uses a multi-layer perceptron as a multipath discriminator to construct the adversarial neural network GANs model; Among them, when the model construction module improves the deep learning network U-Net model to obtain an improved deep learning network U-Net model, two position feature extraction channels are added to the input end of the deep learning network U-Net model, and the first layer network of the deep learning network U-Net model is modified to a 3*3 convolutional layer to obtain an improved deep learning network U-Net model.

Images