Millimeter-wave radar sparse point cloud reconstruction method based on multi-step neural network structure

Abstract

The present invention discloses a millimeter wave radar sparse point cloud reconstruction method based on a multi-step neural network structure, comprising step S1: obtaining first information including speed, angle and distance through a millimeter wave radar and performing Fourier transform processing on the obtained speed and distance, so as to use the generated speed and distance spectrum as the first target feature. The present invention discloses a millimeter wave radar sparse point cloud reconstruction method based on a multi-step neural network structure, which directly naturalizes radar sparse point cloud information to predict high-resolution point cloud information through a feed forward neural network, and provides a new algorithm for the millimeter wave radar sparse point cloud problem.

Description

Technical Field

[0001] The present invention belongs to the technical field of millimeter wave radar point cloud, and in particular relates to a millimeter wave radar sparse point cloud reconstruction method based on a multi-step neural network structure. Background Art

[0002] With the rapid development of intelligent perception technology, millimeter-wave radar, as one of the intelligent sensors, has been widely used in object detection and recognition tasks. In the field of autonomous driving, millimeter-wave radar is often used as an auxiliary sensor in conjunction with cameras and lidar to achieve advanced autonomous driving functions. Compared with lidar and cameras, the advantages of millimeter-wave radar are that it can detect object speed and distance information, is low-cost, and is suitable for various weather and lighting conditions. However, a very big drawback of millimeter-wave radar is its low resolution, and the point cloud obtained in the field of autonomous driving is very sparse.

[0003] If the high-resolution point cloud information of millimeter-wave radar can be effectively improved, the market application prospects of millimeter-wave radar can be greatly improved, and the scale is huge. In the literature, there is currently no effective method to reconstruct the sparse point cloud of millimeter-wave radar.

[0004] Therefore, further improvements are made to the above problems. Summary of the invention

[0005] The main purpose of the present invention is to provide a millimeter-wave radar sparse point cloud reconstruction method based on a multi-step neural network structure, which directly naturalizes the radar sparse point cloud information through a forward transfer neural network (Feed Forward Neural Network) to predict high-resolution (high-density) point cloud information, providing a new algorithm for the millimeter-wave radar sparse point cloud problem.

[0006] To achieve the above objectives, the present invention provides a millimeter wave radar sparse point cloud reconstruction method based on a multi-step neural network structure, comprising the following steps:

[0007] Step S1: obtaining first information including speed, angle and distance through a millimeter wave radar and performing Fourier transform processing on the obtained speed and distance to generate a speed and distance spectrum diagram as a first target feature;

[0008] Step S2: extracting the spatial coordinate point cloud information corresponding to the speed and distance spectrum graph;

[0009] Step S3: clustering the spatial coordinate point cloud information so that the coordinates of all the center points do not change, thereby obtaining cluster points;

[0010] Step S4: Input the clustered points into the multi-step forward neural network model to perform naturalized prediction and generate supplementary spatial coordinate point cloud information, thereby obtaining dense point cloud information.

[0011] As a further preferred technical solution of the above technical solution, in step S1, the continuous wave signal is modulated by the radar frequency, and the relationship between the distance and the speed is:

[0012]

[0013] Among them, c is the propagation speed of electromagnetic waves in the air, R is the distance between the object and the radar, v is the relative speed, τ is the propagation time, and to obtain R and v:

[0014]

[0015] As a further preferred technical solution of the above technical solution, in step S1, the speed and distance are subjected to Fourier transform processing by the following formula:

[0016]

[0017] Where b[n,m,k] refers to the nth sampling point of the mth chirp signal received on the kth antenna, T fast is the sampling interval for sampling the signal within one frequency sweep cycle;

[0018] Perform a two-dimensional discrete Fourier transform on N sampling points within a period to obtain the corresponding distance. The formula is:

[0019]

[0020] As a further preferred technical solution of the above technical solution, in step S2, the distance frequency component and the speed frequency component with non-zero values ​​are extracted from step S1, and the following is calculated:

[0021] x=Rsinθ,y=Rcosθ,

[0022] To obtain the spatial coordinate point cloud information (x, y, z coordinates) of the millimeter wave radar.

[0023] As a further preferred technical solution of the above technical solution, in step S3, the spatial coordinate point cloud information is subjected to K-means clustering processing, which is specifically implemented as follows:

[0024] Step S3.1: Randomly select K = 5 initial point information, denoted as C1, C2, C3, C4, C5;

[0025] Step S3.2: Find the Euclidean distance between all points of the spatial coordinate point cloud information and the initial point information. The distance formula is:

[0026]

[0027] X=(x1,x2,x3),C=(y1,y2,y3);

[0028] Step S3.3: Assign the point corresponding to each spatial coordinate point cloud information to the nearest center by obtaining the distance:

[0029]

[0030] Step S3.4: Recalculate the new center point of the assigned points using the following formula:

[0031]

[0032] Step S3.5: Repeat steps S3.2 to S3.4 to ensure that the coordinates of all center points (cluster points) remain unchanged.

[0033] As a further preferred technical solution of the above technical solution, in step S4, the obtained cluster points are input into a multi-step forward neural network model, which is specifically implemented as follows:

[0034] Step S4.1: The multi-step forward neural network model includes an input layer, a hidden layer, and an output layer. L Let the input data point of the Lth layer and the output result of the L+1th layer be X L+1 for:

[0035] Z L =F(X L )=W L *X L +b L ;

[0036]

[0037] J=||AX L+1 ||;

[0038] Among them, W L ,b L is the weight and bias parameter of the Lth layer, A is the real labeled data for training, and its calculation formula is:

[0039]

[0040]

[0041] Step S4.2: The gradient is calculated as:

[0042]

[0043]

[0044] Step S4.3: According to the multi-step neural network format of differential equations, the transmission of the forward neural network is modeled as a mathematical differential equation, the formula is:

[0045]

[0046] Where: T is the number of layers of the neural network model, and the differential equation is expanded to obtain:

[0047]

[0048] Where: i is the coefficient.

[0049] To achieve the above objectives, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the millimeter-wave radar sparse point cloud reconstruction method based on a multi-step neural network structure are implemented.

[0050] To achieve the above objectives, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the millimeter-wave radar sparse point cloud reconstruction method based on a multi-step neural network structure.

[0051] The beneficial effects of the present invention are:

[0052] 1. It is proposed to use clustering method combined with neural network to reconstruct sparse millimeter-wave radar point cloud, which is the first time to propose this algorithm-based process.

[0053] 2. Propose a multi-step forward propagation neural network structure. Different from the general forward propagation network structure, this network structure is derived from rigorous mathematical theories and formulas, and has produced good results in actual experiments.

[0054] 3. Using the K-mean clustering algorithm to cluster the original radar point cloud information can effectively reduce the data dimension and improve the training effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 The invention discloses a point cloud prediction diagram based on Kmean clustering and a multi-step feedforward neural network of a millimeter wave radar sparse point cloud reconstruction method based on a multi-step neural network structure.

[0056] Figure 2 It is the original point cloud information collected by millimeter wave radar.

[0057] Figure 3 It is a point cloud information graph predicted by a neural network in the millimeter wave radar sparse point cloud reconstruction method based on a multi-step neural network structure of the present invention. DETAILED DESCRIPTION

[0058] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.

[0059] In the preferred embodiments of the present invention, those skilled in the art should note that the electronic device and storage medium involved in the present invention may be regarded as prior art.

[0060] Preferred embodiments.

[0061] The present invention discloses a millimeter wave radar sparse point cloud reconstruction method based on a multi-step neural network structure, comprising the following steps:

[0062] Step S1: obtaining first information including speed, angle and distance through a millimeter wave radar and performing Fourier transform processing on the obtained speed and distance to generate a speed and distance spectrum diagram as a first target feature;

[0063] Step S2: extracting the spatial coordinate point cloud information corresponding to the speed and distance spectrum graph;

[0064] Step S3: clustering the spatial coordinate point cloud information so that the coordinates of all the center points do not change, thereby obtaining cluster points;

[0065] Step S4: Input the clustered points into the multi-step forward neural network model to perform naturalized prediction and generate supplementary spatial coordinate point cloud information, thereby obtaining dense point cloud information.

[0066] Specifically, in step S1, the radar frequency is used to modulate the continuous wave signal, and the relationship between distance and speed is:

[0067]

[0068] Among them, c is the propagation speed of electromagnetic waves in the air, R is the distance between the object and the radar, v is the relative speed, τ is the propagation time, and to obtain R and v:

[0069]

[0070] More specifically, in step S1, the speed and distance are subjected to Fourier transform processing using the following formula:

[0071]

[0072] Where b[n,m,k] refers to the nth sampling point of the mth chirp signal received on the kth antenna, T fast is the sampling interval for sampling the signal within one frequency sweep cycle;

[0073] Perform a two-dimensional discrete Fourier transform on N sampling points within a period to obtain the corresponding distance. The formula is:

[0074]

[0075] Furthermore, in step S2, the distance frequency component and the speed frequency component with non-zero values ​​are extracted from step S1, and the following is calculated:

[0076] x=Rsinθ,y=Rcosθ,

[0077] To obtain the spatial coordinate point cloud information (x, y, z coordinates) of the millimeter wave radar.

[0078] Furthermore, in step S3, the spatial coordinate point cloud information is subjected to K-means clustering processing, which is specifically implemented as follows:

[0079] Step S3.1: Randomly select K = 5 initial point information, denoted as C1, C2, C3, C4, C5;

[0080] Step S3.2: Find the Euclidean distance between all points of the spatial coordinate point cloud information and the initial point information. The distance formula is:

[0081]

[0082] X=(x1,x2,x3),C=(y1,y2,y3);

[0083] Step S3.3: Assign the point corresponding to each spatial coordinate point cloud information to the nearest center by obtaining the distance:

[0084]

[0085] Step S3.4: Recalculate the new center point of the assigned points using the following formula:

[0086]

[0087] Step S3.5: Repeat steps S3.2 to S3.4 to ensure that the coordinates of all center points (cluster points) remain unchanged.

[0088] Preferably, in step S4, the obtained cluster points are input into a multi-step forward neural network model, which is specifically implemented by the following steps:

[0089] Step S4.1: The multi-step forward neural network model includes an input layer, a hidden layer, and an output layer. L Let the input data point of the Lth layer and the output result of the L+1th layer be X L+1 for:

[0090] Z L =F(X L )=W L *X L +b L ;

[0091]

[0092] J=||AX L+1 ||;

[0093] Among them, W L ,b L is the weight and bias parameter of the Lth layer, A is the real labeled data for training, and its calculation formula is:

[0094]

[0095]

[0096] Step S4.2: The gradient is calculated as:

[0097]

[0098]

[0099] Step S4.3: According to the multi-step neural network format of differential equations, the transmission of the forward neural network is modeled as a mathematical differential equation, the formula is:

[0100]

[0101] Where: T is the number of layers of the neural network model, and the differential equation is expanded to obtain:

[0102]

[0103] Where: i is the coefficient.

[0104] The present invention also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the millimeter wave radar sparse point cloud reconstruction method based on a multi-step neural network structure are implemented.

[0105] The principle of the present invention is:

[0106] The present invention provides a new algorithm to solve the problem of sparse millimeter wave radar point cloud. The algorithm can effectively improve the wide application range of reconstructing millimeter wave radar high-density point cloud through the output results of existing radars.

[0107] The specific process of the real-time detection algorithm is divided into:

[0108] 1. Obtain information such as speed, angle, and distance through millimeter-wave radar signals.

[0109] Using radar frequency modulated continuous wave (FMCW) signal, the relationship between distance and speed is:

[0110]

[0111] Where c is the propagation speed of electromagnetic waves in the air, R is the distance between the object and the radar, v is the relative speed, and τ is the propagation time.

[0112]

[0113] 2. Perform Fourier transform on the obtained speed and distance to generate a distance-speed spectrum as the first target feature.

[0114]

[0115] b[n,m,k] refers to the nth sampling point of the mth chirp received on the kth antenna, T fast is the sampling interval of the signal in one frequency sweep cycle. The 2-transmit 4-receive radar antenna used is shown in Figure 1 , N is 256, M is 128, K is 4, and the frame rate is 20fps. Performing a two-dimensional discrete Fourier transform on the N sampling points in one cycle, the corresponding distance is obtained as formula (4).

[0116]

[0117] 3. Extract the spatial coordinate point cloud information obtained from the radar signal

[0118] Extract the distance frequency component and speed frequency component with non-zero value, and calculate according to formula (2):

[0119] x=Rsinθ,y=Rcosθ (6)

[0120] That is the radar space point cloud information (x, y, z coordinates).

[0121] 4. Perform K-mean clustering on the obtained point cloud (x, y, z). In order to reduce the input cost and improve the calculation efficiency, set K=5. The specific algorithm of K-mean is as follows:

[0122] (1) Randomly select K = 5 initial points, denoted as C1, C2, C3, C4, C5

[0123] (2) Find the Euclidean distance d(X,C) between all points and the initial point. The distance formula is:

[0124]

[0125] X=(x1,x2,x3),C=(y1,y2,y3)

[0126] (3) Using the distance obtained above, assign each point to the center with the closest distance.

[0127]

[0128] (4) Recalculate the new center point of the assigned points using the following formula:

[0129]

[0130] (5) Repeat steps 2 to 4 until the coordinates of all center points no longer change.

[0131] 5. Input the obtained cluster points (K=5) into the multi-step forward neural network model for normalization prediction to generate more point cloud information. The basic structure of the forward neural network includes input layer, hidden layer, and output layer.

[0132] The basic structure of the feedforward neural network, including input layer, hidden layer, and output layer;

[0133] Its mathematical formula is:

[0134] Input layer: X L is the input data point of the Lth layer, and the output result X of the L+1th layer L+1 Z L =F(X L )=W L *X L +b L ,

[0135]

[0136] J=||AX L+1 ||

[0137] Where W L ,b L is the weight and bias parameter of the Lth layer, A is the real labeled data for training, and its calculation formula is

[0138]

[0139]

[0140] The gradient is calculated as:

[0141]

[0142]

[0143] One of the biggest features of the present invention is that it is based on the theory of feedforward neural network and proposes a multi-step neural network format for differential equations. The transmission of the feedforward neural network is modeled as a mathematical differential equation as shown below:

[0144]

[0145] Time T is the number of layers in the neural network model. Using the Adams-Bashforth formula AB (abbreviated as AB format), the differential equation is expanded to obtain

[0146]

[0147] where λ i As coefficients, the 3-step AB format is used in the experiment, and its mathematical formula is,

[0148]

[0149] 6. Get the result and produce more dense point cloud information, such as Figure 3 As shown, and the algorithm flow of the present invention is as follows Figure 1 shown.

[0150] The present invention also discloses a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the millimeter-wave radar sparse point cloud reconstruction method based on a multi-step neural network structure. It is worth mentioning that the technical features of the electronic device and storage medium involved in the patent application of the present invention should be regarded as the prior art, and the specific structure, working principle, and control method and spatial arrangement method that may be involved in these technical features can be selected conventionally in the field, and should not be regarded as the inventive point of the present patent, and the present patent will not be further specifically described.

[0151] For those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A millimeter wave radar sparse point cloud reconstruction method based on a multi-step neural network structure, characterized in that: The following steps are involved: Step S1: obtaining first information including speed, angle and distance through a millimeter wave radar and performing Fourier transform processing on the obtained speed and distance to generate a speed and distance spectrum diagram as a first target feature; In step S1, the speed and distance are Fourier transformed using the following formula: Where b[n,m,k] refers to the nth sampling point of the mth linear frequency modulation signal received by the kth antenna, T fast is the sampling interval for sampling the signal within one frequency sweep cycle; Perform a two-dimensional discrete Fourier transform on N sampling points within a period to obtain the corresponding distance. The formula is: Step S2: extracting the spatial coordinate point cloud information corresponding to the speed and distance spectrum graph; Step S3: clustering the spatial coordinate point cloud information so that the coordinates of all the center points do not change, thereby obtaining cluster points; Step S4: Input the clustered points into the multi-step forward neural network model to perform naturalized prediction and generate supplementary spatial coordinate point cloud information, thereby obtaining dense point cloud information, specifically including modeling the transmission of the forward neural network as a mathematical differential equation according to the multi-step neural network format of the differential equation.

2. The millimeter wave radar sparse point cloud reconstruction method based on a multi-step neural network structure according to claim 1 is characterized in that: In step S1, the radar frequency modulates the continuous wave signal, and the relationship between distance and speed is: Among them, c is the propagation speed of electromagnetic waves in the air, R is the distance between the object and the radar, v is the relative speed, τ is the propagation time, and to obtain R and v:

3. The millimeter wave radar sparse point cloud reconstruction method based on a multi-step neural network structure according to claim 2 is characterized in that: In step S2, the distance frequency component and the speed frequency component with non-zero values ​​are extracted from step S1, and the following is calculated: x=Rsinθ,y=Rcosθ, To obtain the spatial coordinate point cloud information of the millimeter wave radar.

4. The millimeter wave radar sparse point cloud reconstruction method based on a multi-step neural network structure according to claim 3 is characterized in that: In step S3, the spatial coordinate point cloud information is subjected to K-means clustering processing, which is specifically implemented as follows: Step S3.1: Randomly select K = 5 initial point information, denoted as C1, C2, C3, C4, C5; Step S3.2: Find the Euclidean distance between all points of the spatial coordinate point cloud information and the initial point information. The distance formula is: X=(x1,x2,x3),C=(y1,y2,y3); Step S3.3: Assign the point corresponding to each spatial coordinate point cloud information to the nearest center by obtaining the distance: Step S3.4: Recalculate the new center point of the assigned points using the following formula: Step S3.5: Repeat steps S3.2 to S3.4 to ensure that the coordinates of all center points do not change.

5. The millimeter wave radar sparse point cloud reconstruction method based on a multi-step neural network structure according to claim 4 is characterized in that: In step S4, the obtained cluster points are input into the multi-step forward neural network model, which is specifically implemented as follows: Step S4.1: The multi-step forward neural network model includes an input layer, a hidden layer, and an output layer. L Let the input data point of the Lth layer and the output result of the L+1th layer be X L+1 for: Z L =F(X L )=W L *X L +b L ; J=||A-X L+1 ||; Among them, W L ,b L is the weight and bias parameter of the Lth layer, A is the real labeled data for training, and its calculation formula is: Step S4.2: The gradient is calculated as: Step S4.3: According to the multi-step neural network format of differential equations, the transmission of the forward neural network is modeled as a mathematical differential equation, the formula is: Where: T is the number of layers of the neural network model, and the differential equation is expanded to obtain: Where: i is the coefficient.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the millimeter-wave radar sparse point cloud reconstruction method based on a multi-step neural network structure are implemented as described in any one of claims 1 to 5.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the millimeter wave radar sparse point cloud reconstruction method based on a multi-step neural network structure as described in any one of claims 1 to 5 are implemented.

Images