4D millimeter wave point cloud completion method based on ShapeInversion++ network

Abstract

The present invention discloses a 4D millimeter wave point cloud completion method based on ShapeInversion++ network, comprising: establishing a GAN architecture to generate a ShapeInversion++ network; a pre-trained generator G uses a potential vector z to generate a complete 3D point cloud target shape x c ; for x c Perform coordinate correction to obtain the 3D point cloud target shape x c '; Through the degenerate function M, x c 'Converted into the corresponding incomplete 3D point cloud target x p , according to x in and x p Update the loss function M‑Loss of the degradation function M; find the optimal latent vector z through the gradient descent method based on M‑Loss, and adjust the parameters of the generator G, and complete the incomplete 3D point cloud shape x based on the optimal latent vector z and the generator G in The point cloud target completed by the method of the present invention is smoother, has relatively fewer burrs, and is more consistent with the real point cloud target.

Description

Technical Field

[0001] The present invention relates to the field of deep learning and provides a 4D millimeter-wave point cloud completion method based on the ShapeInversion++ network. It is a point cloud completion neural network model based on unsupervised learning. The model optimization and algorithm are proposed based on the intrinsic characteristics of 4D millimeter-wave radar point clouds. Background Art

[0002] In recent years, millimeter-wave radar has been widely used in advanced driver assistance systems (ADAS), becoming a focus of attention for component suppliers. ADAS systems require the coordinated operation of multiple sensors to improve driving safety and traffic quality. Millimeter-wave radar, with its advantages of all-weather operation, compact size, light weight, and low price, can offset the shortcomings of lidar, such as its inability to detect distant targets, its susceptibility to inclement weather, and its high cost. It also addresses the shortcomings of other sensors, such as cameras and ultrasonic radar, in environmental monitoring during ADAS-assisted driving. Therefore, it has become an indispensable core sensor component of ADAS and a key sensor for autonomous and unmanned driving.

[0003] In the past, 3D millimeter-wave radar could only obtain target distance, speed, and azimuth information, failing to reflect road traffic information in real three-dimensional space. However, 4D millimeter-wave radar can additionally obtain target altitude information, potentially replacing lidar in future in-vehicle environmental monitoring systems, reducing hardware costs and offering broad application prospects. Point clouds, a collection of massive points on a target's surface, are typically captured in the real world using devices such as lidar and stereo cameras. Compared to lidar, 4D millimeter-wave radar's point cloud density is sparse. To better perceive the environment using 4D radar alone, it is necessary to accurately enhance the point cloud distribution density of 4D millimeter-wave radar.

[0004] Recent breakthroughs in deep learning technology have spurred the development of point cloud completion methods. The ShapeInversion network, for example, pioneered the application of GAN inverse mapping to 3D point cloud completion. However, the sparsity, irregularity, and density non-uniformity of 4D radar point clouds complicate feature extraction. Directly inputting point clouds into the completion network results in poor completion of point cloud objects of the same type due to the large variance in their coordinate values. Therefore, it is necessary to explore methods suitable for 4D millimeter-wave radar point cloud completion. Summary of the Invention

[0005] The purpose of the present invention is to address the problems and shortcomings of the existing technology and provide a 4D millimeter-wave point cloud completion algorithm based on the ShapeInversion++ network. By completing the preprocessed sparse 4D millimeter-wave radar point cloud data, a dense point cloud similar to the real point cloud is obtained.

[0006] The technical solution to achieve the purpose of the present invention is as follows: On the one hand, a 4D millimeter wave point cloud completion method based on the ShapeInversion++ network is provided, and the method comprises the following steps:

[0007] S1, establish the GAN architecture, generate the ShapeInversion++ network based on the ShapeInversion network, including the generator G and the discriminator D;

[0008] S2, the pre-trained generator G uses a latent vector z to generate a complete 3D point cloud target shape x c ;

[0009] S3, for the 3D point cloud target shape x c Perform coordinate correction to obtain the corrected 3D point cloud target shape x c ';

[0010] S4, based on 3D point cloud target shape x c 'And the incomplete 3D point cloud shape x to be completed in , the 3D point cloud target shape x is transformed into c 'Converted into the corresponding incomplete 3D point cloud target x p , and according to x in and x p Update the loss function M-Loss of the degradation function M;

[0011] S5, according to M-Loss, find the best potential vector z through gradient descent method, and adjust the parameters of generator G, based on the best potential vector z and generator G to complete the incomplete 3D point cloud shape x in .

[0012] Furthermore, the generator G in the ShapeInversion++ network in S1 is a point cloud generation network Tree-GAN based on tree-structured graph convolution, which includes 7 tree layers. Each tree layer executes a point amplification module, namely the Branching module, and a graph-like convolution module, namely the GraphConv module.

[0013] Among them, the expression of the GraphConv module is:

[0014]

[0015] Where σ(.) is the activation unit, Respectively represent the i-th node of the l-th tree layer and the l+1-th tree layer, that is, the coordinates of the three-dimensional point cloud, is a fully connected layer containing K nodes, express All ancestors, linear mapping relationship Integration j Information about all ancestors, b l is the external bias term of the lth tree layer, l ranges from 1 to 6;

[0016] Among them, the expression of the Branching module is:

[0017] for j = 1,…,d l

[0018] Where, is the transformation matrix, [*] j represents the jth column of matrix *, d l Represents the number of expansion points at the lth tree layer;

[0019] The Tree-GAN adopts the unified loss PatchVariance as the loss function G-Loss, which is defined as

[0020]

[0021] Where [D(G(z))] represents the result value of the fake point cloud generated by the generator G input to the discriminator D, λ patch is a weighting parameter, ranging from [0,1]. n represents the number of seeds on the target surface, ρ j represents the average distance between each seed j and its k nearest neighbors.

[0022] Furthermore, the discriminator D in step 1 is a point cloud authenticity identification network built based on the r-GAN structure, including a convolutional layer part and a linear layer part. The convolutional layer part includes five layers of convolution, each layer of convolution is followed by a LeakyReLU function, and the convolution numbers of the five layers of convolution are {3, 64, 128, 256, 256, 512} respectively; the linear layer part is a fully connected layer, which classifies the convolution results through three FC-ReLU structures; the linear layer part is connected to the Sigmoid function;

[0023] The loss function D-Loss of the discriminator is defined as

[0024]

[0025] Where, represents the real point cloud distribution, represents the expected value of the fake point cloud generated by the generator input to the discriminator result, Represents the expected value of the real point cloud input to the discriminator result, Indicates sampling from between true and false point clouds, The Wasserstein distance at which the gradient of the result of the input discriminator of the sampling point becomes 0, yes The mean square error, λ gp is a weighting parameter ranging between [0,1].

[0026] Furthermore, the pre-trained generator G in S2 uses a latent vector z to generate a complete 3D point cloud target shape x c , specifically including:

[0027] S2-1, simultaneously collects 4D millimeter-wave radar point cloud and lidar point cloud of the same vehicle;

[0028] S2-2, cropping, filtering, clustering, and coordinate conversion are performed on the 4D millimeter-wave radar point cloud to serve as an input incomplete 3D point cloud to be completed;

[0029] S2-3, based on the lidar point cloud of S2-1, uses the pre-trained generator G to generate a complete 3D point cloud target shape x using a potential vector z c .

[0030] Furthermore, S3 describes the 3D point cloud target shape x c Perform coordinate correction, including:

[0031] S3-1, in the radar coordinate system, for each 3D point cloud target shape, the boundary values ​​along the x, y, and z directions are calculated respectively to obtain the 3D bounding box of the 3D point cloud target, denoted as the matrix box, which is expressed as:

[0032]

[0033] Where x min 、z min Represents the minimum value in the x and z directions respectively, x max 、y max 、z max Represent the maximum values ​​in the x, y, and z directions respectively;

[0034] Then obtain the coordinate value P of the center point of the 3D point cloud target center and the yaw angle θ yaw for:

[0035]

[0036]

[0037] Among them, [box] ij Represents the i-th row and j-th column of the matrix box;

[0038] S3-2, according to the rotation theory, the rotation correction matrix of the point cloud target is obtained, which is expressed as:

[0039]

[0040] According to the rotation correction matrix [r], the magnification correction coefficient scale of the point cloud target is obtained:

[0041] [box new ]=[box]·[r]

[0042] scale=[box new ] 30 -[box new ] 00

[0043] Combining the above formulas, the coordinate transformation expression of the point cloud target is obtained as follows:

[0044]

[0045] Furthermore, in S4, the target shape x is based on the 3D point cloud c 'And the incomplete 3D point cloud shape x to be completed in , the 3D point cloud target shape x is transformed into c 'Converted into the corresponding incomplete 3D point cloud target x p , specifically:

[0046]

[0047] Where, Represents the input incomplete 3D point cloud shape x in Every point p i , from the 3D point cloud target shape x c 'Find its k nearest neighbor points; Represents x in A set of n points in .

[0048] Furthermore, the loss function M-Loss of the degradation function M in S4 is expressed as:

[0049]

[0050] in,

[0051]

[0052]

[0053] Where, is x in and x pThe bulldozer distance EMD, is x in and x p Characteristic distance FD between φ:x p →x in is x p to x in Deflection; ||D(x p )-D(x in )||1 is x in with x p The Manhattan distance difference.

[0054] Furthermore, in S5, the optimal latent vector z is found by gradient descent method based on M-Loss, and the parameters of the generator G are adjusted to complete the incomplete 3D point cloud shape x based on the optimal latent vector z and the generator G. in , specifically including:

[0055] Update the latent vector z and the parameters θ of the generator G by gradient descent:

[0056]

[0057] Where θ * represents the parameters of the adjusted generator G, z * represents the best latent vector, Represents the point cloud x when the loss function is minimum p The inverse mapping of M(G(z;θ)) represents the point cloud target x after the degradation function M p ;

[0058] By θ * ,z * Generate x in The corresponding best completed point cloud

[0059]

[0060] In another aspect, a 4D millimeter-wave point cloud completion system based on a ShapeInversion++ network is provided, the system comprising:

[0061] The first module is used to establish the GAN architecture and generate the ShapeInversion++ network based on the ShapeInversion network, including the generator G and the discriminator D;

[0062] The second module is used to implement the pre-trained generator G to generate a complete 3D point cloud target shape x using a potential vector z c ;

[0063] The third module is used to analyze the 3D point cloud target shape xc Perform coordinate correction to obtain the corrected 3D point cloud target shape x c ';

[0064] The fourth module is used to calculate the target shape based on 3D point cloud x c 'And the incomplete 3D point cloud shape x to be completed in , the 3D point cloud target shape x is transformed into c 'Converted into the corresponding incomplete 3D point cloud target x p , and according to x in and x p Update the loss function M-Loss of the degradation function M;

[0065] The fifth module is used to find the optimal latent vector z through the gradient descent method based on M-Loss, and adjust the parameters of the generator G to complete the incomplete 3D point cloud shape x based on the optimal latent vector z and the generator G. in .

[0066] Compared with the prior art, the present invention has the following significant advantages:

[0067] 1) The loss function of the ShapeInversion++ discriminator introduces the Wasserstein distance to improve the optimization details of GAN to stabilize the GAN training process. It can better measure the difference between the point cloud generated by the generator and the real point cloud, thereby improving the completion effect.

[0068] 2) The loss function of the degradation module M-Loss uses EMD distance. Compared with traditional Euclidean distance or other common distance metrics, EMD distance is more sensitive to details and density distribution, making the completed point cloud target smoother, with relatively fewer burrs, and more closely aligned with the actual point cloud target.

[0069] 3) A 3D coordinate transformation formula is designed to transform point cloud targets at different positions to the same position, solving the problem of poor completion effect for point cloud targets of the same type due to large differences in point cloud coordinate values.

[0070] 4) The present invention uses a deep learning method to complete the sparse and irregular 4D millimeter-wave radar point cloud. The completed point cloud target is smoother, has relatively fewer burrs, and is more consistent with the real point cloud target.

[0071] The present invention is further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a flowchart of the 4D millimeter wave point cloud completion method based on the ShapeInversion++ network of the present invention.

[0073] Figure 2 FIG. 1 is a network structure diagram of a generator G in an embodiment.

[0074] Figure 3 FIG. 4 is a diagram of the network structure of the discriminator D in an embodiment.

[0075] Figure 4 It is a schematic diagram of the effect of pre-training parameter adjustment in an embodiment.

[0076] Figure 5 The figure is a diagram of the overall structure of the ShapeInversion++ network in one embodiment.

[0077] Figure 6 The figure is a schematic diagram of the point cloud completion effect in an embodiment, wherein Figure (a) and Figure (b) are point cloud completion effect diagrams obtained by three-dimensional reconstruction using the traditional method and the method of the present invention, respectively. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0079] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications will also change accordingly.

[0080] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features specified as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that ordinary technicians in this field can implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0081] In one embodiment, combined Figure 1 , provides a 4D millimeter wave point cloud completion method based on ShapeInversion++ network, the method comprising the following steps:

[0082] Step S1: Establish a GAN architecture and generate a ShapeInversion++ network based on the ShapeInversion network, including a generator G and a discriminator D.

[0083] Combine Figure 2 The generator G in the ShapeInversion++ network in S1 is a point cloud generation network Tree-GAN based on tree-structured graph convolution, which includes 7 tree layers. Each tree layer executes a point amplification module, namely the Branching module, and a graph-like convolution module, namely the GraphConv module. The generator Tree-G converts a single point that conforms to the Gaussian distribution into As input;

[0084] In the GraphConv part, the network introduces a multi-layer graph convolution of the first-order approximation of Chebyshev expansion, which is expressed as follows:

[0085]

[0086] Where σ(·) is the activation unit, Represents the i-th node of the l-th layer in the graph, that is, the coordinates of the three-dimensional point cloud, yes The jth neighbor of yes The set of all neighbors. The first and second items are called loops and neighbors, respectively. z and x' can be expressed as and L is the last layer, and n is the number of point clouds at L. During training, graph convolution finds the optimal weight W for each layer. l and U l and the optimal bias b l , and then generate a fake point cloud similar to the real point cloud through these parameters.

[0087] The main feature of the improved graph convolution is that it refers to the value of its ancestor for each vertex instead of the value of its neighbors, while traditional graph convolution often refers to the values ​​of neighbors on a single layer. Therefore, the expression of the improved GraphConv module is:

[0088]

[0089] Where σ(.) is the activation unit, Respectively represent the i-th node of the l-th tree layer and the l+1-th tree layer, that is, the coordinates of the three-dimensional point cloud, is a fully connected layer containing K nodes, express All ancestors, linear mapping relationship Integration j Information about all ancestors, bl It is the external bias term of the l-th tree layer, and l ranges from 1 to 6.

[0090] The improved graph convolution expression has two main differences from the original expression: one is the use of subnet This improves upon the regular loop term, called a K-supported loop. A second difference is that the values ​​of all ancestors in the tree are used to update the current vertex. For complex graphs, a single parameter is insufficient to describe the complex distribution of points. Therefore, GraphConv uses a K-supported loop term to describe complex point distributions.

[0091] In GraphConv, an ancestor item is defined in Indicates all The ancestor of Integration j Information of all ancestors. Each ancestor is in a different layer and different feature space. The ancestor item integrates all the information of the previous layer and different feature spaces. In order to generate the next point, the current point finds the best mapping relationship by referring to the ancestors of different feature spaces. Effectively incorporates ancestral information.

[0092] Among them, the Branching module is an operation to increase the total number of points, similar to the upsampling of two-dimensional convolution. In Branching, Conversion point For child node d l ,therefore:

[0093] for j = 1,…,d l

[0094] Where, is the transformation matrix, [*] j represents the jth column of matrix *, d l Represents the number of expansion points at the lth tree layer;

[0095] The Tree-GAN adopts the unified loss PatchVariance as the loss function G-Loss, which is defined as

[0096]

[0097] Where [D(G(z))] represents the result value of the fake point cloud generated by the generator G input to the discriminator D, λ patch is a weighting parameter, ranging from [0,1]. n represents the number of seeds on the target surface, ρj represents the average distance between each seed j and its k nearest neighbors.

[0098] Combine Figure 3 The discriminator in ShapeInversion++ is a point cloud authenticity identification network built based on the r-GAN structure. It includes convolutional layers and linear layers. The convolutional layers include five layers of convolution, each of which is followed by a LeakyReLU function. The convolution numbers of the five convolution layers are {3, 64, 128, 256, 256, 512}, respectively, mapping the point cloud data to the hidden feature space. LeakyReLU is used here to replace the original ReLU activation function. By introducing a leakage value in the negative half of the function, the problem of the neural network being unable to update parameters caused by ReLU when the input value is negative is avoided. Compared with the original network ShapeInversion, the maximum pooling layer is discarded to ensure better feature extraction.

[0099] The linear layer is a fully connected layer that uses three FC-ReLU structures to classify the convolution results and map the learned "distributed feature representation" to the sample label space. Finally, a sigmoid function is used to map the results to a range between 0 and 1 through logistic regression.

[0100] The loss function D-Loss of the discriminator is defined as

[0101]

[0102] Where, represents the real point cloud distribution, represents the expected value of the fake point cloud generated by the generator input to the discriminator result, Represents the expected value of the real point cloud input to the discriminator result, Indicates sampling from between true and false point clouds, The Wasserstein distance at which the gradient of the result of the input discriminator of the sampling point becomes 0, yes The mean square error, λ gp is a weighting parameter ranging between [0,1].

[0103] Here, the discriminator's loss function, D-Loss, considers both the positive feedback of real point cloud inputs and the negative feedback of fake point cloud inputs. Compared to the discriminator loss function definition in GANs, D-Loss removes the logarithm, addressing the vanishing gradient problem of the generator in the original GAN, where the better the discriminator, the more serious the generator. The Wasserstein distance is introduced to address the problems in the original GAN ​​where the generator and discriminator losses fail to indicate the training progress and the generated samples lack diversity.

[0104] Step S2, combining Figure 4 , the pre-trained generator G uses a latent vector z to generate a complete 3D point cloud target shape x c ; Specifically include:

[0105] S2-1, simultaneously collects 4D millimeter-wave radar point cloud and lidar point cloud of the same vehicle;

[0106] S2-2, cropping, filtering, clustering, and coordinate conversion are performed on the 4D millimeter-wave radar point cloud to serve as an input incomplete 3D point cloud to be completed;

[0107] S2-3, based on the lidar point cloud of S2-1, uses the pre-trained generator G to generate a complete 3D point cloud target shape x using a potential vector z c .

[0108] Step S3, 3D point cloud target shape x c Perform coordinate correction to obtain the corrected 3D point cloud target shape x c ';Specifically include:

[0109] Step S3-1: In the radar coordinate system, for each 3D point cloud target shape, the boundary values ​​along the x, y, and z directions are calculated respectively to obtain the 3D bounding box of the 3D point cloud target, which is denoted as the matrix box and expressed as:

[0110]

[0111] Where x min 、z min Represents the minimum value in the x and z directions respectively, x max 、y max 、z max Represent the maximum values ​​in the x, y, and z directions respectively;

[0112] Then obtain the coordinate value P of the center point of the 3D point cloud target center and the yaw angle θ yaw for:

[0113]

[0114]

[0115] Among them, [box] ij Represents the i-th row and j-th column of the matrix box;

[0116] Step S3-2: Obtain the rotation correction matrix of the point cloud target according to the rotation theory, which is expressed as:

[0117]

[0118] According to the rotation correction matrix [r], the magnification correction coefficient scale of the point cloud target is obtained:

[0119] [box new ]=[box]·[r]

[0120] scale=[box new ] 30 -[box new ] 00

[0121] Combining the above formulas, the coordinate transformation expression of the point cloud target is obtained as follows:

[0122]

[0123] Step S4, based on the 3D point cloud target shape x c 'And the incomplete 3D point cloud shape x to be completed in , the 3D point cloud target shape x is transformed into c 'Converted into the corresponding incomplete 3D point cloud target x p , and according to x in and x p Update the loss function M-Loss of the degradation function M.

[0124] Here, ShapeInversion++ introduces an accurate and robust mask as the degradation function M, called k-Mask. k-Mask dynamically obtains the generated point cloud x according to the Euclidean distance c And a specific input point cloud x in-s At the same time, considering the ambiguity of the correspondence, k-Mask selects multiple points to achieve robustness. Specifically, the corresponding input point cloud x in Every point p i , all generated from point cloud x c 'Find its k nearest neighbor points, expressed as

[0125]

[0126] Where, Represents the input incomplete 3D point cloud shape x in Every point p i , from the 3D point cloud target shape x c 'Find its k nearest neighbor points; Represents x in The set of n points in

[0127] The degradation module loss function M-Loss not only considers the low-level regularity of the point cloud, but also performs feature matching in the observation space, so that the geometric shape can be adjusted more semantically. The expression is:

[0128]

[0129] in,

[0130]

[0131]

[0132] Where, is x in and x p The bulldozer distance EMD, is x in and x p Characteristic distance FD between; φ:x p →x in is x p to x in Deflection; ||D(x p )-D(x in )||1 is x in with x p The Manhattan distance difference.

[0133] Step S5: Find the best potential vector z by gradient descent method according to M-Loss, and adjust the parameters of generator G. Complete the incomplete 3D point cloud shape x based on the best potential vector z and generator G. in .

[0134] Combine Figure 5 , the ShapeInversion++ network first randomly samples hundreds of latent vectors and selects the latent vector z with the smallest L value as the initial value of the generator fine-tuning to generate a complete 3D point cloud target shape x c ShapeInversion++ fine-tunes the generator parameters θ while updating the latent vector z in real time. This method can effectively improve the results of GAN inverse mapping.

[0135] Step S5 specifically includes:

[0136] Update the latent vector z and the parameters θ of the generator G by gradient descent:

[0137]

[0138] Where θ * represents the parameters of the adjusted generator G, z * represents the best latent vector, Represents the point cloud x when the loss function is minimum p The inverse mapping of M(G(z;θ)) represents the point cloud target x after the degradation function M p ;

[0139] By θ * ,z * Generate x in The corresponding best completed point cloud

[0140]

[0141] like Figure 6 The figure below compares the point cloud completion results obtained by the proposed method and the traditional method for 3D reconstruction. It can be seen that the proposed method uses deep learning to complete sparse and irregular 4D millimeter-wave radar point clouds. The completed point cloud objects are smoother, have fewer burrs, and are more consistent with the actual point cloud objects.

[0142] In one embodiment, a 4D millimeter wave point cloud completion system based on a ShapeInversion++ network is provided, the system comprising:

[0143] The first module is used to establish the GAN architecture and generate the ShapeInversion++ network based on the ShapeInversion network, including the generator G and the discriminator D;

[0144] The second module is used to implement the pre-trained generator G to generate a complete 3D point cloud target shape x using a potential vector z c ;

[0145] The third module is used to analyze the 3D point cloud target shape x c Perform coordinate correction to obtain the corrected 3D point cloud target shape x c ';

[0146] The fourth module is used to calculate the target shape based on 3D point cloud x c 'And the incomplete 3D point cloud shape x to be completed in , the 3D point cloud target shape x is transformed into c 'Converted into the corresponding incomplete 3D point cloud target x p , and according to xin and x p Update the loss function M-Loss of the degradation function M;

[0147] The fifth module is used to find the optimal latent vector z through the gradient descent method based on M-Loss, and adjust the parameters of the generator G to complete the incomplete 3D point cloud shape x based on the optimal latent vector z and the generator G. in .

[0148] For the specific limitations of the 4D millimeter-wave point cloud completion system based on the ShapeInversion++ network, please refer to the limitations of the 4D millimeter-wave point cloud completion method based on the ShapeInversion++ network above, which will not be repeated here. Each module in the above-mentioned 4D millimeter-wave point cloud completion system based on the ShapeInversion++ network can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0149] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0150] S1, establish the GAN architecture, generate the ShapeInversion++ network based on the ShapeInversion network, including the generator G and the discriminator D;

[0151] S2, the pre-trained generator G uses a latent vector z to generate a complete 3D point cloud target shape x c ;

[0152] S3, for the 3D point cloud target shape x c Perform coordinate correction to obtain the corrected 3D point cloud target shape x c ';

[0153] S4, based on 3D point cloud target shape x c 'And the incomplete 3D point cloud shape x to be completed in , the 3D point cloud target shape x is transformed into c 'Converted into the corresponding incomplete 3D point cloud target x p , and according to x in and x p Update the loss function M-Loss of the degradation function M;

[0154] S5, according to M-Loss, find the best potential vector z through gradient descent method, and adjust the parameters of generator G, based on the best potential vector z and generator G to complete the incomplete 3D point cloud shape x in .

[0155] For the specific limitations of each step, please refer to the limitations of the 4D millimeter-wave point cloud completion method based on the ShapeInversion++ network above, which will not be repeated here.

[0156] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only illustrative of the principles of the present invention. Without departing from the spirit and scope of the present invention, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A 4D millimeter wave point cloud completion method based on ShapeInversion++ network, characterized by: The method comprises the following steps: S1, establish the GAN architecture, generate the ShapeInversion++ network based on the ShapeInversion network, including the generator G and the discriminator D; S2, the pre-trained generator G uses a latent vector z to generate a complete 3D point cloud target shape x c ; S3, for the 3D point cloud target shape x c Perform coordinate correction to obtain the corrected 3D point cloud target shape x c '; S4, based on 3D point cloud target shape x c 'And the incomplete 3D point cloud shape x to be completed in , the 3D point cloud target shape x is transformed into c 'Converted into the corresponding incomplete 3D point cloud target x p , and according to x in and x p Update the loss function M-Loss of the degradation function M; S5, according to M-Loss, find the best potential vector z through gradient descent method, and adjust the parameters of generator G, based on the best potential vector z and generator G to complete the incomplete 3D point cloud shape x in ; The generator G in the ShapeInversion++ network in S1 is a point cloud generation network Tree-GAN based on tree-structured graph convolution, which includes 7 tree layers. Each tree layer executes a point amplification module (Branching module) and a graph-like convolution module (GraphConv module). Among them, the expression of the GraphConv module is: Where σ(.) is the activation unit, Respectively represent the i-th node of the l-th tree layer and the l+1-th tree layer, that is, the coordinates of the three-dimensional point cloud, is a fully connected layer containing K nodes, express All ancestors, linear mapping relationship Integration j Information about all ancestors, b l is the external bias term of the lth tree layer, l ranges from 1 to 6; Among them, the expression of the Branching module is: Where, is the transformation matrix, [*] j represents the jth column of matrix *, d l Represents the number of expansion points at the lth tree layer; The Tree-GAN adopts the unified loss PatchVariance as the loss function G-Loss, which is defined as Where [D(G(z))] represents the result value of the fake point cloud generated by the generator G input to the discriminator D, λ patch is a weighting parameter, ranging from [0,1]. n represents the number of seeds on the target surface, ρ j represents the average distance between each seed j and its k nearest neighbors.

2. The 4D millimeter wave point cloud completion method based on ShapeInversion++ network according to claim 1 is characterized in that: In step 1, the discriminator D is a point cloud authenticity identification network built based on the r-GAN structure, which includes a convolutional layer and a linear layer. The convolutional layer includes five layers of convolution, each of which is followed by a LeakyReLU function. The convolution numbers of the five layers are {3, 64, 128, 256, 256, 512} respectively; the linear layer is a fully connected layer that classifies the convolution results through three FC-ReLU structures; the linear layer is followed by a Sigmoid function. The loss function D-Loss of the discriminator is defined as Where, represents the real point cloud distribution, represents the expected value of the fake point cloud generated by the generator input to the discriminator result, Represents the expected value of the real point cloud input to the discriminator result, Indicates sampling from between true and false point clouds, The Wasserstein distance at which the gradient of the result of the input discriminator of the sampling point becomes 0, yes The mean square error, λ gp is a weighting parameter ranging between [0,1].

3. The 4D millimeter wave point cloud completion method based on ShapeInversion++ network according to claim 2, characterized in that: The pre-trained generator G in S2 uses a latent vector z to generate a complete 3D point cloud target shape x c , specifically including: S2-1, simultaneously collects 4D millimeter-wave radar point cloud and lidar point cloud of the same vehicle; S2-2, cropping, filtering, clustering, and coordinate conversion are performed on the 4D millimeter-wave radar point cloud to serve as an input incomplete 3D point cloud to be completed; S2-3, based on the lidar point cloud of S2-1, uses the pre-trained generator G to generate a complete 3D point cloud target shape x using a potential vector z c .

4. The 4D millimeter wave point cloud completion method based on ShapeInversion++ network according to claim 3 is characterized in that: S3 is the 3D point cloud target shape x c Perform coordinate correction to obtain the corrected 3D point cloud target shape x c ', specifically including: S3-1, in the radar coordinate system, for each 3D point cloud target shape, the boundary values ​​along the x, y, and z directions are calculated respectively to obtain the 3D bounding box of the 3D point cloud target, denoted as the matrix box, which is expressed as: Where x min 、z min Represents the minimum value in the x and z directions respectively, x max 、y max 、z max Represent the maximum values ​​in the x, y, and z directions respectively; Then obtain the coordinate value P of the center point of the 3D point cloud target center and the yaw angle θ yaw for: Among them, [box] ij Represents the i-th row and j-th column of the matrix box; S3-2, according to the rotation theory, the rotation correction matrix of the point cloud target is obtained, which is expressed as: According to the rotation correction matrix [r], the magnification correction coefficient scale of the point cloud target is obtained: [box new ]=[box]·[r] scale=[box new ] 30 -[box new ] 00 Combining the above formulas, the coordinate transformation expression of the point cloud target is obtained as follows:

5. The 4D millimeter wave point cloud completion method based on ShapeInversion++ network according to claim 4, characterized in that: S4 based on 3D point cloud target shape x c 'And the incomplete 3D point cloud shape x to be completed in , the 3D point cloud target shape x is transformed into c 'Converted into the corresponding incomplete 3D point cloud target x p , specifically: Where, Represents the input incomplete 3D point cloud shape x in Every point p i , from the 3D point cloud target shape x c 'Find its k nearest neighbor points; Represents x in A set of n points in .

6. The 4D millimeter wave point cloud completion method based on ShapeInversion++ network according to claim 5, characterized in that: The loss function M-Loss of the degradation function M in S4 is expressed as: in, Where, is x in and x p The bulldozer distance EMD, is x in and x p Characteristic distance FD between φ:x p →x in is x p to x in Deflection; ||D(x p )-D(x in )||1 is x in with x p The Manhattan distance difference.

7. The 4D millimeter wave point cloud completion method based on ShapeInversion++ network according to claim 6, characterized in that: In S5, the optimal latent vector z is found by gradient descent method based on M-Loss, and the parameters of generator G are adjusted. Based on the optimal latent vector z and generator G, the incomplete 3D point cloud shape x is completed. in , specifically including: Update the latent vector z and the parameters θ of the generator G by gradient descent: Where θ * represents the parameters of the adjusted generator G, z * represents the best latent vector, Represents the point cloud x when the loss function is minimum p The inverse mapping, M(G(x;θ)) represents the point cloud target x after the degradation function M p ; By θ * ,z * Generate x in The corresponding best completed point cloud 8. A 4D millimeter wave point cloud completion system based on a ShapeInversion++ network according to any one of claims 1 to 7, characterized in that: The system comprises: The first module is used to establish the GAN architecture and generate the ShapeInversion++ network based on the ShapeInversion network, including the generator G and the discriminator D; The second module is used to implement the pre-trained generator G to generate a complete 3D point cloud target shape x using a potential vector z c ; The third module is used to analyze the 3D point cloud target shape x c Perform coordinate correction to obtain the corrected 3D point cloud target shape x c '; The fourth module is used to calculate the target shape based on 3D point cloud x c 'And the incomplete 3D point cloud shape x to be completed in , the 3D point cloud target shape x is transformed into c 'Converted into the corresponding incomplete 3D point cloud target x p , and according to x in and x p Update the loss function M-Loss of the degradation function M; The fifth module is used to find the optimal latent vector z through the gradient descent method based on M-Loss, and adjust the parameters of the generator G to complete the incomplete 3D point cloud shape x based on the optimal latent vector z and the generator G. in .

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images