End-to-end cloud completion method based on generative adversarial network
By employing an end-to-end point cloud completion method based on generative adversarial networks and utilizing a multi-scale Transformer encoder and feature decoder, the problem of low point cloud completion accuracy is solved, achieving high-precision point cloud completion while preserving the geometric semantics and details of the original point cloud.
Patent Information
- Application Number
- CN202210173243.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-24
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-02-24
AI Technical Summary
Existing point cloud completion methods have low accuracy and cannot effectively solve the problem of data incompleteness caused by occlusion, light reflection and other issues in applications such as autonomous driving and medicine.
An end-to-end point cloud completion method based on generative adversarial networks is adopted. By acquiring the ModelNet40 dataset, point cloud segmentation and downsampling are performed to construct generator and discriminator networks. Multi-scale Transformer encoder and feature decoder are used for point cloud completion, and network parameters are optimized to improve accuracy.
It improves the accuracy of point cloud completion, preserves the geometric semantic information and details of the original point cloud, solves the problem of missing geometric semantics and poor details in the results of point cloud completion methods, and has good robustness.
Smart Images

Figure CN114638751B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of computer vision and autonomous driving technology, and in particular to an end-to-end cloud completion method based on a generative adversarial network. Background Art
[0002] In the field of 3D vision, three-dimensional data representation methods include voxels, point clouds, and binary grids. Among them, point clouds have the unique advantages of small data volume and high resolution and are widely used in 3D data processing. LiDAR, stereo cameras, or RGBD depth cameras are common ways to obtain point clouds in real life. Traditional laser scanners have disadvantages such as high cost, low efficiency, and poor quality. Among them, stereo cameras or depth cameras are frequently used, but in the actual acquisition process of point clouds, the acquired point cloud data is basically incomplete due to the obstruction of objects, light reflection on the surface of objects, transparency of objects, and limitations of camera viewing angles. Therefore, in order to enable point cloud data to be applied in fields such as autonomous driving and medicine, it is a very important task to complete the information of point clouds.
[0003] Currently, point clouds and voxels are the main data sources for 3D shape completion. "Shape inpainting using 3d generative adversarial network and recurrent convolutional networks" proposed a voxel grid-based algorithm 3D-ED-GAN to achieve point cloud repair. However, due to the resolution limitation of the voxel-based method, the amount of calculation will be greatly increased during the network training process. "Pointnet: deep learning on point sets for 3D classification and segmentation" proposed PointNet, which allows point cloud data to be directly used as input to the neural network and realizes the modeling, classification and segmentation of point cloud shapes. PointNet has been looking for a symmetric function to solve the problem of point cloud disorder. In the end, they chose to use the maximum pooling in the neural network to solve the problem and achieved good results. After that, the algorithm for point cloud completion tasks has been greatly developed. "Learning representations and generative models for 3d point clouds》 proposed the first method to use the encoder-decoder framework (L-GAN) to solve the point cloud completion task. Because the architecture of this method is not mainly built for the task of point cloud repair, it can only repair the point cloud to a certain extent. Therefore, the current point cloud completion method has low accuracy. Summary of the invention
[0004] Based on this, it is necessary to provide an end-to-end cloud completion method based on a generative adversarial network that can improve the accuracy of the point cloud completion method in response to the above technical problems.
[0005] An end-to-end cloud completion method based on a generative adversarial network, the method comprising:
[0006] S1. Obtain the public dataset in ModelNet40 and construct the required original point cloud dataset;
[0007] S2: First, perform a cutting and deletion operation on the original point cloud in the original point cloud dataset, and delete a part of the point cloud in the original point cloud to construct the missing point cloud G. 1 (N, 3), N represents the number of points in the point cloud, and the cut part of the point cloud P is retained GT and missing point cloud G 1 (N, 3) is used as data for model training;
[0008] S3, missing point cloud G 1 (N, 3) Use the random point sampling method to perform downsampling operations to obtain two point clouds G with a resolution less than 1024×3 2 (N * , 3) and point cloud G 3 (N ** ,3),N>N * >N ** ;
[0009] S4. constructing a generator network and a discriminator network, wherein the generator network includes a multi-scale Transformer encoder and a feature decoder;
[0010] S5. The missing point cloud G 1 (N, 3), point cloud G 2 (N * , 3) and point cloud G 3 (N ** , 3) as the input of the multi-scale Transformer encoder, the global feature vector F is obtained p ;
[0011] S6, the global feature vector F p Input into the feature decoder and output the point cloud P of the missing part out ;
[0012] S7, the cut part point cloud P GT And the point cloud P of the missing part out As the input of the discriminator, the point cloud P of the missing part is output out score;
[0013] S8, according to the point cloud P of the missing partout The loss function is used to calculate the loss, and the network parameters are optimized by back propagation until the preset accuracy is reached. The model training is completed to obtain the point cloud completion model.
[0014] S9, obtaining a residual point cloud and inputting it into a point cloud completion model to complete the point cloud, and outputting a residual point cloud of the residual point cloud;
[0015] S10, splicing the incomplete point cloud of the incomplete point cloud with the incomplete point cloud to obtain a completed point cloud.
[0016] In one embodiment, the original point cloud in the original point cloud data set is first cut and deleted, and a part of the point cloud is deleted from the original point cloud to construct the missing point cloud G. 1 The step (N, 3) comprises:
[0017] Find 5 points in the three-dimensional coordinate system. These 5 points are O 1 (0, 0, 1), O 2 (1, 0, 0), O 3 (-1, 0, 0), O 4 (1, 0, 1), O 5 (-1, 1, 0), randomly select a point from these 5 points as the center point O j , j∈(1, 2…5), and then calculate each point P in the original point cloud i (x i ,y i , z i ), i∈(1, 2…N) to the center point O j The sum of the squares of the distances D i , i∈(1,2…N), delete D i The first 512 points with the smallest median distance are deleted to obtain the point cloud G 1 (N, 3), where x i is the coordinate value of the x-axis in the three-dimensional coordinate system, and y i is the coordinate value of the y-axis in the three-dimensional coordinate system, z i is the coordinate value of the z-axis in the three-dimensional coordinate system.
[0018] In one embodiment, the missing point cloud G 1 (N, 3) Use the random point sampling method to perform downsampling operations to obtain two point clouds G with a resolution less than 1024×3 2 (N * , 3) and point cloud G 3 (N ** , 3) steps, comprising:
[0019] First, in the missing point cloud G 1A point is randomly selected from (N, 3), and the circle is centered at that point. * A circle is formed with a radius of , and the first R points closest to the center of the circle are deleted from the points within the circle, and this cycle repeats N times * Second, we get a low-resolution point cloud G 2 (N * , 3);
[0020] In the point cloud G 2 (N * , 3) randomly select a point, take this point as the center of the circle, and set R = N * / N ** A circle is formed with a radius of , and the first R points closest to the center of the circle are deleted from the points within the circle, and this cycle repeats N times ** Second, we get a low-resolution point cloud G 3 (N ** , 3).
[0021] In one embodiment, the multi-scale Transformer encoder is composed of three Transformer encoders with the same structure, which are used to transform the missing point cloud G 1 (N, 3), point cloud G 2 (N * , 3) and point cloud G 3 (N ** , 3) are used as the input of the three Transformer encoders, and the output contains G 1 , G 2 , G 3 The global feature F of point cloud information p , thereby extracting the characteristic information of residual cloud.
[0022] In one embodiment, the Transformer encoder is mainly composed of a point embedding module and a superimposed self-attention mechanism module. The point embedding module is composed of two connected CBRs, and the two connected CBR structures are (128, 128). The point embedding module regards each point as a word in a sentence and places points with similar position semantics in a close position in space; G 1 (N, 3) as an example, two connected CBRs generate [128×128] point embedding feature information F e , point embedding feature information F e It will be used as the input of the superimposed attention mechanism module. The superimposed self-attention mechanism module consists of four connected self-attention mechanisms with the same structure. The self-attention mechanism is mainly composed of CBR. After four connected self-attention mechanisms, each point is mapped to the dimension of [128-128-128-128], and these four dimensions are spliced together to form G 1 Global features Its size is [128×512]; similarly, through G 2 (N * ,3) The global features obtained and through G 3 (N ** ,3) The global features obtained The size of the splicing is [128×512], and Then the global feature vector F is obtained through maximum pooling. p , F p The dimension is [1×1536].
[0023] In one embodiment, the feature decoder is mainly composed of a feed-forward neural network, which mainly processes the global feature F p Decoding, the feature decoder converts the global feature vector F p As its input, it consists of a linear layer and CBR, and takes the global feature vector F of dimension [1×1536] p Decoded into [M×3] point cloud P Out , P out represents the point cloud of the missing part, and M is the number of missing points.
[0024] In one embodiment, the discriminator consists of three CBR layers, two LBR layers, and a linear layer. The discriminator divides the missing part of the point cloud P Out As input, three consecutive CBRs generate intermediate features of [64-128-256] dimensions, which are concatenated and then max-pooled to obtain [1×448] features. Then, two LBRs and a linear layer are used to obtain a scalar V, which represents the point cloud P of the missing part generated by the generator. out score.
[0025] In one embodiment, the loss function is:
[0026]
[0027] Among them, L CD (S 1 , S 2 ) is the point cloud P of the missing part out And cut part of the point cloud P GT The gap between 1 is the point cloud P of the missing part out , S 2 To cut part of the point cloud P GT , x is the point cloud P of the missing part out A point of y is the cut part of the point cloud P GT a certain point.
[0028] The end-to-end cloud completion method based on the generative adversarial network constructs the required original point cloud dataset by acquiring the dataset disclosed in ModelNet40; firstly, the original point cloud in the original point cloud dataset is cut and deleted, a part of the point cloud is deleted from the original point cloud to construct the missing point cloud, and the cut part of the point cloud and the missing point cloud are retained as data for model training; the missing point cloud is downsampled using a random point sampling method to obtain two low-resolution point clouds and point clouds; a generator network and a discriminator network are constructed, wherein the generator network includes a multi-scale Transformer encoder and a feature decoder; the missing point cloud, point cloud and point cloud are used as multi-scale Transformer. ormer encoder to obtain a global feature vector; input the global feature vector into the feature decoder, and output the point cloud of the missing part; use the cut part point cloud and the missing part point cloud as the input of the discriminator, and output the score of the point cloud of the missing part; according to the score of the point cloud of the missing part, use the loss function to calculate the loss, optimize the network parameters by back propagation until the preset accuracy is reached, complete the model training, and obtain the point cloud completion model; obtain the incomplete point cloud and input it into the point cloud completion model to complete the point cloud, and output the incomplete part point cloud of the incomplete point cloud; splice the incomplete part point cloud of the incomplete point cloud with the incomplete point cloud to obtain the completed point cloud. The architecture of the generative adversarial network ensures the stability of network training and can learn global context information. The point cloud of the missing part is directly generated end-to-end through the multi-scale Transformer encoder and feature decoder, so that the completed point cloud not only does not lose geometric semantic information but also retains the details of the original point cloud, thereby improving the accuracy of the point cloud completion model, further solving the problems of missing geometric semantics and poor details in the results of the point cloud completion method, and improving the accuracy of the point cloud completion method. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 An overall network structure diagram of model training in an end-to-end cloud completion method based on a generative adversarial network in one embodiment;
[0030] Figure 2 A network structure diagram of a generator in one embodiment;
[0031] Figure 3 is a network structure diagram of a discriminator in one embodiment;
[0032] Figure 4 This is a visual comparison of the effect of completing the missing point cloud in this application;
[0033] Figure 5 This is a visual comparison of the completion effects of point clouds with different degrees of missingness in this application. DETAILED DESCRIPTION
[0034] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0035] In one embodiment, Figure 1 As shown, an end-to-end cloud completion method based on a generative adversarial network is provided, and the method is applied to a terminal as an example for explanation, including the following steps:
[0036] An end-to-end cloud completion method based on a generative adversarial network, the method comprising:
[0037] S1. Obtain the public dataset in ModelNet40 and construct the required original point cloud dataset.
[0038] Among them, the ModelNet40 public data set has 40 categories of three-dimensional point cloud data. Each point cloud data is composed of N points, N is not the same, and each point contains three-dimensional coordinate information; there are 80,000 point cloud data in total, and this application uses 64,000 point clouds as training sets and 16,000 point clouds as test sets.
[0039] S2: First, perform a cutting and deletion operation on the original point cloud in the original point cloud dataset, and delete a part of the point cloud in the original point cloud to construct the missing point cloud G. 1 (N, 3), N represents the number of points in the point cloud, and the cut part of the point cloud P is retained GT and missing point cloud G 1 (N, 3) is used as data for model training.
[0040] S3, missing point cloud G 1 (N, 3) Use the random point sampling method to perform downsampling operations to obtain two low-resolution point clouds G 2 (N * , 3) and point cloud G 3 (N ** ,3),N>N * >N ** .
[0041] Among them, N * is the point cloud G 2 The number of data points, N*<1024, N ** is the point cloud G 3 The number of points in the data, N**<1024. Low-resolution point cloud refers to point cloud with less than 1024 points. For example, if N=1024, N*=512, N**=256.
[0042] S4. Construct a generator network and a discriminator network, wherein the generator network includes a multi-scale Transformer encoder and a feature decoder.
[0043] Among them, Figure 2 As shown, the generator network includes Encoder T (Multi-scale Transformer Encoder) and Decoder M (Feature decoder). T It consists of three Transformer encoders with the same structure. The Transformer encoder is a point cloud feature extraction network based on Transformer. It mainly consists of a point embedding module and a superimposed self-attention mechanism module. The point embedding module consists of two connected CBRs (Convolution, BatchNorm, ReLU). The two connected CBR structures are (128, 128). The point embedding module regards each point as a word in a sentence. Its function is to place points with similar position semantics in close positions in space. 1 (N, 3) as an example, two connected CBRs generate [128×128] point embedding feature information F e . F e It will be used as the input of the superimposed attention mechanism module. The superimposed self-attention mechanism module consists of four connected self-attention mechanisms with the same structure. The self-attention mechanism is mainly composed of CBR. Assuming the input size is Row×Col, first, each self-attention machine obtains the Query matrix, Key matrix, and Value matrix through the same linear layer. The sizes of the three matrices are Row×Col respectively. * 、Col * ×Row, Row×Col, where the Key matrix needs to be transposed before it is obtained, and then the Query matrix and the Key matrix are matrix-multiplied to obtain the attention weight (Attention weight), whose size is Row×Row. The size of the attention weight represents the semantic similarity and correlation between each point and other points. Finally, the obtained attention weight is normalized and then matrix-multiplied with the Value matrix to obtain the attention feature (Attention feature), whose size is Row×Col. In this way, the output and input after a layer of self-attention mechanism are the same size Row×Col. We stacked 4 layers of self-attention mechanisms in the attention mechanism module. After four connected self-attention mechanisms, each point is mapped to the dimension of [128-128-128-128], and these 4 dimensions are spliced together to form G 1 Global features Its size is [128×512]; similarly, through G2 (N * ,3) The global features obtained and through G 3 (N ** ,3) The global features obtained The size of the splicing is [128×512], and Then the global feature vector F is obtained through maximum pooling. p , F p The dimension is [1×1536]; the feature decoder is composed of a feedforward neural network. The feature decoder converts the global feature vector F p As its input, it consists of a linear layer and CBR, and takes the global feature vector F of dimension [1×1536] p Decoded into [M×3] point cloud P Out , represents the point cloud that completes the missing area, and M is the number of missing points.
[0044] S5. The missing point cloud G 1 (N, 3), point cloud G 2 (N * , 3) and point cloud G 3 (N ** , 3) as the input of the multi-scale Transformer encoder, the global feature vector F is obtained p .
[0045] S6, the global feature vector F p Input into the feature decoder and output the point cloud P of the missing part out .
[0046] S7, the cut part point cloud P GT And the point cloud P of the missing part out As the input of the discriminator, the point cloud P of the missing part is output out score.
[0047] S8, according to the point cloud P of the missing part out The loss function is used to calculate the loss, and the network parameters are optimized by back propagation until the preset accuracy is reached. The model training is completed to obtain the point cloud completion model.
[0048] S9, obtaining the incomplete point cloud and inputting it into the point cloud completion model to complete the point cloud, and outputting the incomplete point cloud of the incomplete point cloud.
[0049] S10, splicing the incomplete point cloud of the incomplete point cloud with the incomplete point cloud to obtain a completed point cloud.
[0050] In one embodiment, the original point cloud in the original point cloud data set is first cut and deleted, and a part of the point cloud is deleted from the original point cloud to construct the missing point cloud G. 1 The step (N, 3) comprises:
[0051] Find 5 points in the three-dimensional coordinate system. These 5 points are O 1 (0, 0, 1), O 2 (1, 0, 0), O 3 (-1, 0, 0), O 4 (1, 0, 1), O 5 (-1, 1, 0), randomly select a point from these 5 points as the center point O j , j∈(1, 2…5), and then calculate each point P in the original point cloud i (x i ,y i , z i ), i∈(1, 2…N) to the center point O j The sum of the squares of the distances D i , i∈(1,2…N), delete D i The first 512 points with the smallest median distance are deleted to obtain the point cloud G 1 (N, 3), where xi is the coordinate value of the x-axis in the three-dimensional coordinate system, y i is the coordinate value of the y-axis in the three-dimensional coordinate system, z i is the coordinate value of the z-axis in the three-dimensional coordinate system.
[0052] In one embodiment, the missing point cloud G 1 (N, 3) Use the random point sampling method to perform downsampling operations to obtain two low-resolution point clouds G 2 (N * , 3) and point cloud G 3 (N ** , 3) steps, comprising:
[0053] First, in the missing point cloud G 1 A point is randomly selected from (N, 3), and the circle is centered at that point. * A circle is formed with a radius of , and the first R points closest to the center of the circle are deleted from the points within the circle, and this cycle repeats N times * Second, we get a low-resolution point cloud G 2 (N * , 3);
[0054] In the point cloud G 2 (N * , 3) randomly select a point, take this point as the center of the circle, and set R = N * / N **A circle is formed with a radius of , and the first R points closest to the center of the circle are deleted from the points within the circle, and this cycle repeats N times ** Second, we get a low-resolution point cloud G 3 (N ** , 3).
[0055] In one embodiment, the multi-scale Transformer encoder is composed of three Transformer encoders with the same structure, which 1 (N, 3), point cloud G 2 (N * , 3) and point cloud G 3 (N ** , 3) are used as the input of the three Transformer encoders, and the output contains G 1 , G 2 , G 3 The global feature F of point cloud information p , thereby extracting the characteristic information of residual cloud.
[0056] Among them, the multi-scale Transformer encoder can extract local detail features of point clouds with different resolutions, thereby effectively improving the accuracy of point cloud completion.
[0057] In one embodiment, the Transformer encoder is mainly composed of a point embedding module and a superimposed self-attention mechanism module. The point embedding module is composed of two connected CBRs, and the two connected CBR structures are (128, 128). The point embedding module regards each point as a word in a sentence and places points with similar position semantics in a close position in space. 1 (N, 3) as an example, two connected CBRs generate [128×128] point embedding feature information F e , point embedding feature information F e It will be used as the input of the superimposed attention mechanism module. The superimposed self-attention mechanism module consists of four connected self-attention mechanisms with the same structure. The self-attention mechanism is mainly composed of CBR. After four connected self-attention mechanisms, each point is mapped to the dimension of [128-128-128-128], and these four dimensions are spliced together to form G 1 Global features Its size is [128×512]; similarly, through G 2 (N * ,3) The global features obtained and through G 3 (N ** ,3) The global features obtained The size of the splicing is [128×512], and Then the global feature vector F is obtained through maximum pooling. p , F p The dimension is [1×1536].
[0058] Among them, CBR refers to the convolution normalization module, which is composed of Convolution, BatchNorm and ReLU connected in sequence.
[0059] In one embodiment, the feature decoder is mainly composed of a feedforward neural network, which mainly processes the global feature F p Decoding, the feature decoder converts the global feature vector F p As its input, it consists of a linear layer and CBR, and takes the global feature vector F of dimension [1×1536] p Decoded into [M×3] point cloud P Out , P out represents the point cloud of the missing part, and M is the number of missing points.
[0060] In one embodiment, Figure 3 As shown, the discriminator consists of three CBRs, two LBRs and a linear layer. The discriminator will miss the point cloud P Out As input, three consecutive CBRs generate intermediate features of [64-128-256] dimensions, which are concatenated and then max-pooled to obtain [1×448] features. Then, two LBRs and a linear layer are used to obtain a scalar V, which represents the point cloud P of the missing part generated by the generator. out score.
[0061] Among them, LBR refers to the linear normalization module, which is composed of Linear, BatchNorm and ReLU connected in sequence.
[0062] In one embodiment, the loss function is:
[0063]
[0064] Among them, L CD (S 1 , S 2 ) is the point cloud P of the missing part out And cut part of the point cloud P GT The gap between 1 is the point cloud P of the missing part out , S 2 To cut part of the point cloud P GT , x is the point cloud P of the missing part out A point of y is the cut part of the point cloud P GT a certain point.
[0065] Among them, the chamfer distance (CD distance) can measure the difference between the point cloud shape after repair and completion and the real point cloud shape. The closer the value is to zero, the higher the similarity between the point clouds is, and the better the repair and completion effect is.
[0066] The end-to-end cloud completion method based on the generative adversarial network constructs the required original point cloud dataset by acquiring the dataset disclosed in ModelNet40; firstly, the original point cloud in the original point cloud dataset is cut and deleted, a part of the point cloud is deleted from the original point cloud to construct the missing point cloud, and the cut part of the point cloud and the missing point cloud are retained as data for model training; the missing point cloud is downsampled using a random point sampling method to obtain two low-resolution point clouds and point clouds; a generator network and a discriminator network are constructed, wherein the generator network includes a multi-scale Transformer encoder and a feature decoder; the missing point cloud, point cloud and point cloud are used as multi-scale Transformer. ormer encoder to obtain a global feature vector; input the global feature vector into the feature decoder, and output the point cloud of the missing part; use the cut part point cloud and the missing part point cloud as the input of the discriminator, and output the score of the point cloud of the missing part; according to the score of the point cloud of the missing part, use the loss function to calculate the loss, optimize the network parameters by back propagation until the preset accuracy is reached, complete the model training, and obtain the point cloud completion model; obtain the incomplete point cloud and input it into the point cloud completion model to complete the point cloud, and output the incomplete part point cloud of the incomplete point cloud; splice the incomplete part point cloud of the incomplete point cloud with the incomplete point cloud to obtain the completed point cloud. The architecture of the generative adversarial network ensures the stability of network training and can learn global context information. The point cloud of the missing part is directly generated end-to-end through the multi-scale Transformer encoder and feature decoder, so that the completed point cloud not only does not lose geometric semantic information but also retains the details of the original point cloud, thereby improving the accuracy of the point cloud completion model, further solving the problems of missing geometric semantics and poor details in the results of the point cloud completion method, and improving the accuracy of the point cloud completion method.
[0067] Furthermore, the present application can also effectively repair point clouds with different missing ratios and has good robustness.
[0068] The present invention also carries out result analysis through experiment, such as Figure 4 As shown, the completion effect of this application is intuitively displayed. Figure 4 By comparing with the true value map (i.e. the actual complete point cloud), it can be seen that the present application can effectively repair the incomplete parts of objects such as airplanes, while making the generated point cloud evenly distributed. Figure 5 As shown, the robustness effect of this application is intuitively demonstrated. This application tests point cloud shape restoration experiments with different degrees of missingness and different missing parts, such as Figure 5The figure shows the effect of completing the point cloud repair when the missing degree of an aircraft sample point cloud is 50%, 25%, and 35% respectively. The experimental results show that under different missing degrees, the method of this application can complete the missing parts.
[0069] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0070] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. An end-to-end cloud completion method based on generative adversarial networks, characterized in that: The method comprises: S1. Obtain the public dataset in ModelNet40 and construct the required original point cloud dataset; S2, first cut and delete the original point cloud in the original point cloud data set, delete a part of the point cloud in the original point cloud to construct the missing point cloud G1 (N, 3), N represents the number of points in the point cloud, and retain the cut part of the point cloud P GT And the missing point cloud G1(N,3) is used as the data for model training; S3, use the random point sampling method to downsample the missing point cloud G1(N,3) to obtain two point clouds G2(N with a resolution less than 1024×3 * , 3) and point cloud G3(N ** ,3),N>N * >N ** ; S4. constructing a generator network and a discriminator network, wherein the generator network includes a multi-scale Transformer encoder and a feature decoder; S5, missing point cloud G1 (N, 3), point cloud G2 (N * , 3) and point cloud G3(N ** , 3) as the input of the multi-scale Transformer encoder, the global feature vector F is obtained p ; S6, the global feature vector F p Input into the feature decoder and output the point cloud P of the missing part out ; S7, the cut part point cloud P GT And the point cloud P of the missing part out As the input of the discriminator, the point cloud P of the missing part is output out score; S8, according to the point cloud P of the missing part out The loss function is used to calculate the loss, and the network parameters are optimized by back propagation until the preset accuracy is reached. The model training is completed to obtain the point cloud completion model. S9, obtaining a residual point cloud and inputting it into a point cloud completion model to complete the point cloud, and outputting a residual point cloud of the residual point cloud; S10, splicing the incomplete point cloud of the incomplete point cloud with the incomplete point cloud to obtain a completed point cloud.
2. The method according to claim 1, characterized in that: The step of first performing a cutting and deleting operation on the original point cloud in the original point cloud data set and deleting a part of the point cloud in the original point cloud to construct the missing point cloud G1 (N, 3) includes: Find 5 points in the three-dimensional coordinate system. These 5 points are O1(0, 0, 1), O2(1, 0, 0), O3(-1, 0, 0), O4(1, 0, 1), O5(-1, 1, 0). Randomly select a point from these 5 points as the center point O. j , j∈(1, 2…5), and then calculate each point P in the original point cloud i (x i ,y i , z i ), i∈(1, 2…N) to the center point O j The sum of the squares of the distances D i , i∈(1,2…N), delete D i The first 512 points with the smallest median distance are deleted to obtain the point cloud G1(N, 3), where x i is the coordinate value of the x-axis in the three-dimensional coordinate system, and y i is the coordinate value of the y-axis in the three-dimensional coordinate system, z i is the coordinate value of the z-axis in the three-dimensional coordinate system.
3. The method according to claim 1, characterized in that The missing point cloud G1(N,3) is downsampled using a random point sampling method to obtain two point clouds G2(N) with a resolution less than 1024×3. * , 3) and point cloud G3(N ** , 3) steps, comprising: First, randomly select a point in the missing point cloud G1(N,3), take this point as the center of the circle, and use R=N / N * A circle is formed with a radius of , and the first R points closest to the center of the circle are deleted from the points within the circle, and this cycle repeats N times * times, and obtain a point cloud G2 (N with a resolution less than 1024×3 * , 3); In the point cloud G2(N * , 3) randomly select a point, take this point as the center of the circle, and set R = N * / N ** A circle is formed with a radius of , and the first R points closest to the center of the circle are deleted from the points within the circle, and this cycle repeats N times ** times, and obtain a point cloud G3 (N with a resolution less than 1024×3 ** , 3).
4. The method according to claim 1, characterized in that: The multi-scale Transformer encoder consists of three Transformer encoders with the same structure, which convert the missing point cloud G1(N,3), the point cloud G2(N * , 3) and point cloud G3(N ** , 3) are used as the input of the three Transformer encoders respectively, and the global feature F containing the point cloud information of G1, G2, and G3 is output p , thereby extracting the characteristic information of residual cloud.
5. The method according to claim 4, characterized in that The Transformer encoder is mainly composed of a point embedding module and a superimposed self-attention mechanism module. The point embedding module consists of two connected CBRs with a structure of (128, 128). The point embedding module regards each point as a word in a sentence and places points with similar position semantics in a similar position in space. Taking G1(N, 3) as an example, two connected CBRs generate point embedding feature information F of [128×128] e , point embedding feature information F e It will serve as the input of the superimposed attention mechanism module. The superimposed self-attention mechanism module consists of four connected self-attention mechanisms with the same structure. The self-attention mechanism is mainly composed of CBR. After four connected self-attention mechanisms, each point is mapped to the dimension of [128-128-128-128]. These four dimensions are spliced together to form the global features of G1. Its size is [128×512]; similarly, through G2(N * ,3) The global features obtained and through G3(N ** ,3) The global features obtained The size of the splicing is [128×512], and Then the global feature vector F is obtained through maximum pooling. p , F p The dimension is [1×1536].
6. The method according to claim 4, characterized in that The feature decoder is mainly composed of a feedforward neural network, which mainly processes the global feature F p Decoding, the feature decoder converts the global feature vector F p As its input, it consists of a linear layer and CBR, and takes the global feature vector F of dimension [1×1536] p Decoded into [M×3] point cloud P Out , P out represents the point cloud of the missing part, and M is the number of missing points.
7. The method according to claim 1, characterized in that The discriminator consists of three CBRs, two LBRs and a linear layer. The discriminator divides the missing part of the point cloud P Out As input, three consecutive CBRs generate intermediate features of [64-128-256] dimensions, which are concatenated and then max-pooled to obtain [1×448] features. Then, two LBRs and a linear layer are used to obtain a scalar V, which represents the point cloud P of the missing part generated by the generator. out score.
8. The method according to claim 1, characterized in that The loss function is: Among them, L CD (S1, S2) is the point cloud P of the missing part out And cut part of the point cloud P GT The gap between them, S1 is the point cloud P of the missing part out , S2 is the cut part point cloud P GT , x is the point cloud P of the missing part out A point of y is the cut part of the point cloud P GT a certain point.
