A multi-scale greenhouse plant point cloud completion method based on inverse mapping of generative adversarial networks
Through GAN-based inverse mapping technology, a multi-scale greenhouse plant point cloud completion network is built, which solves the problems of missing point cloud data and poor completion effects in the existing technology, and achieves higher accuracy and generalization.
Patent Information
- Application Number
- CN202211060340.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-30
- Publication Date
- 2025-04-18
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art has severe missing point cloud data due to the performance limitations of lidar sensors or depth cameras when acquiring greenhouse plant point cloud data, especially in complex shapes and edge details, and poor generalization performance in real-world scanned data sets.
A multi-scale greenhouse plant point cloud completion method based on generative adversarial network (GAN) inverse mapping is adopted. By building a pre-trained network TreeGAN, a graph convolution and iterative furthest point sampling technology is used, and a multi-scale degradation mechanism and a multi-discriminator mechanism are combined to perform point cloud completion.
The accuracy of point cloud completion in complex shapes and edge details is improved, the generalization of real-world scanned data is enhanced, and better completion effect is achieved.
Smart Images
Figure CN115439490B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of greenhouse plant point cloud completion, and in particular to a multi-scale greenhouse plant point cloud completion method based on generative adversarial network inverse mapping. Background Art
[0002] In the real world, in order to maximize benefits in the agricultural field, low-cost, low-precision lidar sensors or depth cameras are often used to obtain point cloud data. Due to their performance reasons, point cloud data is usually severely missing during acquisition. In actual applications, there are many occlusions, light reflections and other factors, which lead to incomplete point clouds.
[0003] With the rapid development of machine vision and image technology, 3D reconstruction technology has different application directions in more and more fields. Compared with 3D expressions such as voxels, point clouds can use smaller amounts of data to express models. 3D reconstruction technology based on target point clouds is increasingly widely used in various fields and is developing rapidly. However, its application in agriculture is relatively immature.
[0004] Traditional point cloud completion methods generally use 3D software based on the structural features of the target, or collect multiple point cloud images from different perspectives for stitching and other operations. This method has a more obvious completion effect on targets with fixed structures. Early deep learning methods relied too much on datasets with incomplete and complete paired data. This deep learning method can well complete the point cloud in the dataset, but it does not perform well in real-world scanned datasets, and has poor generalization, and the completion results lack details. Summary of the invention
[0005] In order to solve the above-mentioned problems, the present invention provides a multi-scale greenhouse plant point cloud completion method based on generative adversarial network inverse mapping, which improves the accuracy of point cloud completion technology in complex shapes and the completion effect of edge details of complex shapes, and can also have good generalization for real-world scans with only residual point clouds.
[0006] The present invention provides a multi-scale greenhouse plant point cloud completion method based on generative adversarial network inverse mapping, and its technical solution is:
[0007] S1. Use 3D scanners to obtain complete point cloud data for greenhouse crops to construct data sets;
[0008] S2, cutting the complete point cloud data to different degrees, and saving the cut point cloud data as residual point cloud data;
[0009] S3, preprocess the data and construct the training data set, test data set and validation data set;
[0010] S4. Construct a pre-trained network for multi-scale greenhouse plant point cloud completion based on inverse mapping of generative adversarial networks;
[0011] S5, using the preprocessed point cloud data as the input of a multi-scale greenhouse plant point cloud completion pre-trained network based on inverse mapping of a generative adversarial network, to generate a generator and a discriminator containing semantic information and structural information of the greenhouse plant point cloud;
[0012] S6. Construct a multi-scale greenhouse plant point cloud completion network based on the inverse mapping of the generative adversarial network;
[0013] S7. Use the point cloud completion network to complete the incomplete greenhouse plant point cloud data and obtain the completion result.
[0014] Furthermore, based on a handheld 3D scanner, the 3D structure of greenhouse crops is reconstructed using the 3D scanner to obtain a complete 3D structural map of the greenhouse crops, and the 3D structural map obtained by scanning is converted into a point cloud using a written script.
[0015] Furthermore, the complete point cloud obtained by the 3D scanner is cut to different degrees, and 25%, 50%, and 75% of the original complete point cloud are deleted to form the residual point cloud data sets of M(N, 3), M1(N1, 3), and M2(N2, 3). N represents the number of residual point clouds with a loss of 25%, N1 represents the number of residual point clouds with a loss of 50%, and N2 represents the number of residual point clouds with a loss of 75%.
[0016] Furthermore, the data in the incomplete data set and the complete data set are preprocessed, the origin of all point cloud data is transferred to the center of the circle as the origin, and the values of all data are limited to the range of [-1, 1]. The preprocessed incomplete point cloud data sets M, M1, and M2 are respectively combined with the preprocessed complete point cloud data set to form three sets of training data sets, test data sets, and verification data sets, and the data sets are divided into training sets and test sets in a ratio of 8:2, where the data of the verification set is the same as the data of the test set, and all data formats are converted into h5 files.
[0017] Furthermore, the idea of graph convolution is used to construct a tree-structured graph convolution (TreeGAN), which includes a generator and a discriminator. The generator uses complete point cloud data as input. The output is a complete point cloud. The discriminator distinguishes between the real point cloud and the generated point cloud, and optimizes the generator to produce a point cloud that is closer to the real one.
[0018] Furthermore, a set of point cloud data containing only greenhouse plants is used as the input of the point cloud completion pre-trained network, and the pre-trained network is used to train a GAN (including a generator and a discriminator) with semantic feature vectors and structural feature vectors of greenhouse plants.
[0019] Furthermore, based on the pre-trained GAN generator, the residual point cloud data is mapped into the latent space of the GAN generator, so that the generator generates an initial complete point cloud shape.
[0020] Furthermore, based on the initial complete point cloud shape generated by the generator, the initial complete point cloud shape is downsampled at different resolutions through iterative farthest point sampling (IFPS) to obtain downsampled point clouds that can reflect the structural characteristics of the initial complete point cloud at different scales.
[0021] Furthermore, a multi-scale degradation mechanism (Mk-mask) simultaneously receives point clouds at different scales, and utilizes the degradation mechanism to degenerate the point clouds at different scales into point cloud shapes similar to the residual point cloud.
[0022] Furthermore, the total loss function of the generated network is defined as the multi-stage completion loss, which measures the degraded point cloud x out With incomplete input x in The degree of proximity of the points in the space between them, the multi-scale feature matching loss (Multi-scale Feature Distance), measures the degraded point cloud x out With incomplete input x in The degree of matching between the feature values.
[0023] Furthermore, the incomplete greenhouse plant point cloud is input into the point cloud completion network to complete the incomplete point cloud.
[0024] The beneficial effects of the present invention are as follows: compared with the traditional method, the present invention uses an unsupervised learning GAN inverse mapping method, uses a pre-trained network to take the complete point cloud as input to train a set of generators and discriminators containing greenhouse plant features, and maps the incomplete input to the pre-trained generator to generate a complete point cloud. Then, a multi-layer degradation mechanism and a multi-discriminator mechanism are used to back-transmit fine-tuning parameters to the generator, so that the complete point cloud generated by the generator can focus on the completion of the overall structure while taking care of the details. The use of unsupervised methods can have better generalization and can have better completion effects when dealing with incomplete data obtained by real-world scans. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 This is a flow chart of a multi-scale greenhouse plant point cloud completion method based on generative adversarial network inverse mapping;
[0026] Figure 2 This is the network structure diagram of the pre-trained network (TreeGAN);
[0027] Figure 3 Schematic diagram of the Iterative Farthest Point Sampling (IFPS) method;
[0028] Figure 4 Complete the network architecture diagram for point cloud;
[0029] Figure 5 Schematic diagram of multi-scale degradation mechanism
[0030] Figure 6 This is a multi-stage loss function architecture diagram
[0031] Figure 7 The effect diagram of the point cloud completion method DETAILED DESCRIPTION
[0032] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments to facilitate a clear understanding of the present invention, but they do not constitute a limitation on the present invention.
[0033] like Figure 1 As shown, a multi-scale greenhouse plant point cloud completion method based on inverse mapping of a generative adversarial network comprises the following steps:
[0034] Step 1: Use a 3D scanner to obtain complete point cloud data for greenhouse crops to build a dataset
[0035] Step 101: Based on a handheld 3D scanner, the 3D structure of the greenhouse crops is reconstructed by using the 3D scanner to obtain a complete 3D structure diagram of the greenhouse crops.
[0036] Step 2: Cut the complete point cloud data to different degrees and save the cut point cloud data as residual point cloud data.
[0037] Step 201, the complete point cloud obtained by the 3D scanner is cut to different degrees, and 25%, 50%, and 75% of the original complete point cloud are deleted to form residual point cloud data sets of M(N, 3), M1(N1, 3), and M2(N2, 3). N represents the number of residual point clouds with a loss of 25%, N1 represents the number of residual point clouds with a loss of 50%, and N2 represents the number of residual point clouds with a loss of 75%.
[0038] Step 3: Preprocess the data and build training, test, and validation datasets
[0039] Step 301: preprocess the data in the incomplete data set and the complete data set, transfer the origin of all point cloud data to the center of the circle as the origin, and limit the values of all data to the range of [-1, 1]. The preprocessed incomplete point cloud data sets M, M1, M2 and the preprocessed complete point cloud data set are respectively used to form three sets of training data sets, test data sets and verification data sets, and the data sets are divided into training sets and test sets in a ratio of 8:2, where the data of the verification set is the same as the data of the test set, and all data formats are converted into h5 files.
[0040] Step 4: Constructing a multi-scale greenhouse plant point cloud completion pre-training network based on the inverse mapping of the generative adversarial network
[0041] Step 401: Use the idea of graph convolution to construct a tree-structured graph convolution (TreeGAN), which includes a generator and a discriminator. Figure 2 The generator takes a single point from a Gaussian distribution, Z∈R 96, as input. For each layer of the generator, GraphConv and Branching operations are performed, producing a set of l-th points p1. The points generated after performing GraphConv and Branching operations on each layer are stored and superimposed, where p L is the set of points in the final layer L, and n is the total number of points. Discriminating between real and generated point clouds optimizes the generator to produce points that are closer to the real thing.
[0042] Step 5: Use the preprocessed point cloud data as the input of the multi-scale greenhouse plant point cloud completion pre-trained network based on the inverse mapping of the generative adversarial network to generate a generator and discriminator containing the semantic information and structural information of the greenhouse plant point cloud.
[0043] Step 501: Use a set of point cloud data containing only greenhouse plants as input to a point cloud completion pre-trained network, and use the pre-trained network to train a GAN (including a generator and a discriminator) with semantic feature vectors and structural feature vectors of greenhouse plants.
[0044] Step 6: Figure 4 The multi-scale greenhouse plant point cloud completion network based on the inverse mapping of the generative adversarial network is constructed.
[0045] Step 601: Create a generator that can be dynamically fine-tuned. By mapping the incomplete point cloud data into the latent space of the GAN generator, the pre-trained generator can contain rich semantic and structural information of greenhouse plants. The generator searches for a latent vector that matches the incomplete input in the latent space and uses the latent vector to reconstruct the initial complete point cloud shape x. c , and use the back propagation parameters of the loss function to optimize the generator.
[0046] Step 602: The initial complete point cloud x generated by the generator c Through Figure 3 Iterative farthest point sampling (IFPS) c Sampling is performed separately as x c1 、x c2 、x c3 Point cloud, where x c1 The scale is [2048,3], x c2 The scale is [1024, 3], x c3 The scale is [512,3], and x c1 、x c2 、x c3 The three resolutions are used as the input of the multi-scale degradation mechanism, which can not only take into account the high- and low-level features at the same time, but also make the degraded part have both the overall structural information and the detailed structural information.
[0047] Step 603: Figure 5 , the x obtained by iterative farthest point sampling c1 、x c2 、x c3 The point clouds of three resolutions are input to the multi-scale degradation mechanism. In order to enhance the accuracy and stability of our degradation mechanism, we input the residual point cloud x in , using the idea of KNN and using Euclidean distance as the distance metric, the complete point cloud is gradually degraded into a point cloud x similar to the incomplete input out1 、x out2 and x out3 .
[0048] Step 604: Figure 6 The multi-stage loss function includes multi-stage reconstruction loss and multi-scale feature matching loss, where the multi-stage reconstruction loss (Multi-stage Completion Loss) measures the degraded point cloud x at multiple resolutions out With incomplete input x in The proximity of the midpoints in space
[0049]
[0050] where ε CD (x out , x in ) is the residual cloud x in Point cloud degraded with degradation mechanism x out The distance difference between them, a is the point cloud x degraded by the degradation mechanism out A point in the ina point in .
[0051] The incomplete input X is obtained by iterative farthest point sampling (IFPS). in Sampled as X in1 , X in2 and X in3 , which are incomplete input feature points of scale Ni*3 (Ni=2048, 1024, 512). And the point cloud x degraded by the multi-scale degradation mechanism out1 、x out2 and x out3 has the same scale as the incomplete input, so the multi-level reconstruction loss will be given by ε CD1 and ε which loads the hyperparameters α and β CD2 , ε CD3 The first term calculates the degenerate point cloud x out1 The detail points and input true residual cloud X in1 The second and third terms calculate the main structural point x out2 With the secondary structure point x out3 With the real incomplete input X in2 , X in3 The square distance.
[0052] T C =ε CD1 (x out1 ,X in1 )+αε CD2 (x out2 ,X in2 )+βε CD3 (x out3 ,X in3 )
[0053] Multi-scale feature matching loss (), using the discriminator (Descriminator) of the pre-trained GAN structure to design feature matching loss:
[0054] T D =||D 1 (x out1 )-D 1 (X in1 )|| 1 +||D 2 (x out2 )-D 2 (X in2 )|| 1 +||D 3 (x out3 )-D 3 (X in3 )|| 1
[0055] Where D 1 , D 2 and D 3 For the discriminator of the training segment, the idea of multiple discriminators is adopted to capture more geometric detail features at multiple scales.
[0056] Our overall loss function is:
[0057] T=h C T C +h D T D
[0058] The overall loss function is used for shape generation and complete shape reconstruction, where h C and h D is the weight of reconstruction loss and feature matching loss, following h C +h D =1.
[0059] The above shows and describes the basic principles, main features and advantages of the present invention. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claim involved.
[0060] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A multi-scale greenhouse plant point cloud completion method based on generative adversarial network inverse mapping, characterized in that: The method comprises the following steps: S1. Use 3D scanners to obtain complete point cloud data for greenhouse crops to construct data sets; S2, cutting the complete point cloud data to different degrees, and saving the cut point cloud data as residual point cloud data; S3, preprocess the data and construct the training data set, test data set and validation data set; S4. Construct a pre-trained network for multi-scale greenhouse plant point cloud completion based on inverse mapping of generative adversarial networks; Wherein, the S4 specifically includes: The idea of graph convolution is used to construct a tree-structured graph convolution, which includes a generator and a discriminator. The generator uses complete point cloud data as input and outputs a complete point cloud. The discriminator distinguishes between real point clouds and generated point clouds and optimizes the generator to produce point clouds that are closer to the real point cloud. S5, using the preprocessed point cloud data as the input of a multi-scale greenhouse plant point cloud completion pre-trained network based on inverse mapping of a generative adversarial network, to generate a generator and a discriminator containing semantic information and structural information of the greenhouse plant point cloud; S6. Construct a multi-scale greenhouse plant point cloud completion network based on the inverse mapping of the generative adversarial network; Wherein, the S6 specifically includes: Based on the pre-trained GAN generator, the residual point cloud data is mapped to the latent space of the GAN generator, so that the generator generates the initial complete point cloud shape; Based on the initial complete point cloud shape generated by the generator, the initial complete point cloud shape is downsampled at different resolutions by iterative farthest point sampling to obtain downsampled point clouds that can reflect the structural features of the initial complete point cloud at different scales; The multi-scale degradation mechanism simultaneously receives point clouds at different scales, and utilizes the degradation mechanism to degrade the point clouds at different scales into point cloud shapes similar to the residual point cloud; The total loss function of the generated network is defined as the multi-stage reconstruction loss, which measures the degraded point cloud With incomplete input The closeness of the points in the space between them, the multi-scale feature matching loss, and the measurement of the degraded point cloud With incomplete input The matching degree between the eigenvalues; S7. Use the point cloud completion network to complete the incomplete greenhouse plant point cloud data and obtain the completion result.
2. The multi-scale greenhouse plant point cloud completion method based on generative adversarial network inverse mapping according to claim 1 is characterized by: S1, using a 3D scanner to obtain complete point cloud data for greenhouse crops for constructing a data set, specifically includes: Based on a handheld 3D scanner, the 3D structure of greenhouse crops is reconstructed using the 3D scanner to obtain a complete 3D structure diagram of greenhouse crops, and the 3D structure diagram obtained by scanning is converted into a point cloud; Among them, for S2, cutting the complete point cloud data to different degrees and saving the cut point cloud data as residual point cloud data specifically includes: The complete point cloud obtained by the 3D scanner is cut to different degrees, and 25%, 50%, and 75% of the original complete point cloud are deleted to form the residual point cloud data sets of M (N, 3), M1 (N1, 3), and M2 (N2, 3), where N represents the number of residual point clouds with a loss of 25%, N1 represents the number of residual point clouds with a loss of 50%, and N2 represents the number of residual point clouds with a loss of 75%; The S3, preprocessing data and constructing training data set, test data set and validation data set, specifically includes: The data in the incomplete data set and the complete data set are preprocessed, the origin of all point cloud data is translated to the center of the circle as the origin, and the values of all data are limited to the range of [-1, 1]; the preprocessed incomplete point cloud data sets M, M1, and M2 are respectively combined with the preprocessed complete point cloud data set to form three sets of training data sets, test data sets, and verification data sets, and the data sets are divided into training sets and test sets in a ratio of 8:2, where the data of the verification set is the same as the data of the test set, and all data formats are converted into h5 format files.
3. The multi-scale greenhouse plant point cloud completion method based on generative adversarial network inverse mapping according to claim 1 is characterized by: The S5 uses the preprocessed point cloud data as the input of the multi-scale greenhouse plant point cloud completion pre-training network based on the inverse mapping of the generative adversarial network, and generates a generator and a discriminator containing the semantic information and structural information of the greenhouse plant point cloud, specifically including: A set of point cloud data containing only greenhouse plants is used as the input of the point cloud completion pre-trained network, and the pre-trained network is used to train the GAN generator and GAN discriminator which contain the semantic feature information and structural feature information of greenhouse plants in the latent space.
4. The multi-scale greenhouse plant point cloud completion method based on generative adversarial network inverse mapping according to claim 1 is characterized by: Based on the pre-trained GAN generator, the residual point cloud data is mapped to the latent space of the GAN generator so that the generator generates an initial complete point cloud shape, specifically including: Create a generator that can be dynamically fine-tuned. By mapping the incomplete point cloud data into the latent space of the GAN generator, the pre-trained generator can contain the semantic and structural information of the greenhouse plants. The generator searches for a latent vector that matches the incomplete input in the latent space and uses the latent vector to reconstruct the initial complete point cloud shape. , and use the back propagation parameters of the loss function to optimize the generator; Among them, the initial complete point cloud shape generated by the generator , by iteratively sampling the farthest point, the initial complete point cloud shape is sampled at different resolutions to obtain downsampled point clouds that can reflect the structural characteristics of the initial complete point cloud at different scales, including: The initial complete point cloud generated by the generator By iteratively sampling the farthest point, Sampling is performed separately , , point cloud, where The scale is [2048,3], The scale is [1024, 3], The scale is [512,3], and , , The three resolutions are used as inputs of the multi-scale degradation mechanism, taking into account the high- and low-level features while making the degraded part have both overall structural information and detailed structural information; The multi-scale degradation mechanism simultaneously receives point clouds at different scales, and uses the degradation mechanism to degenerate point clouds at different scales into point cloud shapes similar to the residual point cloud, specifically including: The farthest point sampling obtained by iterative , , Point clouds with three resolutions are input to the multi-scale degradation mechanism to input residual point clouds. , using the idea of KNN and using Euclidean distance as the distance metric, the complete point cloud is gradually degraded into a point cloud similar to the incomplete input , and ; The total loss function of the generated network is defined as the multi-stage reconstruction loss, which measures the degraded point cloud With incomplete input The closeness of the points in the space between them, the multi-scale feature matching loss, and the measurement of the degraded point cloud With incomplete input The degree of matching between the characteristic values includes: Multi-stage reconstruction loss to measure the degraded point cloud at multiple resolutions With incomplete input The proximity of the midpoints in space: in For the residual cloud Degraded point cloud with degradation mechanism The distance difference between is the point cloud degraded by the degradation mechanism At a certain point in It's a residual cloud A point in The incomplete input is converted into Sampling is , and , , and It is the incomplete input feature points of scale Ni*3 (Ni=2048, 1024, 512), and the point cloud degraded by the multi-scale degradation mechanism , and has the same scale as the defective input, so that the multi-level reconstruction loss is and loading hyperparameters α and β , Composition; the first item calculates the degenerate point cloud The detail points and input true residual point cloud The second and third terms calculate the main structural points With secondary structural points With real incomplete input , The square distance of Multi-scale feature matching loss, using the discriminator of the pre-trained GAN structure to design feature matching loss: in , and For the discriminator of the training segment, the idea of multiple discriminators is adopted to capture more geometric detail features at multiple scales; The overall loss function is: The overall loss function is used for shape generation and complete shape reconstruction, where and is the weight of reconstruction loss and feature matching loss, following + =1.