A crop organ point cloud segmentation method based on dynamic graph convolution and transfer learning

Abstract

The application discloses a crop organ point cloud segmentation method based on dynamic graph convolution and transfer learning, and belongs to the technical field of three-dimensional point cloud instance segmentation. The method firstly designs a brand-new point cloud instance segmentation network PointDGCNet based on graph convolution operation. Then, a transfer learning method is adopted to fine-tune parameters of a pre-training segmentation model, so that an instance segmentation model suitable for monocotyledon plant point cloud is obtained, the segmentation accuracy of monocotyledon plant point cloud is effectively improved (average accuracy 68.98%, average intersection over union 66.53%), the training time of the model is greatly shortened, and overfitting is avoided. After training, a resampling method is further adopted to improve the segmentation accuracy (average accuracy 72.10%). Finally, it is proved through further experiments that the method has certain generalization ability for monocotyledon plant point cloud, such as corn and wheat.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional point cloud instance segmentation, and in particular relates to a crop organ point cloud segmentation method based on dynamic graph convolution and transfer learning. Background Art

[0002] Currently, the amount of food and biofuels needed for human survival and production is increasing dramatically, and resource shortages are widespread worldwide. To meet the needs of human activities and ensure food supply within existing resources, it is necessary to identify and quantify various plant characteristics and cultivate more productive plant varieties. Plant phenomics analyzes the interaction between a plant's intrinsic genotype and the external environment to determine plant growth. Using computer technology to accurately and rapidly analyze phenotypes, this approach improves understanding of complex plant genetic traits, accelerates breeding, and accelerates the development of precision agriculture. It has become a key research area in biology.

[0003] Generally speaking, data for plant phenotyping primarily comes from plant organs, such as leaves, rhizomes, and fruits. Therefore, accurate and rapid segmentation of plant organs is one of the current challenges in plant phenomics. Initially, a significant amount of manual labor was required to collect plant organ data, and some traits even required destructive sampling of growing plants, which is impractical under current circumstances. Other work has used clustering, region growing, or convolutional neural networks to automatically extract phenotypic features based on two-dimensional image data. However, two-dimensional images can only record features of plants at fixed angles, losing spatial structural information. In recent years, the rapid development of three-dimensional sensor technology has accelerated the progress of three-dimensional plant phenotyping. Analysis methods based on three-dimensional plant point sets in space have continued to emerge. These works can not only fully describe the spatial distribution of plants, but also extract statistically significant phenotypic data.

[0004] Due to the inherent disorder and irregularity of crop 3D point cloud data, current segmentation methods are primarily based on deep neural networks. However, mainstream neural networks often fail to accurately capture the geometric features between points, and local features are crucial for accurate crop point cloud segmentation. Graph convolution, a method based on graph theory, is naturally adapted to point cloud data with irregular spatial distributions. It leverages the topological relationships between points to calculate the edge features of each point relative to its neighbors, thereby improving the neural network's ability to perceive the local features of the point cloud. Furthermore, graph convolution ensures permutation invariance of the point cloud and can update the structure in different feature spaces to adapt to the feature data in the neural network.

[0005] To achieve more accurate point cloud organ segmentation results, training neural networks typically requires a very large amount of data. Manually labeling this massive amount of data is prohibitively laborious, which directly leads to the limited generalization capabilities of existing models. Transfer learning methods can learn the similarities between the source and target domains and, by fine-tuning parameters, maximize the use of existing pre-trained models for analyzing 3D plant data that has not undergone extensive training. Therefore, transfer learning methods can achieve the same or even better results as existing models using less data for training. Summary of the Invention

[0006] To address the low accuracy and generalization issues of current crop point cloud segmentation models, this paper proposes a crop organ point cloud segmentation method based on dynamic graph convolution and transfer learning. This method designs a novel crop organ point cloud segmentation network, PointDGCNet, based on dynamic graph convolution. This network is used to train a sorghum point cloud segmentation model, and its parameters are fine-tuned to produce a monocot point cloud segmentation model. This method addresses the overfitting, long training time, and slow convergence of traditional neural network models when limited data is available, thereby improving the segmentation efficiency and generalization capabilities of the plant point cloud organ segmentation model.

[0007] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A crop organ point cloud segmentation method based on dynamic graph convolution and transfer learning, the method comprising:

[0009] Step 1: Data acquisition and dataset creation

[0010] The dataset is divided into two parts: (1) 3D scanners were used to non-contact scan sorghum plants at the seedling stage under five different environments, obtaining 3D point clouds of sorghum with micron-level resolution; (2) 3D scanners were used to non-contact scan monocotyledonous plants at the seedling stage under six environments, obtaining 3D point clouds of corn with sub-millimeter-level resolution;

[0011] Step 2: Establish PointDGCNet point cloud segmentation network

[0012] The PointDGCNet network adopts an encoder-decoder architecture and is designed based on graph convolution operations. The sampling convolution block in the network consists of an edge convolution layer and a sampling grouping layer.

[0013] The edge convolution layer constructs a graph data structure based on the input point cloud information, where each independent point is regarded as a vertex in the graph, and the relationship operation h between each vertex and its neighboring points is defined as edge convolution. The relationship function h is defined as follows:

[0014] h(x i ,x j)=max(mlp(x i -x j ))

[0015] Among them, x i For the focus, x j The neighboring points around the point of interest;

[0016] The sampling grouping layer uses the FPS downsampling algorithm to select some representative points PC' from the input point cloud PC, and uses the k-nearest neighbor algorithm to aggregate the features of neighboring points into the point set PC';

[0017] The feature encoder of the PointDGCNet network consists of three cascaded sampling convolution blocks, which are responsible for high-level feature abstraction of the input point cloud; the feature decoder consists of three cascaded upsampling interpolation layers, which are responsible for gradually restoring the abstract high-dimensional features to each independent point; finally, the average pooling layer is used to output the point cloud segmentation result; the formula for the average pooling operation is

[0018] f({x1,x2,…,x k})=avg(w(x1),w(x2),…,w(x k ))

[0019] Among them, w is the feature abstraction operation;

[0020] The sorghum point cloud data is input into the PointDGCNet network for neural network training. After the training is completed, a sorghum point cloud instance segmentation model is obtained;

[0021] Step 3: Parameter fine-tuning

[0022] Fine-tune the parameters of the sorghum model trained in step 2 to adapt to the monocot data. Specifically, fine-tune the parameters of the neural network layer related to label prediction to obtain a suitable monocot point cloud segmentation model;

[0023] The output layer of the pre-trained network PointDGCNet is modified as follows:

[0024] output=conv2'(dropout(relu(batch_normal(conv1(points)))))

[0025] Among them, conv2' is the modified two-dimensional convolution layer, dropout is the random inactivation layer, relu is the activation function, batch_normal is the batch normalization layer, conv1 is the one-dimensional convolution layer, and points is the abstract feature output by the fp1' layer.

[0026] Furthermore, in step 1, the point cloud dataset is normalized, the point cloud coordinate scale is fixed within [-1, 1], and the point cloud center is moved to the origin of the coordinate axis, (x i ,y i ,z i ) represents the original coordinates of the i-th point in the point cloud, center represents the center of the point cloud, bias i Represents the coordinate deviation between the coordinate point and the center of the point cloud, m represents the scaling ratio, (x i ',y i ',z i ') represents the new coordinates of the i-th point after normalization. The specific formula is as follows:

[0027]

[0028]

[0029]

[0030]

[0031] Where x is the average of all x, y is the average of all y, z is the average of all z, and n is the number of independent points in the point cloud of a single plant;

[0032] Data augmentation is performed on the normalized point cloud above. First, the original point cloud P is randomly downsampled to 10,000 points to obtain P′. Then, the coordinates of each point in P′ are perturbed in a random direction with a perturbation distance of 0.001. The above operation is repeated 10 times for each point cloud in the dataset to obtain a well-labeled plant point cloud dataset S.

[0033] Furthermore, in step 2, the hyperparameters of the PointDGCNet network are set as follows: batch size is 32, epochs are set to 100, number of point cloud samples is set to 2048, initial learning rate is 0.001, and Adam optimization algorithm is used.

[0034] Furthermore, in step 3, the specific method of fine-tuning the neural network layer parameters related to label prediction is as follows:

[0035] (1) Extract all parameters from the sorghum pre-trained segmentation model and fix the gradients so that they do not participate in training;

[0036] (2) Replace the fully connected layer fp1 and the two-dimensional convolutional layer conv2 of the feature decoder in the pre-trained model with the fp1′ and conv2′ layers suitable for monocot point cloud data;

[0037] (3) Changing the one-hot encoding in the sorghum pre-trained segmentation model to the one-hot encoding of the monocot point cloud data;

[0038] (4) Randomly initialize the weights of the fp1′ and conv2′ layers, and set the bias term to 0.

[0039] Furthermore, the method further comprises step 4: resampling

[0040] The original monocot point cloud was resampled using two ratios. Specifically, the parameter r1 was set as the ratio of the number of skeleton points to the total number of point clouds, and the parameter r2 was set as the ratio of the number of internal points to the total number of point clouds. The parameters r1 and r2 were modified in steps of 0.1, and the samples were resampled on the monocot point cloud test set. The segmentation accuracy was then retested using the transfer learning model in step 3.

[0041] The beneficial effects of the present invention compared to the prior art are as follows: First, a new point cloud instance segmentation network PointDGCNet is designed based on graph convolution operations. Then, the parameters of the pre-trained segmentation model are fine-tuned by the transfer learning method to obtain an instance segmentation model suitable for monocotyledonous plant point clouds, which effectively improves the segmentation accuracy of monocotyledonous plant point clouds (average accuracy of 68.98%, average intersection-over-union ratio of 66.53%), while greatly shortening the training time of the model and avoiding the occurrence of overfitting. After training, the resampling method is used to further improve the segmentation accuracy (average accuracy of 72.10%). Finally, further experiments have shown that this method has a certain generalization ability for monocotyledonous plant point clouds, such as corn, wheat, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is the overall flow chart of the present invention;

[0043] Figure 2 This is the PointDGCNet network architecture diagram of the present invention;

[0044] Figure 3 This is a diagram of the fine-tuning network architecture of the present invention;

[0045] Figure 4 This is a partial sorghum point cloud data diagram used in the present invention;

[0046] Figure 5 This is a partial corn point cloud data map used in the present invention;

[0047] Figure 6 This is the segmentation effect diagram of corn point cloud using the transfer learning method of the present invention;

[0048] Figure 7 This is a scatter plot of the accuracy obtained after resampling at different ratios in the present invention;

[0049] Figure 8This is the segmentation effect diagram of the transfer learning method in the present invention applied to the wheat point cloud. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0051] Example 1:

[0052] A crop organ point cloud segmentation method based on dynamic graph convolution and transfer learning. The overall process is as follows Figure 1 As shown, it is characterized in that: the method is:

[0053] Step 1: Data acquisition and dataset creation

[0054] The dataset is divided into two parts: (1) Using a 3D scanner to perform contactless scanning of sorghum plants at the seedling stage under five different environments (ambient light, shadow, high temperature, strong light and drought), 111 micron-level resolution sorghum 3D point clouds were obtained ( Figure 4 ); (2) A 3D scanner was used to perform contactless scanning of corn plants at the seedling stage under six different environments, and 47 high-definition 3D point clouds of corn plants were obtained ( Figure 5 ); In order to illustrate the diversity of data, multiple environments are used. Because crops grown in different environments have large differences in morphological structure, data from multiple environments can illustrate the robustness of this method.

[0055] The point cloud data obtained above were manually annotated at the organ level. Since even experienced professionals cannot distinguish the boundaries between the stems and leaves in the sorghum seedling stage, the cylindrical parts in the point cloud were marked as stems and the non-cylindrical parts were marked as leaves.

[0056] Normalize the above point cloud dataset, fix the point cloud coordinate scale within [-1,1], and move the point cloud center to the origin of the coordinate axis, (x i ,y i ,z i ) represents the original coordinates of the i-th point in the point cloud, center represents the center of the point cloud, bias i Represents the coordinate deviation between the coordinate point and the center of the point cloud, m represents the scaling ratio, (x i ',y i ',z i ') represents the new coordinates of the i-th point after normalization. The specific formula is as follows:

[0057]

[0058]

[0059]

[0060]

[0061] is the average of all x, is the mean of all y, is the average of all z;

[0062] Data augmentation is performed on the normalized point cloud. First, the original point cloud P is randomly downsampled to 10,000 points to obtain P′. Then, the coordinates of each point in P′ are perturbed in a random direction with a perturbation distance of 0.001. The above operation is repeated 10 times for each point cloud in the dataset, resulting in a well-labeled plant point cloud dataset S, which includes 1,110 sorghum point clouds and 470 corn point clouds.

[0063] Step 2: Self-created PointDGCNet point cloud segmentation network

[0064] This paper first designs a new crop point cloud segmentation network PointDGCNet based on graph convolution ( Figure 2 ), and based on this training, a pre-trained model for sorghum point cloud segmentation was obtained. The PointDGCNet network adopts an encoder-decoder architecture and is designed based on graph convolution operations. The most important module in the network is the sampling convolution block, which consists of an edge convolution layer and a sampling grouping layer. Specifically, the edge convolution layer constructs a graph data structure based on the input point cloud information, where each independent point is regarded as a vertex in the graph, and the relationship operation h between each vertex and its neighboring points is defined as an edge convolution. The relationship function h in this invention is defined as follows:

[0065] h(x i ,x j )=max(mlp(x i -x j ))

[0066] Among them, x i For the focus, x j The neighboring points around the focus point; it should be noted that this neighbor relationship between points is not limited to Euclidean space; as the network continues to deepen, this neighbor search is also applicable to high-dimensional feature spaces; the sampling grouping layer uses the FPS downsampling algorithm to select some representative points PC' from the input point cloud PC, and uses the k-nearest neighbor algorithm to aggregate the features of the neighboring points into the point set PC'; while reducing the number of points in the point cloud, it also increases the dimension of the features contained in each point;

[0067] The feature encoder of the PointDGCNet network consists of three cascaded sampling convolution blocks, which are responsible for high-level feature abstraction of the input point cloud; the feature decoder is relatively simple, consisting of three cascaded upsampling interpolation layers, which are responsible for gradually restoring the abstract high-dimensional features to each individual point; finally, an average pooling layer is used to output the point cloud segmentation result. The formula for the average pooling operation is as follows, where w represents the feature abstraction operation:

[0068] f({x1,x2,...,x k})=avg(w(x1),w(x2),...,w(x k ))

[0069] Sorghum point cloud data was fed into the PointDGCNet network for neural network training. The specific experimental setup was as follows: the data was split into training, validation, and test sets at an 8:1:1 ratio, with each instance containing at least three organs and a maximum of six. Computational experiments were conducted on a single server with a 12-core, 20-thread CPU, 64GB of memory, and an Nvidia GeForce RTX 3090Ti GPU. The experiments were run using the PyTorch framework on a Windows operating system. During the training phase, the network input consisted only of 3D coordinates and the corresponding point annotations, with a maximum of 4096 point cloud samples. The training batch size was set to 48, the initial learning rate to 0.5, and the network was optimized using stochastic gradient descent. Due to the small data size, training was performed for 50 epochs, with the learning rate halved every 10 epochs. The weight decay was set to 0.01, and the momentum was set to 0.9. After training, a sorghum point cloud instance segmentation model was obtained.

[0070] Alternatively, the point clouds of 1110 well-labeled sorghum plants are divided into training and test sets in a ratio of 9:1. The maximum number of organs in all sorghum point clouds is 6. The hyperparameters of the PointDGCNet network are set as follows: the batch size is 32, 100 epochs are set, the number of point cloud samples is set to 2048, the initial learning rate is 0.001, and the Adam optimization algorithm is used. Finally, an instance segmentation model of the sorghum point cloud is obtained with an average segmentation accuracy of 77.23% and an average intersection-over-union ratio of 73.29%.

[0071] Table 1 Comparison of segmentation accuracy between PointDGCNet and PointNet++ in this invention

[0072] Model Training time (seconds) Average intersection-merge ratio (%) Accuracy (%) PointNet++ (Gaoliang) 2702 68.76 62.17 PointNet++ (Corn) 1522 62.23 63.11 PointDGCNet (Gaoliang) 2901 73.29 77.23 PointDGCNet (Corn) 2066 66.53 68.98

[0073] Step 3: Parameter fine-tuning

[0074] Since the amount of corn point cloud data is small, it is unrealistic to train a model with tens of millions of parameters from scratch, and overfitting is inevitable. Figure 3 The architecture of the feature decoder was modified to modify the network layers related to point classification, so that the number of input and output channels of each layer, such as the one-dimensional and two-dimensional convolution, matched the corn point cloud. The parameters of the trained sorghum model were fine-tuned to adapt to the corn data and reduce the time and labor costs of training.

[0075] In the sorghum organ segmentation model, point-level feature extraction is achieved through multi-channel one-dimensional and two-dimensional convolution operations. Since the features perceived by the low-level feature extraction layer on any three-dimensional point cloud object are similar, the present invention only fine-tunes the parameters of the neural network layer related to label prediction to quickly obtain a suitable corn point cloud segmentation model.

[0076] The specific steps are as follows:

[0077] (1) Extract all parameters from the sorghum pre-trained segmentation model and fix the gradients so that they do not participate in training;

[0078] (2) Replace the fully connected layer fp1 and the two-dimensional convolutional layer conv2 of the feature decoder in the pre-trained model with the fp1′ and conv2′ layers suitable for corn data;

[0079] (3) Changing the one-hot encoding in the sorghum pre-trained segmentation model to the one-hot encoding of the corn data;

[0080] (4) Randomly initialize the weights of the fp1′ and conv2′ layers, and set the bias term to 0;

[0081] The output layer of the pre-trained network PointDGCNet is modified as follows, where points is the abstract feature output by the fp1′ layer, conv1 is the one-dimensional convolution layer, batch_normal is the batch normalization layer, relu is the activation function, dropout is the random inactivation layer, and conv2′ is the modified two-dimensional convolution layer.

[0082] output=conv2'(dropout(relu(batch_normal(conv1(points)))))

[0083] Table 2 Segmentation results of corn point cloud data trained by transfer learning method and PointDGCNet respectively

[0084] Model Training time (seconds) Average intersection-merge ratio (%) Accuracy (%) PointDGCNet 1004 66.53 68.98 Transfer Learning Model 940 66.11 72.10

[0085] Step 4: Resampling

[0086] Figure 6This is the segmentation effect diagram of corn point cloud. In order to further improve the accuracy of organ instance segmentation, the present invention uses two ratios to resample the original corn point cloud. Specifically, the parameter r1 is set as the ratio of the number of skeleton points to the total number of point cloud, and the parameter r2 is set as the ratio of the number of internal points to the total number of point cloud. The parameters r1 and r2 are modified with a step size of 0.1, and the corn point cloud test set is resampled and the segmentation accuracy is retested. The data shows that ( Figure 7 ), the resampled point cloud can improve the segmentation accuracy by up to 0.77%. Figure 8 The following figure shows the results of the method applied to wheat point cloud data. It can be seen that the method also has a certain degree of generalization ability on monocotyledonous crops. It can be applied not only to the corn point cloud in the above experiment, but also to wheat, and achieved good results.

Claims

1. A crop organ point cloud segmentation method based on dynamic graph convolution and transfer learning, characterized by: The method is: Step 1: Data acquisition and dataset creation The dataset is divided into two parts: (1) 3D scanners were used to non-contact scan sorghum plants at the seedling stage under five different environments, obtaining 3D point clouds of sorghum with micron-level resolution; (2) 3D scanners were used to non-contact scan monocotyledonous plants at the seedling stage under six environments, obtaining 3D point clouds of corn with sub-millimeter-level resolution; Step 2: Establish PointDGCNet point cloud segmentation network The PointDGCNet network adopts an encoder-decoder architecture and is designed based on graph convolution operations. The sampling convolution block in the network consists of an edge convolution layer and a sampling grouping layer. The edge convolution layer constructs a graph data structure based on the input point cloud information, where each independent point is regarded as a vertex in the graph, and the relationship operation h between each vertex and its neighboring points is defined as edge convolution. The relationship function h is defined as follows: h(x i ,x j )=max(mlp(x i -x j )) Among them, x i For the focus, x j The neighboring points around the point of interest; The sampling grouping layer uses the FPS downsampling algorithm to select some representative points PC' from the input point cloud PC, and uses the k-nearest neighbor algorithm to aggregate the features of neighboring points into the point set PC'; The feature encoder of the PointDGCNet network consists of three cascaded sampling convolution blocks, which are responsible for high-level feature abstraction of the input point cloud; the feature decoder consists of three cascaded upsampling interpolation layers, which are responsible for gradually restoring the abstract high-dimensional features to each independent point; finally, the average pooling layer is used to output the point cloud segmentation result; the formula for the average pooling operation is f({x1,x2,…,x k })=avg(w(x1),w(x2),...,w(x k )) Among them, w is the feature abstraction operation; The sorghum point cloud data is input into the PointDGCNet network for neural network training. After the training is completed, a sorghum point cloud instance segmentation model is obtained; Step 3: Parameter fine-tuning Fine-tune the parameters of the sorghum model trained in step 2 to adapt to the monocot data. Specifically, fine-tune the parameters of the neural network layer related to label prediction to obtain a suitable monocot point cloud segmentation model; The output layer of the pre-trained network PointDGCNet is modified as follows: output=conv2'(dropout(relu(batch_normal(conv1(points))))) Among them, conv2' is the modified two-dimensional convolution layer, dropout is the random inactivation layer, relu is the activation function, batch_normal is the batch normalization layer, conv1 is the one-dimensional convolution layer, and points is the abstract feature output by the fp1' layer.

2. The crop organ point cloud segmentation method based on dynamic graph convolution and transfer learning according to claim 1, characterized in that: In step 1, the point cloud dataset is normalized, the point cloud coordinate scale is fixed within [-1, 1], and the point cloud center is moved to the origin of the coordinate axis, (x i ,y i ,z i ) represents the original coordinates of the i-th point in the point cloud, center represents the center of the point cloud, bias i Represents the coordinate deviation between the coordinate point and the center of the point cloud, m represents the scaling ratio, (x i ',y i ',z i ') represents the new coordinates of the i-th point after normalization. The specific formula is as follows: in, is the average of all x, is the mean of all y, is the average of all z, n is the number of independent points in the point cloud of a single plant; Data augmentation is performed on the normalized point cloud above. First, the original point cloud P is randomly downsampled to 10,000 points to obtain P′. Then, the coordinates of each point in P′ are perturbed in a random direction with a perturbation distance of 0.

001. The above operation is repeated 10 times for each point cloud in the dataset to obtain a well-labeled plant point cloud dataset S.

3. The crop organ point cloud segmentation method based on dynamic graph convolution and transfer learning according to claim 1, characterized in that: In step 2, the PointDGCNet network hyperparameters are set as follows: batch size is 32, epochs are set to 100, number of point cloud samples is set to 2048, initial learning rate is 0.001, and Adam optimization algorithm is used.

4. The crop organ point cloud segmentation method based on dynamic graph convolution and transfer learning according to claim 1, characterized in that: In step 3, the specific steps for fine-tuning the parameters of the neural network layer related to label prediction are as follows: (1) Extract all parameters from the sorghum pre-trained segmentation model and fix the gradients so that they do not participate in training; (2) Replace the fully connected layer fp1 and the two-dimensional convolutional layer conv2 of the feature decoder in the pre-trained model with the fp1′ and conv2′ layers suitable for monocot point cloud data; (3) Changing the one-hot encoding in the sorghum pre-trained segmentation model to the one-hot encoding of the monocot point cloud data; (4) Randomly initialize the weights of the fp1′ and conv2′ layers, and set the bias term to 0.

5. The crop organ point cloud segmentation method based on dynamic graph convolution and transfer learning according to any one of claims 1 to 4, characterized in that: The method further comprises step 4: resampling The original monocot point cloud was resampled using two ratios. Specifically, the parameter r1 was set as the ratio of the number of skeleton points to the total number of point clouds, and the parameter r2 was set as the ratio of the number of internal points to the total number of point clouds. The parameters r1 and r2 were modified in steps of 0.1, and the samples were resampled on the monocot point cloud test set. The segmentation accuracy was then retested using the transfer learning model in step 3.

Images