Plant organ point cloud segmentation method based on attention mechanism and graph convolution

Abstract

A plant organ point cloud segmentation method based on attention mechanism and graph convolution belongs to the technical field of three-dimensional point cloud instance segmentation. A double-branch parallel instance segmentation network TRGCN based on point attention mechanism and spatial graph convolution directly inputs three-dimensional point cloud, the double-branch respectively focuses on local feature extraction and global feature extraction, and fuses the two kinds of features through a T-G feature coupling layer. Taking tomato, corn, tobacco, sorghum and wheat five kinds of plant point cloud data as research objects, the double-branch parallel neural network architecture TRGCN can simultaneously capture the local features and global features of the point cloud, is used for training a high-robustness instance segmentation model, can improve the segmentation precision of plant point cloud, has good generalization ability, and can provide good data support for fast, efficient and accurate plant phenotype analysis.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional point cloud instance segmentation, and specifically relates to a dual-branch parallel plant organ point cloud segmentation method based on attention mechanism and spatial graph convolution. Background Art

[0002] With the widespread adoption of LiDAR devices and the emergence of various consumer-grade depth sensors, point cloud data is increasingly being used in various fields, such as robotics, autonomous driving, and urban planning. In phenotyping research, three-dimensional point clouds, as low-resolution representations of the real world, have become the most direct and effective data format for studying plant structure and morphology. Many studies have used three-dimensional plant structures for organ segmentation, growth monitoring, and variety evaluation. Points in a three-dimensional coordinate system, the most basic unit of a point cloud, are similar to pixels in a two-dimensional image, but can accommodate more high-dimensional semantic information. In phenotyping studies, the morphological structure of plant organs is a highly intuitive and important trait, reflecting a plant's adaptability to external conditions and growth status, such as photosynthesis efficiency and water absorption efficiency. Plant organ point cloud segmentation, the process of semantically segmenting plants into different organs (such as stems, leaves, and fruits), is the foundation for subsequent in-depth understanding of point cloud data and is crucial for understanding plant functional structure. It is currently a challenging research direction.

[0003] Traditional plant point cloud segmentation algorithms require manual pre-description of features, making the segmentation process complex and tedious. With the advent of the big data era, traditional processing methods are unable to meet the demands of fast and accurate analysis. Therefore, the demand for automated segmentation methods is growing. With the rapid growth of computer graphics processing unit (GPU) performance, deep learning, a leading AI technology, has been successfully applied to various two-dimensional vision problems. However, due to the disorder and complexity of point clouds in spatial aggregation, the application of deep learning methods to point clouds still faces many challenges. Convolutional neural networks (CNNs), which have performed well in visual segmentation tasks, use shared kernel convolution to extract features, improving model efficiency. CNNs' inherent translation invariance enables more precise capture of local features. However, CNNs typically have a small receptive field, are relatively weak in capturing global features, and cannot directly operate on raw point cloud data. Another neural network architecture for point cloud data is the graph convolutional network (GCN). This treats each individual point in the point cloud as a vertex in a graph data structure, allowing local features to be extracted by performing convolution-like operations directly on the point cloud data. The Transformer, which has achieved outstanding results in natural language processing, can also effectively capture global features. Its core attention mechanism is also well-suited for processing point cloud data. These deep learning methods have achieved satisfactory segmentation results on many public point cloud datasets, demonstrating the effectiveness of deep learning methods for point cloud data segmentation.

[0004] However, the relatively complex structure of plant point clouds leads to a greater amount of semantic information that needs to be identified in organ segmentation tasks. When acquiring point clouds, occlusion between leaves often causes the loss of part of the point cloud, resulting in holes and sparseness. In addition, the similarity between plant organs is very high, and different leaf instances often have the same color, morphological structure, texture and other features. This highly repetitive feature is not friendly to neural network learning. Finally, different plant varieties have different geometric morphological characteristics. Even the same variety of plants have different phenotypic characteristics under different growth environments, and even there will be significant differences, which requires a high level of generalization ability of the network. In summary, the current segmentation accuracy of plant point clouds cannot meet the requirements. Summary of the Invention

[0005] The purpose of the present application is to solve the problem of accurate organ segmentation in complex plant point clouds, and to provide a double-branch parallel plant organ point cloud segmentation method based on attention mechanism and spatial graph convolution. The method provides a reliable and efficient organ segmentation method for plant three-dimensional point clouds with complex structures. Tomato, corn, tobacco, sorghum and wheat point cloud data are used as research objects. A double-branch parallel neural network architecture TRGCN is newly designed based on attention mechanism and spatial graph convolution, which can capture local features and global features of point clouds at the same time, and is used to train a high-robustness instance segmentation model, which can improve the segmentation accuracy of plant point clouds and provide data support for fast, efficient and accurate plant phenotype analysis.

[0006] To achieve the above purpose, the technical solutions adopted by the present application are as follows:

[0007] A plant organ point cloud segmentation method based on attention mechanism and graph convolution, the method comprises:

[0008] Step one: the feature encoder takes the original point cloud as input, maps the features to a high-dimensional space using a multi-layer perceptron, and preliminarily extracts the features using a point cloud attention mechanism. Then, the initial feature data is input into the TRGCN block, which can be stacked multiple times to deepen the understanding of high-dimensional features. The feature aggregation layer in the TRGCN block extracts the neighborhood features while down-sampling the point cloud, and then enters the double-branch parallel network part, which includes a local feature capture branch composed of spatial graph convolution and a global feature learning branch composed of point attention mechanism. Finally, the feature data is input into the T-G feature coupling layer to obtain the target number of point clouds and the corresponding high-dimensional abstract features. The encoder part extracts high-dimensional feature information from the original plant point cloud by stacking TRGCN blocks, which is used for segmentation tasks;

[0009] Step two: the feature decoder part also stacks three cascaded TRGCN blocks and receives the output of the three TRGCN blocks in the encoder, but replaces the feature aggregation layer with an interpolation layer. The interpolation layer restores the features of the high-dimensional point set to the low-dimensional point set, but still outputs the grouping results of the K-nearest neighbor algorithm for the two branches of TRGCN to calculate. For segmentation result prediction, the decoder sets an independent interpolation layer after the TRGCN block and uses a single point attention layer to ensure information integrity. The network finally uses a multi-layer perceptron to output the segmentation results of the point cloud;

[0010] Step 3: Network training: All experiments in this study were conducted on an independent server equipped with a 12-core 20-thread CPU, 64GB of memory and an Nvidia GeForce RTX 3090Ti GPU; an independent server was used for neural network training. During the training phase, all plant point cloud segmentation models adopted the same hyperparameters, which were: the training batch size was set to 32, the initial learning rate was set to 0.001, the network was optimized using the Adam method, and a total of 100 cycles were trained. The learning rate was halved every 20 cycles, the weight decay was set to 0.0001, the momentum was set to 0.9, the K value of the K nearest neighbor algorithm was set to 12, and the feature dimension of the point attention layer was set to 256.

[0011] Furthermore, the step 1 is specifically as follows:

[0012] (1) Feature aggregation layer

[0013] The specific process of feature aggregation is as follows: input x points with feature dimensions, first perform random farthest point sampling, then use K-nearest neighbor algorithm to group the point cloud, input it into the multi-layer perceptron to aggregate the neighbor point features to the center point, and finally use the maximum pooling operation to calculate y points with feature' dimension features;

[0014] The feature aggregation layer uses the K-nearest neighbor algorithm to sample and group the input point set; the feature aggregation layer outputs the calculated K-nearest neighbor matrix and shares it with subsequent parallel branches;

[0015] (2) Local feature capture branch

[0016] This branch is built based on dynamic spatial graph convolution and is used to extract local features from the input plant point cloud. First, a feature graph G = (V, E) is constructed based on the point set V and the neighbor information E. Edge convolution is used to extract features in the input feature space. i The formula of the feature is:

[0017] f i =? h(x i ,y i )

[0018] Among them, x j is point x i One of the neighboring points of the candidate point, ? and h represent a certain aggregation function and a certain relational operation respectively; that is, a relational operation is used to aggregate the features of the neighboring points around the candidate point, that is, to obtain the feature information of the candidate point. The relational operation is defined as edge convolution;

[0019] Max pooling is used as the aggregation function. The specific process is as follows:

[0020] convi = Max(MLP(h(x i , x i - x j ))

[0021] The relationship operation h is defined as a linear combination between the feature difference of the point x i , x i and its neighbor point x j , and the output value of the point x i ;

[0022] (3) Global feature learning branch

[0023] The vector attention mechanism in the local neighborhood is used to extract features, and the calculation formula is:

[0024]

[0025] where x j is one of the K neighbor points of the point x i , X is the independent point set in each single plant point cloud, p is the regularization function, g is the mapping function, b is a certain relationship operation, which is defined as the difference between the neighbor point and the point of interest in this study, f, a is a point-level feature transformation method, which respectively obtains Q, K, V (Query, Key, Value, which are the exclusive terms in the attention mechanism, corresponding to Chinese query, key, value) values in the self-attention mechanism, and d is a position encoding function. According to the above attention mechanism, a point attention layer is proposed, and the improved calculation formula is:

[0026]

[0027] (4) T-G feature coupling layer

[0028] After the above processing, two feature matrices with the same dimensions and shapes are obtained: matrix G with significant local features and matrix T with complete global features; After concatenating G and T, the target feature matrix is obtained by inputting the feature coupling layer:

[0029] TG = Linear(ReLU(Linear(T, G)))

[0030] The T-G feature coupling layer is designed by using two linear layers and a ReLU activation layer, so that the network can learn the more important information of the two matrices respectively, and combine them into the target feature matrix.

[0031] The present invention offers the following advantages over existing technologies: It designs a novel dual-branch parallel instance segmentation network (TRGCN) based on a point-attention mechanism and spatial graph convolution. It directly inputs a 3D point cloud, with two branches focusing on local and global feature extraction, respectively, and fusing these two features via a TG feature coupling layer. Results demonstrate that TRGCN achieves excellent performance on diverse plant point clouds, achieving higher accuracy than other mainstream point cloud segmentation networks. It demonstrates excellent generalization capabilities and can provide robust data support for rapid, efficient, and accurate plant phenotyping. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a network architecture diagram of the TRGCN of the present invention;

[0033] Figure 2 This is a structural diagram of the TRGCN block of the present invention;

[0034] Figure 3 This is the architecture diagram of the global feature learning layer of the TRGCN block of the present invention;

[0035] Figure 4 This is the segmentation result diagram of the point cloud of five kinds of plants according to the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0037] Example 1:

[0038] Based on the point cloud self-attention mechanism and spatial graph convolution, this study innovatively proposes a dual-branch parallel network Transformer Graph Convolution Network (TRGCN) with an encoder-decoder architecture design. Figure 1 ).

[0039] The feature encoder takes the original point cloud as input, uses a multi-layer perceptron to map the features to a high-dimensional space (32 dimensions by default), and uses the point cloud attention mechanism to initially extract features. The initial feature data is then fed into the TRGCN module ( Figure 2), this module can be cascaded and superimposed in multiple layers to continuously deepen the understanding of high-dimensional features. Specifically, the feature aggregation layer in the TRGCN block extracts neighborhood features while downsampling the point cloud, and then enters the dual-branch parallel network part, which is a local feature capture branch composed of spatial graph convolution and a global feature learning branch composed of a point attention mechanism. Finally, the feature data is input into a specially designed TG feature coupling layer to obtain the target number of point clouds and the corresponding high-dimensional abstract features. The encoder part extracts high-dimensional feature information from the original plant point cloud by superimposing TRGCN blocks for segmentation tasks.

[0040] (1) Feature aggregation layer

[0041] The function of the feature aggregation layer in the TRGCN block is to reduce the cardinality of the input point set while abstracting higher-dimensional feature vectors during the stacking of multiple modules. For example, from the original input to the first TRGCN block, the number of points is reduced from N to N / 4, and the feature dimension of the point cloud is increased from F to 2F.

[0042] The specific process of feature aggregation is as follows: input x points with feature dimension, first perform random farthest point sampling, then use K nearest neighbor algorithm to group the point cloud, input it into the multi-layer perceptron to aggregate the neighbor point features to the center point, and finally use the maximum pooling operation to calculate y points with feature' dimension features (default y = x / 4, feature' = 2*feature).

[0043] The feature aggregation layer uses the K-nearest neighbor algorithm to sample and group the input point set. In addition, to save video memory space during training, this layer outputs the calculated K-nearest neighbor matrix and shares it with subsequent parallel branches.

[0044] (2) Local feature capture branch

[0045] This branch is built based on dynamic spatial graph convolution and is used to extract local features from the input plant point cloud. First, a feature graph G = (V, E) is constructed based on the point set V and the neighbor information E, and edge convolution is used to extract features in the input feature space. i The formula for the feature is as follows:

[0046] f i =? h(x i ,y i )

[0047] Among them, x j Representative point x iOne of the neighboring points of a candidate point, ? and h represent an aggregation function and a relational operation, respectively. Specifically, a relational operation is used to aggregate the features of neighboring points around the candidate point to obtain the feature information of the candidate point. This relational operation is defined as edge convolution. To enhance the understanding of local features in the point cloud, this study uses maximum pooling as the aggregation function. The specific process is as follows:

[0048] conv i =Max(MLP(h(x i ,x i -x j )))

[0049] The relational operation h is defined as the point x i x i and its neighbor point x j The characteristic difference and point x i The linear combination of the output values. This choice not only retains the characteristics of the local point concentration that influence each other, but also partially takes into account the overall global characteristics.

[0050] (3) Global feature learning branch

[0051] like Figure 3 As shown in Figure 2, this branch is built based on the point cloud attention mechanism and is very suitable for processing point cloud data. In essence, point cloud data can be regarded as word vectors embedded in the attention space. This study uses the vector attention mechanism in the local neighborhood to extract features. The calculation formula is as follows:

[0052]

[0053] Among them, x j It is point x i One of the K neighbor points of α is a point-level feature transformation method, which respectively obtains the Q, K, and V values ​​in the self-attention mechanism. δ is the position encoding function, ρ is the regularization function, γ is the mapping function, and β is a certain relational operation, which is defined in this study as the difference between the neighboring points and the focus point. Based on the above attention mechanism, this study proposes a point attention layer. The improved calculation formula is as follows:

[0054]

[0055] Unlike conventional attention mechanisms, positional encoding is also incorporated into the α function to enhance feature understanding. Building on the point-wise attention layer, the TRGCN encoder constructs a residual structure within the global feature learning branch. A linear layer is added before and after the point-wise attention layer, and the final output is residually connected to the input. This facilitates information exchange, accelerates network convergence, and opens the possibility of training deeper networks.

[0056] (4) T-G feature coupling layer

[0057] After the above processing, two-dimensional and shape identical feature matrices can be obtained: matrix G with significant local features and matrix T with complete global features. The target feature matrix is obtained by concatenating G and T and inputting them into the feature coupling layer:

[0058] The T-G feature coupling layer is designed with two linear layers and one ReLU activation layer, allowing the network to learn more important information from each of the two matrices and combine them into a target feature matrix.

[0059] The T-G feature coupling layer is designed with two linear layers and one ReLU activation layer, allowing the network to learn more important information from each of the two matrices and combine them into a target feature matrix.

[0060] In summary, the TRGCN network feature encoder part can adapt to different visual tasks by changing the number of TRGCN blocks. Fewer TRGCN blocks can be used for lightweight classification networks, while more cascaded TRGCN blocks can be used for more fine-grained tasks such as point cloud segmentation and object recognition.

[0061] The feature decoder part also stacks three cascaded TRGCN blocks and receives the output of the three TRGCN blocks in the encoder, but replaces the feature aggregation layer with an interpolation layer. Unlike the feature aggregation layer, the interpolation layer in the decoder restores the features of the high-dimensional point set to the low-dimensional point set, but still outputs the grouping results of the K-nearest neighbor algorithm for the two branches of TRGCN to calculate. For instance segmentation result prediction, the decoder sets an independent interpolation layer after the TRGCN block and uses a single point attention layer to ensure information integrity. The network finally uses a multi-layer perceptron to output the segmentation result of the point cloud.

[0062] Network training. All experiments in this study were conducted on an independent server equipped with a 12-core 20-thread CPU, 64GB of memory, and an Nvidia GeForce RTX 3090Ti GPU. In the training phase, the five plant point cloud segmentation models used the same hyperparameters: the training batch size was set to 32, the initial learning rate was set to 0.001, the Adam method was used to optimize the network, a total of 100 cycles were trained, the learning rate was halved every 20 cycles, the weight decay was set to 0.0001, the momentum was set to 0.9, the K value of the K-nearest neighbor algorithm was set to 12, and the feature dimension of the point attention layer was set to 256.

[0063] Organ instance segmentation tests were conducted on 5 types of plant point cloud data, achieving the highest average intersection-over-union ratio of 86.38% and an average accuracy of 88.58%. In order to verify the segmentation ability of TRGCN, three mainstream point cloud segmentation networks were selected for comparison with TRGCN. Among the 5 segmentation tasks, TRGCN was ahead of the other three methods in 9 indicators, and achieved the best accuracy in most segmentation tasks, especially on sorghum leaves. The accuracy improvement was more obvious, which showed that TRGCN was better at processing monocot point clouds. Since the canopy structure of dicot crops is relatively crowded and prone to occlusion problems, the segmentation effect of tobacco and tomato point clouds is not as good as that of monocot crops, but the segmentation effect is still better than the other three segmentation networks. The specific test results are shown in Table 1. Figure 4 This is the segmentation effect diagram of five types of plant point clouds.

[0064] This study also used sorghum point clouds as the research object and explored the number of TRGCN pooling layers and cascaded TRGCN blocks. The results showed that the network using maximum pooling achieved the best segmentation performance, with an accuracy approximately 2% higher than average pooling and sum pooling. When the number of cascaded TRGCN blocks was three, the network reached the optimal training time and segmentation effect, sacrificing some time for higher segmentation accuracy. The specific test results are shown in Tables 2 and 3.

[0065]

[0066]

[0067] Table 1 is a comparison table of the segmentation accuracy of the TRGCN network of the present invention and other mainstream networks

[0068] Pooling layer Training time (seconds) Mean intersection over union (%) Mean accuracy (%) Max pooling 2082 78.9292 84.9198 Mean pooling 2085 75.7748 80.6104 Sum pooling 2086 76.3709 82.9647

[0069] Table 2 Segmentation effects of different pooling layers in ablation experiment 1 of the present invention

[0070] Number of TRGCN blocks Training time (seconds) Mean intersection over union (%) 2 1728 73.7120 3 2082 78.9292 4 2202 75.5498

[0071] Table 3. Segmentation effect of different stacking numbers of TRGCN blocks in ablation experiment 2 of the present invention.

Claims

1. A plant organ point cloud segmentation method based on attention mechanism and graph convolution, characterized by: The method is: Step 1: The feature encoder takes the original point cloud as input, uses a multi-layer perceptron to map the features to a high-dimensional space, and uses a point cloud attention mechanism to initially extract features. The initial feature data is then input into the TRGCN block. This module can be cascaded to continuously deepen the understanding of high-dimensional features. The feature aggregation layer in the TRGCN block extracts neighborhood features while downsampling the point cloud. It then enters the dual-branch parallel network part, which consists of a local feature capture branch composed of spatial graph convolution and a global feature learning branch composed of a point attention mechanism. Finally, the feature data is input into the TG feature coupling layer to obtain the target number of point clouds and the corresponding high-dimensional abstract features. Step 2: The feature decoder also stacks three cascaded TRGCN blocks and receives the outputs of the three TRGCN blocks in the encoder respectively, but replaces the feature aggregation layer with an interpolation layer. The interpolation layer restores the features of the high-dimensional point set to a low-dimensional point set, but still outputs the grouping results of the K-nearest neighbor algorithm for the calculation of the two branches of TRGCN. For segmentation result prediction, the decoder sets an independent interpolation layer after the TRGCN block and uses a single point attention layer to ensure information integrity. The network finally uses a multi-layer perceptron to output the segmentation result of the point cloud. Step 3: Network training: An independent server was equipped with a 12-core 20-thread CPU, 64GB of memory, and an Nvidia GeForce RTX 3090Ti GPU. The neural network was trained using the independent server. During the training phase, all plant point cloud segmentation models adopted the same hyperparameters, specifically: the training batch size was set to 32, the initial learning rate was set to 0.001, the network was optimized using the Adam method, and a total of 100 cycles were trained. The learning rate was halved every 20 cycles, the weight decay was set to 0.0001, the momentum was set to 0.9, the K value of the K nearest neighbor algorithm was set to 12, and the feature dimension of the point attention layer was set to 256.

2. The plant organ point cloud segmentation method based on attention mechanism and graph convolution according to claim 1, characterized in that: The step 1 is specifically as follows: (1) Feature aggregation layer The specific process of feature aggregation is as follows: input x points with feature dimensions, first perform random farthest point sampling, then use K-nearest neighbor algorithm to group the point cloud, input it into the multi-layer perceptron to aggregate the neighbor point features to the center point, and finally use the maximum pooling operation to calculate y points with feature' dimension features; The feature aggregation layer uses the K-nearest neighbor algorithm to sample and group the input point set; the feature aggregation layer outputs the calculated K-nearest neighbor matrix and shares it with subsequent parallel branches; (2) Local feature capture branch This branch is built based on dynamic spatial graph convolution and is used to extract local features from the input plant point cloud. First, a feature graph G = (V, E) is constructed based on the point set V and the neighbor information E. Edge convolution is used to extract features in the input feature space. i The formula of the feature is: f i =?h(x i ,y i ) Among them, x j is point x i One of the neighboring points of the candidate point, ? and h represent a certain aggregation function and a certain relational operation respectively; that is, a relational operation is used to aggregate the features of the neighboring points around the candidate point, that is, to obtain the feature information of the candidate point. The relational operation is defined as edge convolution; Max pooling is used as the aggregation function. The specific process is as follows: conv i =Max(MLP(h(x i ,x i -x j ))) The relational operation h is defined as the point x i , x i and its neighbor point x j The characteristic difference and point x i Linear combinations of output values; (3) Global feature learning branch The vector attention mechanism in the local neighborhood is used to extract features. The calculation formula is: Among them, x j It is point x i One of the K neighbor points of , X is a set of independent points in each single plant point cloud, ρ is the regularization function, γ is the mapping function, β is the difference between the neighborhood point and the focus point, φ, α is the point-level feature transformation method, which obtains the Q, K, and V values ​​in the self-attention mechanism respectively. δ is the position encoding function. Based on the above attention mechanism, a point attention layer is proposed. The improved calculation formula is: (4)TG characteristic coupling layer After the above processing, two feature matrices with exactly the same dimensions and shapes are obtained: a matrix G with significant local features and a matrix T with complete global features. G and T are concatenated and input into the feature coupling layer to obtain the target feature matrix: TG=Linear(ReLU(Linear(T,G))) The TG feature coupling layer is designed using two linear layers and a ReLU activation layer, which enables the network to learn the more important information of the two matrices and combine them into the target feature matrix.

Images