Plant semantic instance joint segmentation system and method based on three-dimensional point cloud
By using a joint segmentation system for plant semantic instances based on 3D point clouds, and employing dual-weighted self-attention and multi-window pooling techniques, the system solves the problems of low segmentation accuracy and information loss in existing plant point cloud technologies, achieving more efficient semantic and instance segmentation results and improving segmentation accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGDONG UNIV OF TECH
- Filing Date
- 2023-07-17
- Publication Date
- 2026-07-21
AI Technical Summary
Existing technologies for 3D point cloud segmentation of plants suffer from problems such as low segmentation accuracy, oversegmentation, excessive reliance on manually set thresholds, and high computational complexity. Furthermore, deep learning methods suffer from information loss and insufficient segmentation accuracy when dealing with 3D occlusion problems.
A joint segmentation system for plant semantic instances based on 3D point clouds is adopted, including a location encoding module, an attention module, a spatial pyramid pooling module, an upsampling module, a downsampling module, an MLP module, and a feature aggregation module. Global and local features of plant point clouds are extracted through a dual-weight self-attention mechanism and multi-window pooling, and semantic and instance segmentation is performed by combining the Softmax activation function and the MeanShift clustering algorithm.
It improves the segmentation accuracy and robustness of plant point clouds, enhances generalization ability, and achieves more efficient semantic and instance segmentation results, especially significantly improving segmentation performance when dealing with plants with complex structures.
Smart Images
Figure CN116883662B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical field of three-dimensional point cloud segmentation of plants, and in particular to a joint segmentation system and method for semantic instances of plants based on three-dimensional point clouds. Background Technology
[0002] Plant phenotype is the physical, physiological, and biochemical characteristics and traits of a plant that are determined or influenced by genes and environmental factors during its growth process, reflecting the plant's structure and composition, growth and development process and results [1]. The morphological phenotype of a plant includes its leaves, stems, fruits, etc., among which leaf organs constitute the majority of the traits of the plant morphological phenotype. In modern agricultural scientific research, plant phenotype is particularly important for studying plant growth and development, physiological metabolism, environmental adaptation, and thus improving crop yield. Traditional crop phenotypic trait measurements are generally done manually, which is not objective and cannot meet the needs of research. Therefore, in order to obtain more accurate plant phenotypic information, it is necessary to divide the plant model into individual organs and develop accurate plant organ segmentation methods to provide key technical means for obtaining phenotypic information.
[0003] Segmentation methods based on 3D point clouds have gradually become a hot topic in agricultural technology research. Hu Chunhua et al. used SegNet to segment poplar seedlings from field data obtained through RGB and depth data fusion, and then used the kd-tree algorithm to classify individual leaves. Peng Cheng et al. segmented tomato point clouds into stem and leaf sub-skeletons after Laplace transform correction, and then used MeanShift to segment leaves and petioles. Lai Yibin et al. used region growing, edge extraction, supervoxel clustering, and concavity / convexity methods to segment 3D plant point clouds of cucumber, tobacco, and corn. Elnashef et al. proposed a segmentation algorithm based on first- and second-order tensors to achieve stem and leaf segmentation of plant point clouds, and used density-based clustering to segment individual leaves. Li et al. proposed a single-leaf segmentation method for dense plant point clouds with excessive facet segmentation and facet region growth, and conducted experiments on greenhouse ornamental plants. Mirande et al. proposed a graph-based two-level method that calculates local features on the neighborhood graph of points based on botanical knowledge to segment individual leaves of spinach and tomato plant point clouds. Liu et al. reconstructed 3D point clouds of sugar beets using the Structure from Motion (SFM) algorithm. They used a region growing algorithm based on multi-scale tensor voting to segment individual leaves and overlapping leaves, and then used surface boundary filtering to segment overlapping leaves to obtain all leaves. Li et al. used the SFM algorithm to reconstruct 3D corn image sequences from multiple views. They then used clustering, color filtering, and voxel filtering algorithms to process the corn plant point cloud to obtain a corn plant model, and finally used a region growing algorithm to segment individual leaves and stems. Traditional point cloud segmentation methods have many disadvantages. Edge detection methods are prone to losing 3D geometric information and oversegmentation. Region growing segmentation methods rely heavily on manual intervention, requiring manual pre-selection of seed points or manual setting of thresholds, which significantly impacts the objectivity of segmentation. Feature-based clustering segmentation methods mainly calculate the distance between points in the feature space. In large, dense point clouds, calculating the feature information of points is highly complex, thus significantly increasing the time cost. Traditional segmentation algorithms are relatively mature in point cloud processing tasks, but their shortcomings are also obvious.
[0004] In recent years, with the development of deep learning, combining deep learning with point cloud segmentation has gradually become a new and popular research direction. Currently, deep learning-based point cloud processing methods include multi-view methods, voxel-based methods, and point-based methods. Ni et al. used a Mask R-CNN network model to segment individual blueberries from 2D images and established a 2D-3D correspondence to obtain the number, volume, and maturity of berries. Li et al. used a PSegNet network to segment the leaves and stems of tobacco, tomato, and sorghum plants after sampling the voxelized farthest point. However, using voxelization consumes a lot of memory, and using sparse convolution leads to the loss of much spatial structural information in the point cloud. Shi et al. used a fully convolutional network (FCN) and a masked R-CNN network to perform semantic and instance segmentation on 2D plant images, and then combined different viewpoints to achieve 3D point cloud segmentation. Since occlusion is unavoidable in 3D point clouds, and 2D images are projected from 3D images, information loss occurs, which affects segmentation accuracy to some extent. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a plant semantic instance joint segmentation system with high segmentation accuracy based on three-dimensional point clouds.
[0006] To achieve the above objectives, the technical solution provided by this invention is as follows:
[0007] A joint segmentation system for plant semantic instances based on 3D point clouds includes a location encoding module, an attention module, a spatial pyramid pooling module, an upsampling module, a downsampling module, an MLP module, and a feature aggregation module.
[0008] The location encoding module receives the plant point cloud and performs location encoding, then divides it into two branches: the left branch is the global feature encoding and decoding branch of the point cloud, and the right branch is the local feature encoding and decoding branch of the point cloud.
[0009] The attention module, spatial pyramid pooling module, and MLP module are used together in the global feature encoding and decoding branch of the plant point cloud to obtain global decoded features.
[0010] The attention module, upsampling module, downsampling module, and MLP module are used together in the local feature encoding and decoding branch of the plant point cloud to obtain local decoded features;
[0011] After receiving global and local decoded features, the feature aggregation module, with the cooperation of the branching MLP module, outputs semantic segmentation results and instance segmentation results respectively.
[0012] Furthermore, the attention module is a dual-weight self-attention module, and its dual-weight attention mechanism weight matrix is as follows:
[0013]
[0014] K1 = MLP(x), K2 = MLP(x) p )
[0015] Where Q is the similarity matrix, This represents a matrix concatenation operation, where x is the output of the layer preceding the attention layer. p This is the output of the position encoding module.
[0016] Furthermore, the feature aggregation module uses the following aggregation formula:
[0017] F lg =f(p(f(F) l )+F g )+F l )
[0018] Among them, F l For local decoding features, F g This is a global decoding feature.
[0019] To achieve the above objectives, this invention further provides a joint segmentation method for plant semantic instances based on 3D point clouds, which is implemented using the aforementioned joint segmentation system for plant semantic instances based on 3D point clouds, and includes the following steps:
[0020] The plant point cloud containing N points and C feature dimensions is input into the position encoding module for position encoding, and then divided into two branches: the left side is the global feature encoding and decoding branch of the plant point cloud, and the right side is the local feature encoding and decoding branch of the plant point cloud.
[0021] The attention module, spatial pyramid pooling module, and MLP module are used together in the global feature encoding and decoding branch of the plant point cloud to obtain global decoded features.
[0022] The attention module, upsampling module, downsampling module, and MLP module are used together in the local feature encoding and decoding branch of the plant point cloud to obtain local decoded features;
[0023] After receiving global and local decoded features, the feature aggregation module, with the cooperation of the branching MLP module, outputs semantic segmentation results and instance segmentation results respectively.
[0024] Furthermore, the attention module, spatial pyramid pooling module, and MLP module are used in combination in the global feature encoding and decoding branch of the plant point cloud to obtain global decoded features, including:
[0025] In the attention module, the input of each layer is the output of the previous attention layer. Finally, the inputs of multiple attention layers are concatenated to obtain the output of the hierarchical attention block.
[0026] The spatial pyramid pooling module uses multi-window pooling of different sizes to obtain global information of different dimensions and to extract features from plant point clouds in multiple aspects.
[0027] Finally, the global decoding features are obtained through the MLP module.
[0028] Furthermore, the formula used in the spatial pyramid pooling module is as follows:
[0029]
[0030] Where f represents the input feature, S n The value represents the pooling window size, p represents max pooling, c represents concatenating the pooling outputs of different scales, and finally spatial pyramid pooling is completed through MLP.
[0031] Furthermore, the downsampling module first samples by sampling the farthest point to reduce the size of the input point set. Then, it filters the points step by step through layering. Next, it uses the ball query algorithm to find the neighbors of all the center points and obtain a local neighborhood of each center point.
[0032] Furthermore, the upsampling module performs multiple consecutive upsampling operations on the plant point cloud, employing a hierarchical propagation method based on distance interpolation and cross-layer skip links. The upsampling process is shown in the following equation:
[0033]
[0034]
[0035] Using an inverse distance-weighted average based on k-nearest neighbors, k = 3, where x i w is a nearest neighbor of x. i It is x i Weights relative to distance.
[0036] Furthermore, after receiving the global and local decoded features, the feature aggregation module uses the Softmax activation function in one branch to obtain the semantic segmentation result, and the MeanShift clustering algorithm in the other branch to obtain the instance segmentation result.
[0037] Compared with existing technologies, the principles and advantages of this solution are as follows:
[0038] 1. This solution simultaneously extracts semantic features and instance features of plant point clouds in the field of 3D point cloud deep learning and combines them to improve the abstract representation of plant point clouds and enhance generalization.
[0039] 2. Pyramid pooling is used to obtain global information of point clouds in different dimensions, enabling feature extraction at different scales from the point cloud, which helps to improve segmentation accuracy. Attached Figure Description
[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the services required in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0041] Figure 1 This is a flowchart illustrating the principle of the plant semantic instance joint segmentation method based on 3D point cloud in this invention (LGF-SegNet network architecture).
[0042] Figure 2 This is a structural diagram of the feature aggregation module in the plant semantic instance joint segmentation system based on 3D point cloud of the present invention;
[0043] Figure 3 These are multiple views of the plants used in the experiment. Detailed Implementation
[0044] The present invention will be further described below with reference to specific embodiments:
[0045] The plant semantic instance joint segmentation system based on 3D point cloud described in this embodiment includes a location encoding module, an attention module, a spatial pyramid pooling module, an upsampling module, a downsampling module, an MLP module, and a feature aggregation module.
[0046] like Figure 1 As shown (DSM is the downsampling module), its working principle is as follows:
[0047] The plant point cloud containing N points and C feature dimensions is input into the location encoding module for location encoding, where C=9, including coordinates (x,y,z), color (r,g,b), and normal vector (n). x n y ,n z Then it is divided into two branches: the left branch is the global feature encoding and decoding branch of plant point cloud, and the right branch is the local feature encoding and decoding branch of plant point cloud.
[0048] The attention module, spatial pyramid pooling module, and MLP module are used together in the global feature encoding and decoding branch of the plant point cloud to obtain global decoded features.
[0049] The attention module, upsampling module, downsampling module, and MLP module are used together in the local feature encoding and decoding branch of the plant point cloud to obtain local decoded features;
[0050] After receiving global and local decoded features, the feature aggregation module, with the cooperation of the branching MLP module, outputs semantic segmentation results and instance segmentation results respectively.
[0051] In the above, the attention module is a two-weighted self-attention module, while the core implementation of the traditional attention mechanism is shown in the following equation:
[0052]
[0053] in,
[0054]
[0055] Q=MLP1(x), K=MLP2(x), V=MLP3(x)
[0056] In shallow layers of the network, each point does not learn enough features to obtain similarity between points, making the geometric relationships between points more important. Therefore, the attention module in this embodiment improves upon the self-attention mechanism by increasing the number of weight matrices and incorporating them into the network, called the dual-weight self-attention module. Its dual-weight attention mechanism weight matrix is as follows:
[0057]
[0058] K1 = MLP(x), K2 = MLP(x) p )
[0059] Where Q is the similarity matrix, This represents a matrix concatenation operation, where x is the output of the layer preceding the attention layer. p This is the output of the position encoding module.
[0060] for Figure 1 The stacked hierarchical attention blocks in the left branch have each layer's input being the output of the previous attention layer. Finally, the inputs of the four attention layers are concatenated, which helps the network learn the semantic features of the points and obtain the output of the hierarchical attention block.
[0061] Furthermore, in traditional max pooling or average pooling, due to the fixed pooling window size, only global feature information of a fixed dimension can be obtained. Spatial pyramid pooling uses multi-window pooling of different sizes to obtain global information of different dimensions, enabling multi-faceted feature extraction from point clouds and improving the accuracy and robustness of the segmentation algorithm. The specific implementation formula is as follows:
[0062]
[0063] Where f represents the input feature, S n Let represent the pooling window size, p represent max pooling, and c represent concatenating the pooling outputs at different scales. Finally, spatial pyramid pooling is performed using an MLP. Specifically, when inputting NxD point cloud features, the pooling window sizes are N / s1, ..., N / s. n Finally, features from different dimensions are aggregated to obtain (s1+s2+…+s n Features in the xD dimension.
[0064] Furthermore, in the above process, the downsampling module first samples by sampling the farthest point to reduce the input point set to a smaller size. Then, it filters the points step by step through layering. Next, it uses the ball query algorithm to find the neighboring points of all center points to obtain a local neighborhood of each center point.
[0065] The upsampling module performs multiple consecutive upsampling operations on the plant point cloud, employing a hierarchical propagation method based on distance interpolation and cross-layer skip links. The upsampling process is shown in the following equation:
[0066]
[0067]
[0068] Using an inverse distance-weighted average based on k-nearest neighbors, k = 3, where x i w is a nearest neighbor of x. i It is x i Weights relative to distance.
[0069] The feature aggregation module is a key module that combines local and global features. 3D point clouds are highly structured data, and variations in noise can affect algorithm performance. Aggregating global information first improves the algorithm's robustness, and secondly, generates more abstract and complex feature representations, which helps the network learn point cloud features better and improves segmentation performance. Figure 2 As shown, the local features F obtained after decoding the network features are respectively l and global features F g The input is fed into the feature aggregation module, as shown in the following formula, to perform feature fusion operations on local and global features.
[0070] F lg =f(p(f(F) l )+F g )+F l )
[0071] Among them, F l For local decoding features, F g This is a global decoding feature;
[0072] The local and global features are fused to output a higher-level feature, which is then input into two parallel branches with an MLP module. One branch uses the Softmax activation function to obtain the semantic segmentation result, and the other branch uses the MeanShift clustering algorithm to obtain the instance segmentation result.
[0073] To demonstrate the effectiveness and superiority of the method described in this embodiment, the following experiment was conducted:
[0074] The experimental dataset for this study was derived from a seedling plant in a flower cultivation base in Guangzhou. The collected multi-view 2D images of the plant were preprocessed using 3D reconstruction and data filtering. The processed data is shown below. Figure 3 As shown (partial data illustration).
[0075] Semantic segmentation performance comparison:
[0076] Regarding the semantic segmentation performance of the LGF-SegNet network architecture on plant point clouds, this study compares it with other popular point cloud deep learning networks with semantic segmentation capabilities. The quantitative comparative analysis is conducted based on the evaluation metrics of IoU, Prec, and F1 score. "Other" represents non-plant parts, i.e., flower pots.
[0077] method Mean leaf stem other PointNet
[16] 73.84 80.54 63.42 77.56 PointNet++
[17] 81.64 82.45 72.81 89.65 DGCNN
[18] 77.73 83.18 64.53 85.49 JSNet
[19] 75.31 77.23 66.32 82.39 PCT
[20] 81.51 81.56 79.72 83.26 RPNet
[21] 81.70 81.48 76.35 87.26 LGF-SegNet 85.76 84.85 80.34 92.09
[0078] Table 1. Comparison of semantic segmentation IoU (%) between the method of this invention and other networks.
[0079]
[0080]
[0081] Table 2 Comparison of the semantic segmentation Prec (%) of the method of this invention with other networks
[0082] method Mean leaf stem other PointNet
[16] 83.76 91.35 70.60 89.33 PointNet++
[17] 90.67 98.14 76.04 97.84 DGCNN
[18] 89.14 96.74 74.33 96.34 JSNet
[19] 89.22 97.31 73.64 96.72 PCT
[20] 88.15 95.64 72.36 96.45 RPNet
[21] 89.74 97.29 74.06 97.86 LGF-SegNet 91.08 98.01 77.21 98.03
[0083] Table 3 Comparison of the semantic segmentation F1 score (%) of the method of this invention with other networks
[0084] The results of the plant point cloud semantic segmentation experiment are shown in Tables 1, 2, and 3. The LGF-SegNet network architecture proposed in this invention outperforms other networks in almost all metrics, and the average values of the three evaluation metrics are also superior to those of other networks. It is not difficult to find that the segmentation effect of the stem is significantly lower than that of the other two classes. The main reason for this phenomenon is that the biological characteristics of the stem are slender and not as wide as the leaves, containing fewer points than the leaves. Secondly, the stem is sandwiched between the flowerpot and the leaves, making stem segmentation more difficult.
[0085] Instance segmentation results comparison:
[0086] The instance segmentation results are shown in Table 4. Therefore, LGF-Net outperforms other networks in all metrics for plant instance segmentation, with all metrics exceeding 75%. The network proposed in this invention shows good performance in leaf instance segmentation. The main difficulty in instance segmentation of plant data in this experiment lies in the overlapping of leaf point clouds when there are many leaves on the plant. This inevitably leads to the possibility that the points where leaf point clouds meet may be incorrectly classified as the point clouds of another leaf. However, compared with other methods, the method of this invention reduces this missegmentation, so the experimental results show higher metrics.
[0087] method mPrec mCov mWCov SGPN
[22] 77.48 65.92 65.13 JSNet
[19] 79.24 70.34 71.58 PCT
[20] 84.33 77.52 78.82 SoftGroup
[23] 81.62 75.26 76.59 LGF-SegNet 85.27 78.46 79.63
[0088] Table 4 Comparison of instance segmentation performance of the method of the present invention with other networks
[0089] The above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Therefore, any changes made in accordance with the shape and principle of the present invention should be covered within the protection scope of the present invention.
Claims
1. A joint segmentation system for plant semantic instances based on 3D point clouds, characterized in that, It includes a location encoding module, a hierarchical attention module, a dual-weight attention module, a spatial pyramid pooling module, an upsampling module, a downsampling module, an MLP module, and a feature aggregation module; The location encoding module receives the plant point cloud and performs location encoding, then divides it into two branches: the left branch is the global feature encoding and decoding branch of the point cloud, and the right branch is the local feature encoding and decoding branch of the point cloud. The hierarchical attention module, spatial pyramid pooling module, and MLP module are used together in the global feature encoding and decoding branch of the plant point cloud to obtain global decoded features. The dual-weighted attention module, upsampling module, downsampling module, and MLP module are used together in the local feature encoding and decoding branch of the plant point cloud to obtain local decoded features; After receiving global and local decoded features, the feature aggregation module, with the cooperation of the branching MLP module, outputs semantic segmentation results and instance segmentation results respectively. The weight matrix of the dual-weight attention mechanism in the dual-weight attention module is as follows: ， in, It is a similarity matrix, where ⊕ indicates matrix concatenation. It is the output of the layer above attention. This is the output of the position encoding module; The formula used in the spatial pyramid pooling module is as follows: ， in, Represents the features of the input. Indicates the pooling window size. denoted as max pooling, c represents concatenating the pooling outputs of different scales, and finally completing spatial pyramid pooling through MLP.
2. The plant semantic instance joint segmentation system based on 3D point cloud according to claim 1, characterized in that, The feature aggregation module uses the following aggregation formula: ， in, For local decoding features, This is a global decoding feature.
3. A joint segmentation method for plant semantic instances based on 3D point clouds, characterized in that, The plant semantic instance joint segmentation system based on 3D point cloud as described in claim 1 is implemented by including the following steps: The plant point cloud containing N points and C feature dimensions is input into the position encoding module for position encoding, and then divided into two branches: the left side is the global feature encoding and decoding branch of the plant point cloud, and the right side is the local feature encoding and decoding branch of the plant point cloud. The hierarchical attention module, spatial pyramid pooling module, and MLP module are used together in the global feature encoding and decoding branch of plant point cloud to obtain global decoded features; The dual-weighted attention module, upsampling module, downsampling module, and MLP module are used together in the local feature encoding and decoding branch of the plant point cloud to obtain local decoded features; After receiving global and local decoded features, the feature aggregation module, with the cooperation of the branching MLP module, outputs semantic segmentation results and instance segmentation results respectively.
4. The plant semantic instance joint segmentation method based on 3D point cloud according to claim 3, characterized in that, The hierarchical attention module, spatial pyramid pooling module, and MLP module are used in combination in the global feature encoding and decoding branch of plant point clouds to obtain global decoded features, including: In the hierarchical attention module, the input of each layer is the output of the previous attention layer. Finally, the inputs of multiple attention layers are concatenated to obtain the output of the hierarchical attention module. The spatial pyramid pooling module uses multi-window pooling of different sizes to obtain global information of different dimensions and to extract features from plant point clouds in multiple aspects. Finally, the global decoding features are obtained through the MLP module.
5. The plant semantic instance joint segmentation method based on 3D point cloud according to claim 3, characterized in that, The downsampling module first samples by sampling the farthest point to reduce the size of the input point set. Then, it filters the points step by step through layering. Next, it uses the ball query algorithm to find the neighbors of all the center points and obtain a local neighborhood of each center point.
6. The plant semantic instance joint segmentation method based on 3D point cloud according to claim 3, characterized in that, The upsampling module performs multiple consecutive upsampling operations on the plant point cloud, employing a hierarchical propagation method based on distance interpolation and cross-layer skip links. The upsampling process is shown in the following equation: ， = ， We use an inverse distance-weighted average based on k-nearest neighbors, where k=3, where... yes The nearest neighbor, yes Weights relative to distance.
7. The plant semantic instance joint segmentation method based on 3D point cloud according to claim 3, characterized in that, After receiving global and local decoded features, the feature aggregation module uses the Softmax activation function in one branch to obtain the semantic segmentation result, and the MeanShift clustering algorithm in the other branch to obtain the instance segmentation result.