Point cloud leaf segmentation method and system based on geometric interaction and adaptive graph convolution
Patent Information
- Application Number
- CN202311726240.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-14
- Publication Date
- 2026-09-25
- Estimated Expiration
- 2043-12-14
AI Technical Summary
但是，点之间的特征对应关系进行相同的卷积处理会导致特征学习的局限性，也无法很好地描述中心点与其邻域之间的几何结构信息
[0040]有益效果：本发明通过邻域搜索构造每个点的局部图，将局部图中的点分别嵌入到欧几里德空间和双曲空间后进行自适应图卷积操作，得到点云在欧几里得和双曲空间的潜在的几何结构信息。最后，利用门控机制自适应地整合这两个几何特征，得到点云的复杂几何空间结构。与现有技术相比，其显著优点为：1)本发明除了考虑点之间的几何关系，还引入局部点云在不同几何空间的几何结构。2)利用门控机制将点云在两个几何空间的特征进行融合，使分割模型更好地学习点云复杂的几何结构。3)本发明在自然场景下对单株植物的表型特征进行测量，避免对其自然生长带来影响。4)损失函数由交叉熵和三元组损失组成，有助于提高模型的泛化性能，使得不同类别的点更容易区分。
Smart Images
Figure CN118135207B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of point cloud segmentation, and in particular to a method and system for segmenting point cloud leaves based on geometric interaction and adaptive graph convolution. Background Technology
[0002] A current limitation in crop breeding and production is the evaluation of plant performance under different environments and management practices. Plant phenotypes describe plant morphology, such as leaf area. Among plant organs, leaves are the site of many important physiological processes, such as photosynthesis, respiration, and transpiration. Therefore, extracting leaf characteristic parameters such as leaf area, color, and shape is crucial for improving plant breeding and production.
[0003] With the development of deep learning, organ segmentation of plant point clouds has become a viable frontier research area. Influenced by the successful application of graphs and other nonlinear structures in deep learning, Graph Convolutional Networks (GCNs) have received increasing attention in recent years. Unlike the regular pixels in two-dimensional images, three-dimensional point clouds consist of a series of disordered points in non-Euclidean space. Due to their irregular and sparse structure, typical two-dimensional convolutions cannot be directly applied to 3D point clouds. Therefore, Graph Convolutional Neural Networks (GCNs) construct a local region for each point, extending 2D convolutions to 3D data. Dynamic Graph Convolutional Neural Networks (DGCNNs) introduce a novel edge convolution, dynamically computing the graph structure at each network layer and pooling the features of nodes in the local graph with their corresponding edge features. However, applying the same convolutional processing to the feature correspondences between points leads to limitations in feature learning and fails to adequately describe the geometric structure information between the center point and its neighborhood. Summary of the Invention
[0004] Purpose of the invention: To address the technical problems existing in the above-mentioned methods, the purpose of this invention is to provide a point cloud leaf segmentation method and system based on geometric interaction and adaptive graph convolution, which can capture the geometric relationships and geometric structure information of point clouds in different geometric spaces, thereby improving the accuracy of point cloud segmentation.
[0005] Technical solution: To achieve the above-mentioned objectives, the present invention adopts the following technical solution:
[0006] A point cloud leaf segmentation method based on geometric interaction and adaptive graph convolution includes the following steps:
[0007] Obtain point cloud data of individual plants and label leaves and stems to obtain a training dataset;
[0008] Point cloud data is input into a point cloud segmentation model for training. The point cloud segmentation model first searches for neighboring points and connects the neighboring points with the center point to form a local graph. Then, the points in the local graph are embedded into Euclidean space and hyperbolic space, and local feature information of the point cloud in different geometric spaces is extracted through an adaptive graph convolutional network. Finally, a gating mechanism is used to fuse these two local features. The loss function for model training is cross-entropy loss plus triplet loss.
[0009] The trained model is used to predict the test samples to obtain the point cloud segmentation results of a single plant leaf.
[0010] The segmented leaf point cloud is separated using a clustering algorithm to obtain individual leaves.
[0011] As a preferred method, the process of acquiring point cloud data of a single plant includes: building a point cloud acquisition platform, taking pictures of a single plant to obtain images from different angles at 360°; and generating point cloud data from the acquired images using the motion structure recovery method.
[0012] As a preferred option, the processing of point cloud data also includes: using color filtering to remove points unrelated to plants; using a uniform downsampling method to control the number of points within a preset range, thereby reducing data complexity while preserving the main features and shape information of the point cloud; and performing coordinate normalization processing on each point.
[0013] As a preferred approach, the input point cloud is denoted as X = {x} i |i=1,2,……,N}∈R N×3 x i This represents the 3D coordinate position of the i-th point in the point cloud in Euclidean space, where N is the number of points; point x i The corresponding eigenvector is defined as F = {f i |i=1,2,……,N},f i Let x be the 3D coordinates and normal vector combination of the i-th point in Euclidean space; use the kd-tree method to search for the distance point x. i The nearest k neighborhood points x ij , neighboring point x ij With center point x i Connecting them forms a local graph.
[0014] As a preferred approach, the input features of the point cloud are represented in Euclidean space. The local map is then embedded into hyperbolic space, and the feature vectors of the points are transformed using the following formula.
[0015]
[0016] Where v is the eigenvector of the point in Euclidean space, c is the curvature of the space, and ||·|| is the Euclidean norm.
[0017] Preferably, the step of extracting local feature information of point clouds in different geometric spaces using an adaptive graph convolutional network includes:
[0018] By sharing the point x in the local graph of the multilayer perceptron i x ij and their corresponding features f i f ij A convolution kernel e is dynamically generated on top. ij Then, perform a convolution operation with the corresponding points; finally, obtain the connection point (x). i x ij Edge features h between ) ij ;
[0019] e ij =g([f i ,f i -f ij ])
[0020] h ij =σ <e ij ,[x i ,x i -x ij ]>
[0021] Where g(·) denotes a shared multilayer perceptron, [·,·] denotes a vector concatenation operation, <·,·> denotes the inner product of two vectors, and σ denotes a nonlinear activation function;
[0022] The center point x is defined by aggregating the features of all edges within the neighborhood. i Output characteristics:
[0023] f i ′=max h ij
[0024] Where max represents the max pooling function.
[0025] As a preferred approach, a gating mechanism is used to fuse the local features of point clouds obtained from two different geometric spaces:
[0026] g=σ(W[f i ′,s′ i ])
[0027] k′ i =g⊙f i ′+(1-g)⊙s′ i
[0028] Where ⊙ denotes the scalar multiplication operation of vectors, W represents the weight matrix, σ represents the activation function, and f i' represents a local feature of Euclidean space, s' i This represents the local features of hyperbolic space.
[0029] As a preferred approach, the loss function for model training is the cross-entropy plus the triplet loss:
[0030] L = L entropy +L triplet
[0031]
[0032]
[0033] Where N represents the number of points, C represents the number of categories, and y i,c The label p indicates whether the i-th point belongs to category c. i,c f represents the probability that the i-th point belongs to category c; a Let f represent the eigenvector of the i-th point. + f represents the feature vector of points belonging to the same category as the i-th point. - Let represent the feature vector of a point that belongs to a different category from the i-th point, and ∈ represent the distance between the anchor point and the positive sample.
[0034] Preferably, the method further includes calculating plant phenotypic characteristic parameters for isolated individual leaves using a ball rotation algorithm and formula, the specific process of which includes:
[0035] The plant height H is obtained by measuring the height of the bounding box surrounding the plant point cloud, and the maximum value z of the point cloud on the z-axis is used. max Subtract the minimum value z min ;
[0036] The length and width of the blade are measured by calculating the distance between the farthest points on the blade along the x and y axes.
[0037] The surface of the blade was reconstructed using a sphere rotation algorithm. It consists of a large number of triangles. The area of the triangles was calculated using Heron's formula, and the total area of the fitted surface of the blade was obtained by statistical analysis.
[0038] A point cloud leaf segmentation system based on geometric interaction and adaptive graph convolution includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the point cloud leaf segmentation method based on geometric interaction and adaptive graph convolution.
[0039] Based on the same inventive concept, the present invention provides a point cloud leaf segmentation system based on geometric interaction and adaptive graph convolution, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, it implements the point cloud leaf segmentation method based on geometric interaction and adaptive graph convolution.
[0040] Beneficial Effects: This invention constructs a local graph for each point through neighborhood search. Points in the local graph are then embedded into Euclidean and hyperbolic spaces respectively, followed by adaptive graph convolution to obtain the potential geometric structure information of the point cloud in both Euclidean and hyperbolic spaces. Finally, a gating mechanism is used to adaptively integrate these two geometric features to obtain the complex geometric spatial structure of the point cloud. Compared with existing technologies, its significant advantages are: 1) This invention considers not only the geometric relationships between points but also the geometric structure of the local point cloud in different geometric spaces. 2) The gating mechanism fuses the features of the point cloud in the two geometric spaces, enabling the segmentation model to better learn the complex geometric structure of the point cloud. 3) This invention measures the phenotypic features of a single plant in a natural scene, avoiding any impact on its natural growth. 4) The loss function consists of cross-entropy and triplet loss, which helps improve the generalization performance of the model, making it easier to distinguish points of different categories. Attached Figure Description
[0041] Figure 1 This is a flowchart illustrating an embodiment of the present invention.
[0042] Figure 2 This is a visual flowchart of an embodiment of the present invention.
[0043] Figure 3 This is a schematic diagram of the point cloud segmentation method in an embodiment of the present invention.
[0044] Figure 4 This is a schematic diagram of the adaptive graph convolution method in an embodiment of the present invention. Detailed Implementation
[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. It should be understood that the specific embodiments described herein are only for explaining this invention and are not intended to limit this invention.
[0046] like Figure 1As shown in the figure, the present invention discloses a point cloud leaf segmentation method based on geometric interaction and adaptive graph convolution. First, point cloud data of a single plant is acquired and the leaves and stems are labeled to obtain a training dataset. Then, the point cloud data is input into a point cloud segmentation model for training. Next, the trained model is used to predict test samples to obtain the point cloud segmentation results of a single plant leaf. Then, the segmented leaf point cloud is separated by a clustering algorithm to obtain individual leaves. Finally, based on this, plant phenotypic feature parameters can be calculated for the separated individual leaves.
[0047] The following uses cucumber seedling point cloud leaf segmentation as an example to illustrate the specific steps of this invention. It should be noted that the method of this invention is not only applicable to cucumber seedling point cloud leaf segmentation, but also to any plant with layered leaves. This invention discloses a point cloud leaf segmentation method based on geometric interaction and adaptive graph convolution, comprising the following steps:
[0048] Step 1: Acquire image data of cucumber seedlings from multiple perspectives. A point cloud acquisition platform was set up, consisting of a tripod, camera, 42cm diameter motorized turntable, small photography studio, and computer. The camera was placed on the tripod and its angle was adjusted so that it was directly facing the cucumber seedling, maintaining a horizontal distance of 0.7m from the center of the turntable. The turntable was rotated using a remote control, pausing every 30° to take pictures of the cucumber seedling, thus achieving 360° image capture from different angles around the seedling. In this study, the shooting time for each cucumber seedling was 3-5 minutes, acquiring 12 angles, with approximately 200 images collected per seedling at a time.
[0049] Step 2 involves inputting the acquired images into VisualSFM (Structure from Motion) software, which uses a motion structure reconstruction method, to generate point cloud data of the cucumber seedlings. By analyzing the geometric relationships between the images, the 3D structure of the object is inferred. The exported point cloud data can be in common 3D model formats such as .ply for further processing and analysis in other software.
[0050] Step 3: For better visualization, color filtering is used to remove points unrelated to the plant, such as flowerpots and soil. Next, a radius filter is used to filter noise. The basic idea is to calculate the number of points within a certain radius around each point; if the number is less than a set threshold, the point is deleted. Since the reconstructed cucumber seedling point cloud is very dense, with each point cloud containing approximately 10,000 to 12,000 points, a uniform downsampling method is used to control the number of points to around 4096, reducing data complexity while preserving the main features and shape information of the point cloud. To further improve the training efficiency of the segmentation model, coordinate normalization is performed on each point cloud, scaling the xyz coordinate values to the range [-1, 1]. However, the above downsampling and normalization operations will affect the accuracy of subsequent phenotypic feature extraction; therefore, in subsequent work, the output point cloud of the segmentation model will be upsampled and inversely normalized.
[0051] CloudCompare, an open-source 3D point cloud processing software, was used to label the training data for leaf segmentation. It can visualize, edit, and process large 3D point clouds. During labeling, leaves and stems of cucumber seedlings were manually selected, with leaves labeled as 0 and stems as 1. Finally, mimicking the format of the ShpaeNet dataset, the point cloud was saved as a txt file, with each line representing the x, y, and z coordinates, label number, and normal vector of a point. The dataset constructed in this study includes 1800 point cloud samples of cucumber seedlings, of which 1600 were used for the training set, 100 for the test set, and 100 for the validation set.
[0052] Step 4: Input the point cloud into the point cloud segmentation model for training to obtain the trained weight file;
[0053] Step 5: Use the trained model to predict the test set samples to obtain the semantic segmentation results of cucumber seedling point cloud;
[0054] Step 6: Use the K-means point cloud clustering algorithm to solve the adhesion problem between leaf point clouds, thereby separating individual leaves;
[0055] Step 7: Calculate the phenotypic characteristic parameters of cucumber seedlings using a ball rotation algorithm and formula based on the separated individual leaves.
[0056] In step 4, the point cloud segmentation model is as follows: Figure 3 As shown, the specific process includes:
[0057] Step 4-1: Use the kd-tree method to search for neighborhood points, connecting the neighborhood points with the center point to form a local graph. Denote the input point cloud as X = {x} i |i=1,2,……,N}∈R N×3 x iThis represents the 3D coordinate position of the i-th point in the point cloud in Euclidean space, where N is the number of points; point x i The corresponding eigenvector is defined as F = {f i |i=1,2,……,N},f i Let x be the 3D coordinates and normal vector combination of the i-th point in Euclidean space; use the kd-tree method to search for the distance point x. i The nearest k neighborhood points x ij , neighboring point x ij With center point x i Connecting them forms a local graph.
[0058] Step 4-2: Since the input features of the point cloud are represented in Euclidean space, it is only necessary to embed the input features into hyperbolic space. This embodiment uses the classic hyperbolic space model, the Poincaré sphere, to embed the local map of the point cloud. The Poincaré model is a hyperbolic space model with a unit radius and a constant negative curvature of -1. Compared to the Euclidean space model, it often requires only a small number of dimensions to model more complex data. In this model, the input features are mapped onto the hyperbolic surface, and the specific mapping formula is as follows:
[0059]
[0060] Where v is the eigenvector of the point in Euclidean space, c is the curvature of the space, and ||·|| is the Euclidean norm.
[0061] Step 4-3: Perform adaptive graph convolution calculation to obtain local geometric features:
[0062] (1) By sharing the point x in the local graph of the multilayer perceptron i x ij and their corresponding features f i f ij A convolution kernel e is dynamically generated on top. ij Then, perform a convolution operation with the corresponding points; finally, obtain the connection point (x). i x ij Edge features h between ) ij ;
[0063] e ij =g([f i ,f i -f ij ])
[0064] h ij =σ <e ij ,[x i ,x i -x ij ]>
[0065] Where g(·) denotes a shared multilayer perceptron, [·,·] denotes a vector concatenation operation, <·,·> denotes the inner product of two vectors, and σ denotes a nonlinear activation function;
[0066] (2) Aggregate all edge features within the neighborhood to define the center point x. i Output characteristics:
[0067] f i ′=max h ij
[0068] Where max represents the max pooling function.
[0069] Characteristic calculation of hyperbolic space and x i x ij Input features f in Euclidean space i f ij Input features s in hyperbolic space i s ij That's all.
[0070] Step 4-4: Use a gating mechanism to fuse the local features of point clouds obtained in two different geometric spaces.
[0071] g=σ(W[f i ′,s′ i ])
[0072] k′ i =g⊙f i ′+(1-g)⊙s′ i
[0073] Where ⊙ denotes the scalar multiplication operation of vectors, W represents the weight matrix, σ represents the activation function, and f i ' represents a local feature of Euclidean space, s' i This represents the local features of hyperbolic space.
[0074] Steps 4-5: For model training loss function, we add triplet loss to the traditional cross-entropy:
[0075] L = L entropy +L triplet
[0076]
[0077]
[0078] Where N represents the number of points, C represents the number of categories, and y i,c The label p indicates whether the i-th point belongs to category c.i,c f represents the probability that the i-th point belongs to class c. a Let f represent the eigenvector of the i-th point. + f represents the feature vector of points belonging to the same category as the i-th point. - Let represent the feature vector of a point that belongs to a different category from the i-th point, and ∈ represent the distance between the anchor point and the positive sample.
[0079] Steps 4-6: The evaluation metrics for the point cloud segmentation model include mean intersection-over-union ratio (MIRR), overall accuracy, and mean accuracy. The calculation formulas are shown below:
[0080]
[0081]
[0082]
[0083] Where n represents the number of categories, TP i TN represents correctly classifying points belonging to the i-th category into that category. i FN represents correctly classifying points that do not belong to category i into other categories. i This indicates that a point of category i was incorrectly classified into another category, FP i This indicates that point cloud points of other categories were incorrectly segmented into category i.
[0084] Step 7 includes the following specific processes:
[0085] Step 7-1, the plant height is determined by measuring the height of the bounding box surrounding the cucumber seedling's point cloud, and is determined by subtracting the minimum z-coordinate value from the maximum z-coordinate value of the point cloud:
[0086] H = z max -z min
[0087] Step 7-2, the blade length and width are measured by calculating the distance between the farthest points on the blade along the x and y axes:
[0088]
[0089] Step 7-3: The surface of the blade was reconstructed using a sphere rotation algorithm. It consists of a large number of triangles. The area of the triangles was calculated using Heron's formula, and the total area of the fitted surface of the blade was obtained by statistical analysis.
[0090]
[0091] Where n represents the number of triangles in the 3D model mesh of the blade, S iLet S represent the area of the i-th triangle point cloud, and let S represent the area of the leaf.
[0092] Based on the same inventive concept, the present invention discloses a cucumber seedling point cloud leaf segmentation system based on geometric interaction and adaptive graph convolution, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements a point cloud leaf segmentation method based on geometric interaction and adaptive graph convolution.
[0093] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
Claims
1. A point cloud leaf segmentation method based on geometric interaction and adaptive graph convolution, characterized in that, Includes the following steps: Obtain point cloud data of individual plants and label leaves and stems to obtain a training dataset; Point cloud data is input into a point cloud segmentation model for training. The model first searches for neighboring points, connecting them to the center point to form a local graph. Then, points in the local graph are embedded into Euclidean and hyperbolic spaces, and an adaptive graph convolutional network is used to extract local feature information of the point cloud in different geometric spaces. Finally, a gating mechanism is used to fuse these two local features. The loss function for model training is cross-entropy loss plus triplet loss. The input point cloud is denoted as X = {x...} i |i=1,2,……,N}∈R N×3 , This represents the 3D coordinate position of the i-th point in the point cloud in Euclidean space, where N is the number of points; The corresponding eigenvector is defined as F={f i |i=1,2,……,N}, Let the i-th point be the combination of its 3D coordinates and normal vector in Euclidean space; use the kd-tree method to search for distance points. The k nearest neighbors Neighboring points With the center point The connection forms a local graph; the input features of the point cloud are represented in Euclidean space, and the local graph is then embedded into hyperbolic space, and the feature vectors of the points are transformed using the following formula; ;in, It is the eigenvector of a point in Euclidean space. It is the curvature of space, and ||·|| is the Euclidean norm; The extraction of local feature information of point clouds in different geometric spaces through an adaptive graph convolutional network includes: extracting points from the local graph using a shared multilayer perceptron. , and their corresponding characteristics , Dynamically generate a convolution kernel. Then, perform a convolution operation with the corresponding points; finally, obtain the connection points ( Edge features between ) ; ; Where g(·) represents a shared multilayer perceptron, [·,·] represents a vector concatenation operation, <·,·> represents the inner product of two vectors, and σ represents a nonlinear activation function; the center point is defined by aggregating all edge features within the neighborhood. Output characteristics: Where max represents the max pooling function; The trained model is used to predict the test samples to obtain the point cloud segmentation results of a single plant leaf; The segmented leaf point cloud is separated using a clustering algorithm to obtain individual leaves.
2. The point cloud leaf segmentation method based on geometric interaction and adaptive graph convolution according to claim 1, characterized in that, The process of acquiring point cloud data of a single plant includes: building a point cloud acquisition platform, taking pictures of a single plant to obtain images from different angles at 360°; and generating point cloud data from the acquired images using the motion structure recovery method.
3. The point cloud leaf segmentation method based on geometric interaction and adaptive graph convolution according to claim 2, characterized in that, Also includes: Color filtering is used to remove points unrelated to plants; uniform downsampling is used to control the number of points within a preset range, reducing data complexity while preserving the main features and shape information of the point cloud; and coordinate normalization is performed on each point.
4. The point cloud leaf segmentation method based on geometric interaction and adaptive graph convolution according to claim 1, characterized in that, The gating mechanism is used to fuse local features of point clouds obtained in two different geometric spaces: ； ； in, This represents the scalar multiplication operation of vectors, where W represents the weight matrix. Representing local features of Euclidean space, This represents the local features of hyperbolic space.
5. The point cloud leaf segmentation method based on geometric interaction and adaptive graph convolution according to claim 1, characterized in that, The loss function for model training is the cross-entropy plus the triplet loss: ； ； ； Where C represents the number of categories, The label indicating whether the i-th point belongs to category c. This represents the probability that the i-th point belongs to category c; This represents the feature vector of the i-th point. This represents the feature vector of a point that belongs to the same category as the i-th point. This represents the feature vector of a point that belongs to a different category than the i-th point. This indicates the distance between the anchor point and the positive sample.
6. The point cloud leaf segmentation method based on geometric interaction and adaptive graph convolution according to claim 1, characterized in that, It also includes calculating plant phenotypic parameters for isolated individual leaves using a spherical rotation algorithm and formula. The specific process includes: The plant height H is obtained by measuring the height of the bounding box surrounding the plant point cloud, using the maximum value of the point cloud on the z-axis. Subtract the minimum value ; The length and width of the blade are measured by calculating the distance between the farthest points on the blade along the x and y axes. The surface of the blade was reconstructed using a sphere rotation algorithm. It consists of a large number of triangles. The area of the triangles was calculated using Heron's formula, and the total area of the fitted surface of the blade was obtained by statistical analysis.
7. A point cloud leaf segmentation system based on geometric interaction and adaptive graph convolution, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is loaded into the processor, it implements the point cloud leaf segmentation method based on geometric interaction and adaptive graph convolution as described in any one of claims 1-6.