A method for segmenting plant leaves based on three-dimensional point clouds

Abstract

The present application belongs to the field of agricultural information technology, and particularly relates to a plant leaf segmentation method based on three-dimensional point cloud, and specifically comprises the following steps: step 1, data set establishment and optimization; step 2, design and optimization of a plant leaf segmentation algorithm; and step 3, model training test and comparison. The present application constructs a labeled three-dimensional synthetic point cloud data set, solves the problem of lack of sufficient labeled plant data set for training model in the current three-dimensional point cloud research, and designs a local aggregation operator which can explicitly encode shape information and capture different local geometric structures from low-level position relations and high-level feature associations. The present application can achieve good effects in plant leaf segmentation and object component segmentation in common scenes, and has strong practicability.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of agricultural information technology, and particularly relates to a plant leaf segmentation method based on three-dimensional point cloud. BACKGROUND

[0002] In the process of human cognition and understanding of the world, it is very important to analyze and separate various objects and their internal relations in a complex scene, and to interact with them. In fact, many of human daily behaviors are based on the segmentation of different parts of objects, and then the application of their functions. It is meaningful to guide machines to learn to segment different parts of objects in real scenes through computer vision, which helps the application of object function prediction and simulation in the field of intelligent robots, computer vision and graphics.

[0003] Multi-view based part segmentation method

[0004] Multi-view based methods first project three-dimensional point clouds into multiple two-dimensional views at different angles, and then use traditional convolutional neural networks (CNN) for view feature learning. However, these methods rely heavily on tangent estimation, which inevitably causes loss of three-dimensional geometric information. Voxel-based methods usually map three-dimensional point clouds into a regular three-dimensional grid, which can apply standard three-dimensional CNNs, but its limitations are in the large computational and memory costs, especially when dealing with dense three-dimensional data. Recently, hierarchical and compact structures such as K-d trees or octrees have been introduced to alleviate some resolution problems, but these methods still rely on the subdivision of boundary volumes rather than local geometry.

[0005] Point cloud based part segmentation method

[0006] Global aggregation method. Recent work focuses on directly processing unstructured and irregular point clouds. PointNet is a pioneering work that learns point-wise features independently through shared multi-layer perceptrons (MLPs) and aggregates global information through max-pooling. DeepSets realizes envelope invariance and extracts features for each point in PointNet individually. Although these methods have achieved impressive performance on some point cloud analysis tasks, they cannot encode the local environment between points, limiting the semantic understanding ability.

[0007] Local aggregation methods. To overcome the above shortcomings, some follow-up work adopts local aggregation operators to capture the fine geometric structure of neighboring points. Specifically, this kind of method first transforms the features of neighboring points and their relative coordinates into new features, and then accumulates all the transformed neighboring features with a reduction function. The point-based MLP method applies several point-wise transformation layers after concatenating relative positions and point features, which incorporates neighboring features. The adaptive feature-based method generally generates pseudo-features on a regular grid and then processes them regularly, among which the KPConv method selects grid points on a consistent spherical domain and simultaneously computes the corresponding pseudo-features. The adaptive weight-based method directly learns relative position-based convolution filters and obtains aggregation weights between points in the local subset. To simplify the local aggregation operator, the PosPool method combines the features of points and relative positions by element multiplication, which has no learnable weights but similar performance to existing complex operators. In addition, other methods construct a graph from the neighbors of each point and use a typical graph-based network to learn local features.

[0008] The point cloud-based component segmentation method mainly has the following problems:

[0009] 1. Problems of traditional segmentation methods.

[0010] Traditional mathematical-based point cloud model segmentation methods include surface growing method, edge detection method, scan line algorithm, graph-based segmentation method, clustering method, etc. Typical supervised learning algorithms commonly used include support vector machine, random forest and naive Bayes, but this kind of method does not fully consider the correlation between local point clouds, which leads to the result being easily affected by noise. Using multi-view method, three-dimensional space is projected into two-dimensional space, although it can fully utilize the currently relatively perfect two-dimensional algorithm model, but it is inevitable to lose three-dimensional geometric detail information. Although the voxel model can map three-dimensional point clouds into a three-dimensional space network, the computational and memory cost is large, and it depends on the subdivision of boundary voxels rather than geometric information. In the point cloud-based method, it is difficult to capture object details by aggregating only globally, which is very important for component segmentation tasks.

[0011] 2. Problems of plant dataset for point cloud component segmentation.

[0012] In plant model segmentation and classification tasks, a large amount of labeled point cloud data is needed for deep learning model training and quantitative analysis. However, currently only small-scale datasets can be found in specific literature—the ROSE-X dataset contains 11 real 3D models of rose bushes obtained through X-ray tomography, with point clouds labeled according to different organs. The Pheno4D dataset contains 49 labeled maize point clouds and 77 labeled tomato point clouds. Therefore, proposing new benchmark datasets suitable for measuring plant phenotypic parameters is beneficial for building high-quality deep learning models, thereby promoting efficient automated monitoring of plant traits, growth dynamics, and characteristic indicators, and contributing to the modernization of agriculture.

[0013] 3. Point cloud information fusion problem

[0014] Contextual information is crucial for extracting global latent representations and recognizing implicit shape patterns. Previous studies have employed multi-scale grouping strategies to acquire and combine context within the same layer; and used dot-product self-attention to dynamically learn attention weights and transformed representations. In these methods, features are computed by concatenating the outputs of all preceding layers to leverage dense contextual information. However, these methods either lack local-to-global semantic learning or are structurally complex, making it difficult to combine local structure with global context for accurate recognition. Summary of the Invention

[0015] This invention addresses the problem of limited 3D point cloud datasets for plants by constructing a point cloud dataset for a 3D synthetic plant model. It also proposes a component segmentation method based on 3D point clouds, applicable to plant leaf segmentation and component segmentation of common objects, effectively improving segmentation accuracy and timeliness.

[0016] A method for segmenting plant leaves based on 3D point clouds includes the following steps:

[0017] Step 1: Dataset creation and optimization;

[0018] Step 2: Design and optimization of plant leaf segmentation algorithm;

[0019] Step 3: Model training, testing, and comparison.

[0020] Preferably, the dataset creation and optimization in step 1 specifically includes:

[0021] Step 1.1: Virtual plant model creation; A three-dimensional synthetic plant model is generated based on L-systems rules, and then triangles are inlaid on the surface of the virtual plant model using primitives to obtain a triangular mesh plant model;

[0022] Step 1.2: Random point sampling; based on the barycentric coordinate system, a point cloud uniformly distributed on the surface of the triangular network plant model is constructed by random sampling method, and then the points inside the geometric primitives are removed by filtering to obtain the final point cloud model of the plant surface.

[0023] Preferably, in the process of generating the three-dimensional synthetic plant model in step 1.1, a cylinder and a sphere are used as geometric primitives to generate the three-dimensional synthetic plant model.

[0024] Preferably, the process of step 1.2 is as follows:

[0025] Step 1.2.1: Construct the barycentric coordinate system. For an affine space, the vertices in the space are represented as (v1, v2, …, vn). n If for a point P

[0026]

[0027] The coefficient set (α1, …, αn) is the barycentric coordinate of P with respect to (v1, v2, …, vn). n n

[0028] Step 1.2.2: Construct the point cloud. For a triangle with a vertex labeled (v1, v2, v3), define a random point m inside the triangle as

[0029]

[0030] In the above formula, α1, α2 and α3 satisfy and α3 can be calculated by randomly generated α1, α2, i.e. α3 = 1-(α1+α2).

[0031] Step 1.2.3: Filter and remove points inside the geometric primitives.

[0032] Since the points generated by the above process may be inside the primitives such as cylinders and spheres, there is a problem that the structure of the plant model is represented by multiple primitives, which may cause the intersection of space between the primitives. In this case, the points inside the primitives need to be removed. The geometric position of each point can be detected as follows:

[0033] For a cylindrical primitive, let the radius of the cylinder be r, and the two endpoints of the central axis be Given a random point To determine whether the point is inside the cylinder, the following judgment can be made:

[0034] First, determine whether is located between the upper and lower bases of the cylinder, which can be determined by the following two conditions​​

[0035]

[0036] If the above condition is true, the next step is to calculate the vertical distance from the point to the central axis. If the distance is less than the radius r of the cylinder itself at this time, the point is an internal point, and the formula is expressed as formula 3-4:

[0037]

[0038] For the sphere primitive, compared with the cylinder primitive, the sphere primitive is simpler to detect the interior: given the sphere center point coordinates and the sphere radius r, for the point to be tested only need to make the following judgment:

[0039]

[0040] If formula 3-5 is true, then that is, it is inside the primitive.

[0041] Preferably, the three-dimensional point cloud-based plant leaf segmentation method, the design and optimization of the plant leaf segmentation algorithm of step 2, specifically includes:

[0042] Step 2.1: Introduce random noise, randomly extract part of the sample data in the training set, and randomly generate a part of the noise point data, the number ratio of sample data and noise point data is 9:1;

[0043] Step 2.2: Design a local operator, design a local aggregation operator PFConv, for a given input point cloud P={p i |i=1,2,…,N}∈R N×3 , and the feature corresponding to each point in P can be represented as: F={f i |i=1,2,…,N}∈R N×D , then PFConv can be expressed as:

[0044]

[0045] Where N(p i ) is the point set contained in the neighborhood of the given center point p i ; p ij refers to each point in the neighborhood point set, f ij refers to the feature vector; the low-level positional relationship l i between the center point p ij and each point p ij in its neighborhood is calculated by the function Low(), and the feature f i is calculated by the function High().ij high-level feature association h ij The two results are fused by calculating the Hadamard product element by element or group by group, and then passing through a reduction function to reduce the dimension scale to obtain the output feature g of the center point i ;

[0046] Step 2.3: Construct a neural network model AFNet, the structure of AFNet is a five-layer multi-resolution encoder, each relationship-driven content fusion module, namely RCF module, is composed of three position-guided modules based on a local aggregation operator PFConv, each module uses a different encoding function, and a one-connection strategy is applied to the modules; for the network backend, a decoder is used for specific semantic segmentation;

[0047] Step 2.4: Optimize the neural network model, adjust the complexity of the model by setting the channel width d and the bottleneck rate w, and then optimize the performance of AFNet.

[0048] Preferably, a plant leaf segmentation method based on three-dimensional point cloud, the model training test and comparison of step 3 specifically comprises: the proposed AFNet is tested and evaluated on three part segmentation benchmarks containing different scenes, including the public dataset PartNet, ShapeNetPart and the dataset PlantSeg3D proposed by the application, the average intersection over union mcIoU and the average intersection over union mIoU are used to evaluate the effectiveness of each method.

[0049] The application has the advantages that:

[0050] 1. An annotated three-dimensional synthetic point cloud dataset is constructed, solving the problem of lack of sufficient annotated plant datasets for training models in current three-dimensional point cloud research.

[0051] 2. A local aggregation operator is designed, which can explicitly encode shape information and capture different local geometric structures from low-level positional relationships and high-level feature associations. The operator can be integrated into a deep hierarchical structure as a core unit to solve the problem of complex shape learning difficulty.

[0052] The application has a wide range of applications and can achieve good results in plant leaf segmentation and object part segmentation in common scenes, and has strong practicality. DETAILED DESCRIPTION

[0053] Figure 1 is a flowchart of a plant leaf segmentation method based on three-dimensional point cloud;

[0054] Figure 2 is a three-dimensional point cloud model of Arabidopsis thaliana;

[0055] Figure 3 is a local aggregation operator architecture diagram;

[0056] Figure 4 is an AFNet model architecture diagram;

[0057] Figure 5 is a sample display diagram of plant segmentation by different methods. DETAILED DESCRIPTION

[0058] The present application proposes a point cloud-based component segmentation model: AFNet. Through the network of local aggregation and content fusion, the spatial relationship and content semantics of points can be comprehensively acquired and understood. The model can acquire low-level position relationship and high-level feature association through a local aggregation operator. By constructing a deep network architecture, the features of each intermediate layer are collected in the feature output layer at one time, and complex shapes can be better segmented.

[0059] The overall process is shown in Figure 1 The process will be described in detail below with reference to the accompanying drawings.

[0060] Step 1: dataset establishment and optimization

[0061] 1) Virtual plant modeling

[0062] In the L-system rule, a plant can be defined by a symbolic string called L-string, and each symbol represents a different organ of the plant, such as A representing the plant tip / apex (Apex), I representing the plant internode / intermediate part (Internode), and L representing the plant leaf (Leaf). Each symbol is attached to a set of parameters to describe the state of the corresponding plant organ. An internode I can be modeled as a cylinder containing two parameters of height h and radius r, a leaf L can be represented by a polygon with s sides, and an apex A can contain an age parameter t. Square brackets [] are used to distinguish different branch components of an L-string.

[0063] The initial state of the L-string is called the axiom, which defines the initial state before the growth of a certain plant. For example, the axiom can initially consist of only one apex A(t) and can be represented as:

[0064] Axiom: A(0) # represents a plant at time 0, containing only one apex A

[0065] Then, under the L-system rule, a series of rules will make the initial L-string continuously grow and evolve. For example:

[0066] A(t)-->I(r,h,2)[+(a)[L(s,1,n)]A(t+dt)]A(t+dt)

[0067] The rule represents that every time period dt, the initial top A generates an internode I, a lateral bud A and a leaf L, and the original top A increases the height from the ground by h. In the rule, a represents the angle, which can be adjusted according to the plant growth time t.

[0068] During the generation process, the present invention uniformly labels the internode as "1" label; and for each leaf, it has its own label (2, n), where n represents the total number of leaves currently existing. As can be seen, each leaf has a unique label when it is generated, which facilitates observation during the experiment.

[0069] For the leaves and internodes in the above formula, the subsequent growth process can be described by the following rules respectively:

[0070] I(r,h,1)-->I(r+a*dt,r+(a+b)*dt,1) # depicts the growth of internodes

[0071] L(s)-->L(s') # with s'=s+c*dt # depicts the growth of leaves

[0072] After establishing the above generation rules, the symbols represented by the previous L-string also need to be translated into geometric models, and here "interpretation rules" are used, for example:

[0073] A(t)-->Sphere(radius(t))

[0074] I(r,h,1)-->Cylinder(r,h,1) # r represents the radius, h represents the height, and 1 is the label of the internode L(size,2,n)-->Polygon(size,2,n) # size is the number of polygon edges, and (2, n) is the label of the leaf

[0075] At present, the existing model is a mesh model represented by triangles of different sizes, which needs to be randomly sampled on the surface of the triangle. The random sampling method is to randomly select a triangle in each iteration process, and then generate a point at a random position inside the triangle. That is, if you want to generate a point cloud model with a size of n, you need to loop the random triangle selection process n times, and finally get n points. However, due to the size difference of the triangles themselves, the selection process will result in more points in the parts that contain more triangles. Therefore, the area of the triangle can be calculated as the selection probability during sampling, and finally a relatively uniform point cloud model is generated.

[0076] 2) Random point sampling

[0077] To perform point acquisition, a barycentric coordinate system is first introduced. The barycentric coordinate system is widely used in ray tracing in computer graphics. Using the barycentric coordinate system, the position of any point within a triangle can be represented. For an affine space, its vertices are represented as (v1, v2, ..., v...). n If, for component point P in this space, the following equation holds:

[0078]

[0079] Then, the coefficient set (α1,…,α) n ) means P for (v1, v2, ..., v n The centroid coordinates of ).

[0080] Specifically, for a tessellated triangle with vertices labeled (v1, v2, v3), a random point m inside the triangle can be defined as:

[0081]

[0082] Equation 3-2 contains and Established.

[0083] At this point, α3 can be calculated using randomly generated α1 and α2: α3 = 1 - (α1 + α2). This results in a uniformly distributed plant point cloud model containing a specific number of points. The constructed Arabidopsis thaliana 3D point cloud model is shown below. Figure 2 As shown. Figure 3 Data on plant leaf quantity from day 5 to day 40 were collected. PlantSeg3D can be applied to many tasks, providing basic data references for point cloud segmentation, plant shape analysis, phenotypic parameter calculation, and other fields, which is beneficial to the interdisciplinary development of computer vision, biology, and agronomy.

[0084] Step 2: Model Creation and Optimization

[0085] This invention proposes a plant leaf segmentation method based on computer vision and deep learning technologies, and designs a point cloud segmentation network based on local aggregation and content fusion. The design and implementation process of the network is as follows:

[0086] 1) Using the PlantSeg3D dataset generated above as the training set for the model, 9,000 points were extracted from each Arabidopsis thaliana simulation model during the experiment, and then 1,000 noise points were randomly generated and added to the model. By introducing random noise, the effect of real sensor data acquisition was simulated, and the data was input into the target network for model training.

[0087] 2) Based on the above dataset, design the local aggregation operator PFConv, such as... Figure 4 As shown, for a given input point cloud P = {p i |i=1,2,…,N}∈R N×3 Given N points in P, the feature corresponding to each point can be represented as follows: F = {f i |i=1,2,…,N}∈R N×D Based on this, for each point, the local aggregation operator PFConv, which computes high-level feature relations and low-level positional relations, is expressed as:

[0088]

[0089] Wherein, N(p) i Given a center point p i The set of points contained in its neighborhood; p ij f refers to each point within the neighborhood set. ij This refers to the eigenvector. In other words, the center point p is calculated using the Low() function. i With each point p in its neighborhood ij Low-level positional relationship l ij And the function High() calculates the feature f i with f ij High-level feature association h ij The two results are then fused by calculating the Hadamard product element-wise or group-wise (i.e., multiplying corresponding elements of the two matrices separately). Finally, the dimensionality is reduced using a reduction function to obtain the output feature g of the center point. i .

[0090] For p i Each neighbor point p ij Specify mapping function To establish the neighbor point and the center point p i The geometrical positional association of p. Different encoding functions can be used to establish p. i and p ij Three geometric priors between:

[0091] a. Position Embedding: Using sine and cosine functions, the coordinates of points in different 3D XYZ dimensions can be transformed and embedded into vectors, thus representing the features of points in the same dimension. The k-th position embedding... The following formula can be used for calculation:

[0092]

[0093] In the above formula, is the relative distance; d is the number of channels;

[0094] b. Coordinates and distances: here, the coordinates and distances are not only calculated three-dimensional relative distance Δp ij , but also fully consider the distance between two points and add it to the calculation:

[0095]

[0096] c. Angle and distance: in this variable, by comprehensively using angle and distance, the following encoding function can be constructed:

[0097]

[0098] 3) Based on the above local aggregation operator, the network is constructed. As shown in the figure, the skeleton part of the constructed AFNet network is composed of a five-layer multi-resolution encoder. Each relationship-driven content fusion module (RCF module) is composed of three position-guided modules based on the local aggregation operator PFConv, each module uses different encoding functions, and a one-connection strategy is applied to these modules. For the network backend, through the use of a decoder, specific semantic segmentation is carried out. Figure 5 4) Through experimental verification, by setting the channel width d and the bottleneck rate w, the complexity of the model can be adjusted, when the simplified configuration of d=48, ω=2 is adopted, the network is about 10 times more efficient than the standard AFNet, which fully embodies the stability and adaptability of the PFConv local aggregation operator and the RCF block proposed in the application.

[0099] Step 3: Model training test and comparison

[0100]

[0101] ​The proposed AFNet is tested on three part segmentation benchmarks containing different scenarios, including the public dataset PartNet, ShapeNetPart and the dataset PlantSeg3D proposed in the application. For the ShapeNetPart dataset, the application uses the mean class intersection over union mcIoU and the mean intersection over union mIoU to evaluate the effectiveness of each method. It can be seen that the mcIoU of the AFNet model proposed in the application is 85.6%, and the mIoU is 86.8%, both of which are the highest values at present. There is still a large difference in the mIoU obtained by the current model on different types of objects. Although the current PAConv has achieved good results in mcIoU and mIoU, AFNet exceeds it by 1.0% and 0.7% in mIoU and mcIoU, respectively. Since AFNet can start from low-level positional relationships and high-level feature associations, and learn complex structures in point clouds, it achieves good results of 56.1% and 52.8% on the challenging PlantSeg3D and PartNet, respectively, which improves the mIoU by 4.1% and the mIoU by 6.4% compared with previous methods. Through the above comparative tests, it can be known that the application further improves the accuracy of the part segmentation task of objects in a general scenario.

Claims

1. A method for segmenting plant leaves based on three-dimensional point clouds, characterized in that, Includes the following steps: Step 1: Dataset creation and optimization; Step 1, dataset creation and optimization, specifically includes: Step 1.1: Virtual plant model creation; A three-dimensional synthetic plant model is generated based on L-systems rules, and then triangles are inlaid on the surface of the virtual plant model using primitives to obtain a triangular mesh plant model; Step 1.2: Random point sampling; Based on the centroid coordinate system, a uniformly distributed point cloud is constructed on the surface of the triangular network plant model by random sampling method. Then, the points inside the geometric primitives are removed by filtering to obtain the final point cloud model of the plant surface. The process in step 1.2 is as follows: Step 1.2.1: Construct a barycentric coordinate system. For an affine space, represent its vertices as (v1, v2, ..., v...). n If for a point P in this space Then, the coefficient set (α1,…,α) n ) means P for (v1, v2, ..., v n The centroid coordinates of ) Step 1.2.2: Construct a point cloud. For a triangle with vertices labeled (v1, v2, v3), define a random point m inside the triangle as... The above formula contains and It is true; α3 can be calculated using randomly generated α1 and α2, i.e., α3 = 1 - (α1 + α2); Step 1.2.3: Filtering to remove internal points of geometric primitives: For a cylindrical primitive, the following judgment is made: In the above formula, r is the radius of the cylinder's ground surface. These are the two endpoints of the central axis. For a random point, if all three equations above are true, then Inside the cylindrical element; For a sphere primitive, the following judgment is made: In the above formula, the coordinates of the center point of the sphere are... A sphere with radius r, at any point If the above expression is true, then Inside the spherical element; Step 2: Design and optimization of plant leaf segmentation algorithm; The design and optimization of the plant leaf segmentation algorithm in step 2 specifically includes: Step 2.1: Introduce random noise. Randomly extract a portion of sample data from the training set and then randomly generate a portion of noise data. The ratio of sample data to noise data is 9:

1. Step 2.2: Design local operators, specifically the local aggregation operator PFConv, for a given input point cloud P = {p i |i=1,2,…,N}∈R N×3 Given N points in P, the feature corresponding to each point can be represented as: F = {f i |i=1,2,…,N}∈R N×D Then PFConv can be represented as: Wherein, N(p) i ) is a given center point p i The set of points contained within the neighborhood; p ij f refers to each point within the neighborhood set. ij The eigenvector is used to calculate the center point p using the Low() function. i With each point p in its neighborhood ij Low-level positional relationship l ij The feature f is calculated using the function High(). i with f ij High-level feature association h ij The two results are fused by calculating the Hadamard product element-wise or group-wise, and then the dimensionality is reduced by a reduction function to obtain the output feature g of the center point. i ; Step 2.3: Construct the neural network model AFNet. The structure of AFNet is a five-layer multi-resolution encoder. Each relation-driven content fusion module, i.e., RCF module, is composed of three position-guided modules based on the local aggregation operator PFConv. Each module uses a different encoding function, and a one-time connection strategy is applied to these modules. For the network backend, a decoder is used to perform specific semantic segmentation. Step 2.4: Optimize the neural network model by adjusting the complexity of the model by setting the channel width d and the bottleneck ratio w, thereby optimizing the performance of AFNet. Step 3: Model training, testing, and comparison.

2. The plant leaf segmentation method based on three-dimensional point clouds according to claim 1, characterized in that, In the process of generating the three-dimensional synthetic plant model described in step 1.1, cylinders and spheres are used as geometric primitives to generate the three-dimensional synthetic plant model.

3. The plant leaf segmentation method based on three-dimensional point clouds according to claim 1, characterized in that, The model training, testing, and comparison in step 3 specifically includes: evaluating and testing the proposed AFNet on three part segmentation benchmarks containing different scenarios, including the public datasets PartNet, ShapeNetPart, and PlantSeg3D, and evaluating the effectiveness of each method using the mean inter-class intersection-union ratio (mcIoU) and the mean intersection-union ratio (mIoU).

Images