A method for measuring the leaf area of seedlings under occlusion conditions

Abstract

The present invention relates to a method for measuring the leaf area of seedlings under occlusion conditions. This method takes seedlings as the object, uses the KINECT platform to obtain the top-view depth image of the seedlings, and then generates the point cloud of a single seedling at this perspective. This method improves the accuracy of leaf area measurement from two aspects. On the one hand, a method of neighborhood space constraint is proposed to effectively filter the suspended points and outlier noises in the point cloud, significantly improving the quality of the point cloud data. On the other hand, a new method for point cloud segmentation and completion based on the neighborhood aggregation strategy and the neighborhood interaction fusion module, abbreviated as the MIX-Net network, is proposed. This method can simultaneously achieve point cloud segmentation and completion, and a good balance is achieved between the two. This method surpasses other technical solutions in the prior art and obtains more accurate measurement results. This method forms a complete set of automated phenotypic data processing solutions including high-throughput seedling collection, automatic segmentation of seedling organs, completion of occluded leaves, and extraction of phenotypic data.

Description

Technical Field

[0001] The invention belongs to the field of agricultural information technology and automated crop phenotype measurement, and in particular relates to a method for measuring seedling leaf area under shielding conditions. Background Art

[0002] Leaves are an important part of the external morphology of plants and are also the main organs for plants to carry out physiological functions. The traditional leaf area measurement method uses a leaf area meter, which calculates leaf parameters based on the pixel points generated by the two-dimensional projection of the leaf. In fact, factors such as growth morphology and deformation make it impossible for the leaf to be an absolute plane, and direct measurement using two-dimensional images will lead to measurement errors. Only based on the morphological configuration of the leaf in three-dimensional space can more accurate measurements be achieved. With the rapid development of sensing technology and the improvement of computing performance, rapid data acquisition and phenotype extraction on a three-dimensional scale have become possible. LiDAR, low-cost depth cameras, and multi-view imaging technologies are widely used to obtain 3D plant data. In general, current plant three-dimensional data processing technology is very time-consuming and requires a lot of manual interaction, resulting in a large amount of raw data accumulation. Therefore, there is an urgent need to design and develop new methods to improve the automation of three-dimensional plant phenotyping. However, in order to achieve this goal, three challenges must be addressed.

[0003] First, we need to solve the quality problem of point cloud data. Specifically, the data obtained from the RGB-D sensor is relatively rough, and outlier noise and floating points are relatively serious. It is difficult to filter out these noisy point clouds using traditional methods such as radius filtering and straight-through filtering.

[0004] Secondly, it is to improve the segmentation accuracy of plant organs, which plays a fundamental role in plant leaf area measurement. In particular, 2D-based methods can use traditional image processing and deep convolutional neural networks (CNNs) to achieve segmentation. Compared with 2D images, 3D models not only contain information about color and texture, but also carry the most important depth information. In recent years, phenotypic measurement based on 3D models has attracted more and more research. However, directly using these methods to achieve point cloud data segmentation of seedlings will significantly reduce performance due to the complex structure of seedlings and the mutual occlusion between leaves.

[0005] Finally, it is necessary to deal with the missing leaf data due to occlusion. Although this technology is an emerging field in the application of plant 3D phenotyping, it has always been an important research problem in the graphics and vision community. Point cloud completion methods based on deep learning have made some research progress, but still face some challenges, such as high computational effort and low resolution. Summary of the invention

[0006] 1. Technical issues to be resolved

[0007] The object of the present invention is to overcome the deficiencies of the above-mentioned prior art, and provide a method for measuring the leaf area of seedlings under occlusion conditions, which can effectively filter out suspended points and noise points in the point cloud, and can also achieve high-precision point cloud segmentation for seedlings with occlusion and complete the missing leaves, realizing automatic and accurate measurement of the leaf area of seedlings.

[0008] (II) Technical solution

[0009] To solve the above problems, the present invention provides the following technical solution, and proposes a method for measuring the leaf area of seedlings under occlusion conditions, specifically as follows.

[0010] A method for measuring the leaf area of seedlings under occlusion conditions includes the following steps:

[0011] S1, Use a depth camera to obtain the depth image of the seedlings, and convert the depth image into a point cloud;

[0012] S2, Adopt the neighborhood space constraint method to filter out the suspended points and outlier noise in the point cloud;

[0013] S3, Based on the seedling point cloud data processed in step S2, use cloud compare to construct a seedling segmentation data set, and use the missing simulation method to construct a seedling leaf completion data set;

[0014] S4, Construct an encoder-decoder seedling segmentation network model and an encoder-decoder leaf completion network model; in the encoders of both network models, feature extraction is performed using the neighborhood aggregation strategy and the feature is processed by the neighborhood interaction fusion module, and in the decoders, both include upsampling to the same resolution as the input and the neighborhood interaction fusion module for decoding; the differences between the seedling segmentation network model and the leaf completion network model include that the final output of the decoder of the former first performs max pooling on the seedling point cloud and then reduces the dimension through a multi-layer perceptron to predict a category for each point;

[0015] S5, Use the seedling segmentation data set and the seedling leaf completion data set in step S3 to train on the two network models constructed in step S4 respectively to determine the weights of the seedling segmentation network model and the leaf completion network model;

[0016] S6, Use the occluded seedlings to be tested as the input, separate the stems and leaves of the seedlings based on the weights of the seedling network segmentation model determined in step S5, and then complete the missing leaves due to occlusion through the weights of the leaf completion network model determined in step S5, and finally measure the leaf area by greedy triangulation.

[0017] Preferably, in step S1, a depth camera connected to an external computer is installed directly above the seedlings to collect high-throughput data, obtain a 1024*1024 depth map, and convert the depth image into a point cloud. The filtering process in step S2 specifically includes the following steps:

[0018] S21, the original point cloud is filtered through a straight-through filter to obtain a point cloud containing only the plant area;

[0019] S22, set a threshold N, use the K nearest neighbor algorithm to find N neighborhoods around each center point, and find the average value D of the Euclidean distance between the center point and the neighborhood;

[0020] S23, using the set threshold N, using the least squares method to fit the plane to predict the normal vector of each center point, and obtaining the angle W between the normal vector and the z-axis;

[0021] S24, repeat S22 and S23, if D ≥ d, it is judged as a floating point, if W ≥ c, it is judged as an outlier, and the floating point and the outlier are deleted, where d is the set distance threshold and c is the set angle threshold;

[0022] S25, traverse the entire point cloud and remove all floating points and outlier points.

[0023] Preferably, when filtering out floating point and outlier noise, the threshold N is set to 12, d is set to 0.0034, and c is set to 60°.

[0024] Preferably, the specific implementation method of step S3 is to use cloud compare to manually annotate stems and leaves to produce a training data set for seedling segmentation, with stem points marked with 0, leaf points marked with 1, and needles marked with 2; when using the missing simulation method to produce a seedling leaf completion data set, with a complete leaf object in the seedling, assuming that a seedling leaf point cloud S is selected, and randomly select [1, 1, 1], [1, 1, -1], [1, -1, 1], [1, -1, -1], [-1, 1, -1], [-1, -1, 1], [-1, -1, 1], [-1, -1, 1 ], [-1, -1, -1], [0, 0, 0] as the viewpoint V, calculate the distance from the viewpoint V to each point in the seedling point cloud S as Dvs, and sort them from small to large; randomly select one of the values 128, 256, 512 as the missing point value of the seedling point cloud leaf, and set the point coordinate value from the minimum point to the missing point value in Dvs to 0; finally, the seedling leaf point cloud after the above processing and the complete seedling leaf S constitute a data pair and put them into the seedling leaf completion data set.

[0025] Preferably, in step S4, a neighborhood aggregation strategy is used to extract features. The specific implementation method is as follows: Assume that the neighborhood feature aggregation layer is a layer with N points and corresponding features F nThe seedling point cloud is used as input, and a sampled seedling point cloud with Ns points and its corresponding aggregated feature F are output s ; First, the farthest point sampling algorithm is used to downsample the number of the seedling point cloud from N to Ns and the feature F n Downsample to F i ; Then, with each sampled point in F i as the center, find the nearest k points in the feature F n to form a neighborhood and use F ik to represent its neighborhood feature, and output the feature F s The calculation method is shown in formula (1):

[0026] F s = MP(LBR(LBR(concat(F i - F ik , RP(F i , k))))) (1)

[0027] Among them, MP is the max pooling operator, RP(F i , k) is the operator that repeats the vector F i k times to form a matrix, LBR contains a Linear layer, a BatchNorm layer and a ReLU layer, and concat means concatenating two features. In step S4, the neighborhood interaction fusion module processes the features. The specific implementation method is: the sequence of non-overlapping point cloud groups F s generated by the seedling point cloud through the local neighborhood aggregation strategy is used as input, and the sequence of point cloud groups is linearly projected to the dimension F o using the same projection matrix; among them, the network neighborhood interaction fusion module consists of two layers of the same size, and each layer consists of two MLP blocks; the first layer is the intra-group mixing of the point cloud group: it acts between the inside of a certain point cloud group and maps the sequence of point cloud groups F s to F c ; the second layer is the channel mixing: it acts between the sequences of each point cloud group and maps F c back to the same dimension as F s and is represented by F o ; each MLP block contains two fully connected layers and a non-linear activation function layer; the whole process is shown in formula (2):

[0028] ;

[0029] T is the flipping operation, F c and F o are the adjustable hidden features in the intra-group mixing and the channel mixing respectively, and LayerNorm represents layer normalization

[0030] Preferably, the model of step S5 constructs an encoder-decoder shaped point cloud segmentation and completion method, referred to as MIX-Net; wherein the encoder adopts multi-resolution progressive (2048, 1024, 512, 256) feature extraction, and the seedling point cloud processing in each resolution is the same as the encoder processing method in step S4; the decoder adopts multi-resolution progressive (256, 512, 1024, 2048) to predict the complete seedling point cloud; the seedling point cloud prediction in each resolution is the same as the decoder processing method in step S4; the difference between the seedling segmentation network model and the leaf completion network model includes that the final output seedling point cloud of the encoder is first maximum pooled and then reduced in dimension through a multi-layer perceptron to predict a category for each point; finally, the seedling segmentation and leaf completion data sets constructed in step S3 are respectively input into MIX-Net for training to obtain the seedling segmentation network model weights and the leaf completion network model weights.

[0031] Preferably, step S6 accurately measures the leaf area of the seedlings, specifically by collecting depth images of single plants or whole trays of seedlings with occlusion, and processing them according to S2, and obtaining seedlings with separated stems and leaves based on the processed seedling point cloud according to the weights of the seedling segmentation network model trained in step S5; then, the leaves missing due to occlusion are completed according to the weights of the leaf completion network model trained in step S5, and finally, the leaf area phenotype of the single plant or the whole tray is accurately measured through greedy triangulation.

[0032] (III) Beneficial effects

[0033] Compared with the prior art, the method for measuring seedling leaf area under shielding conditions provided by the present invention has significant positive technical effects, which are specifically manifested in the following aspects.

[0034] (1) The present invention adopts step S2 to solve the quality problem of point cloud data and obtains a processing effect that is significantly better than the prior art. The seedling data obtained from the RGB-D sensor is relatively rough, and outlier noise and suspended points are often more serious. Traditional methods use radius filtering and pass-through filtering to filter out these noisy point clouds. The present invention proposes a neighborhood space constraint method, which creatively combines the spatial distance and normal vector constraints within the seedling point cloud to accurately and efficiently filter out the suspended point and outlier noise in the point cloud.

[0035] (2) This application uses the seedling point cloud segmentation model in step S4 and the seedling point cloud segmentation dataset constructed in step S2 to solve the problem of seedling segmentation under occlusion conditions. Compared with traditional two-dimensional and three-dimensional segmentation methods, the present invention is an accurate and effective encoder-decoder seedling segmentation network model. This model is fused with a high-precision point cloud completion model, and with the help of the efficient feature extraction ability of the point cloud completion model, point cloud segmentation in more complex situations is realized. Even if the seedlings have complex structures and the leaves are occluded from each other, the stems and leaves of the seedlings can be separated, and the performance is significantly improved.

[0036] (3) This application uses the seedling point cloud completion model in step S4 and the seedling point cloud completion dataset constructed in step S2 to solve the problem of accurate measurement of seedling leaf area under occlusion conditions. In the prior art, the problem of accurate measurement of seedling leaf area under occlusion conditions has not been well solved. Most of them only complement and predict the missing leaves according to the inherent shape of the leaves, and then measure the leaf area of the leaves. However, this method has low efficiency, low applicability and poor effect. The encoder-decoder seedling completion network model proposed by the present invention can extract rich features of the leaves through the neighborhood aggregation strategy and the neighborhood interaction fusion module, and can easily complement various missing-shaped leaves, making the measurement of seedling leaf area more accurate. Moreover, the end-to-end method of the present invention has significantly higher efficiency than the prior art. Description of the Drawings

[0037] Figure 1 It is a process diagram of seedling data acquisition;

[0038] Figure 2 It is a process diagram of seedling point cloud filtering process;

[0039] Figure 3 It is a process diagram of the production of the seedling segmentation dataset and the leaf completion dataset;

[0040] Figure 4 It is a diagram of the seedling segmentation network model and the leaf completion network model;

[0041] Figure 5 It is a diagram of the neighborhood aggregation strategy;

[0042] Figure 6 It is a diagram of the neighborhood interaction fusion module;

[0043] Figure 7 It is a comparison diagram of leaf area measurement results. Detailed Embodiments

[0044] The present invention will be further described below in conjunction with the drawings and embodiments.

[0045] Figure 1 It is a process diagram of seedling data acquisition. Starting from Figure 1(a) Obtain seedlings from the greenhouse for planting seedlings, and then put them into Figure 1 (b)'s data acquisition device to obtain Figure 1 (c)'s depth image, and finally convert the depth image into Figure 1 (d)'s three-dimensional point cloud.

[0046] Figure 2 This is a process diagram for filtering the point cloud of seedlings. Filter the obtained point cloud of seedlings. In Figure 2 (a) represents the original point cloud of seedlings. Subsequently, filter out the background and the ground through direct filtering. Figure 2 (b) represents the point cloud after filtering out the background and the ground. Then, filter out the suspended points and noise points through neighborhood space constraint as Figure 2 (c) shows.

[0047] Figure 3 This is a process diagram for making the seedling segmentation data set and the leaf completion data set. As Figure 3 (1) shows, use cloudcompare to manually annotate the stems and leaves to make the training data set for seedling segmentation. The stem points are marked with 0, the leaf points are marked with 1, and the needle leaves are marked with 2; as Figure 3 (2) shows, use the missing simulation method to generate the missing leaves. Using the above methods, construct the seedling segmentation data set and the seedling leaf completion data set respectively.

[0048] Figures 4 - 6 This is a diagram of the seedling segmentation network model and the leaf completion network model, the neighborhood aggregation strategy diagram, and the neighborhood interaction fusion module diagram. Figure 4 This is to construct an encoder-decoder seedling segmentation network model and an encoder-decoder leaf completion network model, which includes the neighborhood aggregation strategy diagram (Neighbor Point Aggregation), the neighborhood interaction fusion module (Point-mixer), and upsampling (Up-conv). Input represents the input, Output represents the output, [B, 128] and [B, 128, 256] represent the feature dimensions, Skip Connections represents feature splicing, Max-pooling represents max pooling, and Repeat represents repeating the feature. In the encoders of both network models, feature extraction is performed using the neighborhood aggregation strategy as Figure 5 shown and the feature is processed by the neighborhood interaction fusion module as Figure 6 shown. In the decoders, both include upsampling to the same resolution as the input and the neighborhood interaction fusion module as Figure 6 shown for decoding; the differences between the seedling segmentation network model and the leaf completion network model include that the final output point cloud of seedlings in the decoder first undergoes max pooling and then dimensionality reduction through a multi-layer perceptron to predict a class for each point. Figure 5Among them, BxNxD represents the feature dimension, FPS represents the farthest point sampling algorithm, K-NN represents the K-nearest neighbor algorithm, and Matrix sub represents feature subtraction. Figure 6 Among them, Skip-connections means adding the features at both ends.

[0049] Using the seedling segmentation dataset and the seedling leaf completion dataset, training is respectively carried out on Figure 4 two constructed network models to determine the weights of the seedling segmentation network model and the leaf completion network model.

[0050] Taking the occluded seedlings to be tested as the input, separating the stems and leaves of the seedlings based on the determined weights of the seedling network segmentation model, then completing the missing leaves due to occlusion through the weights of the leaf completion network model, and finally measuring the leaf area through greedy triangulation. Figure 7 (1) shows the leaf area correlation index between measuring the leaf area by greedy triangulation and measuring the leaf area by a manual leaf area meter when 40 seedlings are under occlusion conditions without completing the missing leaves; Figure 7 (2) shows the leaf area correlation index between measuring the leaf area by greedy triangulation and measuring the leaf area by a manual leaf area meter after completing the missing leaves by adopting the technical solution of the present invention; compared with the two, R in the leaf area correlation index 2 increases, while MSE decreases; where R 2 represents variance, and the larger the value, the better; MSE represents mean, and the smaller the value, the better. It can be seen from this that the present application can achieve accurate measurement of the seedling leaf area under occlusion conditions.

[0051] The specific examples described in the application are only illustrative of the spirit of the present invention. Those skilled in the technical field to which the present invention pertains can make various modifications or supplements to the specific examples described in the present invention, or use alternative methods of the same type, but will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A method for measuring the leaf area of seedlings under occlusion conditions, characterized in that, The steps include: S1, using a depth camera to obtain a depth image of the seedlings, and converting the depth image into a point cloud; S2, using the neighborhood space constraint method to filter out the floating points and outlier noise in the point cloud; S3, based on the seedling point cloud data processed in step S2, using cloud compare to construct a seedling segmentation dataset, and using a missing simulation method to construct a seedling leaf completion dataset; S4, constructing an encoder-decoder seedling segmentation network model and an encoder-decoder leaf completion network model; the encoders of the two network models both include a neighborhood aggregation strategy for feature extraction and a neighborhood interaction fusion module for feature processing, and the decoders both include upsampling to the same resolution as the input and a neighborhood interaction fusion module for decoding; The difference between the seedling segmentation network model and the leaf completion network model is that the final output of the decoder of the former is first max-pooled and then reduced in dimension by a multi-layer perceptron to predict a category for each point; S5, using the seedling segmentation data set and the seedling leaf completion data set in step S3, respectively training on the two network models constructed in step S4 to determine the seedling segmentation network model weight and the leaf completion network model weight; S6, using the occluded seedlings to be tested as input, separating the stems and leaves of the seedlings based on the seedling network segmentation model weights determined in step S5, then completing the leaves missing due to occlusion using the leaf completion network model weights determined in step S5, and finally measuring the leaf area through greedy triangulation; The specific implementation of step S3 is to use cloud compare to manually annotate stems and leaves to produce a training data set for seedling segmentation, with stem points marked with 0, leaf points marked with 1, and needles marked with 2; When the missing simulation method is used to produce the seedling leaf completion dataset, the complete leaf object in the seedling is taken as an example. Assume that a seedling leaf point cloud S is selected, and a point in the space [1, 1, 1], [1, 1, -1], [1, -1, 1], [1, -1, -1], [-1, 1, 1], [-1, 1, -1], [-1, -1, 1], [-1, -1, -1], [0, 0, 0] is randomly selected as the viewpoint V, and the distance from the viewpoint V to each point in the seedling point cloud S is calculated and recorded as Dvs, and sorted from small to large; one of the values 128, 256, and 512 is randomly selected as the missing point value of the seedling point cloud leaf, and the point coordinate value from the minimum point in Dvs to the missing point value is set to 0; finally, the seedling leaf point cloud and the complete seedling leaf S after the above processing constitute a data pair and are put into the seedling leaf completion dataset.

2. The method for measuring the leaf area of seedlings under occlusion conditions according to claim 1, wherein In step S1, a depth camera connected to an external computer is installed directly above the seedlings to collect high-throughput data, obtain a 1024*1024 depth map, and convert the depth image into a point cloud.

3. The method for measuring the leaf area of seedlings under occlusion conditions according to claim 1, wherein, The filtering process in step S2 specifically includes the following steps: S21, the original point cloud is filtered through a straight-through filter to obtain a point cloud containing only the plant area; S22. Set the threshold N, use the K-nearest neighbor algorithm to find N neighborhoods around each center point, and calculate the average value D of the Euclidean distance between the center point and the neighborhoods. S23. Through the set threshold N, use the least squares method to fit a plane to predict the normal vector of each center point, and obtain the angle W between the normal vector and the z-axis. S24. Repeat S22 and S23. If D≥d, it is judged as a floating point; if W≥c, it is judged as an outlier, and the floating points and outliers are deleted, where d is the set distance threshold and c is the set angle threshold. S25. Traverse the entire point cloud and remove all floating points and outliers.

4. The method for measuring the leaf area of seedlings under the occlusion condition according to claim 3, wherein When filtering floating point and outlier noise, the threshold N is set to 12, d is set to 0.0034, and c is set to 60°.

5. The method for measuring the leaf area of seedlings under occlusion conditions according to claim 1, wherein In step S4, a neighborhood aggregation strategy is adopted for feature extraction. The specific implementation method is as follows: Assume that the neighborhood feature aggregation layer takes a seedling point cloud with N points and corresponding features F n as input, and outputs a sampled seedling point cloud with Ns points and its corresponding aggregated features F s ; First, the farthest point sampling algorithm is used to downsample the number of the seedling point cloud from N to Ns, and the feature F n is downsampled to F i ; Then, with each sampling point in F i as the center, find the nearest k points in the feature F n to form a neighborhood and use F ik to represent its neighborhood features Output feature F s is calculated as shown in formula (1): F s = MP(LBR(LBR(concat(F i -F ik , RP(F i , k))))) (1) Among them, MP is the max pooling operator, and RP(F i , k) is the operator that repeats the vector F i k times to form a matrix. LBR includes a Linear layer, a BatchNorm layer, and a ReLU layer. Concat means concatenating two features.

6. The method for measuring the leaf area of seedlings under occlusion conditions according to claim 1, wherein In step S4, the neighborhood interaction and fusion module processes the features. The specific implementation method is as follows: The sequence of non-overlapping point cloud groups F generated by the seedling point cloud through the local neighborhood aggregation strategy s is used as the input. The sequence of point cloud groups is linearly projected to dimension F using the same projection matrix o ; The network neighborhood interaction and fusion module consists of two layers of the same size, and each layer consists of two MLP blocks; The first layer is the in-group mixing of point clouds: it acts within a certain point cloud group, and maps the sequence of point cloud groups F s to F c ; The second layer is the channel mixing: it acts between the sequences of point cloud groups, and maps F c back to the same dimension as F s and is represented by F o ; Each MLP block contains two fully connected layers and a non-linear activation function layer; The whole process is shown in formula (2): ； T is the flipping operation, F c and F o are adjustable hidden features in intra-group mixing and channel mixing, respectively, and LayerNorm represents layer normalization.

7. The method for measuring the leaf area of seedlings under occlusion conditions according to claim 1, wherein The method for constructing the encoder-decoder-shaped point cloud segmentation and completion in step S5 is abbreviated as MIX-Net. Among them, the encoder uses multi-resolution progressive (2048, 1024, 512, 256) feature extraction, and the processing of the seedling point cloud in each resolution is the same as the processing method of the encoder in step S4. The decoder uses multi-resolution progressive (256, 512, 1024, 2048) to predict the complete seedling point cloud. The prediction of the seedling point cloud in each resolution is the same as the processing method of the decoder in step S4. The differences between the seedling segmentation network model and the leaf completion network model include that the final output seedling point cloud of the encoder is first max-pooled and then dimension-reduced through a multi-layer perceptron to predict a category for each point. Finally, the seedling segmentation and leaf completion datasets constructed in step S3 are respectively input into MIX-Net for training to obtain the weights of the seedling segmentation network model and the weights of the leaf completion network model.

8. The method for measuring the leaf area of seedlings under occlusion conditions according to claim 1, characterized in that Step S6 accurately measures the leaf area of the seedlings. Specifically, collect the depth images of single plants or whole trays of seedlings with occlusion, and process them according to S2. The processed seedling point cloud is based on the weights of the seedling segmentation network model trained in step S5 to obtain the seedlings with separated stems and leaves. Then, the missing leaves due to occlusion are completed according to the weights of the leaf completion network model trained in step S5. Finally, the leaf area phenotype of single plants or whole trays is accurately measured through greedy triangulation.

Images