A non-destructive measurement method for morphological indicators of Masson pine seedlings

Abstract

The present invention discloses a non-destructive measurement method for the morphological indexes of Masson pine seedlings, comprising: collecting and correcting Masson pine images; reconstructing the collected images into 3D point clouds and pre-processing the point clouds; creating a dataset for semantic segmentation of the Masson pine point clouds, calculating class balance weights for different classes in the dataset; constructing a densely weighted semantic segmentation network model; assigning the class balance weights to the loss function of the model; training the densely weighted semantic segmentation network model using the dataset; obtaining the original skeleton points of the stem based on the points of the Masson pine stem in the Masson pine point cloud segmented by the densely weighted semantic segmentation network model, and refining and extracting the skeleton points of the stem; and extracting the morphological index values of the Masson pine to be measured based on the refined skeleton points of the stem. The present invention can accurately measure the morphological indexes of Masson pine seedlings, no longer relying on manual labor and specialized instrument measurement equipment, and reducing measurement costs and difficulty.

Description

Technical Field

[0001] The present invention relates to the field of smart forestry, and in particular to a non-destructive measurement method for morphological indicators of Masson pine seedlings. Background Art

[0002] As an important tree species in my country, Masson pine boasts strong adaptability and a high degree of comprehensive utilization. It is not only a pioneering species for afforestation on barren hills, but is also used in construction, papermaking, and even medicinally. Selecting different grades of Masson pine seedlings for different applications can fully utilize its excellent properties. Therefore, studying the morphological indicators of Masson pine seedlings to construct a Masson pine seedling evaluation system is of great theoretical and practical significance.

[0003] Traditionally, the measurement of morphological indicators of Masson pine seedlings has relied on manual labor. However, labor costs are high nowadays, and long-term measurements will lead to increased errors. More importantly, the accurate measurement of some morphological indicators requires specialized instruments and measuring equipment, which undoubtedly increases the difficulty and cost of measurement. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a non-destructive measurement method for the morphological indicators of Masson pine seedlings in response to the deficiencies of the above-mentioned existing technology. This non-destructive measurement method for the morphological indicators of Masson pine seedlings can accurately measure the morphological indicators of Masson pine seedlings, no longer relying on manual and professional instrument measuring equipment, and reducing measurement costs and measurement difficulty.

[0005] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0006] A non-destructive measurement method for morphological indicators of Masson pine seedlings comprises the following steps:

[0007] Step 1: Collect 2D images of Masson pine from multiple viewing angles and perform correction on the 2D images;

[0008] Step 2: Reconstruct the collected 2D image into a 3D point cloud of the Masson pine and preprocess the point cloud;

[0009] Step 3: Create a dataset for semantic segmentation of Masson pine point cloud and calculate the class balance weights of different categories in the dataset;

[0010] Step 4: Construct the feature point adaptive weighting module and dense mapping block, and build a dense weighted semantic segmentation network model based on PointNet++; assign the class balance weights calculated in step 3 to the loss function of the dense weighted semantic segmentation network model;

[0011] Step 5: Train a dense weighted semantic segmentation network model using the Masson pine point cloud semantic segmentation dataset; use the methods of Steps 1 and 2 to collect the Masson pine point cloud of the Masson pine to be tested, and input the Masson pine point cloud of the Masson pine to be tested into the trained dense weighted semantic segmentation network model to achieve the segmentation of stems, leaves, and other parts in the Masson pine point cloud;

[0012] Step 6: Obtain the original skeleton points of the stem based on the obtained stem points in the Masson pine point cloud, and refine the skeleton points of the stem based on the original skeleton points to obtain the refined skeleton points of the stem;

[0013] Step 7: Extract the morphological index values of the tested Masson pine based on the refined skeleton points of the stem.

[0014] As a further improved technical solution of the present invention, the step 1 specifically includes:

[0015] 1.1. Build a black box, surround it with light-absorbing cloth, and install light sources on both sides. Place a rotating stage inside the black box. Place a Masson pine seedling pot and a rectangular reference block with known length, width, and height in the center of the rotating stage. Place two cameras in front of the black box. One camera photographs the Masson pine seedling pot from a level perspective, and the other camera photographs the Masson pine seedling pot from a 45° downward perspective.

[0016] 1.2. The microcontroller controls the movement of the rotating stage and the two cameras. The cameras are calibrated to obtain their internal and external parameters. The stage rotates 5° at a time. After a 15-second wait, the microcontroller controls the cameras to capture an image. The image is saved to the computer before the next rotation is performed. A total of 72 rotations are performed. Finally, the 2D image of the potted Masson pine seedlings is corrected based on the obtained internal and external camera parameters.

[0017] As a further improved technical solution of the present invention, the step 2 specifically includes:

[0018] 2.1. Use the structure-from-motion algorithm to reconstruct the rectified 2D image of the potted Masson pine seedlings into a 3D sparse point cloud of the pine. Then use the patch-based multi-view stereo geometry algorithm to process the sparse point cloud of the pine to obtain a dense point cloud of the pine.

[0019] 2.2. Use the color threshold segmentation algorithm to remove the black noise generated during the three-dimensional reconstruction of Pinus massoniana, and use the statistical outlier removal algorithm to remove outliers in the point cloud of Pinus massoniana.

[0020] As a further improved technical solution of the present invention, the step 3 specifically includes:

[0021] 3.1. Input the Masson pine point cloud that needs manual annotation into Cloudcompare and assign labels to each point in the Masson pine point cloud, including stem, leaf, and other labels.

[0022] 3.2. Obtain multiple Masson pine point clouds and labels for each point according to the methods of steps 1, 2, and 3.1. Use the multiple Masson pine point clouds with assigned labels as the Masson pine point cloud semantic segmentation dataset. Divide the Masson pine point cloud semantic segmentation dataset into a training set and a test set in proportion.

[0023] 3.3. Calculate the total number of points labeled stem, the total number of points labeled leaf, and the total number of points labeled other in the training set, and calculate the proportion of each label:

[0024]

[0025] Among them, Point i is the total number of points with label i in the training set; Sum is the total number of all points in the training set; i = 0, 1, 2, 0 represents stem, 1 represents leaf, and 2 represents others; F i Indicates the proportion of label i in the training set;

[0026] 3.4. Average the proportion of stems, leaves, and other labels in the training set to get the mean proportion according to and F i Calculate the class balance weights for the stem label, the leaf label, and the other label in the training set respectively:

[0027]

[0028] Weight i represents the class balance weight of label i.

[0029] As a further improved technical solution of the present invention, the step 4 specifically includes:

[0030] 4.1. The feature point adaptive weighting module includes an adaptive average pooling layer, an adaptive maximum pooling layer, four 1D convolutional layers, two LeakyReLU activation function layers, and a Sigmoid activation function layer;

[0031] The processing process of the feature point adaptive weighting module is:

[0032] 4.1.1. Use an adaptive average pooling layer to aggregate the input feature information, pass it through a 1D convolutional layer with an output channel number of 3, then through a LeakyReLU activation function layer, and then through a 1D convolutional layer with an output channel number equal to the initial input channel number to obtain the average pooling feature; use an adaptive max pooling layer to aggregate the input feature information, pass it through a 1D convolutional layer with an output channel number of 3, then through a LeakyReLU activation function layer, and then through a 1D convolutional layer with an output channel number equal to the initial input channel number to obtain the max pooling feature;

[0033] 4.1.2. Add the average pooling feature and the maximum pooling feature element by element, and then pass it through the Sigmoid activation function layer to obtain the 1D channel attention;

[0034] 4.2. The dense mapping block constructed includes a 1D convolutional layer, a 1D BatchNorm layer, a ReLU activation function layer, a feature point adaptive weighting module, and a Dropout layer. The probability of inactivation of each neuron in the Dropout layer is set to 0.1.

[0035] 4.3. Assign the class balance weights calculated in step 3.4 to the loss function of the densely weighted semantic segmentation network model. The loss function uses the existing common formula. In the code, the loss function must have a weight parameter to call class balance.

[0036] 4.4. Build a dense weighted semantic segmentation network model based on PointNet++. The dense weighted semantic segmentation network model includes the SA1-Msg module, SA2-Msg module, SA3 module, FP1++ module, FP2++ module, FP3++ module, 4 dense mapping blocks, a 1D convolutional layer, and a Log_softmax layer.

[0037] Among them, the FP1++ module includes the FP module and the feature point adaptive weighting module with the number of input channels and the number of output channels both being 256; the FP2++ module includes the FP module and the feature point adaptive weighting module with the number of input channels and the number of output channels both being 128; the FP3++ module includes the FP module and the feature point adaptive weighting module with the number of input channels and the number of output channels both being 128.

[0038] As a further improved technical solution of the present invention, the step 6 specifically includes:

[0039] 6.1. Based on the point cloud segmentation, the original skeleton points of the stem are extracted using the Laplace shrinkage algorithm. The original skeleton points consist of a point set O = {O1, O2, ..., O n}, where n is the number of original skeleton points. Before using the multi-iteration plane method based on Euclidean distance to refine the skeleton points of the stem, O must be sorted first. The sorting method is:

[0040] 6.1.1. Create an empty set P, traverse each element in the point set O, and find the element with the smallest y coordinate value;

[0041] 6.1.2. Label the element with the smallest y-coordinate value as P1, add P1 to the empty set P, and delete it from the point set O;

[0042] 6.1.3. Set the initial value of j to 1, i.e. j = 1;

[0043] 6.1.4. Traverse each element in the updated point set O and calculate the difference between each element and P j The Euclidean distance of point set O is found, and the element that minimizes the Euclidean distance is recorded as P j+1 , P j+1 Append it to P and delete it from the point set O;

[0044] 6.1.5. Add 1 to the value of j, i.e. j = j + 1;

[0045] 6.1.6. Repeat steps 6.1.4 and 6.1.5 until the point set O becomes an empty set. Then the sorted point set P = {P1, P2, ..., P n};

[0046] .2. Use the multi-iteration planar method based on Euclidean distance to refine the skeleton points of the stem:

[0047] 6.2.1. Perform refinement operations on the original skeleton points P1P2 as follows:

[0048] 6.2.1.1. Starting from points P1 and P2, execute step 6.2.1.2.

[0049] 6.2.1.2. Using the line segment connecting the two points as the normal vector, construct two planes perpendicular to the normal vector through the two points. Using these two planes as constraints, we can obtain the set of points S in the stem's point cloud.

[0050] 6.2.1.3. Using the minimum bounding box of point set S as the constraint, traverse all elements in it in steps of 0.0001 on the X, Y, and Z axes, and find the element (X, Y, Z) that minimizes the distance. The distance is calculated as follows:

[0051]

[0052] Among them, (X, Y, Z) is the coordinate of the current element during traversal, (x i ,y i , z i) is the coordinate of the i-th element in the space constrained by the minimum bounding box of the point set S, and m is the number of elements contained in the space;

[0053] 6.2.1.4. Let this element be A.

[0054] 6.2.1.5. Operate P1 and A according to steps 6.2.1.2 and 6.2.1.3 to obtain new refined skeleton points. A and P2 are operated according to steps 6.2.1.2 and 6.2.1.3 to obtain new refined skeleton points The original skeleton P1P2 is refined into

[0055] 6.2.2, follow the method in step 6.2.1 to traverse P2, P3, P3, P4, ... and P n-1 , P n , and then the original skeleton points P2, P3, P3, P4, ... and P n-1 , P n Perform refined operations.

[0056] As a further improved technical solution of the present invention, the step 7 specifically includes:

[0057] 7.1. Place the bottom surface of the stem's refined skeleton points on the XOZ plane. Project the stem's refined skeleton points and leaf points onto the YOZ plane. The maximum value minus the minimum value of the points along the Y axis is the seedling height.

[0058] 7.2. Slice the stem based on the two refined skeleton points at the bottom. The maximum value minus the minimum value of the points in the slice along the X-axis is the ground diameter.

[0059] 7.3. Divide the seedling height obtained in step 7.1 by the ground diameter obtained in step 7.2 to get the height-to-diameter ratio.

[0060] 7.4. For the refined skeleton points obtained in step 6, calculate the Euclidean distance of each segment in the refined skeleton points in turn, and add up all the obtained Euclidean distances to obtain the stem length;

[0061] 7.5. Use a spatial curve to fit the refined skeleton points at the bend of the stem. Calculate the curvature of the spatial curve at that point to obtain the degree of curvature of the stem at that point.

[0062] 7.6. Project the stem and leaves of a Masson pine onto the XOZ plane. Use the Graham algorithm to obtain the convex hull points. Calculate the area of the polygon formed by connecting these convex hull points to obtain the projected crown area.

[0063] 7.7. Calculate the calibration coefficient based on the cuboid reference block with known length, width and height and the point cloud reconstructed from it Where L is the length of the rectangular reference block, and L1 is the length of the rectangular reference block displayed after reconstruction in the computer. This coefficient is used to calibrate the morphological index values obtained in steps 7.1-7.6.

[0064] In order to achieve the above technical objectives, another technical solution adopted by the present invention is:

[0065] A non-destructive measurement system for morphological indicators of Masson pine seedlings, comprising:

[0066] An image acquisition module, used to acquire 2D Masson pine images from multiple viewing angles and to correct the 2D images;

[0067] The 3D reconstruction module is used to reconstruct the rectified 2D image into a sparse point cloud of Masson pine, and then process the sparse point cloud of Masson pine to obtain a dense point cloud of Masson pine;

[0068] Point cloud preprocessing module, used to remove black noise generated during the 3D reconstruction of Masson pine and outliers in the point cloud of Masson pine;

[0069] The dataset creation module is used to assign labels to each point in the Masson pine point cloud, including stem, leaf, and others. The labeled Masson pine point cloud can be used as the Masson pine point cloud semantic segmentation dataset. The Masson pine point cloud semantic segmentation dataset is divided into training and test sets according to the proportion.

[0070] The point cloud segmentation module is used to segment the Masson pine point cloud using the trained dense weighted semantic segmentation network model to obtain stems, leaves, and others;

[0071] The stem skeleton extraction module is used to refine the stem skeleton points obtained from point cloud segmentation;

[0072] The phenotypic information extraction module is used to extract the morphological index values of Masson pine, which include seedling height, ground diameter, height-to-diameter ratio, stem length, stem curvature and crown projection area.

[0073] The beneficial effects of the present invention are:

[0074] (1) The feature point adaptive weighting module of the present invention optimizes the upsampling capability of the Feature Propagation module in PointNet++. By calculating the weights of the point features output by the FP module in different channels and multiplying them with the original features, the expression of the original features is improved, thereby improving the classification accuracy of each subsequent point.

[0075] (2) The dense mapping block of the present invention enhances the classification capability of the FC layers in PointNet++ by gradually reducing the output dimension and optimizing the feature weights. This further improves the classification accuracy of each point in the network. In addition, to address the problem of a large disparity in the number of points in the reconstructed stem, leaves, and other three parts, the weights of these three parts are calculated and assigned to the loss function, thereby optimizing the class imbalance in the dataset.

[0076] (3) The dense weighted semantic segmentation network based on PointNet++ of the present invention can well segment the point cloud of Pinus massoniana into stems, leaves and others. Since the needles of Pinus massoniana are dense and closely connected to the stems, and the surface of the stems is green, the traditional clustering segmentation and threshold segmentation are not effective. The dense weighted semantic segmentation network based on PointNet++ can well complete the segmentation task.

[0077] (4) The multi-iteration planar method based on Euclidean distance of the present invention extracts the skeleton points of the stem in a refined manner. The refined skeleton points improve the measurement accuracy of the subsequent ground diameter, height-to-diameter ratio, stem length and degree of stem bending. In addition, for some Masson pine stems with more complex morphological structures, the number of iterations can be increased, and the method can be used multiple times between two skeleton points to obtain skeleton points that can accurately reflect the growth morphology of the Masson pine stem.

[0078] (5) The present invention makes the measurement of the morphological indicators of Masson pine seedlings no longer rely on manual labor and professional instruments and equipment, saving measurement costs and improving the measurement accuracy of the morphological indicators of Masson pine seedlings, especially the morphological indicators that are difficult to measure accurately, such as stem length, stem bending degree, and crown projection area. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 This is a flow chart of the non-destructive measurement method for the morphological indicators of Masson pine seedlings.

[0080] Figure 2 This is the structural diagram of the feature point adaptive weighting module.

[0081] Figure 3 This is a structural diagram of the dense weighted semantic segmentation network built based on PointNet++.

[0082] Figure 4 This is the algorithm flow chart of the multi-iteration planar method based on Euclidean distance.

[0083] Figure 5 This is the effect of semantic segmentation of the Masson pine point cloud using the dense weighted semantic segmentation network model.

[0084] Figure 6 This is the effect of using the multi-iteration plane method based on Euclidean distance and Laplace shrinkage to extract the stem skeleton. DETAILED DESCRIPTION

[0085] The specific embodiments of the present invention are further described below with reference to the accompanying drawings:

[0086] This embodiment provides a non-destructive measurement method for the morphological indexes of Masson pine seedlings. Figure 1 As shown, the following steps are included:

[0087] Step 1: Build an image acquisition platform, collect 2D images of Masson pine from multiple perspectives and angles, and correct the 2D images.

[0088] The step 1 specifically includes:

[0089] Step 1.1: Build a black box (0.8m × 0.48m × 0.48m), surround it with light-absorbing cloth, and install light sources on both sides. Place a rotating stage inside the black box. Place a Masson pine seedling pot and a rectangular reference block with known length, width, and height in the center of the stage. Place two cameras in front of the black box: one camera photographs the Masson pine seedling pot from a level perspective, and the other camera photographs the Masson pine seedling pot from a 45° downward perspective.

[0090] Step 1.2: The microcontroller controls the movement of the rotating stage and the two cameras. The cameras are calibrated to obtain their internal and external parameters. The stage rotates 5° at a time. After each rotation, the microcontroller controls the cameras to capture an image. The image is saved to the computer before the next rotation is performed. A total of 72 rotations are performed. Finally, the 2D image of the potted Masson pine seedlings is corrected based on the obtained internal and external camera parameters.

[0091] Step 2: Reconstruct the collected 2D image into a 3D point cloud of the Masson pine and preprocess the point cloud.

[0092] The step 2 specifically includes:

[0093] Step 2.1: Use the Structure from Motion (SFM) algorithm to reconstruct the rectified 2D image of the potted Masson pine seedlings into a 3D sparse point cloud of the Masson pine. Then, use the Patch-based Multiple View Stereo (PMVS) algorithm to process the sparse point cloud of the Masson pine to obtain a dense point cloud of the Masson pine.

[0094] Step 2.2: Use the color threshold segmentation algorithm to remove the black noise generated during the 3D reconstruction of the Masson pine, and use the statistical outlier removal algorithm to remove outliers in the Masson pine point cloud.

[0095] Step 3: Create a dataset for the semantic segmentation of Masson pine point cloud and calculate the class balance weights of different categories in the dataset.

[0096] The step 3 specifically includes:

[0097] Step 3.1. Input the Masson pine point cloud that needs to be manually labeled into Cloudcompare and assign a label to each point in the Masson pine point cloud. The labels include stem: 0, leaf: 1, and others: 2.

[0098] Step 3.2: Obtain the corresponding Masson pine point clouds and labels of each point according to the methods of steps 1, 2, and 3.1. Use the labeled Masson pine point clouds as the Masson pine point cloud semantic segmentation dataset. Split the Masson pine point cloud semantic segmentation dataset into a training set and a test set in a ratio of 8:2 for subsequent training of the densely weighted semantic segmentation network based on PointNet++.

[0099] Step 3.3: Calculate the total number of points labeled stem, the total number of points labeled leaf, and the total number of points labeled other in the training set, and calculate the proportion of each label:

[0100]

[0101] Among them, Point i is the total number of points with label i in the training set; Sum is the total number of all points in the training set; i = 0, 1, 2, 0 represents stem, 1 represents leaf, and 2 represents others; F i Indicates the proportion of labels i in the training set;

[0102] Step 3.4: Average the proportion of stems, leaves, and other labels in the training set to get the mean proportion. according to and F i Calculate the class balance weights for the stem label, the leaf label, and the other label in the training set respectively:

[0103]

[0104] Weight i represents the class balance weight of label i.

[0105] Step 4: Construct the feature point adaptive weighting module and dense mapping block, and build a dense weighted semantic segmentation network model based on PointNet++; assign the class balance weight calculated in step 3 to the loss function of the dense weighted semantic segmentation network model.

[0106] The step 4 specifically includes:

[0107] Step 4.1, such as Figure 2As shown in the figure, the feature point adaptive weighting module includes 1 adaptive average pooling layer, 1 adaptive maximum pooling layer, 4 1D convolution layers, 2 LeakyReLU activation function layers and 1 Sigmoid activation function layer;

[0108] The processing process of the feature point adaptive weighting module is:

[0109] Step 4.1.1. Use an adaptive average pooling layer to aggregate the input feature information, reduce its dimensionality to 3D by a 1D convolutional layer, then pass it through a LeakyReLU activation function layer, and then increase its dimensionality to the initial input dimension by a 1D convolutional layer to obtain the average pooling feature; use an adaptive max pooling layer to aggregate the input feature information, reduce its dimensionality to 3D by a 1D convolutional layer, then pass it through a LeakyReLU activation function layer, and then increase its dimensionality to the initial input dimension by a 1D convolutional layer to obtain the max pooling feature;

[0110] Step 4.1.2, add the average pooling feature and the maximum pooling feature element by element, and then pass the Sigmoid activation function layer to obtain the 1D channel attention;

[0111] Step 4.2: The dense mapping block constructed includes a 1D convolutional layer, a 1D BatchNorm layer, a ReLU activation function layer, a feature point adaptive weighting module, and a Dropout layer. The probability of inactivation of each neuron in the Dropout layer is set to 0.1.

[0112] Step 4.3. Assign the class balance weight calculated in step 3.4 to the loss function of the dense weighted semantic segmentation network model. The loss function uses the existing common formula. In the code, the loss function needs to have a weight parameter to call class balance. The purpose of assigning class balance weight to the loss function of the dense weighted semantic segmentation network model is because after the Masson pine point cloud is segmented into stems, leaves, and other three parts, the number of points contained in each part varies greatly. For example, the leaves and other parts have many points, while the stems have very few points. Therefore, assigning more weight to the stems through class balancing technology can help improve the accuracy of the model.

[0113] Step 4.4: Build a dense weighted semantic segmentation network model based on PointNet++:

[0114] Step 4.4.1. Modify the second Feature Propagation module (FP2) in PointNet++, mlp = [256, 128] is changed to mlp = [128, 128], and insert the feature point adaptive weighting module with an input and output dimension of 256 into the first Feature Propagation module (FP1) in PointNet++ to obtain FP1++; insert the feature point adaptive weighting module with an input and output dimension of 128 into the third Feature Propagation module (FP3) in FP2 and PointNet++ to obtain FP2++ and FP3++;

[0115] Step 4.4.2: Delete the 1D convolutional layer, 1D BatchNorm layer, and Dropout layer in the FC layers after FP3, and add four dense mapping blocks, where the input and output dimensions of the four dense mapping blocks decrease in sequence, i.e., 128 → 64 → 32 → 16; after the four dense mapping blocks, add another 1D convolutional layer with an input dimension of 16 and an output dimension equal to the number of categories;

[0116] like Figure 3 As shown in the figure, the final dense weighted semantic segmentation network model includes 1 SA1-Msg module, 1 SA2-Msg module, 1 SA3 module, 1 FP1++ module, 1 FP2++ module, 1 FP3++ module, 4 dense mapping blocks, 1 1D convolution layer and 1 Log_softmax layer.

[0117] The SA1-Msg module, SA2-Msg module, SA3 module, and FP module in the dense weighted semantic segmentation network built based on PointNet++ are existing modules in the existing technology PointNet++.

[0118] Among them, the SA1-Msg module consists of the SA1 module and the Msg module, the SA2-Msg module consists of the SA2 module and the Msg module, and the SA1 module, SA2 module and SA3 module are all composed of a sampling layer, a grouping layer and a PointNet layer. The sampling layer uniformly samples a certain number of points in the input point set through the farthest point sampling algorithm. The grouping layer takes each sampling point as the center point to form a local neighborhood with a radius of R. The PointNet layer consists of a ReLU activation function layer, a 2D convolution layer and a 2D BatchNorm layer. The Msg module extracts multiple local area features by constructing multiple radii, and then splices these local features of different radii together.

[0119] The PointNet layer in the SA1 module consists of 9 ReLU activation function layers, 9 2D convolution layers and 9 2D BatchNorm layers. The PointNet layer in the SA2 module consists of 6 ReLU activation function layers, 6 2D convolution layers and 6 2D BatchNorm layers. The PointNet layer in the SA3 module consists of 3 ReLU activation function layers, 3 2D convolution layers and 3 2D BatchNorm layers.

[0120] The FP1++ module includes the FP module and a feature point adaptive weighting module with 256 input and output channels; the FP2++ module includes the FP module and a feature point adaptive weighting module with 128 input and output channels; and the FP3++ module includes the FP module and a feature point adaptive weighting module with 128 input and output channels. The FP module consists of a ReLU activation function layer, a 1D convolution layer, and a 1D BatchNorm layer. The FP1++ module, FP2++ module, and FP3++ module contain, in order, a 1D convolution layer, a 1D BatchNorm layer, a ReLU activation function layer, a feature point adaptive weighting module, a 1D convolution layer, a 1D BatchNorm layer, a ReLU activation function layer, and a feature point adaptive weighting module.

[0121] Add four dense mapping blocks after the FP3++ module, where the input and output dimensions of the four feature point adaptive weighting modules decrease in sequence, i.e., 128 → 64 → 32 → 16; add a 1D convolutional layer after the four dense mapping blocks, with an input dimension of 16 and an output dimension equal to the number of categories.

[0122] Step 5: Train a dense weighted semantic segmentation network model using the Masson pine point cloud semantic segmentation dataset; use the methods of steps 1 and 2 to collect the Masson pine point cloud of the Masson pine to be tested, and input the Masson pine point cloud of the Masson pine to be tested into the trained dense weighted semantic segmentation network model to achieve the segmentation of stems, leaves and other parts in the Masson pine point cloud, such as Figure 5 As shown, the green ones are stems, the red ones are leaves, and the blue ones are others, such as "pots".

[0123] Step 6: Based on the points of the stem in the obtained Masson pine point cloud, the original skeleton points of the stem are obtained, and the skeleton points of the stem are refined based on the original skeleton points to obtain the refined skeleton points of the stem.

[0124] The step 6 specifically includes:

[0125] Step 6.1: Based on the point cloud segmentation, the original skeleton points of the stem are extracted using the Laplace shrinkage algorithm. The point set O composed of the original skeleton points is O = {O1, O2, ..., On}, where n is the number of original skeleton points. Before using the multi-iteration plane method based on Euclidean distance to refine the skeleton points of the stem, O must be sorted first, such as Figure 4 As shown, the sorting method is:

[0126] Step 6.1.1. Create an empty set P, traverse each element in the point set O, and find the element with the smallest y coordinate value;

[0127] Step 6.1.2. Record the element with the smallest y-coordinate value as P1, add P1 to the empty set P, and delete it from the point set O;

[0128] Step 6.1.3, set the initial value of j to 1, i.e. j = 1;

[0129] Step 6.1.4, traverse each element in the updated point set O, and calculate the difference between each element and P j The Euclidean distance of point set O is found, and the element that minimizes the Euclidean distance is recorded as P j+1 , P j+1 Append it to P and delete it from the point set O;

[0130] Step 6.1.5, add 1 to the value of j, i.e. j = j + 1;

[0131] Step 6.1.6, loop steps 6.1.4 and 6.1.5 until the point set O becomes an empty set, then the sorted point set P = {P1, P2, ..., P n};

[0132] Step 6.2: Use the multi-iteration planar method based on Euclidean distance to refine the skeleton points of the stem. Figure 6 As shown in the figure, blue and green represent slicing layers, and red represents refined skeleton points. It mainly represents the content of the multi-iteration plane method based on Euclidean distance, which slices the original skeleton points and then finds new refined skeleton points.

[0133] Step 6.2.1: Perform refinement operations on the original skeleton points P1P2 as follows:

[0134] Step 6.2.1.1. Starting from points P1 and P2, execute step 6.2.1.2.

[0135] Step 6.2.1.2: Using the line segment connecting the two points as the normal vector, construct two planes perpendicular to the normal vector through the two points. Using these two planes as constraints, we can obtain the point set S of the stem point cloud between the two planes.

[0136] Step 6.2.1.3: Using the minimum bounding box of point set S as a constraint, traverse all elements in it with a step size of 0.0001 on the X, Y, and Z axes, and find the element (X, Y, Z) that minimizes the distance. For each point (X, Y, Z), the sum of its Euclidean distances with all points in point set S can be calculated. A total of m distances can be calculated. The minimum distance is found, and the corresponding (X, Y, Z) is the coordinate to be searched, which is the center of point set S. The distance is calculated as follows:

[0137]

[0138] Among them, (X, Y, Z) is the coordinate of the current element during traversal, (x i ,y i , z i ) is the coordinate of the i-th element in the space constrained by the minimum bounding box of the point set S, and m is the number of elements contained in the space;

[0139] Step 6.2.1.4, record the element as A;

[0140] Step 6.2.1.5: Operate P1 and A according to steps 6.2.1.2 and 6.2.1.3 to obtain new refined skeleton points. A and P2 are operated according to steps 6.2.1.2 and 6.2.1.3 to obtain new refined skeleton points The original skeleton points P1P2 are refined into

[0141] Step 6.2.2: Follow the method in step 6.2.1 to traverse P2, P3, P3, P4, ... and P n-1 , P n , and then the original skeleton points P2, P3, P3, P4, ... and P n-1 , P n Perform refined operations separately.

[0142] Step 7: Extract the morphological index values of the tested Masson pine based on the refined skeleton points of the stem.

[0143] The step 7 specifically includes:

[0144] Step 7.1. Place the bottom surface of the refined skeleton points of the stem on the XOZ plane. Project the refined skeleton points of the stem and the leaves of the Masson pine on the YOZ plane. The maximum value minus the minimum value of the points in the Y-axis direction is the seedling height.

[0145] Step 7.2: Slice the stem based on the two refined skeleton points at the bottom. The maximum value minus the minimum value of the points in the slice along the X-axis is the ground diameter.

[0146] Step 7.3: Divide the seedling height obtained in step 7.1 by the ground diameter obtained in step 7.2 to obtain the height-to-diameter ratio.

[0147] Step 7.4: For the refined skeleton points obtained in step 6, calculate the Euclidean distance of each segment in the refined skeleton points in turn, and add up all the calculated Euclidean distances to obtain the stem length;

[0148] Step 7.5: Use the spatial curve to fit the refined skeleton points at the bend of the stem. Calculate the curvature of the spatial curve at that point to obtain the degree of curvature of the stem at that point.

[0149] Step 7.6: Project the stem and leaves of the Masson pine onto the XOZ plane. Use the Graham algorithm to obtain the convex hull points. Calculate the area of the polygon formed by connecting these convex hull points to obtain the crown projection area.

[0150] Step 7.7: Calculate the calibration coefficient based on the rectangular reference block with known length, width and height and the point cloud reconstructed from it, and use the coefficient to calibrate the morphological index values obtained in steps 7.1-7.6.

[0151] The specific values obtained in steps 7.1-7.6 are not the true values of the morphological indicators of Masson pine, because after the Masson pine is reconstructed into a three-dimensional model, the values reflected in the computer do not have actual units and the numerical values are also scaled with the true values of the Masson pine indicators. For example, if the length, width and height of a rectangular block are known to be L centimeters, after it is reconstructed in the computer, the length is displayed as L1. Then when the height of the reconstructed Masson pine seedling is displayed as D in the computer, its true morphological indicator value is actually cm, of which The purpose is to find the actual morphological index value of Masson pine, and the calibrated value should be selected in the end.

[0152] This embodiment also provides a non-destructive measurement system for morphological indicators of Masson pine seedlings, comprising:

[0153] An image acquisition module, used to acquire 2D Masson pine images from multiple viewing angles and to correct the 2D images;

[0154] The 3D reconstruction module is used to reconstruct the rectified 2D image into a sparse point cloud of Masson pine, and then process the sparse point cloud of Masson pine to obtain a dense point cloud of Masson pine;

[0155] Point cloud preprocessing module, used to remove black noise generated during the 3D reconstruction of Masson pine and outliers in the point cloud of Masson pine;

[0156] The dataset creation module is used to assign a label to each point in the Masson pine point cloud. The labels include stem: 0, leaf: 1, and others: 2. The Masson pine point cloud with assigned labels can be used as the Masson pine point cloud semantic segmentation dataset. The Masson pine point cloud semantic segmentation dataset is divided into a training set and a test set in a ratio of 8:2;

[0157] The point cloud segmentation module is used to segment the Masson pine point cloud using the trained dense weighted semantic segmentation network model to obtain stems, leaves, and others;

[0158] The stem skeleton extraction module is used to refine the stem skeleton points obtained from point cloud segmentation;

[0159] The phenotypic information extraction module is used to extract the morphological index values of Masson pine, which include seedling height, ground diameter, height-to-diameter ratio, stem length, stem curvature and crown projection area.

[0160] The protection scope of the present invention includes but is not limited to the above embodiments. The protection scope of the present invention is based on the claims. Any replacement, deformation, and improvement of the technology that can be easily thought of by those skilled in the art fall within the protection scope of the present invention.

Claims

1. A non-destructive measurement method for morphological indicators of Masson pine seedlings, characterized in that: The following steps are involved: Step 1: Collect 2D images of Masson pine from multiple viewing angles and perform correction on the 2D images; Step 2: Reconstruct the collected 2D image into a 3D point cloud of the Masson pine and preprocess the point cloud; Step 3: Create a dataset for semantic segmentation of Masson pine point cloud and calculate the class balance weights of different categories in the dataset; Step 4: Construct the feature point adaptive weighting module and dense mapping block, and build a dense weighted semantic segmentation network model based on PointNet++; assign the class balance weights calculated in step 3 to the loss function of the dense weighted semantic segmentation network model; Step 5: Train a dense weighted semantic segmentation network model using the Masson pine point cloud semantic segmentation dataset; use the methods of Steps 1 and 2 to collect the Masson pine point cloud of the Masson pine to be tested, and input the Masson pine point cloud of the Masson pine to be tested into the trained dense weighted semantic segmentation network model to achieve the segmentation of stems, leaves, and other parts in the Masson pine point cloud; Step 6: Obtain the original skeleton points of the stem based on the obtained stem points in the Masson pine point cloud, and refine the skeleton points of the stem based on the original skeleton points to obtain the refined skeleton points of the stem; Step 7: Extract the morphological index values of the tested Masson pine based on the refined skeleton points of the stem.

2. the morphological index nondestructive measurement method of Masson pine seedlings according to claim 1, is characterized in that, The step 1 specifically includes: 1.

1. Build a black box, surround it with light-absorbing cloth, and install light sources on both sides. Place a rotating stage inside the black box. Place a Masson pine seedling pot and a rectangular reference block with known length, width, and height in the center of the rotating stage. Place two cameras in front of the black box. One camera photographs the Masson pine seedling pot from a level perspective, and the other camera photographs the Masson pine seedling pot from a 45° downward perspective. 1.

2. The microcontroller controls the movement of the rotating stage and the two cameras. The cameras are calibrated to obtain their internal and external parameters. The stage rotates 5° at a time. After a 15-second wait, the microcontroller controls the cameras to capture an image. The image is saved to the computer before the next rotation is performed. A total of 72 rotations are performed. Finally, the 2D image of the potted Masson pine seedlings is corrected based on the obtained internal and external camera parameters.

3. the morphological index nondestructive measurement method of Masson pine seedlings according to claim 2, is characterized in that, The step 2 specifically includes: 2.

1. Use the structure-from-motion algorithm to reconstruct the rectified 2D image of the potted Masson pine seedlings into a 3D sparse point cloud of the pine. Then use the patch-based multi-view stereo geometry algorithm to process the sparse point cloud of the pine to obtain a dense point cloud of the pine. 2.

2. Use the color threshold segmentation algorithm to remove the black noise generated during the three-dimensional reconstruction of Pinus massoniana, and use the statistical outlier removal algorithm to remove outliers in the point cloud of Pinus massoniana.

4. the morphological index nondestructive measurement method of Masson pine seedlings according to claim 3, is characterized in that, The step 3 specifically includes: 3.

1. Input the Masson pine point cloud that needs manual annotation into Cloudcompare and assign labels to each point in the Masson pine point cloud, including stem, leaf, and other labels. 3.

2. Obtain multiple Masson pine point clouds and labels for each point according to the methods of steps 1, 2, and 3.

1. Use the multiple Masson pine point clouds with assigned labels as the Masson pine point cloud semantic segmentation dataset. Divide the Masson pine point cloud semantic segmentation dataset into a training set and a test set in proportion. 3.

3. Calculate the total number of points labeled stem, the total number of points labeled leaf, and the total number of points labeled other in the training set, and calculate the proportion of each label: Among them, Point i is the total number of points with label i in the training set; Sum is the total number of all points in the training set; i = 0, 1, 2, 0 represents stem, 1 represents leaf, and 2 represents others; F i Indicates the proportion of label i in the training set; 3.

4. Average the proportion of stems, leaves, and other labels in the training set to get the mean proportion according to and F i Calculate the class balance weights for the stem label, the leaf label, and the other label in the training set respectively: Weight i represents the class balance weight of label i.

5. the morphological index nondestructive measurement method of Masson pine seedlings according to claim 4, is characterized in that, The step 4 specifically includes: 4.

1. The feature point adaptive weighting module includes an adaptive average pooling layer, an adaptive maximum pooling layer, four 1D convolutional layers, two LeakyReLU activation function layers, and a Sigmoid activation function layer; The processing process of the feature point adaptive weighting module is: 4.1.

1. Use an adaptive average pooling layer to aggregate the input feature information, pass it through a 1D convolutional layer with an output channel number of 3, then through a LeakyReLU activation function layer, and then through a 1D convolutional layer with an output channel number equal to the initial input channel number to obtain the average pooling feature; use an adaptive max pooling layer to aggregate the input feature information, pass it through a 1D convolutional layer with an output channel number of 3, then through a LeakyReLU activation function layer, and then through a 1D convolutional layer with an output channel number equal to the initial input channel number to obtain the max pooling feature; 4.1.

2. Add the average pooling feature and the maximum pooling feature element by element, and then pass it through the Sigmoid activation function layer to obtain the 1D channel attention; 4.

2. The dense mapping block constructed includes a 1D convolutional layer, a 1D BatchNorm layer, a ReLU activation function layer, a feature point adaptive weighting module, and a Dropout layer. The probability of inactivation of each neuron in the Dropout layer is set to 0.

1. 4.

3. Assign the class balance weights calculated in step 3.4 to the loss function of the densely weighted semantic segmentation network model; 4.

4. Build a dense weighted semantic segmentation network model based on PointNet++. The dense weighted semantic segmentation network model includes the SA1-Msg module, SA2-Msg module, SA3 module, FP1++ module, FP2++ module, FP3++ module, 4 dense mapping blocks, a 1D convolutional layer, and a Log_softmax layer. in, The FP1++ module includes the FP module and a feature point adaptive weighting module with 256 input channels and 256 output channels; the FP2++ module includes the FP module and a feature point adaptive weighting module with 128 input channels and 128 output channels; the FP3++ module includes the FP module and a feature point adaptive weighting module with 128 input channels and 128 output channels.

6. the morphological index nondestructive measurement method of Masson pine seedlings according to claim 1, is characterized in that, The step 6 specifically includes: 6.

1. Based on the point cloud segmentation, the original skeleton points of the stem are extracted using the Laplace shrinkage algorithm. The original skeleton points consist of a point set O = {O1, O2, ..., O n }, where n is the number of original skeleton points. Before using the multi-iteration plane method based on Euclidean distance to refine the skeleton points of the stem, O must be sorted first. The sorting method is: 6.1.

1. Create an empty set P, traverse each element in the point set O, and find the element with the smallest y coordinate value; 6.1.

2. Label the element with the smallest y-coordinate value as P1, add P1 to the empty set P, and delete it from the point set O; 6.1.

3. Set the initial value of j to 1, i.e. j = 1; 6.1.

4. Traverse each element in the updated point set O and calculate the difference between each element and P j The Euclidean distance of point set O is found, and the element that minimizes the Euclidean distance is recorded as P j+1 , P j+1 Append it to P and delete it from the point set O; 6.1.

5. Add 1 to the value of j, i.e. j = j + 1; 6.1.

6. Repeat steps 6.1.4 and 6.1.5 until the point set O becomes an empty set. Then the sorted point set P = {P1, P2, ..., P n }; 6.

2. Use the multi-iteration planar method based on Euclidean distance to refine the skeleton points of the stem: 6.2.

1. Perform refinement operations on the original skeleton points P1P2 as follows: 6.2.1.

1. Starting from points P1 and P2, execute step 6.2.1.

2. 6.2.1.

2. Using the line segment connecting the two points as the normal vector, construct two planes perpendicular to the normal vector through the two points. Using these two planes as constraints, we can obtain the set of points S in the stem's point cloud. 6.2.1.

3. Using the minimum bounding box of point set S as the constraint, traverse all elements in it in steps of 0.0001 on the X, Y, and Z axes, and find the element (X, Y, Z) that minimizes the distance. The distance is calculated as follows: in, (X, Y, Z) is the coordinate of the current element during traversal, (x i ,y i , z i ) is the coordinate of the i-th element in the space constrained by the minimum bounding box of the point set S, and m is the number of elements contained in the space; 6.2.1.

4. Record the element found in step 6.2.1.3 as A; 6.2.1.

5. Perform operations on points P1 and R according to steps 6.2.1.2 and 6.2.1.3 to find the element that minimizes the distance. This element is recorded as the new refined skeleton point. Operate points A and P2 according to steps 6.2.1.2 and 6.2.1.3 to find the element that minimizes the distance. This element is recorded as the new refined skeleton point. The original skeleton P1P2 is refined into 6.2.2, follow the method in step 6.2.1 to traverse P2, P3, P3, P4, ... and P n-1 , P n , and then the original skeleton points P2, P3, P3, P4, ... and P n-1 , P n Perform refined operations.

7. the morphological index nondestructive measurement method of Masson pine seedlings according to claim 2, is characterized in that, The step 7 specifically includes: 7.

1. Place the bottom surface of the stem's refined skeleton points on the XOZ plane. Project the stem's refined skeleton points and leaf points onto the YOZ plane. The maximum value minus the minimum value of the points along the Y axis is the seedling height. 7.

2. Slice the stem based on the two refined skeleton points at the bottom. The maximum value minus the minimum value of the points in the slice along the X-axis is the ground diameter. 7.

3. Divide the seedling height obtained in step 7.1 by the ground diameter obtained in step 7.2 to get the height-to-diameter ratio. 7.

4. For the refined skeleton points obtained in step 6, calculate the Euclidean distance of each segment in the refined skeleton points in turn, and add up all the obtained Euclidean distances to obtain the stem length; 7.

5. Use a spatial curve to fit the refined skeleton points at the bend of the stem. Calculate the curvature of the spatial curve at that point to obtain the degree of curvature of the stem at that point. 7.

6. Project the stem and leaves of a Masson pine onto the XOZ plane. Use the Graham algorithm to obtain the convex hull points. Calculate the area of the polygon formed by connecting the convex hull points to obtain the projected crown area. 7.

7. Calculate the calibration coefficient based on the cuboid reference block with known length, width and height and the point cloud reconstructed from it Where L is the length of the rectangular reference block, and L1 is the length of the rectangular reference block displayed after reconstruction in the computer. This coefficient is used to calibrate the morphological index values obtained in steps 7.1-7.

6.

8. A non-destructive measurement system for morphological indicators of Masson pine seedlings, characterized in that: include: An image acquisition module, used to acquire 2D Masson pine images from multiple viewing angles and to correct the 2D images; The 3D reconstruction module is used to reconstruct the rectified 2D image into a sparse point cloud of Masson pine, and then process the sparse point cloud of Masson pine to obtain a dense point cloud of Masson pine; Point cloud preprocessing module, used to remove black noise generated during the 3D reconstruction of Masson pine and outliers in the point cloud of Masson pine; The dataset creation module is used to assign labels to each point in the Masson pine point cloud, including stem, leaf, and others. The labeled Masson pine point cloud can be used as the Masson pine point cloud semantic segmentation dataset. The Masson pine point cloud semantic segmentation dataset is divided into training and test sets according to the proportion. The point cloud segmentation module is used to segment the Masson pine point cloud using the trained dense weighted semantic segmentation network model to obtain stems, leaves, and others; The stem skeleton extraction module is used to refine the stem skeleton points obtained from point cloud segmentation; The phenotypic information extraction module is used to extract the morphological index values of Masson pine, which include seedling height, ground diameter, height-to-diameter ratio, stem length, stem curvature and crown projection area.

Images