Soybean plant phenotype information extraction method based on three-dimensional point cloud

Abstract

The present application relates to the technical field of soybean plant information analysis, and particularly relates to a soybean plant phenotype information extraction method based on three-dimensional point cloud. The method comprises the following steps: collecting three-dimensional point cloud data of soybean plants, performing standardization pretreatment to obtain a training set; constructing a SofGroup three-dimensional point cloud instance segmentation model, and fine-tuning the model based on the training set; inputting three-dimensional point cloud of soybean plants to be measured into the trained segmentation model, removing pod parts and retaining plant main body point cloud; based on the segmented plant main body point cloud, using an algorithm based on skeleton analysis, extracting main stems and branch skeletons representing plant topological structure, and calculating various plant phenotypic traits. The method has the advantages that: through three-dimensional point cloud pretreatment and SoftGroup model adaptation fine-tuning, the pods are removed and the plant main body is separated; combined with the skeleton analysis algorithm to extract the main stems and branches, the plant phenotype is automatically calculated, the problems such as low efficiency of manual measurement are solved, and the extraction accuracy and practicability are improved.

Description

Technical Field

[0001] This invention relates to the field of soybean plant information analysis technology, and in particular to a method for extracting soybean plant phenotypic information based on three-dimensional point clouds. Background Technology

[0002] Soybeans are an important food and economic crop in my country, and soybean plant traits (such as the number of pods, pod opening status, pod length / width / thickness, plant height, and number of branches) are core indicators for evaluating the quality of soybean varieties and predicting yield. Therefore, trait analysis of soybean plants is a crucial link in soybean breeding and cultivation research. Traditional trait analysis of soybean plants relies on manual peeling, counting, and visual judgment, which is not only time-consuming and labor-intensive, but also prone to inaccurate data due to human subjective errors or mutual obstruction between pods and between pods and the main body of the plant. This makes it difficult to meet the needs of large-scale breeding and precision agriculture.

[0003] To address the drawbacks of manual soybean variety evaluation, the industry is gradually exploring the use of machine vision technology to assist in the evaluation process. Current technologies are mostly based on two-dimensional images: two-dimensional images of soybean plants are acquired using a camera, image segmentation algorithms are used to extract the pod region and the main body of the soybean plant, and feature extraction models are then used to identify pod bursting status, estimate pod size, and calculate the soybean plant's phenotypic data. However, two-dimensional images only reflect the planar projection information of the soybean plant and are easily affected by environmental factors such as changes in light, leaf occlusion, and shooting angle. For example, leaf occlusion can lead to incomplete segmentation of the pod region, and side lighting can cause stretching and deformation of pod size. These problems directly hinder the improvement of the accuracy of two-dimensional image-based variety evaluation, failing to meet the needs of high-precision breeding.

[0004] In recent years, three-dimensional point cloud technology has become an important direction for crop phenotypic detection because it can completely characterize the spatial structure of plants. Existing technologies have the following problems: (1) Three-dimensional point cloud acquisition mostly relies on single-view scanning or general registration algorithms, which easily leads to problems such as missing point clouds of soybean plants (especially single plants in the field) and large registration errors; (2) Point cloud labeling mostly adopts the method of labeling the pod state and the number of seeds separately, which requires manual maintenance of two types of label systems, resulting in low labeling efficiency (labeling a single plant takes more than 30 minutes) and poor label correlation; (3) The original point cloud has problems such as scale differences (caused by different plant sizes and scanning distances) and data redundancy. Directly inputting it into a deep learning model easily leads to low training efficiency and poor recognition generalization, making it difficult to achieve efficient and accurate joint recognition of pod traits.

[0005] In summary, current soybean plant trait analysis techniques still face problems such as low efficiency, poor accuracy, and weak multi-task collaborative capabilities. There is an urgent need for an automated and high-precision seed evaluation technology that can integrate the advantages of three-dimensional point clouds and achieve multi-trait collaborative detection to meet the actual needs of modern soybean breeding research. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides a method for extracting soybean plant phenotypic information based on three-dimensional point clouds.

[0007] The purpose of this invention is to provide a method for extracting phenotypic information of soybean plants based on three-dimensional point clouds, which specifically includes the following steps:

[0008] S1. Collect three-dimensional point cloud data of soybean plants, perform standardized preprocessing, and obtain the training set;

[0009] S2. Construct a SofGroup 3D point cloud instance segmentation model and fine-tune the model based on the training set;

[0010] S3. Input the three-dimensional point cloud of the soybean plant to be tested into the trained segmentation model, remove the soybean pods, and retain the main point cloud of the plant.

[0011] S4. Based on the segmented plant main point cloud, the main stem and branch skeleton representing the plant topology are extracted using a skeleton analysis-based algorithm, and various plant phenotypic traits are calculated.

[0012] Preferably, in step S4, the plant skeleton is extracted from the main point cloud of the plant, and the phenotypic data of the plant skeleton are obtained, specifically including:

[0013] The PC-Skeletor algorithm was applied to the main point cloud of the plant to generate the original topological skeleton.

[0014] Treating the topological framework as a graph, a graphical method is used to extract the main stem path of the plant from the topological framework.

[0015] Subtract all nodes and edges on the main stem path from the topological skeleton, and the remaining part is the branch skeleton;

[0016] Calculate the height of the main plant body based on the main stem path;

[0017] A series of normal cutting planes are generated along the main stem path. For each normal cutting plane, the RANSAC algorithm is used to fit each normal cutting plane to a circle. The diameter of each fitted circle is the stem thickness of the plant body at the corresponding normal cutting plane.

[0018] Each connection point between the branch skeleton and the main stem skeleton is defined as a node, and the total number of nodes is the number of main stem nodes of the plant.

[0019] In the branch skeleton connected to the main stem skeleton, the branch skeleton with a length greater than the set branch length threshold is defined as a valid branch, and the number of valid branches is the number of branches of the plant body.

[0020] For each effective branch, take a segment of the skeleton vector near the connection point of the main stem skeleton, and take a segment of the skeleton vector of the main stem skeleton at that connection point. Calculate the branching angle between the two skeleton vectors, calculate the average value of all branching angles, and obtain the overall opening degree of the plant body.

[0021] Preferably, the main stem path of the plant body is extracted from the topological skeleton using a graphical method, including:

[0022] The main stem path is obtained by solving the following formula:

[0023] ;

[0024] in, Indicates the main stem path. Represents the topological skeleton from the root node The set of paths to all leaf nodes. Representing an edge The weight, This represents the point cloud within the topological skeleton.

[0025] Preferably, step S1 includes the following sub-steps:

[0026] S11. Perform multi-view scanning on the target soybean plant, simultaneously acquire depth and color images from each view, and place markers on the soybean plant for spatial reference.

[0027] S12. Spatial alignment of depth image data is performed to unify all depth image data into the same global coordinate system;

[0028] S13. Based on the registered depth image data and color image data, an initial color three-dimensional point cloud of soybean plants is generated. The initial color three-dimensional point cloud is then subjected to post-processing such as noise removal, data simplification and surface smoothing to obtain an optimized three-dimensional point cloud model.

[0029] S14. Perform instance segmentation and annotation on the optimized 3D point cloud model, and simultaneously complete annotation cleaning and quality verification to ensure the purity and integrity of the main point cloud of the plant.

[0030] S15. Point cloud standardization: The labeled point cloud data that has passed the S14 verification is resampled and spatially normalized, and the coordinates are mapped to the [-1,1] interval;

[0031] S16. Data partitioning: Divide the standardized point cloud into training and testing sets according to the proportion, and maintain a balanced distribution of the proportion of plant subjects and non-plant subjects.

[0032] Preferably, in step S11, a three-dimensional scanning device is used to scan the target soybean plant from multiple preset scanning stations, and depth images and color images are acquired simultaneously at each preset scanning station; 4-6 marker points are set on a single soybean plant, and the marker points are respectively pasted on the base, different branches in the middle and the top main stem of the soybean plant, and the marker points do not cover the pod area.

[0033] Step S12 specifically includes the following sub-steps:

[0034] S121. Coarse registration: Based on marker points and geometric feature points extracted from depth images or appearance feature points extracted from color images, feature matching is performed between different images, and the rigid body transformation matrix between adjacent viewpoint images is estimated to achieve preliminary alignment.

[0035] S122. Fine registration and global optimization: The iterative nearest point algorithm is used to refine the coarse registration results; the bundle adjustment method is used to globally optimize the rigid body transformation matrix of all scanning stations to minimize the cumulative error, and finally unify all depth images to the same global coordinate system.

[0036] Preferably, step S13 specifically includes the following sub-steps:

[0037] S131. For each registered depth image, the pixel coordinates are calculated based on the camera intrinsic parameters using the principle of coordinate back projection. u , v ) and the depth value is converted to a 3D point in the global coordinate system. X , Y , Z The data is then processed and fused with the (R,G,B) color information of the corresponding color image to synthesize a complete initial color 3D point cloud.

[0038] S132. For the reconstructed initial color 3D point cloud, apply statistical filtering or radius filtering to remove outlier noise points; use voxelized mesh downsampling method to simplify the data while maintaining the model shape; use moving least squares method to smooth the point cloud surface to obtain the 3D point cloud data of soybean plants.

[0039] Preferably, step S15 specifically includes the following sub-steps:

[0040] S151. Resampling: Set the number of target points N; use a random sampling algorithm to unify the point cloud scale. When the number of points exceeds N, downsampling is performed. When the number of points is less than N, the original data is retained. During the sampling process, spatial coordinates, normal vectors and label data are synchronously associated.

[0041] S152. Normalization: Calculate the geometric center point of the point cloud and translate the entire point cloud to a reference system with the center point as the origin; then, through proportional scaling, map all coordinates to the [-1,1] interval to eliminate scale differences.

[0042] S153. Format Conversion and Storage: Convert the processed spatial coordinates, normal vectors, semantic tags, and instance tags into tensor format, combine and encapsulate them, and then serialize them into binary files for storage, with each plant as a unit.

[0043] Preferably, the plant phenotypic traits include main stem-related traits, branching-related traits, and overall structural traits;

[0044] Main stem related traits include main stem length, number of main stem nodes, average internode spacing, stem diameter of each segment of the main stem, and average stem diameter;

[0045] Branching-related traits include the number of effective branches, the longest branch length, and the average branch angle.

[0046] Overall structural traits include plant height, canopy width, and overall openness.

[0047] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0048] (1) Based on three-dimensional data, the whole soybean plant is analyzed. Compared with two-dimensional data, it has a stronger spatial expression ability, can capture plant structural features more comprehensively and accurately, and improves the accuracy of the analysis data.

[0049] (2) Using a deep learning model for point cloud segmentation to segment soybean plants and pods can effectively learn the local and global structural features in the point cloud, and achieve efficient and precise segmentation of soybean plants and pods.

[0050] (3) Through examples, individual phenotypic analysis of each pod can be performed, providing a data foundation for subsequent pod screening, superior breeding, etc.

[0051] (4) Extracting the skeleton of the plant and removing invalid point cloud noise can more effectively analyze the characteristics of the main trunk and lateral branches.

[0052] (5) The entire technical solution has a high degree of automation, high accuracy and high stability, which solves the inefficiency of previous manual analysis and has strong generalization. Attached Figure Description

[0053] Figure 1 This is a flowchart of a soybean plant phenotypic information extraction method based on three-dimensional point clouds according to an embodiment of the present invention.

[0054] Figure 2This is a flowchart of the PC-Skeletor algorithm provided according to an embodiment of the present invention. Detailed Implementation

[0055] In the following description, embodiments of the invention will be described with reference to the accompanying drawings. In the description below, the same modules are denoted by the same reference numerals. Where the same reference numerals are used, their names and functions are also the same. Therefore, their detailed description will not be repeated.

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0057] This invention provides a method for extracting soybean plant phenotypic information based on three-dimensional point clouds, specifically including the following steps:

[0058] S1. Collect 3D point cloud data of soybean plants and perform standardized preprocessing. The specific steps are as follows:

[0059] S11. Multi-view data acquisition: Using a 3D scanner, the target soybean plant is scanned from multiple preset scanning stations; at each scanning station, the 3D scanner simultaneously acquires a depth image and a color image from that viewpoint (depth data and color data are acquired synchronously to ensure pixel-level spatial alignment); the depth image records the distance information of each pixel in the scene relative to the scanner, and the color image provides the corresponding texture color information; during the scanning process, marker points are placed on the plant or its supporting device as a common reference for subsequent registration.

[0060] In some embodiments, the device selection and parameter settings are as follows: Select a 3D scanner (model: FAROFocusS70), set the scanning accuracy to 0.1mm, the dot pitch to 0.5mm, and the scanning distance range to 1.5~3m; Simultaneously enable the color image acquisition function, set the color image resolution to 2048×1536 pixels, and the frame rate to 30fps to ensure pixel-level alignment between the depth image and the color image;

[0061] Use circular reflective markers with a diameter of 5mm (e.g., PET reflective film, reflectivity ≥85%). Place 4 markers on a single plant: 1 at the base of the plant (10cm from the ground on the stem), 2 on different branches in the middle (30-40cm from the base, avoiding the pods), and 1 on the top main stem (5cm from the top). The center of the marker must be flat and attached to the stem to avoid feature extraction errors caused by wrinkles.

[0062] Six scanning stations were evenly distributed in a 360° pattern on the horizontal plane, with the center of the soybean plant as the center (the angle between adjacent stations was 60°), and each station was 1.8m away from the center of the plant. For the distribution of pods in the vertical direction of the plant, each station was scanned in two layers (the lower layer scan height was 0-40cm, and the upper layer scan height was 40-80cm) to ensure that the top and bottom pods were completely covered. The scanning time for each station was 15s. During the scanning process, the scanner was kept still. If the plant shook slightly (amplitude ≤2cm), the station was scanned again once, and the overlap of the two scans was used for subsequent processing.

[0063] S12. Multi-source data registration: Spatial alignment of the acquired multi-view, independent coordinate system depth images; specifically including the following sub-steps:

[0064] S121. Coarse registration: Based on marker points and geometric or appearance feature points extracted from depth or color images, feature matching is performed between different images, and the rigid body transformation matrix between adjacent viewpoint images is estimated to achieve preliminary alignment.

[0065] In some embodiments, the compute_convex_hull convex hull function of the Open3D library is used to extract geometric feature points such as pod vertices and stem bifurcation points in the depth image (20-30 feature points are extracted from each depth image); the Canny edge detection algorithm (threshold: low threshold 50, high threshold 150) is used to extract appearance feature points in the color image (30-40 texture edge points are extracted from each color image); the extracted geometric feature points and appearance feature points are merged into a joint feature point set;

[0066] The marker matching method is as follows: Identify reflective markers (connected regions with grayscale values ​​≥ 240) in each depth image and obtain the three-dimensional coordinates of the four markers; using the coordinates of the markers at the first scanning station as a reference, calculate the Euclidean distance between the markers at the other five scanning stations and the reference marker, filter matching pairs with a distance error ≤ 0.3 mm, and preliminarily determine the positional relationship between adjacent stations;

[0067] The rigid body transformation matrix estimation method is as follows: Based on the joint feature point set and the marker point matching results, the Random Sample Consensus (RANSAC) algorithm (1000 iterations, inlier threshold 0.2mm) is used to estimate the rigid body transformation matrix (including translation vector and rotation matrix) of the depth images of adjacent stations, and complete the coarse registration. At this time, the error of the overlapping area of ​​the point cloud of adjacent stations is controlled within 1-1.5mm.

[0068] S122. Fine registration and global optimization: The iterative nearest point algorithm is used to refine the coarse registration results; the bundle adjustment method is used to globally optimize the rigid body transformation matrix of all scanning stations to minimize the cumulative error, and finally unify all depth images to the same global coordinate system.

[0069] In some embodiments, the fine-tuning method of the Iterative Closest Point (ICP) algorithm is as follows: taking the coarsely registered point cloud as input, the ICP algorithm is used for fine registration; the iteration termination condition is set: the number of iterations reaches 50 or the average distance error between two adjacent iterations is ≤0.05mm; in each iteration, 1000 random points in the overlapping area are selected as sampling points, the distance error of the corresponding point pairs is calculated, and the rigid body transformation matrix is ​​updated by the least squares method, so that the overlap error of the point clouds of adjacent stations is reduced to within 0.3mm;

[0070] The global optimization method of the bundle adjustment method is as follows: The point clouds of the 6 stations and their corresponding rigid body transformation matrices are input into the bundle adjustment algorithm. The optimization objective is to minimize the sum of squared cumulative distance errors of the point clouds of all stations. The constraint is the consistency of the transformation matrices of adjacent stations (the error between the product of adjacent matrices and the directly calculated cross-station matrix is ​​≤0.02mm). The optimization problem is solved by the Levenberg-Marquardt algorithm, and the globally optimal transformation matrix is ​​output. The depth images of all stations are unified to the same global coordinate system (the origin of the coordinate system is the center of the scanner of the first scanning station).

[0071] S13. Point cloud generation and post-processing: Generate the final three-dimensional point cloud model based on the registered data; specifically, generate the initial color three-dimensional point cloud of soybean plants based on the registered depth image data and color image data, and perform post-processing operations such as noise removal, data simplification and surface smoothing on the initial color three-dimensional point cloud to obtain the optimized three-dimensional point cloud model.

[0072] Specifically, it includes the following sub-steps:

[0073] S131. Color 3D Point Cloud Reconstruction: For each registered depth image, the pixel coordinates are reconstructed using the coordinate back projection principle and based on the camera intrinsic parameters. u , v ) and the depth value is converted to a 3D point in the global coordinate system. X , Y , Z The data is then processed and fused with the (R,G,B) color information of the corresponding color image to synthesize a complete initial color 3D point cloud.

[0074] In some embodiments, the (R, G, B) values ​​(ranging from 0 to 255) of the corresponding color image are read and compared with the three-dimensional spatial points ( X , Y , Z The points are associated one by one to generate an initial colored 3D point cloud of a single plant (approximately 80,000-100,000 points), which is then stored as a .pcd file.

[0075] S132. Post-process the reconstructed initial colored 3D point cloud to obtain the 3D point cloud data of soybean plants; this includes: applying statistical filtering or radius filtering to remove outlier noise points; using a voxelized mesh downsampling method to simplify the data while maintaining the model shape; and using algorithms such as moving least squares to smooth the point cloud surface. After the above processing, the 3D point cloud data of soybean plants can be obtained.

[0076] In some embodiments, the outlier noise removal method is as follows: a statistical filtering algorithm is used, the number of neighboring points is set to 10, the average distance between each point and its neighboring points is calculated, and points with a distance greater than "average distance + 2 times standard deviation" are identified as outliers (mostly dust and light spots in the scanning environment). After removal, the number of points in the point cloud is retained to about 90%; if there is local dense noise (such as reflective interference points on the surface of the stem), radius filtering is added (points with a search radius of 5mm and a number of neighboring points ≤ 3 are removed).

[0077] The voxel downsampling method is as follows: Set the voxel side length to 1mm, replace all points in the same voxel with the voxel center coordinates, and take the average (R,G,B) of all points in the voxel as the color value of the center coordinates, so that the number of point clouds is reduced to 20,000-25,000, reducing the amount of data while maintaining the pod morphological characteristics (such as pod length and curvature).

[0078] The surface smoothing method is as follows: using the moving least squares method, setting the neighborhood radius to 5mm and the polynomial order to 2, the surface of the downsampled point cloud is fitted to correct the small protrusions on the point cloud surface (such as the burrs on the surface of the bean pod caused by scanning noise), so that the surface roughness of the point cloud is ≤0.1mm, and finally the optimized color 3D point cloud is obtained.

[0079] S14. Plant Main Body Point Cloud Separation, Labeling, and Quality Verification: Based on 3D point cloud processing software, the optimized 3D point cloud model is segmented and labeled, and label cleaning and quality verification are performed simultaneously to ensure the purity and integrity of the plant main body point cloud; the specific operations are as follows:

[0080] S141. Instance Segmentation and Labeling: Open CloudCompare software, import the 3D point cloud file optimized in step S13, and enable the instance segmentation and labeling function; use the point cloud selection tool (rectangular selection + manual fine-tuning) to select instance objects within a single plant one by one: if the instance object is a stem, branch, or leaf, select all its point clouds and label them uniformly as the "main plant" category; if the instance object is a pod, select all its point clouds and label them as the "non-main plant" category; during the labeling process, if the same main plant structure is segmented into multiple point cloud clusters, use the software's instance merging function to integrate them into a single instance; if the main plant and pod point clouds are stuck together, separate them using the point cloud cutting tool (based on a distance threshold of 2mm) to ensure the independence and integrity of the "main plant" instance.

[0081] S142. Label Cleaning: Targeted cleaning of the labeled raw data, specifically including:

[0082] Tag cleaning: Remove redundant tags (such as tags that are repeatedly labeled in the same instance) and incorrect tags (such as tags that mistakenly label the background as a bean pod);

[0083] Point cloud cleaning: merge broken points cloud segments of the same instance, separate incorrectly connected points cloud instances, and remove extremely small point cloud clusters with ≤50 points (such point clouds do not have characteristic representation significance).

[0084] S143. Quality Verification: Verify the annotation effect through quantitative indicators, calculating the instance segmentation intersection-over-union ratio (IoU), particle count annotation accuracy, and pod bursting state annotation accuracy of the annotated data; details are as follows:

[0085] Instance segmentation IoU: Calculate the intersection-union ratio (IoU) between the labeled pod instances and the real pod regions, requiring IoU ≥ 90%;

[0086] Particle count accuracy: Compare the labeled particle count with the actual particle count counted by manual peeling, and the accuracy rate should be ≥95%;

[0087] Accuracy of labeling the pod breakage status: The accuracy rate of labeling the pod breakage status is required to be ≥96% compared with the actual status as determined by human visual inspection.

[0088] Samples that do not meet the above indicators are returned to S142 for re-labeling and cleaning until all samples meet the quality requirements.

[0089] S15. Point Cloud Standardization Processing: The labeled point cloud data that has passed the verification in S14 is resampled and spatially normalized to unify the data size and scale, providing standardized input for deep learning models; specifically including the following sub-steps:

[0090] S151. Resampling: To meet the requirement of consistent input data scale for deep learning models, the point cloud of each plant after integration is resampled: a predefined target number of points N is set, and a random sampling algorithm is used to adjust the scale of the point cloud data; for point clouds with a total number of points exceeding the target number of points N, random downsampling is performed to uniformly reduce the amount of data; for point clouds with a total number of points less than the target number of points N, all original data points are retained; the sampling process is applied simultaneously to spatial coordinates, normal vectors, and all label data to ensure that the correspondence between various types of data remains unchanged after sampling, and to maintain the integrity of point cloud morphological features and annotation structure while unifying the data scale; in a specific embodiment, the target number of points N is set to 22000 (to adapt to the input requirements of the improved SoftGroup model), and a uniform random sampling algorithm is used to adjust the scale of the point cloud:

[0091] For the main point cloud area of ​​the plant: retain the original point cloud proportionally, and give priority to retaining key morphological feature points such as stem and branch bifurcation points and main stem apex;

[0092] For point cloud areas that are not the main body of the plant: these areas are removed during the sampling process, and only the point cloud of the main body of the plant is retained;

[0093] During sampling, spatial coordinates, normal vectors, semantic labels, instance labels, and other data are synchronously associated to ensure that the correspondence between various data types does not shift after sampling. S152. Normalization: Spatial coordinate normalization is performed on the resampled point cloud data to eliminate spatial scale uncertainties introduced by differences in measurement distance, angle, or plant size. This process first calculates the geometric center point of all spatial coordinates of the point cloud and then translates the entire point cloud to a reference system with this center point as the origin, i.e.: P center = P raw -C;

[0094] In the formula, P center The coordinates of the centered point cloud; P raw The coordinates of the original point cloud are N×3; C is the geometric center point; the expression for C is as follows:

[0095] ;

[0096] Calculate the absolute maximum value of each coordinate axis component of the point cloud in the translated coordinate system, and use this as the scaling factor. All spatial coordinates are scaled proportionally, ultimately normalizing the spatial distribution range of the entire point cloud to a standardized numerical range, resulting in the normalized point cloud coordinates. All of its coordinate values ​​are within the range [-1, 1].

[0097] S153. Format Conversion and Storage: Convert the normalized point cloud data into a format suitable for deep learning frameworks: Convert the spatial coordinate data representing the spatial location of the point cloud, the normal vector data representing the surface geometric characteristics, the semantic label data representing the object category, and the instance label data representing the uniqueness of the instance into the corresponding tensor formats respectively; combine and encapsulate each tensor according to the preset structure, and serialize them into binary data files (such as pth format) for storage, with each plant as the unit, to provide direct input for the training and inference of the SoftGroup model in the future.

[0098] S16. Data Partitioning: To ensure the generalization ability of model training and the effectiveness of testing, the standardized point cloud data stored in S15 is scientifically partitioned as follows:

[0099] S161. Sample Selection: Standardized point cloud data of several potted soybean samples were selected. The samples covered multiple mainstream soybean varieties and the growth period covered the vegetative growth stage to the pod-setting stage (V3-R6 stage). This ensured the diversity and representativeness of the samples and met the data distribution requirements for model training.

[0100] S162. Splitting rule: The selected samples are divided into training and test sets in an 8:2 ratio:

[0101] Training set: accounting for 80%, containing balanced samples of each variety, used for parameter training and optimization of the SoftGroup model;

[0102] Test set: accounting for 20%, with no overlap with the training set, containing representative samples of each variety, used for independent validation of model performance;

[0103] During the partitioning process, the class distribution of samples must be kept balanced (the ratio of plant subjects to non-plant subjects should be consistent with the overall sample) to avoid poor model training performance due to data bias.

[0104] In some embodiments, the sample selection is specifically as follows: standardized point cloud data of 100 potted soybean samples are selected, covering 5 mainstream soybean varieties (20 plants for each variety) to ensure the rationality of the sample distribution; after corresponding division, the training set contains 80 samples (16 plants for each variety) and the test set contains 20 samples (4 plants for each variety) to further ensure the reliability of model training and validation.

[0105] S2. Construct a SofGroup 3D point cloud instance segmentation model (using a publicly available model), and fine-tune the model based on the training set to adapt the model to the feature distribution of soybean plant point clouds, ensuring the segmentation accuracy between the main plant body and non-plant body (pods); specifically including:

[0106] S21. Model Configuration: Select the SoftGroup 3D point cloud instance segmentation model (existing). Fine-tune the model based on the training set and test set divided in step S1 to adapt the model to the feature distribution of soybean plant point clouds. The model input format is (N is the number of points, 3 corresponds to X / Y / Z 3D coordinates), and the output is the instance segmentation result (including the category label and instance mask of each instance).

[0107] S22. Model Fine-tuning Training: Configure model training hyperparameters: Batch size (BatchSize) is set to 8 (to adapt to GPU memory capacity), initial learning rate is 0.001, cosine annealing learning rate decay strategy is adopted (Tmax=100), training iterations are 100 rounds, and early stopping strategy is set: when the loss on the validation set does not decrease for 10 consecutive rounds, training is terminated early to avoid overfitting.

[0108] The loss function used is the DiceLoss function, which constrains the overlap between the instance clusters output by the model and the real instance mask to ensure segmentation accuracy;

[0109] After fine-tuning and training, the model achieved an IoU of ≥90% for the main plant segment and a rejection rate of ≥98% for non-plant segments (pods), providing a clean data foundation for subsequent plant phenotypic extraction.

[0110] S3. Plant Main Body Point Cloud Segmentation Based on SoftGroup Model: Using the 3D point cloud of the soybean plant to be tested as input, the trained segmentation model performs instance segmentation inference to achieve accurate segmentation of the main body of the plant from non-plant parts (pods), removing the pods and retaining the pure main body point cloud of the plant; the specific process is as follows:

[0111] S31. Preprocessing of the point cloud to be tested: For the soybean plants to be tested, the standardized point cloud data is obtained by following the collection and preprocessing process in step S1, ensuring that the data size and format are consistent with the training set (number of target points N=22000, coordinate range [-1,1]).

[0112] S32. Segmentation Inference Execution: Input the standardized point cloud to be tested into the fine-tuned SoftGroup model. The model outputs the category label ("plant subject" or "non-plant subject") and instance mask of each point through forward propagation. The "plant subject" label corresponds to the point cloud of stem, branch and leaf, and the "non-plant subject" label corresponds to the point cloud of pod.

[0113] S33. Pod Removal and Main Body Preservation: Based on the category labels output by the model, filter out all instance masks labeled as "main body of the plant" and extract all point clouds corresponding to the mask; at the same time, remove the point clouds labeled as "non-main body of the plant" (pods) to obtain a pure soybean plant main body point cloud (containing only the main stem, branches, and leaves), and then extract phenotypic traits based only on this point cloud.

[0114] S34. Segmentation result verification: Perform quality verification on the main point cloud of the plant after removing the pods: calculate the point cloud purity (the proportion of the main plant points to the total number of points), which should be ≥99%; if there are residual pod point clouds (the number of points in a single cluster is ≥30), remove them using the manual fine-tuning tool to ensure that subsequent phenotypic extraction is not interfered with.

[0115] S4. Extraction of soybean plant phenotypic information: Based on the segmented and purified point cloud of the main plant, an algorithm based on skeleton analysis is used to extract the main stem and branch skeleton representing the plant's topological structure, and then various plant phenotypic traits are calculated; the specific steps are as follows:

[0116] S41. Initial skeleton extraction: The PC-Skeletor algorithm is applied to the main point cloud of the plant to generate the original topological skeleton;

[0117] Specifically, the PC-Skeletor algorithm is applied to the main point cloud of the plant to generate the original topological skeleton. The skeleton It consists of a series of nodes and connecting edges, capturing the overall topological structure of the plant. The core idea of ​​the PC-Skeletor algorithm is to shrink the dense point cloud into a central line that represents its topological structure. See also Figure 2 The algorithm flow is as follows:

[0118] Point cloud shrinkage initialization: Using the original plant main point cloud P (N points) as input, a K-nearest neighbor graph is constructed to establish neighborhood relationships between points. Based on these neighborhood relationships, a Laplacian matrix L is constructed. This matrix, along with the Laplacian operator, causes the center of each point's neighbor points to shift, resulting in a shrinkage energy function that gradually shrinks the point cloud.

[0119] ;

[0120] Among them, E - l represents the contraction energy function, and X represents the point cloud coordinate matrix;

[0121] To maintain the overall shape of the point cloud during the shrinkage process, a shrinkage constraint is added. This constraint ensures that the shrinkage occurs inward along the normal vector direction, preventing shape distortion. Specifically:

[0122] ;

[0123] Among them, E - c represents the shrinkage constraint term, X0 represents the original point cloud coordinates (i.e., the original main point cloud P of the plant), and W represents the weight matrix.

[0124] Based on the contraction energy function E - l and contraction constraint term E- c, construct the complete energy function E as follows:

[0125] ;

[0126] in, α and β The hyperparameter representing the weights of the two energy terms;

[0127] By solving the linear system of the following equation, the position of each point cloud after shrinkage can be obtained: ;

[0128] After multiple iterations, the original dense point cloud P of the plant body is shrunk into a sparse and disordered skeleton point set S that represents its topological structure.

[0129] For the obtained skeleton point set S, the minimum spanning tree algorithm is used to construct the initial connectivity graph, and then the connectivity graph is optimized according to geometric rules to obtain the topological skeleton. .

[0130] S42. Main stem path extraction based on graph theory: Treat the topological skeleton as a graph and use a graphical method to extract the main stem path of the plant from the topological skeleton;

[0131] Specifically, the topological skeleton Consider the graph G(V,E), where V represents the set of nodes and E represents the set of edges. The main stem corresponds to the longest node in graph G(V,E) that is closest to the root node (defined as the skeleton node closest to the lowest point in the point cloud). The shortest path to a certain terminal leaf node; the main stem path is obtained by solving the following formula:

[0132] ;

[0133] in, Indicates the main stem path. Represents the topological skeleton from the root node The set of paths to all leaf nodes. Let represent the weight of edge e in the graph, and P represent the point cloud in the topological skeleton. In this embodiment of the invention, the weights... Defined as a function of the point cloud density of the line segment represented by edge e in graph G(V,E), to obtain a thicker and more continuous main stem path.

[0134] S43. From the topological skeleton Subtract the main stem path All nodes and edges on the graph, and the remaining part is the branch skeleton.

[0135] S44. Based on the obtained main stem path and branching framework, obtain the phenotypic data of the plant framework. The phenotypic data of the plant framework include plant height, stem diameter, number of main stem nodes, number of branches, and branching angle. The corresponding process for obtaining the phenotypic data of the plant framework includes:

[0136] Plant height: Calculate the height of the main body of the plant based on the main stem path, specifically the height along the main stem path. Top of the node With the root node The Euclidean distance between them is the plant height. H ,Right now ;

[0137] Stem thickness: A series of N normal cutting planes are generated along the main stem path. For each normal cutting plane, the RANSAC algorithm is used to fit a circle to each normal cutting plane. The diameter of each fitted circle is... This refers to the stem diameter of the plant body at the corresponding normal cutting plane; in this embodiment of the invention, the average of all diameters is taken. The thickness of the stem of the entire plant is called the stem diameter. ;

[0138] Number of main stem nodes: Each connection point between the branch framework and the main stem framework is defined as a node, and the total number of nodes is the number of main stem nodes of the plant.

[0139] Branch count and branching angle: Among the branch skeletons connected to the main stem skeleton, those with a length greater than a set branch length threshold are defined as valid branches. The number of valid branches is the number of branches in the main body of the plant. For each valid branch, a segment of the skeleton vector near the connection point with the main stem skeleton is taken. And take a segment of the skeleton vector of the main stem skeleton at the connection point. Calculate the branch angle between the two skeleton vectors. Calculate the average value of all branch angles to obtain the overall opening degree of the plant body.

[0140] S45. The extracted soybean plant phenotypic traits are organized and output according to the preset format. The output content includes:

[0141] Basic information: Soybean variety, growth stage, sampling time, and sampling environment to be analyzed;

[0142] Main stem characteristics: main stem length (mm), number of main stem nodes, average internode spacing (mm);

[0143] Branching characteristics: number of effective branches (number), longest branch length (mm), average branch angle (°).

[0144] Overall structural characteristics: plant height (mm), canopy width (mm);

[0145] Visualization results: point cloud model of the main plant body, schematic diagram of the extraction of the main stem and branch skeleton, and phenotypic trait annotation diagram.

[0146] The output formats support Excel spreadsheets, PDF reports, and 3D visualization files (.pcd format), which facilitates subsequent data statistical analysis and breeding research applications.

[0147] The key technical point of this invention is:

[0148] 1. Accurate segmentation and standardized preprocessing techniques for plant main body point clouds: The "instance segmentation annotation + quantization quality verification" strategy is adopted to clearly distinguish between "plant main body (stem, branch, leaf)" and "non-plant main body (pod)". Combined with the quantitative indicators of instance segmentation IoU ≥ 90% and plant main body extraction accuracy ≥ 96%, the problems of blurred plant and pod boundaries and uncontrollable extraction purity in traditional annotation are solved. At the same time, random resampling (unified point cloud scale N=22000), spatial normalization (coordinate mapping to [-1,1]) and tensor format encapsulation are used to construct a standardized input dataset adapted to the SoftGroup model, which eliminates scale interference caused by differences in scanning distance and plant size, and ensures the consistency of model training and inference.

[0149] 2. The publicly available SoftGroup 3D point cloud instance segmentation model is adopted. The model parameters are optimized based on a soybean plant-specific dataset. The DiceLoss loss function is used to enhance the segmentation boundary accuracy, achieving efficient separation of the plant body and pods. The pod removal rate is ≥98% and the point cloud purity of the plant body is ≥99%. Through targeted fine-tuning, it can be adapted to the structural features of soybean plants, reducing training costs and engineering difficulty while ensuring segmentation accuracy.

[0150] 3. The original topological skeleton of the plant's main point cloud is generated using the PC-Skeletor algorithm. The main stem path is solved using a graphical method (the longest path is selected by maximizing the edge weight, and the weight is defined as the density function of the line segment point cloud) to obtain a robust and continuous main stem structure. Furthermore, through a closed-loop process of "main stem stripping - branch selection - trait calculation", effective branches are automatically identified, the number of main stem nodes is counted, and core phenotypic traits such as plant height, main stem length, branch angle, and overall spread are calculated. This solves the problems of low efficiency and strong subjectivity in traditional manual measurement, and realizes the automated and high-precision extraction of plant topological structure and phenotypic parameters.

[0151] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0152] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for extracting soybean plant phenotypic information based on three-dimensional point clouds, characterized in that: Specifically, the steps include the following: S1. Collect 3D point cloud data of soybean plants, perform standardized preprocessing, and obtain a training set; including the following sub-steps: S11. Using a 3D scanning device, the target soybean plant is scanned from multiple preset scanning stations. At each preset scanning station, depth and color images are simultaneously acquired from the corresponding viewpoint. 4-6 marker points are set on a single soybean plant. The marker points are affixed to the base, different branches in the middle, and the top main stem of the soybean plant, and the marker points do not obstruct the pod area. S12. Spatial alignment of depth image data is performed to unify all depth image data into the same global coordinate system; S13. Based on the registered depth image data and color image data, an initial color three-dimensional point cloud of soybean plants is generated. The initial color three-dimensional point cloud is then subjected to post-processing, including noise removal, data simplification, and surface smoothing, to obtain an optimized three-dimensional point cloud model. S14. Perform instance segmentation and annotation on the optimized 3D point cloud model, and simultaneously complete annotation cleaning and quality verification to ensure the purity and integrity of the main point cloud of the plant. S15. Point cloud standardization: The labeled point cloud data that has passed the S14 verification is resampled and spatially normalized, and the coordinates are mapped to the [-1,1] interval; S16. Data partitioning: Divide the standardized point cloud into training and testing sets according to the proportion, and maintain a balanced distribution of the proportion of plant subjects and non-plant subjects. S2. Construct a SofGroup 3D point cloud instance segmentation model and fine-tune the model based on the training set; S3. Input the 3D point cloud of the soybean plant to be tested into the trained segmentation model, remove the pods, and retain the main point cloud of the plant; the details are as follows: S31. Preprocessing of the point cloud to be tested: For the soybean plant to be tested, the standardized point cloud data is obtained by following the collection and preprocessing process in step S1. S32. Segmentation Inference Execution: Input the standardized point cloud to be tested into the fine-tuned SoftGroup model. The model outputs the category label and instance mask of each point through forward propagation. The category label includes "plant body" or "non-plant body". The "plant body" label corresponds to the point cloud of stem, branch and leaf, and the "non-plant body" label corresponds to the point cloud of pod. S33. Pod Removal and Main Body Preservation: Based on the category labels output by the model, filter out all instance masks labeled "main body of the plant" and extract all point clouds corresponding to the mask; at the same time, remove the point clouds labeled "non-main body of the plant" to obtain a pure soybean plant main body point cloud. S34. Segmentation result verification: Perform quality verification on the main point cloud of the plant after removing the pods: calculate the purity of the point cloud, which should be ≥99%; if there are residual pod point clouds, remove them using the manual fine-tuning tool to ensure that subsequent phenotypic extraction is not interfered with; the residual pod point cloud should have a single cluster of points ≥30; S4. Based on the segmented plant main point cloud, the main stem and branch skeleton representing the plant topology are extracted using a skeleton analysis-based algorithm, and various plant phenotypic traits are calculated.

2. The method for extracting soybean plant phenotypic information based on three-dimensional point clouds according to claim 1, characterized in that: In step S4, the plant skeleton is extracted from the main point cloud of the plant, and the phenotypic data of the plant skeleton are obtained, specifically including: The PC-Skeletor algorithm was applied to the main point cloud of the plant to generate the original topological skeleton. Treating the topological framework as a graph, a graphical method is used to extract the main stem path of the plant from the topological framework. Subtract all nodes and edges on the main stem path from the topological skeleton, and the remaining part is the branch skeleton; Calculate the height of the main plant body based on the main stem path; A series of normal cutting planes are generated along the main stem path. For each normal cutting plane, the RANSAC algorithm is used to fit each normal cutting plane to a circle. The diameter of each fitted circle is the stem thickness of the plant body at the corresponding normal cutting plane. Each connection point between the branch skeleton and the main stem skeleton is defined as a node, and the total number of nodes is the number of main stem nodes of the plant. In the branch skeleton connected to the main stem skeleton, the branch skeleton with a length greater than the set branch length threshold is defined as a valid branch, and the number of valid branches is the number of branches of the plant body. For each effective branch, take a segment of the skeleton vector near the connection point of the main stem skeleton, and take a segment of the skeleton vector of the main stem skeleton at that connection point. Calculate the branching angle between the two skeleton vectors, calculate the average value of all branching angles, and obtain the overall opening degree of the plant body.

3. The method for extracting soybean plant phenotypic information based on three-dimensional point clouds according to claim 2, characterized in that: The main stem path of the plant body is extracted from the topological skeleton using a graphical method, including: The main stem path is obtained by solving the following formula: ； in, Indicates the main stem path. Represents the topological skeleton from the root node The set of paths to all leaf nodes. Representing an edge The weight, This represents the point cloud within the topological skeleton.

4. The method for extracting soybean plant phenotypic information based on three-dimensional point clouds according to claim 1, characterized in that: Step S12 specifically includes the following sub-steps: S121. Coarse registration: Based on marker points and geometric feature points extracted from depth images or appearance feature points extracted from color images, feature matching is performed between different images, and the rigid body transformation matrix between adjacent viewpoint images is estimated to achieve preliminary alignment. S122. Fine registration and global optimization: The iterative nearest point algorithm is used to refine the coarse registration results; the bundle adjustment method is used to globally optimize the rigid body transformation matrix of all scanning stations to minimize the cumulative error, and finally unify all depth images to the same global coordinate system.

5. The method for extracting soybean plant phenotypic information based on three-dimensional point clouds according to claim 1, characterized in that: Step S13 specifically includes the following sub-steps: S131. For each registered depth image, the pixel coordinates are calculated based on the camera intrinsic parameters using the principle of coordinate back projection. u , v ) and the depth value is converted to a 3D point in the global coordinate system. X , Y , Z The data is then processed and fused with the (R,G,B) color information of the corresponding color image to synthesize a complete initial color 3D point cloud. S132. For the reconstructed initial color 3D point cloud, apply statistical filtering or radius filtering to remove outlier noise points; use voxelized mesh downsampling method to simplify the data while maintaining the model shape; use moving least squares method to smooth the point cloud surface to obtain the 3D point cloud data of soybean plants.

6. The method for extracting soybean plant phenotypic information based on three-dimensional point clouds according to claim 1, characterized in that: Step S15 specifically includes the following sub-steps: S151. Resampling: Set the number of target points N; use a random sampling algorithm to unify the point cloud scale. When the number of points exceeds N, downsampling is performed. When the number of points is less than N, the original data is retained. During the sampling process, spatial coordinates, normal vectors and label data are synchronously associated. S152. Normalization: Calculate the geometric center point of the point cloud and translate the entire point cloud to a reference system with the center point as the origin; then, through proportional scaling, map all coordinates to the [-1,1] interval to eliminate scale differences. S153. Format Conversion and Storage: Convert the processed spatial coordinates, normal vectors, semantic tags, and instance tags into tensor format, combine and encapsulate them, and then serialize them into binary files for storage, with each plant as a unit.

7. The method for extracting soybean plant phenotypic information based on three-dimensional point clouds according to claim 1, characterized in that: The plant phenotypic traits include main stem-related traits, branching-related traits, and overall structural traits; Main stem related traits include main stem length, number of main stem nodes, average internode spacing, stem diameter of each segment of the main stem, and average stem diameter; Branching-related traits include the number of effective branches, the longest branch length, and the average branch angle. Overall structural traits include plant height, canopy width, and overall openness.

Images