Soybean pod state joint labeling method based on three-dimensional point cloud

Abstract

This invention relates to the field of soybean plant information analysis technology, and particularly to a joint annotation method for soybean pod status based on three-dimensional point clouds. The method includes: multi-view scanning of the target soybean plant, simultaneously acquiring depth and color images from each viewpoint, and placing marker points on the soybean plant; spatially aligning the depth image data and unifying it to the same global coordinate system; generating an initial color three-dimensional point cloud of the soybean plant after registration, performing post-processing to obtain an optimized three-dimensional point cloud model; setting structured binary encoding rules, performing instance segmentation and annotation on the optimized three-dimensional point cloud model according to the encoding rules, and assigning a binary code conforming to the encoding rules to each pod instance; verifying and cleaning the annotated data, and then converting the processed point cloud data into a binary file through resampling and spatial normalization, extracting the binary code corresponding to the pod instance, and quickly parsing to obtain the pod bursting status and grain count information.

Description

Technical Field

[0001] This invention relates to the field of soybean plant information analysis technology, and in particular to a method for joint annotation of soybean pod status based on three-dimensional point clouds. Background Technology

[0002] Soybeans, as an important food and economic crop in my country, have pod bursting state (open / closed) and number of seeds as core traits for assessing yield, quality, and harvest timing, directly affecting variety breeding and field management efficiency. Traditional pod trait detection relies on manual peeling, counting, and visual judgment, which is not only time-consuming and labor-intensive but also prone to data inaccuracies due to human subjective errors or pod occlusion, making it difficult to meet the needs of large-scale breeding and precision agriculture. With the development of computer vision technology, two-dimensional image recognition methods have been gradually applied to pod trait detection. However, limited by two-dimensional perspective occlusion and light reflection interference, they cannot accurately restore the three-dimensional morphology of pods. The accuracy of identifying bursting state (requiring differentiation of surface crack depth) and number of seeds (requiring judgment of the spatial stacking relationship of seeds) is generally low, thus limiting its practicality.

[0003] Three-dimensional point cloud technology has become an important direction for crop phenotypic detection because it can completely characterize the spatial structure of plants. Existing technologies have the following problems: (1) Three-dimensional point cloud acquisition mostly relies on single-view scanning or general registration algorithms, which easily leads to problems such as missing point clouds of soybean plants (especially single plants in the field) and large registration errors; (2) Point cloud labeling mostly adopts the method of labeling the pod state and the number of seeds separately, which requires manual maintenance of two types of label systems, resulting in low labeling efficiency (labeling a single plant takes more than 30 minutes) and poor label correlation; (3) The original point cloud has problems such as scale differences (caused by different plant sizes and scanning distances) and data redundancy. Directly inputting it into a deep learning model easily leads to low training efficiency and poor recognition generalization, making it difficult to achieve efficient and accurate joint recognition of pod traits.

[0004] In summary, given the current pain points of soybean pod trait detection technology, such as low manual efficiency, insufficient two-dimensional recognition accuracy, cumbersome three-dimensional point cloud processing procedures, and poor compatibility between annotation and model, there is an urgent need for an integrated method that can combine three-dimensional point cloud acquisition, joint annotation, and data standardization. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a method for joint annotation of soybean pod states based on 3D point clouds.

[0006] The purpose of this invention is to provide a joint annotation method for soybean pod state based on 3D point cloud, which specifically includes the following steps:

[0007] S1. Perform multi-view scanning on the target soybean plant, simultaneously acquire depth and color images from each view, and place markers on the soybean plant for spatial reference.

[0008] S2. Spatial alignment of the depth image data is performed to unify all depth image data into the same global coordinate system;

[0009] S3. Based on the registered depth image data and color image data, an initial color 3D point cloud of soybean plants is generated. The initial color 3D point cloud is then subjected to post-processing, including noise removal, data simplification, and surface smoothing, to obtain an optimized 3D point cloud model.

[0010] S4. Set structured binary encoding rules, perform instance segmentation and annotation on the optimized 3D point cloud model according to the encoding rules, and assign a binary code that conforms to the encoding rules to each pod instance;

[0011] S5. The labeled point cloud data is checked and cleaned, and then resampled and spatially normalized to convert the processed point cloud data into a format suitable for deep learning and serialize it into a binary file for storage.

[0012] S6. Read the binary file, extract the binary code corresponding to the pod instance, and quickly parse to obtain the pod bursting status and number of pods for each pod.

[0013] Preferably, in step S1, a three-dimensional scanning device is used to scan the target soybean plant from multiple preset scanning stations, and depth images and color images from that perspective are acquired simultaneously at each scanning station; 4-6 marker points are set on a single soybean plant, and the marker points are respectively pasted on the base, different branches in the middle and the top main stem of the soybean plant, and the marker points do not cover the pod area.

[0014] Preferably, step S2 specifically includes the following sub-steps:

[0015] S21. Coarse registration: Based on marker points and geometric feature points extracted from depth images or appearance feature points extracted from color images, feature matching is performed between different images, and the rigid body transformation matrix between adjacent viewpoint images is estimated to achieve preliminary alignment.

[0016] S22. Fine registration and global optimization: The iterative nearest point algorithm is used to refine the coarse registration results; the bundle adjustment method is used to globally optimize the rigid body transformation matrix of all scanning stations to minimize the cumulative error, and finally unify all depth images to the same global coordinate system.

[0017] Preferably, step S3 specifically includes the following sub-steps:

[0018] S31. For each registered depth image, the pixel coordinates are calculated based on the camera intrinsic parameters using the principle of coordinate back projection. u , v ) and the depth value is converted to a 3D point in the global coordinate system.X , Y , Z The data is then processed and fused with the (R,G,B) color information of the corresponding color image to synthesize a complete initial color 3D point cloud.

[0019] S32. For the reconstructed initial color 3D point cloud, apply statistical filtering or radius filtering to remove outlier noise points; use voxelized mesh downsampling method to simplify the data while maintaining the model shape; use moving least squares method to smooth the point cloud surface to obtain the 3D point cloud data of soybean plants.

[0020] Preferably, step S4 further includes constructing a correspondence table according to structured binary encoding rules; the correspondence table includes the association information of decimal label, binary representation, number of grains and pod bursting state;

[0021] The encoding rule is as follows: the least significant bit of the binary number represents the pod-exploding state, 0 = no pod-exploding, 1 = pod-exploding; the index of the position of 1 in the second and above bits from the right represents the number of pods.

[0022] Based on the encoding rules and the corresponding relationship table, the optimized 3D point cloud model is segmented and labeled to distinguish between three semantic categories: popped pods, non-popped pods, and background. The number of grains in each pod is labeled, and a binary code conforming to the encoding rules is assigned to each pod instance.

[0023] The preferred correspondence table is as follows:

[0024] .

[0025] Preferably, step S4 further includes constructing a correspondence table according to structured binary encoding rules; step S5 specifically includes the following sub-steps:

[0026] S51. Data Verification and Cleaning: Manually verify and clean the original labeled data, removing redundant and erroneous labels; clean the problematic data found during verification; read the cleaned point cloud data file; parse the semantic category and number identifier of the instance object, map the semantic category identifier to an integer label, assign a unique instance identifier, and integrate the spatial coordinates, normal vector and corresponding label data of a single plant to reconstruct the three-dimensional point cloud scene of the complete plant.

[0027] S52. Resampling: Set the number of target points N; use a random sampling algorithm to unify the point cloud scale. When the number of points exceeds N, downsampling is performed. When the number of points is less than N, the original data is retained. During the sampling process, spatial coordinates, normal vectors and label data are synchronously associated.

[0028] S53. Normalization: Calculate the geometric center point of the point cloud and translate the entire point cloud to a reference system with the center point as the origin; then, through proportional scaling, map all coordinates to the [-1,1] interval to eliminate scale differences.

[0029] S54. Format Conversion and Storage: Convert the processed spatial coordinates, normal vectors, semantic tags, and instance tags into tensor format, combine and encapsulate them, and then serialize them into binary files for storage, with each plant as a unit.

[0030] Preferably, step S4 further includes constructing a corresponding relationship table according to structured binary encoding rules; the data cleaning in step S51 includes: merging multiple point cloud parts of the same broken instance, separating point clouds of different instances that are incorrectly connected, and removing extremely small point cloud clusters with too few points and not representative.

[0031] Preferably, step S4 further includes constructing a correspondence table according to structured binary encoding rules; step S53 specifically includes: calculating the geometric center point of all spatial coordinates of the point cloud, and translating the entire point cloud to a reference system with the center point as the origin, that is: P center = P raw -C;

[0032] In the formula, P center The coordinates of the centered point cloud; P raw The coordinates of the original point cloud are N×3; C is the geometric center point; the expression for C is as follows:

[0033] ;

[0034] Calculate the absolute maximum value of each coordinate axis component of the point cloud in the translated coordinate system, and use it as the scaling factor. All spatial coordinates are scaled proportionally, ultimately normalizing the spatial distribution range of the entire point cloud to a standardized numerical range, resulting in the normalized point cloud coordinates. All coordinate values ​​are within the range [-1, 1].

[0035] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0036] (1) This invention achieves joint labeling and parsing of pod bursting state and number of seeds through structured binary coding rules. It eliminates the need to maintain multiple labeling systems, solves the problems of information fragmentation and complex management caused by traditional separate labeling, significantly shortens the labeling time of a single plant, and the coding is strongly bound to the traits, avoiding label misalignment, and subsequent parsing is efficient and convenient.

[0037] (2) High quality and standardization of point cloud data: Multi-view acquisition combined with marker point-assisted registration, along with a series of processing such as noise removal, resampling, and normalization, not only solved the problems of missing and spatial misalignment of soybean plant point clouds, but also achieved uniform point cloud scale and elimination of scale differences, providing high-quality and consistent input data for deep learning models, and significantly improving the morphological recognition accuracy compared with two-dimensional image recognition schemes.

[0038] (3) Strong scene adaptability and practicality: It takes into account both potted and field soybean plant scenarios. Through standardized data processing flow and file storage format, it can directly connect to mainstream 3D point cloud recognition models without additional format conversion, and has a wide range of applications.

[0039] (3) Easy to reproduce and low threshold: The core logic of the key processing links (verification and cleaning, resampling and normalization) is clear and quantifiable, which solves the problems of complex operation and ambiguous parameters in the existing 3D point cloud annotation methods, and facilitates the promotion and application of the technology. Attached Figure Description

[0040] Figure 1 This is a flowchart of a soybean pod state joint annotation method based on three-dimensional point cloud provided by an embodiment of the present invention. Detailed Implementation

[0041] In the following description, embodiments of the invention will be described with reference to the accompanying drawings. In the description below, the same modules are denoted by the same reference numerals. Where the same reference numerals are used, their names and functions are also the same. Therefore, their detailed description will not be repeated.

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0043] This invention provides a method for joint annotation of soybean pod state based on 3D point cloud, specifically including the following steps:

[0044] S1. Multi-view data acquisition: Using a 3D scanner, the target soybean plant is scanned from multiple preset scanning stations; at each scanning station, the 3D scanner simultaneously acquires a depth image and a color image from that viewpoint (depth data and color data are acquired synchronously to ensure pixel-level spatial alignment); the depth image records the distance information of each pixel in the scene relative to the scanner, and the color image provides the corresponding texture and color information; during the scanning process, marker points are placed on the plant or its supporting device as a common reference for subsequent registration.

[0045] In some embodiments, the device selection and parameter settings are as follows: Select a 3D scanner (model: FAROFocusS70), set the scanning accuracy to 0.1mm, the dot pitch to 0.5mm, and the scanning distance range to 1.5~3m; Simultaneously enable the color image acquisition function, set the color image resolution to 2048×1536 pixels, and the frame rate to 30fps to ensure pixel-level alignment between the depth image and the color image;

[0046] Use circular reflective markers with a diameter of 5mm (e.g., PET reflective film, reflectivity ≥85%). Place 4 markers on a single plant: 1 at the base of the plant (10cm from the ground on the stem), 2 on different branches in the middle (30-40cm from the base, avoiding the pods), and 1 on the top main stem (5cm from the top). The center of the marker must be flat and attached to the stem to avoid feature extraction errors caused by wrinkles.

[0047] Six scanning stations were evenly distributed in a 360° pattern on the horizontal plane, with the center of the soybean plant as the center (the angle between adjacent stations was 60°), and each station was 1.8m away from the center of the plant. For the distribution of pods in the vertical direction of the plant, each station was scanned in two layers (the lower layer scan height was 0-40cm, and the upper layer scan height was 40-80cm) to ensure that the top and bottom pods were completely covered. The scanning time for each station was 15s. During the scanning process, the scanner was kept still. If the plant shook slightly (amplitude ≤2cm), the station was scanned again once, and the overlap of the two scans was used for subsequent processing.

[0048] S2. Multi-source data registration: Spatial alignment of the acquired multi-view, independent coordinate system depth images; specifically including the following sub-steps:

[0049] S21. Coarse registration: Based on marker points and geometric or appearance feature points extracted from depth or color images, feature matching is performed between different images, and the rigid body transformation matrix between adjacent viewpoint images is estimated to achieve preliminary alignment.

[0050] In some embodiments, the compute_convex_hull convex hull function of the Open3D library is used to extract geometric feature points such as pod vertices and stem bifurcation points in the depth image (20-30 feature points are extracted from each depth image); the Canny edge detection algorithm (threshold: low threshold 50, high threshold 150) is used to extract appearance feature points in the color image (30-40 texture edge points are extracted from each color image); the extracted geometric feature points and appearance feature points are merged into a joint feature point set;

[0051] The marker matching method is as follows: Identify reflective markers (connected regions with grayscale values ​​≥ 240) in each depth image and obtain the three-dimensional coordinates of the four markers; using the coordinates of the markers at the first scanning station as a reference, calculate the Euclidean distance between the markers at the other five scanning stations and the reference marker, filter matching pairs with a distance error ≤ 0.3 mm, and preliminarily determine the positional relationship between adjacent stations;

[0052] The rigid transformation matrix estimation method is as follows: Based on the joint feature point set and the marker point matching results, the Random Sample Consensus (RANSAC) algorithm (1000 iterations, inlier threshold 0.2mm) is used to estimate the rigid transformation matrix (including translation vector and rotation matrix) of the depth images of adjacent stations, and complete the coarse registration. At this time, the error of the overlapping area of ​​the point cloud of adjacent stations is controlled within 1-1.5mm.

[0053] S22. Fine registration and global optimization: The iterative nearest point algorithm is used to refine the coarse registration results; the bundle adjustment method is used to globally optimize the rigid body transformation matrix of all scanning stations to minimize the cumulative error, and finally unify all depth images to the same global coordinate system.

[0054] In some embodiments, the fine-tuning method of the Iterative Closest Point (ICP) algorithm is as follows: taking the coarsely registered point cloud as input, the ICP algorithm is used for fine registration; the iteration termination condition is set: the number of iterations reaches 50 or the average distance error between two adjacent iterations is ≤0.05mm; in each iteration, 1000 random points in the overlapping area are selected as sampling points, the distance error of the corresponding point pairs is calculated, and the rigid body transformation matrix is ​​updated by the least squares method, so that the overlap error of the point clouds of adjacent stations is reduced to within 0.3mm;

[0055] The global optimization method of the bundle adjustment method is as follows: The point clouds of the 6 stations and their corresponding rigid body transformation matrices are input into the bundle adjustment algorithm. The optimization objective is to minimize the sum of squared cumulative distance errors of the point clouds of all stations. The constraint is the consistency of the transformation matrices of adjacent stations (the error between the product of adjacent matrices and the directly calculated cross-station matrix is ​​≤0.02mm). The optimization problem is solved by the Levenberg-Marquardt algorithm, and the globally optimal transformation matrix is ​​output. The depth images of all stations are unified to the same global coordinate system (the origin of the coordinate system is the center of the scanner of the first scanning station).

[0056] S3. Point Cloud Generation and Post-processing: Generate the final 3D point cloud model based on the registered data; specifically, generate the initial color 3D point cloud of soybean plants based on the registered depth image data and color image data, and perform post-processing operations such as noise removal, data simplification and surface smoothing on the initial color 3D point cloud to obtain the optimized 3D point cloud model.

[0057] Specifically, it includes the following sub-steps:

[0058] S31. Color 3D Point Cloud Reconstruction: For each registered depth image, the pixel coordinates are reconstructed using the coordinate back projection principle and based on the camera intrinsic parameters. u , v ) and the depth value is converted to a 3D point in the global coordinate system. X , Y , Z The data is then processed and fused with the (R,G,B) color information of the corresponding color image to synthesize a complete initial color 3D point cloud.

[0059] In some embodiments, the (R, G, B) values ​​(ranging from 0 to 255) of the corresponding color image are read and compared with the three-dimensional spatial points ( X , Y , Z The points are associated one by one to generate an initial colored 3D point cloud of a single plant (approximately 80,000-100,000 points), which is then stored as a .pcd file.

[0060] S32. Post-process the reconstructed initial colored 3D point cloud to obtain the 3D point cloud data of soybean plants; this includes: applying statistical filtering or radius filtering to remove outlier noise points; simplifying the data by using a voxelized mesh downsampling method while maintaining the model shape; and smoothing the point cloud surface using algorithms such as moving least squares. After the above processing, the 3D point cloud data of soybean plants can be obtained.

[0061] In some embodiments, the outlier noise removal method is as follows: a statistical filtering algorithm is used, the number of neighboring points is set to 10, the average distance between each point and its neighboring points is calculated, and points with a distance greater than "average distance + 2 times standard deviation" are identified as outliers (mostly dust and light spots in the scanning environment). After removal, the number of points in the point cloud is retained to about 90%; if there is local dense noise (such as reflective interference points on the surface of the stem), radius filtering is added (points with a search radius of 5mm and a number of neighboring points ≤ 3 are removed).

[0062] The voxel downsampling method is as follows: Set the voxel side length to 1mm, replace all points in the same voxel with the voxel center coordinates, and take the average (R,G,B) of all points in the voxel as the color value of the center coordinates, so that the number of point clouds is reduced to 20,000-25,000, reducing the amount of data while maintaining the pod morphological characteristics (such as pod length and curvature).

[0063] The surface smoothing method is as follows: using the moving least squares method, setting the neighborhood radius to 5mm and the polynomial order to 2, the surface of the downsampled point cloud is fitted to correct the small protrusions on the point cloud surface (such as the burrs on the surface of the bean pod caused by scanning noise), so that the surface roughness of the point cloud is ≤0.1mm, and finally the optimized color 3D point cloud is obtained.

[0064] S4. Joint annotation of pod traits: Set structured binary encoding rules and build a correspondence table; Based on the binary encoding rules and the correspondence table, perform instance segmentation and annotation on the optimized 3D point cloud model: distinguish between three semantic categories: broken pods, non-broken pods, and background, annotate the number of pods in each pod, and assign a binary code that conforms to the encoding rules to each pod instance;

[0065] Structured binary coding is used to integrate the labeled information. The binary code includes at least two key trait information categories: pod bursting state and seed count. Multiple traits are jointly represented through different positions and values ​​of the coding bits. Specifically, CloudCompare software is used to segment and label instances based on the 3D point cloud data of soybean plants, as follows: point clouds of pods in a cracked state are labeled "bursted pods," and point clouds of pods with intact shells and in a closed state are labeled "non-bursted pods." To achieve seed count, each pod point cloud is labeled with "seed count," while the stems and branches of the soybean plant are uniformly labeled as "background." The structured binary coding scheme is used to... The pod traits are jointly labeled. This encoding simultaneously contains two key types of information: pod bursting and number of beans. The label value is a binary number, which is interpreted from right to left as follows: the least significant bit represents the pod bursting trait, with 0 representing non-bursting pods and 1 representing bursting pods; starting from the second bit from the right, it represents the number of beans contained in the pod. Specifically, the index of the position of the 1 in the binary number represents the number of beans; for example, binary 1001 (decimal 9), the third bit is 1, which means three pods, and the least significant bit is 1, which means three pods bursting, and so on. This single-integer multi-shape encoding scheme greatly simplifies the data labeling and management process, and realizes efficient joint labeling of multiple classification attributes (as shown in Table 1).

[0066] Table 1 Binary Encoding Correspondence Table

[0067]

[0068] In some embodiments, open the CloudCompare software, import the point cloud file optimized in step S3, and enable the instance segmentation and annotation function: use the point cloud selection tool (rectangular selection + manual fine-tuning) to select instance objects within a single plant one by one: if the instance object is a "split pod", select all its point clouds and label it as "split pod"; if the instance object is a "closed pod", select all its point clouds and label it as "non-split pod"; if the instance object is a stem, branch, or leaf, select all its point clouds and label it as "background"; during the annotation process, if the same pod is divided into 2 point cloud clusters, merge them into 1 instance using the instance merging function; if different pod point clouds are stuck together, separate them using the point cloud cutting tool (based on a distance threshold of 2mm) to ensure the independence of each instance object.

[0069] S5. Point Cloud Data Processing: The labeled point cloud data is checked and cleaned. Then, the point cloud scale is unified through resampling, and spatial normalization is used to eliminate scale differences. The processed point cloud data is converted into a format suitable for deep learning and serialized into binary files for storage. This includes the following sub-steps:

[0070] S51. Data Verification and Cleaning: Manually verify and clean the original labeled data, removing redundant and erroneous labels; clean the problematic data found during verification; read the cleaned point cloud data file; parse the semantic category and identifier of the instance objects, map the semantic category identifier to an integer label, assign a unique instance identifier, and integrate the spatial coordinates, normal vector, and corresponding label data of a single plant to reconstruct the complete 3D point cloud scene of the plant; the specific operations are as follows:

[0071] S501. Annotation Data Verification and Correction: Manually verify and clean the original annotation data generated by 3D annotation software, removing redundant and erroneous labels; clean the problematic data found during verification: merge multiple point cloud parts of the same broken instance (such as pods or background), separate incorrectly connected point clouds of different instances, and remove extremely small point cloud clusters with too few points or that are not representative (such as fragmented point clouds with ≤50 points).

[0072] S502. Point Cloud Data Reading and Parsing: Reads the cleaned point cloud data file from the specified storage path; the original data is stored in units of a single soybean plant, each containing multiple txt files, and each txt file corresponds to a labeled instance object (including background structures such as pods, stems, and branches); the naming format of each txt file is automatically parsed to extract the semantic category identifier (such as "exploded pod", "non-exploded pod", "background") and instance number identifier of the instance object.

[0073] S503. Data Integration and Reconstruction: Map textual semantic category identifiers to integer numerical labels and assign a unique instance identifier to each instance object; integrate the data of all instance objects of the same plant, including spatial coordinate data (coord), normal vector data (normal), and their corresponding semantic labels (semantic_gt) and instance labels (instance_gt), to reconstruct a complete 3D point cloud scene of a single plant and complete annotation information.

[0074] S52. Resampling: To meet the requirements of deep learning models for consistency in input data scale, the point cloud of each plant after integration is resampled: a predefined target number of points N is set, and a random sampling algorithm is used to adjust the scale of the point cloud data; for point clouds with a total number of points exceeding the target number of points N, random downsampling is performed to uniformly reduce the amount of data; for point clouds with a total number of points less than the target number of points N, all original data points are retained; the sampling process is applied synchronously to spatial coordinates, normal vectors, and all label data to ensure that the correspondence between various types of data remains unchanged after sampling, and to maintain the integrity of point cloud morphological features and annotation structure while unifying the data scale.

[0075] S53. Normalization: Spatial coordinate normalization is performed on the resampled point cloud data to eliminate spatial scale uncertainties introduced by differences in measurement distance, angle, or plant size. This process first calculates the geometric center point of all spatial coordinates of the point cloud and then translates the entire point cloud to a reference frame with this center point as the origin, i.e.: P center = P raw -C;

[0076] In the formula, P center The coordinates of the centered point cloud; P raw The coordinates of the original point cloud are N×3; C is the geometric center point; the expression for C is as follows:

[0077] ;

[0078] Calculate the absolute maximum value of each coordinate axis component of the point cloud in the translated coordinate system, and use this as the scaling factor. All spatial coordinates are scaled proportionally, ultimately normalizing the spatial distribution range of the entire point cloud to a standardized numerical range, resulting in the normalized point cloud coordinates. All of its coordinate values ​​are within the range [-1, 1].

[0079] S54. Format Conversion and Storage: Convert the normalized point cloud data into a format compatible with the deep learning framework: Convert the spatial coordinate data representing the spatial location of the point cloud, the normal vector data representing the surface geometric characteristics, the semantic label data representing the object category, and the instance label data representing the uniqueness of the instance into the corresponding tensor formats respectively; combine and encapsulate each tensor according to the preset structure, and serialize them into binary data files (such as pth format) for storage, with each plant as the unit, to support subsequent model reading and use.

[0080] S6. Pod Characteristics Analysis: Read the binary file stored in S5, extract the binary code corresponding to the pod instance, and quickly parse the pod breakage status (non-breakage / breakage) and number of pods for each pod by querying the pre-set correspondence table in S4 (Table 1).

[0081] The advantages of this invention are: (1) It breaks through the traditional mode of separately labeling the pod breakage state and the number of grains. By pre-setting the structured binary coding rules and the corresponding relationship table, it realizes the integrated operation of "semantic labeling - grain number labeling - binary code allocation" for a single pod instance. It does not require the maintenance of multiple label systems, and the time spent labeling a single soybean pod is shorter than that of the traditional method. At the same time, the coding is strongly bound to the traits, avoiding label misalignment. Subsequent analysis only needs to query the pre-set table to quickly associate the pod breakage state and the number of grains. (2) The multi-view acquisition step sets up marker points and simultaneously acquires depth and color data. Combined with the three-level registration strategy of "coarse registration - fine registration - global optimization", the multi-view point cloud registration error is controlled within 0.3mm, which effectively solves the problem of missing points and spatial misalignment of soybean plants (especially single plants in the field). Subsequent point cloud post-processing (noise removal, data simplification) and pre-processing (resampling, normalization) further ensure data integrity and standardization, providing a high-quality and consistent data foundation for the input of deep learning models. (3) From multi-view data acquisition and point cloud registration generation to joint annotation, data preprocessing, and then to morphology analysis, each step relies on the same set of coding rules, and the logical chain is complete and without redundancy; key parameters (such as scanning accuracy, registration iteration threshold, and number of resampled target points) are clear and quantifiable, solving the problems of ambiguous parameters and high operation threshold in existing 3D point cloud annotation methods.

[0082] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0083] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for joint annotation of soybean pod states based on 3D point clouds, characterized in that: Specifically, the steps include the following: S1. Using a 3D scanning device, the target soybean plant is scanned from multiple preset scanning stations from multiple perspectives. At each scanning station, depth and color images are acquired simultaneously from that perspective. Marker points for spatial reference are placed on the soybean plant. 4-6 marker points are placed on a single soybean plant. The marker points are affixed to the base, different branches in the middle, and the top main stem of the soybean plant, and the marker points do not obstruct the pod area. S2. Spatial alignment of the depth image data, unifying all depth image data to the same global coordinate system; specifically including the following sub-steps: S21. Coarse registration: Based on marker points and geometric feature points extracted from depth images or appearance feature points extracted from color images, feature matching is performed between different images, and the rigid body transformation matrix between adjacent viewpoint images is estimated to achieve preliminary alignment. S22. Fine registration and global optimization: The iterative nearest point algorithm is used to refine the coarse registration results; the bundle adjustment method is used to globally optimize the rigid body transformation matrix of all scanning stations to minimize the cumulative error, and finally unify all depth images to the same global coordinate system. S3. Based on the registered depth image data and color image data, generate an initial color 3D point cloud of soybean plants. Perform post-processing on the initial color 3D point cloud, including noise removal, data simplification, and surface smoothing, to obtain an optimized 3D point cloud model. This includes the following sub-steps: S31. For each registered depth image, the pixel coordinates are calculated based on the camera intrinsic parameters using the principle of coordinate back projection. u , v ) and the depth value is converted to a 3D point in the global coordinate system. X , Y , Z The data is then processed and fused with the (R,G,B) color information of the corresponding color image to synthesize a complete initial color 3D point cloud. S32. For the reconstructed initial color 3D point cloud, apply statistical filtering or radius filtering to remove outlier noise points; use voxelized mesh downsampling method to simplify the data while maintaining the model shape; use moving least squares method to smooth the point cloud surface to obtain the 3D point cloud data of soybean plants; S4. Set structured binary encoding rules, perform instance segmentation and annotation on the optimized 3D point cloud model according to the encoding rules, and assign a binary code that conforms to the encoding rules to each pod instance; S5. The labeled point cloud data is checked and cleaned, and then resampled and spatially normalized to convert the processed point cloud data into a format suitable for deep learning and serialize it into a binary file for storage. S6. Read the binary file, extract the binary code corresponding to the pod instance, and quickly parse to obtain the pod bursting status and number of pods for each pod.

2. The method for joint annotation of soybean pod state based on three-dimensional point cloud according to claim 1, characterized in that: Step S4 further includes constructing a correspondence table according to structured binary encoding rules; the correspondence table contains association information of decimal labels, binary representations, number of grains, and pod bursting status; The encoding rule is as follows: the least significant bit of the binary number represents the pod-exploding state, 0 = no pod-exploding, 1 = pod-exploding; the index of the position of 1 in the second and above bits from the right represents the number of pods. Based on the encoding rules and the corresponding relationship table, the optimized 3D point cloud model is segmented and labeled to distinguish between three semantic categories: popped pods, non-popped pods, and background. The number of grains in each pod is labeled, and a binary code conforming to the encoding rules is assigned to each pod instance.

3. The method for joint annotation of soybean pod state based on three-dimensional point cloud according to claim 2, characterized in that: The specific correspondence table is as follows: 。 4. The method for joint annotation of soybean pod state based on three-dimensional point cloud according to claim 1, characterized in that: Step S5 specifically includes the following sub-steps: S51. Data Verification and Cleaning: Manually verify and clean the original labeled data, removing redundant and incorrect labels; clean the problematic data found during verification; read the cleaned point cloud data file. The semantic category and number identifier of the instance object are parsed, the semantic category identifier is mapped to an integer label, a unique instance identifier is assigned, and the spatial coordinates, normal vector and corresponding label data of a single plant are integrated to reconstruct the three-dimensional point cloud scene of the complete plant. S52. Resampling: Set the number of target points N; use a random sampling algorithm to unify the point cloud scale. When the number of points exceeds N, downsampling is performed. When the number of points is less than N, the original data is retained. During the sampling process, spatial coordinates, normal vectors and label data are synchronously associated. S53. Normalization: Calculate the geometric center point of the point cloud and translate the entire point cloud to a reference system with the center point as the origin; then, through proportional scaling, map all coordinates to the [-1,1] interval to eliminate scale differences. S54. Format Conversion and Storage: Convert the processed spatial coordinates, normal vectors, semantic tags, and instance tags into tensor format, combine and encapsulate them, and then serialize them into binary files for storage, with each plant as a unit.

5. The method for joint annotation of soybean pod state based on three-dimensional point cloud according to claim 4, characterized in that: The data cleaning in step S51 includes: merging multiple point cloud parts of the same broken instance, separating point clouds of different instances that are incorrectly connected, and removing extremely small point cloud clusters that have too few points and are not representative.

6. The method for joint annotation of soybean pod state based on three-dimensional point cloud according to claim 4, characterized in that: Step S53 specifically includes: calculating the geometric center point of all spatial coordinates of the point cloud, and translating the entire point cloud to a reference system with that center point as the origin, i.e.: P center = P raw -C; In the formula, P center The coordinates of the centered point cloud; P raw The coordinates of the original point cloud are N×3; C is the geometric center point; the expression for C is as follows: ； Calculate the absolute maximum value of each coordinate axis component of the point cloud in the translated coordinate system, and use it as the scaling factor. All spatial coordinates are scaled proportionally, ultimately normalizing the spatial distribution range of the entire point cloud to a standardized numerical range, resulting in the normalized point cloud coordinates. All coordinate values ​​are within the range [-1, 1].

Images