A method and system for ear point cloud segmentation based on deep learning and geometric correction

Abstract

This invention discloses a method and system for wheat ear point cloud segmentation based on deep learning and geometric correction. The wheat ear point cloud segmentation method includes: cropping individual wheat point clouds with category labels from a population of wheat point clouds; constructing a wheat sample library using these individual wheat point clouds; selecting wheat sample point clouds from the wheat sample library and planting them in virtual plots to obtain a virtual wheat population point cloud; extracting training point cloud data from the virtual wheat population point cloud and inputting it into a deep neural network to obtain a deep learning model; segmenting the real wheat point cloud using the deep learning model to obtain wheat ear labels; extracting individual wheat ears from the real wheat point cloud using a spatial clustering algorithm based on the wheat ear labels; and extracting the principal axis of each individual wheat ear using a Laplace shrinkage algorithm and performing region growing to obtain a complete wheat ear point cloud. The technical solution of this invention solves the problem that existing data-driven and model-driven methods are not suitable for wheat ear segmentation tasks, and that wheat ear learning is not stable enough.

Description

Technical Field

[0001] This invention relates to the field of lidar technology, and in particular to a method and system for segmenting wheat ear point clouds based on deep learning and geometric correction. Background Technology

[0002] As the reproductive organ of wheat, the ear of wheat is a major limiting factor in the aggregation of photosynthetic products and plays a decisive role in determining maximum wheat yield. Therefore, biologists and breeders have been dedicated to developing wheat varieties with ideal plant type characteristics. One of the most important characteristics of wheat varieties with ideal plant type characteristics is a large, upright ear of wheat. Currently, most field measurements of ear morphology are done manually. However, manual measurement is highly subjective, time-consuming, inefficient, and often destructive. The lack of suitable methods for automatically measuring ear morphology characteristics makes it difficult to reduce reliance on manual measurements, resulting in slow progress in wheat breeding.

[0003] As a primary data source for high-throughput phenotyping, LiDAR possesses rich geometric, shape, and scale information, providing a data source for rapid and efficient measurement of wheat ear morphological features. Organ segmentation is a prerequisite for obtaining wheat ear morphological features based on point cloud data. By segmenting point cloud data using voxel-based and mean-shift methods, wheat ears can be detected in field plots. For example, existing technologies provide the Otsu algorithm for segmenting wheat ear point clouds and use adaptive density estimation to calculate the density of mature rice plants. However, these traditional model-driven point cloud segmentation methods all use driving models based on manually designed feature extractors. Manually designing features requires extensive feature design skills and domain expertise. Using a general automated learning process can avoid these issues, and automated learning is a key advantage of deep learning technology.

[0004] Deep learning technology has shown great promise in point cloud segmentation, but current DNN (Deep Neural Network) models still face several challenges when applied to point cloud segmentation tasks. First, as a data-driven approach, high-quality datasets are crucial for DNN training and testing; poor dataset quality or limited quantity can lead to overfitting and poor classification accuracy. However, collecting and labeling datasets is a tedious and costly task; therefore, quickly acquiring high-quality labeled data is essential for DNN training. Second, as a black-box model, DNN models implicitly learn the geometric features of point clouds, but the complexity of wheat ear data often results in unstable learning. Summary of the Invention

[0005] This invention provides a wheat ear point cloud segmentation scheme based on deep learning and geometric correction, aiming to solve the problem that the existing DNN models have limited performance improvement and always implicitly learn the geometric features of the point cloud, resulting in unstable DNN learning.

[0006] To achieve the above objectives, according to a first aspect of the present invention, the present invention provides a method for wheat ear point cloud segmentation based on deep learning and geometric correction, comprising:

[0007] Individual wheat point clouds with category labels are cropped from the collective wheat point cloud, and a wheat sample library is constructed using the individual wheat point clouds with category labels.

[0008] Select wheat sample point clouds from the wheat sample library, and plant the wheat sample point clouds onto virtual plots using a random transformation method to obtain virtual wheat population point clouds;

[0009] Training point cloud data with category labels is extracted from virtual wheat population point cloud and input into a deep neural network for training and validation to obtain a trained deep learning model.

[0010] A deep learning model was used to segment real wheat point clouds to obtain wheat ear labels;

[0011] Based on the wheat ear labels, individual wheat ears are extracted from the real wheat point cloud using a spatial clustering algorithm; the main axis of the individual wheat ear is extracted using a Laplace shrinkage algorithm, and region growth is performed along the main axis to obtain a complete wheat ear point cloud.

[0012] Preferably, in the above-mentioned wheat ear point cloud segmentation method, the step of cropping individual wheat point clouds with category labels from the population wheat point cloud and constructing a wheat sample library using the individual wheat point clouds with category labels includes:

[0013] Single wheat plant point clouds were extracted from a group of wheat point clouds;

[0014] Wheat ear markers and non-wheat ear markers were obtained by marking single wheat plant point clouds;

[0015] A wheat sample library was constructed using point clouds of single wheat plants, including those with and without ear markers.

[0016] Preferably, in the above wheat ear point cloud segmentation method, the step of planting wheat sample point clouds onto virtual plots using a random transformation method to obtain virtual wheat population point clouds includes:

[0017] Single wheat point clouds with ear-marked structures were selected from the wheat sample library and used as wheat sample point clouds.

[0018] The wheat sample point cloud is randomly raised and lowered within a predetermined distance range, and randomly rotated within a predetermined angle range;

[0019] Plant wheat sample point clouds onto virtual plots;

[0020] Adjust the wheat plant spacing of the virtual plot within a predetermined plant spacing range, and adjust the planting row spacing of the virtual plot within a predetermined row spacing range to obtain a virtual wheat population point cloud.

[0021] Preferably, in the above-mentioned wheat ear point cloud segmentation method, the step of extracting training point cloud data with category labels from the virtual wheat population point cloud and inputting it into a deep neural network for training and validation to obtain a trained deep learning model includes:

[0022] Using a sliding window to sample virtual wheat population point clouds, training point cloud data with and without wheat ear labels were obtained;

[0023] The deep neural network is trained using training point cloud data to obtain the classification labels of the training point cloud data.

[0024] The classification labels are tested using wheat ear markers and non-wheat ear markers in the training point cloud data to obtain the trained deep learning model.

[0025] Preferably, in the above-mentioned wheat ear point cloud segmentation method, the step of extracting individual wheat ears from the real wheat point cloud based on the wheat ear tags using a spatial clustering algorithm includes:

[0026] Determine the vertical direction of the actual wheat point cloud based on the wheat ear labels;

[0027] By filtering out the portion below the wheat ear along the vertical direction of the real wheat point cloud, we obtain the wheat ear point cloud.

[0028] Spatial clustering algorithm is used to divide the wheat ear point cloud into individual wheat ears.

[0029] Preferably, in the above-mentioned wheat ear point cloud segmentation method, the step of extracting the principal axis of a single wheat ear according to the Laplace shrinkage algorithm and performing region growth along the principal axis to obtain a complete wheat ear point cloud includes:

[0030] The main axis of a single wheat ear is extracted using the Laplace shrinkage algorithm;

[0031] Search for neighboring point clouds within a predetermined radius along the main axis of the wheat ears;

[0032] The neighbor point clouds within a predetermined radius of the wheat ear's main axis are corrected using a region growing algorithm to obtain a complete wheat ear point cloud.

[0033] Preferably, the above-mentioned wheat ear point cloud segmentation method further includes, after obtaining the complete wheat ear point cloud, the following:

[0034] The point cloud of the complete wheat ear was projected onto a two-dimensional plane using principal component analysis to obtain a projected image of the complete wheat ear.

[0035] The length and width of the wheat ear in a complete projected image were measured using a quadratic curve fitting method; and...

[0036] The surface area and volume of a wheat ear are calculated using convex hull in three-dimensional space.

[0037] According to a second aspect of the present invention, the present invention also provides a wheat ear point cloud segmentation system based on deep learning and geometric correction, comprising:

[0038] The wheat sample library construction module is used to cut out individual wheat point clouds with category labels from the wheat population point cloud, and to construct a wheat sample library using the individual wheat point clouds with category labels.

[0039] The wheat point cloud planting module is used to extract wheat sample point clouds from the wheat sample library and plant the wheat sample point clouds onto virtual plots using a random transformation method to obtain virtual wheat population point clouds.

[0040] The network model training module is used to extract training point cloud data with category labels from the virtual wheat population point cloud, input it into the deep neural network for training and validation, and obtain the trained deep learning model.

[0041] The wheat point cloud segmentation module is used to segment real wheat point cloud data using a deep learning model to obtain wheat ear labels;

[0042] The single wheat ear extraction module is used to extract single wheat ears from real wheat point clouds based on wheat ear tags and spatial clustering algorithms.

[0043] The complete wheat ear acquisition module is used to extract the main axis of a single wheat ear according to the Laplace shrinkage algorithm, and perform regional growth along the main axis to obtain a complete wheat ear point cloud.

[0044] Preferably, in the above-mentioned wheat ear point cloud segmentation system, the network model training module includes:

[0045] The population point cloud sampling submodule is used to sample virtual wheat population point clouds using a sliding window to obtain training point cloud data with and without wheat ear labels;

[0046] The neural network training submodule is used to train a deep neural network using training point cloud data to obtain the classification labels of the training point cloud data.

[0047] The classification label verification submodule is used to verify the classification labels using wheat ear markers and non-wheat ear markers in the training point cloud data in order to obtain the trained deep learning model.

[0048] The wheat ear label assignment submodule is used to assign wheat ear labels to the sampled points of the virtual wheat population point cloud using the nearest neighbor interpolation method based on the classification labels obtained from the deep learning model.

[0049] Preferably, in the above-mentioned wheat ear point cloud segmentation system, the complete wheat ear acquisition module includes:

[0050] The wheat ear spindle extraction submodule is used to extract the wheat ear spindle of a single wheat plant using the Laplace shrinkage algorithm;

[0051] The neighbor point cloud search submodule is used to search for the number of neighbor point clouds within a predetermined radius along the main axis of the wheat ear;

[0052] The neighbor point cloud correction submodule is used to perform category correction on the neighbor point cloud within a predetermined radius of the wheat ear main axis using a region growing algorithm to obtain a complete wheat ear point cloud.

[0053] In summary, the wheat ear point cloud segmentation scheme based on deep learning and geometric correction described above in this invention obtains a population of wheat point clouds, then extracts individual wheat point clouds with category labels from these populations. This allows for the construction of a wheat sample library using these labeled individual wheat point clouds, as the wheat samples in the library are constructed using individual wheat point clouds from each population, resulting in high similarity among the wheat samples. Then, the wheat sample point clouds from the wheat sample library are randomly transformed and planted onto virtual plots, thus obtaining a virtual wheat population point cloud. Since the virtual wheat population point cloud uses the same wheat sample... The point clouds are obtained through random transformations, resulting in highly similar wheat population point clouds with similar category labels. Inputting these virtual wheat population point clouds into a deep neural network (DNN) significantly improves the recognition performance of the DNN model, leading to a deep learning model with excellent segmentation performance. Using this deep learning model to segment real wheat point clouds and obtain ear labels, individual wheat ears can be extracted from the real wheat population point clouds using spatial clustering algorithms. The principal axis of each individual ear can then be extracted using the Laplace shrinkage algorithm, and region growing along this axis yields a complete wheat ear point cloud. This ensures that the complete wheat ear point cloud possesses complete explicit geometric features, guaranteeing stable learning performance of the DNN model. In summary, the above-described method of this application enables the construction of datasets based on individual and group wheat plants, meeting the data requirements for wheat ear segmentation tasks using DNNs. Furthermore, geometric correction optimizes the DNN results, combining the advantages of data-driven and model-driven approaches to achieve excellent point cloud segmentation performance. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0055] Figure 1 This is a flowchart illustrating the first wheat ear point cloud segmentation method based on deep learning and geometric correction provided in this embodiment of the invention.

[0056] Figure 2 This is a schematic diagram of a complete wheat ear clustering point provided in an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of the structure of a wheat ear after category correction by a region growth algorithm, provided in an embodiment of the present invention;

[0058] Figure 4 This is a schematic diagram of a complete wheat ear dot cloud structure provided in an embodiment of the present invention.

[0059] Figure 5 This is a flowchart illustrating the second wheat ear point cloud segmentation method based on deep learning and geometric correction provided in this embodiment of the invention.

[0060] Figure 6 yes Figure 5 The illustrated embodiment provides a flowchart of a method for cropping point clouds of a single wheat plant.

[0061] Figure 7 yes Figure 5 The illustrated embodiment provides a flowchart of a method for obtaining virtual wheat population point clouds;

[0062] Figure 8 yes Figure 5 The illustrated embodiment provides a flowchart of a method for obtaining wheat ear tags;

[0063] Figure 9 yes Figure 5 The illustrated embodiment provides a flowchart of a method for determining wheat ear cluster points;

[0064] Figure 10 yes Figure 5 The illustrated embodiment provides a flowchart of a method for obtaining a complete wheat ear point cloud;

[0065] Figure 11 This is a flowchart illustrating the second wheat ear point cloud segmentation method based on deep learning and geometric correction provided in this embodiment of the invention.

[0066] Figure 12 This is a schematic diagram of the structure of the first wheat ear point cloud segmentation system based on deep learning and geometric correction provided in the embodiments of the present invention;

[0067] Figure 13 yes Figure 12 The illustrated embodiment provides a schematic diagram of the structure of a network model training module;

[0068] Figure 14 yes Figure 12 The illustrated embodiment provides a structural schematic diagram of a wheat ear point cloud segmentation module;

[0069] Figure 15 This is a schematic diagram of the structure of the second type of wheat ear point cloud segmentation system provided in the embodiments of the present invention.

[0070] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0071] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0072] The main technical problem solved by the embodiments of the present invention is:

[0073] Deep learning technology has shown great promise in point cloud segmentation, but current DNN (Deep Neural Network) models still face several challenges when applied to point cloud segmentation tasks. First, as a data-driven approach, high-quality datasets significantly impact DNN training and testing. Poor dataset quality or limited quantity can lead to overfitting and low classification accuracy. However, collecting and labeling datasets is a tedious and costly task; therefore, quickly acquiring high-quality labeled data is crucial for DNN training. Data augmentation is a common method for data collection and labeling, but the reconstructed data is inherently highly similar to the original data, limiting the performance improvement of DNN models. Therefore, it is necessary to generate highly similar but entirely new labeled data. Second, as a black-box model, DNN models implicitly learn the geometric features of point clouds. However, due to the complexity of wheat ear data and the insufficient learning ability of DNN models, their learning is often unstable.

[0074] To address the aforementioned problems, the following embodiments of the present invention provide a wheat ear point cloud segmentation scheme based on deep learning and geometric correction. (See also...) Figure 1 As shown, the wheat ear point cloud segmentation scheme specifically includes three parts: breeding data simulation, wheat ear classification, and optimization and segmentation of wheat ear cluster points. Among them,

[0075] In the data simulation section, a wheat sample library (i.e., a point cloud of single wheat plants with category labels) is constructed. Figure 1 The wheat single-plant sample library in the wheat sample library is randomly transformed (e.g., raised or lowered within 15cm and rotated within 360 degrees), and then planted into virtual plots by adjusting the planting density to obtain virtual wheat population point clouds.

[0076] In the wheat ear classification part, virtual wheat cell data, namely the virtual wheat population point cloud mentioned above, is used as input into the deep neural network model. Its core convolutional layer and upper pooling layer are used for feature matching to obtain classification labels (including labels for wheat ear points and non-wheat ear points), thus completing the training of the deep learning model.

[0077] In the optimization and segmentation of wheat ear clustering points, the aforementioned deep learning model is used to segment the real wheat point cloud to obtain wheat ear labels. Then, using these wheat ear labels, a spatial clustering algorithm is used to cluster the real wheat point cloud to obtain wheat ear clustering points, thereby extracting individual wheat ears. The Laplace shrinkage algorithm is then used to extract the main axis of each individual wheat ear. Region growth is performed along the main axis to correct misclassified and missed classification points, resulting in a complete wheat ear point cloud.

[0078] To achieve the above objectives, please refer to Figure 5 , Figure 5 This is a flowchart illustrating the first wheat ear point cloud segmentation method based on deep learning and geometric correction provided in this embodiment of the invention. Figure 5 As shown, this wheat ear point cloud segmentation method based on deep learning and geometric correction includes:

[0079] S110: Obtain the wheat point cloud of the population, crop out the single wheat point cloud with category labels from the wheat point cloud of the population, and use the single wheat point cloud with category labels to construct a wheat sample library. In order to meet the requirement of a large amount of labeled data for training the DNN model, this embodiment of the application crops out the single wheat point cloud from the wheat point cloud of the population, thereby constructing a single wheat sample library.

[0080] Specifically, as a preferred embodiment, such as Figure 6 As shown, the steps of obtaining a wheat population point cloud and cropping out individual wheat plant point clouds with category labels from the wheat population point cloud include:

[0081] S111: Extracting individual wheat stalk point clouds from a group of wheat point clouds.

[0082] S112: Mark wheat ears and non-wheat ears in the point cloud of a single wheat plant.

[0083] S113: Construct a wheat sample library using single-plant wheat point clouds that include both ear-marked and non-ear-marked wheat plants.

[0084] The technical solution provided in this application involves cropping individual wheat point clouds from a wheat population point cloud, obtaining point clouds of individual wheat point clouds, and then loading these point clouds into CloudCompare software. This allows for a clear view of point clouds containing wheat ears and non-wheat ears. Using the labeling function, wheat ear and non-wheat ear labels (collectively referred to as category labels) can be obtained. Thus, a wheat sample library can be constructed using these labeled individual wheat point clouds. Because the wheat sample library is generated from individual wheat point clouds within each wheat population point cloud, it exhibits high similarity and contains entirely new labeled data. Therefore, using this data can improve the performance of the DNN model.

[0085] Figure 5 The wheat ear point cloud segmentation method provided in the illustrated embodiment further includes the following steps after constructing the wheat sample library:

[0086] S120: Select wheat sample point clouds from the wheat sample library and plant them into virtual plots using random transformations to obtain virtual wheat population point clouds. Specifically, the wheat sample point clouds are planted into virtual plots with different planting densities through random transformations (such as elevation and rotation), thus obtaining virtual wheat population point clouds. These virtual wheat population point clouds are all obtained by random transformations of the same wheat sample point cloud, therefore they have high similarity and novel labeled data, which can improve the performance of the DNN model. Furthermore, the density of the virtual plots can be changed when planting them.

[0087] Specifically, as a preferred embodiment, such as Figure 7 As shown, the steps for planting wheat sample point clouds from a wheat sample library onto virtual plots using a random transformation method to obtain a virtual wheat population point cloud include:

[0088] S121: Select single wheat point clouds with ear markers from the wheat sample library as wheat sample point clouds;

[0089] S122: Randomly raise and lower the wheat sample point cloud within a predetermined distance range, and randomly rotate the wheat sample point cloud within a predetermined angle range;

[0090] S123: Plant wheat sample point clouds into virtual plots;

[0091] S124: Adjust the wheat plant spacing of the virtual plot within the predetermined plant spacing range, and adjust the planting row spacing of the virtual plot within the predetermined row spacing range to obtain a virtual wheat population point cloud.

[0092] The technical solution provided in this application selects single wheat point clouds with ear-shaped markers as wheat sample point clouds, thus ensuring high similarity among the wheat sample point clouds. Then, the wheat point clouds are randomly raised and lowered within a predetermined distance range (e.g., within 30cm) and randomly rotated within a predetermined angle range (e.g., 0-360 degrees), resulting in a large number of wheat sample point clouds with different heights and growth directions, which are then planted in a virtual plot. Finally, the spacing between wheat plants in the virtual plot is adjusted within a predetermined plant spacing range (e.g., 10-20cm), and the planting row spacing is changed within a predetermined row spacing range (e.g., 10-30cm) to control the planting density, thereby obtaining a virtual wheat population point cloud. Combined with... Figure 1 As shown, the method involves controlling planting density by varying wheat plant spacing within a 10-20 cm range and row spacing within a 10-30 cm range, and controlling random variations by randomly raising and lowering plants within a 10-30 cm range and randomly rotating plants within a 0-360° range. These individuals exhibit significant differences in ear shape (upright or curved), ear number (2-6), leaf-to-ear ratio (0.15-0.30), and plant structure (compact or loose). For example, a training dataset consisting of 17 individual wheat plants and 153 wheat populations (i.e., wheat plots) was created.

[0093] Figure 5 The wheat ear point cloud segmentation method provided in the illustrated embodiment further includes the following steps after obtaining the virtual wheat population point cloud:

[0094] S130: Extract training point cloud data with category labels from the virtual wheat population point cloud, input it into a deep neural network for training and validation, and obtain a trained deep learning model. The virtual wheat population point clouds exhibit high similarity; therefore, inputting them into a deep neural network (DNN) for training can improve the DNN model's recognition performance, thereby obtaining highly accurate wheat ear labels for the sampled points in the virtual wheat population point cloud.

[0095] Specifically, as a preferred embodiment, such as Figure 8 As shown in the embodiments of this application, the steps of extracting training point cloud data with category labels from virtual wheat population point clouds, inputting it into a deep neural network for training and validation, and obtaining a trained deep learning model include:

[0096] S131: Use a sliding window to sample virtual wheat population point clouds to obtain training point cloud data with and without wheat ear labels.

[0097] The purpose of using a sliding window is to segment the point cloud of a wheat patch, dividing it into blocks for training. Otherwise, if the data from a single wheat patch is too large, it will cause hardware limitations, GPU memory explosion, and prevent the model from training. After the deep neural network model is trained, it is necessary to stitch together the wheat point clouds cropped from different sliding windows belonging to the same patch.

[0098] S132: Use training point cloud data to train a deep neural network model and obtain the classification labels of the training point cloud data.

[0099] S133: Test the classification labels based on the wheat ear markers and non-wheat ear markers in the training point cloud data to obtain the trained deep learning model.

[0100] In the technical solution provided in this application, the deep neural network model samples the virtual wheat population point cloud using a sliding window sampling method before training, which can improve the training efficiency of the model. Combined with... Figure 1 As shown, for each wheat patch (i.e., the virtual wheat patch data mentioned above), a fixed-size sliding window (with length, width, and height of 0.25m, 0.25m, and 0.3m respectively) is used for sampling, and the sampled wheat point cloud data is used as training data. Then, a deep neural network (DNN) model is selected, such as the Kernel Point Convolution (Thomas et al., 2019) provided by Torch-Points3d (Chaton et al., 2020) as the deep neural network (DNN) model. All parameters are set to default values ​​and trained using the above training data. The classification labels are verified by the original category labels (wheat ear labels and non-wheat ear labels) of the wheat point cloud, thus obtaining the trained deep learning model.

[0101] Figure 5 The wheat ear point cloud segmentation method provided in the illustrated embodiment further includes the following steps after obtaining the trained deep learning model:

[0102] S140: Use a deep learning model to segment real wheat point clouds and obtain wheat ear labels.

[0103] After the DNN model assigns classification labels (ear labels and non-ear labels) to the real wheat point cloud, nearest neighbor interpolation is used to assign labels to the sampled points. A large portion of the sampled points in the real wheat point cloud are not assigned classification labels; therefore, assigning ear labels to all sampled points in the virtual wheat point cloud using nearest neighbor interpolation yields the labels for all ears of wheat in the entire real wheat point cloud. Finally, the ear point clouds of the same wheat patch sampled in a fixed window are merged back into the original patch before pruning. Because the real wheat point cloud was obtained using sliding window sampling, which is equivalent to pruning, it is necessary to merge the ear point clouds of the same wheat patch back into the original patch before pruning to obtain complete wheat patches with ear labels.

[0104] Figure 5 The wheat ear point cloud segmentation method provided in the illustrated embodiment further includes the following steps after obtaining the wheat ear labels:

[0105] S150: Extract individual wheat ears from real wheat point clouds using spatial clustering algorithms based on wheat ear labels.

[0106] Specifically, based on the wheat ear labels at the sampled points, wheat ear clusters are determined in the virtual wheat population point cloud. Combined with... Figure 2 As shown, after obtaining the initial results of the DNN, it is assumed that no wheat ears will appear in the lower part of the wheat patch point cloud. Therefore, by extracting certain outliers that deviate from this assumption based on the distribution of points along the Z-axis, we can obtain the wheat ear clusters in the virtual wheat population point cloud. The assumption that wheat ears never grow at the bottom and always grow at the top of the wheat is unrealistic and constitutes a classification error. Therefore, we need to correct this classification error; the aforementioned outliers represent these misclassifications.

[0107] As a preferred embodiment, such as Figure 9 As shown, the steps for extracting individual wheat ears from a real wheat point cloud using a spatial clustering algorithm based on wheat ear labels include:

[0108] S151: Determine the vertical direction of the actual wheat point cloud based on the wheat ear labels.

[0109] S152: Filter out the part below the wheat ear along the vertical direction of the real wheat point cloud to obtain the wheat ear point cloud.

[0110] S153: Use a spatial clustering algorithm to cluster the wheat ear point cloud and divide the wheat ear point cloud into individual wheat ears.

[0111] The technical solution provided in this application determines the vertical direction (Z-axis direction) of the real wheat point cloud based on the wheat ear labels of the sampled points. By filtering out the lower part of the wheat ears along this vertical direction, the wheat ear point cloud can be obtained. Clustering this point cloud yields complete wheat ear clusters with stalks removed. Finally, density-based spatial clustering (DBCAN) is used to cluster the wheat ear point cloud, resulting in point clouds of individual wheat ears. This method, combining geometric correction and a deep neural network model, produces accurate and complete wheat ear point clouds.

[0112] Figure 5 The wheat ear point cloud segmentation method provided in the illustrated embodiment further includes the following steps after determining the wheat ear cluster points in the virtual wheat population point cloud:

[0113] S160: Extract the principal axis of a single wheat ear using the Laplace shrinkage algorithm, and perform region growth along the principal axis to obtain a complete wheat ear point cloud. Combined with... Figure 2 As shown in the embodiment, the above-mentioned wheat ear cluster points are defined as seed points. In this embodiment, the Laplace shrinkage algorithm is used to extract the principal axis of each wheat ear cluster point. The number of neighbors within a fixed radius range (0.01m) along the principal axis is searched. The results are corrected and a threshold is determined by the region growth method. After reaching the threshold, the planting growth conditions are determined, thereby obtaining a complete wheat ear.

[0114] Specifically, as a preferred embodiment, such as Figure 10 As shown, the steps for obtaining the complete wheat ear point cloud by performing a neighbor search on the wheat ear cluster points according to the Laplace shrinkage algorithm include:

[0115] S161: Extract the main axis of a single wheat ear using the Laplace shrinkage algorithm.

[0116] S162: Search for neighboring point clouds within a predetermined radius along the main axis of the wheat ears.

[0117] S163: The neighbor point cloud within a predetermined radius of the wheat ear main axis is corrected by the region growing algorithm to obtain the complete wheat ear point cloud.

[0118] The technical solution provided in this application defines these initial wheat ear cluster points as seed points, extracts the principal axis of each wheat ear cluster point using the Laplace shrinkage algorithm, searches for the number of neighboring point clouds within a fixed radius (0.01 m) along the principal axis, corrects the classification results (wheat ear or non-wheat ear) through region growing, and determines the termination condition for growing by combining a threshold, thereby obtaining a complete wheat ear point cloud (specifically combined with...). Figure 2As shown, the DNN obtains the initial wheat ear cluster point structure for classification. Noise is removed based on the distribution of cluster points. The results are then clustered. Laplace shrinkage is performed on each cluster point to obtain the principal axis. Region growth is performed along the principal axis to correct missed classification errors. The region growth algorithm is stopped when a threshold is determined.

[0119] In addition, as a preferred embodiment, such as Figure 11 As shown, the above wheat ear point cloud segmentation method further includes the following after step S160: obtaining the complete wheat ear point cloud:

[0120] S170: Use principal component analysis to project the complete wheat ear point cloud onto a two-dimensional plane to obtain a complete wheat ear projection image;

[0121] S180: Measure the length and width of a wheat ear in a complete wheat ear projection image using the quadratic curve fitting method;

[0122] S190: Calculate the surface area and volume of the wheat ear using the reconstructed convex hull in three-dimensional space.

[0123] Combination Figure 3 As shown in the wheat ear image, after obtaining a complete wheat ear instance (i.e., the single wheat ear point cloud mentioned above), principal component analysis (PCA) is used to project the wheat ear points onto a two-dimensional (2D) plane, thus obtaining a complete wheat ear projection image. Figure 4 As shown, a quadratic curve is fitted to measure the length and width of the wheat ear in the projected image of the complete wheat ear, and the reconstructed convex hull is used to calculate the surface area and volume of the wheat ear. After obtaining the length and width of the wheat ear, and calculating the surface area and volume, applications such as predicting wheat yield can be realized based on the above data.

[0124] In summary, the wheat ear point cloud segmentation method based on deep learning and geometric correction provided in this invention obtains a population wheat point cloud, then cuts out individual wheat plant point clouds from it. This allows the construction of a wheat sample library using these individual plant point clouds, as the wheat samples in the library are constructed using the individual wheat plant point clouds from each population, resulting in high similarity among the wheat samples. Then, the wheat sample point clouds from the wheat sample library are randomly transformed and planted onto virtual plots, thus obtaining a virtual wheat population point cloud. Because the virtual wheat population point cloud is obtained by randomly transforming the same wheat sample point cloud, it also exhibits high similarity and similarity among the wheat population point clouds. By using nearby labels to input the virtual wheat population point cloud into a deep neural network model, the recognition performance of the DNN model can be greatly improved, thus obtaining the ear labels of the sampled points in the virtual wheat population point cloud. Then, using these ear labels, the ear clusters in the virtual wheat population point cloud are determined. The Laplace shrinkage algorithm is then used to perform a neighbor search on these clusters, resulting in a complete ear point cloud. This ensures that the ear point cloud has complete explicit geometric features, guaranteeing the stable learning performance of the DNN model. Finally, a spatial clustering algorithm is used to segment the complete ear point cloud, yielding complete individual ear point clouds with precise shapes. In summary, the above method of this application can construct datasets based on individual and group wheat plants, meeting the data requirements of DNN for ear segmentation tasks. Furthermore, by optimizing the DNN results through geometric correction, combining the advantages of data-driven and model-driven approaches, and abandoning the existing end-to-end point cloud segmentation method, excellent point cloud segmentation performance is achieved.

[0125] In addition, based on the same concept of the above method embodiments, the present invention also provides a wheat ear point cloud segmentation system based on deep learning and geometric correction, which is used to implement the above method of the present invention. Since the principle and method of solving the problem in this system embodiment are similar, it has at least all the beneficial effects brought about by the technical solutions of the above embodiments, and will not be described in detail here.

[0126] See Figure 12 , Figure 12 A schematic diagram of a wheat ear point cloud segmentation system based on deep learning and geometric correction provided in an embodiment of the present invention; as shown. Figure 12 As shown, the wheat ear point cloud segmentation system includes:

[0127] The wheat sample library construction module 110 is used to acquire a population wheat point cloud, crop out individual wheat point clouds with category labels from the population wheat point cloud, and construct a wheat sample library using the individual wheat point clouds with category labels.

[0128] The wheat point cloud planting module 120 is used to extract wheat sample point clouds from the wheat sample library and plant the wheat sample point clouds in the wheat sample library onto virtual plots using a random transformation method to obtain virtual wheat population point clouds.

[0129] The network model training module 130 is used to extract training point cloud data with category labels from the virtual wheat population point cloud, input it into the deep neural network for training and testing, and obtain the trained deep learning model.

[0130] The wheat point cloud segmentation module 140 is used to segment real wheat point cloud data using a deep learning model to obtain wheat ear labels;

[0131] The single wheat ear extraction module 150 extracts single wheat ears from real wheat point clouds based on wheat ear tags and a spatial clustering algorithm.

[0132] The complete wheat ear acquisition module 160 is used to extract the main axis of a single wheat ear according to the Laplace shrinkage algorithm, and perform regional growth along the main axis to obtain a complete wheat ear point cloud.

[0133] In summary, the wheat ear point cloud segmentation system based on deep learning and geometric correction provided in this invention obtains a group of wheat point clouds, then cuts out individual wheat point clouds from these group point clouds. This allows the construction of a wheat sample library using these individual wheat point clouds, as the wheat samples in the library are constructed using the individual wheat point clouds from each group, resulting in high similarity among the wheat samples. Then, the wheat sample point clouds from the wheat sample library are randomly transformed and planted onto virtual plots, thus obtaining a virtual wheat population point cloud. Because the virtual wheat population point cloud is obtained by randomly transforming the same wheat sample point cloud, it also exhibits high similarity and similarity among the wheat population point clouds. By using nearby labels to input the virtual wheat population point cloud into a deep neural network model, the recognition performance of the DNN model can be greatly improved, thus obtaining the ear labels of the sampled points in the virtual wheat population point cloud. Then, using these ear labels, the ear clusters in the virtual wheat population point cloud are determined. The Laplace shrinkage algorithm is then used to perform a neighbor search on these clusters, resulting in a complete ear point cloud. This ensures that the ear point cloud has complete explicit geometric features, guaranteeing the stable learning performance of the DNN model. Finally, a spatial clustering algorithm is used to segment the complete ear point cloud, yielding complete individual ear point clouds with precise shapes. In summary, the above method of this application can construct datasets based on individual and group wheat plants, meeting the data requirements of DNN for ear segmentation tasks. Furthermore, by optimizing the DNN results through geometric correction, and combining the advantages of data-driven and model-driven approaches, excellent point cloud segmentation performance is achieved.

[0134] As a preferred embodiment, such as Figure 13 As shown, in the above wheat ear point cloud segmentation system, the network model training module 130 includes:

[0135] The population point cloud sampling submodule 131 is used to sample virtual wheat population point clouds using a sliding window to obtain training point cloud data with and without wheat ear labels;

[0136] The neural network training submodule 132 is used to train the deep neural network model using training point cloud data to obtain the classification labels of the training point cloud data.

[0137] The classification label testing submodule 133 is used to test the classification labels using wheat ear markers and non-wheat ear markers in the training point cloud data in order to obtain the trained deep learning model.

[0138] The wheat ear label allocation submodule 134 is used to obtain classification labels based on the deep learning model and assign wheat ear labels to the sampled points of the real wheat point cloud using the nearest neighbor interpolation method.

[0139] As a preferred embodiment, such as Figure 14 As shown, in the above wheat ear point cloud segmentation system, the complete wheat ear acquisition module 160 includes:

[0140] The wheat ear main axis extraction submodule 161 is used to extract the wheat ear main axis of a single wheat plant using the Laplace shrinkage algorithm;

[0141] The neighbor point cloud search submodule 162 is used to search for neighbor point clouds within a predetermined radius along the main axis of the wheat ear;

[0142] The neighbor point cloud correction submodule 163 is used to correct the neighbor point cloud within a predetermined radius of the main axis of the wheat ear using a region growing algorithm to obtain a complete wheat ear point cloud.

[0143] In addition, the aforementioned wheat sample library construction module is specifically used to: crop out individual wheat point clouds from the population wheat point cloud; mark the individual wheat point clouds to obtain ear markers and non-ear markers; and use the individual wheat point clouds including ear markers and non-ear markers to construct a wheat sample library.

[0144] In addition, the aforementioned wheat point cloud planting module is specifically used to select single wheat point clouds with ear markers from the wheat sample library as wheat sample point clouds; randomly raise and lower the wheat sample point clouds within a predetermined distance range and randomly rotate the wheat sample point clouds within a predetermined angle range; plant the wheat sample point clouds in virtual plots; adjust the wheat plant spacing of the virtual plots within a predetermined plant spacing range and adjust the planting row spacing of the virtual plots within a predetermined row spacing range to obtain virtual wheat population point clouds.

[0145] In addition, the aforementioned single-plant wheat extraction module is also used to determine the vertical direction of the real wheat point cloud based on the wheat ear label; filter out the part below the wheat ear along the vertical direction of the real wheat point cloud to obtain the wheat ear point cloud; and use a spatial clustering algorithm to divide the wheat ear point cloud into single wheat ears.

[0146] In addition, such as Figure 15 As shown, the above wheat ear point cloud segmentation system also includes:

[0147] The wheat ear point cloud projection submodule 170 is used to project the complete wheat ear point cloud onto a two-dimensional plane using principal component analysis to obtain a complete wheat ear projection image.

[0148] The wheat ear projection measurement submodule 180 is used to measure the length and width of the wheat ear in a complete wheat ear projection image using a quadratic curve fitting method; and...

[0149] The wheat ear area calculation submodule 190 is used to calculate the surface area and volume of wheat ears in three-dimensional space using convex hull.

[0150] In summary, the wheat ear point cloud segmentation scheme based on deep neural networks and geometric correction (DNN-GEO) provided in the above embodiments of this application has the following advantages compared with existing technologies:

[0151] (1) This method designs a new training data generation method and constructs a dataset based on individual and group wheat plants to meet the data requirements of the DNN-based wheat ear segmentation task.

[0152] (2) The results of DNN are optimized by geometric correction, which combines the advantages of data-driven and model-driven methods and abandons the previous approach of focusing solely on end-to-end point cloud segmentation.

[0153] (3) This DNN-GEO-based method has achieved good performance in the ear segmentation task. This is the first work to use high-throughput data (LiDAR data) for wheat organ segmentation and phenotypic trait extraction. Therefore, all the phenotypic traits of the wheat ears extracted here are significantly related to yield, and there are large differences among varieties. These parameters can be used to predict yield.

[0154] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0158] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0159] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0160] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for segmenting wheat ear point clouds based on deep learning and geometric correction, characterized in that, include: Individual wheat point clouds with category labels are cropped from the collective wheat point cloud, and a wheat sample library is constructed using the individual wheat point clouds with category labels. Select wheat sample point clouds from the wheat sample library, and plant the wheat sample point clouds onto virtual plots using a random transformation method to obtain virtual wheat population point clouds; Training point cloud data with the category label is extracted from the virtual wheat population point cloud and input into a deep neural network for training and validation to obtain a trained deep learning model. The deep learning model is used to segment real wheat point clouds to obtain wheat ear labels; Based on the wheat ear tags, single wheat ears are extracted from the real wheat point cloud using a spatial clustering algorithm; The main axis of the single wheat ear is extracted using the Laplace shrinkage algorithm, and regional growth is performed along the main axis to obtain a complete wheat ear point cloud.

2. The wheat ear point cloud segmentation method according to claim 1, characterized in that, The step of cropping individual wheat point clouds with category labels from a group of wheat point clouds and constructing a wheat sample library using these category-labeled individual wheat point clouds includes: Individual wheat plant point clouds were cut out from the aforementioned wheat point cloud population; Wheat ear markers and non-wheat ear markers were obtained by marking the single wheat plant point cloud; The wheat sample library is constructed using the single-plant wheat point cloud that includes the wheat ear markers and non-wheat ear markers.

3. The wheat ear point cloud segmentation method according to claim 1, characterized in that, The step of planting the wheat sample point cloud onto virtual plots using a random transformation method to obtain a virtual wheat population point cloud includes: Single wheat point clouds with ear-marked plants are selected from the wheat sample library as the wheat sample point clouds; The wheat sample point cloud is randomly raised and lowered within a predetermined distance range, and randomly rotated within a predetermined angle range; The wheat sample point cloud is planted into the virtual plot; The wheat plant spacing of the virtual plot is adjusted within a predetermined plant spacing range, and the planting row spacing of the virtual plot is adjusted within a predetermined row spacing range to obtain the virtual wheat population point cloud.

4. The wheat ear point cloud segmentation method according to claim 2, characterized in that, The step of extracting training point cloud data with the category label from the virtual wheat population point cloud, inputting it into a deep neural network for training and validation, and obtaining a trained deep learning model includes: The virtual wheat population point cloud is sampled using a sliding window to obtain training point cloud data with and without wheat ear labels; The deep neural network is trained using the training point cloud data to obtain the classification labels of the training point cloud data; The classification labels are validated using wheat ear markers and non-wheat ear markers in the training point cloud data to obtain the trained deep learning model.

5. The wheat ear point cloud segmentation method according to claim 1, characterized in that, The step of extracting individual wheat ears from the real wheat point cloud based on the wheat ear tags using a spatial clustering algorithm includes: Based on the wheat ear labels, determine the vertical direction of the actual wheat point cloud; The portion below the wheat ear is filtered out along the vertical direction of the real wheat point cloud to obtain the wheat ear point cloud; The point cloud of wheat ears was divided into individual wheat ears using a spatial clustering algorithm.

6. The wheat ear point cloud segmentation method according to claim 1, characterized in that, The steps of extracting the principal axis of the single wheat ear according to the Laplace shrinkage algorithm, and performing region growth along the principal axis to obtain a complete wheat ear point cloud include: The main axis of the individual wheat ear was extracted using the Laplace shrinkage algorithm; Search for neighboring point clouds within a predetermined radius along the main axis of the wheat ear; The complete wheat ear point cloud is obtained by classifying the neighbor point clouds within a predetermined radius of the wheat ear main axis using a region growing algorithm.

7. The wheat ear point cloud segmentation method according to claim 1, characterized in that, After obtaining the complete wheat ear point cloud, the method further includes: The point cloud of the complete wheat ear was projected onto a two-dimensional plane using principal component analysis to obtain a projected image of the complete wheat ear. The length and width of the wheat ear in the complete wheat ear projection image were measured using a quadratic curve fitting method; and... The surface area and volume of the wheat ear are calculated using convex hull in three-dimensional space.

8. A wheat ear point cloud segmentation system based on deep learning and geometric correction, characterized in that, include: A wheat sample library construction module is used to cut out individual wheat point clouds with category labels from a population of wheat point clouds, and to construct a wheat sample library using the individual wheat point clouds with category labels. The wheat point cloud planting module is used to extract wheat sample point clouds from the wheat sample library and plant the wheat sample point clouds onto virtual plots using a random transformation method to obtain virtual wheat population point clouds. The network model training module is used to extract training point cloud data with the category label from the virtual wheat population point cloud, input it into the deep neural network for training and verification, and obtain the trained deep learning model. The wheat point cloud segmentation module is used to segment real wheat point cloud data using the deep learning model to obtain wheat ear labels; The single wheat ear extraction module is used to extract single wheat ears from the real wheat point cloud based on the wheat ear tags using a spatial clustering algorithm. The complete wheat ear acquisition module is used to extract the main axis of the single wheat ear according to the Laplace shrinkage algorithm, and perform regional growth along the main axis to obtain a complete wheat ear point cloud.

9. The wheat ear point cloud segmentation system according to claim 8, characterized in that, The network model training module includes: The population point cloud sampling submodule is used to sample the virtual wheat population point cloud using a sliding window to obtain training point cloud data with and without wheat ear labels; The neural network training submodule is used to train the deep neural network using the training point cloud data to obtain the classification labels of the training point cloud data. The classification label verification submodule is used to verify the classification labels using wheat ear markers and non-wheat ear markers in the training point cloud data, so as to obtain the trained deep learning model.

10. The wheat ear point cloud segmentation system according to claim 8, characterized in that, The complete wheat ear acquisition module includes: The wheat ear spindle extraction submodule is used to extract the wheat ear spindle of a single wheat plant using the Laplace shrinkage algorithm; The neighbor point cloud search submodule is used to search for neighbor point clouds within a predetermined radius along the main axis of the wheat ear; The neighbor point cloud correction submodule is used to perform category correction on the neighbor point cloud within a predetermined radius of the wheat ear main axis using a region growing algorithm to obtain the complete wheat ear point cloud.

Images