Unsupervised three-dimensional leaf and spike feature extraction method for crop point cloud image data

Abstract

The invention discloses an unsupervised three-dimensional leaf and spike feature extraction method for crop point cloud image data. According to the method, global similarity loss, intra-class similarity loss and inter-class separation loss are utilized to jointly construct an unsupervised joint loss function, and unsupervised training of a segmentation model is realized. Then, taking an unsupervised weight obtained by unsupervised training as an initialization parameter of a segmentation model in supervised learning fine tuning training, and further performing fine tuning on the initialized segmentation model by using a small number of labeled crop point cloud training samples; and finally, obtaining a model which can be directly used for extracting a three-dimensional segmentation result of the leaf point cloud feature and the spike point cloud feature. The model provided by the invention can reduce the requirements of the model for the training sample and the labeling degree while keeping relatively high segmentation precision. In addition, the method can be suitable for leaf and spike identification and segmentation of wheat and rice crop point clouds at the same time.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image data processing, and particularly relates to a method for unsupervised three-dimensional leaf and ear feature extraction of crop point cloud image data. BACKGROUND

[0002] Leaves and ears are the main source and sink organs of rice and wheat crops, and the composition of leaves and ears is a key factor affecting crop yield formation. Therefore, high-yield plant type design needs to rely on accurate segmentation and quantitative research of plant leaf and ear three-dimensional structure, and research on leaf and ear organ distribution characteristics provides a theoretical basis for crop high-yield technology.

[0003] However, due to the complex three-dimensional structure of crops in the field and the high density of point clouds, the current research often faces the dilemma of difficult labeling and high labeling cost. At the same time, the scarcity of public high-quality crop point cloud datasets restricts the generalization ability of existing crop leaf and ear segmentation algorithms.

[0004] Most existing algorithms are only adapted to a specific network architecture, and their effectiveness often highly depends on the structure design of the algorithm model, lacking sufficient universality and migratability, which greatly limits the effective application of unsupervised learning technology in point cloud semantic segmentation, especially in specific environments such as field scenes. Therefore, how to provide an efficient three-dimensional leaf and ear feature extraction technology for crops in the field is still a research challenge to be overcome. SUMMARY

[0005] The problem to be solved by the present application is that in the field scene, the existing technology cannot clearly obtain the structural features and geometric boundaries of the three-dimensional structure of plant leaves and ears; most existing point cloud image feature extraction technologies are usually optimized only for specific deep networks, and the representation ability and stability of the segmentation model differ significantly under different network structures or different input densities, lacking universality and flexibility, and lacking cross-domain generalization ability; in the leaf and ear segmentation task, due to the complex three-dimensional structure of crops in the field, mutual occlusion of plants, and high density of point clouds, large-scale point cloud labeling is time-consuming and laborious, and manual labeling inevitably leads to mislabeling and missing labeling, which greatly limits the development and application of unsupervised learning models.

[0006] To solve the above technical problems, the technical scheme adopted by the present application is: an unsupervised three-dimensional ear feature extraction method for crop point cloud image data, comprising the following steps: after obtaining the crop point cloud, inputting it into a model obtained by sequentially passing through unsupervised pre-training and supervised learning fine-tuning training, and obtaining a three-dimensional segmentation result of leaf point cloud features and ear point cloud features by processing the model; wherein, the unsupervised pre-training is performed on unannotated crop point cloud training samples, first performing branch point cloud transformation on the training samples to generate at least two unannotated branch point clouds; then extracting semantic features of each unannotated branch point cloud by a segmentation model; inputting the semantic features as an unsupervised joint loss function, adjusting the parameter weight in the segmentation model through the unsupervised joint loss function, and determining a set of unsupervised weights; the supervised learning fine-tuning training initializes the segmentation model according to the unsupervised weights obtained above, uses crop point cloud training samples with annotations as the input of the initialized segmentation model, and uses cross-entropy loss as the loss function to fine-tune the initialized segmentation model, thereby obtaining a model for segmenting leaf point cloud features and ear point cloud features; wherein, the unsupervised joint loss function superimposes three components of global similarity loss, intra-class similarity loss and inter-class separation loss.

[0007] Optionally, the unsupervised three-dimensional ear feature extraction method for crop point cloud image data according to any one of the above, wherein the crop point cloud is obtained by processing two-dimensional images taken at multiple angles, or is obtained by a depth camera, or is obtained by a structured light camera according to crop surface light pattern measurement processing, or is obtained by laser radar scanning; the steps of obtaining the crop point cloud by processing the two-dimensional images taken at multiple angles include: extracting local features from each image respectively, matching the local feature points between multiple images to find the corresponding relationship of different views displayed in different images at the same spatial position; based on the matched feature points, obtaining the three-dimensional coordinates of the feature points matched in the images taken by different cameras through triangulation and bundle adjustment processing, and restoring the scene sparse point cloud; further using the consistency and occlusion relationship of pixel colors in the image to perform pixel-level three-dimensional inference on the scene to obtain a dense point cloud or a surface model; performing coordinate unification, redundancy removal, filtering and hole filling on the dense point cloud corresponding to the images at different angles to obtain the crop point cloud.

[0008] Optionally, the unsupervised three-dimensional ear feature extraction method of crop point cloud image data as claimed in any one of the above, wherein in the unsupervised pre-training, the semantic mutual correspondence between each unlabeled branch point cloud generated by the branch point cloud transformation on the training sample; the specific steps of the branch point cloud transformation on the training sample include: taking the unlabeled crop point cloud training sample as input, selecting a group of anchor points by farthest point sampling, determining the anchor point corresponding to each point in the crop point cloud training sample according to the Euclidean distance, and dividing all points into several local point clouds according to the corresponding anchor points; respectively performing similarity transformation on each local point cloud; performing smoothing interpolation processing on the local point clouds obtained after similarity transformation to generate at least two unlabeled branch point clouds.

[0009] Optionally, the unsupervised three-dimensional ear feature extraction method of crop point cloud image data as claimed in any one of the above, wherein the unsupervised joint loss function is: , wherein, is the unsupervised joint loss function; is a global similarity loss, which first normalizes the semantic features of each unlabeled branch point cloud, then constructs a global similarity loss reflecting the global point-by-point semantic similarity between the normalized semantic features, and constrains the segmentation model to learn the similarity of the global semantic features between each unlabeled branch point cloud by minimizing the global similarity loss; is an intra-class similarity loss, which constructs a neighborhood point group by farthest point sampling, constructs an intra-class similarity loss reflecting the feature distance between points in the neighborhood point group and the redundancy correlation between feature dimensions according to the feature similarity between each intra-class center point and the sample points in the group and the covariance matrix of the feature vectors in the neighborhood point group, and constrains the segmentation model to narrow the points with the same semantic features in the feature space by minimizing the intra-class similarity loss; is an inter-class separation loss, which constructs an inter-class separation loss reflecting the separation degree between the clustering point groups by calculating the ratio of the semantic feature similarity in the clustering point group to the semantic feature difference between different clustering point groups, and constrains the segmentation model to distinguish points with different semantic features by maximizing the inter-class separation loss.

[0010] Optionally, the unsupervised three-dimensional ear feature extraction method of crop point cloud image data as claimed in any one of the above, wherein the global similarity loss function ; wherein, represents the number of points in the unlabeled branch point cloud, , represents the total number of points in a point cloud, the fixed value of 4096 can be taken, and respectively represent the normalized semantic features of different unlabeled branch point clouds, a soft cosine similarity between semantic features, a batch size, may be set to 8.

[0011] Optionally, the unsupervised three-dimensional spike feature extraction method of crop point cloud image data according to any one of the above, wherein the intra-class similarity loss function ; wherein, represents a multiple value adopted by downsampling, may be set to 16, represents a neighborhood point group of one of the branch point clouds, represents a neighborhood point group of the other branch point cloud, represents a given multiple value, may be 200, and represents a cosine similarity between each intra-class center point and each sample point in the group in the two branch point clouds, represents the number of intra-class center points, h = [1, 2, 3, …, H], H represents the total number of intra-class center points, represents a regularization loss between neighborhood point groups, a batch size, may be set to 8.

[0012] Optionally, the unsupervised three-dimensional spike feature extraction method of crop point cloud image data according to any one of the above, wherein the inter-class separation loss function ; wherein, and respectively represent the clustering point groups of the first iteration of the two branch point clouds, represents the number of iterations, represents the inter-class separation degree of the two branch point clouds, represents the regularization loss between the clustering point groups of the first iteration, represents the total number of inter-class center points, represents the number of any inter-class center point, , a batch size, set to 8.

[0013] Optionally, the unsupervised three-dimensional spike feature extraction method of crop point cloud image data according to any one of the above, wherein the segmentation model adopts any one of the following: a dynamic graph convolution segmentation model, a point cloud transformer segmentation model, a point cloud upgrade segmentation model, a point cloud improved segmentation model, a point cloud pure residual multilayer perceptron segmentation model, and a point cloud convolution segmentation model.

[0014] Optionally, the unsupervised three-dimensional ear feature extraction method of crop point cloud image data as any one of the above describes, wherein none of the segmentation models used in the unsupervised pre-training contains a segmentation head; the segmentation models used in the supervised learning fine-tuning training all contain a segmentation head; the segmentation models used in the unsupervised pre-training and the segmentation models used in the supervised learning fine-tuning training all have the same backbone feature extraction network.

[0015] Optionally, the unsupervised three-dimensional ear feature extraction method of crop point cloud image data as any one of the above describes, wherein the proportion of the number of points with labels to the total number of points in all point cloud training samples is 1% to 100%. Beneficial effects

[0016] The unsupervised three-dimensional ear feature extraction method of crop point cloud image data provided in the present application has high computational complexity when performing segmentation and identification on three-dimensional plant ear structures, relies on a large amount of labeled data, and in actual use, the three-dimensional structure of field crops is complex, the point cloud density is high, and it is difficult to label and the training cost is high. The global similarity loss, intra-class similarity loss and inter-class separation loss are used to jointly construct an unsupervised joint loss function to realize the unsupervised training of the segmentation model. Then, the unsupervised weight obtained by the unsupervised training is used as the initialization parameter of the segmentation model in the supervised learning fine-tuning training, and only a small amount of crop point cloud training samples with labels are used to further fine-tune the initialized segmentation model, so as to finally obtain a model that can directly extract three-dimensional segmentation results of leaf point cloud features and ear point cloud features. The model of the present application can reduce the demand for training samples and labeling degree while maintaining high segmentation accuracy. Moreover, it can be simultaneously applied to the identification and segmentation of wheat and rice crop point clouds, has strong generalization ability, and the model of the present application does not need to be limited to a specific network architecture, providing an important reference for unsupervised point cloud semantic segmentation under the condition of small sample and low labeling, and having important significance for accelerating three-dimensional phenotype analysis. BRIEF DESCRIPTION OF DRAWINGS

[0017] The embodiments of the present application will be described in further detail below in combination with the drawings and the embodiments of the present application: Figure 1 The training process of the unsupervised three-dimensional ear feature extraction model for crop point cloud image data provided in the present application is shown in the following figure: Figure 1 A in the figure is a schematic diagram of the training method of the model, Figure 1 B in the figure is a schematic diagram of the calculation process of the global similarity loss used in the present application, Figure 1 C in the figure is a schematic diagram of the calculation process of the intra-class similarity loss used in the present application, Figure 1 D in the figure is a schematic diagram of the calculation process of the inter-class separation loss used in the present application; Figure 2A schematic diagram of the execution process of the branch point cloud transformation used in the present application; Figure 3 A visualization diagram of the qualitative results obtained by the sensitivity analysis of the present application for training samples. DETAILED DESCRIPTION

[0018] For the existing crop leaf ear recognition technology, there are the following problems: (1) It relies on a large amount of labeled data, but the three-dimensional structure of field crops is complex, the point cloud density is high, the labeling work is difficult, and the labeling cost is high; (2) Most of the existing unsupervised point cloud segmentation models are usually specific to a certain segmentation model, and the representation ability and stability of the framework differ significantly under different network structures and different input densities, lacking universality and flexibility; (3) Compared with building or urban scenes, crop point clouds usually lack clear structural and geometric boundaries, and are often accompanied by a large amount of random noise, occluded areas, and point cloud sparsification, which can seriously interfere with the accuracy of unsupervised feature extraction and segmentation. The present embodiment provides an unsupervised three-dimensional leaf ear feature extraction method for crop point cloud image data. After obtaining the crop point cloud by processing the two-dimensional images taken at multiple angles, or after receiving the crop point cloud obtained directly by using a depth camera, a structured light camera, a laser radar or the like, the crop point cloud is input into a model obtained by sequentially pre-training unsupervised and fine-tuning supervised learning, and the three-dimensional segmentation results of the leaf point cloud features and the ear point cloud features can be obtained by processing the crop point cloud through the model.

[0019] The model of the present application can achieve such recognition effect mainly because, in the unsupervised pre-training, for the unlabeled crop point cloud training sample, the training sample is first subjected to branch point cloud transformation to generate at least two unlabeled branch point clouds; then the semantic features of each unlabeled branch point cloud are extracted by the segmentation model; the semantic features are input into the unsupervised joint loss function, the parameter weights in the segmentation model are adjusted by the unsupervised joint loss function, and a set of unsupervised weights are determined. Then, in the supervised learning fine-tuning training, the segmentation model is initialized according to the unsupervised weights obtained above, and the crop point cloud training sample with labels is used as the input of the initialized segmentation model, and the cross-entropy loss is used as the loss function to fine-tune the initialized segmentation model, so as to finally obtain a model capable of effectively segmenting the leaf point cloud features and the ear point cloud features.

[0020] Thus, through the two-stage training steps described above, the present application can construct a suitable unsupervised joint loss function for unsupervised training by superimposing the three components of global similarity loss, intra-class similarity loss, and inter-class separation loss, to realize three-dimensional ear segmentation, thereby reducing the demand for training sample size and labeling degree of the model while maintaining segmentation accuracy. The training method described above can be applied to crops such as rice and wheat whose ear features have relatively significant differences. Moreover, the two-stage training steps described above can reduce the demand for labeled data in the present application. In the training process of the present application, any sample point with more than 1% of the total number of points in the entire point cloud training sample can be labeled according to the convenience of operation. The proportion of labeled points in the entire point cloud training sample does not significantly affect the recognition effect of the present application. Thus, the labeling cost in the model training and use process can be effectively reduced.

[0021] The segmentation model obtained by the training method of the present application does not have any requirements for the acquisition method of crop point cloud data. Whether in the training phase or in the use phase of directly recognizing the acquired crop point cloud data, the crop point cloud obtained by processing two-dimensional images taken from multiple angles or the crop point cloud acquired directly by using a depth camera, a structured light camera, a laser radar, etc. can be selected as the model input.

[0022] Among them, in the way of obtaining crop point cloud by using multi-angle two-dimensional images through multi-view point cloud reconstruction technology, the following processing steps need to be performed on the multi-angle two-dimensional images: Step s1, after taking a large number of scene photos from different angles using a single-lens reflex camera, a drone, etc., feature extraction and matching are performed, local features are extracted for each image, and matching of local feature points between multiple images is performed to find the corresponding relationship of different angles displayed in different images at the same spatial position; Step s2, camera pose estimation, based on the matched feature points, the external parameters of each camera are optimized through algorithms such as triangulation and bundle adjustment, and the three-dimensional coordinates of the feature points matched in the images taken by different cameras are processed to restore the sparse point cloud of the scene; Step s3, dense reconstruction (also known as: Multi-View Stereo, abbreviated as "MVS"), further utilizing the consistency of pixel color and occlusion relationship in the image, pixel-level three-dimensional inference is performed on the scene to obtain a dense point cloud or a surface model; Step s4, point cloud fusion and post-processing, the dense point clouds corresponding to the images under different angles are processed for coordinate unification, redundancy removal, filtering, hole filling, etc. to obtain a complete crop point cloud that can reflect the three-dimensional structure of the plant.

[0023] The above steps can recover the scene structure and generate a three-dimensional point cloud through three-dimensional geometric reasoning based on the two-dimensional images captured from multiple angles. The core idea is to infer the three-dimensional coordinates from the images of the same scene or object under different perspectives through feature matching and spatial geometric constraints, and to realize three-dimensional reconstruction. However, considering the image-based point cloud reconstruction: 1) sensitive to light and material, strong light, shadow, reflection, and low texture areas will cause feature matching to fail, affecting the reconstruction quality; 2) the effect of image-based point cloud reconstruction is poor when there is serious occlusion, and matching is difficult in dense vegetation areas. The structure of vegetation and other structures is repeated, and the texture is weak, which causes unstable feature matching, limited point cloud density and precision; 3) high requirements for camera calibration and pose estimation, high-precision camera calibration and multi-perspective coverage are required, otherwise the spatial coordinate recovery error is large; the crop point cloud can also be obtained by the structure light camera. The structure light camera projects a specific pattern, such as a stripe or dot matrix, onto the surface of the target object, captures the deformed light pattern, and uses triangulation to recover the depth information to obtain a three-dimensional point cloud. The structure light camera has the advantages of small device size, low cost, high imaging resolution, and good detail performance, and is suitable for fine construction of static scenes and small objects. However, it is easily affected by environmental light, and is easily disabled in strong light or outdoor conditions, and the effective ranging range is short, generally only 1-3 meters. Therefore, the depth camera can also be flexibly selected to calculate the depth value of each pixel point by emitting infrared light and measuring the time of flight of the reflected infrared light from the object surface. The depth camera can directly output a dense depth map and output a dense point cloud in real time, which is suitable for dynamic scenes; has the advantages of small device size and easy integration, and is suitable for indoor, robot grabbing, gesture recognition and other applications. However, its ranging range is still limited, generally less than 10 meters, and the depth precision decreases with the increase of distance, and is easily affected by multipath effect, infrared interference, etc. Therefore, the laser radar can also be flexibly used according to the requirements of the use scene to obtain a high-density point cloud by emitting a laser beam and receiving the signal reflected from the target, and calculating the spatial position of each point through the time of flight or phase difference. The laser radar can accurately obtain the three-dimensional structure in a large range and complex environment, and has a long range, up to hundreds of meters. In addition, it has high resolution, strong anti-light interference ability, is not sensitive to object material and surface color, has wide applicability, and is suitable for outdoor, autonomous driving, remote sensing and other large scenes and dynamic occasions.

[0024] In a more preferred implementation, to reduce the demand of the model for the training sample size and the labeling degree, the present application designs an unsupervised semantic segmentation model (also referred to as: Unsupervised semantic segmentation framework, abbreviated as: 3D-USF). Through the two-stage training mode of unsupervised pre-training and supervised fine-tuning, it can realize the three-dimensional segmentation and trait extraction of crop leaf and ear organs. In the first stage, through unsupervised pre-training, it learns the three-dimensional spatial distribution and semantic features of the leaf and ear point cloud using unlabeled training samples. This stage is composed of four core algorithms: branch point cloud transformation, global similarity, intra-class similarity, and inter-class separation degree. In the second stage, the present application uses a supervised learning method to initialize the corresponding parameters of the segmentation model based on the unsupervised weights obtained from unsupervised pre-training. It can be flexibly adjusted according to the use scenario to fine-tune the training on different training sample sizes and different labeling degrees of labeled data, so that the trained model can achieve high-precision segmentation of leaf and ear point clouds. The present application provides an important reference for unsupervised point cloud semantic segmentation under the conditions of small sample and low labeling, and has important significance for accelerating three-dimensional phenotype analysis.

[0025] Referring to Figure 1 As shown in the training steps of the segmentation model of the present application, in the first stage, the present application uses unlabeled crop point cloud data as the input for unsupervised pre-training. Through branch point cloud transformation, the original unlabeled point cloud is generated into at least two unlabeled branch point clouds with corresponding semantics. Then, the semantic features of each branch point cloud are extracted by a segmentation model without a segmentation head. Each semantic feature is used as the input of the unsupervised joint loss function (Formula 1) to optimize the training process of the unsupervised learning stage in the unsupervised semantic segmentation model (3D-USF), and a set of unsupervised weights is obtained. Then, in the second stage, the labeled crop point cloud is used as the input for supervised fine-tuning, and the set of unsupervised weights obtained from unsupervised pre-training is used to initialize the segmentation model. The model with a segmentation head is fine-tuned. In the training process of this stage, the cross-entropy loss is used as the loss function for model training, and the segmented green leaf point cloud and yellow ear point cloud are obtained.

[0026] Specifically, the branch point cloud transformation used in the first stage can refer to the method shown in Figure 2 The following steps are used to generate several unlabeled branch point clouds with corresponding semantics to provide global and local feature comparison supervision signals and enhance the robustness of the unsupervised learning framework to complex point clouds and noisy data: Step c1, taking the unlabeled crop point cloud training sample as input, a set of anchor points is selected by farthest point sampling, and the anchor point corresponding to each point in the crop point cloud training sample is determined according to the Euclidean space distance, and all points are divided into several local point clouds according to the corresponding anchor points; Step c2, respectively, the local point clouds are subjected to similarity transformation such as combination translation, rotation, flip and scaling to generate diversified new training data; thus, the diversity of the generated point cloud and its features is ensured, the model generalization ability is enhanced, and the overfitting risk is reduced, thereby providing diversified feature input for subsequent global and local loss calculation; Step c3, the local point clouds obtained after similarity transformation are subjected to smoothing interpolation processing to generate at least two unlabeled branch point clouds.

[0027] In the above two-stage training, the segmentation model used in unsupervised pre-training and the segmentation model used in supervised learning fine-tuning have the same backbone feature extraction network. The backbone feature extraction network can be flexibly selected as follows according to needs: dynamic graph convolution segmentation model (also known as: Dynamic graph cnn for learning on point clouds, abbreviated as: DGCNN), point cloud transformer segmentation model (also known as: Point Cloud Transformer, abbreviated as: PCT), point cloud upgrade segmentation model (also known as: Deep hierarchical feature learning on point sets in a metric space, abbreviated as: PointNet++), point cloud improvement segmentation model (also known as: Revisiting pointnet++ with improved training and scaling strategies, abbreviated as: PointNeXt), point cloud pure residual multilayer perceptron segmentation model (also known as: pure residual MLP network, abbreviated as: PointMLP), and point cloud convolution segmentation model (also known as: Deep convolutional networks on 3d point clouds, abbreviated as: PointConv).

[0028] The unsupervised joint loss function (also known as: Unsupervised joint loss function, abbreviated as: ) used in the present application is specifically composed of: global similarity loss (also known as: Global similar loss, abbreviated as: ), class-intra loss (also referred to as: class-intra loss, abbreviated as: ) and class-inter separation loss (also referred to as: class-inter loss, abbreviated as: ), the three components together constitute: (Equation 1) Wherein: Global similarity loss , used to normalize the semantic features of each unlabeled branch point cloud first, and then construct a global similarity loss reflecting the global point-by-point semantic similarity between the normalized semantic features, and constrain the segmentation model to learn the similarity of the global semantic features between the branch point clouds by minimizing the global similarity loss; Intra-class similarity loss , used to construct a neighborhood point group by farthest point sampling, and construct an intra-class similarity loss reflecting the feature distance between points in the neighborhood point group and the redundant correlation between feature dimensions according to the feature similarity between each intra-class center point and each sample point in the group and the covariance matrix of the feature vectors in the neighborhood point group, and minimize the intra-class similarity loss to constrain the segmentation model to bring the points with the same semantic features in the feature space closer and minimize the meaningless feature combinations learned by the segmentation model; Class-inter separation loss , used to construct a class-inter separation loss reflecting the separation degree between the clustering point groups by calculating the ratio of the semantic feature similarity in the clustering point group to the semantic feature difference between different clustering point groups, and maximize the class-inter separation loss to constrain the segmentation model to distinguish points with different semantic features; In the above loss function: the point-by-point semantic similarity in the global similarity loss is based on the entire point cloud. The point-by-point semantic similarity in the intra-class similarity loss is calculated in the clustering point group; both are calculated using the same operation to calculate the feature similarity. In the intra-class similarity loss, the clustering point group can be formed by K-Nearest Neighbors (also referred to as: K-Nearnest Neighbors, abbreviated as: KNN) operation; in the class-inter separation loss, the clustering point group can be formed by K-Means++.

[0029] Therefore, the model obtained by the training process described above can reduce the dependence of the model on labeled data and the degree of labeling, and also shows robust segmentation on wheat and rice crop point clouds with obvious differences in spike features compared to the unsupervised method of pseudo-label generation and region growing.

[0030] The following will fully demonstrate the construction and training process of the model of the present application through a specific example.

[0031] The unannotated crop point cloud data is transformed into two unannotated branch point clouds in the following steps to generate two semantically corresponding unannotated branch point clouds: With the original unannotated point cloud as input, the number of anchor points is defined as 4, 4 center anchor points are selected by farthest point sampling (also referred to as: Farthest point sampling, abbreviated as: FPS), the nearest center anchor point of each sample point in the entire point cloud is found by Euclidean space distance, and the entire point cloud is divided into four local point clouds. Then, take the four local point clouds as input, respectively, and perform rotation, scaling and translation three kinds of similarity transformation. In the similarity transformation, the rotation interval is defined as [-10, 10], the scaling interval is defined as [1, 3], and the translation interval is defined as [-0.25, 0.25]. Then, random values are taken in the defined intervals of rotation, scaling and translation to respectively transform the four local point clouds, and four transformed local point clouds are obtained. Finally, taking the four transformed local point clouds as input, based on the distance from each sample point to the anchor point, a Nadaraya-Watson kernel regression function (also referred to as: Nadaraya-Watson kernel regression, abbreviated as: N-W kernel regression) is used to perform smooth interpolation in three-dimensional space local transformation, thereby obtaining two transformed branch point clouds. Since the similarity transformation process does not affect the three-dimensional features between sample points, the two branch point clouds can maintain high semantic feature similarity in the feature space, which can be used for unsupervised training.

[0032] The unsupervised semantic segmentation model (3D-USF) used for training in the present application can utilize Figure 1 the global similarity loss provided in the inner B, by calculating the point-by-point semantic similarity between the two unannotated branch point clouds, to construct a global similarity loss function . The function aims to evaluate the semantic mutual correspondence of the two branch point clouds in the global feature space. The global similarity module helps the model to make more accurate segmentation strategies by combining global context features, and improves the segmentation performance of the boundary or transition area. Since the translation, scaling and rotation operations of the three-dimensional point cloud do not change the three-dimensional structure of the point cloud, the global loss function aims to evaluate the semantic mutual correspondence of the two branch point clouds in the global feature space. The global similarity module helps the model to make more accurate segmentation strategies by combining global context features, and avoids the backbone feature extraction network from paying too much attention to local features and ignoring the overall structure of the point cloud.

[0033] Taking the two branch point clouds obtained by the branch point cloud transformation as an example, the input of the global similarity loss function is the features of the two branch point clouds, which are marked as feature 1 and feature 2, and the feature spaces of feature 1 and feature 2 maintain consistent global semantic representations. The calculation process of the global similarity module is as follows: first, the two sets of semantic features are respectively subjected to norm normalization (also referred to as: L2 Normalization, for short: L2) (formula 2), to obtain the normalized semantic features of the two branch point clouds and Then, the point-by-point semantic similarity (also referred to as: Point-by-point semantic similarity, for short: ) between the normalized semantic features and is calculated by the soft cosine similarity (also referred to as: Soft cosine similarity) of formula 3; finally, the global similarity loss (also referred to as: Global similar loss, for short: ) is constructed: , (formula 2) , (formula 3) (formula 4) wherein, represents the number of points of the unlabeled branch point cloud, , represents the total multiplier value in a point cloud, and respectively represent the normalized semantic features of different unlabeled branch point clouds, the calculation method of is the same, represents the soft cosine similarity between the semantic features, represents the batch size.

[0034] The global similarity loss function above has a value range of (0, 1), and the smaller the value, the more similar the point-by-point semantic information of feature 1 and feature 2 is. Minimizing the global similarity loss (also referred to as: Global similar loss, for short: ) can effectively constrain the model to learn the corresponding global semantic features between the branch point clouds, which helps to improve the segmentation performance of the model in processing boundary or transition regions.

[0035] The above unsupervised semantic segmentation model (3D-USF) can also utilize Figure 1In the same way as the class-intra loss provided in the inner C, the class-intra similarity loss function is constructed by calculating the feature similarity between the center point and the sample points in each neighborhood point group , which is used to measure the local feature learning ability of the model in the unsupervised pre-training stage. The core idea of the class-intra similarity loss function is that in the feature space, sample points with similar semantic features, such as sample points of the same leaf or the same ear, should maintain a close geometric distribution, that is, sample points in the same neighborhood point group should have a higher feature similarity, so that the local context features can be enhanced by aggregating sample points with similar semantic features, so that points in the feature space that may have similar semantic information are close to each other.

[0036] Taking the two branch point clouds obtained by the branch point cloud transformation as an example, the input of the class-intra similarity loss function is the features of the two branch point clouds, which are marked as feature 1 and feature 2. Fusion features 1 (also referred to as: Fusion features 1, abbreviated as: F1') and fusion features 2 (also referred to as: Fusion features 2, abbreviated as: F2') are obtained by concatenating feature 1 with its position feature 1 (also referred to as: position feature 1, abbreviated as: F1pos) and feature 2 with its position feature 2 (also referred to as: position feature 2, abbreviated as: F2pos), respectively. In specific implementation, the matrix size of feature 1 is [4096, 128], and the matrix size of position feature F1pos is [4096, 3]. By directly concatenating position feature F1pos to the column vector of feature 1, the matrix size of fusion feature 1 obtained is [4096, 131]. In the same way, feature 2 and position feature F2pos are concatenated, and fusion feature 2 with a matrix size of [4096, 131] is obtained by concatenating in the same way; Then, H class-intra center points are selected from fusion feature F1' and fusion feature F2' according to formula 5 by farthest point sampling. Then, based on the H class-intra center points, K- nearest neighbor algorithm (also referred to as: K-Nearnest Neighbors, abbreviated as: KNN) is used to select the K nearest sample points to each class-intra center point in the feature space of fusion feature F1' and F2', thereby constructing H neighborhood point groups, each containing K sample points. Finally, for the H neighborhood point groups of each branch, the semantic feature similarity between the neighborhood point group and its K sample points is calculated, and the class-intra similarity loss function is constructed according to the following calculation process : Referring to formula 8, the class-intra loss (also referred to as: class-intra loss, abbreviated as: ) is constructed by the local feature similarity in the neighborhood point group provided by formula 6 and the regularization loss between neighborhood point groups provided by formula 7 comprises. The average feature distance between the intra-class center point and all sample points in each neighborhood point group is calculated, and the minimum The feature distance between points in the group can be shortened, so that points that may have the same semantic label are clustered in the feature space. Covariance regularization , by constraining the covariance matrix of the feature vector within the point group, minimizing the non-diagonal elements in the matrix, so as to approach a diagonal matrix, which helps to reduce the redundant correlation between feature dimensions, prevent the model from learning meaningless feature combinations in the unsupervised pre-training phase, and thus reduce the risk of overfitting. Intra-class similarity loss The value range of is (0, 1), which is closer to 0, indicating that the semantic features within the neighborhood point group are more similar, and the minimum can effectively shorten the distance between points in the feature space that may have the same semantic label, and promote the close aggregation of intra-class features: (formula 5) (formula 6) (formula 7) (formula 8) In the above formulas, represents the multiple value used for downsampling, which is set to 16 here; represents the logarithmic operation; represents the determinant, that is, the content in the parentheses is converted into a determinant; represents the hth intra-class center point; represents the kth sample point of the hth intra-class center point; represents a positive decimal number, which is set to 0.001; represents the neighborhood point group of one of the branch point clouds, represents the neighborhood point group of the other branch point cloud; represents the transpose of Z; represents the feature dimension of Z, which is 131 here; represents an identity matrix with a size of ; represents a given multiple value, which is set to here; respectively represent the cosine similarity between each intra-class center point and each sample point in the group in the two branch point clouds; represents the number of intra-class center points, h=[1,2,3,…,H], H represents the total number of intra-class center points; represents the regularization loss between neighborhood point groups; denotes the batch size, which is set to 8 here.

[0037] The unsupervised semantic segmentation model (3D-USF) can also utilize Figure 1 the inter-class separation loss provided in the inner D to construct an inter-class separation loss function by calculating the separation degree between different unsupervised clustering point groups . This function aims to evaluate the ability of the model in the unsupervised learning stage to distinguish feature points that may have different semantics within the local neighborhood. The design of this function is based on the fact that although the leaf points and ear points of the wheat canopy have similarities in three-dimensional spatial distribution, there are still differences in local geometric information and semantic features. Therefore, in addition to the intra-class similarity, the unsupervised semantic segmentation model (3D-USF) also needs to increase the feature difference between clustering point groups that may have different semantic labels through the inter-class separation loss function. At the same time, this function can make the three-dimensional coordinates of the two branch point clouds initially clustered. Since the Euclidean space of the two branch point clouds should have high similarity, it can provide spatial prior knowledge for the iterative clustering in the subsequent supervised fine-tuning training, providing a favorable initialization basis.

[0038] Taking the two branch point clouds obtained by the branch point cloud transformation as an example, the input of the inter-class separation loss function is the features of the two branch point clouds, which are labeled as feature 1 and feature 2, and are divided into initial clustering based on position feature 1 (also referred to as: position feature 1, abbreviated as: F1pos) and position feature 2 (also referred to as: position feature 2, abbreviated as: F2pos) and iterative clustering based on fusion feature 1 (also referred to as: Fusion features1, abbreviated as: F1') and fusion feature 2 (also referred to as: Fusion features2, abbreviated as: F2'): In the initial clustering based on position feature 1 (also referred to as: position feature 1, abbreviated as: F1pos) and position feature 2 (also referred to as: position feature 2, abbreviated as: F2pos), first, the initial clustering center points are selected by farthest point sampling (also referred to as: Farthest point sampling, abbreviated as: FPS); secondly, each point is assigned to its nearest initial center point in Euclidean distance, and the initial clustering point group is constructed; finally, the geometric center of the initial point group is recalculated to obtain the updated clustering center point .This stage uses only position feature 1 (also known as position feature 1, or F1pos) and position feature 2 (also known as position feature 2, or F2pos) to obtain a geometrically driven, relatively stable initial cluster. This effectively alleviates the problems of initial center point randomness and outlier sensitivity in the upgraded version of the K-means algorithm (also known as KMeans++) and preserves the original structural information of the point cloud to the greatest extent.

[0039] In iterative clustering based on fusion features 1 (also known as Fusion features1, or F1') and fusion features 2 (also known as Fusion features2, or F2'), the resulting updated cluster centers are first... As prior knowledge, each point in fusion feature 1 (also known as Fusion features1, or F1') and fusion feature 2 (also known as Fusion features2, or F2') is assigned to the current cluster center with the nearest feature distance, thus constructing an unsupervised cluster point group. Then, update each unsupervised cluster group. Cluster center Finally, the difference in the sum of squared errors (also known as: Error squared and difference, abbreviated as: Subsse) (Formula 9) and the maximum number of iterations are used as the threshold conditions for terminating clustering. That is, clustering is completed when the maximum number of clustering iterations is met or (SSE < 0.01 ∩ Subsse < 0). The difference in the sum of squared errors (also known as: Error squared and difference, abbreviated as: Subsse) (Formula 10) is the difference in the sum of squared errors between two clustering iterations. The sum of squared errors is a statistical method used to measure the total difference between the predicted value and the actual value. A lower sum of squared errors indicates that the points in the point group are closer to the center point, and the clustering effect is better; conversely, a higher sum of squared errors indicates that the points in the point group are more loosely distributed, and the clustering effect is poor. (Formula 9) (Formula 10) Where v represents the v-th iteration, Indicates the number of cluster points. Indicates the cluster center number. , This represents the number of all sample points in a cluster. Indicates the first The first cluster of cluster points One sample point, represent the centroid of the th cluster point group.

[0040] class-inter loss (also referred to as: class-inter loss, for short: ) is used to evaluate the semantic feature difference between unsupervised cluster point groups. Its core role is to make point groups with different semantic labels far away from each other in the feature space, forming clearer cluster boundaries. The class-inter loss (also referred to as: class-inter loss, for short: ) (formula 12) is mainly composed of the degree of separation between classes (also referred to as: Degree of separation between classes, for short: ) (formula 11) and covariance regularization (formula 6), where respectively represent the degree of separation between classes of the two branch point clouds. The degree of separation between classes loss is constructed based on the Calinski-Harabasz Index (also referred to as: Calinski-Harabasz Index, for short: CH) idea, which is used to quantify the clustering quality. Considering the tightness within the point group and the separation between the point groups, the ratio of the point-by-point semantic feature similarity within the point group to the feature difference between the point groups is calculated, the feature difference between the cluster point groups is maximized, and the similarity between each point within the point group and the center point is ensured. The value range of is (0, 1), and the smaller the value, the better the clustering effect. At the same time, the covariance regularization loss is introduced to reduce the redundant correlation between feature dimensions and prevent the model from overfitting during unsupervised learning: (formula 11) (formula 12) wherein and represent the th iteration of the cluster point group of the two branch point clouds; represents the set of all points in the th cluster point group, and tr represents the sum of all diagonal elements, represents the sample center point of all points; represents the number of iterations, and represent the degree of separation between classes of the two branch point clouds, represents the regularization loss between the cluster point groups of the th iteration, and the batch size is consistent with the foregoing setting, represents the total number of point clouds, and the value is 4096, which is consistent with the foregoing setting.

[0041] This application utilizes the aforementioned unsupervised joint loss function The segmentation model is iteratively trained, and a set of unsupervised weights is determined when the training time reaches 100. Then, in the subsequent supervised learning fine-tuning training, the segmentation model is initialized according to the aforementioned unsupervised weights. The labeled point cloud is used as the input of the initialized segmentation model. Fine-tuning training is carried out flexibly as needed on labeled data with different training sample sizes and different levels of labeling. Cross-entropy loss is used as the loss function for this second stage to achieve high-precision segmentation of leaf spike point clouds.

[0042] To verify the effectiveness of the segmentation model obtained by this application through the above training steps, this application also conducted the following experiments to verify its recognition effect on crop leaves and ears.

[0043] (1) Data preparation.

[0044] The crop point cloud datasets used in this experiment consist of two sets: a wheat crop point cloud dataset and a rice crop point cloud dataset. The wheat crop point cloud dataset was developed in 2020 at the Nanjing Agricultural University Baima Experimental Base in Baima Town, Lishui District, Jiangsu Province. Each field breeding plot had a planting area of ​​1 m² (1 m × 1 m), a plot spacing of 0.5 m, and was divided into four planting rows with a row spacing of 0.25 m. Wheat was sown on November 15, 2019, and data was collected from 17:00 to 23:00 on April 23, 2020. A total of 170 wheat plots were labeled. After cropping these plots, 20,654 wheat point cloud samples were obtained, with each sample containing 4,096 points randomly selected. To ensure that the unsupervised dataset, training set, test set, and validation set contain the leaf-to-ear ratio (LPR) of all samples as comprehensively as possible, we first count the number of leaves and ears in all pruned samples, and then calculate the LPR for each sample according to Equation 13: (Formula 13) in, This represents the points in the point cloud that belong to the leaf. This represents the points in the point cloud that belong to the ear of grain. This indicates the leaf-to-panicle ratio.

[0045] The unsupervised dataset, training set, test set, and validation set were adjusted to cover as many leaf-to-ear ratio (LPR) distributions as possible. Then, a small number of samples were selected as the unsupervised training set according to the above principles. Finally, the remaining point cloud samples were divided into training, validation, and test sets in a 5:2:3 ratio, resulting in the sample distribution of rice and wheat point clouds shown in Table 1. Table 1 Sample distribution of rice and wheat point cloud

[0046] The crop point cloud dataset-wheat is used for full annotation samples and full annotation degree, and the segmentation model (3D-USF) of the present application is compared with the fully supervised learning model. In the comparison, first, based on the crop point cloud dataset-wheat dataset, sub-datasets with different training sample amounts are constructed, and the proportion of the training sample amount is set to [10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%]. The leaf and stem ratio (LPR) distribution of all samples in the training set and the validation set is counted, and the corresponding proportion of point cloud samples is randomly selected under each leaf and stem ratio distribution. For example, when constructing a 10% sample amount dataset, samples are randomly selected according to a proportion of 10% in the leaf and stem ratio distribution of the training set and the validation set. Finally, [10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%] datasets are constructed for sensitivity analysis of the segmentation model (3D-USF) to the training sample amount under different training sample amounts.

[0047] Based on the crop point cloud dataset-wheat dataset, different annotation degree datasets are constructed under two datasets respectively, and the annotation proportion is set to [1%, 5%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%]. The annotation point number is selected in the point cloud training sample by farthest point sampling (also known as: Farthest point sampling, abbreviated as: FPS) to construct the annotation degree dataset: the annotation point number in each point cloud sample of the training set and the validation set is [1%, 5%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%] of all point numbers, and the points in the test set are fully annotated. At the same time, based on the nearest neighbor distance method, the average point spacing (also known as: Average dot spacing, abbreviated as: APS) of the training sample is calculated by formula 13. The sensitivity analysis of the segmentation model (3D-USF) of the present application to the sample annotation degree under different annotation degrees is verified by using the dataset: (Formula 13) Wherein, is the i-th point in the point cloud, is the other point in the point cloud, N is the total number of points in the point cloud, which is set to 4096; n represents the number of points, n = [1, 2, 3, …, N], ‖·‖ represents the Euclidean space distance, represents the average point spacing (Average point spacing, APS).

[0048] Crop point cloud dataset-rice is a field experiment conducted in Baima Experimental Base of Nanjing Agricultural University in Baima Town, Lishui District, Jiangsu Province in 2021. In this experiment, the planting area of each field breeding plot is 1.8m 2 (1m x 1.8m), and the plant spacing and row spacing in the plot are 13cm and 25cm, respectively, with 3 seedlings per hole. In the experiment, the high-throughput field plant phenotyping platform developed by the Dutch Phenospex Company (also known as: High-throughput field plant phenotyping platform, abbreviated as: FieldScan) was used to collect laser radar point cloud data of rice. The collection time is September 22 and October 12, 2021, and 4 times of data are collected automatically every day, with the 4 times of data collection periods being 6:00-11:00, 11:00-17:00, 17:00-23:00 and 23:00-05:00. There are 29 rice plots labeled, and 8504 rice point cloud samples are obtained after cropping 29 rice plots. Each point cloud sample is randomly sampled to 4096 points.

[0049] In the rice experiment, the same division method as the wheat point cloud dataset is used. First, a small number of samples are selected as the unsupervised training set, and then the remaining point cloud samples are divided into training set, validation set and test set according to the ratio of 5:2:3, which is used for model comparison of rice point cloud. And in the same way as the wheat training sample amount sub-dataset, the rice point cloud dataset with different training sample amounts is constructed, which is used for the sensitivity analysis of the training sample amount of rice point cloud. At the same time, in the same way as the wheat labeling degree sub-dataset, the rice point cloud dataset with different training sample amounts is constructed, which is used for the sensitivity analysis of the labeling degree of rice point cloud.

[0050] (2) Experimental method.

[0051] Model training: The segmentation model (3D-USF) provided in the present application can be implemented on the framework of PyTorch1.10.1, and trained on a single NVIDIA GeForce RTX 3090 GPU with 24 GB of random access memory (also referred to as: Random Access Memory, abbreviated as: RAM). During training, the batch size of the wheat and rice datasets is set to 8, the training epoch is set to 100, and the initial learning rate is 0.001; the Stochastic Gradient Descent (also referred to as: SGD) optimizer is used, wherein the weight decay is set to 0.001 and the momentum coefficient is set to 0.9; the learning rate scheduler type used is “step”, the interval period of learning rate decay is 10, and the factor of learning rate scaling is 0.7 each time the decay occurs. The accuracy, average cross-to-match ratio (also referred to as: mIoU), and recall rate are selected as three quantitative indicators to evaluate the segmentation performance of the segmentation model (3D-USF) for rice and wheat leaf ear point clouds. These indicators are used to evaluate the effectiveness of the model from different angles.

[0052] Model comparison: In order to evaluate the improvement of the segmentation model (3D-USF) of the present application on the segmentation performance of different types of deep learning models, the present experiment compares the segmentation performance of six backbone feature extraction networks based on the crop point cloud dataset-wheat dataset. First, six models are selected, including a representative dynamic graph convolution (also known as: Dynamic graph cnn for learning on point clouds, abbreviated as: DGCNN), a point cloud transformer (also known as: Point Cloud Transformer, abbreviated as: PCT), a point cloud upgrade segmentation model (also known as: Deep hierarchical feature learning on point sets in a metric space, abbreviated as: PointNet++), a point cloud improvement (also known as: Revisiting pointnet++ with improved training and scaling strategies, abbreviated as: PointNeXt), a point cloud pure residual multilayer perceptron (also known as: pure residual MLP network, abbreviated as: PointMLP) and a point cloud convolution (also known as: Deep convolutional networks on 3d point clouds, abbreviated as: PointConv). Then, the aforementioned unsupervised pre-training and supervised learning fine-tuning training method is used to train the above six models, and the segmentation accuracy of each model is evaluated.

[0053] Training sample size sensitivity analysis: In order to explore the sensitivity of the segmentation model (3D-USF) of the present application and the full supervision learning of different models to the training sample size, the present application applies the sample size interval of 10%-100% of the crop point cloud dataset-wheat dataset, compares the segmentation performance of the full supervision learning model and the segmentation model (3D-USF) of the present application when using the aforementioned six backbone feature extraction networks, and evaluates the sensitivity of different models to the training sample size under the training method of the present application.

[0054] Labeling sensitivity analysis: In order to explore the sensitivity of the segmentation model (3D-USF) of the present application and the full supervision learning of the model to the sample labeling degree, the present application further applies the labeling degree interval of 10%-100% of the crop point cloud dataset-wheat dataset, compares the segmentation performance that can be achieved by the six backbone feature extraction networks under the training method of the present application, evaluates the sensitivity of different segmentation models to the sample labeling degree, and the average point distance and the change trend of the model segmentation performance with the labeling degree.

[0055] Ablation analysis: In order to explore the influence of the four modules of branch point cloud transformation, global similarity, intra-class similarity and inter-class separation on the segmentation performance of the unsupervised semantic segmentation model, the present application first selects the model with the best performance in leaf and ear segmentation and the highest sample size sensitivity based on model comparison experiments and training sample size sensitivity analysis experiments. Then, using the 10% training sample size dataset of the crop point cloud dataset-wheat, the ablation experiment is carried out under four training conditions, which include all four modules and remove one module at a time.

[0056] Cross-crop adaptability: In order to explore the expansibility of the segmentation model (3D-USF) provided by the present application to rice point cloud, the present application also evaluates the segmentation performance, training sample size sensitivity and annotation size sensitivity of the model based on the crop point cloud dataset-rice dataset. First, full supervised learning comparison training of six models is realized on the crop point cloud dataset-rice dataset. Then, based on the model with the best segmentation performance in the model comparison experiment, the sensitivity of the full supervised learning model and the segmentation model (3D-USF) of the present application to the sample size of rice is compared within the 10%-100% sample size interval of the crop point cloud dataset-rice. Finally, based on the model with the best segmentation performance in the model comparison experiment, the sensitivity of the full supervised learning model and the segmentation model (3D-USF) of the present application to the annotation degree is compared within the 1%-100% sample degree interval of the crop point cloud dataset-rice.

[0057] (3) Experimental results.

[0058] Under the condition of 100% training sample size of the crop point cloud dataset-wheat, the quantitative comparison results of the segmentation results of the segmentation framework (3D-USF) provided by the present application and the six models under full supervised learning using the same backbone feature extraction network are as shown in the following table: Table 2 Model comparison results

[0059] The model comparison results provided in Table 2 show that compared with full supervised learning, the segmentation model (3D-USF) of the present application can improve the leaf and ear segmentation accuracy of the six backbone feature extraction networks, and the segmentation performance of PointConv is best improved. Especially for the "ear" points, the segmentation performance of PointConv is best improved, and the average cross-to-match ratio (also known as: Average cross-to-match ratio, abbreviated as: mIoU) increases by 1.17%; while the segmentation performance of PointNeXt is the lowest, and the average cross-to-match ratio (mIoU) only increases by 0.04%. This shows that the segmentation model (3D-USF) of the present application can effectively improve the segmentation performance of the six backbone feature extraction networks based on points and graphs.

[0060] The quantitative comparison results of the six models using different backbone feature extraction networks under the full supervision learning in the 1-100% training sample amount interval of the crop point cloud dataset-wheat are shown in Table 3. Table 3 Sample amount sensitivity analysis based on full supervision learning

[0061] The quantitative comparison results of the six models using different backbone feature extraction networks under the segmentation framework (3D-USF) provided by the present application in the 1-100% training sample amount interval of the crop point cloud dataset-wheat are shown in Table 4. Table 4 Model training sample amount sensitivity analysis based on the segmentation framework (3D-USF) of the present application

[0062] The training sample amount sensitivity analysis results shown in Table 3 and Table 4 show that: compared with full supervision learning, (1) as the training sample amount increases, the average intersection over union (mIoU) change trend of the segmentation model (3D-USF) obtained by the present application is relatively slow, which indicates that the unsupervised semantic segmentation model obtained by the present application is more robust to the change of the training sample amount. (2) The segmentation models (3D-USF) obtained based on the six backbone feature extraction networks in the present application can achieve the same segmentation accuracy as full supervision using less training sample amount, which indicates that the unsupervised semantic segmentation model of the present application can effectively reduce the demand of the model for training sample amount, especially in the use of point cloud convolution (PointConv), the unsupervised semantic segmentation model (3D-USF) has the best model improvement effect, and in the 10%-40% less training sample amount interval, the segmentation performance improvement effect on the "ear" category is the best. (3) Referring to the qualitative results of the segmentation model using point cloud convolution (PointConv) shown in Table 5, in the scenarios of high or low ear, the segmentation model (3D-USF) provided by the present application effectively reduces the occurrence of missegmentation points of "leaf" and "ear" in the six models, especially when the sample amount is small. It can be proved that the unsupervised semantic segmentation model provided by the present application is applicable in scenarios of high or low ear, and has robustness in improving the segmentation performance of different models, especially the segmentation performance improvement in the 10%-40% less sample amount interval is more significant. Figure 3

[0063] ​In the 1%-100% annotation degree interval of the crop point cloud dataset-wheat, the backbone feature extraction network based on point cloud convolution (PointConv) is trained based on the unsupervised semantic segmentation framework (3D-USF) provided in the present application. Table 5 shows the annotation degree sensitivity analysis results: Table 5 Sensitivity analysis for sample annotation degree

[0064] The results of Table 5 show that compared with full supervision learning, the segmentation model (3D-USF) obtained based on unsupervised semantic training provided in the present application is more robust to changes in annotation degree when using point cloud convolution (PointConv). Especially at low annotation degree, the unsupervised semantic segmentation model (3D-USF) provided in the present application has more significant segmentation performance improvement for point cloud convolution (PointConv): only 10% of the annotation degree, i.e. the average point distance is 0.5371 cm, can achieve the segmentation accuracy equivalent to full supervision learning at full annotation degree.

[0065] The ablation analysis results shown in Table 6 show that: Table 6 Ablation analysis results of four modules under 10% training sample size of crop point cloud dataset-wheat

[0066] Compared with the unsupervised semantic segmentation model (3D-USF) containing four modules in the present application, on the "leaf" and "ear" categories, (1) the segmentation model lacking point cloud transformation, the average intersection over union (mIoU) decreases by 0.3%, 4.83% respectively; (2) the segmentation model lacking global loss, the average intersection over union decreases by 0.17%, 4.98% respectively; (3) the segmentation model lacking intra-class loss, the average intersection over union can decrease by 1.45%, 9.88% respectively; (4) the segmentation model lacking inter-class separation degree, the average intersection over union can decrease by 3.11%, 13.87% respectively. Therefore, the unsupervised semantic segmentation model containing four modules provided in the present application can exhibit the best segmentation performance.

[0067] The cross-crop adaptability test obtained the results shown in Tables 7 and 8. Among them: Table 7 Analysis and comparison results of 6 models under 100% training sample size of crop point cloud dataset-rice

[0068] Table 8 compares the sensitivity of the model obtained in the present application: Table 8 Sensitivity analysis of rice sample size

[0069] The unsupervised semantic segmentation framework (3D-USF) provided in the present application can obtain the following quantitative comparison results compared with the point cloud pure residual multilayer perceptron (PointMLP) of full supervision learning under different training sample amounts of rice: (1) Based on the model comparison results of the rice point cloud shown in Table 7, it is shown that among the unsupervised semantic segmentation models based on six backbone feature extraction networks, the point cloud pure residual multilayer perceptron (PointMLP) shows the best segmentation effect on the two categories of leaves and ears; (2) The training sample amount sensitivity analysis results based on the rice point cloud provided in Table 8 show that the unsupervised semantic segmentation model (3D-USF) based on the rice point cloud provided in the present application can effectively improve the average intersection over union of the point cloud pure residual multilayer perceptron (PointMLP), and the improvement of the segmentation performance is more significant when the sample amount is small. At the same time, it only needs 40% of the training sample amount of rice to achieve the segmentation accuracy comparable to full supervision learning, and it can effectively reduce the demand of the model for the sample amount of rice; (3) The labeling degree sensitivity analysis results based on the rice point cloud shown in Table 9 show that the unsupervised semantic segmentation model (3D-USF) provided in the present application is more robust to the change of the labeling degree of the rice sample, and the improvement of the segmentation performance of the model is more significant when the labeling degree is low. At the same time, it only needs 50% of the labeling degree to achieve the average intersection over union of leaves and ears comparable to full supervision learning. Therefore, the unsupervised semantic segmentation model (3D-USF) provided in the present application can effectively reduce the demand of the model for the labeling degree: Table 9 Quantitative results of the point cloud pure residual multilayer perceptron based on the unsupervised semantic segmentation framework and full supervision learning under different labeling degrees of rice

[0070] In summary, the segmentation model (3D-USF) provided in the present application shows good generalization on the rice point cloud, especially in the case of small sample training and low labeling, the segmentation performance is significantly improved, which proves that the model provided in the present application can have high transferability and generalization on the rice point cloud. Through the above research, the present application can provide an efficient and feasible solution for three-dimensional leaf and ear segmentation of crops, and provide important support and technical guarantee for three-dimensional data extraction and analysis in the field of agriculture.

[0071] The comparative results of the segmentation model (3D-USF) of the application under complete labeled samples and complete labeling degree, the sensitivity analysis results of the segmentation model of the application under different training sample amounts, the sensitivity analysis results of the segmentation model of the application under different labeling degrees, and the ablation analysis results of the four processing modules in the model all show that the application realizes the sensitivity design of the training sample amount and the labeling degree of the rice and wheat point cloud, which confirms that the unsupervised semantic segmentation model (3D-USF) provided by the application can effectively reduce the demand of the model for the training sample amount and the labeling degree while ensuring the segmentation accuracy of the model, which has important significance for accelerating three-dimensional phenotype analysis.

[0072] The basic principles, main features and advantages of the application are shown and described above. Those skilled in the art should understand that the application is not limited to the above examples, and the above examples and descriptions in the specification are only to illustrate the principles of the application. Without departing from the spirit and scope of the application, various changes and improvements can be made to the application, and these changes and improvements all fall within the scope of the claimed application. The scope of protection of the application is defined by the appended claims and their equivalents.

Claims

1. An unsupervised three-dimensional leaf and spike feature extraction method for crop point cloud image data, characterized in that the steps include: include: After obtaining the crop point cloud, it is input into the model that has been pre-trained through unsupervised pre-training and supervised learning fine-tuning in sequence. The model then processes the data to obtain the three-dimensional segmentation results of the leaf point cloud features and the ear point cloud features. In this process, unsupervised pre-training is performed on unlabeled crop point cloud training samples. First, the training samples are transformed into branch point clouds to generate at least two unlabeled branch point clouds. Then, the segmentation model extracts the semantic features of each unlabeled branch point cloud. The semantic features are used as the input of the unsupervised joint loss function. The parameter weights in the segmentation model are adjusted through the unsupervised joint loss function to determine a set of unsupervised weights. Supervised learning fine-tuning training initializes the segmentation model based on the unsupervised weights obtained above. It uses labeled crop point cloud training samples as input to the initialized segmentation model and uses cross-entropy loss as the loss function to fine-tune the initialized segmentation model to obtain a model for segmenting leaf point cloud features and ear point cloud features. The unsupervised joint loss function consists of three components: global similarity loss, intra-class similarity loss, and inter-class separation loss.

2. The unsupervised three-dimensional leaf and spike feature extraction method for crop point cloud image data as described in claim 1, characterized in that, The crop point cloud is obtained by processing two-dimensional images captured from multiple angles, or by acquiring them with a depth camera, or by measuring and processing them according to the light pattern on the crop surface with a structured light camera, or by scanning with a lidar. The steps for processing 2D images taken from multiple angles to obtain crop point clouds include: Local features are extracted from each image, and local feature points are matched between multiple images to find the correspondence of the same spatial location under different viewpoints in different images; Based on the matched feature points, the three-dimensional coordinates of the matching feature points in images taken by different cameras are obtained through triangulation and bundle adjustment, and the sparse point cloud of the scene is restored. By further utilizing the consistency of pixel colors and occlusion relationships in the image, pixel-level 3D inference of the scene can be performed to obtain dense point clouds or surface models. By performing coordinate unification, redundancy removal, filtering, and hole filling on the dense point clouds corresponding to images from different viewpoints, crop point clouds are obtained.

3. The unsupervised three-dimensional leaf and spike feature extraction method for crop point cloud image data as described in claim 1, characterized in that, In unsupervised pre-training, the unlabeled branch point clouds generated by performing branch point cloud transformation on the training samples correspond semantically to each other. The specific steps for performing branch point cloud transformation on the training samples include: Using unlabeled crop point cloud training samples as input, a set of anchor points is selected by sampling the farthest point. Based on the Euclidean spatial distance, the nearest anchor point to each point in the crop point cloud training samples is determined. All points are then divided into several local point clouds according to their respective anchor points. Perform similarity transformations on each local point cloud separately; The local point cloud obtained after similarity transformation is smoothed by interpolation to generate at least two unlabeled branch point clouds.

4. The unsupervised three-dimensional leaf and spike feature extraction method for crop point cloud image data as described in claim 3, characterized in that, The unsupervised joint loss function is: , in, For unsupervised joint loss function; The global similarity loss first normalizes the semantic features of each unlabeled branch point cloud, and then constructs a global similarity loss that reflects the global point-by-point semantic similarity based on the soft cosine similarity between the normalized semantic features. The segmentation model is constrained to learn the similarity of global semantic features between each unlabeled branch point cloud by minimizing this global similarity loss. The intra-class similarity loss constructs a neighborhood point group by sampling the farthest point. Based on the feature similarity between each intra-class center point and each sample point in the group and the covariance matrix of the feature vectors in the neighborhood point group, an intra-class similarity loss is constructed that reflects the redundant correlation between feature distances and feature dimensions among points in the neighborhood point group. By minimizing this intra-class similarity loss, the segmentation model is constrained to bring points with the same semantic features in the feature space closer together. The inter-class separation loss is constructed by calculating the ratio of semantic feature similarity within a cluster to semantic feature difference between different clusters, thus reflecting the separation degree between clusters. Maximizing this inter-class separation loss constrains the segmentation model to distinguish points with different semantic features.

5. The unsupervised three-dimensional leaf and spike feature extraction method for crop point cloud image data as described in claim 4, characterized in that, Global similarity loss function ; in, This represents the number of points in the unlabeled branch point cloud. , This represents the total number of points in a point cloud. and These represent the semantic features after normalization of different unlabeled branch point clouds. The soft cosine similarity represents the similarity between semantic features. Indicates batch size.

6. The unsupervised three-dimensional leaf and spike feature extraction method for crop point cloud image data as described in claim 4, characterized in that, Intra-class similarity loss function ; in, This indicates the multiplier used for downsampling. This represents the neighborhood point group of one of the branch point clouds. This represents the neighboring point group of another branch of the point cloud. To represent a given multiple value, and This represents the cosine similarity between the in-class center point and the in-group sample point in the point clouds of the two branches. Indicates the number of the center point within the class. H represents the total number of center points within the class. This represents the regularization loss between neighboring point groups. Indicates batch size.

7. The unsupervised three-dimensional leaf and spike feature extraction method for crop point cloud image data as described in claim 4, characterized in that, Inter-class separation loss function ; in, and These represent the first two branches of the point cloud. Clustering point groups in the next iteration Indicates the number of iterations. and This represents the inter-class separation degree of the point clouds of the two branches. Indicates the first The regularization loss between cluster point groups in each iteration. This represents the total number of center points between classes. This represents the number of the center point between any two classes. , Indicates batch size.

8. The unsupervised three-dimensional leaf and spike feature extraction method for crop point cloud image data as described in claim 4, characterized in that, The segmentation model can be any of the following: dynamic graph convolution segmentation model, point cloud transformer segmentation model, point cloud upgraded segmentation model, point cloud improved segmentation model, point cloud pure residual multilayer perceptron segmentation model, or point cloud convolution segmentation model.

9. The unsupervised three-dimensional leaf and spike feature extraction method for crop point cloud image data as described in claim 5, characterized in that, The segmentation models used in unsupervised pre-training do not contain segmentation heads; the segmentation models used in supervised learning fine-tuning all contain segmentation heads; the segmentation models used in both unsupervised pre-training and supervised learning fine-tuning have the same backbone feature extraction network.

10. The unsupervised three-dimensional leaf and spike feature extraction method for crop point cloud image data as described in any one of claims 1-9, characterized in that, In all point cloud training samples, the proportion of labeled points ranges from 1% to 100% of the total points.

Images