A crop point cloud instance segmentation method, device, equipment, medium and product

Abstract

This application discloses a method, apparatus, device, medium, and product for instance segmentation of crop point clouds, relating to the field of agricultural informatics technology. The method comprises: obtaining a crop point cloud set; the crop point cloud set includes the number of points in the crop point cloud and the real coordinates of each point in three-dimensional space; employing a plant data enhancement method to perform data enhancement processing on the crop point cloud set, thereby obtaining enhanced point cloud data; and performing organ-level instance segmentation of the crop point cloud based on the enhanced point cloud data, based on a point cloud instance segmentation model, thereby obtaining segmentation information data; the segmentation information data includes leaf organs and stem organs; the point cloud instance segmentation model is obtained by deep learning based on a crop point cloud set with labeled data using a soft grouping algorithm; the labeled data is the segmentation information data. This application aims to achieve accurate organ-level instance segmentation of crop point clouds.

Description

Technical Field

[0001] The present application relates to the field of agricultural informatics technology, and in particular to a crop point cloud instance segmentation method, device, equipment, medium and product. Background Art

[0002] To address the challenge of accurate instance segmentation in crop point clouds due to occlusion and intersection, researchers have proposed many automated crop point cloud instance segmentation methods, aiming to reduce reliance on manual parameter settings and improve robustness to complex scenes. One common method is the instance segmentation algorithm based on deep learning.

[0003] However, current methods typically require large amounts of point cloud data for model training. Traditional point cloud data augmentation methods simply perform simple operations such as rotation, translation, and scaling on point cloud data. While these methods can increase data diversity to a certain extent, they still have limitations and make it difficult to fully realize the data's potential.

[0004] Therefore, it is crucial to effectively improve the model's leaf instance segmentation capability for complex crops and occlusion situations in order to improve the accuracy and robustness of crop point cloud instance segmentation. Summary of the Invention

[0005] The purpose of this application is to provide a crop point cloud instance segmentation method, device, equipment, medium and product, which can achieve organ-level accurate instance segmentation of crop point clouds.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides a crop point cloud instance segmentation method, comprising:

[0008] Acquire a crop point cloud point set; the crop point cloud point set includes the number of points in the crop point cloud and the real coordinates of each point in three-dimensional space;

[0009] Using a plant data enhancement method, performing data enhancement processing on the crop point cloud according to the crop point cloud point set to obtain enhanced point cloud data;

[0010] Based on the point cloud instance segmentation model, organ-level instance segmentation of the crop point cloud is performed according to the enhanced point cloud data to obtain segmentation information data; the segmentation information data includes: leaf organs and stem organs; the point cloud instance segmentation model is obtained by using a soft grouping algorithm and deep learning based on a crop point cloud point set with labeled data; the labeled data is the segmentation information data.

[0011] Optionally, a plant data enhancement method is used to perform data enhancement processing on the crop point cloud according to the crop point cloud point set to obtain enhanced point cloud data, specifically including:

[0012] Performing principal component analysis on the crop point cloud point set to obtain a crop point cloud analysis point set; the principal component analysis includes a first principal component analysis and a second principal component analysis; the direction of the first principal component analysis is the direction of leaf length; the direction of the second principal component analysis is the direction of leaf width;

[0013] A plant data enhancement method is adopted to set the expected number of crop point clouds after amplification, and data transformation and data enhancement processing are performed on the crop point cloud analysis point set to obtain enhanced point cloud data.

[0014] Optionally, a plant data enhancement method is used to set the desired number of crop point clouds after amplification, and data transformation and data enhancement processing are performed on the crop point cloud analysis point set to obtain enhanced point cloud data, specifically including:

[0015] Use plant data augmentation methods to set the expected number of crop point clouds after augmentation , the crop point cloud analysis point set is sequentially Transform the point cloud data by performing the transformations; each transformation includes: leaf length transformation, leaf width transformation and leaf angle transformation;

[0016] Merge all transformed point cloud data to obtain enhanced point cloud data;

[0017] The leaf length transformation is to scale the point cloud density along the direction of the first principal component analysis; the leaf width transformation is to scale the point cloud density along the direction of the second principal component analysis; the leaf angle transformation is to transform the leaf inclination angle by rotating along the z-axis in three-dimensional space;

[0018] The scaling ratio interval corresponding to the leaf length transformation is [0.5, 2];

[0019] The scaling ratio interval corresponding to the leaf width transformation is [0.5, 2];

[0020] The rotation angle interval corresponding to the leaf angle transformation is [-30°, 30°].

[0021] Optionally, the point cloud instance segmentation model includes a bottom-up grouping module and a top-down refinement module; the grouping module and the refinement module are connected;

[0022] Wherein, the grouping module includes a point-by-point prediction network and a soft grouping module connected in sequence;

[0023] The point-by-point prediction network is used to:

[0024] voxelization is performed on the enhanced point cloud data to convert disordered points into ordered volume meshes;

[0025] Based on the U-Net backbone network, point features are determined according to the ordered volume grid;

[0026] Constructing branches based on point features to output point-by-point semantic scores and offsets; the branches include: semantic branches and offset branches;

[0027] The soft grouping module is used to receive the point-by-point semantic score and the offset and shift according to the offset, and iterate based on the set semantic score according to the point-by-point semantic score. Class, at each class index, slice the point cloud data set with point-by-point semantic scores higher than the threshold, and create links between points whose geometric distance is less than the set grouping bandwidth to obtain instance proposals;

[0028] The refinement module is used to classify and refine the instance proposals to obtain segmentation information data.

[0029] Optionally, in the refinement module, classification and refinement processing are performed based on the classification branch, the segmentation branch, and the mask scoring branch;

[0030] The classification branch uses a global average pooling layer to aggregate the features of all points in the instance proposal, and uses MLP to predict the classification score, outputting the category and classification confidence score;

[0031] The segmentation branch uses a two-layer point-by-point MLP to perform instance mask prediction on the instance proposal to obtain the instance mask;

[0032] The mask scoring branch is used to determine a mask score based on the instance proposal;

[0033] The final confidence score is determined based on the mask score and the classification confidence score;

[0034] According to the final confidence score, an instance mask corresponding to a set proportion is selected from the instance masks as the segmentation information data.

[0035] Optionally, the method for determining the point cloud instance segmentation model specifically includes:

[0036] Acquire training data; the training data includes: a training crop point cloud point set and corresponding label data;

[0037] The plant data enhancement method is used to perform data enhancement processing on the training crop point cloud point set to obtain enhanced training point cloud data;

[0038] Build a point cloud instance segmentation network;

[0039] Inputting the training data into the point cloud instance segmentation network, and performing deep learning training on the point cloud instance segmentation network using a soft grouping algorithm to obtain a trained point cloud instance segmentation network;

[0040] The trained point cloud instance segmentation network is determined as the point cloud instance segmentation model.

[0041] In a second aspect, the present application provides a crop point cloud instance segmentation device, comprising:

[0042] A data acquisition module is used to acquire a crop point cloud point set; the crop point cloud point set includes the number of points in the crop point cloud and the real coordinates of each point in three-dimensional space;

[0043] a data enhancement module, configured to perform data enhancement processing on the crop point cloud according to the crop point cloud point set using a plant data enhancement method to obtain enhanced point cloud data;

[0044] A segmentation module is used to perform organ-level instance segmentation of the crop point cloud according to the enhanced point cloud data based on the point cloud instance segmentation model to obtain segmentation information data; the segmentation information data includes: leaf organs and stem organs; the point cloud instance segmentation model is obtained by using a soft grouping algorithm and deep learning based on the crop point cloud point set with labeled data; the labeled data is the segmentation information data.

[0045] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned crop point cloud instance segmentation method.

[0046] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned crop point cloud instance segmentation method.

[0047] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the above-mentioned crop point cloud instance segmentation method.

[0048] According to the specific embodiments provided in this application, this application has the following technical effects:

[0049] The present application provides a method, apparatus, device, medium and product for crop point cloud instance segmentation, which adopts a plant data enhancement method, performs data enhancement processing on the crop point cloud based on the crop point cloud point set, and obtains enhanced point cloud data. It can use the feature information of the point cloud instance to design an enhancement strategy, provide data diversity for subsequent point cloud instance segmentation, and give full play to the data potential to improve the performance and robustness of the point cloud instance segmentation model. Since the point cloud instance segmentation model is obtained by deep learning based on the crop point cloud point set with labeled data using a soft grouping algorithm, it can usually better adapt to crop scenes of different types and complexities, and has higher accuracy and robustness. The soft grouping algorithm is used to achieve accurate segmentation of crop point cloud data in complex scenes. Therefore, the present application can achieve accurate instance segmentation of crop point clouds at the organ level. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 Flowchart of the crop point cloud instance segmentation method;

[0052] Figure 2 This is the operational flow chart of the crop point cloud instance segmentation method in practical applications;

[0053] Figure 3 Schematic diagram of principal component analysis of leaf-based point cloud;

[0054] Figure 4 Schematic diagram of data enhancement using the Plant Augment method; Figure 4 (a) Schematic diagram of PlantAugment transformation - leaf length is reduced by 20%; Figure 4 (b) Schematic diagram of Plant Augment transformation - leaf width increased by 20%; Figure 4 (c) Schematic diagram of Plant Augment transformation - leaf inclination angle reduced by 20°;

[0055] Figure 5 Schematic diagram of the point cloud instance segmentation model structure using the Soft Group algorithm;

[0056] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0058] The principle of deep learning-based point cloud segmentation is to automatically learn features from point cloud data using deep neural networks and classify the point clouds into different groups or categories based on these features. This method is generally better able to adapt to crop scenes of different types and complexities, and has higher accuracy and robustness. The point cloud instance segmentation model using the soft grouping algorithm (Soft Group) achieves accurate segmentation of crop point cloud data in complex scenes through soft grouping and a top-down refinement module (Refinement Module). This application applies it to the crop instance segmentation scenario, aiming to obtain more accurate crop point cloud organ instances, thereby providing reliable technical support for the automated calculation of crop phenotypic data.

[0059] In addition, this application proposes a point cloud enhancement algorithm called Plant Augment, which enhances point cloud data based on point cloud instance information, aiming to improve the performance and robustness of deep learning-based point cloud instance segmentation algorithms. The core idea of ​​Plant Augment is to use the feature information of point cloud instances to design an enhancement strategy, including three operations: leaf length change, leaf width change, and leaf angle change, to generate richer and more representative training data. This enhancement method can effectively improve the model's leaf instance segmentation capabilities for complex crops and occlusion situations, enhancing the model's generalization and robustness.

[0060] The Soft Group algorithm, combined with Plant Augment point cloud enhancement technology, can effectively overcome the limitations of traditional methods, improve the accuracy and robustness of crop point cloud instance segmentation, and thus more accurately calculate crop phenotypic data, providing more reliable technical support for applications in the agricultural field.

[0061] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0062] In an exemplary embodiment, Figure 1 As shown, a crop point cloud instance segmentation method is provided. The method is executed by a computer device, specifically a terminal or server, alone, or jointly. In the embodiments of this application, the method is described using a server as an example, and includes the following steps.

[0063] like Figure 1 As shown, the crop point cloud instance segmentation method provided in the embodiment of the present application includes:

[0064] Step 100: Obtain a crop point cloud point set. The crop point cloud point set includes the number of points in the crop point cloud and the real coordinates of each point in three-dimensional space.

[0065] Step 200: Using a plant data enhancement method, perform data enhancement processing on the crop point cloud according to the crop point cloud point set to obtain enhanced point cloud data.

[0066] Step 300: Based on the enhanced point cloud data, the point cloud instance segmentation model performs organ-level instance segmentation on the crop point cloud, generating segmentation information data. The segmentation information data includes leaf and stem organs. The point cloud instance segmentation model is derived by deep learning using a soft grouping algorithm based on a crop point cloud dataset with labeled data. The labeled data is the segmentation information data.

[0067] In one embodiment, a plant data enhancement method is used to perform data enhancement processing on a crop point cloud according to a crop point cloud point set to obtain enhanced point cloud data, specifically including:

[0068] Principal component analysis is performed on the crop point cloud point set to obtain the crop point cloud analysis point set; the principal component analysis includes the first principal component analysis and the second principal component analysis; the direction of the first principal component analysis is the leaf length direction; the direction of the second principal component analysis is the leaf width direction.

[0069] The plant data enhancement method is used to set the expected number of crop point clouds after amplification, and the crop point cloud analysis point set is transformed and enhanced to obtain the enhanced point cloud data.

[0070] As an optional implementation, a plant data enhancement method is used to set the desired number of crop point clouds after amplification, and perform data transformation and data enhancement processing on the crop point cloud analysis point set to obtain enhanced point cloud data, specifically including:

[0071] Use plant data augmentation methods to set the expected number of crop point clouds after augmentation , the crop point cloud analysis point set is performed in sequence The transformed point cloud data is obtained by performing the transformations; each transformation includes: leaf length transformation, leaf width transformation and leaf angle transformation.

[0072] All transformed point cloud data are merged to obtain enhanced point cloud data.

[0073] Among them, the leaf length transformation is to scale the point cloud density along the direction of the first principal component analysis; the leaf width transformation is to scale the point cloud density along the direction of the second principal component analysis; and the leaf angle transformation is to rotate along the z-axis in three-dimensional space to transform the leaf inclination angle.

[0074] The scaling ratio range for leaf length transformation is [0.5, 2]. The scaling ratio range for leaf width transformation is [0.5, 2]. The rotation angle range for leaf angle transformation is [-30°, 30°].

[0075] The point cloud instance segmentation model includes a bottom-up grouping module and a top-down refinement module; the grouping module is connected to the refinement module; wherein the grouping module includes a point-by-point prediction network and a soft grouping module connected in sequence.

[0076] The point-by-point prediction network is used to voxelize the enhanced point cloud data to convert disordered points into ordered volume meshes; the point-by-point prediction network is used to determine point features based on the ordered volume meshes based on the U-Net backbone network; the point-by-point prediction network is used to construct branches based on point features to output point-by-point semantic scores and offsets; the branches include: semantic branches and offset branches.

[0077] The soft grouping module is used to receive the point-by-point semantic score and the offset and shift according to the offset, and iterate based on the set semantic score according to the point-by-point semantic score. At each class index, the point cloud data set with point-by-point semantic scores higher than the threshold is sliced, and links are created between points whose geometric distance is less than the set grouping bandwidth to obtain instance proposals.

[0078] The refinement module is used to classify and refine instance proposals to obtain segmentation information data.

[0079] Specifically, in the refinement module, classification and refinement processing are performed based on the classification branch, the segmentation branch, and the mask scoring branch.

[0080] The classification branch uses a global average pooling layer to aggregate the features of all points in the instance proposal, and uses a multilayer perceptron (MLP) to predict the classification score and output the category and classification confidence score.

[0081] The segmentation branch uses a two-layer point-by-point MLP to predict instance masks for instance proposals to obtain instance masks; the mask scoring branch is used to determine the mask score based on the instance proposal; the final confidence score is determined based on the mask score and the classification confidence score; according to the final confidence score, the instance mask corresponding to the set proportion is selected in the instance mask as the segmentation information data.

[0082] In one embodiment, a method for determining a point cloud instance segmentation model specifically includes:

[0083] Obtain training data; the training data includes: training crop point cloud point set and corresponding label data.

[0084] The plant data enhancement method is used to perform crop point cloud data enhancement processing on the training crop point cloud point set to obtain enhanced training point cloud data; a point cloud instance segmentation network is constructed; the training data is input into the point cloud instance segmentation network, and the soft grouping algorithm is used to perform deep learning training on the point cloud instance segmentation network to obtain the trained point cloud instance segmentation network; the trained point cloud instance segmentation network is determined as the point cloud instance segmentation model.

[0085] This application combines a data augmentation method for crop point cloud instances, namely the PlantAugment method, to enhance crop point clouds. Subsequently, a crop point cloud instance segmentation model based on the soft grouping algorithm (Soft Group) is used to perform fine-grained organ-level instance segmentation. This application can accurately perform organ-level instance segmentation for a variety of crops.

[0086] In practical applications, such as Figure 2 As shown in Figure 2, the crop point cloud instance segmentation method based on Soft Group specifically includes:

[0087] Step 1: Input the crop point cloud point set with labeled instance labels. The instance labels are divided into two categories: leaf organs and stem organs. It can be expressed as ,in, Refers to the number of points in the input point cloud, is a real number space, and each point in three-dimensional space is represented by It consists of three real coordinates.

[0088] Step 2: Combined with the crop point cloud example, the crop point cloud enhancement is completed based on the Plant Augment method.

[0089] Wherein, step 2 includes:

[0090] S21: Perform principal component analysis on the point cloud part labeled as leaf organ, such as Figure 3 The first principal component direction (the direction of the first principal component analysis) is the direction of leaf length, and the second principal component direction (the direction of the second principal component analysis) is the direction of leaf width.

[0091] Step S21 includes:

[0092] S211: Leaf length transformation: The leaf blades are density-scaled along principal component 1 (the direction of the first principal component analysis) to transform the leaf length, and the random scaling ratio of the leaf length transformation ranges from [0.5, 2].

[0093] S212: Leaf width transformation: The leaf blade performs point cloud density scaling along principal component 2 (the direction of the second principal component analysis) to transform the leaf width, and the leaf width transformation random scaling ratio range is [0.5, 2].

[0094] S213: Leaf angle transformation: A blade is rotated along the Z axis to transform the leaf inclination angle. The range of the random rotation angle of the leaf angle transformation is [-30°, 30°].

[0095] S22: Set the expected number of crop point clouds M after amplification, and perform The operation of step S21 is to obtain the transformed point cloud data , All organ point clouds in the file are merged into one point cloud data file, which is the first A data enhanced point cloud file, that is, enhanced point cloud data, is generated to complete the data enhancement of the crop point cloud, such as Figure 4 As shown. Among them, Figure 4 (a) Schematic diagram of Plant Augment transformation - leaf length reduced by 20%; Figure 4 (b) Schematic diagram of Plant Augment transformation - leaf width increased by 20%; Figure 4 (c) Schematic diagram of Plant Augment transformation - leaf inclination angle reduced by 20°.

[0096] Step 3: Based on the Soft Group algorithm, the crop point cloud instance segmentation network (point cloud instance segmentation model) completes the crop organ level fine instance segmentation, such as Figure 5 shown.

[0097] Step 3 specifically includes:

[0098] S31: bottom-up grouping stage, corresponding to Figure 5 Bottom-up grouping in

[15] . The point-by-point prediction network takes the point cloud (augmented point cloud data) as input and generates point-by-point semantic labels (point-by-point semantic scores) and offset vectors, and the soft grouping module processes these outputs to generate preliminary instance proposals.

[0099] Step S31 includes:

[0100] S311: Point-by-point prediction network. The network input is a set of points, each represented by its coordinates and color. The input point cloud data is first voxelized, converting the unordered points into an ordered volume mesh. The ordered volume mesh is fed into the U-Net backbone to obtain point features. From these point features, two branches are constructed to output point-by-point semantic scores and offsets.

[0101] Step S311 includes:

[0102] S3111: Semantic branch. This branch is constructed by a two-layer MLP and learns to output On a class Semantic score of each point The network directly groups the semantic scores instead of converting them into a one-hot encoded semantic prediction. is the total number of categories in the semantic segmentation task, which is used to indicate the range of semantic categories that the network model needs to predict.

[0103] S3112: Offset branch. This branch runs parallel to the semantic branch and uses a two-layer MLP to learn the offset vector , represents the vector from each point to the geometric center of the instance to which the point belongs. Based on the learned offset vector, the point is moved to the center of the corresponding instance to perform grouping more effectively. Cross entropy loss and regression loss They are used to train the semantic branch and bias branch respectively, and the calculation formulas are as follows:

[0104] .

[0105] .

[0106] in, It is The semantic label of the point, It means the The offset label of the vector from a point to the geometric center of the instance to which the point belongs. For the The predicted semantic score of each point; For the The predicted offset vector of points; For the Cross entropy loss of each point; is the L1 norm, which is used to measure the difference between the predicted offset vector and the true offset vector.

[0107] It is the indicator point Indicator function of whether it belongs to the current instance. The calculation formula is as follows:

[0108] .

[0109] S312: Soft grouping module. This module receives semantic scores (point-by-point semantic scores) and offsets as input and generates instance proposals. First, the offset vector (offset) is used to shift the points toward the corresponding instance center. Given a semantic score , iteration At each class index, we slice the subset of points (point cloud data set) of the entire scene with scores above a threshold and create links between points whose geometric distance is less than the grouping bandwidth to obtain instance proposals. For each iteration, grouping is performed on the subset of points in the entire scan to ensure fast reasoning.

[0110] S32: Top-down refinement phase, corresponding to Figure 5 The refinement module performs classification and refinement based on instance proposals, extracting corresponding features from the backbone and using them to predict the class, instance mask, and mask score as the final segmentation result to obtain segmentation information data.

[0111] Step S32 includes:

[0112] S321: Classify and refine the instance proposals output by the bottom-up grouping stage. A feature extractor processes each proposal to extract its corresponding backbone features. The extracted features are fed into a tiny U-Net network, which then predicts the classification score, instance mask, and mask score on subsequent branches.

[0113] Step S321 includes:

[0114] S3211: Classification branch. This branch first uses a global average pooling layer to aggregate the features of all points in the instance proposal. It then uses an MLP to predict the classification score. The object category and classification confidence score are directly derived from the output of the classification branch. The SoftGroup algorithm directly uses the output of the classification branch as the instance class. The classification branch aggregates all point features in the instance proposal and classifies the instance using a single label, resulting in more reliable predictions.

[0115] S3212: Segmentation branch. Instance proposals contain both foreground and background points. The SoftGroup algorithm constructs a segmentation branch to predict the instance mask for each instance proposal. This branch is a two-layer point-wise MLP that outputs an instance mask for each instance.

[0116] S3213: Mask scoring branch. This branch has the same structure as the classification branch. This branch outputs a mask score, which estimates the intersection-over-union ratio of the predicted mask to the true value. The mask score (mask score) is multiplied by the classification score (classification confidence score) to get the final confidence score.

[0117] S322: All the instances with true value Instance proposals with a probability of more than 50% are considered positive samples, and the rest are considered negative samples. Each positive sample is assigned to The highest true instance. The classification target of the positive sample is the category of the corresponding true instance. The segmentation branch and the mask scoring branch are only trained on positive samples. The mask target of the positive sample is the mask of the paired true instance. The target of the mask scoring is the difference between the predicted mask and the true mask. , and then complete instance segmentation.

[0118] Based on the same inventive concept, embodiments of the present application also provide a crop point cloud instance segmentation device for implementing the aforementioned crop point cloud instance segmentation method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more crop point cloud instance segmentation device embodiments provided below can be found in the limitations of the crop point cloud instance segmentation method described above and will not be further elaborated here.

[0119] In an exemplary embodiment, a crop point cloud instance segmentation apparatus is provided, comprising:

[0120] The data acquisition module is used to obtain a crop point cloud point set; the crop point cloud point set includes the number of points in the crop point cloud and the real coordinates of each point in three-dimensional space.

[0121] The data enhancement module is used to perform data enhancement processing on the crop point cloud according to the crop point cloud point set using the Plant Augment method to obtain enhanced point cloud data.

[0122] The segmentation module is used to perform organ-level instance segmentation of the crop point cloud based on the enhanced point cloud data based on the point cloud instance segmentation model to obtain segmentation information data; the segmentation information data includes: leaf organs and stem organs; the point cloud instance segmentation model is obtained by deep learning based on the crop point cloud point set with labeled data using the SoftGroup algorithm; the labeled data is the segmentation information data.

[0123] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, memory and input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store crop point cloud instance segmentation data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a crop point cloud instance segmentation method is implemented.

[0124] Those skilled in the art will understand that Figure 6 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application and does not constitute a limitation on the computer device to which the solution of the present application is applied. A specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the above-mentioned method embodiments when executing the computer program.

[0125] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0126] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0127] In this application, all actions to obtain signals, information, or data are performed in compliance with the relevant data protection laws and policies of the country in which they are located and with the authorization of the corresponding device owner. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, storage, and display, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of the relevant data must comply with relevant regulations.

[0128] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0129] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0130] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A crop point cloud instance segmentation method, characterized in that: The crop point cloud instance segmentation method comprises: Acquire a crop point cloud point set; the crop point cloud point set includes the number of points in the crop point cloud and the real coordinates of each point in three-dimensional space; Using a plant data enhancement method, performing data enhancement processing on the crop point cloud according to the crop point cloud point set to obtain enhanced point cloud data; Based on the point cloud instance segmentation model, organ-level instance segmentation of the crop point cloud is performed based on the enhanced point cloud data to obtain segmentation information data; the segmentation information data includes: leaf organs and stem organs; the point cloud instance segmentation model is obtained by using a soft grouping algorithm and performing deep learning on a crop point cloud point set with labeled data; the labeled data is the segmentation information data; The plant data enhancement method is used to perform data enhancement processing on the crop point cloud according to the crop point cloud point set to obtain enhanced point cloud data, specifically including: Performing principal component analysis on the crop point cloud point set to obtain a crop point cloud analysis point set; the principal component analysis includes a first principal component analysis and a second principal component analysis; the direction of the first principal component analysis is the direction of leaf length; the direction of the second principal component analysis is the direction of leaf width; Using a plant data enhancement method to set the desired number of crop point clouds after amplification, performing data transformation and data enhancement processing on the crop point cloud analysis point set to obtain enhanced point cloud data; The plant data enhancement method is used to set the expected number of crop point clouds after amplification, and the crop point cloud analysis point set is subjected to data transformation and data enhancement processing to obtain enhanced point cloud data, specifically including: Use plant data augmentation methods to set the expected number of crop point clouds after augmentation , the crop point cloud analysis point set is sequentially Transform the point cloud data by performing the transformations; each transformation includes: leaf length transformation, leaf width transformation and leaf angle transformation; Merge all transformed point cloud data to obtain enhanced point cloud data; The leaf length transformation is to scale the point cloud density along the direction of the first principal component analysis; the leaf width transformation is to scale the point cloud density along the direction of the second principal component analysis; the leaf angle transformation is to transform the leaf inclination angle by rotating along the z-axis in three-dimensional space; The scaling ratio interval corresponding to the leaf length transformation is [0.5, 2]; The scaling ratio interval corresponding to the leaf width transformation is [0.5, 2]; The rotation angle interval corresponding to the leaf angle transformation is [-30°, 30°].

2. The crop point cloud instance segmentation method according to claim 1, characterized in that: The point cloud instance segmentation model includes a bottom-up grouping module and a top-down refinement module; the grouping module and the refinement module are connected; Wherein, the grouping module includes a point-by-point prediction network and a soft grouping module connected in sequence; The point-by-point prediction network is used to: voxelization is performed on the enhanced point cloud data to convert disordered points into ordered volume meshes; Based on the U-Net backbone network, point features are determined according to the ordered volume grid; Constructing branches based on point features to output point-by-point semantic scores and offsets; the branches include: semantic branches and offset branches; The soft grouping module is used to receive the point-by-point semantic score and the offset and shift according to the offset, and iterate based on the set semantic score according to the point-by-point semantic score. Class, at each class index, slice the point cloud data set with point-by-point semantic scores higher than the threshold, and create links between points whose geometric distance is less than the set grouping bandwidth to obtain instance proposals; The refinement module is used to classify and refine the instance proposals to obtain segmentation information data.

3. The crop point cloud instance segmentation method according to claim 2, characterized in that: In the refinement module, classification and refinement processing are performed based on the classification branch, the segmentation branch, and the mask scoring branch; The classification branch uses a global average pooling layer to aggregate the features of all points in the instance proposal, and uses MLP to predict the classification score, outputting the category and classification confidence score; The segmentation branch uses a two-layer point-by-point MLP to perform instance mask prediction on the instance proposal to obtain the instance mask; The mask scoring branch is used to determine a mask score based on the instance proposal; The final confidence score is determined based on the mask score and the classification confidence score; According to the final confidence score, an instance mask corresponding to a set proportion is selected from the instance masks as the segmentation information data.

4. The crop point cloud instance segmentation method according to claim 1, characterized in that: The method for determining the point cloud instance segmentation model specifically includes: Acquire training data; the training data includes: a training crop point cloud point set and corresponding label data; The plant data enhancement method is used to perform data enhancement processing on the training crop point cloud point set to obtain enhanced training point cloud data; Build a point cloud instance segmentation network; Inputting the training data into the point cloud instance segmentation network, and performing deep learning training on the point cloud instance segmentation network using a soft grouping algorithm to obtain a trained point cloud instance segmentation network; The trained point cloud instance segmentation network is determined as the point cloud instance segmentation model.

5. A crop point cloud instance segmentation device, characterized in that: The crop point cloud instance segmentation device comprises: A data acquisition module is used to acquire a crop point cloud point set; the crop point cloud point set includes the number of points in the crop point cloud and the real coordinates of each point in three-dimensional space; a data enhancement module, configured to perform data enhancement processing on the crop point cloud according to the crop point cloud point set using a plant data enhancement method to obtain enhanced point cloud data; a segmentation module for performing organ-level instance segmentation of the crop point cloud based on the enhanced point cloud data based on a point cloud instance segmentation model to obtain segmentation information data; the segmentation information data includes leaf organs and stem organs; the point cloud instance segmentation model is obtained by deep learning based on a crop point cloud point set with labeled data using a soft grouping algorithm; the labeled data is the segmentation information data; The plant data enhancement method is used to perform data enhancement processing on the crop point cloud according to the crop point cloud point set to obtain enhanced point cloud data, specifically including: Performing principal component analysis on the crop point cloud point set to obtain a crop point cloud analysis point set; the principal component analysis includes a first principal component analysis and a second principal component analysis; the direction of the first principal component analysis is the direction of leaf length; the direction of the second principal component analysis is the direction of leaf width; Using a plant data enhancement method to set the desired number of crop point clouds after amplification, performing data transformation and data enhancement processing on the crop point cloud analysis point set to obtain enhanced point cloud data; The plant data enhancement method is used to set the expected number of crop point clouds after amplification, and the crop point cloud analysis point set is subjected to data transformation and data enhancement processing to obtain enhanced point cloud data, specifically including: Use plant data augmentation methods to set the expected number of crop point clouds after augmentation , the crop point cloud analysis point set is sequentially Transform the leaf to obtain the transformed point cloud data; each transformation includes: leaf length transformation, leaf width transformation and leaf angle transformation; Merge all transformed point cloud data to obtain enhanced point cloud data; The leaf length transformation is to scale the point cloud density along the direction of the first principal component analysis; the leaf width transformation is to scale the point cloud density along the direction of the second principal component analysis; the leaf angle transformation is to transform the leaf inclination angle by rotating along the z-axis in three-dimensional space; The scaling ratio interval corresponding to the leaf length transformation is [0.5, 2]; The scaling ratio interval corresponding to the leaf width transformation is [0.5, 2]; The rotation angle interval corresponding to the leaf angle transformation is [-30°, 30°].

6. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the crop point cloud instance segmentation method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the crop point cloud instance segmentation method according to any one of claims 1 to 4 is implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the crop point cloud instance segmentation method according to any one of claims 1 to 4 is implemented.

Images