Crop point cloud instance segmentation method, device, equipment, medium and product

Abstract

The invention discloses a crop point cloud instance segmentation method, device and equipment, a medium and a product, and relates to the technical field of agricultural informatics. The method comprises the following steps: acquiring a crop point cloud point set; the crop point cloud point set comprises the number of points of the crop point cloud and real number coordinates of each point in a three-dimensional space; performing data enhancement processing on the crop point cloud according to the crop point cloud point set by adopting a plant data enhancement method to obtain enhanced point cloud data; based on a point cloud instance segmentation model, performing organ-level instance segmentation of the crop point cloud according to the enhanced point cloud data to obtain segmentation information data; the segmentation information data comprises leaf organs and stalk organs; the point cloud instance segmentation model is obtained by adopting a soft grouping algorithm and performing deep learning based on a crop point cloud point set with label data; the label data is the segmentation information data. The objective of the invention is to realize organ-level accurate instance segmentation of crop point clouds.

Description

Technical Field

[0001] The present application relates to the technical field of agricultural informatics, and particularly to a method, device, equipment, medium and product for crop point cloud instance segmentation. Background Art

[0002] To address the challenge of accurately segmenting crop point clouds due to occlusion and intersection, researchers have proposed many automated crop point cloud instance segmentation methods, aiming to reduce the dependence on manual parameter settings and improve the robustness to complex scenarios. One common method is the instance segmentation algorithm based on deep learning.

[0003] However, the currently adopted methods usually require a large amount of point cloud data for model training. Traditional point cloud data augmentation methods only perform simple operations such as rotation, translation, and scaling on point cloud data. Although they can increase the diversity of data to a certain extent, they still have limitations and are difficult to fully exploit the data potential.

[0004] Therefore, it is crucial to effectively improve the leaf instance segmentation ability of the model for complex crops and occlusion situations to enhance the accuracy and robustness of crop point cloud instance segmentation. Summary of the Invention

[0005] The purpose of the present application is to provide a method, device, equipment, medium and product for crop point cloud instance segmentation, which can achieve organ-level accurate instance segmentation of crop point clouds.

[0006] To achieve the above purpose, the present application provides the following solutions: In the first aspect, the present application provides a method for crop point cloud instance segmentation, including: Obtaining a crop point cloud point set; the crop point cloud point set includes the number of points in the crop point cloud and the real coordinates of each point in three-dimensional space; Using a plant data augmentation method to perform data augmentation processing on the crop point cloud according to the crop point cloud point set to obtain enhanced point cloud data; Based on a point cloud instance segmentation model, performing organ-level instance segmentation on the crop point cloud according to the enhanced point cloud data to obtain segmentation information data; the segmentation information data includes: leaf organs and stem organs; the point cloud instance segmentation model is obtained by performing deep learning on a crop point cloud point set with labeled data using a soft grouping algorithm; the labeled data is the segmentation information data.

[0007] Optionally, using a plant data augmentation method to perform data augmentation processing on the crop point cloud according to the crop point cloud point set to obtain enhanced point cloud data, specifically including: Perform principal component analysis on the crop point cloud point set to obtain a crop point cloud analysis point set; the principal component analysis includes a first principal component analysis and a second principal component analysis; the direction of the first principal component analysis is the leaf length direction; the direction of the second principal component analysis is the leaf width direction; Adopt a plant data augmentation method to perform data transformation and data augmentation processing on the crop point cloud analysis point set with the expected number of crop point clouds after setting the amplification, so as to obtain enhanced point cloud data.

[0008] Optionally, adopt a plant data augmentation method to perform data transformation and data augmentation processing on the crop point cloud analysis point set with the expected number of crop point clouds after setting the amplification, so as to obtain enhanced point cloud data, which specifically includes: Adopt a plant data augmentation method with the expected number of crop point clouds after setting the amplification to perform the following operations on the crop point cloud analysis point set in sequence times of transformation to obtain transformed point cloud data; each transformation includes: leaf length transformation, leaf width transformation and leaf angle transformation; Merge all the transformed point cloud data to obtain enhanced point cloud data; Among them, the leaf length transformation is to perform point cloud density scaling along the direction of the first principal component analysis; the leaf width transformation is to perform point cloud density scaling along the direction of the second principal component analysis; the leaf angle transformation is to perform rotation along the z-axis in three-dimensional space to transform the leaf inclination angle; The scaling ratio range corresponding to the leaf length transformation is [0.5, 2]; The scaling ratio range corresponding to the leaf width transformation is [0.5, 2]; The rotation angle range corresponding to the leaf angle transformation is [-30°, 30°].

[0009] Optionally, the point cloud instance segmentation model includes a bottom-up grouping module and a top-down refinement module; the grouping module is connected to the refinement module; Among them, the grouping module includes a point-by-point prediction network and a soft grouping module connected in sequence; The point-by-point prediction network is used for: Perform voxelization processing on the enhanced point cloud data to convert disordered points into an ordered volume grid; Based on the U-Net backbone network, determine point features according to the ordered volume grid; Construct branches according to the point features to output point-by-point semantic scores and offsets; the branches include: a semantic branch and an offset branch; The soft grouping module is used to receive the point-by-point semantic scores and offsets, shift according to the offsets, and based on the point-by-point semantic scores, iterate based on the set semantic scores For each class, at each class index, slice the set of point cloud data with pointwise semantic scores higher than the threshold, and create links between points with geometric distances less than the set grouping bandwidth to obtain instance proposals; The refinement module is used to classify and refine the instance proposals to obtain segmentation information data.

[0010] Optionally, in the refinement module, classification and refinement are performed based on a classification branch, a segmentation branch, and a mask scoring branch; Among them, the classification branch uses a global average pooling layer to aggregate the features of all points in the instance proposal, and uses an MLP to predict the classification score, outputting the class and the classification confidence score; The segmentation branch uses a two-layer pointwise MLP to perform instance mask prediction on the instance proposal to obtain an instance mask; The mask scoring branch is used to determine the mask score according to the instance proposal; The final confidence score is determined based on the mask score and the classification confidence score; Select the instance mask corresponding to the set proportion in the instance mask according to the final confidence score as the segmentation information data.

[0011] Optionally, the method for determining the point cloud instance segmentation model specifically includes: Obtain training data; the training data includes: the training crop point cloud point set and the corresponding label data; Adopt the plant data augmentation method to perform data augmentation processing on the training crop point cloud point set to obtain the augmented training point cloud data; Construct a point cloud instance segmentation network; Input the training data into the point cloud instance segmentation network, and use the soft grouping algorithm to perform deep learning training on the point cloud instance segmentation network to obtain the trained point cloud instance segmentation network; Determine the trained point cloud instance segmentation network as the point cloud instance segmentation model.

[0012] In a second aspect, the present application provides a crop point cloud instance segmentation device, including: A data acquisition module, configured to acquire a crop point cloud point set; the crop point cloud point set includes the number of points in the crop point cloud and the real coordinates of each point in three-dimensional space; A data augmentation module, configured to use the plant data augmentation method to perform data augmentation processing on the crop point cloud according to the crop point cloud point set to obtain the augmented point cloud data; A segmentation module, which is used to perform organ-level instance segmentation of crop point clouds based on a point cloud instance segmentation model according to the enhanced point cloud data, and obtain segmentation information data; the segmentation information data includes: leaf organs and stem organs; the point cloud instance segmentation model is obtained by performing deep learning on a crop point cloud point set with labeled data by using a soft grouping algorithm; the labeled data is the segmentation information data.

[0013] In a third aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the above-mentioned crop point cloud instance segmentation method.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the above-mentioned crop point cloud instance segmentation method.

[0015] In a fifth aspect, the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the above-mentioned crop point cloud instance segmentation method.

[0016] According to the specific embodiments provided by the present application, the present application has the following technical effects: The present application provides a crop point cloud instance segmentation method, device, equipment, medium and product. By using a plant data enhancement method, data enhancement processing of crop point clouds is performed according to a crop point cloud point set to obtain enhanced point cloud data. The feature information of point cloud instances can be used to design an enhancement strategy, providing data diversity for subsequent point cloud instance segmentation, giving full play to the data potential, and improving the performance and robustness of the point cloud instance segmentation model. Since the point cloud instance segmentation model is obtained by performing deep learning on a crop point cloud point set with labeled data by using a soft grouping algorithm, it can usually better adapt to different types and complexities of crop scenes and has higher accuracy and robustness. The soft grouping algorithm is used to achieve precise segmentation of crop point cloud data in complex scenes. Therefore, the present application can achieve precise organ-level instance segmentation of crop point clouds. Description of the Drawings

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the following-described drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 It is a flowchart of the crop point cloud instance segmentation method; Figure 2 In practical applications, it is the operation flowchart of the crop point cloud instance segmentation method; Figure 3 It is the schematic diagram of the principal component analysis of the leaf-based point cloud; Figure 4 It is the schematic diagram of data augmentation using the Plant Augment method; among them, Figure 4 (a) is the schematic diagram of the PlantAugment transformation - the leaf length is reduced by 20%; Figure 4 (b) is the schematic diagram of the Plant Augment transformation - the leaf width is increased by 20%; Figure 4 (c) is the schematic diagram of the Plant Augment transformation - the leaf inclination angle is reduced by 20°; Figure 5 It is the schematic diagram of the structure of the point cloud instance segmentation model using the Soft Group algorithm; Figure 6 It is the schematic diagram of the structure of a computer device provided by an embodiment of the present application. Specific implementation manners

[0019] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0020] The principle of point cloud segmentation based on deep learning is to automatically learn features from point cloud data using a deep neural network and classify the point cloud into different groups or categories according to these features. This method can usually better adapt to different types and complexities of crop scenes and has higher accuracy and robustness. The point cloud instance segmentation model using the soft grouping algorithm (Soft Group) realizes the accurate segmentation of crop point cloud data in complex scenes through soft grouping and top-down refinement modules (refinement modules). The present application applies it to the crop instance segmentation scenario, aiming to obtain crop point cloud organ instances with higher accuracy, so as to provide reliable technical support for the automated calculation of crop phenotype data.

[0021] In addition, this application proposes a point cloud augmentation algorithm called Plant Augment, which augments point cloud data based on point cloud instance information, aiming to improve the performance and robustness of deep learning-based point cloud instance segmentation algorithms. The core idea of Plant Augment is to use the feature information of point cloud instances to design augmentation strategies, including three operations: leaf length change, leaf width change, and leaf angle change, so as to generate richer and more representative training data. This augmentation method can effectively improve the leaf instance segmentation ability of the model for complex crops and occlusion situations, and enhance the generalization ability and robustness of the model.

[0022] The Soft Group algorithm combines the Plant Augment point cloud augmentation technology, which can effectively overcome the limitations of traditional methods, improve the accuracy and robustness of crop point cloud instance segmentation, and then calculate crop phenotype data more precisely, providing more reliable technical support for applications in the agricultural field.

[0023] To make the above objects, features, and advantages of this application more obvious and understandable, the following further details this application in conjunction with the accompanying drawings and specific embodiments.

[0024] In an exemplary embodiment, as Figure 1 shown, a method for crop point cloud instance segmentation is provided. This method is executed by a computer device, which can be specifically executed by a computer device such as a terminal or a server alone, or jointly executed by a terminal and a server. In the embodiments of this application, taking the application of this method to a server as an example for illustration, it includes the following steps.

[0025] As Figure 1 shown, the method for crop point cloud instance segmentation provided by the embodiments of this application includes: Step 100: Obtain a crop point cloud point set. The crop point cloud point set includes the number of points in the crop point cloud and the real coordinates of each point in three-dimensional space.

[0026] Step 200: Use the plant data augmentation method to perform data augmentation processing on the crop point cloud according to the crop point cloud point set to obtain augmented point cloud data.

[0027] Step 300: Based on the point cloud instance segmentation model, perform organ-level instance segmentation on the crop point cloud according to the augmented point cloud data to obtain segmentation information data. The segmentation information data includes: leaf organs and stem organs; the point cloud instance segmentation model is obtained by deep learning based on the crop point cloud point set with label data using the soft grouping algorithm; the label data is the segmentation information data.

[0028] In one embodiment, a plant data augmentation method is adopted to perform data augmentation processing on the crop point cloud according to the crop point cloud point set, and the augmented point cloud data is obtained, which specifically includes: Perform principal component analysis on the crop point cloud point set to obtain the crop point cloud analysis point set; the principal component analysis includes the first principal component analysis and the second principal component analysis; the direction of the first principal component analysis is the leaf length direction; the direction of the second principal component analysis is the leaf width direction.

[0029] Adopt the plant data augmentation method to perform data transformation and data augmentation processing on the crop point cloud analysis point set with the expected number of crop point clouds after setting the amplification, so as to obtain the augmented point cloud data.

[0030] As an alternative implementation, adopt the plant data augmentation method to perform data transformation and data augmentation processing on the crop point cloud analysis point set with the expected number of crop point clouds after setting the amplification, so as to obtain the augmented point cloud data, which specifically includes: Adopt the plant data augmentation method with the expected number of crop point clouds after setting the amplification , and perform times of transformation on the crop point cloud analysis point set in sequence to obtain the transformed point cloud data; each transformation includes: leaf length transformation, leaf width transformation and leaf angle transformation.

[0031] Merge all the transformed point cloud data to obtain the augmented point cloud data.

[0032] Among them, the leaf length transformation is to perform point cloud density scaling along the direction of the first principal component analysis; the leaf width transformation is to perform point cloud density scaling along the direction of the second principal component analysis; the leaf angle transformation is to rotate along the z-axis in three-dimensional space to transform the leaf inclination angle.

[0033] The scaling ratio interval corresponding to the leaf length transformation is [0.5, 2]. The scaling ratio interval corresponding to the leaf width transformation is [0.5, 2]. The rotation angle interval corresponding to the leaf angle transformation is [-30°, 30°].

[0034] The point cloud instance segmentation model includes a bottom-up grouping module and a top-down refinement module; the grouping module is connected to the refinement module; among them, the grouping module includes a point-by-point prediction network and a soft grouping module connected in sequence.

[0035] The point-by-point prediction network is used to perform voxelization processing on the augmented point cloud data to convert the disordered points into an ordered volume grid; the point-by-point prediction network is used to determine the point features based on the U-Net backbone network according to the ordered volume grid; the point-by-point prediction network is used to construct branches according to the point features to output the point-by-point semantic score and offset; the branches include: a semantic branch and an offset branch.

[0036] The soft grouping module is used to receive per-point semantic scores and offsets, shift according to the offsets, and iteratively based on the per-point semantic scores and a set semantic score, for a number of classes. At each class index, slice the set of point cloud data with per-point semantic scores higher than a threshold, and create links between points with geometric distances less than a set grouping bandwidth to obtain instance proposals.

[0037] The refinement module is used to classify and refine the instance proposals to obtain segmentation information data.

[0038] Specifically, in the refinement module, classification and refinement are performed based on a classification branch, a segmentation branch, and a mask scoring branch.

[0039] Among them, the classification branch uses a global average pooling layer to aggregate the features of all points in the instance proposals, and uses a Multilayer Perceptron (MLP) to predict classification scores, outputting the class and classification confidence scores.

[0040] The segmentation branch uses a two-layer per-point MLP to perform instance mask prediction on the instance proposals to obtain instance masks; the mask scoring branch is used to determine mask scores according to the instance proposals; the final confidence score is determined based on the mask scores and the classification confidence scores; a set proportion of the corresponding instance masks are selected from the instance masks according to the final confidence score as the segmentation information data.

[0041] In an embodiment, a method for determining a point cloud instance segmentation model specifically includes: Obtain training data; the training data includes: a set of training crop point clouds and corresponding label data.

[0042] Adopt a plant data augmentation method to perform data augmentation processing on the set of training crop point clouds to obtain augmented training point cloud data; construct a point cloud instance segmentation network; input the training data into the point cloud instance segmentation network, and use a soft grouping algorithm to perform deep learning training on the point cloud instance segmentation network to obtain a trained point cloud instance segmentation network; determine the trained point cloud instance segmentation network as the point cloud instance segmentation model.

[0043] This application combines a data augmentation method for crop point cloud instances, namely the PlantAugment method, to complete crop point cloud augmentation; then, based on a crop point cloud instance segmentation model using the Soft Group algorithm, it completes organ-level fine instance segmentation. This application can accurately complete organ-level instance segmentation of various crops.

[0044] In practical applications, such as Figure 2As shown, the crop point cloud instance segmentation method based on Soft Group specifically includes: Step 1: Input the labeled crop point cloud point set with instance labels, that is, the point set with labeled data. The instance labels are divided into two categories: leaf organs and stem organs. The crop point cloud point set can be expressed as , where refers to the number of points in the input point cloud, is the real number space, and each point in the three-dimensional space consists of three real number coordinates.

[0045] Step 2: Combine the crop point cloud instances and complete the crop point cloud enhancement based on the Plant Augment method.

[0046] Among them, Step 2 includes: S21: Conduct principal component analysis on the point cloud part with the label of leaf organs, as Figure 3 shown. Among them, the first principal component direction (the direction of the first principal component analysis) is the leaf length direction, and the second principal component direction (the direction of the second principal component analysis) is the leaf width direction.

[0047] Step S21 includes: S211: Leaf length transformation: Sequentially perform point cloud density scaling on the th leaf along the principal component 1 (the direction of the first principal component analysis) to transform the leaf length. The random scaling ratio range of the leaf length transformation is [0.5, 2].

[0048] S212: Leaf width transformation: Sequentially perform point cloud density scaling on the th leaf along the principal component 2 (the direction of the second principal component analysis) to transform the leaf width. The random scaling ratio range of the leaf width transformation is [0.5, 2].

[0049] S213: Leaf angle transformation: Sequentially perform rotation on the th leaf along the Z-axis to transform the leaf inclination angle. The random rotation angle range of the leaf angle transformation is [-30°, 30°].

[0050] S22: Set the expected number of crop point clouds M after amplification, and perform the operations of Step S21 on the input crop point cloud point set P times in sequence to obtain the transformed point cloud data . All the organ point clouds in are merged into a point cloud data save file, which is the Figure 4 th data-enhanced point cloud file after data enhancement, that is, the enhanced point cloud data, thus completing the data enhancement of the crop point cloud, as Figure 4(a) Schematic diagram of the Plant Augment transformation - leaf length reduced by 20%; Figure 4 (b) Schematic diagram of the Plant Augment transformation - leaf width increased by 20%; Figure 4 (c) Schematic diagram of the Plant Augment transformation - leaf inclination angle reduced by 20°.

[0051] Step 3: Based on the Soft Group algorithm for crop point cloud instance segmentation network (point cloud instance segmentation model), complete the fine instance segmentation at the crop organ level, as Figure 5 shown.

[0052] Step 3 specifically includes: S31: Bottom-up grouping stage, corresponding to Figure 5 the bottom-up grouping in. The point-by-point prediction network takes the point cloud (enhanced point cloud data) as input and generates point-by-point semantic labels (point-by-point semantic scores) and offset vectors. The soft grouping module processes these outputs to generate preliminary instance proposals.

[0053] Step S31 includes: S311: Point-by-point prediction network. The input of this network is a set of points, and each point is represented by its coordinates and color. The input point cloud data is first voxelized to convert the unordered points into an ordered volume grid. The ordered volume grid is fed into the U-Net backbone network to obtain point features. Then, starting from the point features, two branches are constructed to output the point-by-point semantic scores and offsets.

[0054] Step S311 includes: S3111: Semantic branch. This branch is constructed by a two-layer MLP and learns to output semantic scores for points on classes. This network directly groups the semantic scores instead of converting the semantic scores into a one-hot semantic prediction. is the total number of classes in the semantic segmentation task, which is used to represent the range of semantic classes that the network model needs to predict.

[0055] S3112: Offset branch. This branch is parallel to the semantic branch and uses a two-layer MLP to learn the offset vector , which represents the vector from each point to the geometric center of the instance to which the point belongs. Based on the learned offset vector, the points are moved to the center of the corresponding instance to perform grouping more effectively. Cross-entropy loss and regression loss are used to train the semantic branch and the bias branch respectively, and the calculation formulas are as follows: .

[0056] 。

[0057] Among them, is the semantic label of the th point, is the offset label of the vector representing the offset from the th point to the geometric center of the instance to which the point belongs, is the predicted semantic score of the th point; is the predicted offset vector of the th point; is the cross-entropy loss of the th point; is the L1 norm, which is used to measure the difference between the predicted offset vector and the true offset vector.

[0058] is an indicator function that indicates whether the point belongs to the current instance, and the calculation formula is as follows: 。

[0059] S312: Soft grouping module. This module receives semantic scores (per-point semantic scores) and offsets as inputs and generates instance proposals. First, the points are shifted towards the corresponding instance centers using the offset vectors (offsets). Given the semantic score , iterate through classes. At each class index, slice the subset of points in the entire scene with scores higher than the threshold (the point cloud data set), and create links between points with a geometric distance less than the grouping bandwidth to obtain instance proposals. For each iteration, group the subset of points in the entire scan to ensure fast inference.

[0060] S32: Top-down refinement stage, corresponding to the top-down refinement in Figure 5 . Based on the refinement module, classify and refine according to the instance proposals, that is, extract the corresponding features from the backbone and use them to predict the class, instance mask, and mask score as the final segmentation result to obtain the segmentation information data.

[0061] Step S32 includes: S321: Classify and refine the instance proposals output by the bottom-up grouping stage. A feature extractor processes each proposal to extract its corresponding backbone features. The extracted features are fed into a tiny U-Net network, and then the classification score, instance mask, and mask score are predicted on the subsequent branches.

[0062] Step S321 includes: S3211: Classification Branch. This branch first uses a global average pooling layer to aggregate the features of all points in the instance proposal, and then uses an MLP to predict the classification scores. The object category and classification confidence scores are directly obtained from the output of the classification branch, that is, the classification branch outputs the category and classification confidence scores. The SoftGroup algorithm directly uses the output of the classification branch as the instance class. The classification branch aggregates all point features of the instance proposal and classifies the instance with a single label, thus obtaining more reliable predictions.

[0063] S3212: Segmentation Branch. The instance proposal contains both foreground points and background points. The SoftGroup algorithm constructs a segmentation branch to predict the instance mask in each instance proposal. This branch is a two-layer point-wise MLP, which outputs the instance mask for each instance.

[0064] S3213: Mask Scoring Branch. This branch has the same structure as the classification branch. This branch outputs a mask score, which estimates the intersection over union of the predicted mask and the ground truth. . Multiply the mask score (mask score) by the classification score (classification confidence score) to obtain the final confidence score.

[0065] S322: Consider all instance proposals with an intersection over union higher than 50% with the ground truth instances as positive samples, and the rest as negative samples. Each positive sample is assigned to the highest ground truth instance. The classification target of the positive sample is the category of the corresponding ground truth instance. The segmentation branch and the mask scoring branch are only trained on positive samples. The mask target of the positive sample is the mask of the paired ground truth instance. The target of mask scoring is the between the predicted mask and the ground truth mask, thus completing instance segmentation.

[0066] Based on the same inventive concept, the embodiment of the present application also provides a crop point cloud instance segmentation device for implementing the above-mentioned crop point cloud instance segmentation method. The implementation solutions provided by this device to solve problems are similar to the implementation solutions described in the above method. Therefore, the specific limitations in one or more embodiments of the following crop point cloud instance segmentation device can refer to the limitations on the crop point cloud instance segmentation method in the above text, and will not be repeated here.

[0067] In an exemplary embodiment, a crop point cloud instance segmentation device is provided, including: A data acquisition module, configured to acquire a crop point cloud point set; the crop point cloud point set includes the number of points in the crop point cloud and the real coordinates of each point in three-dimensional space.

[0068] A data augmentation module, which is used to perform data augmentation processing on crop point clouds according to the crop point cloud point set by using the Plant Augment method to obtain the augmented point cloud data.

[0069] A segmentation module, which is used to perform organ-level instance segmentation on crop point clouds based on a point cloud instance segmentation model according to the augmented point cloud data to obtain segmentation information data; the segmentation information data includes: leaf organs and stem organs; the point cloud instance segmentation model is obtained by deep learning based on the crop point cloud point set with labeled data by using the SoftGroup algorithm; the labeled data is the segmentation information data.

[0070] In an exemplary embodiment, a computer device is provided. The computer device can be a server or a terminal, and its internal structure diagram can be as Figure 6 shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store crop point cloud instance segmentation data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a method for crop point cloud instance segmentation.

[0071] Those skilled in the art can understand that Figure 6 the structure shown in

[0072] is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, it implements the steps in the above method embodiments.

[0073] In an exemplary embodiment, a computer program product is provided, including a computer program which, when executed by a processor, implements the steps in the above method embodiments.

[0074] In this application, all actions of obtaining signals, information or data are carried out on the premise of complying with the corresponding data protection regulations and policies of the country where it is located and obtaining the authorization given by the owner of the corresponding device. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant regulations.

[0075] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. Among them, any reference to a memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memories can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0076] In each of the embodiments provided in this application, the database involved may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., and is not limited thereto. In each of the embodiments provided in this application, the processor may be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., and is not limited thereto.

[0077] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0078] In this article, specific examples are used to elaborate on the principles and implementation manners of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A method for instance segmentation of crop point clouds, characterized in that, The crop point cloud instance segmentation method includes: Obtaining a crop point cloud point set; the crop point cloud point set includes the number of points in the crop point cloud and the real coordinates of each point in three-dimensional space; Using a plant data augmentation method, performing data augmentation processing on the crop point cloud according to the crop point cloud point set to obtain augmented point cloud data; Based on a point cloud instance segmentation model, performing organ-level instance segmentation on the crop point cloud according to the augmented point cloud data to obtain segmentation information data; the segmentation information data includes: leaf organs and stem organs; the point cloud instance segmentation model is obtained by performing deep learning on the crop point cloud point set with labeled data using a soft grouping algorithm; the labeled data is the segmentation information data.

2. The crop point cloud instance segmentation method according to claim 1, characterized in that Using a plant data augmentation method, performing data augmentation processing on the crop point cloud according to the crop point cloud point set to obtain augmented point cloud data, specifically including: Performing principal component analysis on the crop point cloud point set to obtain a crop point cloud analysis point set; the principal component analysis includes a first principal component analysis and a second principal component analysis; the direction of the first principal component analysis is the leaf length direction; the direction of the second principal component analysis is the leaf width direction; Using a plant data augmentation method, with the expected number of crop point clouds after setting the amplification, performing data transformation and data augmentation processing on the crop point cloud analysis point set to obtain augmented point cloud data.

3. The method for segmenting crop point cloud instances according to claim 2, wherein Using a plant data augmentation method, with the expected number of crop point clouds after setting the amplification, performing data transformation and data augmentation processing on the crop point cloud analysis point set to obtain augmented point cloud data, specifically including: Adopt a plant data augmentation method to set the expected number of crop point clouds in the later stage of amplification , and perform the following operations on the analysis point set of the crop point cloud in sequence transformations to obtain the transformed point cloud data; each transformation includes: leaf length transformation, leaf width transformation, and leaf angle transformation; Merging all the transformed point cloud data to obtain augmented point cloud data; Among them, the leaf length transformation is to perform point cloud density scaling along the direction of the first principal component analysis; the leaf width transformation is to perform point cloud density scaling along the direction of the second principal component analysis; the leaf angle transformation is to perform rotation along the z-axis in three-dimensional space to transform the leaf inclination angle; The scaling ratio interval range corresponding to the leaf length transformation is [0.5, 2]; The scaling ratio interval range corresponding to the leaf width transformation is [0.5, 2]; The rotation angle interval range corresponding to the leaf angle transformation is [-30°, 30°].

4. The crop point cloud instance segmentation method according to claim 1, wherein The point cloud instance segmentation model includes a bottom-up grouping module and a top-down refinement module; the grouping module and the refinement module are connected; Among them, the grouping module includes a point-by-point prediction network and a soft grouping module connected in sequence; The point-by-point prediction network is used for: Performing voxelization processing on the augmented point cloud data to convert disordered points into an ordered volume grid; Based on the U-Net backbone network, determining point features according to the ordered volume grid; Constructing branches according to the point features to output point-by-point semantic scores and offsets; the branches include: a semantic branch and an offset branch; The soft grouping module is used to receive per-point semantic scores and offsets, shift according to the offsets, and based on the per-point semantic scores, iterate based on a set semantic score classes. At each class index, slice the set of point cloud data with per-point semantic scores higher than the threshold, and create links between points with a geometric distance less than the set grouping bandwidth to obtain instance proposals; The refinement module is used for classifying and refining the instance proposals to obtain segmentation information data.

5. The crop point cloud instance segmentation method according to claim 4, characterized in that In the refinement module, classification and refinement processing are performed based on a classification branch, a segmentation branch, and a mask scoring branch; Among them, the classification branch uses a global average pooling layer to aggregate the features of all points in the instance proposal, and uses an MLP to predict the classification scores, outputting the category and the classification confidence score; The segmentation branch uses a two-layer pointwise MLP to perform instance mask prediction on the instance proposal, obtaining an instance mask; The mask scoring branch is used to determine the mask score according to the instance proposal; The final confidence score is determined based on the mask score and the classification confidence score; According to the final confidence score, the instance mask corresponding to a set ratio is selected from the instance mask as the segmentation information data.

6. The method for segmenting crop point cloud instances according to claim 1, wherein The method for determining the point cloud instance segmentation model specifically includes: Obtaining training data; the training data includes: the training crop point cloud point set and the corresponding label data; Using a plant data augmentation method to perform data augmentation processing on the training crop point cloud point set to obtain the augmented training point cloud data; Constructing a point cloud instance segmentation network; Inputting the training data into the point cloud instance segmentation network, and using a soft grouping algorithm to perform deep learning training on the point cloud instance segmentation network to obtain the trained point cloud instance segmentation network; Determining the trained point cloud instance segmentation network as the point cloud instance segmentation model.

7. A crop point cloud instance segmentation device, characterized in that, The crop point cloud instance segmentation device includes: A data acquisition module for acquiring a crop point cloud point set; the crop point cloud point set includes the number of points in the crop point cloud and the real coordinates of each point in three-dimensional space; A data augmentation module for using a plant data augmentation method to perform data augmentation processing on the crop point cloud according to the crop point cloud point set to obtain the augmented point cloud data; A segmentation module for performing organ-level instance segmentation of the crop point cloud based on the point cloud instance segmentation model according to the augmented point cloud data to obtain segmentation information data; the segmentation information data includes: leaf organs and stem organs; the point cloud instance segmentation model is obtained by performing deep learning based on the crop point cloud point set with label data using a soft grouping algorithm; the label data is the segmentation information data.

8. A computer device, comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the crop point cloud instance segmentation method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the crop point cloud instance segmentation method according to any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the crop point cloud instance segmentation method according to any one of claims 1-6.

Images