A zero-annotation target segmentation method and system based on point cloud data optimization

Abstract

The application discloses a zero-label target segmentation method and system based on point cloud data optimization. The method comprises the following steps: obtaining optimized actual production scene target point cloud data by optimizing the actual production scene target point cloud data to be processed through a point cloud data optimization method based on self-supervised training; the optimization is performed from the aspects of structural integrity, data density and data smoothness; training a standard point cloud segmentation network based on labeled indoor target point cloud data to obtain a trained standard point cloud segmentation model; the labeled indoor target point cloud data has the characteristics of complete structural information of a target, high data density and high data smoothness; inputting the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model, obtaining the segmentation result of different parts of the target, and obtaining related phenotype parameters. The point cloud data labeling problem of the actual production scene is avoided, and the ability of accurately segmenting different parts of the target is ensured.

Description

Technical Field

[0001] The present invention relates to the technical fields of deep learning, point cloud data optimization processing and target segmentation, and in particular to a zero-labeled target segmentation method and system based on point cloud data optimization. Background Art

[0002] Currently, computer vision-based target phenotypic parameter calculations are mostly based on two-dimensional image analysis, with relevant parameters acquired through algorithms such as pattern recognition, machine learning, and deep learning. However, studies using two-dimensional images capture images from only a single perspective. Due to the complex morphological structure of the target (e.g., plant), severe occlusion and overlap, and the susceptibility of captured images to lighting, they cannot fully describe the complete spatial distribution of the target (e.g., plant) structure, resulting in unreliable statistical significance of the measured phenotypic traits.

[0003] In contrast, three-dimensional data can retain more detailed spatial information of the research object, solving the problem of incomplete information of two-dimensional images. Based on three-dimensional data, the overall morphological characteristics of the target can be quantitatively evaluated. By modeling the target (e.g., plant) with three-dimensional point cloud data and processing the obtained point cloud data, an effective approach is provided for high-precision morphological feature analysis of the target (e.g., plant). One key step in this process is to segment different parts of the target (e.g., stems and leaves of the plant) from the point cloud data. For example, there have been related studies using point cloud traditional algorithms to segment the stems and leaves of the plant. However, they highly depend on pre-defined rules and prior knowledge of the segmentation target, requiring a tedious parameter tuning process to achieve more accurate parameter calculation results, which is time-consuming and labor-intensive. Moreover, due to the complex morphologies of different plant individuals, these methods lack generalizability, leading to unstable segmentation results. Therefore, more research tends to use deep learning-based point cloud processing algorithms to address the shortcomings of traditional algorithms. Compared to traditional point cloud algorithms that highly depend on prior knowledge and manual design, deep learning methods can automatically extract complex structural information from a large number of plant point clouds, automatically learn features from raw plant point clouds, and have strong adaptability and accuracy. Since quickly and accurately segmenting plant organs is crucial for plant phenotype parameter acquisition, researchers have conducted studies in this area, most of which are focused on ideal state plant point cloud data collected in laboratory environments. These plants are standardized and managed with precision, have simple morphologies, high data density, little noise, clear edges, and rich extractable features, making the data collected in laboratory environments an ideal starting point for many studies and providing an ideal condition for algorithm development and optimization. For example, the Eff-3DPSeg weakly supervised network framework is used to realize the segmentation of plant stems and leaves and parameter calculation; the FF-Net network based on feature fusion is used to realize the segmentation of plant point clouds; and a deep learning framework based on SqueezeNet is used to realize the segmentation and parameter calculation of point cloud plants. However, although research on this data helps researchers master the technology of phenotype parameter acquisition, the ultimate goal of research is to solve practical problems, which often requires applying research results to actual greenhouse production scenarios. However, due to the natural growth state of plants in production scenarios, their morphologies are diverse, and the environment is complex, influenced by various external factors, resulting in low point cloud density, much noise, blurred edges, and missing features, making it difficult for segmentation algorithms to process. Methods designed for data collected in experimental environments are difficult to apply in practice. Therefore, a few researchers have conducted research facing actual production scenarios, including: manually processing and labeling point cloud data collected in actual production scenarios, dividing the data into training and test sets, and using pointnet++ to realize the segmentation of plant stems and leaves.However, the point cloud data processing and annotation process is tedious and complex, consuming a lot of manpower and time. Therefore, how to achieve accurate segmentation of actual production scene targets (for example, plant stems and leaves) while reducing the need for annotation has become an unresolved issue.

[0004] In summary, there are still two key challenges in the current segmentation of actual production scene targets (e.g., plant stems and leaves):

[0005] (1) Low quality of point cloud data: Due to the morphological complexity and irregularity of targets in actual scenarios (e.g., plant growth), the collected point cloud data inevitably contains a lot of noise, resulting in blurred edges and large incompleteness. The low quality of point cloud data affects the segmentation accuracy of different target areas (e.g., stems and leaves of plants).

[0006] (2) High labeling cost: The morphology of targets under different conditions (for example, plants under different growth conditions) varies greatly, so the same model cannot be applied in different scenarios. As a result, data needs to be collected independently for labeling and model retraining in different application tasks, which results in high manual labeling costs. Summary of the Invention

[0007] In order to solve the problems existing in the prior art, a zero-labeled target segmentation method and system based on point cloud data optimization is provided, which solves the following technical problems, including:

[0008] (1) Optimization of point cloud data collected in actual production scenarios: To rationally guide the process of optimizing point cloud data collected in actual production scenarios toward high-quality point clouds, while ensuring that the optimized data can fully and accurately reflect the structural information of the target (e.g., plant), this paper proposes a point cloud data cascade optimization network (TP-PCONet) based on self-supervised training, based on the morphological and data characteristics analysis of the target (e.g., plant) in actual production scenarios. The point cloud data of actual production scenarios is optimized in terms of structural integrity, data density, and data smoothness, so that the segmentation model can effectively process the optimized high-quality point cloud data.

[0009] (2) Segmentation framework for different parts of target (e.g., plant stems and leaves) collected in actual production scenes based on zero annotation: In order to solve the problem of high annotation cost of point cloud data collected in actual production scenes, the present invention first uses a set of annotated high-quality target (e.g., plant) point clouds to train a point cloud segmentation model (collectively referred to as a standard point cloud segmentation model), and then guides the target (e.g., plant) point cloud data collected in actual production scenes to learn the features of the annotated indoor high-quality target (e.g., plant) point cloud data (with complete structural information), and inputs the optimized actual production scene point cloud data into the above-mentioned standard point cloud segmentation model to segment different parts of the target (e.g., plant stems and leaves), thereby providing the possibility of obtaining three-dimensional phenotypic parameters (e.g., plant leaf area, stem diameter, etc.). The point of this invention is that it effectively avoids the annotation problem of point cloud data collected in actual production scenes, while effectively ensuring the ability of different parts of the target (e.g., plant stems and leaves) to be accurately segmented.

[0010] Taking the crop tomato as a typical example, phenotypic analysis of the morphological traits of tomato plants can help researchers analyze and track the growth of plants, which is of great significance to tomato breeding, planting guidance and growth monitoring. Among them, the surface coverage of the leaves is the largest, and the calculation of parameters such as leaf area can reflect the photosynthesis intensity of the plant. The stem is the basic structure of the plant. By linking it with other organs such as flowers and fruits, and calculating parameters such as stem diameter, we can gain a deep understanding of the stress level of the plant. Traditional phenotypic traits are generally measured manually, which requires a lot of manpower and material resources, is inefficient, highly subjective, and often destroys the plant structure, causing irreversible losses. With the development and popularization of computer technology, the method of obtaining stem and leaf phenotypic parameters based on computer vision has gradually replaced manual measurement through non-invasive phenotypic measurement methods and high-throughput parameter extraction effects. However, the technical defects mentioned in the background technology section still exist.

[0011] A first aspect of the present invention is to provide a zero-labeled target segmentation method based on point cloud data optimization, comprising:

[0012] S1, optimizing the actual production scene target point cloud data to be processed by a point cloud data optimization method based on self-supervised training to obtain optimized actual production scene target point cloud data; wherein the data characteristics of the actual production scene target point cloud data to be processed are one or more of incompleteness, sparseness, and edge fuzziness; the data characteristics of the optimized actual production scene target point cloud data are one or more of completeness, density, and smoothness; the optimization is performed based on structural integrity, data density, and data smoothness;

[0013] S2, training a standard point cloud segmentation network based on the labeled indoor target point cloud data to obtain a trained standard point cloud segmentation model; the data characteristics of the labeled indoor target point cloud data are complete target structure information, high data density, and high data smoothness;

[0014] S3, inputting the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model, obtaining the segmentation results of different parts of the target, and obtaining related phenotypic parameters.

[0015] Preferably, the S1 includes:

[0016] S11, training the point cloud data cascade optimization network TP-PCONet through data pair construction strategy and self-supervision method;

[0017] S12, through the three steps of point cloud completion based on snowflake diffusion, point cloud upsampling based on gradual refinement, and point cloud refinement based on edge thermal value, the actual production scene target point cloud data to be processed is sequentially constrained in terms of backbone, surface, and contour to obtain the optimized actual production scene target point cloud data.

[0018] Preferably, the S11 includes:

[0019] (1) Taking indoor high-quality point cloud data as input, obtaining simulated production scene point cloud data through a data pair construction strategy based on the point cloud characteristics of the actual production scene, and constructing a training data pair based on the simulated production scene point cloud data; the data pair construction strategy of the actual production scene point cloud characteristics includes: taking indoor high-quality point cloud data as input, and presetting the defect coefficient n, sparse coefficient m and noise coefficient k during processing to respectively implement surface defect simulation, overall sparseness and noise addition, thereby simulating the occlusion, shaking of the target and the point cloud data changes caused during the data acquisition process, and processing the simulated production scene point cloud data obtained after processing. The method comprises the following steps: first, normalizing the indoor high-quality point cloud data; then randomly selecting n points in the indoor high-quality point cloud data as the center of a cube generated in the point cloud space, and setting an edge length; then removing the points in the cube from the input indoor high-quality point cloud data to obtain processed residual point cloud data; finally, randomly downsampling the processed residual point cloud data by m times and adding Gaussian noise with k times the standard deviation to obtain the simulated production scene point cloud data;

[0020] (2) Training the point cloud data cascade optimization network TP-PCONet based on the training data pair and the self-supervised learning method, including: setting the upsampling network PUCRN as the baseline model; introducing a point cloud completion module based on snowflake diffusion and a point cloud refinement module based on edge thermal value on the basis of the point cloud upsampling module based on step-by-step refinement in the baseline model to form the point cloud data cascade optimization network TP-PCONet, the input and output of the point cloud completion module based on snowflake diffusion, the point cloud upsampling module based on step-by-step refinement and the point cloud refinement module based on edge thermal value correspond to the low-quality point cloud data before optimization and the high-quality point cloud data after optimization for the same target in the training data pair; training the point cloud data cascade optimization network TP-PCONet based on the training data pair and the self-supervised learning method, the training data pair is divided into the network input value of the point cloud data cascade optimization network TP-PCONet before training and the training true value, and the training is used to mine the internal inherent structure and mapping relationship of the training data pair to obtain the point cloud data cascade optimization network TP-PCONet.

[0021] Preferably, the point cloud completion module based on snowflake diffusion is used to predict the complete form of the point cloud of the actual production scene to be optimized, and provide complete data for subsequent point cloud optimization, including three steps: point cloud backbone feature extraction, missing part seed point generation and snowflake point diffusion; the point cloud backbone feature extraction is used to perform backbone feature extraction on the point cloud to be optimized, and the encoder in the Transformer architecture is used to extract and encode the input point cloud to obtain the backbone feature vector of the point cloud; the missing part seed point generation includes: based on the extracted backbone feature vector, the Transformer decoder predicts the seed point set of the missing part, and the seed points in the seed point set of the missing part represent the approximate position and shape of the missing area; the snowflake point diffusion includes: using multiple SPD modules in the SnowFlakeNet network, with the seed point in the seed point set of the missing part as the center, gradually diffusing outward in a snowflake shape to generate new points until the entire point cloud data becomes complete and complete point cloud data is obtained.

[0022] Preferably, the point cloud upsampling module based on gradual refinement adopts feature expansion of point cloud data and coordinate reconstruction of point cloud data to obtain the final dense point cloud data; the feature expansion of the point cloud data includes: after the completed complete point cloud data is input into the point cloud upsampling module based on gradual refinement, feature expansion is performed through a multi-layer perceptron MLP to obtain an extended point cloud feature vector; the coordinate reconstruction of the point cloud data includes: fitting the distribution and regularity of the existing point cloud data, and again gradually generating a new point cloud subset from the point cloud feature vector that has undergone the feature expansion through the multi-layer perceptron MLP to fill the sparse area in the original point cloud to obtain the final dense point cloud data.

[0023] Preferably, the point cloud refinement module based on edge heat value is used to receive the dense target point cloud data, and refine it to generate a smoother point cloud to provide data for subsequent target part segmentation; the point cloud refinement module based on edge heat value takes the dense point cloud data as input, and outputs the smoothed point cloud data based on gradient value calculation, edge heat map generation and edge point contraction; the gradient value calculation includes: using KD tree to perform nearest neighbor point query on the input dense point cloud data to obtain a set of nearest neighbor points of each point in the point cloud; the heat map generation includes: calculating the geometric center of the set of nearest neighbor points of each point as the center coordinate of the center point of the current point, calculating the distance between each point and its corresponding center point to obtain the position offset of each point in the point cloud, and calculating the difference between the offset value of the current point and the offset value of its nearest neighbor points, the maximum difference being the gradient value of each point; the calculated point cloud gradient value is normalized to [0, 1] to obtain the edge heat value of the point cloud, and the closer the heat value is to 1, the closer the point is to the edge, thereby forming the edge heat map; the edge point contraction includes: based on the generated edge heat map, the edge points are contracted and the finally optimized point cloud is obtained.

[0024] Preferably, the S3 comprises:

[0025] S31, inputting the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model to obtain the segmentation result of different parts of the target;

[0026] S32, based on the segmentation result of different parts of the target and the morphological analysis of the target, obtaining the related phenotype parameters of the target; the phenotype parameters include one or more of the area, diameter, inclination angle, length and width of the different parts.

[0027] The second aspect of the application provides a zero-annotation target segmentation system based on point cloud data optimization, which is used to implement the zero-annotation target segmentation method based on point cloud data optimization of the first aspect, and comprises:

[0028] The point cloud data optimization module based on self-supervised training is used to optimize the to-be-processed actual production scene target point cloud data to obtain the optimized actual production scene target point cloud data by using the point cloud data optimization method based on self-supervised training; wherein the data characteristics of the to-be-processed actual production scene target point cloud data are one or more of defects, sparseness and edge blur; the data characteristics of the optimized actual production scene target point cloud data are one or more of completeness, density and smoothness; the optimization is performed from three aspects of structural completeness, data density and data smoothness;

[0029] A model training module is used to train a standard point cloud segmentation network based on labeled indoor target point cloud data to obtain a trained standard point cloud segmentation model; the data characteristics of the labeled indoor target point cloud data are complete target structure information, high data density and high data smoothness;

[0030] The standard point cloud segmentation module is used to input the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model, obtain the segmentation results of different parts of the target, and obtain related phenotypic parameters.

[0031] The third aspect of the present invention is to provide an electronic device, including a processor and a memory, wherein the memory stores multiple instructions, and the processor is used to read the instructions and execute the zero-labeled target segmentation method based on point cloud data optimization as described in the first aspect.

[0032] The fourth aspect of the present invention is to provide a computer-readable storage medium, which stores multiple instructions, and the multiple instructions can be read by a processor and executed by the zero-labeled target segmentation method based on point cloud data optimization as described in the first aspect.

[0033] The method, system, electronic device, and computer-readable storage medium provided by the present invention have the following beneficial technical effects:

[0034] A zero-label target segmentation method based on point cloud data optimization is proposed. The segmentation framework of different parts of the target collected in actual production scenarios based on zero-labeling is used to provide the possibility for phenotypic parameter calculation. At the same time, the point cloud data collected in actual production scenarios is optimized, and a point cloud cascade optimization network TP-PCONet based on self-supervised training is designed. Experimental results show that the proposed TP-PCONet achieves a CD distance of 8.45 and an F1-score of 73.5%, respectively. In addition, using a standard segmentation model, the performance of point cloud data optimized by TP-PCONet is also better than other state-of-the-art optimization models. These results show that the proposed method has the potential to significantly improve the segmentation accuracy and efficiency of different parts of the target (for example, the stems and leaves of a plant). BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of the zero-annotation target segmentation method based on point cloud data optimization according to the present invention;

[0036] Figure 2 This is a schematic diagram of the principle of the zero-labeled target segmentation method based on point cloud data optimization according to the present invention;

[0037] Figure 3 Schematic diagram of the point cloud optimization method based on self-supervised training according to the present invention;

[0038] Figure 4 This is an example diagram of the point cloud data characteristics of an actual production scenario described in the present invention;

[0039] Figure 5 This is a schematic diagram of the structure of the point cloud data cascade optimization network TP-PCONet according to the present invention;

[0040] Figure 6 This is a schematic diagram of the point cloud edge heat map of the present invention;

[0041] Figure 7 This is a schematic diagram of the boundary between the stem and leaves of the plant point cloud according to the present invention;

[0042] Figure 8 Schematic diagram of data pair composition in the self-supervised training phase of the point cloud data optimization network according to the present invention;

[0043] Figure 9 This is a sample schematic diagram of the point cloud dataset of an actual production scenario to be optimized according to the present invention;

[0044] Figure 10 This is a diagram showing the visualization effect on the test set described in the present invention. The circle in the figure shows the completion of the incomplete part;

[0045] Figure 11 This is a schematic diagram of the optimization results of different optimization models for the actual production scenario point cloud described in the present invention. The circles in the figure represent the completion effect of the incomplete parts;

[0046] Figure 12 This is a schematic diagram of the plant point cloud segmentation results in an actual production scenario described in the present invention. Dark colors represent leaves, and light colors represent stems.

[0047] Figure 13 This is a schematic diagram of the segmentation results of different segmentation models for a plant point cloud in an actual production scene after TP-PCONet optimization. Dark colors represent leaves, and light colors represent stems.

[0048] Figure 14 This is an architecture diagram of the zero-labeled object segmentation system based on point cloud data optimization according to the present invention;

[0049] Figure 15 This is a structural diagram of the electronic device described in the present invention. DETAILED DESCRIPTION

[0050] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0051] Example 1

[0052] like Figure 1As shown, this embodiment provides a zero-labeled target segmentation method based on point cloud data optimization, including:

[0053] S1, optimizing the actual production scene target point cloud data to be processed by a point cloud data optimization method based on self-supervised training to obtain optimized actual production scene target point cloud data; wherein the data characteristics of the actual production scene target point cloud data to be processed are one or more of incompleteness, sparseness, and edge fuzziness; the data characteristics of the optimized actual production scene target point cloud data are one or more of completeness, density, and smoothness; the optimization is performed from three aspects: structural integrity, data density, and data smoothness;

[0054] In this embodiment, Figure 2 As shown in ①, the point cloud data optimization method based on self-supervised training is used to solve the problem that the point cloud collected in the actual production scene cannot effectively extract structural features due to quality factors such as incompleteness, sparseness and blurred edges.

[0055] As a preferred embodiment, the S1 includes:

[0056] S11, training the point cloud data cascade optimization network TP-PCONet through data pair construction strategy and self-supervision method;

[0057] In this embodiment, the optimization task for point cloud data of actual production scenarios requires that the model be able to learn the morphological characteristics of the plants, and have the ability to extract and characterize complete and detailed three-dimensional structural information of the plants. How to set plant morphological supervision information for the model is the primary consideration. Among various supervision methods, the self-supervised learning method uses the data itself as a supervision signal, deeply mines the information contained in the data through the network, and updates the knowledge representation in real time. It can accurately capture the subtle spatial structural characteristics of tomato plants and is suitable for the optimization task of point cloud data of actual production scenarios. Therefore, the present invention uses the self-supervised training method to achieve high optimization of point clouds of actual production scenarios, such as Figure 3 The method includes a data pair construction strategy based on the point cloud characteristics of actual production scenarios and a point cloud data cascade optimization network TP-PCONet, which are introduced below.

[0058] During the model training phase, high-quality indoor point clouds are first used as input. A data pair construction strategy based on the characteristics of actual production scene point clouds is used to generate simulated production scene point clouds for training data pairs. Secondly, the generated training data pairs are used for self-supervised training of the point cloud data cascade optimization network TP-PCONet. During the model application phase, the actual production scene point cloud to be processed is input into the trained point cloud data optimization model to generate the optimized actual production scene point cloud.

[0059] As a preferred embodiment, the S11 includes:

[0060] (1) Taking indoor high-quality point cloud data as input, obtaining simulated production scene point cloud data through a data pair construction strategy based on the point cloud characteristics of the actual production scene, and constructing a training data pair based on the simulated production scene point cloud data; the data pair construction strategy of the actual production scene point cloud characteristics includes: taking indoor high-quality point cloud data as input, and presetting the defect coefficient n, sparse coefficient m and noise coefficient k during processing to respectively implement surface defect simulation, overall sparseness and noise addition, thereby simulating the occlusion, shaking of the target and the point cloud data changes caused during the data acquisition process, and processing the simulated production scene point cloud data obtained after processing. The point cloud data of the simulated production scene is obtained by constructing a data pair strategy based on the point cloud characteristics of the actual production scene, including: first, normalizing the indoor high-quality point cloud data; then randomly selecting n points in the indoor high-quality point cloud data as the center of the cube generated in the point cloud space, and setting the edge length; then removing the points belonging to the cube from the input indoor high-quality point cloud data to obtain the processed residual point cloud data; finally, randomly downsampling the processed residual point cloud data by m times, and adding Gaussian noise with k times the standard deviation to obtain the simulated production scene point cloud data.

[0061] In this embodiment, the point cloud data optimization network based on self-supervised training requires sample training pairs of low-quality pre-optimization and high-quality post-optimization point clouds for the same target, serving as network input and training ground truth, respectively, to exploit the inherent structure and mapping relationships within the data. However, due to the complex environment of the collected point cloud data in actual production scenarios, it is impossible to obtain high-quality point clouds as supervisory ground truth. Therefore, it is necessary to utilize a high-quality point cloud dataset available under indoor conditions as supervisory ground truth to provide sufficient supervisory information to guide the model for correct self-learning. Furthermore, in order to enable the network to optimize the tomato plant point cloud in actual production scenarios, the present invention provides a simulated production scenario dataset based on the characteristics of the point cloud data in actual production scenarios.

[0062] Examples of point cloud data characteristics in actual production scenarios include: Figure 4 As shown in the figure, there are mainly the following aspects: First, due to the mutual occlusion between leaves and between leaves and stems, the collected point cloud is incomplete [(A): incomplete characteristic]; second, due to the complex environment in which the plant is located and the irregular growth form, it is difficult to collect a uniform and dense point cloud [(B): sparse characteristic]; in addition, in the actual production scene collection process, external factors such as wind disturbance and shaking of the collection equipment will make it impossible for the plant to fix a posture, resulting in blurred edges of the collected point cloud [(C): blurred edge characteristic].

[0063] Therefore, based on the above analysis of the characteristics of actual production scene point cloud data, and the specific manifestation of edge blurring, where points at the boundary are not distributed in a linear shape but rather locally exhibit a Gaussian noise distribution, this method performs a series of processing on the input high-quality indoor point cloud, including surface defect simulation, global thinning, and noise addition, to simulate plant occlusion, swaying, and point cloud data changes that may occur during data acquisition. This results in simulated production scene point cloud data with the characteristics of actual production scene point clouds. The strategy for constructing the simulated production scene point cloud dataset is shown in Algorithm 1.

[0064]

[0065]

[0066] As shown in Algorithm 1, during the data pair construction process, the present invention uses a high-quality indoor point cloud as input, and simultaneously presets the processing incompleteness coefficient n (set to 5 in this embodiment of the present invention), the sparseness coefficient m (set to 4 in this embodiment of the present invention), and the noise coefficient k (set to 0.02 in this embodiment of the present invention). The processed simulated production scene point cloud is then output. In this process, the input point cloud data is first normalized to facilitate subsequent processing. Then, n points are randomly selected from the input point cloud as the center of a cube generated in the point cloud space, and the edge length is set to 0.05 (an empirical value in this embodiment of the present invention). Points within the cube are then removed from the input point cloud to obtain a processed incomplete point cloud. Finally, the point cloud is randomly downsampled m times and Gaussian noise with k times the standard deviation is added to obtain a simulated production scene point cloud. Through the above operations, a sample of plant point clouds from a simulated production scene is obtained, thereby constructing a pair of low-quality pre-optimization and high-quality post-optimization point cloud data that can be used for training a point cloud optimization network.

[0067] (2) Training the point cloud data cascade optimization network TP-PCONet based on the training data pair and the self-supervised learning method, including: setting the upsampling network PUCRN as the baseline model; introducing a point cloud completion module based on snowflake diffusion and a point cloud refinement module based on edge thermal value on the basis of the point cloud upsampling module based on step-by-step refinement in the baseline model to form the point cloud data cascade optimization network TP-PCONet, the input and output of the point cloud completion module based on snowflake diffusion, the point cloud upsampling module based on step-by-step refinement and the point cloud refinement module based on edge thermal value correspond to the low-quality point cloud data before optimization and the high-quality point cloud data after optimization for the same target in the training data pair; training the point cloud data cascade optimization network TP-PCONet based on the training data pair and the self-supervised learning method, the training data pair is divided into the network input value of the point cloud data cascade optimization network TP-PCONet before training and the training true value, and the training is used to mine the internal inherent structure and mapping relationship of the training data pair to obtain the point cloud data cascade optimization network TP-PCONet.

[0068] In this embodiment, the self-supervised learning method enables the network to actively learn the mapping relationship between data pairs, enabling it to optimize point clouds in real-world production scenarios. To further guide the network's learning direction from optimizing low-quality point clouds to high-quality point clouds, capture the fine structure of the target (in this embodiment, a tomato plant), and avoid optimization distortion, this embodiment designed the network structure as follows.

[0069] First, this embodiment uses the upsampling network PUCRN as the baseline model. This model effectively avoids distortion and discontinuity caused by overfilling the point cloud at once by gradually increasing the point cloud density, making it suitable for the complex and detailed structure of tomato plants. Second, to enable the optimization network to complete incomplete point clouds, this embodiment incorporates the snowflake diffusion-based point cloud completion module from SnowFlakeNet into the baseline model. This module utilizes the principle of snowflake diffusion to achieve smooth transitions and natural connections in point cloud data, avoiding abrupt completion distortion and addressing the distinct incomplete features of stems and leaves in the plant point cloud. To address the issue of blurred stem and leaf edges caused by the data itself and the upsampling and completion operations, this embodiment designs a point cloud upsampling module based on gradual refinement by constraining the point cloud surface. Furthermore, by constraining the point cloud contours, a point cloud refinement module based on edge thermal values ​​is designed. These three modules constitute the point cloud data cascade optimization network designed in this embodiment, TP-PCONet. Figure 5The specific structural diagram of the point cloud data cascade optimization network TP-PCONet is shown. It is mainly composed of three cascaded sub-modules: (A-1), (A-2), and (A-3). Among them, (A-1): constrains the point cloud backbone to effectively fill in the missing parts and provide complete data for subsequent processing; (A-2): constrains the point cloud surface to optimize sparse data into dense data and improve data density; (A-3): constrains the contour of the completed and dense point cloud to obtain smooth point cloud data.

[0070] S12, through the three steps of point cloud completion based on snowflake diffusion, point cloud upsampling based on gradual refinement, and point cloud refinement based on edge thermal value, the actual production scene target point cloud data to be processed is sequentially constrained in terms of backbone, surface, and contour to obtain the optimized actual production scene target point cloud data.

[0071] In actual application, the optimized actual production scene point cloud data is input into Figure 2 The standard point cloud segmentation training model in ② is used to further segment the target (a tomato plant in this embodiment) into different parts (stem and leaf segmentation in this embodiment).

[0072] In the actual production scenario of tomato plant point cloud data optimization task, it is assumed that the input point cloud subset is represented as Among them, P 0i Represents the i-th point of the point cloud, N is the total number of points in the input point cloud, and the task is to obtain the optimized point cloud subset Where M represents the total number of points in the optimized point cloud, and M>N. The optimized point cloud subset P3 is obtained by formula (1) as follows:

[0073]

[0074] In formula (1), express Figure 5 The completion module processing process based on snowflake diffusion shown in (A-1) is as follows: express Figure 5 (A-2) shows the upsampling module processing process based on gradual refinement, express Figure 5 (A-3) shows the refinement module processing process based on edge thermal value.

[0075] This embodiment optimizes the processing flow after the network is optimized based on the input of point cloud data. The three modules of this embodiment, namely, the point cloud completion module based on snowflake diffusion, the point cloud upsampling module based on gradual refinement, and the point cloud refinement module based on edge thermal value, and their processing procedures include:

[0076] (1) Point cloud completion module based on snowflake diffusion

[0077] Figure 5 Figure (A-1) shows the SnowFlakeNet-based point cloud completion module introduced in this embodiment. This module is used to predict the complete form of the actual production scene point cloud to be optimized, providing complete data for subsequent point cloud optimization. This module takes the point cloud to be optimized as input and obtains the completed point cloud, as shown in Formulas (2) and (3).

[0078]

[0079] in Represents the input point cloud data to be optimized, Represents the output completed point cloud data, N and K represent the number of point cloud points of the input point cloud data to be optimized and the number of point cloud points of the output completed point cloud data, respectively, K>N, It consists of three steps, where MFE(x i ,y i , z i ) represents the point cloud backbone feature extraction, SG(x i ,y i , z i ) indicates the missing seed point generation, SPD(x i ,y i , z i ) represents snowflake point diffusion. The specific steps are as follows.

[0080] Step 1: Extract backbone features of point cloud, which is used to extract backbone features of the point cloud P0 to be optimized. The encoder in the Transformer architecture is used to extract and encode the input point cloud to obtain the backbone feature vector of the point cloud, as shown in formula (4).

[0081] V m =T e (P0) (4)

[0082] in, represents the backbone feature vector, N represents the total number of points in the input point cloud, T e Represents the encoder of Transformer.

[0083] Step 2: Generate missing seed points, including: based on the extracted backbone feature vector V m , the decoder of Transformer predicts the seed point set of the missing part, as shown in formula (5). The seed points in the seed point set of the missing part represent the approximate position and shape of the missing area.

[0084] P s =T d (V m ) (5)

[0085] in, n represents the number of points in the generated point cloud subset, T d Represents the decoder of Transformer.

[0086] Step 3: Snowflake point diffusion, including: using multiple SPD modules in the SnowFlakeNet network, starting with the seed point in the missing part of the seed point set in step 2, gradually diffusing outward in a snowflake shape to generate new points until the entire point cloud data is complete, thus obtaining the complete point cloud data. The module diffusion process is shown in formula (6).

[0087] P s(i+1) =SPD(P si ) (6)

[0088] Among them, P si represents the input point cloud of the i+1th SPD module, P s(i+1) It represents the output point cloud of the i+1th SPD module. In this embodiment, when i=2, the point diffusion is completed.

[0089] The snowflake diffusion-based point cloud completion module samples the true point cloud at the farthest point to obtain the backbone points. This is then compared to the predicted point cloud using CDloss to calculate the loss, and the snowflake diffusion-based point cloud completion module is updated. Through these three steps, a structurally complete point cloud of a plant in an actual production scene can be obtained.

[0090] (2) Point cloud upsampling module based on gradual refinement

[0091] Figure 5 (A-2) shows the progressively refined point cloud upsampling module in the baseline model PUCRN. This module is primarily used to receive the completed plant point cloud and perform upsampling operations on it, providing dense data for subsequent point cloud optimization. This module takes the completed point cloud as input and outputs a dense point cloud, as shown in Equations (7) and (8).

[0092]

[0093] in, Represents the complete point cloud data of the input, represents the output dense point cloud data, r represents the upsampling multiple, which is set to 4 in this embodiment; It consists of two steps, FE(x i ,y i , z i ) represents the feature expansion of point cloud data, CR(x i ,y i , z i) represents the coordinate reconstruction of point cloud data. The specific steps are as follows.

[0094] Step 1: Feature expansion of point cloud data, including: after the completed point cloud data is input into the point cloud upsampling module based on the progressive refinement, feature expansion is performed through the multi-layer perceptron MLP to obtain an expanded point cloud feature vector, as shown in formula (9).

[0095] F up =MLP(P1) (9)

[0096] in, represents the extended feature vector.

[0097] Step 2: coordinate reconstruction of point cloud data, including: fitting the distribution and regularity of the existing point cloud data, and gradually generating new point cloud subsets from the point cloud feature vector after the feature expansion through the multi-layer perceptron MLP again to fill the sparse areas in the original point cloud, as shown in formula (10).

[0098] S=MLP(F up ) (10)

[0099] in, Represents the newly generated point cloud subset. Since this embodiment uses a step-by-step upsampling method, α=r / 2 here. Repeat step 2 to obtain the final dense point cloud data.

[0100] Based on the gradually refined point cloud upsampling module, the CD loss between the ground truth point cloud and the predicted point cloud is directly calculated, and the generated dense point cloud is constrained. Through the above operations, a complete and dense point cloud of plants in actual production scenes can be obtained.

[0101] (3) Point cloud refinement module based on edge thermal value

[0102] Figure 5 (A-3) shows the edge thermal value-based point cloud refinement module designed in this embodiment. It is primarily used to receive the densified plant point cloud data and refine it to generate a smoother point cloud, providing data for subsequent stem and leaf segmentation. The edge thermal value-based point cloud refinement module takes the dense point cloud data as input and outputs smoothed point cloud data, as shown in Equations (11) and (12).

[0103]

[0104] in, represents the input dense point cloud data, Represents the output refined point cloud data, It consists of three steps: GC(x i ,y i, z i ) represents the gradient value calculation, HMG(x i ,y i , z i ) represents the heat map generation, EPC(x i ,y i , z i ) indicates edge point contraction. The specific implementation steps are as follows.

[0105] Step 1: Gradient value calculation, including: using the KD tree to perform nearest neighbor query on the input dense point cloud data to obtain the neighbor point set of each point in the point cloud, as shown in formula (13).

[0106] PN=KDTree(P i ) (13)

[0107] Among them, P i represents the input point cloud, represents the set of neighboring points of each point, and β represents the number of neighboring points of each point, which is set to 350 in this embodiment.

[0108] Step 2: Heat map generation, including: calculating the geometric center PC of the neighboring point set of each point as the center coordinate of the center point of the current point, as shown in formula (14).

[0109]

[0110] Calculate the distance between each point and its corresponding center point to obtain the position offset of each point in the point cloud, and calculate the difference between the offset value of the current point and the offset value of its neighboring points. The largest difference is the gradient value d of each point. i , as shown in formula (15).

[0111] d i =max||P i -PC i |-|PN i -PC i || (15)

[0112] In point cloud data, the larger the gradient value of the point, the closer the point is to the edge. Conversely, the smaller the gradient value, the closer the point is to the inside of the point cloud. The calculated point cloud gradient value is normalized to [0, 1] to obtain the edge thermal value of the point cloud. The closer the thermal value is to 1, the closer the point is to the edge, thus forming a clear edge thermal map. Figure 6 As shown, Figure 6 (A) in the figure represents the original input point cloud; Figure 6 (B) in the figure shows the corresponding heat map visualization generated by encoding.

[0113] Step 3: Edge Point Contraction: This involves contracting edge points based on the generated edge heatmap to reduce noise around the point cloud, sharpen the point cloud outline, and ultimately produce the optimized point cloud. This module calculates the edge heatmap values ​​for the predicted and ground-truth point clouds and applies an L1 loss to constrain the point clouds. This process yields a structurally complete, dense, and smooth point cloud of plants from a real-world production scenario.

[0114] After three modules, the actual production scene point cloud data to be optimized is processed step by step by TP-PCONet, ultimately forming a point cloud with high-quality point cloud distribution characteristics. The optimized point cloud structure is complete, dense, and smooth, thus providing effective data for tomato plant stem and leaf segmentation.

[0115] S2, training a standard point cloud segmentation network based on the labeled indoor target point cloud data to obtain a trained standard point cloud segmentation model; the data characteristic of the labeled indoor target point cloud data is that the target's structural information is complete;

[0116] As a preferred embodiment, Figure 2 The standard point cloud segmentation module shown in ② is used for plant stem and leaf segmentation. The standard point cloud segmentation module proposed in the embodiment of the present invention is trained based on a set of labeled indoor high-quality point cloud dataset Pheno4D.

[0117] S3, inputting the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model, obtaining the segmentation results of different parts of the target, and obtaining related phenotypic parameters.

[0118] In this embodiment, the standard point cloud segmentation model has been trained as follows Figure 2 As shown in ②, the trained standard point cloud segmentation model is trained by a set of labeled indoor high-quality point cloud data sets, which is collectively referred to as the standard point cloud segmentation module in the subsequent system description of the present invention.

[0119] As a preferred embodiment, the S3 includes:

[0120] S31, inputting the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model to obtain segmentation results of different parts of the target;

[0121] S32, based on the segmentation results of different parts of the target and the morphological analysis of the target, obtaining the relevant phenotypic parameters of the target.

[0122] As a preferred embodiment, the phenotypic parameters of S3 include one or more of the area, diameter, inclination, length and width of the different parts.

[0123] In this embodiment, it is specifically manifested as follows: based on the accurate stem and leaf segmentation results of the optimized actual production scene point cloud and the analysis of plant morphology, it is possible to accurately calculate the three-dimensional phenotypic parameters of the plant, such as leaf area, stem diameter, leaf inclination, etc.

[0124] Since the training data and application data of the model are very different in terms of collection environment, plant morphology and data distribution, the point cloud segmentation model needs to have the ability to generalize the model to adapt to different application data. At the same time, in the semantic segmentation task of plant stems and leaves, due to the complex structure and shape of the plant, especially the interlacing and overlapping between the stems and leaves, it is difficult to accurately segment the stem-leaf junction. Therefore, the model should have the ability to accurately segment the stem-leaf junction. Figure 7 shown.

[0125] In this embodiment, the segmentation model SoftGroup is used as the stem-leaf semantic segmentation model, which performs point-by-point semantic label prediction through U-Net, and then adopts a grouping-based method to group the semantic scores of the point cloud through the soft grouping module to reduce incorrect semantic label classification. SoftGroup uses a grouping-based method for point cloud semantic segmentation, which can directly process the entire scene without generating proposals, so that the network can more comprehensively understand the contextual information in the scene. When there is a large difference between the training data and the application data, it can also adapt to the new data distribution by capturing the overall structural characteristics of the scene, thereby improving the generalization ability of the model. At the same time, the grouping strategy can make full use of the spatial information and semantic information in the point cloud, reduce the misjudgment at the junction of the stem and leaves, effectively handle complex situations, accurately identify the stems and leaves, and thus achieve their precise semantic segmentation.

[0126] Verification of technical effects of application examples:

[0127] In order to verify the effectiveness of the method proposed in this invention, we constructed a dataset for point cloud data optimization and a dataset for standard point cloud segmentation, as shown in Table 1.

[0128] Table 1 Dataset introduction

[0129]

[0130]

[0131] 1. Dataset

[0132] (1) Point cloud data optimization module dataset based on self-supervised training

[0133] 1. Training dataset for point cloud data optimization module based on self-supervised training

[0134] The training dataset of the point cloud data optimization module based on self-supervised training consists of low-quality point cloud training data pairs before optimization and high-quality point cloud training data pairs after optimization. The high-quality point cloud dataset uses the tomato plant point cloud dataset in the Pheno4D dataset. This dataset was collected in a laboratory environment, and the point cloud is complete, dense, and has clear edges. Figure 8 As shown in (A) in the figure, the low-quality dataset is obtained by processing the high-quality dataset through the data pair construction strategy based on the point cloud characteristics of the actual production scene, as shown in Figure 8 In this example, 77 tomato plants in the Pheno4D dataset were processed to obtain 77 training data pairs, of which 66 were used as training sets and 11 were used as test sets.

[0135] 2. Point cloud dataset for actual production scenarios to be optimized

[0136] The application dataset of the point cloud data optimization module based on self-supervised training is the point cloud dataset of the actual production scene to be optimized. This embodiment selects CF Petit Puyo Yellow and CF Petit Puyo in actual production as experimental objects, and collects data from the tomato planting base of Tokyo University in Japan (35°44′13.95″N139°32′23.53″E). In this experiment, an onboard RGB camera (1 / 2.3-inch CMOS, lens FC7303, focal length 4.49mm, maximum image resolution size 4000×2250) on a low-cost industrial-grade drone (Mavic Mini 2, DJI) was used to collect data. The flight altitude was set to 0.5~2m and the distance from the tomato plant was 1.5~2m. The captured RGB images were three-dimensionally reconstructed using the commercial software clomap based on the principle of multi-view reconstruction to obtain the reconstructed tomato plant point cloud. A total of 14 tomato plant point clouds were obtained for application testing. The dataset is as follows: Figure 9 As shown. Among them, Figure 9 (A), (B) and (C) represent different plant point cloud samples respectively.

[0137] (2) Standard point cloud segmentation module dataset

[0138] 1. Training dataset for standard point cloud segmentation model

[0139] This example uses the tomato plant point cloud dataset from the Pheno4D dataset as the training dataset for the standard point cloud segmentation model. This dataset provides instance-level annotations for the stems and leaves of tomato plants. This article uses 66 labeled samples as the training set and 11 labeled samples as the test set to train the standard point cloud segmentation network.

[0140] 2. Optimized actual production scene point cloud dataset

[0141] The application dataset of the standard point cloud segmentation model is the optimized actual production scene point cloud dataset, which is the optimized actual production scene point cloud dataset obtained in this embodiment. At the same time, it is annotated with stems and leaves through the commercial point cloud annotation software CloudCompare as the segmentation result to calculate the true value for evaluation.

[0142] 2. Evaluation Indicators

[0143] This embodiment formulates evaluation indicators for the point cloud data optimization experiment and the point cloud segmentation experiment, as follows.

[0144] (1) Point cloud data optimization experiment evaluation indicators

[0145] In the point cloud data optimization experiment, chamfer distance (CD-distance) and F1-Score are used as evaluation indicators to measure the effect of point cloud optimization.

[0146] Chamfer distance: It can measure the similarity between the predicted point cloud and the true value point cloud at the set level. The smaller the chamfer distance, the more similar the two point clouds are. For each sample, the predicted point set P pred and the true value point set P gt Chamfer distance d CD The calculation formula is shown in formula (16).

[0147]

[0148] Among them, |P pred | represents the prediction point set P pred Points, |P gt | represents the true value point set P gt The number of points, ||pg|| represents the distance between p and g. As shown in formula (17), the chamfer distance represents the minimum distance between two sets of points. The larger the distance, the greater the difference between the two sets of point clouds; the smaller the distance, the higher the similarity of the point clouds.

[0149] F1-Score: The harmonic mean of precision and recall measures the similarity between Ppred and Pgt. Precision represents the percentage of predicted points within a certain distance from the true value, representing the accuracy of the prediction; recall represents the percentage of true value points within a certain distance from the predicted point, representing the completeness of the prediction. The specific calculation method is shown in formulas (17) to (19).

[0150]

[0151] Precision(d) and Recall(d) represent the accuracy and recall respectively calculated under the distance threshold d. In the PU1K data format, d is set to 0.02. The higher the F1-Score, the higher the similarity between the two point sets.

[0152] (2) Point cloud segmentation experiment evaluation indicators

[0153] In the plant point cloud stem and leaf segmentation experiment, the intersection over union (IoU) is used as the evaluation index for plant stem and leaf segmentation. The higher the IoU value, the better the segmentation performance. The calculation formula of IoU is shown in formula (20).

[0154]

[0155] Where TP represents the number of true positive points of the current semantic category, FP represents the number of false positive points of the current category, and FN represents the number of false negative points. As shown in formula (20), IoU can reflect the degree of overlap between the predicted area of ​​each semantic category and the corresponding true area. The closer the IoU value is to 1, the higher the semantic segmentation accuracy and the more accurate the segmentation.

[0156] (3) Experimental setup

[0157] This experiment was conducted on a computer platform with an Intel Core i7-10700k CPU processor (64GB RAM). The model was constructed, trained, and validated using a GeForce RTX 3090 GPU (24GB), the Ubuntu 18.04LTS operating system, and the Python 3.7 programming language running on the PyTorch 1.10.0 framework.

[0158] The TP-PCONet model was trained with a batch size of 4 and a learning rate of 0.001 for 100 training cycles. The SoftGroup model was trained with an Adaptive Moment Estimation (Adam) optimizer, with a batch size of 4 and a learning rate of 0.001 for 100 training cycles.

[0159] 3. Conclusion

[0160] (1) Point cloud data optimization experiment

[0161] The point cloud data optimization experiment verifies the point cloud optimization effect of the point cloud data optimization module based on self-supervised training. It consists of a training dataset experiment of the point cloud data optimization module based on self-supervised training and a point cloud data optimization experiment in actual production scenarios.

[0162] 1. Training dataset experiment of point cloud data optimization module based on self-supervised training

[0163] The other classic point cloud data optimization network is selected for comparative experiment in the embodiment, including the baseline design model PUCRN of TP-PCONet, and PU-GAN, PU-GCN, Grad-PU, so as to prove the effectiveness of the designed TP-PCONet in the embodiment. The experimental results are shown in Table 2.

[0164] Table 2 Comparison of experimental results of different data optimization networks based on self-supervised training

[0165]

[0166]

[0167] As shown in Table 2, the TP-PCONet achieves the best effect, which is better than other comparison algorithms, which proves the effectiveness of the point cloud data optimization. The point cloud optimization visualization effect after being processed by different optimization models is shown in Figure 10 (A)-(F) of Figure 10 As shown in (A)-(G) of

[0168] In order to further verify the effectiveness of each improved module in the TP-PCONet, including the point cloud completion module based on snowflake diffusion (denoted as SP) and the point cloud refinement module based on edge heat value (denoted as EH), the ablation experiment is performed in the embodiment, and the results are shown in Table 3, wherein PUCRN+SP+EH is the designed TP-PCONet in the embodiment.

[0169] Table 3 Ablation experiment results of data optimization based on self-supervised training

[0170]

[0171] As shown in Table 3, by using the point cloud completion module based on snowflake diffusion and the point cloud refinement module based on edge heat value, higher accuracy is obtained on the basis of the baseline model, and the best result is achieved when the two modules are combined.

[0172] 2. Point cloud data optimization experiment in actual production scene

[0173] In the embodiment, the actual production scene point cloud data set to be optimized is input into the trained point cloud data optimization model. Since there is no optimized true value for evaluation and calculation for the data set, the visualization effect after optimization is shown in Figure 11 (A)-(F) of Figure 11As can be seen from (A)-(F) in the figure, the tomato plant point cloud collected in the actual production scene is more dense and complete after TP-PCONet data optimization, and the point cloud outline is clearer.

[0174] In summary, the experimental results verify that the point cloud data optimization experiment based on self-supervised training proposed in the embodiment of the present invention can highly optimize the point cloud data and provide an analysis data basis for subsequent steps.

[0175] (2) Point cloud segmentation experiment

[0176] The point cloud segmentation experiment is used to verify the effectiveness and applicability of the optimization method designed in this invention. It consists of a point cloud segmentation experiment in an actual production scenario based on the point cloud data optimization method and a comparative experiment of different segmentation models based on the point cloud optimization method.

[0177] 1. Point cloud segmentation experiment in actual production scenarios based on point cloud data optimization method

[0178] To verify the effectiveness of the proposed TP-PCONet, this example pre-trained SoftGroup using a training dataset from a standard point cloud segmentation module to obtain a standard stem-and-leaf segmentation model. This model then segmented an optimized point cloud dataset from an actual production scenario. This example conducted comparative experiments based on the optimization results of different data optimization networks, as shown in Table 4.

[0179] Table 4 Comparison of experimental results of tomato plant stem and leaf segmentation in actual production scenarios after optimization by different optimization models

[0180]

[0181] As can be seen from Table 4, compared with not performing data optimization, data optimization of the actual production scene point cloud can improve the stem and leaf segmentation effect of the plant point cloud, which proves the effectiveness of the method proposed in this embodiment. In addition, the TP-PCONet of this embodiment achieved the best effect among the comparison algorithms, which proves the effectiveness of TP-PCONet. Among them, after the optimization of TP-PCONet proposed in this embodiment, the leaf segmentation accuracy of the tomato plant in the actual production scene is 0.882, and the stem segmentation accuracy is 0.471. The plant segmentation results of the actual production scene are shown in Figure 4. Figure 12 As shown in (A)-(F) in the figure.

[0182] To further verify the effectiveness of the various improved modules in the TP-PCONet of this embodiment in the stem-leaf segmentation method, including the point cloud completion module based on snowflake diffusion (denoted as SP) and the point cloud refinement module based on edge thermal value (denoted as EH), this embodiment conducted an ablation experiment. The results are shown in Table 5, where PUCRN+SP+SHE is the TP-PCONet designed in this embodiment.

[0183] Table 5 Experimental results of tomato plant stem and leaf segmentation ablation in actual production scenarios

[0184]

[0185] As can be seen from Table 5, through the point cloud completion module based on snowflake diffusion and the point cloud refinement module based on edge thermal value, this embodiment achieves higher accuracy in plant stem and leaf segmentation in actual production scenarios, and achieves the best results when the two are combined.

[0186] 2. Comparative experiment of different segmentation models based on point cloud optimization method

[0187] To verify that the point cloud data optimization method proposed in this example improves the segmentation results of different segmentation models, this example selected different classic point cloud segmentation networks, including SoftGroup, PointNet++, PointTransformer, and GeoSegNet, used in this example, to conduct experiments on actual production scene point cloud data before and after TP-PCONet optimization. Table 6 shows this result.

[0188] Table 6 Comparison of experimental results of tomato plant stem and leaf segmentation in actual production scenarios using different segmentation models

[0189]

[0190] As shown in Table 6, after the optimization method proposed in this embodiment, different segmentation models have achieved improvements in segmentation accuracy. At the same time, the segmentation model SoftGroup used in the method of this embodiment has achieved higher accuracy, which proves the wide applicability of the method proposed in this embodiment. Different segmentation models are used to segment the actual production scene dataset after TP-PCONet optimization. The results are shown in Figure 6. Figure 13 As shown in (A)-(G) in . Figure 13 As can be seen from Table 6, the SoftGroup used in this embodiment achieves a better segmentation effect and can accurately segment the junction of stems and leaves, which proves the effectiveness of the segmentation model finally selected in this embodiment.

[0191] Example 2

[0192] like Figure 14 As shown, this embodiment provides a zero-labeled target segmentation system based on point cloud data optimization, which is used to implement the method of embodiment 1, including:

[0193] The point cloud data optimization module 101 based on self-supervised training is used to optimize the actual production scene target point cloud data to be processed by a point cloud data optimization method based on self-supervised training to obtain optimized actual production scene target point cloud data; wherein the data characteristics of the actual production scene target point cloud data to be processed are one or more of incompleteness, sparseness and edge fuzziness; the data characteristics of the actual production scene target point cloud data after optimization are one or more of completeness, density and smoothness; the optimization is performed from three aspects: structural integrity, data density and data smoothness;

[0194] The model training module 102 is configured to train a standard point cloud segmentation network based on the labeled indoor target point cloud data to obtain a trained standard point cloud segmentation model; the labeled indoor target point cloud data has the characteristics of complete target structure information, high data density, and high data smoothness;

[0195] The standard point cloud segmentation module 103 is used to input the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model, obtain the segmentation results of different parts of the target, and obtain related phenotypic parameters.

[0196] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the method described in the first embodiment.

[0197] like Figure 15 As shown, the present invention also provides an electronic device, including a processor 301 and a memory 302 connected to the processor 301, wherein the memory 302 stores multiple instructions, which can be loaded and executed by the processor to enable the processor to execute the method described in Example 1.

[0198] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the invention. Thus, the present invention is intended to include such changes and modifications as fall within the scope of the claims and their equivalents.

Claims

1. A zero-labeled target segmentation method based on point cloud data optimization, characterized in that: include: S1, optimizing the actual production scene target point cloud data to be processed by a point cloud data optimization method based on self-supervised training to obtain optimized actual production scene target point cloud data; wherein the data characteristics of the actual production scene target point cloud data to be processed are one or more of incompleteness, sparseness, and edge fuzziness; the data characteristics of the optimized actual production scene target point cloud data are one or more of completeness, density, and smoothness; the optimization is performed based on structural integrity, data density, and data smoothness; S2, training a standard point cloud segmentation network based on the labeled indoor target point cloud data to obtain a trained standard point cloud segmentation model; the data characteristics of the labeled indoor target point cloud data are complete target structure information, high data density, and high data smoothness; S3, inputting the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model, obtaining segmentation results of different parts of the target, and obtaining related phenotypic parameters; Said S1 comprises: S11, training the point cloud data cascade optimization network TP-PCONet through data pair construction strategy and self-supervision method; S12, through the three steps of point cloud completion based on snowflake diffusion, point cloud upsampling based on gradual refinement, and point cloud refinement based on edge thermal value, the actual production scene target point cloud data to be processed is sequentially constrained in terms of backbone, surface, and contour, to obtain the optimized actual production scene target point cloud data; The S11 includes: (1) Taking indoor high-quality point cloud data as input, obtaining simulated production scene point cloud data through a data pair construction strategy based on the point cloud characteristics of the actual production scene, and constructing training data pairs based on the simulated production scene point cloud data; the data pair construction strategy of the actual production scene point cloud characteristics includes: taking indoor high-quality point cloud data as input, and presetting the incompleteness coefficient during processing. , sparse coefficient and noise figure The surface defect simulation, overall sparseness and noise addition are respectively implemented to simulate the occlusion and shaking of the target and the change of point cloud data caused by the data acquisition process, and the simulated production scene point cloud data obtained after processing is used as the output; the data pair construction strategy based on the actual production scene point cloud characteristics is used to obtain the simulated production scene point cloud data, which includes: first, normalizing the indoor high-quality point cloud data; then randomly selecting the indoor high-quality point cloud data. points as the center of the cube generated in the point cloud space, and set the edge length; then remove the points belonging to the cube from the input indoor high-quality point cloud data to obtain the processed residual point cloud data; finally, randomly downsample the processed residual point cloud data times and add Gaussian noise with a standard deviation of times, and finally obtaining the point cloud data of the simulated production scene; (2) Training the point cloud data cascade optimization network TP-PCONet based on the training data pair and the self-supervised learning method, including: setting the upsampling network PUCRN as the baseline model; introducing a point cloud completion module based on snowflake diffusion and a point cloud refinement module based on edge thermal value on the basis of the point cloud upsampling module based on step-by-step refinement in the baseline model to form the point cloud data cascade optimization network TP-PCONet, the input and output of the point cloud completion module based on snowflake diffusion, the point cloud upsampling module based on step-by-step refinement and the point cloud refinement module based on edge thermal value correspond to the low-quality point cloud data before optimization and the high-quality point cloud data after optimization for the same target in the training data pair; training the point cloud data cascade optimization network TP-PCONet based on the training data pair and the self-supervised learning method, the training data pair is divided into the network input value of the point cloud data cascade optimization network TP-PCONet before training and the training true value, and the training is used to mine the internal inherent structure and mapping relationship of the training data pair to obtain the point cloud data cascade optimization network TP-PCONet.

2. The zero-labeled target segmentation method based on point cloud data optimization according to claim 1, characterized in that: The point cloud completion module based on snowflake diffusion is used to predict the complete form of the point cloud of the actual production scene to be optimized, and provide complete data for the subsequent point cloud optimization, including three steps: point cloud backbone feature extraction, missing part seed point generation and snowflake point diffusion; the point cloud backbone feature extraction is used to perform backbone feature extraction on the point cloud to be optimized, and the encoder in the Transformer architecture is used to extract and encode the input point cloud to obtain the backbone feature vector of the point cloud; the missing part seed point generation includes: based on the extracted backbone feature vector, the Transformer decoder predicts the seed point set of the missing part, and the seed points in the seed point set of the missing part represent the approximate position and shape of the missing area; the snowflake point diffusion includes: using multiple SPD modules in the SnowFlakeNet network, with the seed point in the seed point set of the missing part as the center, gradually diffusing outward in a snowflake shape to generate new points until the entire point cloud data becomes complete and the complete point cloud data is obtained.

3. The zero-labeled target segmentation method based on point cloud data optimization according to claim 2, characterized in that: The point cloud upsampling module based on gradual refinement uses feature expansion of point cloud data and coordinate reconstruction of point cloud data to obtain the final dense point cloud data; the feature expansion of point cloud data includes: after the completed point cloud data is input into the point cloud upsampling module based on gradual refinement, it is passed through a multi-layer perceptron. Perform feature expansion to obtain an expanded point cloud feature vector; the coordinate reconstruction of the point cloud data includes: fitting the distribution and regularity of the existing point cloud data, and then passing it through a multi-layer perceptron. New point cloud subsets are gradually generated from the point cloud feature vectors that have undergone feature expansion to fill the sparse areas in the original point cloud and obtain final dense point cloud data.

4. The zero-labeled target segmentation method based on point cloud data optimization according to claim 3, characterized in that: The edge thermal value-based point cloud refinement module is used to receive the dense target point cloud data and refine it to generate a smoother point cloud, providing data for the subsequent segmentation of each part of the target; the edge thermal value-based point cloud refinement module takes the dense point cloud data as input and outputs the smoothed point cloud data based on gradient value calculation, edge thermal map generation and edge point shrinkage; The gradient value calculation includes: using the KD tree to perform a nearest neighbor query on the input dense point cloud data to obtain a set of neighbor points for each point in the point cloud; the heat map generation includes: calculating the geometric center of the neighbor point set of each point as the center coordinate of the center point of the current point, calculating the distance between each point and its corresponding center point, obtaining the position offset of each point in the point cloud, and calculating the difference between the offset value of the current point and the offset value of its neighboring points. The largest difference is the gradient value of each point. The calculated point cloud gradient value is normalized to [0,1] to obtain the edge thermal value of the point cloud. The closer the thermal value is to 1, the closer the point is to the edge, thereby forming the edge heat map; the edge point shrinkage includes: based on the generated edge heat map, shrinking the edge points to obtain the final optimized point cloud.

5. The zero-labeled target segmentation method based on point cloud data optimization according to claim 4, characterized in that: The S3 includes: S31, inputting the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model to obtain segmentation results of different parts of the target; S32, based on the segmentation results of the different parts of the target and the morphological analysis of the target, obtaining relevant phenotypic parameters of the target; the phenotypic parameters include one or more of the area, diameter, inclination, length and width of the different parts.

6. A zero-labeled target segmentation system based on point cloud data optimization, used to implement the zero-labeled target segmentation method based on point cloud data optimization according to any one of claims 1 to 5, characterized in that: include: A point cloud data optimization module (101) based on self-supervised training is used to optimize the actual production scene target point cloud data to be processed by a point cloud data optimization method based on self-supervised training to obtain optimized actual production scene target point cloud data; wherein the data characteristics of the actual production scene target point cloud data to be processed are one or more of incomplete, sparse and edge fuzzy; the data characteristics of the actual production scene target point cloud data after optimization are one or more of complete, dense and smooth; the optimization is performed from three aspects: structural integrity, data density and data smoothness; A model training module (102) is used to train a standard point cloud segmentation network based on labeled indoor target point cloud data to obtain a trained standard point cloud segmentation model; the data characteristics of the labeled indoor target point cloud data are complete target structure information, high data density and high data smoothness; The standard point cloud segmentation module (103) is used to input the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model, obtain the segmentation results of different parts of the target, and obtain related phenotypic parameters.

7. An electronic device, characterized in that: It includes a processor and a memory, the memory stores multiple instructions, and the processor is used to read the instructions and execute the zero-labeled target segmentation method based on point cloud data optimization as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, and the plurality of instructions can be read by a processor and executed according to any one of claims 1 to 5, the method for zero-labeled target segmentation based on point cloud data optimization.

Images