Zero-labeling target segmentation method and system based on point cloud data optimization

Abstract

The invention discloses a zero-labeling target segmentation method and system based on point cloud data optimization. The method comprises the following steps: optimizing actual production scene target point cloud data to be processed through a point cloud data optimization method based on self-supervised training to obtain optimized actual production scene target point cloud data; optimization is performed from structural integrity, data density and data smoothness; training the standard point cloud segmentation network based on the marked indoor target point cloud data to obtain a trained standard point cloud segmentation model; the marked indoor target point cloud data has the characteristics that the structural information of the target is complete, the data density is high, and the data smoothness is high; and inputting the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model, obtaining segmentation results of different parts of the target, and obtaining related phenotypic parameters. The marking problem of point cloud data collected in an actual production scene is avoided, and the capability of accurately segmenting different parts of a target is ensured.

Description

Technical Field

[0001] The present invention relates to the technical fields of deep learning, point cloud data optimization processing and target segmentation, and in particular to a zero-labeled target segmentation method and system based on point cloud data optimization. Background Art

[0002] At present, the calculation of target phenotypic parameters based on computer vision is mostly based on two-dimensional image analysis, and the relevant parameters are obtained through algorithms such as pattern recognition, machine learning and deep learning. However, the research using two-dimensional images is only shot from one perspective. Due to the complex morphological structure of the target (such as plants), serious occlusion and overlap, and the image is easily affected by light, it cannot fully describe the complete spatial distribution of the target (such as plants) structure, resulting in the statistical significance of the measured phenotypic traits being less reliable.

[0003] In contrast, three-dimensional data can retain more detailed spatial information of the research object and solve the problem of incomplete information in two-dimensional images. Based on three-dimensional data, the comprehensive morphological characteristics of the target can be quantitatively evaluated. By modeling the target (e.g., plant) with three-dimensional point cloud data and processing based on the acquired point cloud data, an effective way is provided for high-precision morphological characteristic analysis of the target (e.g., plant). A key step in this process is to segment different parts of the target (e.g., stems and leaves of plants) from point cloud data. Taking plants as an example, some related studies have used traditional point cloud algorithms to segment the stems and leaves of plants. However, it is highly dependent on predefined rules and prior knowledge of segmentation targets, and requires a tedious parameter adjustment process to achieve more accurate parameter calculation results, which is time-consuming and labor-intensive. In addition, due to the complex morphology of different plant individuals, these methods are not generalizable enough, resulting in unstable segmentation results. Therefore, more research tends to solve the shortcomings of traditional algorithms by using deep learning point cloud processing algorithms. Compared with traditional point cloud algorithms that are highly dependent on prior knowledge and manual design, deep learning methods can automatically learn features from original plant point clouds by automatically extracting complex structural information from a large number of plant point clouds, and have strong adaptability and accuracy. Since fast and accurate segmentation of plant organs is the key to obtaining plant phenotypic parameters, relevant researchers have been engaged in this research, most of which are aimed at ideal plant point cloud data collected in laboratory environments. Plants are planted in a standardized manner and managed in a refined manner, with simple morphology, high data density, less noise, clear edges, rich extractable features, and low algorithm processing difficulty. These advantages make the data collected in laboratory environments the starting point for many studies and provide ideal conditions for the development and optimization of algorithms. For example: based on the Eff-3DPSeg weakly supervised network framework, the stem and leaf segmentation and parameter calculation of plants are realized; based on the feature fusion FF-Net network, the stem and leaf segmentation of plant point clouds is realized; and the deep learning framework based on SqueezeNet is constructed to realize the segmentation and parameter calculation of point cloud plants. However, although the study of this data helps researchers master the technology of obtaining phenotypic parameters, the ultimate goal of the research is to solve practical problems, which often requires applying the research results to actual greenhouse production scenarios. However, because the plants in the production scenario are in a natural growth state, with diverse morphologies and complex environments, and are affected by multiple factors in the external environment, the density of the collected point cloud is low, there is a lot of noise, and the edges are blurred, and there are incomplete features and unclear features, which makes it difficult for the segmentation algorithm to process. Methods designed for collecting data in experimental environments are difficult to apply in practice. Therefore, a few researchers have conducted research on actual production scenarios, including: manually processing and labeling the point cloud data collected in actual production scenarios, dividing the data into training sets and test sets, and implementing the stem and leaf segmentation of plants through pointnet++.However, the point cloud data processing and annotation process is cumbersome and complex, and consumes a lot of manpower and time. Therefore, how to achieve accurate segmentation of actual production scene targets (for example, plant stems and leaves) while reducing annotation has become a problem to be solved.

[0004] In summary, there are still two key challenges in the current segmentation of actual production scene targets (e.g., plant stems and leaves):

[0005] (1) Low quality of point cloud data: Due to the morphological complexity and irregularity of targets in actual scenarios (e.g., plant growth), the collected point cloud data inevitably contains a lot of noise, resulting in blurred edges and large incompleteness. The low quality of point cloud data affects the segmentation accuracy of different target areas (e.g., stems and leaves of plants).

[0006] (2) High labeling cost: The morphology of targets under different conditions (for example, plants under different growth conditions) varies greatly, so the same model cannot be applied in different scenarios. As a result, data needs to be collected independently for labeling and model retraining in different application tasks, which results in high manual labeling costs. Summary of the invention

[0007] In order to solve the problems existing in the prior art, a zero-labeled target segmentation method and system based on point cloud data optimization is provided, which solves the following technical problems, including:

[0008] (1) Optimization of point cloud data collected in actual production scenarios: In order to reasonably guide the process of optimizing point cloud data collected in actual production scenarios to high-quality point clouds, and at the same time ensure that the optimized data can fully and accurately reflect the structural information of the target (e.g., plants), the present invention proposes a point cloud data cascade optimization network TP-PCONet based on self-supervised training based on the morphology and data characteristics analysis of the target (e.g., plants) in actual production scenarios. The point cloud data of actual production scenarios is optimized in terms of structural integrity, data density, and data smoothness, so that the segmentation model can effectively process the optimized high-quality point cloud data.

[0009] (2) Segmentation framework of different parts of the target (e.g., stems and leaves of plants) collected in actual production scenes based on zero annotation: In order to solve the problem of high annotation cost of point cloud data collected in actual production scenes, the present invention first uses a set of annotated high-quality target (e.g., plant) point clouds to train a point cloud segmentation model (collectively referred to as a standard point cloud segmentation model), and then guides the target (e.g., plant) point cloud data collected in actual production scenes to learn the features of the annotated indoor high-quality target (e.g., plant) point cloud data (with complete structural information), and inputs the optimized actual production scene point cloud data into the above-mentioned standard point cloud segmentation model to segment different parts of the target (e.g., stems and leaves of plants), thereby providing the possibility of obtaining three-dimensional phenotypic parameters (e.g., leaf area, stem diameter, etc. of plants). The point of this invention is that it effectively avoids the annotation problem of point cloud data collected in actual production scenes, while effectively ensuring the ability of different parts of the target (e.g., stems and leaves of plants) to be accurately segmented.

[0010] Taking the crop tomato as a typical example, phenotypic analysis of the morphological traits of tomato plants can help researchers analyze and track the growth of plants, which is of great significance to tomato breeding, planting guidance and growth monitoring. Among them, the surface coverage of the leaves is the largest, and the calculation of parameters such as leaf area can reflect the photosynthesis intensity of the plant. The stem is the basic structure of the plant. It is linked to other organs such as flowers and fruits. Calculation of parameters such as stem diameter can provide a deep understanding of the stress level of the plant. Traditional phenotypic traits are generally measured manually, which requires a lot of manpower and material resources, is inefficient, highly subjective, and often destroys the plant structure, causing irreversible losses. With the development and popularization of computer technology, the method of obtaining stem and leaf phenotypic parameters based on computer vision has gradually replaced manual measurement through non-invasive phenotypic measurement methods and high-throughput parameter extraction effects. However, there are still technical defects mentioned in the background technology section.

[0011] A first aspect of the present invention is to provide a zero-labeled target segmentation method based on point cloud data optimization, comprising:

[0012] S1, optimizing the actual production scene target point cloud data to be processed by a point cloud data optimization method based on self-supervised training to obtain optimized actual production scene target point cloud data; wherein the data characteristics of the actual production scene target point cloud data to be processed are one or more of incompleteness, sparseness and edge fuzziness; the data characteristics of the actual production scene target point cloud data after optimization are one or more of completeness, density and smoothness; the optimization is performed from the perspective of structural integrity, data density and data smoothness;

[0013] S2, training a standard point cloud segmentation network based on the labeled indoor target point cloud data to obtain a trained standard point cloud segmentation model; the data characteristics of the labeled indoor target point cloud data are complete target structure information, high data density and high data smoothness;

[0014] S3, inputting the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model, obtaining the segmentation results of different parts of the target, and obtaining related phenotypic parameters.

[0015] Preferably, the S1 comprises:

[0016] S11, training point cloud data cascade optimization network TP-PCONet through data pair construction strategy and self-supervision method;

[0017] S12, through the three steps of point cloud completion based on snowflake diffusion, point cloud upsampling based on gradual refinement, and point cloud refinement based on edge thermal value, the actual production scene target point cloud data to be processed are constrained in three aspects: backbone, surface, and contour, to obtain the optimized actual production scene target point cloud data.

[0018] Preferably, the S11 includes:

[0019] (1) Taking indoor high-quality point cloud data as input, obtaining simulated production scene point cloud data through a data pair construction strategy based on the point cloud characteristics of the actual production scene, and constructing a training data pair based on the simulated production scene point cloud data; the data pair construction strategy of the actual production scene point cloud characteristics includes: taking indoor high-quality point cloud data as input, and presetting the incompleteness coefficient n, sparseness coefficient m and noise coefficient k during processing to respectively implement surface incompleteness simulation, overall sparseness and noise addition, thereby simulating the occlusion and shaking of the target and the point cloud data changes caused during the data acquisition process, and processing the simulated production scene point cloud data obtained The method comprises: first, normalizing the indoor high-quality point cloud data; then randomly selecting n points in the indoor high-quality point cloud data as the center of the cube generated in the point cloud space, and setting the edge length; then removing the points belonging to the cube from the input indoor high-quality point cloud data to obtain the processed residual point cloud data; finally, randomly downsampling the processed residual point cloud data by m times, and adding Gaussian noise with k times the standard deviation to obtain the simulated production scene point cloud data;

[0020] (2) Training the point cloud data cascade optimization network TP-PCONet based on the training data pair and the self-supervised learning method, including: setting the upsampling network PUCRN as the baseline model; introducing a point cloud completion module based on snowflake diffusion and a point cloud refinement module based on edge thermal value on the basis of the point cloud upsampling module based on stepwise refinement in the baseline model to form the point cloud data cascade optimization network TP-PCONet, the input and output of the point cloud completion module based on snowflake diffusion, the point cloud upsampling module based on stepwise refinement and the point cloud refinement module based on edge thermal value correspond to the low-quality point cloud data before optimization and the high-quality point cloud data after optimization for the same target in the training data pair; training the point cloud data cascade optimization network TP-PCONet based on the training data pair and the self-supervised learning method, the training data pair is divided into the network input value of the point cloud data cascade optimization network TP-PCONet before training and the training true value, and the training is used to mine the internal inherent structure and mapping relationship of the training data pair to obtain the point cloud data cascade optimization network TP-PCONet.

[0021] Preferably, the point cloud completion module based on snowflake diffusion is used to predict the complete form of the point cloud of the actual production scene to be optimized, and provide complete data for the subsequent point cloud optimization, including three steps: point cloud backbone feature extraction, missing part seed point generation and snowflake point diffusion; the point cloud backbone feature extraction is used to perform backbone feature extraction on the point cloud to be optimized, and the encoder in the Transformer architecture is used to extract and encode the input point cloud to obtain the backbone feature vector of the point cloud; the missing part seed point generation includes: based on the extracted backbone feature vector, the Transformer decoder predicts the seed point set of the missing part, and the seed points in the seed point set of the missing part represent the approximate position and shape of the missing area; the snowflake point diffusion includes: using multiple SPD modules in the SnowFlakeNet network, with the seed points in the seed point set of the missing part as the center, gradually diffusing outward in a snowflake shape to generate new points until the entire point cloud data becomes complete, and the complete point cloud data is obtained.

[0022] Preferably, the point cloud upsampling module based on gradual refinement adopts feature expansion of point cloud data and coordinate reconstruction of point cloud data to obtain the final dense point cloud data; the feature expansion of the point cloud data includes: after the completed complete point cloud data is input into the point cloud upsampling module based on gradual refinement, feature expansion is performed through a multi-layer perceptron MLP to obtain an extended point cloud feature vector; the coordinate reconstruction of the point cloud data includes: fitting the distribution and regularity of the existing point cloud data, and gradually generating a new point cloud subset from the point cloud feature vector that has undergone the feature expansion through the multi-layer perceptron MLP again, filling the sparse area in the original point cloud, and obtaining the final dense point cloud data.

[0023] Preferably, the point cloud refinement module based on edge thermal value is used to receive the dense target point cloud data, refine it to generate a smoother point cloud, and provide data for the subsequent segmentation of each part of the target; the point cloud refinement module based on edge thermal value takes the dense point cloud data as input, and outputs the smoothed point cloud data based on gradient value calculation, edge thermal map generation and edge point shrinkage; the gradient value calculation includes: using KD tree to perform nearest neighbor query on the input dense point cloud data to obtain the neighbor point set of each point in the point cloud; the thermal map generation includes: calculating the neighbor point of each point The geometric center of the set is used as the central coordinate of the center point of the current point, and the distance between each point and its corresponding center point is calculated to obtain the position offset of each point in the point cloud. The difference between the offset value of the current point and the offset value of its neighboring points is calculated, and the largest difference is the gradient value of each point. The calculated point cloud gradient value is normalized to [0, 1] to obtain the edge thermal value of the point cloud. The closer the thermal value is to 1, the closer the point is to the edge, thereby forming the edge thermal map; the edge point shrinkage includes: based on the generated edge thermal map, shrinking the edge points to obtain the final optimized point cloud.

[0024] Preferably, S3 includes:

[0025] S31, inputting the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model to obtain segmentation results of different parts of the target;

[0026] S32, based on the segmentation results of the different parts of the target and the morphological analysis of the target, obtaining the relevant phenotypic parameters of the target; the phenotypic parameters include one or more of the area, diameter, inclination, length and width of the different parts.

[0027] The second aspect of the present invention is to provide a zero-label target segmentation system based on point cloud data optimization, which is used to implement the zero-label target segmentation method based on point cloud data optimization of the first aspect, comprising:

[0028] A point cloud data optimization module based on self-supervised training is used to optimize the actual production scene target point cloud data to be processed by a point cloud data optimization method based on self-supervised training to obtain optimized actual production scene target point cloud data; wherein the data characteristics of the actual production scene target point cloud data to be processed are one or more of incompleteness, sparseness and edge fuzziness; the data characteristics of the actual production scene target point cloud data after optimization are one or more of completeness, density and smoothness; the optimization is performed from three aspects: structural integrity, data density and data smoothness;

[0029] A model training module is used to train a standard point cloud segmentation network based on the labeled indoor target point cloud data to obtain a trained standard point cloud segmentation model; the data characteristics of the labeled indoor target point cloud data are complete target structure information, high data density and high data smoothness;

[0030] The standard point cloud segmentation module is used to input the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model, obtain the segmentation results of different parts of the target, and obtain related phenotypic parameters.

[0031] The third aspect of the present invention is to provide an electronic device, including a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is used to read the instructions and execute the zero-labeled target segmentation method based on point cloud data optimization as described in the first aspect.

[0032] The fourth aspect of the present invention is to provide a computer-readable storage medium, which stores multiple instructions, and the multiple instructions can be read by a processor and executed by the zero-labeled target segmentation method based on point cloud data optimization as described in the first aspect.

[0033] The method, system, electronic device, and computer-readable storage medium provided by the present invention have the following beneficial technical effects:

[0034] A zero-label target segmentation method based on point cloud data optimization is proposed. The segmentation framework of different parts of the target collected in the actual production scene based on zero annotation is used to provide the possibility for phenotypic parameter calculation. At the same time, the point cloud data collected in the actual production scene is optimized, and the point cloud cascade optimization network TP-PCONet based on self-supervised training is designed. The experimental results show that the proposed TP-PCONet achieves a CD distance of 8.45 and an F1-score of 73.5%, respectively. In addition, using the standard segmentation model, the performance of the point cloud data optimized by TP-PCONet is also better than other state-of-the-art optimization models. These results show that the proposed method has the potential to significantly improve the segmentation accuracy and efficiency of different parts of the target (e.g., stems and leaves of plants). BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of the zero-annotation target segmentation method based on point cloud data optimization according to the present invention;

[0036] Figure 2 It is a principle diagram of the zero-annotation target segmentation method based on point cloud data optimization described in the present invention;

[0037] Figure 3 Schematic diagram of the point cloud optimization method based on self-supervised training according to the present invention;

[0038] Figure 4 This is an example diagram of the point cloud data characteristics of an actual production scenario described in the present invention;

[0039] Figure 5 This is a schematic diagram of the structure of the point cloud data cascade optimization network TP-PCONet of the present invention;

[0040] Figure 6 This is a schematic diagram of the point cloud edge heat map described in the present invention;

[0041] Figure 7 It is a schematic diagram of the boundary between the stem and leaves of the plant point cloud according to the present invention;

[0042] Figure 8 A schematic diagram of the data pair composition in the self-supervised training phase of the point cloud data optimization network of the present invention;

[0043] Fig. 9 A schematic diagram of a sample of a point cloud data set of an actual production scenario to be optimized according to the present invention;

[0044] Fig.10 This is a schematic diagram of the visualization effect on the test set of the present invention. The circle in the figure shows the completion of the incomplete part;

[0045] Fig.11 This is a schematic diagram of the optimization results of the actual production scene point cloud in different optimization models described in the present invention. The circle in the figure represents the completion effect of the incomplete part;

[0046] Fig.12 This is a schematic diagram of the plant point cloud segmentation result in the actual production scene described in the present invention. The dark color represents leaves, and the light color represents stems;

[0047] Fig.13 This is a schematic diagram of the segmentation results of the plant point cloud in the actual production scene after TP-PCONet optimization in different segmentation models. The dark color represents leaves and the light color represents stems;

[0048] Fig.14 This is an architecture diagram of the zero-labeled target segmentation system based on point cloud data optimization described in the present invention;

[0049] Fig.15 It is a schematic diagram of the structure of the electronic device described in the present invention. DETAILED DESCRIPTION

[0050] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0051] Embodiment 1

[0052] like Figure 1As shown, this embodiment provides a zero-labeled target segmentation method based on point cloud data optimization, including:

[0053] S1, optimizing the actual production scene target point cloud data to be processed by a point cloud data optimization method based on self-supervised training to obtain optimized actual production scene target point cloud data; wherein the data characteristics of the actual production scene target point cloud data to be processed are one or more of incompleteness, sparseness and edge fuzziness; the data characteristics of the actual production scene target point cloud data after optimization are one or more of completeness, density and smoothness; the optimization is performed from three aspects: structural integrity, data density and data smoothness;

[0054] In this embodiment, Figure 2 As shown in ①, the point cloud data optimization method based on self-supervised training is used to solve the problem that the algorithm cannot effectively extract structural features due to quality factors such as incompleteness, sparseness and blurred edges of point clouds collected in actual production scenarios.

[0055] As a preferred embodiment, the S1 includes:

[0056] S11, training point cloud data cascade optimization network TP-PCONet through data pair construction strategy and self-supervision method;

[0057] In this embodiment, the optimization task for point cloud data of actual production scenarios requires that the model be able to learn the morphological characteristics of the plants and have the ability to extract and characterize complete and detailed three-dimensional structural information of the plants. How to set plant morphological supervision information for the model is the primary consideration. Among various supervision methods, the self-supervised learning method uses the data itself as a supervision signal, deeply mines the information contained in the data through the network, and updates the knowledge representation in real time. It can accurately capture the subtle spatial structural characteristics of tomato plants and is suitable for the optimization task of point cloud data of actual production scenarios. Therefore, the present invention uses the self-supervised training method to achieve a high degree of optimization of point clouds in actual production scenarios, such as Figure 3 The method includes a data pair construction strategy based on the point cloud characteristics of actual production scenarios and a point cloud data cascade optimization network TP-PCONet, which are introduced below.

[0058] In the model training stage: first, indoor high-quality point clouds are used as input, and through ① a data pair construction strategy based on the characteristics of actual production scene point clouds, simulated production scene point clouds are obtained to construct training data pairs; secondly, the produced training data pairs are used to self-supervise the training of ② point cloud data cascade optimization network TP-PCONet. In the model application stage: the actual production scene point cloud to be processed is input into the trained point cloud data optimization model to obtain the optimized actual production scene point cloud.

[0059] As a preferred implementation, the S11 includes:

[0060] (1) Taking indoor high-quality point cloud data as input, obtaining simulated production scene point cloud data through a data pair construction strategy based on the point cloud characteristics of the actual production scene, and constructing a training data pair based on the simulated production scene point cloud data; the data pair construction strategy of the actual production scene point cloud characteristics includes: taking indoor high-quality point cloud data as input, and presetting the incompleteness coefficient n, sparseness coefficient m and noise coefficient k during processing to respectively implement surface incompleteness simulation, overall sparseness and noise addition, thereby simulating the occlusion and shaking of the target and the point cloud data changes caused during the data acquisition process, and processing the simulated production scene point cloud data obtained The point cloud data is taken as output; the point cloud data of the simulated production scene is obtained by constructing a data pair strategy based on the point cloud characteristics of the actual production scene, including: first, normalizing the indoor high-quality point cloud data; then randomly selecting n points in the indoor high-quality point cloud data as the center of the cube generated in the point cloud space, and setting the edge length; then removing the points belonging to the cube from the input indoor high-quality point cloud data to obtain the processed residual point cloud data; finally, randomly downsampling the processed residual point cloud data by m times, and adding Gaussian noise with k times the standard deviation to obtain the simulated production scene point cloud data.

[0061] In this embodiment, the point cloud data optimization network based on self-supervised training needs to set sample training pairs of low-quality point clouds before optimization and high-quality point clouds after optimization for the same target, respectively as network input values ​​and training truth values, to mine the inherent structure and mapping relationship within the data. However, in actual production scenarios, the collected point cloud data cannot obtain high-quality point clouds as supervision truth values ​​due to the complex environment. Therefore, it is necessary to use high-quality point cloud data sets that can be obtained under indoor conditions as supervision truth values ​​to provide sufficient supervision information to guide the model to perform correct self-learning. Furthermore, in order to enable the network to have the ability to optimize the tomato plant point cloud in actual production scenarios, the present invention sets a simulated production scenario data set based on the characteristics of the point cloud data in actual production scenarios.

[0062] Examples of point cloud data features in actual production scenarios include: Figure 4 As shown in the figure, there are mainly the following aspects: First, due to the mutual occlusion between leaves and between leaves and stems, the collected point cloud is incomplete [(A): incomplete characteristic]; second, due to the complex environment in which the plant is located and the irregular growth form, it is difficult to collect a uniform and dense point cloud [(B): sparse characteristic]; in addition, in the actual production scene collection process, external factors such as wind disturbance and shaking of the collection equipment will cause the plant to be unable to fix a posture, resulting in blurred edges of the collected point cloud [(C): blurred edge characteristic].

[0063] Therefore, based on the above analysis of the characteristics of the actual production scene point cloud data, the specific manifestation of edge blur is that the points at the boundary are not distributed in a certain linear shape, and the local distribution is Gaussian noise. Therefore, this method performs a series of processing on the input indoor high-quality point cloud, including surface defect simulation, overall sparseness, and noise addition, to simulate plant occlusion, shaking, and point cloud data changes that may be caused during data collection. The simulated production scene point cloud data with the characteristics of the actual production scene point cloud is obtained. The construction strategy of the simulated production scene point cloud data set is shown in Algorithm 1.

[0064]

[0065]

[0066] As can be seen from Algorithm 1, in the process of constructing data pairs, the present invention uses indoor high-quality point clouds as input, and presets the incomplete coefficient n (set to 5 in the embodiment of the present invention), the sparse coefficient m (set to 4 in the embodiment of the present invention) and the noise coefficient k (set to 0.02 in the embodiment of the present invention) during processing, and uses the simulated production scene point cloud obtained after processing as output. In this process, first, the input point cloud data is normalized for subsequent processing; then n points are randomly selected from the input point cloud as the center of the cube generated in the point cloud space, and the edge length is set to 0.05 (the empirical value of the embodiment of the present invention); then the points belonging to the cube are removed from the input point cloud to obtain the processed residual point cloud; finally, it is randomly downsampled m times, and Gaussian noise with k times the standard deviation is added to finally obtain the simulated production scene point cloud. Through the above operations, a sample of the plant point cloud of the simulated production scene can be obtained, thereby constructing a pair of low-quality point cloud data before optimization and high-quality point cloud data after optimization that can be used for point cloud optimization network training.

[0067] (2) Training the point cloud data cascade optimization network TP-PCONet based on the training data pair and the self-supervised learning method, including: setting the upsampling network PUCRN as the baseline model; introducing a point cloud completion module based on snowflake diffusion and a point cloud refinement module based on edge thermal value on the basis of the point cloud upsampling module based on stepwise refinement in the baseline model to form the point cloud data cascade optimization network TP-PCONet, the input and output of the point cloud completion module based on snowflake diffusion, the point cloud upsampling module based on stepwise refinement and the point cloud refinement module based on edge thermal value correspond to the low-quality point cloud data before optimization and the high-quality point cloud data after optimization for the same target in the training data pair; training the point cloud data cascade optimization network TP-PCONet based on the training data pair and the self-supervised learning method, the training data pair is divided into the network input value of the point cloud data cascade optimization network TP-PCONet before training and the training true value, and the training is used to mine the internal inherent structure and mapping relationship of the training data pair to obtain the point cloud data cascade optimization network TP-PCONet.

[0068] In this embodiment, the self-supervised learning method enables the network to actively learn the mapping relationship between data pairs, so that it has the ability to optimize point clouds in actual production scenarios. In order to further guide the network to optimize from low-quality point clouds to high-quality point clouds, capture the fine structure of the target (in this embodiment, tomato plants), and avoid optimization distortion problems, this embodiment designs the network structure as follows.

[0069] First, this embodiment uses the upsampling network PUCRN as the baseline model. This model effectively avoids the distortion and discontinuity caused by excessive filling of point clouds at one time by gradually increasing the density of point clouds, which is suitable for the complex structure and rich details of tomato plants. Secondly, in order to enable the optimization network to have the ability to complete the incomplete point cloud, this embodiment introduces the point cloud completion module based on snowflake diffusion in SnowFlakeNet in the baseline model. This module uses the principle of snowflake diffusion to achieve smooth transition and natural connection of point cloud data, avoiding the problem of abrupt distortion in completion, so as to meet the different incomplete characteristics of stems and leaves in the plant point cloud. In view of the problem of blurred edges of stems and leaves caused by the data itself and the upsampling and completion operations, this embodiment constrains the surface of the point cloud and designs a point cloud upsampling module based on gradual refinement. At the same time, by constraining the contour of the point cloud, a point cloud refinement module based on edge thermal value is designed. The above three modules constitute the point cloud data cascade optimization network TP-PCONet designed in this embodiment. Figure 5The specific structural diagram of the point cloud data cascade optimization network TP-PCONet is shown. It is mainly composed of three sub-modules (A-1), (A-2), and (A-3) in cascade. Among them, (A-1): constrains the point cloud backbone to effectively fill in the missing parts and provide complete data for subsequent processing; (A-2): constrains the point cloud surface to optimize sparse data into dense data and improve data density; (A-3): constrains the contour of the completed and dense point cloud to obtain smooth point cloud data.

[0070] S12, through the three steps of point cloud completion based on snowflake diffusion, point cloud upsampling based on gradual refinement, and point cloud refinement based on edge thermal value, the actual production scene target point cloud data to be processed are constrained in three aspects: backbone, surface, and contour, to obtain the optimized actual production scene target point cloud data.

[0071] In practical applications, the optimized actual production scene point cloud data is input into Figure 2 The standard point cloud segmentation training model in ② is used to further segment the target (the tomato plant in this embodiment) into different parts (the stem and leaf segmentation in this embodiment).

[0072] In the actual production scenario tomato plant point cloud data optimization task, it is assumed that the input point cloud subset is represented as Where P 0i represents the i-th point of the point cloud, N is the total number of points in the input point cloud, and the task aims to obtain an optimized point cloud subset Where M represents the total number of points in the optimized point cloud, and M>N. The optimized point cloud subset P3 is obtained by formula (1), as shown below:

[0073]

[0074] In formula (1), express Figure 5 The completion module processing process based on snowflake diffusion shown in (A-1) is as follows: express Figure 5 (A-2) shows the upsampling module processing process based on step-by-step refinement, express Figure 5 (A-3) shows the refinement module processing process based on edge thermal value.

[0075] In this embodiment, according to the processing flow after the point cloud data input data optimization network, the three modules of the point cloud completion module based on snowflake diffusion, the point cloud upsampling module based on gradual refinement, and the point cloud refinement module based on edge thermal value and their processing process include:

[0076] (1) Point cloud completion module based on snowflake diffusion

[0077] Figure 5 (A-1) shows the point cloud completion module based on snowflake diffusion introduced by SnowFlakeNet in this embodiment, which is used to predict the complete form of the actual production scene point cloud to be optimized and provide complete data for subsequent point cloud optimization. This module takes the point cloud to be optimized as input and obtains the completed point cloud, as shown in formula (2) and formula (3).

[0078]

[0079] in Represents the input point cloud data to be optimized, Represents the output completed point cloud data, N and K represent the number of point cloud points of the input point cloud data to be optimized and the number of point cloud points of the output completed point cloud data, respectively, K>N, It consists of three steps, among which MFE(x i ,y i , z i ) represents the point cloud backbone feature extraction, SG(x i ,y i , z i ) indicates the missing part of the seed point generation, SPD(x i ,y i , z i ) represents snowflake point diffusion. The specific steps are as follows.

[0080] Step 1: Extract the backbone features of the point cloud, which is used to extract the backbone features of the point cloud P0 to be optimized. Use the encoder in the Transformer architecture to extract and encode the input point cloud to obtain the backbone feature vector of the point cloud, as shown in formula (4).

[0081] V m =T e (P0) (4)

[0082] in, represents the backbone feature vector, N represents the total number of points in the input point cloud, T e Represents the encoder of Transformer.

[0083] Step 2: Generate missing seed points, including: based on the extracted backbone feature vector V m , the Transformer decoder predicts the seed point set of the missing part, as shown in formula (5). The seed points in the seed point set of the missing part represent the approximate position and shape of the missing area.

[0084] P s =T d (V m ) (5)

[0085] in, n represents the number of points in the generated point cloud subset, T d Represents the decoder of Transformer.

[0086] Step 3: Snowflake point diffusion, including: using multiple SPD modules in the SnowFlakeNet network, with the seed point in the missing part of the seed point set in step 2 as the center, gradually diffuse outward in a snowflake shape to generate new points until the entire point cloud data becomes complete, and the complete point cloud data is obtained. The module diffusion process is shown in formula (6).

[0087] P s(i+1) =SPD(P si ) (6)

[0088] Among them, P si represents the input point cloud of the i+1th SPD module, P s(i+1) It represents the output point cloud of the i+1th SPD module. In this embodiment, when i=2, the point diffusion is completed.

[0089] The point cloud completion module based on snowflake diffusion obtains the backbone points by sampling the farthest point of the true point cloud, and calculates the loss with the predicted point cloud through CDloss to update the point cloud completion module based on snowflake diffusion. Through the above three steps, a complete point cloud of plants in the actual production scene can be obtained.

[0090] (2) Point cloud upsampling module based on gradual refinement

[0091] Figure 5 (A-2) shows the point cloud upsampling module based on gradual refinement in the baseline model PUCRN, which is mainly used to receive the completed plant point cloud and perform upsampling operations on it to provide dense data for subsequent point cloud optimization. This module takes the completed point cloud as input and outputs the dense point cloud, as shown in formulas (7) and (8).

[0092]

[0093] in, Represents the complete point cloud data of the input, represents the output dense point cloud data, r represents the upsampling multiple, which is set to 4 in this embodiment; It consists of two steps, FE(x i ,y i , z i ) represents the feature expansion of point cloud data, CR(x i ,y i , z i) represents the coordinate reconstruction of point cloud data. The specific steps are as follows.

[0094] Step 1: feature expansion of point cloud data, including: after the completed point cloud data is input into a point cloud upsampling module based on gradual refinement, feature expansion is performed through a multi-layer perceptron MLP to obtain an expanded point cloud feature vector, as shown in formula (9).

[0095] F up =MLP(P1) (9)

[0096] in, Represents the extended feature vector.

[0097] Step 2: coordinate reconstruction of point cloud data, including: fitting the distribution and regularity of the existing point cloud data, and gradually generating new point cloud subsets from the point cloud feature vector after the feature expansion through the multi-layer perceptron MLP again to fill the sparse areas in the original point cloud, as shown in formula (10).

[0098] S=MLP(F up ) (10)

[0099] in, Represents the newly generated point cloud subset. Since this embodiment uses a step-by-step upsampling method, α=r / 2 here; repeat the operation of step 2 to obtain the final dense point cloud data.

[0100] Based on the gradually refined point cloud upsampling module, the CD loss of the true point cloud and the predicted point cloud is directly calculated, and the generated dense point cloud is constrained. Through the above operations, a complete and dense point cloud of the actual production scene plant can be obtained.

[0101] (3) Point cloud refinement module based on edge thermal value

[0102] Figure 5 (A-3) shows the point cloud refinement module based on edge thermal value designed in this embodiment, which is mainly used to receive the dense plant point cloud data and refine it to generate a smoother point cloud to provide data for subsequent stem and leaf segmentation. The point cloud refinement module based on edge thermal value takes dense point cloud data as input and outputs smoothed point cloud data, as shown in formula (11) and formula (12).

[0103]

[0104] in, represents the input dense point cloud data, Represents the output refined point cloud data, It consists of three steps: GC(x i ,y i, z i ) indicates the gradient value calculation, HMG(x i ,y i , z i ) represents the generation of heat map, EPC(x i ,y i , z i ) indicates the edge point shrinkage. The specific implementation steps are as follows.

[0105] Step 1: Gradient value calculation, including: using the KD tree to perform nearest neighbor query on the input dense point cloud data to obtain the neighbor point set of each point in the point cloud, as shown in formula (13).

[0106] PN = KDTree(P i ) (13)

[0107] Where P i represents the input point cloud, represents the neighbor point set of each point, β represents the number of neighbor points of each point, which is set to 350 in this embodiment.

[0108] Step 2: Heat map generation, including: calculating the geometric center PC of the neighboring point set of each point as the center coordinate of the center point of the current point, as shown in formula (14).

[0109]

[0110] Calculate the distance between each point and its corresponding center point to obtain the position offset of each point in the point cloud, and calculate the difference between the offset value of the current point and the offset value of its neighboring points. The largest difference is the gradient value d of each point. i , as shown in formula (15).

[0111] d i =max||P i -PC i |-|PN i -PC i || (15)

[0112] In the point cloud data, the larger the gradient value of the point, the closer the point is to the edge. Conversely, the closer the point is to the inside of the point cloud, the calculated point cloud gradient value is normalized to [0, 1] to obtain the edge thermal value of the point cloud. The closer the thermal value is to 1, the closer the point is to the edge, thus forming a clear edge thermal map. The point cloud edge thermal map is as follows: Figure 6 As shown, Figure 6 (A) in the figure represents the original input point cloud; Figure 6 (B) in the figure shows the corresponding heat map visualization generated by the encoding.

[0113] Step 3: Edge point shrinkage, including: based on the generated edge heat map, shrinking the edge points to reduce the edge noise of the point cloud, making the point cloud outline clear, and obtaining the final optimized point cloud. This module calculates the edge heat value of the predicted point cloud and the true point cloud, and constrains the point cloud through the L1 loss of the two. Through the above operations, a complete, dense and smooth point cloud of plants in actual production scenes can be obtained.

[0114] After three modules, the actual production scene point cloud data to be optimized is processed by TP-PCONet step by step, and finally a point cloud with high-quality point cloud distribution characteristics is formed. The optimized point cloud structure is complete, dense and smooth, thus providing effective data for the segmentation of tomato plant stems and leaves.

[0115] S2, training a standard point cloud segmentation network based on the labeled indoor target point cloud data to obtain a trained standard point cloud segmentation model; the data feature of the labeled indoor target point cloud data is that the structure information of the target is complete;

[0116] As a preferred embodiment, Figure 2 The standard point cloud segmentation module shown in ② is used for plant stem and leaf segmentation. The standard point cloud segmentation module proposed in the embodiment of the present invention is trained based on a set of labeled indoor high-quality point cloud data sets Pheno4D.

[0117] S3, inputting the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model, obtaining the segmentation results of different parts of the target, and obtaining related phenotypic parameters.

[0118] In this embodiment, the standard point cloud segmentation model has been trained as follows Figure 2 As shown in ②, the trained standard point cloud segmentation model is trained by a set of labeled indoor high-quality point cloud data sets, which is collectively referred to as a standard point cloud segmentation module in the subsequent system description of the present invention.

[0119] As a preferred implementation, S3 includes:

[0120] S31, inputting the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model to obtain segmentation results of different parts of the target;

[0121] S32, based on the segmentation results of different parts of the target and the morphological analysis of the target, obtaining the relevant phenotypic parameters of the target.

[0122] As a preferred embodiment, the phenotypic parameters of S3 include one or more of the area, diameter, inclination, length and width of the different parts.

[0123] In this embodiment, it is specifically manifested as follows: based on the accurate stem and leaf segmentation results of the optimized actual production scene point cloud and the analysis of plant morphology, it is possible to accurately calculate the three-dimensional phenotypic parameters of the plant, such as leaf area, stem diameter, leaf inclination, etc.

[0124] Since the training data and application data of the model are very different in terms of collection environment, plant morphology and data distribution, the point cloud segmentation model needs to have the ability to generalize the model to adapt to different application data. At the same time, in the task of semantic segmentation of plant stems and leaves, due to the complex structure and shape of the plant, especially the interlacing and overlapping between the stems and leaves, the junction of the stems and leaves is difficult to be accurately segmented. Therefore, the model should have the ability to accurately segment the junction of the stems and leaves. Figure 7 shown.

[0125] In this embodiment, the segmentation model SoftGroup is used as the stem-leaf semantic segmentation model, which performs point-by-point semantic label prediction through U-Net, and then adopts a grouping-based method to group the semantic scores of the point cloud through the soft grouping module to reduce incorrect semantic label classification. SoftGroup uses a grouping-based method for point cloud semantic segmentation, which can directly process the entire scene without generating proposals, so that the network can more comprehensively understand the contextual information in the scene. When there is a large difference between the training data and the application data, it can also adapt to the new data distribution by capturing the overall structural characteristics of the scene, thereby improving the generalization ability of the model. At the same time, the grouping strategy can make full use of the spatial information and semantic information in the point cloud, reduce the misjudgment of the junction of the stem and leaves, effectively handle complex situations, and accurately identify the stems and leaves, thereby achieving their precise semantic segmentation.

[0126] Application example technical effect verification:

[0127] In order to verify the effectiveness of the method proposed in the present invention, the present invention constructs a data set for point cloud data optimization and a data set for standard point cloud segmentation, as shown in Table 1.

[0128] Table 1 Dataset introduction

[0129]

[0130]

[0131] 1. Dataset

[0132] (I) Point cloud data optimization module dataset based on self-supervised training

[0133] 1. Training dataset for point cloud data optimization module based on self-supervised training

[0134] The training dataset of the point cloud data optimization module based on self-supervised training consists of low-quality point cloud training data pairs before optimization and high-quality point cloud training data pairs after optimization. The high-quality point cloud dataset uses the tomato plant point cloud dataset in the Pheno4D dataset. This dataset was collected in a laboratory environment, and the point cloud is complete, dense, and has clear edges. Figure 8 As shown in (A) in the figure, the low-quality dataset is obtained by processing the high-quality dataset through the data pair construction strategy based on the point cloud characteristics of the actual production scene, as shown in Figure 8 As shown in (B) in the figure, in this example, 77 tomato plants in the Pheno4D dataset are processed to obtain 77 training data pairs, of which 66 are used as training sets and 11 are used as test sets.

[0135] 2. Point cloud dataset of actual production scenarios to be optimized

[0136] The application dataset of the point cloud data optimization module based on self-supervised training is the point cloud dataset of the actual production scene to be optimized. In this embodiment, CF Petit Puyo Yellow and CF Petit Puyo in actual production are selected as experimental objects, and the data are collected from the tomato planting base of the University of Tokyo in Japan (35°44′13.95″N139°32′23.53″E). In this experiment, the onboard RGB camera (1 / 2.3-inch CMOS, lens FC7303, focal length 4.49mm, maximum image resolution size 4000×2250) on a low-cost industrial-grade drone (Mavic Mini 2, DJI) was used to collect data. The flight altitude was set to 0.5~2m and the distance from the tomato plant was 1.5~2m. The captured RGB images were three-dimensionally reconstructed using the commercial software clomap based on the principle of multi-view reconstruction to obtain the reconstructed point cloud of the tomato plant. A total of 14 tomato plant point clouds were obtained for application testing. The dataset is as follows: Fig. 9 As shown. Among them, Fig. 9 (A), (B) and (C) represent different plant point cloud samples respectively.

[0137] (II) Standard point cloud segmentation module dataset

[0138] 1. Training dataset for standard point cloud segmentation model

[0139] This embodiment uses the tomato plant point cloud dataset of the Pheno4D dataset as the training dataset of the standard point cloud segmentation model. The dataset performs instance-level annotation on the stems and leaves of tomato plants. This article uses 66 labeled samples as the training set and 11 labeled samples as the test set to train the standard point cloud segmentation network.

[0140] 2. Optimized point cloud dataset of actual production scene

[0141] The application data set of the standard point cloud segmentation model is the optimized actual production scene point cloud data set, which is the optimized actual production scene point cloud data set obtained in this embodiment. At the same time, it is annotated with stems and leaves through the commercial point cloud annotation software CloudCompare as the segmentation result to calculate the true value for evaluation.

[0142] 2. Evaluation Indicators

[0143] This embodiment formulates evaluation indicators for the point cloud data optimization experiment and the point cloud segmentation experiment, respectively, as follows.

[0144] (I) Point cloud data optimization experiment evaluation indicators

[0145] In the point cloud data optimization experiment, chamfer distance (CD-distance) and F1-Score are used as evaluation indicators to measure the effect of point cloud optimization.

[0146] Chamfer distance: It can measure the similarity between the predicted point cloud and the true point cloud at the set level. The smaller the chamfer distance, the more similar the two point clouds are. For each sample, the predicted point set P pred and the truth point set P gt Chamfer distance d CD The calculation formula is shown in formula (16).

[0147]

[0148] Among them, |P pred | represents the prediction point set P pred Points, |P gt | represents the truth point set P gt The number of points, ||pg|| represents the distance between p and g. From formula (17), we can see that the chamfer distance represents the minimum distance between two sets of point sets. If the distance is larger, the difference between the two sets of point clouds is greater; the smaller the distance is, the higher the similarity of the point clouds is.

[0149] F1-Score: The similarity between Ppred and Pgt is measured by the harmonic mean of precision and recall. Precision represents the percentage of predicted points within a certain distance from the true value point, which represents the accuracy of the prediction; recall represents the percentage of true value points within a certain distance from the predicted point, which represents the completeness of the prediction. The specific calculation method is shown in formula (17)-formula (19).

[0150]

[0151] Among them, Precision(d) and Recall(d) respectively represent the accuracy Precision and recall Recall calculated under the distance threshold d. In the PU1K data format, d is set to 0.02. The higher the F1-Score, the higher the similarity between the two point sets.

[0152] (II) Point cloud segmentation experiment evaluation indicators

[0153] In the plant point cloud stem and leaf segmentation experiment, the intersection over union (IoU) is used as the evaluation index for plant stem and leaf segmentation. The higher the IoU value, the better the segmentation performance. The calculation formula of IoU is shown in formula (20).

[0154]

[0155] Among them, TP represents the number of true positive points of the current semantic category, FP represents the number of false positive points of the current category, and FN represents the number of false negative points. From formula (20), we can see that IoU can reflect the degree of overlap between the predicted area of ​​each semantic category and the corresponding real area. The closer the IoU value is to 1, the higher the semantic segmentation accuracy is and the more accurate the segmentation is.

[0156] (III) Experimental setup

[0157] This experiment is deployed on a computer platform with an Intel Core i7-10700k CPU processor (64GB RAM). The model is built, trained, and verified using a GeForce RTX 3090 GPU (24GB) graphics card, Ubuntu 18.04LTS operating system, and Python 3.7 programming language under the Pytorch 1.10.0 framework.

[0158] Among them, the TP-PCONet model is trained with the batch size set to 4, the learning rate set to 0.001, and a total of 100 training cycles: The SoftGroup model is trained using the adaptive moment estimation (Adam) optimizer, with the batch size set to 4, the learning rate set to 0.001, and a total of 100 training cycles.

[0159] Conclusion

[0160] (I) Point cloud data optimization experiment

[0161] The point cloud data optimization experiment verifies the point cloud optimization effect of the point cloud data optimization module based on self-supervised training, which consists of a training data set experiment of the point cloud data optimization module based on self-supervised training and a point cloud data optimization experiment in an actual production scenario.

[0162] 1. Training data set experiment of point cloud data optimization module based on self-supervised training

[0163] This example selects other classic point cloud data optimization networks for comparative experiments, including the baseline design model PUCRN of TP-PCONet, as well as PU-GAN, PU-GCN, and Grad-PU, to prove the effectiveness of TP-PCONet designed in this paper. The experimental results are shown in Table 2.

[0164] Table 2 Comparison of experimental results of different data optimization networks based on self-supervised training

[0165]

[0166]

[0167] As shown in Table 2, TP-PCONet achieved the best results and was superior to other comparison algorithms, which shows the effectiveness of its point cloud data optimization. The optimization visualization effects of the test set point cloud after being processed by different optimization models are shown in Figure 2. Fig.10 As shown. Fig.10 From (A) to (G), we can see that TP-PCONet achieves better results. It can not only densify the input point cloud, but also complete the incomplete parts and smooth the overall point cloud.

[0168] In order to further verify the effectiveness of each improved module in the TP-PCONet proposed in the present invention, including the point cloud completion module based on snowflake diffusion (denoted as SP) and the point cloud refinement module based on edge thermal value (denoted as EH), the embodiment of the present invention conducted an ablation experiment, and the results are shown in Table 3, where PUCRN+SP+EH is the TP-PCONet designed in this embodiment.

[0169] Table 3. Data optimization ablation experiment results based on self-supervised training

[0170]

[0171] As can be seen from Table 3, through the point cloud completion module based on snowflake diffusion and the point cloud refinement module based on edge thermal value, this paper obtains higher accuracy based on the baseline model, and achieves the best result when the two are combined.

[0172] 2. Point cloud data optimization experiment in actual production scenarios

[0173] In this embodiment, the actual production scene point cloud data set to be optimized is input into the trained point cloud data optimization model. Since the data set does not have the optimized true value for evaluation calculation, the optimized visualization effect is shown here. Fig.11 As shown in (A)-(F) in Figure 1, the point cloud optimization effect is evaluated through the point cloud segmentation experiment results. Fig.11As can be seen from (A)-(F) in the figure, after the TP-PCONet data optimization, the point cloud of the tomato plant collected in the actual production scene has become denser and more complete, and the point cloud contour is clearer.

[0174] In summary, experimental results have verified that the point cloud data optimization experiment based on self-supervised training proposed in the embodiment of the present invention can highly optimize the point cloud data and provide an analysis data basis for subsequent steps.

[0175] (II) Point cloud segmentation experiment

[0176] The point cloud segmentation experiment is used to verify the effectiveness and applicability of the optimization method designed in the present invention, and consists of an actual production scenario point cloud segmentation experiment based on the point cloud data optimization method and a comparison experiment of different segmentation models based on the point cloud optimization method.

[0177] 1. Point cloud segmentation experiment in actual production scenarios based on point cloud data optimization method

[0178] In order to verify the effectiveness of the TP-PCONet proposed in this paper, this embodiment uses the training data set of the standard point cloud segmentation module to pre-train SoftGroup, obtains the standard stem-leaf segmentation model, and segments the optimized actual production scene point cloud data set. Based on the optimization results of different data optimization networks, this embodiment conducts comparative experiments, as shown in Table 4.

[0179] Table 4 Comparison of experimental results of tomato plant stem and leaf segmentation in actual production scenarios after optimization by different optimization models

[0180]

[0181] As can be seen from Table 4, compared with not performing data optimization, data optimization of the actual production scene point cloud can improve the stem and leaf segmentation effect of the plant point cloud, which proves the effectiveness of the method proposed in this embodiment. In addition, the TP-PCONet of this embodiment achieved the best effect among the comparison algorithms, which proves the effectiveness of TP-PCONet. Among them, after the optimization of TP-PCONet proposed in this embodiment, the leaf segmentation accuracy of the tomato plant in the actual production scene is 0.882, and the stem segmentation accuracy is 0.471. The plant segmentation results of the actual production scene are shown in Figure 4. Fig.12 As shown in (A)-(F) in FIG.

[0182] In order to further verify the effectiveness of each improved module in the TP-PCONet of this embodiment in the stem-leaf segmentation method, including the point cloud completion module based on snowflake diffusion (denoted as SP) and the point cloud refinement module based on edge thermal value (denoted as EH), this embodiment conducted an ablation experiment, and the results are shown in Table 5, where PUCRN+SP+SHE is the TP-PCONet designed in this embodiment.

[0183] Table 5 Experimental results of tomato plant stem and leaf segmentation ablation in actual production scenarios

[0184]

[0185] It can be seen from Table 5 that through the point cloud completion module based on snowflake diffusion and the point cloud refinement module based on edge thermal value, this embodiment achieves higher accuracy in plant stem and leaf segmentation in actual production scenarios, and achieves the best result when the two are combined.

[0186] 2. Comparative experiment of different segmentation models based on point cloud optimization method

[0187] In order to verify that the point cloud data optimization method proposed in this embodiment has an improvement effect on the segmentation results of different segmentation models, this embodiment selects different classic point cloud segmentation networks and conducts experiments on actual production scene point cloud data before and after TP-PCONet optimization, including SoftGroup, PointNet++, PointTransformer, and GeoSegNet used in this embodiment. As shown in Table 6.

[0188] Table 6 Comparison of experimental results of tomato plant stem and leaf segmentation in actual production scenarios using different segmentation models

[0189]

[0190] As shown in Table 6, after being processed by the optimization method proposed in this embodiment, different segmentation models have achieved improvements in segmentation accuracy. At the same time, the segmentation model SoftGroup used in the method of this embodiment has achieved higher accuracy, which proves the wide applicability of the method proposed in this embodiment. Different segmentation models are used to segment the actual production scene data set after TP-PCONet optimization. The results are as follows: Fig.13 As shown in (A)-(G) in . Fig.13 As can be seen from Table 6, the SoftGroup used in this embodiment achieves a better segmentation effect and can accurately segment the junction of the stem and leaves, which proves the effectiveness of the segmentation model finally selected in this embodiment.

[0191] Embodiment 2

[0192] like Fig.14 As shown, this embodiment provides a zero-labeled target segmentation system based on point cloud data optimization, which is used to implement the method of embodiment 1, including:

[0193] The point cloud data optimization module 101 based on self-supervised training is used to optimize the actual production scene target point cloud data to be processed by a point cloud data optimization method based on self-supervised training to obtain optimized actual production scene target point cloud data; wherein the data characteristics of the actual production scene target point cloud data to be processed are one or more of incompleteness, sparseness and edge fuzziness; the data characteristics of the actual production scene target point cloud data after optimization are one or more of completeness, density and smoothness; the optimization is performed from three aspects of structural integrity, data density and data smoothness;

[0194] The model training module 102 is used to train the standard point cloud segmentation network based on the labeled indoor target point cloud data to obtain a trained standard point cloud segmentation model; the data characteristics of the labeled indoor target point cloud data are complete target structure information, high data density and high data smoothness;

[0195] The standard point cloud segmentation module 103 is used to input the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model, obtain the segmentation results of different parts of the target, and obtain related phenotypic parameters.

[0196] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the method described in the first embodiment.

[0197] like Fig.15 As shown, the present invention also provides an electronic device, including a processor 301 and a memory 302 connected to the processor 301, wherein the memory 302 stores a plurality of instructions, which can be loaded and executed by the processor so that the processor can execute the method described in Example 1.

[0198] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A zero-labeled target segmentation method based on point cloud data optimization, characterized in that: include: S1, optimizing the actual production scene target point cloud data to be processed by a point cloud data optimization method based on self-supervised training to obtain optimized actual production scene target point cloud data; wherein the data characteristics of the actual production scene target point cloud data to be processed are one or more of incompleteness, sparseness and edge fuzziness; the data characteristics of the actual production scene target point cloud data after optimization are one or more of completeness, density and smoothness; the optimization is performed from the perspective of structural integrity, data density and data smoothness; S2, training a standard point cloud segmentation network based on the labeled indoor target point cloud data to obtain a trained standard point cloud segmentation model; the data characteristics of the labeled indoor target point cloud data are complete target structure information, high data density and high data smoothness; S3, inputting the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model, obtaining the segmentation results of different parts of the target, and obtaining related phenotypic parameters.

2. The zero-labeled target segmentation method based on point cloud data optimization according to claim 1, characterized in that: The S1 includes: S11, training point cloud data cascade optimization network TP-PCONet through data pair construction strategy and self-supervision method; S12, through the three steps of point cloud completion based on snowflake diffusion, point cloud upsampling based on gradual refinement, and point cloud refinement based on edge thermal value, the actual production scene target point cloud data to be processed are constrained in three aspects: backbone, surface, and contour, to obtain the optimized actual production scene target point cloud data.

3. The zero-labeled target segmentation method based on point cloud data optimization according to claim 2, characterized in that: The S11 includes: (1) Taking indoor high-quality point cloud data as input, obtaining simulated production scene point cloud data through a data pair construction strategy based on the point cloud characteristics of the actual production scene, and constructing a training data pair based on the simulated production scene point cloud data; the data pair construction strategy of the actual production scene point cloud characteristics includes: taking indoor high-quality point cloud data as input, and presetting the incompleteness coefficient n, sparseness coefficient m and noise coefficient k during processing to respectively implement surface incompleteness simulation, overall sparseness and noise addition, thereby simulating the occlusion and shaking of the target and the point cloud data changes caused during the data acquisition process, and processing the simulated production scene point cloud data obtained The method comprises: first, normalizing the indoor high-quality point cloud data; then randomly selecting n points in the indoor high-quality point cloud data as the center of the cube generated in the point cloud space, and setting the edge length; then removing the points belonging to the cube from the input indoor high-quality point cloud data to obtain the processed residual point cloud data; finally, randomly downsampling the processed residual point cloud data by m times, and adding Gaussian noise with k times the standard deviation to obtain the simulated production scene point cloud data; (2) Training the point cloud data cascade optimization network TP-PCONet based on the training data pair and the self-supervised learning method, including: setting the upsampling network PUCRN as the baseline model; introducing a point cloud completion module based on snowflake diffusion and a point cloud refinement module based on edge thermal value on the basis of the point cloud upsampling module based on stepwise refinement in the baseline model to form the point cloud data cascade optimization network TP-PCONet, the input and output of the point cloud completion module based on snowflake diffusion, the point cloud upsampling module based on stepwise refinement and the point cloud refinement module based on edge thermal value correspond to the low-quality point cloud data before optimization and the high-quality point cloud data after optimization for the same target in the training data pair; training the point cloud data cascade optimization network TP-PCONet based on the training data pair and the self-supervised learning method, the training data pair is divided into the network input value of the point cloud data cascade optimization network TP-PCONet before training and the training true value, and the training is used to mine the internal inherent structure and mapping relationship of the training data pair to obtain the point cloud data cascade optimization network TP-PCONet.

4. The zero-labeled target segmentation method based on point cloud data optimization according to claim 3 is characterized in that: The point cloud completion module based on snowflake diffusion is used to predict the complete form of the point cloud of the actual production scene to be optimized, and provide complete data for the subsequent point cloud optimization, including three steps: point cloud backbone feature extraction, missing part seed point generation and snowflake point diffusion; the point cloud backbone feature extraction is used to extract backbone features of the point cloud to be optimized, and the encoder in the Transformer architecture is used to extract and encode the input point cloud to obtain the backbone feature vector of the point cloud; the missing part seed point generation includes: based on the extracted backbone feature vector, the Transformer decoder predicts the seed point set of the missing part, and the seed points in the seed point set of the missing part represent the approximate position and shape of the missing area; the snowflake point diffusion includes: using multiple SPD modules in the SnowFlakeNet network, with the seed points in the seed point set of the missing part as the center, gradually diffusing outward in a snowflake shape to generate new points until the entire point cloud data becomes complete and the complete point cloud data is obtained.

5. The zero-labeled target segmentation method based on point cloud data optimization according to claim 4, characterized in that: The point cloud upsampling module based on gradual refinement adopts feature expansion of point cloud data and coordinate reconstruction of point cloud data to obtain the final dense point cloud data; the feature expansion of the point cloud data includes: after the completed complete point cloud data is input into the point cloud upsampling module based on gradual refinement, feature expansion is performed through a multi-layer perceptron MLP to obtain an extended point cloud feature vector; the coordinate reconstruction of the point cloud data includes: fitting the distribution and regularity of the existing point cloud data, and gradually generating a new point cloud subset from the point cloud feature vector after the feature expansion through the multi-layer perceptron MLP again, filling the sparse area in the original point cloud, and obtaining the final dense point cloud data.

6. The zero-labeled target segmentation method based on point cloud data optimization according to claim 5, characterized in that: The point cloud refinement module based on edge thermal value is used to receive the dense target point cloud data, refine it to generate a smoother point cloud, and provide data for the subsequent segmentation of each part of the target; the point cloud refinement module based on edge thermal value takes the dense point cloud data as input, and outputs the smoothed point cloud data based on gradient value calculation, edge thermal map generation and edge point shrinkage; The gradient value calculation includes: using the KD tree to perform a nearest neighbor query on the input dense point cloud data to obtain a neighbor point set for each point in the point cloud; the heat map generation includes: calculating the geometric center of the neighbor point set of each point as the center coordinate of the center point of the current point, calculating the distance between each point and its corresponding center point, obtaining the position offset of each point in the point cloud, and calculating the difference between the offset value of the current point and the offset value of its neighboring points. The largest difference is the gradient value of each point. The calculated point cloud gradient value is normalized to [0, 1] to obtain the edge thermal value of the point cloud. The closer the thermal value is to 1, the closer the point is to the edge, thereby forming the edge thermal map; the edge point shrinkage includes: based on the generated edge thermal map, shrinking the edge points to obtain the final optimized point cloud.

7. The zero-labeled target segmentation method based on point cloud data optimization according to claim 6, characterized in that: The S3 includes: S31, inputting the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model to obtain segmentation results of different parts of the target; S32, based on the segmentation results of the different parts of the target and the morphological analysis of the target, obtaining the relevant phenotypic parameters of the target; the phenotypic parameters include one or more of the area, diameter, inclination, length and width of the different parts.

8. A zero-label target segmentation system based on point cloud data optimization, used to implement the zero-label target segmentation method based on point cloud data optimization according to any one of claims 1 to 7, characterized in that: include: A point cloud data optimization module (101) based on self-supervised training is used to optimize the actual production scene target point cloud data to be processed by a point cloud data optimization method based on self-supervised training to obtain optimized actual production scene target point cloud data; wherein the data characteristics of the actual production scene target point cloud data to be processed are one or more of incompleteness, sparseness and edge fuzziness; the data characteristics of the actual production scene target point cloud data after optimization are one or more of completeness, density and smoothness; the optimization is performed from three aspects of structural integrity, data density and data smoothness; A model training module (102) is used to train a standard point cloud segmentation network based on the labeled indoor target point cloud data to obtain a trained standard point cloud segmentation model; the data characteristics of the labeled indoor target point cloud data are complete target structure information, high data density and high data smoothness; The standard point cloud segmentation module (103) is used to input the optimized actual production scene target point cloud data into the trained standard point cloud segmentation model, obtain the segmentation results of different parts of the target, and obtain related phenotype parameters.

9. An electronic device, characterized in that: It includes a processor and a memory, the memory stores multiple instructions, and the processor is used to read the instructions and execute the zero-labeled target segmentation method based on point cloud data optimization as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the plurality of instructions can be read by a processor and executed according to any one of claims 1 to 7, the method for zero-labeled target segmentation based on point cloud data optimization.

Images