A point cloud data optimization method and system based on self-supervised training

Abstract

The application discloses a kind of point cloud data optimization method and system based on self-supervised training.The optimization method is used to optimize the actual production scene target point cloud data to be processed to obtain the actual production scene target point cloud data after optimization, the characteristics of the actual production scene target point cloud data to be processed are incomplete, sparse or edge blur;The characteristics of the actual production scene target point cloud data after optimization are complete, dense or smooth;From the structural integrity, data density and data smoothness optimization, including: through data pair construction strategy and self-supervised method training point cloud data cascade optimization network TP-PCONet;Through the point cloud completion based on snowflake diffusion, the point cloud upsampling based on step-by-step refinement and the point cloud refinement based on edge thermal value, the actual production scene target point cloud data to be processed is sequentially constrained in three aspects of main body, surface and contour, to obtain the actual production scene target point cloud data after optimization, and the corresponding electronic equipment and computer readable storage medium are also disclosed.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and point cloud data optimization processing, and in particular to a point cloud data optimization method and system based on self-supervised training. Background Art

[0002] In the segmentation of different parts of the target in actual production scenarios, for example, the segmentation of the stems and leaves of plants, the density and completeness of the application data are important influencing factors. Since plants are affected by multiple biological, physical and environmental factors during their natural growth, plants show a high degree of morphological complexity and irregularity, which makes it difficult for the acquisition equipment to obtain uniform and dense point cloud data during the acquisition process. There are currently studies on the optimization of point cloud data, among which the point cloud upsampling method based on deep learning is the current mainstream solution. Point cloud upsampling refers to the task of generating a dense point cloud given a set of sparse point cloud inputs, which is a necessary task to complete downstream visual tasks. These include: using a point cloud upsampling network, PU-Net, which learns the multi-level features of each point and implicitly expands the point set in the feature space through multi-branch convolutional units, thereby achieving point cloud upsampling; or using a PU-Transformer, which improves the connection between the feature maps of points and channels through an encoding and decoding structure and a multi-head self-attention structure, thereby generating a more realistic point cloud; or using a point cloud upsampling network, Grad-PU, which first interpolates the point cloud and then refines the position of the interpolated points through iterative optimization, achieving more accurate point cloud upsampling. It can be seen that point cloud upsampling technology can be used to optimize plant point cloud data collected in actual production scenarios, providing data support for subsequent plant stem and leaf segmentation.

[0003] However, existing point cloud upsampling networks add points to existing points. When the input point cloud is incomplete, resulting in an incomplete morphological structure, these networks are limited by the existing data points and struggle to accurately restore the true structure of the point cloud, thus limiting their application accuracy on complex point clouds. Furthermore, when the input point cloud is noisy, the redundant noise features of the existing data points blur the point cloud edges, interfering with the network's feature extraction and leading to inaccurate upsampling results, thus limiting their effectiveness. The plant point cloud to be optimized, due to its complex leaf structure, the tortuous stem orientation, and the randomness of its overall growth, inevitably suffers from incompleteness during acquisition due to occlusion between leaves and stems. Furthermore, because plants are flexible objects, they can sway during acquisition due to external factors such as wind and equipment movement, resulting in severe point cloud noise and blurred edges. Therefore, existing point cloud upsampling algorithms face difficulties in optimizing plant point clouds. The degree of data optimization directly affects the accuracy of stem and leaf segmentation, hindering accurate stem and leaf segmentation in real-world production scenarios. One of the research focuses of this invention is how to construct a point cloud data optimization model that can achieve highly optimized point clouds in actual production scenarios and provide analyzable data for the segmentation of different parts of subsequent targets (such as the stems and leaves of plants) and the calculation of phenotypic parameters.

[0004] In summary, the following key challenges still exist in point cloud data work at this stage:

[0005] Low point cloud data quality: Due to the morphological complexity and irregularity of targets in actual production scenarios (for example, plant growth), the collected point cloud data inevitably contains a lot of noise, resulting in blurred edges and large incompleteness. The low quality of the point cloud data affects the segmentation accuracy of different parts of the target (for example, the stems and leaves of the plant). Summary of the Invention

[0006] In order to solve the problems existing in the prior art, a point cloud data optimization method and system based on self-supervised training are provided, which solves the technical problem of optimizing point cloud data collected in actual production scenarios. In order to reasonably guide the process of optimizing point cloud data collected in actual production scenarios to high-quality point clouds, and at the same time ensure that the optimized data can fully and accurately reflect the structural information of the target (e.g., plants), the present invention proposes a point cloud data cascade optimization network TP-PCONet based on self-supervised training based on the analysis of the morphology and data characteristics of the target (e.g., plants) in actual production scenarios. The point cloud data of actual production scenarios are optimized in terms of structural integrity, data density, and data smoothness, so that the segmentation model can effectively process the optimized high-quality point cloud data.

[0007] A first aspect of the present invention is to provide a point cloud data optimization method based on self-supervised training, which is used to optimize actual production scene target point cloud data to be processed to obtain optimized actual production scene target point cloud data, wherein the data characteristics of the actual production scene target point cloud data to be processed are one or more of incompleteness, sparseness and edge fuzzy; the data characteristics of the optimized actual production scene target point cloud data are one or more of completeness, density and smoothness; the optimization is performed based on structural integrity, data density and data smoothness, and the method includes:

[0008] S1, training the point cloud data cascade optimization network TP-PCONet through data pair construction strategy and self-supervision method;

[0009] S2, through the three steps of point cloud completion based on snowflake diffusion, point cloud upsampling based on gradual refinement, and point cloud refinement based on edge thermal value, the actual production scene target point cloud data to be processed is sequentially constrained in terms of backbone, surface, and contour to obtain the optimized actual production scene target point cloud data.

[0010] Preferably, the S1 includes:

[0011] S11, using indoor high-quality point cloud data as input, obtaining simulated production scene point cloud data through a data pair construction strategy based on the point cloud characteristics of the actual production scene, and constructing a training data pair based on the simulated production scene point cloud data; the data pair construction strategy of the actual production scene point cloud characteristics includes: using indoor high-quality point cloud data as input, and presetting the defect coefficient n, sparse coefficient m and noise coefficient k during processing to respectively implement surface defect simulation, overall sparseness and noise addition, thereby simulating the occlusion, shaking of the target and the point cloud data changes caused during the data acquisition process, and processing the simulated production scene point cloud obtained Data is output; the method of obtaining the simulated production scene point cloud data by the data pair construction strategy based on the point cloud characteristics of the actual production scene includes: first, normalizing the indoor high-quality point cloud data; then randomly selecting n points in the indoor high-quality point cloud data as the center of a cube generated in the point cloud space, and setting the edge length; then removing the points belonging to the cube from the input indoor high-quality point cloud data to obtain processed residual point cloud data; finally, randomly downsampling the processed residual point cloud data by m times and adding Gaussian noise with k times the standard deviation to obtain the simulated production scene point cloud data;

[0012] S12, training the point cloud data cascade optimization network TP-PCONet based on the training data pair and the self-supervised learning method, including: setting the upsampling network PUCRN as the baseline model; introducing a point cloud completion module based on snowflake diffusion and a point cloud refinement module based on edge thermal value on the basis of the point cloud upsampling module based on step-by-step refinement in the baseline model to constitute the point cloud data cascade optimization network TP-PCONet, the input and output of the point cloud completion module based on snowflake diffusion, the point cloud upsampling module based on step-by-step refinement and the point cloud refinement module based on edge thermal value correspond to the low-quality point cloud data before optimization and the high-quality point cloud data after optimization for the same target in the training data pair; training the point cloud data cascade optimization network TP-PCONet based on the training data pair and the self-supervised learning method, the training data pair being the network input value and the training true value of the point cloud data cascade optimization network TP-PCONet before training, respectively, and the training being used to mine the internal inherent structure and mapping relationship of the training data pair to obtain the point cloud data cascade optimization network TP-PCONet.

[0013] Preferably, the incomplete coefficient n is 5, the sparse coefficient m is 4, and the noise coefficient k is 0.02; the edge length is 0.05; the point cloud data cascade optimization network TP-PCONet uses the upsampling network PUCRN as the baseline model, and the upsampling network PUCRN avoids distortion and discontinuity caused by excessive filling of the point cloud at one time by gradually increasing the point cloud density.

[0014] Preferably, the S2 includes:

[0015] S21, processing of the point cloud completion module based on snowflake diffusion, is used to predict the complete form of the actual production scene point cloud to be optimized to obtain a structurally complete actual production scene target point cloud, and provide complete data for the point cloud optimization, including: taking the point cloud data to be optimized as input to obtain the completed point cloud data, as shown in formula (2) and formula (3):

[0016]

[0017] in Represents the input point cloud data to be optimized, Represents the output completed point cloud data, N and K represent the number of point cloud points of the input point cloud data to be optimized and the number of point cloud points of the output completed point cloud data, respectively, K>N, It consists of three steps, where MFE(x i ,y i , z i ) represents the point cloud backbone feature extraction, SG(x i ,y i , zi ) represents missing part seed point generation, SPD(x i , y i , z i ) represents snowflake point diffusion;

[0018] S22, processing of the point cloud upsampling module based on step-by-step refinement, is used to receive the completed target point cloud after completion and perform an upsampling operation to obtain the final dense point cloud data, which provides the dense point cloud data for subsequent point cloud optimization. The point cloud upsampling module based on step-by-step refinement takes the completed point cloud as input and outputs the dense point cloud, as shown in formulas (7) and (8):

[0019]

[0020] wherein, represents the input complete point cloud data, represents the output dense point cloud data, and r represents the upsampling multiple; contains two steps, FE(x i , y i , z i ) represents feature expansion of point cloud data, and CR(x i , y i , z i ) represents coordinate reconstruction of point cloud data;

[0021] S23, processing of the point cloud refinement module based on edge heat value, is used to receive the dense target point cloud data and refine it to generate a smoother point cloud, which provides data for subsequent segmentation of each part of the target. The point cloud refinement module based on edge heat value takes the dense point cloud data as input and outputs the smoothed point cloud data, as shown in formulas (11) and (12):

[0022]

[0023] wherein, represents the input dense point cloud data, represents the output refined point cloud data, contains three steps, GC(x i , y i , z i ) represents gradient value calculation, HMG(x i , y i , z i ) represents heat map generation, and EPC(x i , y i , z i ) represents edge point contraction.

[0024] Preferably, the S21 includes:

[0025] Step 1: Extract backbone features of point cloud. This is used to extract backbone features of the point cloud P0 to be optimized. The encoder in the Transformer architecture is used to extract and encode the input point cloud to obtain the backbone feature vector of the point cloud, as shown in formula (4):

[0026] V m =T e (P0) (4)

[0027] in, represents the backbone feature vector, N represents the total number of points in the input point cloud, T e Represents the encoder of Transformer;

[0028] Step 2: Generate missing seed points, including: based on the extracted backbone feature vector V m , the Transformer decoder predicts the seed point set of the missing part, as shown in formula (5). The seed points in the seed point set of the missing part represent the approximate position and shape of the missing area;

[0029] P s =T d (V m ) (5)

[0030] in, n represents the number of points in the generated point cloud subset, T d Represents the decoder of Transformer;

[0031] Step 3: Snowflake point diffusion, including: using multiple SPD modules in the SnowFlakeNet network, with the seed point in the seed point set of the missing part as the center, gradually diffusing outward in a snowflake shape to generate new points until the entire point cloud data becomes complete, and obtaining complete point cloud data. The diffusion process of the snowflake-like gradual outward diffusion is shown in formula (6):

[0032] P s(i+1) =SPD(P si ) (6)

[0033] Among them, P si represents the input point cloud of the i+1th SPD module, P s(i+1) Represents the output point cloud of the i+1th SPD module. When i = a predetermined value, the point diffusion is completed to obtain the target point cloud of the actual production scene with a complete structure;

[0034] Among them, the point cloud completion module based on snowflake diffusion obtains the backbone points of the true point cloud through the farthest point sampling, and calculates the loss with the predicted point cloud through CD loss to update the point cloud completion module based on snowflake diffusion.

[0035] Preferably, the S22 includes:

[0036] Step 1: Feature expansion of point cloud data, including: after the completed point cloud data is input into the point cloud upsampling module based on the gradual refinement, feature expansion is performed through the multi-layer perceptron MLP to obtain the expanded point cloud feature vector, as shown in formula (9):

[0037] F up =MLP(P1) (9)

[0038] in, represents the extended feature vector;

[0039] Step 2: coordinate reconstruction of point cloud data, including: fitting the distribution and regularity of the existing point cloud data, and gradually generating new point cloud subsets from the point cloud feature vector after the feature expansion through the multi-layer perceptron MLP again to fill the sparse areas in the original point cloud, as shown in formula (10):

[0040] S=MLP(F up ) (10)

[0041] in, Represents a new point cloud subset generated based on the use of a step-by-step upsampling method, where α = r / 2; repeating the coordinate reconstruction operation of the point cloud data to obtain the final dense point cloud data.

[0042] Based on the gradually refined point cloud upsampling module, the CD loss between the ground truth point cloud and the predicted point cloud is directly calculated, and the generated dense point cloud is constrained. Through the above operations, a complete and dense target point cloud of the actual production scene can be obtained.

[0043] Preferably, the S23 includes:

[0044] Step 1: Gradient value calculation, including: using the KD tree to perform nearest neighbor query on the input dense point cloud data to obtain the neighbor point set of each point in the point cloud, as shown in formula (13):

[0045] PN=KDTree(P i ) (13)

[0046] Among them, P i represents the input point cloud, Represents the set of neighboring points of each point, and β represents the number of neighboring points of each point;

[0047] Step 2: Heat map generation, including: calculating the geometric center PC of the neighboring point set of each point as the center coordinate of the center point of the current point, as shown in formula (14):

[0048]

[0049] Calculate the distance between each point and its corresponding center point to obtain the position offset of each point in the point cloud, and calculate the difference between the offset value of the current point and the offset value of its neighboring points. The largest difference is the gradient value d of each point. i , as shown in formula (15):

[0050] d i =max||P i -PC i |-|PN i -PC i || (15)

[0051] In point cloud data, the larger the gradient value of the point, the closer the point is to the edge. Conversely, the smaller the gradient value, the closer the point is to the inside of the point cloud. The calculated point cloud gradient value is normalized to [0, 1] to obtain the edge thermal value of the point cloud. The closer the thermal value is to 1, the closer the point is to the edge, thus forming an edge thermal map.

[0052] Step 3: shrinking edge points, including: shrinking edge points based on the generated edge heat map, and obtaining a final optimized point cloud;

[0053] Among them, the point cloud refinement module based on edge thermal value calculates the edge thermal value of the predicted point cloud and the true point cloud, and constrains the point cloud through the L1 loss of the two, so as to obtain a target point cloud of the actual production scene with complete structure, dense and smooth.

[0054] A second aspect of the present invention is to provide a point cloud data optimization system based on self-supervised training, which is used to implement the point cloud data optimization method based on self-supervised training described in the first aspect, characterized in that the actual production scene target point cloud data to be processed is optimized to obtain optimized actual production scene target point cloud data, the data characteristics of the actual production scene target point cloud data to be processed are one or more of incompleteness, sparseness and edge fuzzy; the data characteristics of the optimized actual production scene target point cloud data are one or more of completeness, density and smoothness; the optimization is performed based on structural integrity, data density and data smoothness, and the system includes:

[0055] Self-supervised training module, used to train the point cloud data cascade optimization network TP-PCONet through data pair construction strategy and self-supervised method;

[0056] The point cloud data optimization module is used to sequentially constrain the backbone, surface and contour of the actual production scene target point cloud data to be processed through three steps: point cloud completion based on snowflake diffusion, point cloud upsampling based on gradual refinement, and point cloud refinement based on edge thermal value, to obtain the optimized actual production scene target point cloud data.

[0057] A third aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is configured to read the instructions and execute the method described in the first aspect.

[0058] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of instructions, and the plurality of instructions can be read by a processor to execute the method described in the first aspect.

[0059] The method, system, electronic device, and computer-readable storage medium provided by the present invention have the following beneficial technical effects:

[0060] A point cloud data optimization method is proposed. This method uses a target segmentation framework based on zero-labeled real-world production scenarios to segment the data, enabling the calculation of phenotypic parameters. Furthermore, the point cloud data collected from real-world production scenarios is optimized, and a point cloud cascade optimization network, TP-PCONet, based on self-supervised training, is designed. Experimental results show that the proposed TP-PCONet achieves a CD distance of 8.45 and an F1-score of 73.5%, respectively. This method has the potential to significantly improve the segmentation accuracy and efficiency of different target parts (e.g., plant stems and leaves). Similar excellent performance and application scenarios have also been demonstrated in other related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flow chart of the point cloud data optimization method based on self-supervised training according to the present invention;

[0062] Figure 2 This is a schematic diagram of the principle of the zero-labeled target segmentation method based on point cloud data optimization according to the present invention;

[0063] Figure 3 Schematic diagram of the point cloud optimization method based on self-supervised training according to the present invention;

[0064] Figure 4 This is an example diagram of the point cloud data characteristics of an actual production scenario described in the present invention;

[0065] Figure 5 This is a schematic diagram of the structure of the point cloud data cascade optimization network TP-PCONet according to the present invention;

[0066] Figure 6This is a schematic diagram of the point cloud edge heat map of the present invention;

[0067] Figure 7 This is a diagram of the architecture of the point cloud data optimization system based on self-supervised training according to the present invention;

[0068] Figure 8 This is a structural diagram of the electronic device described in the present invention. DETAILED DESCRIPTION

[0069] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0070] The method provided by the present invention can be implemented in the following terminal environment, which may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, which is loaded and executed by the processor to implement the method described in the following embodiments.

[0071] A processor can include one or more processing cores. It connects various components within the terminal using various interfaces and circuits. It executes instructions, programs, code sets, or instruction sets stored in memory, and accesses data stored in memory to perform various terminal functions and process data.

[0072] The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets, or instructions.

[0073] The display is used to show the user interface of each application.

[0074] In addition, those skilled in the art will appreciate that the structure of the terminal described above does not limit the terminal. The terminal may include more or fewer components, or a combination of certain components, or a different arrangement of components. For example, the terminal may also include a radio frequency circuit, an input unit, a sensor, an audio circuit, a power supply, and other components, which will not be described in detail here.

[0075] Example 1

[0076] like Figure 1 As shown, this embodiment provides a point cloud data optimization method based on self-supervised training, which is used to optimize the actual production scene target point cloud data to be processed to obtain optimized actual production scene target point cloud data, including:

[0077] S1, training the point cloud data cascade optimization network TP-PCONet through data pair construction strategy and self-supervision method;

[0078] In this embodiment, Figure 2 As shown in ①, the point cloud data optimization method based on self-supervised training is used to solve the problem that the point cloud collected in the actual production scene cannot effectively extract structural features due to quality factors such as incompleteness, sparseness and blurred edges.

[0079] In this embodiment, the optimization task for point cloud data of actual production scenarios requires that the model be able to learn the morphological characteristics of the plants, and have the ability to extract and characterize complete and detailed three-dimensional structural information of the plants. How to set plant morphological supervision information for the model is the primary consideration. Among various supervision methods, the self-supervised learning method uses the data itself as a supervision signal, deeply mines the information contained in the data through the network, and updates the knowledge representation in real time. It can accurately capture the subtle spatial structural characteristics of tomato plants and is suitable for the optimization task of point cloud data of actual production scenarios. Therefore, the present invention uses the self-supervised training method to achieve high optimization of point clouds of actual production scenarios, such as Figure 3 The method includes a data pair construction strategy based on the point cloud characteristics of actual production scenarios and a point cloud data cascade optimization network TP-PCONet, which are introduced below.

[0080] During the model training phase, high-quality indoor point clouds are first used as input. A data pair construction strategy based on the characteristics of actual production scene point clouds is used to generate simulated production scene point clouds for training data pairs. Secondly, the generated training data pairs are used for self-supervised training of the point cloud data cascade optimization network TP-PCONet. During the model application phase, the actual production scene point cloud to be processed is input into the trained point cloud data optimization model to generate the optimized actual production scene point cloud.

[0081] The data characteristics of the actual production scene target point cloud data to be processed are one or more of incomplete, sparse and edge fuzzy; the data characteristics of the actual production scene target point cloud data after optimization are one or more of complete, dense and smooth; the optimization is performed from three aspects: structural integrity, data density and data smoothness;

[0082] As a preferred embodiment, the S1 includes:

[0083] S11, using indoor high-quality point cloud data as input, obtaining simulated production scene point cloud data through a data pair construction strategy based on the point cloud characteristics of the actual production scene, and constructing a training data pair based on the simulated production scene point cloud data; the data pair construction strategy of the actual production scene point cloud characteristics includes: using indoor high-quality point cloud data as input, and presetting the defect coefficient n, sparse coefficient m and noise coefficient k during processing to respectively implement surface defect simulation, overall sparseness and noise addition, thereby simulating the occlusion, shaking of the target and the point cloud data changes caused during the data acquisition process, and processing the simulated production scene point cloud obtained Data is taken as output; the simulated production scene point cloud data is obtained by the data pair construction strategy based on the point cloud characteristics of the actual production scene, including: first, normalizing the indoor high-quality point cloud data; then randomly selecting n points in the indoor high-quality point cloud data as the center of the cube generated in the point cloud space, and setting the edge length; then removing the points belonging to the cube from the input indoor high-quality point cloud data to obtain the processed residual point cloud data; finally, randomly downsampling the processed residual point cloud data by m times, and adding Gaussian noise with k times the standard deviation to finally obtain the simulated production scene point cloud data.

[0084] In this embodiment, the point cloud data optimization network based on self-supervised training requires sample training pairs of low-quality pre-optimization and high-quality post-optimization point clouds for the same target, serving as network input and training ground truth, respectively, to exploit the inherent structure and mapping relationships within the data. However, due to the complex environment of the collected point cloud data in actual production scenarios, it is impossible to obtain high-quality point clouds as supervisory ground truth. Therefore, it is necessary to utilize a high-quality point cloud dataset available under indoor conditions as supervisory ground truth to provide sufficient supervisory information to guide the model for correct self-learning. Furthermore, in order to enable the network to optimize the tomato plant point cloud in actual production scenarios, the present invention provides a simulated production scenario dataset based on the characteristics of the point cloud data in actual production scenarios.

[0085] Examples of point cloud data characteristics in actual production scenarios include: Figure 4 As shown in the figure, there are mainly the following aspects: First, due to the mutual occlusion between leaves and between leaves and stems, the collected point cloud is incomplete [(A): incomplete characteristic]; second, due to the complex environment in which the plant is located and the irregular growth form, it is difficult to collect a uniform and dense point cloud [(B): sparse characteristic]; in addition, in the actual production scene collection process, external factors such as wind disturbance and shaking of the collection equipment will make it impossible for the plant to fix a posture, resulting in blurred edges of the collected point cloud [(C): blurred edge characteristic].

[0086] Therefore, based on the above analysis of the characteristics of actual production scene point cloud data, and the specific manifestation of edge blurring, where points at the boundary are not distributed in a linear shape but rather locally exhibit a Gaussian noise distribution, this method performs a series of processing on the input high-quality indoor point cloud, including surface defect simulation, global thinning, and noise addition, to simulate plant occlusion, swaying, and point cloud data changes that may occur during data acquisition. This results in simulated production scene point cloud data with the characteristics of actual production scene point clouds. The strategy for constructing the simulated production scene point cloud dataset is shown in Algorithm 1.

[0087]

[0088] As shown in Algorithm 1, during the data pair construction process, the present invention uses a high-quality indoor point cloud as input, presets the processing incompleteness coefficient n (set to 5 in this embodiment), the sparsity coefficient m (set to 4 in this embodiment), and the noise coefficient k (set to 0.02 in this embodiment), and uses the processed simulated production scene point cloud as the output. In this process, the input point cloud data is first normalized to facilitate subsequent processing. Then, n points are randomly selected from the input point cloud as the center of a cube generated in the point cloud space, and the edge length is set to 0.05 (an empirical value in this embodiment). Points within the cube are then removed from the input point cloud to obtain a processed incomplete point cloud. Finally, the point cloud is randomly downsampled m times and Gaussian noise with k times the standard deviation is added to obtain a simulated production scene point cloud. Through the above operations, a sample of plant point clouds from a simulated production scene is obtained, thereby constructing a pair of low-quality pre-optimization and high-quality post-optimization point cloud data that can be used for training the point cloud optimization network.

[0089] S12, training the point cloud data cascade optimization network TP-PCONet based on the training data pair and the self-supervised learning method, including: setting the upsampling network PUCRN as the baseline model; introducing a point cloud completion module based on snowflake diffusion and a point cloud refinement module based on edge thermal value on the basis of the point cloud upsampling module based on stepwise refinement in the baseline model to constitute the point cloud data cascade optimization network TP-PCONet, the input and output of the point cloud completion module based on snowflake diffusion, the point cloud upsampling module based on stepwise refinement and the point cloud refinement module based on edge thermal value correspond to the low-quality point cloud data before optimization and the high-quality point cloud data after optimization for the same target in the training data pair; training the point cloud data cascade optimization network TP-PCONet based on the training data pair and the self-supervised learning method, the training data pair being the network input value and the training true value of the point cloud data cascade optimization network TP-PCONet before training, respectively, and the training being used to mine the internal inherent structure and mapping relationship of the training data pair to obtain the point cloud data cascade optimization network TP-PCONet.

[0090] In this embodiment, the self-supervised learning method enables the network to actively learn the mapping relationship between data pairs, enabling it to optimize point clouds in real-world production scenarios. To further guide the network's learning direction from optimizing low-quality point clouds to high-quality point clouds, capture the fine structure of the target (in this embodiment, a tomato plant), and avoid optimization distortion, this embodiment designed the network structure as follows.

[0091] First, this embodiment uses the upsampling network PUCRN as the baseline model. This model effectively avoids distortion and discontinuity caused by overfilling the point cloud at once by gradually increasing the point cloud density, making it suitable for the complex and detailed structure of tomato plants. Second, to enable the optimization network to complete incomplete point clouds, this embodiment incorporates the snowflake diffusion-based point cloud completion module from SnowFlakeNet into the baseline model. This module utilizes the principle of snowflake diffusion to achieve smooth transitions and natural connections in point cloud data, avoiding abrupt completion distortion and addressing the distinct incomplete features of stems and leaves in the plant point cloud. To address the issue of blurred stem and leaf edges caused by the data itself and the upsampling and completion operations, this embodiment designs a point cloud upsampling module based on gradual refinement by constraining the point cloud surface. Furthermore, by constraining the point cloud contours, a point cloud refinement module based on edge thermal values ​​is designed. These three modules constitute the point cloud data cascade optimization network designed in this embodiment, TP-PCONet. Figure 5 The specific structural diagram of the point cloud data cascade optimization network TP-PCONet is shown. It is mainly composed of three cascaded sub-modules: (A-1), (A-2), and (A-3). Among them, (A-1): constrains the point cloud backbone to effectively fill in the missing parts and provide complete data for subsequent processing; (A-2): constrains the point cloud surface to optimize sparse data into dense data and improve data density; (A-3): constrains the contour of the completed and dense point cloud to obtain smooth point cloud data.

[0092] S2, through the three steps of point cloud completion based on snowflake diffusion, point cloud upsampling based on gradual refinement, and point cloud refinement based on edge thermal value, the actual production scene target point cloud data to be processed is sequentially constrained in terms of backbone, surface, and contour to obtain the optimized actual production scene target point cloud data.

[0093] In actual application, the optimized actual production scene point cloud data is input into Figure 2 The standard point cloud segmentation training model in ② is used to further segment the target (a tomato plant in this embodiment) into different parts (stem and leaf segmentation in this embodiment).

[0094] In the actual production scenario of tomato plant point cloud data optimization task, it is assumed that the input point cloud subset is represented as Among them, P 0i Represents the i-th point of the point cloud, N is the total number of points in the input point cloud, and the task is to obtain the optimized point cloud subset Where M represents the total number of points in the optimized point cloud, and M>N. The optimized point cloud subset P3 is obtained by formula (1) as follows:

[0095]

[0096] In formula (1), express Figure 5 The completion module processing process based on snowflake diffusion shown in (A-1) is as follows: express Figure 5 (A-2) shows the upsampling module processing process based on gradual refinement, express Figure 5 (A-3) shows the refinement module processing process based on edge thermal value.

[0097] In this embodiment, the processing flow after the point cloud data input data optimization network is processed by three modules: a point cloud completion module based on snowflake diffusion, a point cloud upsampling module based on gradual refinement, and a point cloud refinement module based on edge thermal value.

[0098] Therefore, as a preferred embodiment, the S2 includes:

[0099] S21, processing of point cloud completion module based on snowflake diffusion

[0100] Figure 5 Figure (A-1) shows the SnowFlakeNet-based point cloud completion module introduced in this embodiment. This module is used to predict the complete form of the actual production scene point cloud to be optimized, providing complete data for subsequent point cloud optimization. This module takes the point cloud to be optimized as input and obtains the completed point cloud, as shown in Formulas (2) and (3).

[0101]

[0102] in Represents the input point cloud data to be optimized, Represents the output completed point cloud data, N and K represent the number of point cloud points of the input point cloud data to be optimized and the number of point cloud points of the output completed point cloud data, respectively, K>N, It consists of three steps, where MFE(x i ,y i , z i ) represents the point cloud backbone feature extraction, SG(x i ,yi , z i ) indicates the missing seed point generation, SPD(x i ,y i , z i ) represents snowflake point diffusion. The specific steps are as follows.

[0103] Step 1: Extract backbone features of point cloud, which is used to extract backbone features of the point cloud P0 to be optimized. The encoder in the Transformer architecture is used to extract and encode the input point cloud to obtain the backbone feature vector of the point cloud, as shown in formula (4).

[0104] V m =T e (P0) (4)

[0105] in, represents the backbone feature vector, N represents the total number of points in the input point cloud, T e Represents the encoder of Transformer.

[0106] Step 2: Generate missing seed points, including: based on the extracted backbone feature vector V m , the decoder of Transformer predicts the seed point set of the missing part, as shown in formula (5). The seed points in the seed point set of the missing part represent the approximate position and shape of the missing area.

[0107] P s =T d (V m ) (5)

[0108] in, n represents the number of points in the generated point cloud subset, T d Represents the decoder of Transformer.

[0109] Step 3: Snowflake point diffusion, including: using multiple SPD modules in the SnowFlakeNet network, starting with the seed point in the missing part of the seed point set in step 2, gradually diffusing outward in a snowflake shape to generate new points until the entire point cloud data is complete, thus obtaining the complete point cloud data. The module diffusion process is shown in formula (6).

[0110] P s(i+1) =SPD(P si ) (6)

[0111] Among them, P si represents the input point cloud of the i+1th SPD module, P s(i+1) It represents the output point cloud of the i+1th SPD module. In this embodiment, when i=2, the point diffusion is completed.

[0112] The snowflake diffusion-based point cloud completion module samples the true point cloud at the farthest point to obtain the backbone points. This is then compared to the predicted point cloud using CDloss to calculate the loss, and the snowflake diffusion-based point cloud completion module is updated. Through these three steps, a structurally complete point cloud of a plant in an actual production scene can be obtained.

[0113] S22, processing of point cloud upsampling module based on gradual refinement

[0114] Figure 5 (A-2) shows the progressively refined point cloud upsampling module in the baseline model PUCRN. This module is primarily used to receive the completed plant point cloud and perform upsampling operations on it, providing dense data for subsequent point cloud optimization. This module takes the completed point cloud as input and outputs a dense point cloud, as shown in Equations (7) and (8).

[0115]

[0116] in, Represents the complete point cloud data of the input, represents the output dense point cloud data, r represents the upsampling multiple, which is set to 4 in this embodiment; It consists of two steps, FE(x i ,y i , z i ) represents the feature expansion of point cloud data, CR(x i ,y i , z i ) represents the coordinate reconstruction of point cloud data. The specific steps are as follows.

[0117] Step 1: Feature expansion of point cloud data, including: after the completed point cloud data is input into the point cloud upsampling module based on the progressive refinement, feature expansion is performed through the multi-layer perceptron MLP to obtain an expanded point cloud feature vector, as shown in formula (9).

[0118] F up =MLP(P1) (9)

[0119] in, represents the extended feature vector.

[0120] Step 2: coordinate reconstruction of point cloud data, including: fitting the distribution and regularity of the existing point cloud data, and gradually generating new point cloud subsets from the point cloud feature vector after the feature expansion through the multi-layer perceptron MLP again to fill the sparse areas in the original point cloud, as shown in formula (10).

[0121] S=MLP(F up) (10)

[0122] in, Represents the newly generated point cloud subset. Since this embodiment uses a step-by-step upsampling method, α=r / 2 here. Repeat step 2 to obtain the final dense point cloud data.

[0123] Based on the gradually refined point cloud upsampling module, the CD loss between the ground truth point cloud and the predicted point cloud is directly calculated, and the generated dense point cloud is constrained. Through the above operations, a complete and dense point cloud of plants in actual production scenes can be obtained.

[0124] S23, processing of point cloud refinement module based on edge thermal value

[0125] Figure 5 (A-3) shows the edge thermal value-based point cloud refinement module designed in this embodiment. It is primarily used to receive the densified plant point cloud data and refine it to generate a smoother point cloud, providing data for subsequent stem and leaf segmentation. The edge thermal value-based point cloud refinement module takes the dense point cloud data as input and outputs smoothed point cloud data, as shown in Equations (11) and (12).

[0126]

[0127] in, represents the input dense point cloud data, Represents the output refined point cloud data, It consists of three steps: GC(x i ,y i , z i ) represents the gradient value calculation, HMG(x i ,y i , z i ) represents the heat map generation, EPC(x i ,y i , z i ) indicates edge point contraction. The specific implementation steps are as follows.

[0128] Step 1: Gradient value calculation, including: using the KD tree to perform nearest neighbor query on the input dense point cloud data to obtain the neighbor point set of each point in the point cloud, as shown in formula (13).

[0129] PN=KDTree(P i ) (13)

[0130] Among them, P i represents the input point cloud, represents the set of neighboring points of each point, and β represents the number of neighboring points of each point, which is set to 350 in this embodiment.

[0131] Step 2: Heat map generation, including: calculating the geometric center PC of the neighboring point set of each point as the center coordinate of the center point of the current point, as shown in formula (14).

[0132]

[0133] Calculate the distance between each point and its corresponding center point to obtain the position offset of each point in the point cloud, and calculate the difference between the offset value of the current point and the offset value of its neighboring points. The largest difference is the gradient value d of each point. i , as shown in formula (15).

[0134] d i =max||P i -PC i |-|PN i -PC i || (15)

[0135] In the point cloud data, the larger the gradient value of the point, the closer the point is to the edge. Conversely, the smaller the gradient value, the closer the point is to the inside of the point cloud. The calculated point cloud gradient value is normalized to [0,1] to obtain the edge thermal value of the point cloud. The closer the thermal value is to 1, the closer the point is to the edge, thus forming a clear edge thermal map. Figure 6 shown, specifically, Figure 6 Middle (A) represents the original input point cloud; Figure 6 Middle (B) shows the corresponding heat map visualization generated by encoding.

[0136] Step 3: Edge Point Contraction: This involves contracting edge points based on the generated edge heatmap to reduce noise around the point cloud, sharpen its outlines, and ultimately produce the optimized point cloud. This module calculates the edge heatmap values ​​for the predicted and ground-truth point clouds and applies an L1 loss to constrain the point clouds. This process yields a structurally complete, dense, and smooth point cloud of plants from a real-world production scenario.

[0137] After three modules, the actual production scene point cloud data to be optimized is processed step by step by TP-PCONet, ultimately forming a point cloud with high-quality point cloud distribution characteristics. The optimized point cloud structure is complete, dense, and smooth, thus providing effective data for tomato plant stem and leaf segmentation.

[0138] Example 2

[0139] like Figure 7As shown, this embodiment provides a point cloud data optimization system based on self-supervised training, which is used to implement the method of the aforementioned embodiment 1, optimize the actual production scene target point cloud data to be processed to obtain optimized actual production scene target point cloud data, wherein the data characteristics of the actual production scene target point cloud data to be processed are one or more of incompleteness, sparseness and edge fuzzy; the data characteristics of the actual production scene target point cloud data after optimization are one or more of completeness, density and smoothness; the optimization is performed based on structural integrity, data density and data smoothness, and the system includes:

[0140] A self-supervised training module 101 is used to train the point cloud data cascade optimization network TP-PCONet through data pair construction strategy and self-supervised method;

[0141] The point cloud data optimization module 102 is used to sequentially constrain the backbone, surface and contour of the actual production scene target point cloud data to be processed through three steps: point cloud completion based on snowflake diffusion, point cloud upsampling based on gradual refinement, and point cloud refinement based on edge thermal value, to obtain the optimized actual production scene target point cloud data.

[0142] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the method described in the first embodiment.

[0143] like Figure 8 As shown, the present invention also provides an electronic device, including a processor 301 and a memory 302 connected to the processor 301, wherein the memory 302 stores multiple instructions, which can be loaded and executed by the processor to enable the processor to execute the method described in Example 1.

[0144] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the invention. Thus, the present invention is intended to include such changes and modifications as fall within the scope of the claims and their equivalents.

Claims

1. A point cloud data optimization method based on self-supervised training, characterized in that: Used to optimize the actual production scene target point cloud data to be processed to obtain optimized actual production scene target point cloud data, wherein the data characteristics of the actual production scene target point cloud data to be processed are one or more of incomplete, sparse and edge fuzzy; the data characteristics of the actual production scene target point cloud data after optimization are one or more of complete, dense and smooth; The optimization is performed based on structural integrity, data density and data smoothness, and the method comprises: S1, training the point cloud data cascade optimization network TP-PCONet through data pair construction strategy and self-supervision method; S2, through the three steps of point cloud completion based on snowflake diffusion, point cloud upsampling based on gradual refinement, and point cloud refinement based on edge thermal value, the actual production scene target point cloud data to be processed is sequentially constrained in terms of backbone, surface, and contour to obtain the optimized actual production scene target point cloud data; Said S1 comprises: S11, using indoor high-quality point cloud data as input, obtaining simulated production scene point cloud data through a data pair construction strategy based on the point cloud characteristics of the actual production scene, and constructing a training data pair based on the simulated production scene point cloud data; the data pair construction strategy of the actual production scene point cloud characteristics includes: using indoor high-quality point cloud data as input, and presetting the incompleteness coefficient during processing , sparse coefficient and noise figure The surface defect simulation, overall sparseness and noise addition are respectively implemented to simulate the occlusion and shaking of the target and the change of point cloud data caused by the data acquisition process, and the simulated production scene point cloud data obtained after processing is used as the output; the data pair construction strategy based on the actual production scene point cloud characteristics is used to obtain the simulated production scene point cloud data, which includes: first, normalizing the indoor high-quality point cloud data; then randomly selecting the indoor high-quality point cloud data. points as the center of the cube generated in the point cloud space, and set the edge length; then remove the points belonging to the cube from the input indoor high-quality point cloud data to obtain the processed residual point cloud data; finally, randomly downsample the processed residual point cloud data times and add Gaussian noise with a standard deviation of times, and finally obtaining the point cloud data of the simulated production scene; S12, training the point cloud data cascade optimization network TP-PCONet based on the training data pair and the self-supervised learning method, including: setting the upsampling network PUCRN as the baseline model; introducing a point cloud completion module based on snowflake diffusion and a point cloud refinement module based on edge thermal value on the basis of the point cloud upsampling module based on step-by-step refinement in the baseline model to constitute the point cloud data cascade optimization network TP-PCONet, the input and output of the point cloud completion module based on snowflake diffusion, the point cloud upsampling module based on step-by-step refinement and the point cloud refinement module based on edge thermal value correspond to the low-quality point cloud data before optimization and the high-quality point cloud data after optimization for the same target in the training data pair; training the point cloud data cascade optimization network TP-PCONet based on the training data pair and the self-supervised learning method, the training data pair being the network input value and the training true value of the point cloud data cascade optimization network TP-PCONet before training, respectively, and the training being used to mine the internal inherent structure and mapping relationship of the training data pair to obtain the point cloud data cascade optimization network TP-PCONet.

2. The point cloud data optimization method based on self-supervised training according to claim 1, characterized in that: The defect coefficient is 5, the sparse coefficient is 4, the noise figure is 0.02; the edge length is 0.05; the point cloud data cascade optimization network TP-PCONet uses the upsampling network PUCRN as the baseline model. The upsampling network PUCRN avoids the distortion and discontinuity problems caused by excessive filling of the point cloud at one time by gradually increasing the point cloud density.

3. The point cloud data optimization method based on self-supervised training according to claim 2, characterized in that: The S2 includes: S21, the processing of the point cloud completion module based on snowflake diffusion, is used to predict the complete form of the actual production scene point cloud to be optimized to obtain the actual production scene target point cloud with complete structure, and provide complete data for point cloud optimization, including: taking the point cloud data to be optimized as input, obtaining the completed point cloud data, as shown in formula (2) and formula (3): ； ； in Represents the input point cloud data to be optimized, Represents the output completed point cloud data, and They represent the number of points in the input point cloud data to be optimized and the number of points in the output completed point cloud data, respectively. K>N, It includes three steps, including represents the point cloud backbone feature extraction, Indicates that some seed points are missing. Indicates snowflake point spread; S22, based on the processing of the gradually refined point cloud upsampling module, is used to receive the completed complete target point cloud and perform an upsampling operation on it to obtain the final dense point cloud data, providing the dense point cloud data for subsequent point cloud optimization. The gradually refined point cloud upsampling module takes the completed point cloud as input and outputs the dense point cloud, as shown in formula (7) and formula (8): ； ； in, Represents the complete point cloud data of the input, Represents the output dense point cloud data, Indicates the upsampling multiple; It involves two steps, Represents the feature expansion of point cloud data, Represents coordinate reconstruction of point cloud data; S23, processing of the point cloud refinement module based on edge thermal value, is used to receive the dense target point cloud data and refine it to generate a smoother point cloud, providing data for the subsequent segmentation of each part of the target. The point cloud refinement module based on edge thermal value takes the dense point cloud data as input and outputs the smoothed point cloud data, as shown in formula (11) and formula (12): ； ； in, represents the input dense point cloud data, Represents the output refined point cloud data, It involves three steps: Indicates the gradient value calculation, Indicates heat map generation, Indicates that the edge points shrink.

4. The point cloud data optimization method based on self-supervised training according to claim 3, characterized in that: The S21 includes: Step 1: Extract the backbone features of the point cloud to be optimized Perform backbone feature extraction and use the encoder in the Transformer architecture to extract and encode the input point cloud to obtain the backbone feature vector of the point cloud, as shown in formula (4): ； in, represents the backbone eigenvector, Represents the total number of points in the input point cloud, Represents the encoder of Transformer; Step 2: Generate missing seed points, including: based on the extracted backbone feature vector , the Transformer decoder predicts the seed point set of the missing part, as shown in formula (5). The seed points in the seed point set of the missing part represent the approximate position and shape of the missing area; ； in, , Indicates the number of points in the generated point cloud subset, Represents the decoder of Transformer; Step 3: Snowflake point diffusion, including: using multiple SPD modules in the SnowFlakeNet network, with the seed point in the seed point set of the missing part as the center, gradually diffusing outward in a snowflake shape to generate new points until the entire point cloud data becomes complete, and obtaining complete point cloud data. The diffusion process of the snowflake-like gradual outward diffusion is shown in formula (6): ； in, Indicates the The input point cloud of the SPD module, Indicates the The output point cloud of the SPD module is When the point diffusion is completed, the actual production scene target point cloud with complete structure is obtained; Among them, the point cloud completion module based on snowflake diffusion obtains the backbone points of the true point cloud through the farthest point sampling, and calculates the loss with the predicted point cloud through CD loss to update the point cloud completion module based on snowflake diffusion.

5. The point cloud data optimization method based on self-supervised training according to claim 4, characterized in that: The S22 includes: Step 1: Feature expansion of point cloud data, including: inputting the completed point cloud data into the gradually refined point cloud upsampling module, and then passing it through a multi-layer perceptron. Perform feature expansion to obtain the expanded point cloud feature vector, as shown in formula (9): ； in, represents the extended feature vector; Step 2: Coordinate reconstruction of point cloud data, including: fitting the distribution and regularity of existing point cloud data, and then passing it through a multi-layer perceptron again. New point cloud subsets are gradually generated from the feature-expanded point cloud feature vectors to fill the sparse areas in the original point cloud, as shown in formula (10): ； in, Represents the generated new point cloud subset, based on the use of step-by-step upsampling, where Repeat the coordinate reconstruction operation of the point cloud data to obtain the final dense point cloud data; Based on the gradually refined point cloud upsampling module, the CD loss between the true point cloud and the predicted point cloud is directly calculated, and the generated dense point cloud is constrained. Through the above operations, a structurally complete and dense target point cloud of the actual production scene can be obtained.

6. The point cloud data optimization method based on self-supervised training according to claim 5, characterized in that: The S23 includes: Step 1: Gradient value calculation, including: using the KD tree to perform nearest neighbor query on the input dense point cloud data to obtain the neighbor point set of each point in the point cloud, as shown in formula (13): ； in represents the input point cloud, represents the set of neighboring points of each point, Indicates the number of neighboring points of each point; Step 2: Heat map generation, including: calculating the geometric center of each point's neighboring point set , as the center coordinate of the center point of the current point, as shown in formula (14): ； Calculate the distance between each point and its corresponding center point to obtain the position offset of each point in the point cloud, and calculate the difference between the offset value of the current point and the offset value of its neighboring points. The largest difference is the gradient value of each point. , as shown in formula (15): ； In point cloud data, the larger the gradient value of the point, the closer the point is to the edge. Conversely, the smaller the gradient value, the closer the point is to the inside of the point cloud. The calculated point cloud gradient value is normalized to [0, 1] to obtain the edge thermal value of the point cloud. The closer the thermal value is to 1, the closer the point is to the edge, thus forming an edge thermal map. Step 3: shrinking edge points, including: shrinking edge points based on the generated edge heat map, and obtaining a final optimized point cloud; Among them, the point cloud refinement module based on edge thermal value calculates the edge thermal value of the predicted point cloud and the true point cloud, and constrains the point cloud through the L1 loss of the two, so as to obtain a target point cloud of the actual production scene with complete structure, dense and smooth.

7. A point cloud data optimization system based on self-supervised training, used to implement the point cloud data optimization method based on self-supervised training according to any one of claims 1 to 6, characterized in that: Optimizing the actual production scene target point cloud data to be processed to obtain optimized actual production scene target point cloud data, wherein the data characteristics of the actual production scene target point cloud data to be processed are one or more of incomplete, sparse, and edge fuzzy; and the data characteristics of the optimized actual production scene target point cloud data are one or more of complete, dense, and smooth; The optimization is performed from the perspectives of structural integrity, data density, and data smoothness, and the system comprises: A self-supervised training module (101) is used to train the point cloud data cascade optimization network TP-PCONet through data pair construction strategy and self-supervised method; The point cloud data optimization module (102) is used to sequentially constrain the actual production scene target point cloud data to be processed in terms of backbone, surface and contour through three steps: point cloud completion based on snowflake diffusion, point cloud upsampling based on gradual refinement, and point cloud refinement based on edge thermal value, to obtain the optimized actual production scene target point cloud data.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is used to read the instructions and execute the point cloud data optimization method based on self-supervised training according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, which can be read by a processor and executed by the point cloud data optimization method based on self-supervised training according to any one of claims 1 to 6.

Images