A leaf point cloud reconstruction method and system based on projection constraint and mixed supervision

Abstract

This application belongs to the field of computer vision and smart agriculture technology, and discloses a method and system for leaf point cloud reconstruction based on projection constraints and hybrid supervision. The method includes: acquiring and preprocessing multimodal data of seedlings; constructing a point cloud autoencoder network with feature flow consistency to output predicted point clouds; constructing a hierarchical hybrid supervision loss function with integrity awareness, which includes: a dynamic 3D geometric loss that dynamically switches constraint methods based on sample integrity labels; a differentiable projection consistency loss that uses differentiable rendering technology to project the predicted point cloud and calculate it with a 2D segmentation mask to calibrate the edges; and a local topological regularization loss that ensures the smoothness of the completed region; and using a course learning strategy to dynamically adjust the weights of each loss term for network training. This method achieves sub-millimeter-level edge reconstruction using easily obtainable 2D image information, significantly improving the quality of incomplete leaf point cloud reconstruction and the accuracy of leaf area measurement.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and smart agriculture technology, and in particular to a method and system for reconstructing leaf point clouds based on projection constraints and hybrid supervision. Background Technology

[0002] In plant phenomics research and industrialized breeding, leaf area is a key indicator for assessing seedling growth vigor, water stress, and photosynthetic capacity. Currently, acquiring three-dimensional point cloud data of plants using RGB-D cameras is the main method for achieving automated measurement. However, existing three-dimensional point cloud reconstruction technologies face the following insurmountable challenges in practical agricultural applications:

[0003] 1. The Paradox of Truth Value Acquisition: Existing deep learning completion networks strictly rely on complete, high-precision 3D models as "truth values" during training. However, in real agricultural environments, due to leaf occlusion and sensor perspective limitations, the collected data naturally contains a large amount of incompleteness. Traditional fully supervised methods fail when truth values ​​are missing.

[0004] 2. Failure of Traditional Loss Functions: Existing technologies generally use "Bidirectional Chamfer Distance" as the loss function. When the training data itself is incomplete, bidirectional constraints force the model to generate point clouds that fit the incomplete input, causing the model to be unable to learn to fill in the missing parts. On the other hand, simple unidirectional constraints can lead to the problem of divergence in the generated point clouds.

[0005] 3. Edge shrinkage problem: Relying solely on geometric constraints in three-dimensional space, the model tends to generate overly smooth surfaces, causing the leaf edges to shrink inward. This results in the calculated leaf area being smaller than the true value, failing to meet the needs of high-precision agronomic research.

[0006] Therefore, there is an urgent need for a point cloud reconstruction method that can utilize readily available two-dimensional information to assist training in the absence of perfect three-dimensional ground truth, and can effectively correct edge accuracy. This would solve the problem that existing technologies struggle to train high-quality point cloud completion networks in the absence of complete three-dimensional ground truth data, as well as the technical challenges of low blade edge reconstruction accuracy and large area measurement errors caused by simple three-dimensional geometric constraints. Summary of the Invention

[0007] Based on this, the present invention provides a method and system for reconstructing leaf point clouds based on projection constraints and hybrid supervision. The method constructs an integrity-aware hierarchical hybrid supervision framework, uses easily obtainable two-dimensional instance segmentation masks as strong prior information, constructs cross-modal projection consistency constraints through differentiable rendering technology, and dynamically adjusts the calculation logic of the loss function according to the integrity status of the samples, thereby achieving high-quality point cloud completion and high-precision edge reconstruction under the condition of lacking complete three-dimensional ground truth.

[0008] In a first aspect, the present invention provides a method for reconstructing blade point clouds based on projection constraints and hybrid supervision, comprising the following steps:

[0009] Obtain the depth map, RGB image, leaf two-dimensional semantic segmentation mask generated based on the RGB image, and integrity indicator variable indicating the integrity of the point cloud corresponding to the depth map of the seedling to be trained, and convert the depth map into the input point cloud;

[0010] The input point cloud, the two-dimensional semantic segmentation mask of the leaf, and the integrity indicator variable are input together into the point cloud autoencoder network to be trained, and the predicted point cloud is output.

[0011] Calculate the total objective function loss value of the point cloud autoencoder network, the total objective function loss value including:

[0012] The dynamic 3D geometric loss is calculated between the input point cloud and the predicted point cloud based on the integrity indicator variable.

[0013] The predicted point cloud is projected onto a two-dimensional plane using differentiable rendering to generate a prediction mask, and the differentiable projection consistency loss between the prediction mask and the two-dimensional semantic segmentation mask of the blade is calculated.

[0014] And, the local topological regularization loss calculated to constrain the geometry of the predicted point cloud in the missing region;

[0015] The total objective function loss value is obtained and used to update the network parameters of the point cloud autoencoder network through gradient backpropagation.

[0016] As an optional implementation of the first aspect of this application, the step of acquiring a depth map, an RGB image, a leaf two-dimensional semantic segmentation mask generated based on the RGB image, and an integrity indicator variable indicating the integrity of the point cloud corresponding to the depth map, and converting the depth map into an input point cloud, includes: acquiring the depth map and RGB image of the seedling to be trained using an RGB-D acquisition device; locking the region of interest of the leaf in the RGB image using an object detection model; performing a vertical flip mapping on the region of interest to align the two-dimensional image coordinate system with the texture coordinate system of the three-dimensional point cloud, and extracting the leaf two-dimensional semantic segmentation mask, and providing an integrity indicator variable indicating the integrity of the point cloud corresponding to the depth map; mapping the original point cloud to a preset normalized coordinate system through decentralization and unit sphere normalization, the mapping being implemented through a preset mapping matrix to generate the input point cloud, the input point cloud being uniformly resampled to a preset number of points.

[0017] As an optional implementation of the first aspect of this application, the method further includes that the point cloud autoencoder network adopts an encoder-decoder architecture, specifically constructed as follows: the encoder is based on a hierarchical set abstraction architecture, and when extracting local geometric features layer by layer, it converts the first... Local features extracted from layers The input point cloud corresponding to this layer absolute coordinates By concatenating the data along the channel dimension, we obtain enhanced features that include spatial pose information. And finally, the enhanced features are aggregated into a global feature vector. The decoder is based on a folded network architecture, and first initializes a network containing... A standard two-dimensional regular grid with 1 grid point Then the global feature vector Copy N times and combine with the standard two-dimensional regular mesh The coordinates are pieced together point by point to form a combined input. The combined input After spatial deformation operations simulated by a multilayer perceptron, the final output is the predicted point cloud. .

[0018] As an optional implementation of the first aspect of this application, the formula for calculating the dynamic three-dimensional geometric loss is: When the integrity indicator variable When it is 1, it indicates that the input point cloud For a complete sample, the above calculation formula is used for calculation; when the completeness indicator variable When the value is 0, it indicates that the input point cloud For incomplete samples, the second term in the above calculation formula is set to zero before calculation; where, For the predicted point cloud, To predict the number of points contained in a point cloud, For the input point cloud, To input the number of points contained in the point cloud, This represents the square of the Euclidean distance.

[0019] As an optional implementation of the first aspect of this application, the calculation method for the differentiable projection consistency loss is as follows: constructing a virtual camera in three-dimensional space with intrinsic parameters consistent with the real acquisition device; and using a differentiable rasterization function to convert the predicted point cloud... Projected onto a two-dimensional image plane, and smoothed using a mean smoothing kernel of a preset size. Convolutional smoothing is applied to the rasterization result to generate a two-dimensional probability mask. ; Calculate the two-dimensional probability mask With the blade's two-dimensional semantic segmentation mask The crossover-union ratio loss, which serves as the differentiable projective consistency loss, is calculated using the following formula: In the formula, and These represent the width and height of the image, respectively. and Representing pixels The predicted probability value and the actual probability value at the location; To prevent tiny constants with a denominator of zero.

[0020] As an optional implementation of the first aspect of this application, the local topological regularization loss is composed of a Laplace smoothing term. Inter-point repulsion term The weighted calculation method is used to suppress the divergence and noise of the predicted point cloud in the missing regions when the input point cloud is an incomplete sample. The calculation formula is as follows: In the formula, and These are the weighting coefficients for the smoothing term and the compaction term, respectively; the Laplace smoothing... The smoothness of the surface is constrained by the center of the k nearest neighbors, and its calculation formula is as follows: In the formula, Represents the first point in the point cloud One point, Point of Nearest neighbor set This represents the number of points within the neighborhood.

[0021] As an optional implementation of the first aspect of this application, the total objective function loss value is calculated as follows: [The dynamic three-dimensional geometric loss is then applied]. The differentiable projective consistency loss and the local topological regularization loss A weighted summation is performed, where the weights of the dynamic 3D geometric loss and the differentiable projective consistency loss vary with the training epochs. The dynamic weights that change are calculated using the following formula: In the formula, , , According to the training rounds The dynamic weights of the varying 3D geometric loss, differentiable projection consistency loss, and local topological regularization loss. represents the switching coefficient of the regularization term.

[0022] As an optional implementation of the first aspect of this application, the dynamic weight follows a piecewise step function, specifically by setting a first-stage threshold. Second stage threshold , where 0 < < By using a pre-defined piecewise step function, a learning strategy is adopted that focuses on 3D skeleton construction in the early stage, collaborative optimization in the middle stage, and refinement of edge details in the later stage. The pre-defined piecewise step function is: when hour, , ;when hour, ;when hour, .

[0023] Secondly, embodiments of this application provide a blade point cloud reconstruction system based on projection constraints and hybrid supervision, comprising:

[0024] The data acquisition module is used to acquire the depth map, RGB image, leaf two-dimensional semantic segmentation mask generated based on the RGB image, and integrity indicator variable indicating the integrity of the point cloud corresponding to the depth map of the seedling to be trained;

[0025] The point cloud processing module is used to convert the depth map into an input point cloud, and input it together with the leaf two-dimensional semantic segmentation mask and the integrity indicator variable into the point cloud autoencoder network to be trained, and output the predicted point cloud.

[0026] The loss calculation module is used to calculate the total objective function loss value of the point cloud autoencoder network. The loss calculation module includes:

[0027] A dynamic three-dimensional geometric loss calculation unit is used to calculate the dynamic three-dimensional geometric loss between the input point cloud and the predicted point cloud based on the integrity indicator variable.

[0028] The differentiable projection consistency loss calculation unit is used to project the predicted point cloud onto a two-dimensional plane through differentiable rendering to generate a prediction mask, and to calculate the differentiable projection consistency loss between the prediction mask and the two-dimensional semantic segmentation mask of the blade.

[0029] A local topology regularization loss calculation unit is used to calculate the local topology regularization loss to constrain the geometry of the predicted point cloud in the missing region;

[0030] The network parameter update module is used to finally obtain the total objective function loss value and to update the network parameters of the point cloud autoencoder network through gradient backpropagation.

[0031] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the method described in the first aspect.

[0032] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0033] Compared with existing technologies, this invention obtains the seedling's depth map, RGB image, leaf 2D semantic segmentation mask, and integrity indicator variable. Utilizing the semantic information of the 2D image as a strong prior, it overcomes the dependence of traditional methods on complete 3D ground truth, reducing the labor cost of constructing high-precision 3D datasets. The point cloud autoencoder network adopts an encoder-decoder architecture. The encoder stitches local features with the absolute coordinates of the point cloud, while the decoder fuses global feature vectors with a standard 2D regular grid. This allows the output point cloud to be generated under the dual constraints of global shape prior and local topological structure, enhancing the robustness of the geometric structure and enabling reasonable completion of occluded leaf areas. The dynamic 3D geometric loss dynamically switches constraint methods based on the integrity indicator variable, avoiding the geometric collapse problem caused by traditional bidirectional chamfer distance when processing incomplete ground truth. The differentiable projection consistency loss uses differentiable rendering technology to project the predicted point cloud and calculate it with the 2D segmentation mask, introducing high-frequency edge information from the RGB image to solve the leaf edge shrinkage problem, achieving sub-millimeter-level edge reconstruction and improving the accuracy of leaf area measurement. Attached Figure Description

[0034] Figure 1 This is a flowchart of a blade point cloud reconstruction method based on projection constraints and hybrid supervision according to an embodiment of the present invention;

[0035] Figure 2 This is a flowchart of the point cloud autoencoder network (PCAE) based on the encoder-decoder architecture in an embodiment of the present invention;

[0036] Figure 3 This is a structural diagram of a blade point cloud reconstruction system based on projection constraints and hybrid supervision, according to an embodiment of the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0039] Example 1

[0040] Please see Figure 1 This is a flowchart illustrating a blade point cloud reconstruction method based on projection constraints and hybrid supervision, provided by an embodiment of the present invention. The method may include the following steps:

[0041] S1: Obtain the depth map, RGB image, leaf two-dimensional semantic segmentation mask generated based on the RGB image, and integrity indicator variable indicating the integrity of the point cloud corresponding to the depth map of the seedling to be trained, and convert the depth map into the input point cloud.

[0042] Depth maps and RGB images of the seedlings to be trained were acquired using an RGB-D acquisition device. To meet the geometric consistency requirements of the input data for the subsequent point cloud completion network, this invention designed a targeted cross-modal alignment. First, the region of interest (ROI) of the leaf was located in the RGB image using an object detection model. Crucially, to correct the inherent difference between the 2D image coordinate system and the 3D point cloud texture coordinate system, and to align the 2D image coordinate system with the 3D point cloud texture coordinate system, a specific vertical flip mapping was performed. This not only yielded a high-confidence 2D semantic segmentation mask for the leaf. , This represents the original binary mask extracted directly from the RGB image without coordinate system correction, which further ensures the strict correspondence between the two-dimensional semantic information and the three-dimensional geometric space in subsequent projection constraints.

[0043] At the same time, the original depth map is converted into the original point cloud. Furthermore, through decentralization and unit sphere normalization, the original point cloud is mapped to a preset normalized coordinate system. This mapping is achieved through a pre-defined mapping matrix. To achieve this, all samples are uniformly resampled to a preset number of points (e.g., 2048 points) to generate an input point cloud that serves as the network input, providing a standardized density distribution for subsequent decoder grid (FoldingNet) folding. Furthermore, an integrity indicator variable is provided for each sample to indicate whether its corresponding input point cloud is a complete sample.

[0044] S2: Input the input point cloud, the two-dimensional semantic segmentation mask of the leaf, and the integrity indicator variable together into the point cloud autoencoder network to be trained, and output the predicted point cloud.

[0045] This invention constructs a point cloud autoencoder network (PCAE) based on an encoder-decoder architecture, such as... Figure 2 As shown. To address the problem of sparse and severely occluded leaf point clouds in agricultural environments, the network was improved by using an explicit feature transfer mechanism to tightly couple local geometric details with global morphological priors.

[0046] The encoder is based on the PointNet++ architecture of hierarchical set abstraction. Its improvement lies in that, when extracting local geometric features layer by layer, it incorporates the first... Local features extracted from layers The input point cloud corresponding to this layer absolute coordinates Concatenate along the channel dimension to obtain enhanced features that include spatial pose information. Through this step, the network generates a global feature vector. It not only includes shape semantics, but also explicitly encodes the blade's attitude information in three-dimensional space.

[0047] The decoder is based on the FoldingNet architecture. To achieve accurate mapping from a 2D manifold to a 3D surface, the decoder first initializes a standard 2D regular mesh containing N grid points. The key improvement lies in the fact that, during the folding operation, instead of simply inputting a mesh, the global feature vector output by the encoder is used. After copying N times, with the grid The coordinates are pieced together point by point to form a combined input. The combined input undergoes spatial deformation operations simulated by a multilayer perceptron (MLP), ultimately outputting a reconstructed predicted point cloud. This design ensures that every point in the output point cloud is based on the global shape prior (from...). ) and local topology (from It is generated under the dual constraints of ).

[0048] S3: Calculate the total objective function loss of the point cloud autoencoder network, the total objective function loss including: dynamic 3D geometric loss calculated between the input point cloud and the predicted point cloud according to the integrity indicator variable; differentiable projection consistency loss between the predicted point cloud and the leaf 2D semantic segmentation mask generated by projecting the predicted point cloud onto a 2D plane through differentiable rendering; and local topological regularization loss calculated to constrain the geometry of the predicted point cloud in the missing region.

[0049] To address the limitation of existing technologies where a single loss function cannot simultaneously handle both complete and incomplete leaf data, this invention constructs a hybrid supervised loss function where each component works in synergy and complements the others. This function is not a simple summation of individual losses, but rather an organic whole dynamically adjusted through an integrity indicator variable and a training phase time variable. The overall objective function loss value includes dynamic 3D geometric loss, differentiable projection consistency loss, and local topological regularization loss.

[0050] 1. Dynamic 3D geometric loss

[0051] Traditional bidirectional chamfering distance methods often cause the generated point cloud to shrink towards the incomplete parts when processing incomplete ground truth values, leading to geometric collapse. To address this, this invention constructs a conditionally triggered asymmetric geometric loss mechanism. :

[0052] ;

[0053] in, For the predicted point cloud, To predict the number of points contained in a point cloud, For the input point cloud, To input the number of points contained in the point cloud, This represents the square of the Euclidean distance.

[0054] Define integrity indicator variables When the sample is complete =1, incomplete =0.

[0055] when When =1, it indicates that the input point cloud For complete samples, a bidirectional chamfer distance constraint is used, that is, the above calculation formula is used to ensure the accuracy and coverage of the generated shape.

[0056] when When =0, it indicates that the input point cloud For incomplete samples, a one-way chamfer distance constraint is used, which involves setting the second term in the above calculation formula to zero before calculation. This constraint only ensures that points existing in the input point cloud are covered by the predicted point cloud, allowing the model to freely generate in the missing regions of the input point cloud. Automatically cutting off the gradient backpropagation of the backward fitting error provides the necessary mathematical space for the model to perform geometric inference in the missing regions, and is also a prerequisite for the subsequent regularization loss to function effectively.

[0057] 2. Loss of consistency in differentiable projection

[0058] Relying solely on 3D geometric loss for constraint often results in overly smoothed or contracted blade edges in the reconstructed model. To correct this deficiency, this invention introduces a cross-modal projection consistency loss. First, a virtual camera with intrinsic parameters identical to the real acquisition device is constructed in 3D space. Then, the predicted point cloud is processed using a differentiable rasterization function. Projected onto a two-dimensional image plane, and smoothed using a mean smoothing kernel of a preset size. Convolutional smoothing is applied to the rasterization result to generate a two-dimensional probabilistic mask containing transparency information. :

[0059] ;

[0060] In the formula, Represents a non-differentiable rasterization function; For size The mean smoothing kernel is used to achieve convolution smoothing; It is the set of pixels in the image plane; Represents pixel coordinates.

[0061] Based on this, a projection consistency loss is constructed. Through calculation Compared with the two-dimensional semantic segmentation mask of the real leaf extracted in step S1 The Intersection over Union (IoU) loss between the two is calculated using the following formula:

[0062] ;

[0063] in, and These represent the width and height of the image, respectively. and Representing pixels The predicted probability value and the actual probability value at the location; To prevent tiny constants with a denominator of zero.

[0064] This loss utilizes a smoothing kernel. A differentiable approximation of rasterization was achieved, establishing a gradient propagation path from a 2D image mask to 3D point cloud coordinates. Furthermore, the denominator is in sum-of-squares form. Instead of traditional This form has stronger numerical stability during backpropagation of the loss gradient, especially when dealing with gradients close to zero, and can more effectively penalize small differences at the edges.

[0065] 3. Local topological regularization loss

[0066] Because when dealing with incomplete samples, Providing only a one-way coverage constraint makes the generated point cloud prone to divergence in free space or the generation of outlier noise. Therefore, a Laplacian smoothing term is introduced. Inter-point repulsion term Local topological regularization loss :

[0067] ;

[0068] in, and These are the weighting coefficients for the smoothing and compaction terms, respectively.

[0069] Laplace smoothing term The surface smoothness is constrained by the nearest neighbor center, and the calculation formula is as follows:

[0070] ;

[0071] in, Represents the first point in the point cloud One point; Point of Nearest neighbor set ( -nearestneighbors); This represents the number of points within the neighborhood.

[0072] and The unidirectional constraints form a counterbalance: Encourage the point cloud to expand outwards to cover the target, while This restricts the disordered divergence of the point cloud. Together, these two factors ensure that the completed region effectively covers the target outline while maintaining a smooth topological structure consistent with biological characteristics.

[0073] S4: Obtain the total objective function loss value, which is used to update the network parameters of the point cloud autoencoder network through gradient backpropagation.

[0074] This invention constructs a time-varying weighted overall objective function. By employing a dynamic weight adjustment mechanism that evolves with the training process, the cognitive pattern of "holistic first, then local" is simulated. The overall objective function is defined as:

[0075] ;

[0076] in, , , According to the training rounds The dynamic weights of the varying 3D geometric loss, differentiable projection consistency loss, and local topological regularization loss. represents the switching coefficient of the regularization term.

[0077] The dynamic weights follow a piecewise step function with each training epoch t. A threshold is set for the first stage. Second stage threshold (where 0 < < ):

[0078] Skeleton construction phase ( (e.g., Epoch 0-20) , The main spatial structure of the blade is quickly learned using only three-dimensional geometric loss.

[0079] Multimodal collaboration stage ( (e.g., Epochs 20-60) By introducing two-dimensional projection constraints and regularization, we began to repair blade edges and optimize surface quality.

[0080] Fine optimization stage ( (e.g., after Epoch 60) Significantly enhance the weighting of 2D projections and utilize high-precision binary masks to forcibly stretch the edges of point clouds, achieving sub-millimeter-level shape alignment.

[0081] This dynamic control strategy ensures that model training follows a learning path from 3D structure to 2D edge correction, guaranteeing convergence stability under multimodal constraints.

[0082] After training, the point cloud of the seedling to be tested is input into the model, and the completed point cloud is output. The Delaunay triangulation algorithm is used to reconstruct the point cloud into a mesh, and the sum of the areas of all triangular faces is the final leaf area.

[0083] Example 2

[0084] Please see Figure 3 The diagram shown is a schematic representation of a blade point cloud reconstruction system based on projection constraints and hybrid supervision, as proposed in the second embodiment of this application. The system includes the following key modules:

[0085] The data acquisition module 100 is used to acquire the depth map, RGB image, leaf two-dimensional semantic segmentation mask generated based on the RGB image, and integrity indicator variable indicating the integrity of the point cloud corresponding to the depth map of the seedling to be trained;

[0086] The point cloud processing module 200 is used to convert the depth map into an input point cloud, and input it together with the leaf two-dimensional semantic segmentation mask and the integrity indicator variable into the point cloud autoencoder network to be trained, and output the predicted point cloud.

[0087] Loss calculation module 300 is used to calculate the total objective function loss value of the point cloud autoencoder network. The loss calculation module includes:

[0088] The dynamic three-dimensional geometric loss calculation unit 310 is used to calculate the dynamic three-dimensional geometric loss between the input point cloud and the predicted point cloud based on the integrity indicator variable.

[0089] Differentiable projection consistency loss calculation unit 320 is used to project the predicted point cloud onto a two-dimensional plane through differentiable rendering to generate a prediction mask, and calculate the differentiable projection consistency loss between the prediction mask and the two-dimensional semantic segmentation mask of the blade.

[0090] Local topology regularization loss calculation unit 330 is used to calculate local topology regularization loss to constrain the geometry of the predicted point cloud in the missing region;

[0091] The network parameter update module 400 is used to finally obtain the total objective function loss value and to update the network parameters of the point cloud autoencoder network through gradient backpropagation.

[0092] The leaf point cloud reconstruction system based on projection constraints and hybrid supervision in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), etc. This application embodiment does not impose specific limitations.

[0093] The blade point cloud reconstruction system based on projection constraints and hybrid supervision in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0094] The blade point cloud reconstruction system based on projection constraints and hybrid supervision provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method embodiment of the blade point cloud reconstruction method based on projection constraints and hybrid supervision will not be described again here to avoid repetition.

[0095] Optionally, embodiments of this application also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described embodiment of a leaf point cloud reconstruction method based on projection constraints and hybrid supervision, and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0096] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described embodiment of a blade point cloud reconstruction method based on projection constraints and hybrid supervision, and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0097] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0098] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0099] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0100] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for reconstructing blade point clouds based on projection constraints and hybrid supervision, characterized in that, Includes the following steps: Obtain the depth map, RGB image, leaf two-dimensional semantic segmentation mask generated based on the RGB image, and integrity indicator variable indicating the integrity of the point cloud corresponding to the depth map of the seedling to be trained, and convert the depth map into the input point cloud; The input point cloud, the two-dimensional semantic segmentation mask of the leaf, and the integrity indicator variable are input together into the point cloud autoencoder network to be trained, and the predicted point cloud is output. Calculate the total objective function loss value of the point cloud autoencoder network, the total objective function loss value including: The dynamic 3D geometric loss is calculated between the input point cloud and the predicted point cloud based on the integrity indicator variable. The formula for calculating the dynamic 3D geometric loss is as follows: When the integrity indicator variable When it is 1, it indicates that the input point cloud For a complete sample, the above calculation formula is used; when the completeness indicator variable When the value is 0, it indicates that the input point cloud For incomplete samples, the second term in the above calculation formula is set to zero before calculation; where, For the predicted point cloud, To predict the number of points contained in a point cloud, For the input point cloud, To input the number of points contained in the point cloud, Represents the square of the Euclidean distance; The predicted point cloud is projected onto a two-dimensional plane using differentiable rendering to generate a prediction mask. The differentiable projection consistency loss between the prediction mask and the two-dimensional semantic segmentation mask of the blade is calculated. The differentiable projection consistency loss is calculated as follows: a virtual camera with intrinsic parameters consistent with the actual acquisition device is constructed in three-dimensional space; the predicted point cloud is then processed using a differentiable rasterization function. Projected onto a two-dimensional image plane, and smoothed using a mean smoothing kernel of a preset size. Convolutional smoothing is applied to the rasterization result to generate a two-dimensional probability mask. ; Calculate the two-dimensional probability mask With the blade's two-dimensional semantic segmentation mask The crossover-union ratio loss, which serves as the differentiable projective consistency loss, is calculated using the following formula: In the formula, and These represent the width and height of the image, respectively. and Representing pixels The predicted probability value and the actual probability value at the location; To prevent tiny constants with a denominator of zero; And, a local topological regularization loss calculated to constrain the geometry of the predicted point cloud in the missing regions, the local topological regularization loss being composed of a Laplacian smoothing term. Inter-point repulsion term The weighted calculation method is used to suppress the divergence and noise of the predicted point cloud in the missing regions when the input point cloud is an incomplete sample. The calculation formula is as follows: In the formula, and These are the weighting coefficients for the smoothing term and the compaction term, respectively; the Laplace smoothing... The smoothness of the surface is constrained by the center of the k nearest neighbors, and its calculation formula is as follows: In the formula, Represents the first point in the point cloud One point, Point of Nearest neighbor set For this point of The number of nearest neighbor sets; The total objective function loss value is obtained and used to update the network parameters of the point cloud autoencoder network through gradient backpropagation.

2. The method according to claim 1, characterized in that, The steps of acquiring a depth map, an RGB image, a leaf 2D semantic segmentation mask generated from the RGB image, and an integrity indicator variable indicating the integrity of the point cloud corresponding to the depth map, and converting the depth map into an input point cloud, include: The depth map and RGB image of the seedling to be trained are acquired using an RGB-D acquisition device; The region of interest of the leaf is located in the RGB image using an object detection model; For the region of interest, a vertical flip mapping is performed to align the two-dimensional image coordinate system with the texture coordinate system of the three-dimensional point cloud, and the two-dimensional semantic segmentation mask of the leaf is extracted, as well as an integrity indicator variable that indicates the integrity of the point cloud corresponding to the depth map is provided. The original point cloud is decentralized and normalized to a unit sphere, and then mapped to a preset normalized coordinate system. This mapping is achieved through a preset mapping matrix to generate the input point cloud. The input point cloud is then uniformly resampled to a preset number of points.

3. The method according to claim 1, characterized in that, The method further includes the point cloud autoencoder network adopting an encoder-decoder architecture, the specific construction method of which is as follows: The encoder is based on a hierarchical set abstraction architecture. When extracting local geometric features layer by layer, it will... Local features extracted from layers The input point cloud corresponding to this layer absolute coordinates By concatenating the data along the channel dimension, we obtain enhanced features that include spatial pose information. And finally, the enhanced features are aggregated into a global feature vector. ; The decoder is based on a folded network architecture and first initializes a network containing... A standard two-dimensional regular grid with 1 grid point Then the global feature vector Copy N times and combine with the standard two-dimensional regular mesh The coordinates are pieced together point by point to form a combined input. The combined input After spatial deformation operations simulated by a multilayer perceptron, the final output is the predicted point cloud. .

4. The method according to claim 1, characterized in that, The method for calculating the total objective function loss value is as follows: For the dynamic three-dimensional geometric loss The differentiable projective consistency loss and the local topological regularization loss A weighted summation is performed, where the weights of the dynamic 3D geometric loss, the differentiable projective consistency loss, and the local topological regularization loss vary with the training epochs. The dynamic weights that change are calculated using the following formula: ； In the formula, , , According to the training rounds The dynamic weights of the varying 3D geometric loss, differentiable projection consistency loss, and local topological regularization loss. represents the switching coefficient of the regularization term.

5. The method according to claim 4, characterized in that, The dynamic weights follow a piecewise step function, specifically: Set the first stage threshold Second stage threshold , where 0 < < ; By employing a pre-defined piecewise step function, a learning strategy is adopted that focuses on 3D skeleton construction in the early stage of training, collaborative optimization in the middle stage, and refinement of edge details in the later stage. The pre-defined piecewise step function is as follows: when hour, , ; when hour, ; when hour, .

6. A blade point cloud reconstruction system based on projection constraints and hybrid supervision, characterized in that, include: The data acquisition module is used to acquire the depth map, RGB image, leaf two-dimensional semantic segmentation mask generated based on the RGB image, and integrity indicator variable indicating the integrity of the point cloud corresponding to the depth map of the seedling to be trained, and to convert the depth map into the input point cloud; The point cloud processing module is used to input the input point cloud, the two-dimensional semantic segmentation mask of the leaf, and the integrity indicator variable into the point cloud autoencoder network to be trained, and output the predicted point cloud. The loss calculation module is used to calculate the total objective function loss value of the point cloud autoencoder network. The loss calculation module includes: A dynamic 3D geometric loss calculation unit is used to calculate the dynamic 3D geometric loss between the input point cloud and the predicted point cloud based on the integrity indicator variable. The calculation formula for the dynamic 3D geometric loss is as follows: When the integrity indicator variable When it is 1, it indicates that the input point cloud For a complete sample, the above calculation formula is used; when the completeness indicator variable When the value is 0, it indicates that the input point cloud For incomplete samples, the second term in the above calculation formula is set to zero before calculation; where, For the predicted point cloud, To predict the number of points contained in a point cloud, For the input point cloud, To input the number of points contained in the point cloud, Represents the square of the Euclidean distance; The differentiable projection consistency loss calculation unit is used to project the predicted point cloud onto a two-dimensional plane through differentiable rendering to generate a prediction mask, and to calculate the differentiable projection consistency loss between the prediction mask and the two-dimensional semantic segmentation mask of the blade. The calculation method of the differentiable projection consistency loss is as follows: a virtual camera with the same intrinsic parameters as the real acquisition device is constructed in three-dimensional space; the predicted point cloud is then processed by a differentiable rasterization function. Projected onto a two-dimensional image plane, and smoothed using a mean smoothing kernel of a preset size. Convolutional smoothing is applied to the rasterization result to generate a two-dimensional probability mask. ; Calculate the two-dimensional probability mask With the blade's two-dimensional semantic segmentation mask The crossover-union ratio loss, which serves as the differentiable projective consistency loss, is calculated using the following formula: In the formula, and These represent the width and height of the image, respectively. and Representing pixels The predicted probability value and the actual probability value at the location; To prevent tiny constants with a denominator of zero; A local topology regularization loss calculation unit is used to calculate the local topology regularization loss to constrain the geometry of the predicted point cloud in the missing region. The local topology regularization loss is composed of a Laplacian smoothing term. Inter-point repulsion term The weighted calculation method is used to suppress the divergence and noise of the predicted point cloud in the missing regions when the input point cloud is an incomplete sample. The calculation formula is as follows: In the formula, and These are the weighting coefficients for the smoothing term and the compaction term, respectively; the Laplace smoothing... The smoothness of the surface is constrained by the center of the k nearest neighbors, and its calculation formula is as follows: In the formula, Represents the first point in the point cloud One point, Point of Nearest neighbor set For this point of Number of nearest neighbor sets; The network parameter update module is used to obtain the total objective function loss value and to update the network parameters of the point cloud autoencoder network through gradient backpropagation.

7. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the leaf point cloud reconstruction method based on projection constraints and hybrid supervision as described in any one of claims 1-5.

Images