Plant three-dimensional reconstruction method for weak texture image and scale distortion

Abstract

A plant three-dimensional reconstruction method for a weak texture image and scale distortion comprises the following steps: step 1, recovering an initial Gaussian point cloud and camera parameters through an SfM method by using a multi-view RGB image; 2, differential rendering is executed in training iteration, a predicted image is compared with a real image, and a loss function containing various constraints is calculated to serve as a basis for gradient solving and parameter updating; 3, calculating a pixel weighted average gradient and an artifact suppression coefficient of each Gaussian in a visible view angle; 4, updating Gaussian parameters through back propagation, and adaptively triggering Gaussian cloning, splitting or deleting operation based on the pixel gradient, the covariance matrix and transparency; and 5, dynamically adjusting the Gaussian projection matrix according to the focal length and the resolution of the camera during rendering so as to maintain scale consistency, and performing anti-aliasing rendering in combination with pixel-level super-sampling. According to the method, the reconstruction precision and the structure reduction capability of the virtual plant in the weak texture region are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and 3D reconstruction, specifically a method for 3D reconstruction of plants. Background Technology

[0002] Virtual plants are a digital representation of real plants, converted into three-dimensional models using digital modeling and reconstruction techniques. These models can be used for simulation analysis, virtual display, and subsequent computational processing. Virtual plant models not only help in agriculture by simulating crop growth and development, conducting plant shape testing, and predicting yields, but also have wide applications in the gaming and film industries, significantly enhancing the realism and immersive experience of virtual scenes. However, the complex structure of plants and their weak surface texture are precisely the main bottlenecks of current mainstream 3D reconstruction technologies, making high-precision plant reconstruction and high-fidelity plant rendering still a challenge.

[0003] Currently, scholars both domestically and internationally have conducted research on plant modeling and proposed numerous modeling methods, which can be mainly divided into three categories: rule-based plant modeling methods, image-based plant modeling methods, and neural rendering-based plant reconstruction methods. Plant structures exhibit typical fractal characteristics, therefore rule-based modeling methods such as L-systems have long been widely used in virtual plant modeling. Due to the vast variety and intricate structures of plants, while rule-based modeling methods can reproduce the fractal features of plants, their framework-based rules often fail to accurately represent the morphology of real plants. To recreate the structural characteristics of plants in the real world, many researchers have attempted to reconstruct plants by combining images. Image-based modeling methods extract image information to recover plant structures. One type of method relies on structural features such as the plant skeleton, achieving high fidelity, but the model's applicability is limited, making it difficult to cover multiple plant types. Another type of method generates 3D models based on user-drawn sketches, offering high flexibility and rapid modeling. However, this type of technology requires extensive interaction for structurally complex plants and lacks sufficient support for dynamic growth simulation. With the development of artificial intelligence, many researchers have proposed plant reconstruction methods based on neural rendering. These methods can recover plant structure and appearance using only image input, offering advantages such as low barrier to entry and high automation. However, these methods have limited accuracy when reconstructing plants with dense foliage and complex layers, and suffer from high computational overhead and low generation efficiency. Nevertheless, this technology has good scalability and has become one of the main research directions in the field of plant reconstruction.

[0004] Image-based plant modeling methods first extract the 3D skeletal structure of plants from multi-view images, then fill it in using procedurally generated plant organs. While this method can reconstruct the skeletal structure of plants relatively well, it performs poorly for plants without a skeletal structure, and it still cannot reconstruct plants at the organ level. Due to the difficulty of multi-view acquisition, single-image reconstruction schemes have emerged. However, these methods are often limited by specific plant structures and rely on manual intervention, making automated reconstruction difficult. Therefore, researchers have gradually turned their attention to more flexible and intuitive sketching interaction methods. For example, some studies have combined hand-drawn sketches with procedural modeling to generate complete 3D tree models. To improve the realism of freehand sketch-based modeling, the academic community has further attempted to generate more accurate 3D branching structures through probabilistic optimization methods or deep learning techniques. However, densely distributed leaves often obscure the branch structure (especially secondary branches), which makes sketching difficult for non-professional users. In addition, the depth blur problem inherent in 2D sketches also limits their ability to generate complete and deterministic tree models.

[0005] Neural Rendering-Based Plant Reconstruction Methods: In recent years, NeRF-based neural rendering technology has become an important direction for 3D plant modeling and has shown great potential in agricultural phenotypic analysis. However, this type of method suffers from high computational cost, long training time, and low rendering efficiency, making it difficult to meet the needs of practical applications. The introduction of 3D Gaussian Splatting (3DGS) has provided a new technical path for plant reconstruction. However, existing methods still have several key limitations, namely, they cannot overcome the inherent defects of 3DGS, are only applicable to single plant scenes, and rely excessively on segmentation accuracy. To improve the applicability of this technology in plant modeling, it is necessary to focus on solving the bottleneck of insufficient Gaussian point fitting in weakly textured regions. Among the improved methods based on Dust3R, InstantSplat and EasySplat enhance the detail representation by increasing the point cloud density, but these methods only support sparse images as input. Complex regions of plants often result in poor reconstruction results during training due to insufficient fitting information. VGGT proposes a feedforward neural network that can effectively alleviate the above problems, but at the cost of significantly increasing computational overhead and reducing pose accuracy, it still cannot meet the application requirements of efficient reconstruction.

[0006] For high-fidelity plant rendering methods with anti-aliasing: Anti-aliasing mainly involves two strategies: oversampling (improving the effect by increasing the number of samples) and pre-filtering (applying a low-pass filter to the signal to satisfy the Nyquist limit). Recent neural rendering methods have integrated pre-filtering techniques to suppress aliasing. For example, Mip-NeRF introduces integrated position coding to attenuate high-frequency details. However, due to the differences in training mechanisms between NeRF and 3DGS, this scheme cannot be directly transferred to 3DGS. Mip-Splatting reveals that 3DGS has a problem of inconsistent projection scaling during the rasterization stage. This defect can lead to distortion of plant organ structures. Although its scene frequency constraint training strategy alleviates this problem to some extent, its experiments show that it still causes accuracy loss in some scenes and is difficult to meet the high-quality reconstruction requirements of complex plant structures. EWASplatting, on the other hand, generates a band-limited output that conforms to the Nyquist frequency of the image by applying a Gaussian low-pass filter to the two-dimensional Gaussian distribution projected in screen space. In summary, most of these methods only suppress the distortion of plant organs at the image level and do not fundamentally guarantee the consistency of the Gaussian projection process. Therefore, the rendering effect of plant structures is still not ideal.

[0007] Existing 3DGS frameworks are primarily designed for general-purpose scenarios, facing two key challenges in plant modeling tasks. First, 3DGS relies on sparse point clouds generated by Structure of Motion (SfM) as input, but SfM struggles to generate a sufficient number of initial point clouds in areas with repetitive plant textures. Simultaneously, Gaussian's original adaptive density control mechanism fails to effectively trigger cloning or splitting operations in weakly textured regions, resulting in insufficient Gaussian units for fine reconstruction in these areas. Second, when camera parameters change, the original 3DGS rendering method exhibits significant image aliasing. This phenomenon also exists in Neural Radiation Field (NeRF), but it is particularly pronounced in plant reconstruction: specifically, when zooming in, plant stems or leaves fade and disappear, while when zooming out, organs become distorted and deformed, severely impacting the direct application of the reconstruction results in practical scenarios such as virtual plants and agricultural mapping. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, this invention provides a method for 3D reconstruction of plants in weakly textured images and with scale distortion. On one hand, addressing the limitation of Gaussian gradients in effectively splitting weakly textured regions, a texture-aware Gaussian density adaptive control strategy is designed. This strategy, when calculating Gaussian gradients, not only considers the gradient values ​​at each viewpoint but also introduces the number of pixels affected by the Gaussian gradient as a weight to balance the gradient response, thereby enhancing the differentiation and fitting ability of Gaussian gradients in weakly textured plant regions. Simultaneously, by dynamically removing non-contributing Gaussian gradients, the number of Gaussian gradients is kept under control, effectively balancing memory usage and training efficiency. This strategy improves plant reconstruction accuracy while maintaining computational efficiency.

[0009] On the other hand, to address the issue of Gaussian projection scale distortion under different rendering settings, this invention proposes a camera-adaptive cross-scale projection method. This method dynamically adjusts the Gaussian projection process based on the rendering focal length and resolution, ensuring that the size of the Gaussian projection on the screen remains consistent, thereby maintaining the scale stability of plant organs under different rendering settings. Building upon this, this invention further introduces supersampling technology, effectively eliminating edge jaggedness and projection distortion through sub-pixel level detail reconstruction and multi-sampling fusion, significantly improving the visual consistency and geometric realism of plant organs in cross-scale rendering.

[0010] The technical solution adopted by this invention to solve its technical problem is:

[0011] A method for 3D reconstruction of plants in images with weak texture and scale distortion includes the following steps:

[0012] Step 1: Collect plant targets using multi-view RGB images, recover sparse or semi-dense point clouds and camera intrinsic and extrinsic parameters using the SfM method, and construct initial geometric priors in a unified coordinate system; then initialize the position, scale, color and opacity of a 3D Gaussian based on the point cloud, thereby providing reliable input for subsequent differentiable rendering and parameter optimization;

[0013] Step 2: In each iteration, perform a differentiable rendering operation based on the current Gaussian distribution set and camera parameters to generate the corresponding predicted image; then, compare the predicted image with the real multi-view image and calculate the composite loss function, which includes pixel-level reconstruction error, structure-aware constraint term, and parameter regularization term; use the composite loss function to measure the accuracy of the current reconstruction result and use it as the basis for optimizing gradient solution and model parameter update.

[0014] Step 3: Using the pixel gradient information obtained from differentiable rendering, the gradient magnitude of each Gaussian within its visible field of view is weighted and averaged to obtain a gradient index that reflects the difficulty of local fitting and texture intensity; the relationship between this index and the threshold is used to determine whether density enhancement needs to be triggered, thus providing a basis for cloning or splitting.

[0015] Step 4: Backpropagate the loss gradient of the loss function obtained in Step 2 to the Gaussian parameters through backpropagation, and use the optimizer to update its position, covariance, color and opacity attributes; based on the average pixel gradient and Gaussian covariance matrix obtained in Step 3, adaptively trigger the cloning or splitting of Gaussians to enhance the expressive power of weak texture regions; at the same time, combine opacity and scale regularization to automatically shrink or delete low-contribution Gaussians to achieve a controllable and stable density evolution process;

[0016] Step 5: In the rendering stage, the projection scaling factor is calculated based on the camera's focal length parameters and resolution information. The covariance matrix of the 2D Gaussian projection is dynamically adjusted to maintain scale consistency and structural stability under different observation distances and resolutions. At the same time, anti-aliasing rendering is performed in conjunction with a pixel-level supersampling strategy to effectively suppress aliasing artifacts caused by changes in camera parameters and improve the clarity and edge smoothness of detailed areas. Finally, a high-fidelity, multi-view consistent rendering result is output for the visualization and subsequent analysis of virtual plants.

[0017] Furthermore, the process of step 1 is as follows:

[0018] Step 1.1: Use an RGB camera to collect plant data. For a single plant, you can take a fixed-point, circular shot around the plant to obtain K multi-view image data, where K can be set according to actual needs, generally 100 to 200 images. If the plant is large, you still need to supplement the details from other angles in addition to the circular shot. Optionally, if there are limitations in the actual shooting scene or conditions, the shooting method described in Step 1.1 can also be replaced by shooting video and then manually cutting frames.

[0019] Step 1.2: Use the SfM algorithm to extract features, match, and solve poses for multi-view plant images; obtain sparse 3D point clouds through triangulation reconstruction, and generate higher density structural information by combining multi-view constraints as needed; in addition to outputting sparse (or semi-dense) 3D point clouds, the position and attribute parameters of each camera will also be output. These results will serve as the basic input for subsequent 3D GS initialization of Gaussian center position and execution of differentiable rendering optimization. Optionally, the initial point cloud in Step 1.2 can be directly replaced by a random point cloud.

[0020] Furthermore, step 2 is as follows:

[0021] Step 2.1: Based on the camera's intrinsic and extrinsic parameters, project each 3D Gaussian ellipse onto the image plane and convert the corresponding 3D covariance matrix into a 2D Gaussian elliptic kernel to determine its coverage area in screen space. Subsequently, calculate the color contribution of each Gaussian to the target pixel in depth order from front to back. Weight the covered pixels using the 2D Gaussian kernel, and accumulate the color value and opacity layer by layer in a differentiable transparency mixing method. Finally, obtain the pixel color value formed by the superposition of multiple Gaussians in screen space, achieving a continuously differentiable image rendering result.

[0022] Step 2.2: Compare the image generated by Gaussian rendering with the real image from the corresponding viewpoint pixel by pixel, and define the loss function:

[0023]

[0024] in, This represents the pixel-level error between the predicted image and the real image. λ represents the structure-perceived difference based on structural similarity. D-SSIM The weights for both are used to balance the reconstruction accuracy of texture details and global structure. To further constrain the growth of the number of Gaussians in weakly textured regions and improve training stability, opacity and scale regularization terms are introduced on top of the above, forming the complete total loss function, where |o i |1 represents the opacity parameter of the i-th Gaussian, |r i |1 represents the mean of its ellipsoidal scale; λ o With λ r These are the regularization weights. Through this regularization term, the proliferation of invalid or redundant Gaussians can be automatically suppressed during the optimization process, thereby effectively controlling the consumption of computing resources while ensuring reconstruction accuracy.

[0025] Furthermore, step 3 is as follows:

[0026] Step 3.1: For the i-th Gaussian, calculate its average pixel gradient based on the viewpoints it participates in and the number of pixels affected by each viewpoint. When the average gradient value is greater than the threshold, the system determines that the Gaussian has significant reconstruction error or insufficient detail in the current stage, triggering a splitting or cloning operation to improve the fitting accuracy of the local structure. Define the formula for judging the average pixel gradient:

[0027]

[0028] Among them, M i This represents the number of viewpoints visible to the i-th Gaussian during training; This represents the number of pixels covered by the Gaussian in the k-th viewpoint, used to measure its influence range in that viewpoint; and τ represents the horizontal and vertical components of the pixel gradient calculated in the k-th viewpoint, used to reflect the intensity of local texture changes; pos This is the gradient threshold parameter, used to control the triggering conditions for Gaussian splitting or cloning;

[0029] Step 3.2: When the camera is too close to the scene, the average pixel gradient will be abnormally amplified due to the close viewing angle, inducing floating artifacts. Therefore, it is necessary to introduce scene relative scale to normalize and suppress the gradient. The scene relative scale is calculated as follows:

[0030]

[0031] Among them, C i Let C represent the world coordinates of the i-th Gaussian coordinate system. j Let represent the world coordinates of the j-th training camera. The furthest distance between the Gaussian point and the training camera is used as the scene scale boundary.

[0032] Step 3.3: Calculate the gradient suppression coefficient based on the relative scale of the scene. Define the gradient suppression coefficient:

[0033]

[0034] in This represents the depth value of the i-th Gaussian in the camera coordinate system at the k-th viewpoint. Multiplying this gradient suppression coefficient in the average pixel gradient calculation formula in step 3.1 can effectively suppress artifacts caused by erroneous splitting.

[0035] The process of step 4 is as follows:

[0036] Step 4.1: Perform backpropagation on the loss function in 2.2 to obtain the gradient, which is used to update the transparency, position, covariance matrix and spherical harmonic function of the Gaussian, so that each updated Gaussian is more in line with the local geometric surface and the color is more realistic.

[0037] Step 4.2: For each Gaussian point, determine whether its average pixel gradient exceeds the threshold. When the index exceeds the set threshold (that is, when the Gaussian has insufficient fitting ability in the current region and there is a significant error), the cloning or splitting operation is triggered to increase the degree of freedom of expression in the local region, thereby further fitting high-frequency details. Specifically, smaller Gaussians will trigger the cloning operation, while larger Gaussians will trigger the splitting operation.

[0038] Step 4.3: During the training process, a Gaussian clearing operation is performed once every preset number of iterations. Since the loss function described in Step 2.2 includes regularization constraints on opacity and scale, Gaussians that contribute less to the final rendering result can gradually shrink or become transparent during the optimization process. When the opacity of a Gaussian is lower than the set threshold, or its spatial scale is smaller than the preset threshold, it is determined that the Gaussian has basically failed at the current stage and is directly deleted to reduce redundant calculations and save training resources.

[0039] The process of step 5 is as follows:

[0040] Step 5.1: After a certain number of iterations following the above steps, a relatively stable reconstruction model can be obtained. At this point, the reconstruction model can be rendered on different terminals according to different rendering settings. If the rendering settings used are different from those used during training, it may cause distortion of plant organs. To prevent this serious problem, the adaptive projection coefficient q is first calculated based on the intrinsic and extrinsic parameters of the rendering camera. Its expression is:

[0041]

[0042] Where, size t With size r These represent the pixel dimensions during the training and rendering phases, respectively; resolution t With resolution r These refer to image resolution and camera. t With camera r The distance from the camera to the center of the scene;

[0043] Step 5.2: Project the 3D Gaussian kernel onto the image plane and adjust its covariance matrix to obtain the 2D Gaussian kernel. The projection calculation is defined as follows:

[0044]

[0045] in, Let be a two-dimensional covariance matrix, where s is a fixed expansion coefficient, typically taken as 0.03, and p... k With the center of the Gaussian plane and x as the sampling point of the Gaussian probability distribution, this projection achieves geometric scale consistency at different resolutions, preventing distortion caused by excessive contraction or expansion of the Gaussian plane.

[0046] Step 5.3: To reduce aliasing artifacts, perform an oversampling operation on each pixel, dividing the pixel into 3×3 sub-sampling points. Calculate the color value formed by the superposition of multiple Gaussian pixels at each sub-sampling point using the following formula:

[0047]

[0048] Where S is the total number of samples, which is 3 by default. Let C be the opacity of the i-th Gaussian at the s-th subsample point. i The corresponding color value, where N is the total Gaussian count. Cumulative transmittance is defined as:

[0049]

[0050] When i is 0 A value of 1 indicates that the pixel has not acquired any accumulated color at this time. In low-resolution rendering, the expressive power of a single pixel is limited and often cannot represent the mixed rendering results of many Gaussians. The above calculation can effectively alleviate the image distortion problem caused by insufficient sampling, thereby further improving the rendering quality of the image.

[0051] The technical concept of this invention is as follows: To enhance the 3D representation of individual plants and scenes, and to solve the problem of inconsistent plant organ scales in cross-resolution rendering, a Gaussian self-updating module based on pixel gradient awareness is designed to improve the geometric fitting ability of weakly textured regions. This module automatically identifies weakly textured regions by analyzing the gradient information of surrounding pixels in the image and adaptively performs Gaussian cloning and splitting operations, thereby significantly improving point cloud density and structural detail representation. Furthermore, scale and opacity regularization constraints are introduced to effectively suppress Gaussian overgrowth, improving reconstruction quality while avoiding the risk of memory overflow. Further, to achieve scale consistency of plant organs under different rendering settings, a supersampling anti-aliasing module based on rendering consistency is developed. By dynamically adjusting the Gaussian projection scale and combining it with a pixel-level supersampling strategy, the consistency of rendering results is maintained under cross-resolution, focal length, and depth-of-field conditions, significantly suppressing near-field needle artifacts and near-far scale distortion.

[0052] The beneficial effects of this invention are mainly reflected in:

[0053] 1. Significantly improves reconstruction accuracy and structural restoration capability in weakly textured regions: This invention utilizes a texture-aware adaptive density control strategy to continuously evaluate local features of the scene. When the point cloud density and gradient information in weakly textured regions such as leaves are insufficient to support high-quality reconstruction, it proactively triggers Gaussian cloning or splitting operations at the corresponding locations. This mechanism significantly enhances the fitting of key fine structures by increasing local representation strength, thereby effectively overcoming the problems of blurring, breakage, and lack of detail in weakly textured regions, ensuring the continuity and high fidelity of complex structures such as leaf edges and twigs.

[0054] 2. Achieving consistent rendering across scales and effectively suppressing distortion artifacts: This invention proposes a camera-adaptive projection method, combined with a screen-space supersampling strategy, to ensure that Gaussian projection maintains scale consistency under different display conditions. This mechanism significantly suppresses problems such as jagged edges, needle-like artifacts, local deformation, and abnormal organ scaling, ensuring that plant leaves and branches maintain a clear, stable, and consistent visual effect throughout scaling, camera zooming, and cross-terminal rendering.

[0055] 3. Enhancing the model's applicability, stability, and application value: Based on optimized rendering consistency and density control strategies, this invention is applicable to tasks such as single plant modeling, plant community reconstruction, and reconstruction of complex vegetation environments. Under different scene conditions, this method demonstrates robust geometric fitting capabilities and stable image output quality, providing reliable technical support for applications such as virtual plant construction, plant phenotyping, and digital twins, significantly improving the technology's generalization ability and engineering practical value. Attached Figure Description

[0056] Figure 1 This is a flowchart of a method for 3D reconstruction of plants in weakly textured images with scale distortion.

[0057] Figure 2 This is the flowchart for adaptive density control.

[0058] Figure 3 This is a flowchart of the consistency projection process. Detailed Implementation

[0059] The present invention will now be further described with reference to the accompanying drawings.

[0060] Reference Figures 1-3 A method for 3D reconstruction of plants in images with weak texture and scale distortion includes the following steps:

[0061] Step 1: Acquire plant targets using multi-view RGB images, recover sparse or semi-dense point clouds and camera intrinsic and extrinsic parameters using the SfM method, and construct initial geometric priors in a unified coordinate system; then initialize the position, scale, color, and opacity of a 3D Gaussian cloud based on the point cloud, thus providing reliable input for subsequent differentiable rendering and parameter optimization; the process is as follows:

[0062] Step 1.1: Use an RGB camera to collect plant data. For a single plant, you can take a fixed-point, circular shot around the plant to obtain K multi-view image data, where K can be set according to actual needs, generally 100 to 200 images. If the plant is large, you still need to supplement the details from other angles in addition to the circular shot. Optionally, if there are limitations in the actual shooting scene or conditions, the shooting method described in Step 1.1 can also be replaced by shooting video and then manually cutting frames.

[0063] Step 1.2: Use the SfM algorithm to extract features, match, and solve poses for multi-view plant images; obtain sparse 3D point clouds through triangulation reconstruction, and generate higher density structural information by combining multi-view constraints as needed; in addition to outputting sparse (or semi-dense) 3D point clouds, the position and attribute parameters of each camera will also be output. These results will serve as the basic input for subsequent 3D GS initialization of Gaussian center position and execution of differentiable rendering optimization. Optionally, the initial point cloud in Step 1.2 can be directly replaced by a random point cloud.

[0064] Step 2: In each iteration, a differentiable rendering operation is performed based on the current Gaussian distribution set and camera parameters to generate the corresponding predicted image. Then, the predicted image is compared with the real multi-view image, and a composite loss function including pixel-level reconstruction error, structure-aware constraint terms, and parameter regularization terms is calculated. The accuracy of the current reconstruction result is measured using this composite loss function, and this is used as the basis for optimizing gradient calculation and model parameter updates. The process is as follows:

[0065] Step 2.1: Based on the camera's intrinsic and extrinsic parameters, project each 3D Gaussian ellipse onto the image plane and convert the corresponding 3D covariance matrix into a 2D Gaussian elliptic kernel to determine its coverage area in screen space. Subsequently, calculate the color contribution of each Gaussian to the target pixel in depth order from front to back. Weight the covered pixels using the 2D Gaussian kernel, and accumulate the color value and opacity layer by layer in a differentiable transparency mixing method. Finally, obtain the pixel color value formed by the superposition of multiple Gaussians in screen space, achieving a continuously differentiable image rendering result.

[0066] Step 2.2: Compare the image generated by Gaussian rendering with the real image from the corresponding viewpoint pixel by pixel, and define the loss function:

[0067]

[0068] in, This represents the pixel-level error between the predicted image and the real image. λ represents the structure-perceived difference based on structural similarity. D-SSIM The weights for both are used to balance the reconstruction accuracy of texture details and global structure; to further constrain the growth of the Gaussian number in weak texture regions and improve training stability, opacity and scale regularization terms are introduced on the basis of the above, forming a complete total loss function, where |o i |1 represents the opacity parameter of the i-th Gaussian, |r i |1 represents the mean of its ellipsoidal scale; λ o With λ rThese are the regularization weights. Through this regularization term, the proliferation of invalid or redundant Gaussians can be automatically suppressed during the optimization process, thereby effectively controlling the consumption of computing resources while ensuring reconstruction accuracy.

[0069] Step 3: Using the pixel gradient information obtained from differentiable rendering, a weighted average of the gradient magnitude of each Gaussian within its visible view range is calculated to obtain a gradient index reflecting the difficulty of local fitting and texture intensity; the relationship between this index and a threshold is used to determine whether density enhancement needs to be triggered, thus providing a basis for cloning or splitting; the process is as follows:

[0070] Step 3.1: For the i-th Gaussian, calculate its average pixel gradient based on the viewpoints it participates in and the number of pixels affected by each viewpoint. When the average gradient value is greater than the threshold, the system determines that the Gaussian has significant reconstruction error or insufficient detail in the current stage, triggering a splitting or cloning operation to improve the fitting accuracy of the local structure. Define the formula for judging the average pixel gradient:

[0071]

[0072] Among them, M i This represents the number of viewpoints visible to the i-th Gaussian during training; This represents the number of pixels covered by the Gaussian in the k-th viewpoint, used to measure its influence range in that viewpoint; and τ represents the horizontal and vertical components of the pixel gradient calculated in the k-th viewpoint, used to reflect the intensity of local texture changes; pos This is the gradient threshold parameter, used to control the triggering conditions for Gaussian splitting or cloning;

[0073] Step 3.2: When the camera is too close to the scene, the average pixel gradient will be abnormally amplified due to the close viewing angle, inducing floating artifacts. Therefore, it is necessary to introduce scene relative scale to normalize and suppress the gradient. The scene relative scale is calculated as follows:

[0074]

[0075] Among them, C i Let C represent the world coordinates of the i-th Gaussian coordinate system. j Let represent the world coordinates of the j-th training camera. The furthest distance between the Gaussian point and the training camera is used as the scene scale boundary.

[0076] Step 3.3: Calculate the gradient suppression coefficient based on the relative scale of the scene. Define the gradient suppression coefficient:

[0077]

[0078] in This represents the depth value of the i-th Gaussian in the camera coordinate system at the k-th viewpoint. Multiplying this gradient suppression coefficient in the average pixel gradient calculation formula in step 3.1 can effectively suppress artifacts caused by erroneous splitting.

[0079] Step 4: Backpropagate the loss gradient obtained in Step 2 to the Gaussian parameters using the loss function, and update their position, covariance, color, and opacity attributes using the optimizer; based on the average pixel gradient and Gaussian covariance matrix obtained in Step 3, adaptively trigger the cloning or splitting of Gaussians to enhance the expressive power of weak texture regions; simultaneously, combine opacity and scale regularization to automatically shrink or delete low-contribution Gaussians, achieving a controllable and stable density evolution process; the process is as follows:

[0080] Step 4.1: Perform backpropagation on the loss function in 2.2 to obtain the gradient, which is used to update the transparency, position, covariance matrix and spherical harmonic function of the Gaussian, so that each updated Gaussian is more in line with the local geometric surface and the color is more realistic.

[0081] Step 4.2: For each Gaussian point, determine whether its average pixel gradient exceeds the threshold. When the index exceeds the set threshold (that is, when the Gaussian has insufficient fitting ability in the current region and there is a significant error), the cloning or splitting operation is triggered to increase the degree of freedom of expression in the local region, thereby further fitting high-frequency details. Specifically, smaller Gaussians will trigger the cloning operation, while larger Gaussians will trigger the splitting operation.

[0082] Step 4.3: During the training process, a Gaussian clearing operation is performed once every preset number of iterations. Since the loss function described in Step 2.2 includes regularization constraints on opacity and scale, Gaussians that contribute less to the final rendering result can gradually shrink or become transparent during the optimization process. When the opacity of a Gaussian is lower than the set threshold, or its spatial scale is smaller than the preset threshold, it is determined that the Gaussian has basically failed at the current stage and is directly deleted to reduce redundant calculations and save training resources.

[0083] Step 5: In the rendering stage, the projection scaling factor is calculated based on the camera's focal length parameters and resolution information. The covariance matrix of the 2D Gaussian projection is dynamically adjusted to maintain scale consistency and structural stability under different observation distances and resolutions. Simultaneously, anti-aliasing rendering is performed using a pixel-level supersampling strategy to effectively suppress aliasing artifacts caused by camera parameter variations, improving the clarity and edge smoothness of detailed areas. Finally, a high-fidelity, multi-view consistent rendering result is output for the visualization and subsequent analysis of virtual plants. The process is as follows:

[0084] Step 5.1: After a certain number of iterations following the above steps, a relatively stable reconstruction model can be obtained. At this point, the reconstruction model can be rendered on different terminals according to different rendering settings. If the rendering settings used are different from those used during training, it may cause distortion of plant organs. To prevent this serious problem, the adaptive projection coefficient q is first calculated based on the intrinsic and extrinsic parameters of the rendering camera. Its expression is:

[0085]

[0086] Where, size t With size r These represent the pixel dimensions during the training and rendering phases, respectively; resolution t With resolution r These refer to image resolution and camera. t With camera r The distance from the camera to the center of the scene;

[0087] Step 5.2: Project the 3D Gaussian kernel onto the image plane and adjust its covariance matrix to obtain the 2D Gaussian kernel. The projection calculation is defined as follows:

[0088]

[0089] in, Let be a two-dimensional covariance matrix, where s is a fixed expansion coefficient, typically taken as 0.03, and p... k With the center of the Gaussian plane and x as the sampling point of the Gaussian probability distribution, this projection achieves geometric scale consistency at different resolutions, preventing distortion caused by excessive contraction or expansion of the Gaussian plane.

[0090] Step 5.3: To reduce aliasing artifacts, perform an oversampling operation on each pixel, dividing the pixel into 3×3 sub-sampling points. Calculate the color value formed by the superposition of multiple Gaussian pixels at each sub-sampling point using the following formula:

[0091]

[0092] Where S is the total number of samples, which is 3 by default. Let C be the opacity of the i-th Gaussian at the s-th subsample point. i The corresponding color value, where N is the total Gaussian count. Cumulative transmittance is defined as:

[0093]

[0094] When i is 0 A value of 1 indicates that the pixel has not acquired any accumulated color at this time. In low-resolution rendering, the expressive power of a single pixel is limited and often cannot represent the mixed rendering results of many Gaussians. The above calculation can effectively alleviate the image distortion problem caused by insufficient sampling, thereby further improving the rendering quality of the image.

[0095] This embodiment addresses the blurry reconstruction problem caused by sparse point clouds and insufficient gradients in low-texture areas such as plant leaves. It automatically identifies low-texture areas through multi-view pixel gradient statistics. Furthermore, based on gradient magnitude and direction characteristics, Gaussian cloning or splitting operations are triggered respectively, thereby achieving adaptive density enhancement. Simultaneously, the method introduces regularization constraints based on opacity and scale, which can automatically suppress the growth and shrinkage of ineffective Gaussians. This mechanism significantly improves the expressive power of structural details without increasing the GPU memory burden, resulting in clearer geometric reconstruction of fine structures such as plant leaf edges and twigs.

[0096] This embodiment proposes a camera-adaptive cross-scale stable projection method. Addressing the issues of organ-scale distortion, jagged edges, and needle-like artifacts that arise with existing 3D Gaussian splashing techniques at different camera distances, resolutions, or focal lengths, this method introduces a projection scaling factor related to rendering resolution, pixel coverage area, and camera distance to dynamically adjust the scale of the 2D Gaussian projection. Combined with an oversampling-based anti-aliasing strategy, this method ensures rendering consistency across different scales from a fundamental mechanism, ultimately achieving stable visual effects for plant scenes during scaling, camera zooming, and cross-platform rendering.

[0097] This embodiment constructs a 3D Gaussian reconstruction pipeline for plant scenes. The pipeline takes multi-view plant images as input and first initializes the point cloud and camera parameters using the SfM method. Then, during model training and optimization, the pipeline utilizes a weak texture-aware densification module to enhance geometric representation and optimize the model's geometric details. In the subsequent rendering stage, a cross-scale projection consistency module is invoked to dynamically adjust the Gaussian projection to ensure the stability of the output results. Finally, the pipeline generates a 3D plant model with clear details, realistic structure, and cross-scale consistency. This framework is applicable to scenes such as single plants and plant communities, exhibiting good generalization ability and engineering applicability.

[0098] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for 3D reconstruction of plants in images with weak texture and scale distortion, characterized in that, The method includes the following steps: Step 1: Collect plant targets using multi-view RGB images, recover sparse or semi-dense point clouds and camera intrinsic and extrinsic parameters using the SfM method, and construct initial geometric priors in a unified coordinate system; then initialize the position, scale, color and opacity of a 3D Gaussian based on the point cloud, thereby providing reliable input for subsequent differentiable rendering and parameter optimization; Step 2: In each iteration, perform a differentiable rendering operation based on the current Gaussian distribution set and camera parameters to generate the corresponding predicted image; then, compare the predicted image with the real multi-view image and calculate the composite loss function, which includes pixel-level reconstruction error, structure-aware constraint term, and parameter regularization term; use the composite loss function to measure the accuracy of the current reconstruction result and use it as the basis for optimizing gradient solution and model parameter update. Step 3: Using the pixel gradient information obtained from differentiable rendering, the gradient magnitude of each Gaussian within its visible field of view is weighted and averaged to obtain a gradient index that reflects the difficulty of local fitting and texture intensity; the relationship between this index and the threshold is used to determine whether density enhancement needs to be triggered, thus providing a basis for cloning or splitting. Step 4: Backpropagate the loss gradient of the loss function obtained in Step 2 to the Gaussian parameters, and use the optimizer to update its position, covariance, color and opacity properties; Based on the average pixel gradient and Gaussian covariance matrix obtained in step 3, the cloning or splitting of Gaussians is adaptively triggered to enhance the expressive power of weak texture regions; at the same time, by combining opacity and scale regularization, low-contribution Gaussians are automatically shrunk or deleted to achieve a controllable and stable density evolution process. Step 5: In the rendering stage, the projection scaling factor is calculated based on the camera's focal length parameters and resolution information. The covariance matrix of the 2D Gaussian projection is dynamically adjusted to maintain scale consistency and structural stability under different observation distances and resolutions. At the same time, anti-aliasing rendering is performed in conjunction with a pixel-level supersampling strategy to effectively suppress aliasing artifacts caused by changes in camera parameters and improve the clarity and edge smoothness of detailed areas. Finally, a high-fidelity, multi-view consistent rendering result is output for the visualization and subsequent analysis of virtual plants.

2. The method for 3D reconstruction of plants in weakly textured images and with scale distortion as described in claim 1, characterized in that, The process of step 1 is as follows: Step 1.1: Use an RGB camera to collect plant data. For a single plant, you can take a fixed-point shot around the plant to obtain K multi-view image data. Step 1.2: Use the SfM algorithm to extract features, match, and solve poses for multi-view plant images; obtain sparse 3D point clouds through triangulation reconstruction, and generate higher density structural information by combining multi-view constraints as needed; in addition to outputting sparse or semi-dense 3D point clouds, the position and attribute parameters of each camera will also be output. These results will serve as the basic input for subsequent 3D GS initialization of Gaussian center position and execution of differentiable rendering optimization.

3. The method for three-dimensional reconstruction of plants in weakly textured images and with scale distortion as described in claim 1 or 2, characterized in that, The process of step 2 is as follows: Step 2.1: Based on the camera's intrinsic and extrinsic parameters, project each 3D Gaussian ellipse onto the image plane and convert the corresponding 3D covariance matrix into a 2D Gaussian elliptic kernel to determine its coverage area in screen space. Subsequently, calculate the color contribution of each Gaussian to the target pixel in depth order from front to back. Weight the covered pixels using the 2D Gaussian kernel, and accumulate the color value and opacity layer by layer in a differentiable transparency mixing method. Finally, obtain the pixel color value formed by the superposition of multiple Gaussians in screen space, achieving a continuously differentiable image rendering result. Step 2.2: Compare the image generated by Gaussian rendering with the real image from the corresponding viewpoint pixel by pixel, and define the loss function: in, This represents the pixel-level error between the predicted image and the real image. λ represents the structurally perceived difference based on structural similarity. D-SSIM These are the weighting coefficients for both, used to balance the reconstruction accuracy of texture details and global structure; Introducing opacity and scale regularization terms, we construct the complete total loss function, where |o i |1 represents the opacity parameter of the i-th Gaussian, |r i |1 represents the mean of its ellipsoidal scale; λ o With λ r These are the regularization weights.

4. The method for three-dimensional reconstruction of plants in weakly textured images and with scale distortion as described in claim 1 or 2, characterized in that, The process of step 3 is as follows: Step 3.1: For the i-th Gaussian, calculate its average pixel gradient based on the viewpoints it participates in and the number of pixels affected by each viewpoint. When the average gradient value is greater than the threshold, the system determines that the Gaussian has obvious reconstruction error or insufficient detail in the current stage, triggering a split or cloning operation. Define the formula for judging the average pixel gradient: Among them, M i This represents the number of viewpoints visible to the i-th Gaussian during training; This represents the number of pixels covered by the Gaussian in the k-th viewpoint, used to measure its influence range in that viewpoint; and τ represents the horizontal and vertical components of the pixel gradient calculated in the k-th viewpoint, used to reflect the intensity of local texture changes; pos This is the gradient threshold parameter, used to control the triggering conditions for Gaussian splitting or cloning; Step 3.2: Introduce scene relative scale and normalize and suppress the gradient. The scene relative scale is calculated as follows: Among them, C i Let C represent the world coordinates of the i-th Gaussian coordinate. j Let represent the world coordinates of the j-th training camera, that is, the farthest distance between the Gaussian point and the training camera is used as the scene scale boundary; Step 3.3: Calculate the gradient suppression coefficient based on the relative scale of the scene, and define the gradient suppression coefficient: in This represents the depth value of the i-th Gaussian in the camera coordinate system at the k-th viewpoint. It is multiplied by this gradient suppression coefficient in the average pixel gradient calculation formula in step 3.1 to effectively suppress artifacts caused by erroneous splitting.

5. The method for three-dimensional reconstruction of plants in weakly textured images and with scale distortion as described in claim 3, characterized in that, The process of step 4 is as follows: Step 4.1: Perform backpropagation on the loss function in 2.2 to obtain the gradient, which is used to update the transparency, position, covariance matrix and spherical harmonic function of the Gaussian, so that each updated Gaussian is more in line with the local geometric surface and the color is more realistic. Step 4.2: For each Gaussian point, determine whether its average pixel gradient exceeds the threshold. When the index exceeds the set threshold, trigger the cloning or splitting operation to increase the degree of expression freedom in the local region. Step 4.3: During the training process, a Gaussian clearing operation is performed once every preset number of iterations. Since the loss function described in Step 2.2 includes regularization constraints on opacity and scale, Gaussians that contribute less to the final rendering result can gradually shrink or become transparent during the optimization process. When the opacity of a Gaussian is lower than the set threshold, or its spatial scale is smaller than the preset threshold, it is determined that the Gaussian has basically failed at the current stage and is directly deleted.

6. The method for three-dimensional reconstruction of plants in weakly textured images and with scale distortion as described in claim 1 or 2, characterized in that, The process of step 5 is as follows: Step 5.1: First, calculate the adaptive projection coefficient q based on the intrinsic and extrinsic parameters of the rendering camera. Its expression is: Where, size t With size r These represent the pixel dimensions during the training and rendering phases, respectively; resolution t With resolution r These refer to image resolution and camera. t With camera r The distance from the camera to the center of the scene; Step 5.2: Project the 3D Gaussian kernel onto the image plane and adjust its covariance matrix to obtain the 2D Gaussian kernel. The projection calculation is defined as follows: in, Let be a two-dimensional covariance matrix, where s is a fixed expansion coefficient, typically taken as 0.03, and p... k With the center of the Gaussian plane and x as the sampling point of the Gaussian probability distribution, this projection achieves geometric scale consistency at different resolutions, preventing distortion caused by excessive contraction or expansion of the Gaussian plane. Step 5.3: Perform an oversampling operation on each pixel, dividing the pixel into 3×3 sub-sampling points. Calculate the color value formed by the superposition of multiple Gaussian pixels at each sub-sampling point using the following formula: Where S is the total number of samples, which is 3 by default. Let C be the opacity of the i-th Gaussian at the s-th subsample point. i The corresponding color value, where N is the total Gaussian count. Cumulative transmittance is defined as: When i is 0 A value of 1 indicates that the pixel has not received any accumulated color at this time.

Images