Plant physical model building method based on 3D Gaussian

Abstract

The invention discloses a plant physical model modeling method based on 3D Gaussian, and provides a new method for high-precision reconstruction and physical simulation of plants through combination of background segmentation, topological structure constraint and a cantilever beam physical model. Background segmentation is carried out on the plant through a large model generalization fine tuning method, so that the precision of plant organ-level instance segmentation is improved, and the problems of background interference and wrong segmentation are reduced; meanwhile, in combination with a Weibull sorting method and topological structure constraints, high-precision branches more conforming to the geometric morphology of the plant are achieved; finally, based on a cantilever beam model, a rotation and revolution mechanism of a Gaussian ellipsoid is provided, and the authenticity of plant physical simulation and the accuracy of dynamic change are improved. According to the invention, not only is the precision of plant reconstruction improved, but also a more real physical effect can be realized in the process of simulating plant deformation.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional reconstruction and plant simulation, and specifically relates to a 3D Gaussian-based plant physical modeling method, which is suitable for plant three-dimensional modeling, simulation and dynamic deformation simulation. Background Art

[0002] In recent years, 3D plant modeling has rapidly developed as a research direction that integrates multiple disciplines, including botany, agronomy, ecology, mathematics, and computer graphics. This technology can realistically reproduce the 3D structure and growth process of plants, visually revealing their morphological structure and physical laws. Such models have been widely used in a variety of fields, such as the production of 3D animation, the generation of virtual scenes in video games, the creation of film and television special effects and advertising creatives, garden planning, community design, and urban planning.

[0003] Current mainstream plant 3D modeling methods (such as MVS, NeRF, and 3D Gaussian Splatting) typically rely on complex 3D reconstruction algorithms and post-processing techniques to incorporate physical rules, thereby achieving realistic simulations of non-rigid plant deformations. However, the realism of plant 3D modeling depends not only on the driving force of the physical model but also on the accuracy of the reconstruction method and the precision of the plant model itself. Image-based plant reconstruction methods rely on real-world scene images to accurately restore plant morphology and structure, a process that relies heavily on image segmentation techniques. Plant image segmentation, particularly organ-level instance segmentation, plays a crucial role in image-based 3D plant reconstruction. By accurately segmenting and identifying plant parts, instance segmentation not only improves the quality of point cloud data but also provides essential support for subsequent skeleton extraction and high-precision model reconstruction. Despite significant progress in plant 3D modeling over the years, universal instance segmentation methods and their impact on reconstruction quality have been limited. Furthermore, achieving efficient model generation, accurate and controllable 3D model reconstruction, and realistic physical simulation is difficult. In terms of three-dimensional reconstruction, the complex environment, occlusion of branches and leaves, and weak texture characteristics of plants affect the reconstruction accuracy, resulting in the loss of some structures or missing details. At the same time, mainstream 3D reconstruction methods usually focus on the recovery of geometric details, but lack integration with physical-driven methods, which makes it challenging to accurately and effectively simulate the dynamic changes of plant structures. In terms of physical simulation, although the latest physical Gaussian method can simulate various non-rigid deformations, it has limited consideration of plant-specific structural characteristics, which affects the accuracy of reconstruction and simulation. In addition, most of the existing public datasets are static plant morphologies, and there is a lack of dynamic deformation data of plants under external forces, which limits the generalization ability of data-driven methods and physical simulation methods. Therefore, how to balance realistic and efficient reconstruction and simulation with physical mechanisms in complex plant structures remains one of the important challenges in this field.

[0004] Recently, PhysGaussian has incorporated physics into the simulation and rendering of 3D object behavior by combining 3D Gaussian splatting with continuum mechanics, achieving physics-based dynamic effects and highly realistic rendering. This approach opens the door to rapidly reconstructing high-precision 3D plant reconstructions and simulating realistic plant dynamic deformation. Currently, PhysGaussian can simulate the dynamic effects of various materials, such as elastic solids, metals, fractured materials, granular materials, viscoplastic materials, and collisions. However, research on 3D plant reconstruction and the simulation of branch bending deformation under load remains limited. The main challenge lies in the complex structure of plant canopies, where dense leaves and interlocking branches can affect the accuracy of 3D plant reconstructions. Furthermore, for woody plants, branches at different levels have significantly different material properties; even within the same level, material properties gradually change from base to tip. Therefore, only by restoring the hierarchical details of the branches during the reconstruction process can realistic simulation of their dynamic deformation under external forces be achieved. However, existing methods have difficulty in effectively restoring the material details of branches during the three-dimensional reconstruction stage of plants, and are only robust to the reconstruction of simple branch structures. In the subsequent physical simulation stage, they mostly rely on a simple uniform elastic modulus to simulate the elastic deformation of objects, which makes it difficult to achieve realistic effects of branch bending (for example, simulating the bending and swaying of branches under the weight of snow in a natural environment or the swaying of branches in the wind). In summary, the three-dimensional reconstruction of plants and physical simulation are tightly coupled. For an advanced plant modeling method, it is not only required to accurately simulate the morphological details of the plant, but also to be able to realistically reproduce the non-rigid deformation of branches under the action of external forces.

[0005] Plant 3D reconstruction and physical simulation have a history of nearly 40 years. Existing plant modeling methods can be categorized into four types: 1) rule-based plant modeling methods; 2) sketch-based modeling methods; 3) image-based plant reconstruction methods; and 4) plant modeling methods based on Nerf or 3D Gaussian Splatting.

[0006] The first category primarily considers using mathematical models to simulate plant growth or morphological characteristics, including L-systems, reference axis techniques, dual-scale automata, fractal methods, function iteration systems, and particle systems. Dynamic plant geometry modeling based on rule- and grammar-based procedural models makes it difficult to control the final form of the plant model, resulting in a three-dimensional structure that is far from the plant's true structure.

[0007] The second type of sketch-based modeling method uses interactive interfaces and sketching to assist in plant feature extraction and model editing, enabling rapid 3D plant simulation. This method is highly flexible, but its applicability is limited to limited tree species and the branch morphology is monotonous. Modeling entire complex plants is particularly time-consuming and labor-intensive, and similarly, it fails to incorporate real-world data, generally failing to achieve the required level of detailed plant reconstruction.

[0008] With the increasing requirements for plant details and reconstruction accuracy, the third type of method, image-based plant reconstruction methods, has received increasing attention from researchers. This method first requires the input of one or more images of the simulation target, and then uses some prior knowledge and rules to generate a plant model that is relatively similar to the original input image. It relies on image data from real sources and aims to construct a plant model with a realistic branching structure. However, the plant structure in real scenes is complex and the background interference is severe, making it challenging to faithfully restore its branching structure. This method has the advantages of convenient data acquisition, low computational complexity, and strong reconstruction realism. However, it also has some disadvantages, such as: insufficient degree of automation, the need for human interaction, and the established plant models are mostly static models, which cannot realize physical simulation or editing of the dynamic behavior of branches.

[0009] In recent years, 3D scene reconstruction based on Neural Radiance Fields (NeRF) or 3D Gaussian Splatting has attracted increasing attention from researchers. Some have noted the excellent performance and quality of Neural Radiance Fields in high-fidelity 3D reconstruction of plants, and explored the advantages and disadvantages of Instant-NGP and Instant-NSR in 3D plant reconstruction, including obtaining high-quality geometric meshes in complex plant structures. Unlike traditional 3D objects composed of points and surfaces, the implicit features of NeRF make it impossible to perform intuitive editable operations. Although some methods have overcome the limitations of implicit features, it is still difficult to achieve high-quality simulations with physical mechanisms.

[0010] The display feature of 3D Gaussian Splatting offers more possibilities for editable operations. For example, some researchers have introduced Gaussian semantic tracking to achieve controllable Gaussian editing. Based on a mesh-like approach, some researchers parameterize Gaussian components using the vertices of mesh surfaces, achieving real-time rendering of editable Gaussians. However, this approach only enables overall object editing and fails to simulate realistic physical mechanisms. Some researchers have combined the material point method to incorporate physical mechanisms into 3D Gaussians, enabling dynamic simulation and rendering of 3D objects with simple physical behaviors. Although these experiments have provided a physics-driven plant morphology simulation, they fail to consider non-rigid properties, such as tree branches, or the topological relationships between branches. These deficiencies not only result in low tree simulation accuracy but also lead to discontinuous branch bending during the physical simulation. To address this issue, some researchers have constructed Spring-Mass 3D Gaussians for elastic objects like plant branches, further improving the fidelity of non-rigid object simulation. Although this method has made some progress in non-rigid simulation, the elastic coefficient of the simulated object is single and the structural representation is relatively simple. It does not take into account the complex structural distribution constraints of plants and is not suitable for high-fidelity physical simulation of plants. Summary of the Invention

[0011] In order to overcome the shortcomings of the existing technology, the present invention provides a 3D Gaussian-based plant physical model modeling method, which provides a new method for high-precision reconstruction and physical simulation of plants through the combination of background segmentation, topological structure constraints and cantilever beam physical model; the background of plants is segmented by a large model generalization and fine-tuning method, which improves the accuracy of plant organ-level instance segmentation and reduces background interference and mis-segmentation problems; at the same time, combined with the Weibull sorting method and topological structure constraints, high-precision branches and trunks that are more in line with the geometric morphology of plants are achieved; finally, based on the cantilever beam model, a rotation and revolution mechanism of the Gaussian ellipsoid is proposed, which improves the authenticity of plant physical simulation and the accuracy of dynamic changes; through this method, not only the accuracy of plant reconstruction is improved, but also more realistic physical effects can be achieved in the process of simulating plant deformation.

[0012] The technical solution adopted by the present invention to solve its technical problem is:

[0013] A 3D Gaussian-based plant physical modeling method comprises the following steps:

[0014] Step 1: Use the camera to take photos around the target plant object, ensuring coverage of different angles and details;

[0015] Step 2: Input the photo set collected in step 1, reconstruct the sparse point cloud using the SfM and MVS-based methods, and train the 3D Gaussian Splatting model to obtain the Gaussian model and point cloud data;

[0016] Step 3: Extract the skeleton of the point cloud data in step 2, generate a connectivity graph based on Delaunay triangulation, and use the Dijkstra shortest path algorithm to extract the initial skeleton of the plant;

[0017] Step 4: Further optimize the rough plant skeleton obtained in step 3 by using spline interpolation to smooth the skeleton. At the same time, set a minimum branch length threshold and merge or delete branches that are too short to optimize the skeleton structure and make it more consistent with the actual structure of the plant.

[0018] Step 5: Based on the skeleton refinement in Step 4, add a physical mechanism to calculate the elastic modulus, density, and moment of inertia parameters based on the branch radius, length, and plant material properties. Add a cantilever beam model to the skeleton to describe the bending behavior of the branch under gravity or external forces. This allows for large deformation and nonlinear elasticity, thereby more realistically simulating the bending deformation of the plant.

[0019] Step 6: Using the physical skeleton in step 5, calculate the nearest neighbor relationship between each Gaussian point and the skeleton, and establish a mapping relationship between the Gaussian points and the skeleton nodes. When external force is applied, the skeleton is deformed and the positions of the Gaussian points are adjusted accordingly to achieve global deformation. At the same time, to enhance the realism of the deformation, the Gaussian ellipsoid performs local rotation to make the branches bend more naturally.

[0020] Furthermore, in step 1, the number of photos taken is between 50 and 150 to ensure the accuracy of subsequent reconstruction; for the group of photos taken, the photos are pre-processed by a segmentation method based on Mask R-CNN or SAM (Segment Anything Model) to extract pure plant objects, thereby optimizing the subsequent 3D reconstruction results.

[0021] The process of step 2 is as follows:

[0022] Step 2.1: Use Colmap to perform camera pose estimation and feature point matching to generate sparse point cloud data;

[0023] Step 2.2: Use the PatchMatch algorithm to perform dense point cloud matching to generate a high-density point cloud, providing high-quality data for subsequent Gaussian training;

[0024] Step 2.3: Initialize Gaussian points, including parameters such as position, normal vector, color, and transparency;

[0025] Step 2.4: Optimize Gaussian parameters through volume rendering to adapt to image inputs of different perspectives, and use gradient descent to optimize Gaussian point parameters to improve rendering quality;

[0026] Step 2.5: Complete the training and obtain the Gaussian model data.

[0027] In step 2.1, post-processing is performed to delete the outlying point clouds and segment the mask to constrain the point cloud distribution to improve the reconstruction quality.

[0028] The process of step 3 is as follows:

[0029] Step 3.1: Use DBScan method to remove isolated points and noise;

[0030] Step 3.2: Calculate the point cloud normal vector and use principal component analysis to align the point cloud direction;

[0031] Step 3.3: Use the Delaunay triangulation method to convert the point cloud into a connected graph so that all points in the point cloud have a topological relationship;

[0032] Step 3.4: Use Dijkstra's shortest path search algorithm to find the shortest path from the root to other points in the connected graph and construct the preliminary skeleton of the plant.

[0033] Optionally, in step 3.4, a KD-Tree is used during the neighborhood search process to accelerate the query and improve computational efficiency.

[0034] The process of step 4 is as follows:

[0035] Step 4.1: Use spline interpolation to smooth the image, eliminate noise and mutation points, and improve the continuity of the skeleton;

[0036] Step 4.2: Set a minimum branch length threshold and merge or delete branches that are shorter than the threshold to reduce redundant structures and improve the rationality of the skeleton.

[0037] Step 4.3: Calculate the branch radius by the average distance of the point cloud around the skeleton point to ensure that the skeleton can accurately describe the thickness variation of the plant branch;

[0038] Optionally, in step 4.3, a cylinder fitting method is used to improve the accuracy of radius calculation, and combined with curvature analysis to optimize the morphology of the skeleton.

[0039] The process of step 5 is as follows:

[0040] Step 5.1: Using the branch radius and length obtained in step 4, calculate the branch's elastic modulus, density, moment of inertia, and other parameters based on plant material properties and the biomechanical model;

[0041] Step 5.2: Use the cantilever beam model to describe the bending behavior of the branch under gravity or external force and simulate its stress state;

[0042] Step 5.3: Calculate the tempering stiffness segment by segment so that the skeleton can simulate the real plant bending and recovery characteristics.

[0043] Optionally, in step 5.2, a Cosserat Rod model is used to enable the skeleton to support large deformation and nonlinear elasticity, thereby improving the realism of the simulation;

[0044] Furthermore, in the step 5.2, finite element analysis is combined to further improve the simulation accuracy, accurately calculate the deformation of the plant under the action of external force, and improve the simulation accuracy.

[0045] The process of step 6 is as follows:

[0046] Step 6.1: Calculate the nearest neighbor relationship between each Gaussian point and the skeleton, and bind it to the corresponding skeleton node to establish the mapping relationship between the Gaussian point and the skeleton;

[0047] Step 6.2: Apply external force, including its magnitude, direction, and location, to calculate the rotation and deformation of the skeleton and simulate the plant's actual response to the external force.

[0048] Step 6.3: Use the cantilever beam model to simulate branch deformation and calculate the displacement and angle changes of the skeleton nodes;

[0049] Step 6.4: When the skeleton nodes rotate or move, the Gaussian points rotate and translate accordingly, so that the entire model presents a consistent deformation effect;

[0050] Step 6.5: The Gaussian ellipsoid can be locally rotated to simulate higher-quality deformation effects and improve detail expression.

[0051] The technical concept of the present invention is to perform background segmentation on plants through a method of generalization and fine-tuning of large models, and use the branch radius to constrain the Gaussian distribution, thereby improving the consistency of the reconstructed topological structure and the accuracy of the geometric shape. In the 3D reconstruction process of plants, multi-view images often contain complex backgrounds, such as soil, stones and other vegetation. These background interferences will significantly reduce the reconstruction accuracy. The premise of plant physical simulation is high-precision 3D Gaussian plant reconstruction, but the existing reconstruction method based on 3D Gaussian Splatting often ignores the topological structure and geometric shape unique to plant objects, resulting in precision problems such as surface ellipsoid protrusion and burrs in the reconstruction results, and the appearance of organ objects that do not conform to the plant geometry. Based on 3D Gaussian Splatting, combined with segmentation masks and prior knowledge of plants, a branch fitting with radius constrained Gaussian distribution is constructed. This makes the geometric shape of the constructed branch objects reasonable, and the surface Gaussian ellipsoid distribution is within a reasonable range. This modeling strategy can not only maintain the consistency of the plant topological structure, but also provide higher quality surface accuracy during the physical simulation process.

[0052] A physical simulation method for plant branch bending has been developed. Based on the coupling of a cantilever beam and a plant skeleton, this method ensures that branch deformation conforms to physical mechanisms. It also employs dual rotation of a Gaussian ellipsoid to ensure the authenticity of the simulation results. Skeleton constraints are also used to improve the stability and accuracy of the 3D Gaussian Splatting physical simulation. Existing editable 3D Gaussian Splatting methods primarily rely on subjective editing, most of which fail to consider the involvement of physical mechanisms. Furthermore, physical simulations for plant objects are lacking, and the non-rigid material properties of plants are not incorporated. This leads to problems such as exaggerated branch bending and unstable topological structures when simulating plant deformation under stress. To this end, the present invention proposes a plant physical deformation simulation method based on the coupling of a cantilever beam and a plant skeleton, aiming to more realistically simulate the dynamic deformation of plant branches under external forces. The present invention first utilizes the skeleton structure to perform hierarchical mapping of the Gaussian ellipsoid, ensuring that the branch structure maintains topological consistency during deformation. Then, the mechanical properties of the plant branch are calculated based on the cantilever beam model, and the bending parameters are transferred to the Gaussian ellipsoid, thereby achieving bending deformation that is more consistent with slender branches in the 3D Gaussian Splatting physical simulation.

[0053] The beneficial effects of the present invention are mainly manifested in: through the combination of background segmentation, topological structure constraints and cantilever beam physical models, a new method is provided for high-precision reconstruction and physical simulation of plants. The present invention performs background segmentation on plants through the method of generalization and fine-tuning of large models, reducing background interference and mis-segmentation problems. At the same time, combined with the Weibull sorting method and topological structure constraints, high-precision branch and trunk reconstruction that is more in line with the geometric morphology of plants is achieved. Finally, based on the cantilever beam model, a rotation and revolution mechanism of the Gaussian ellipsoid is proposed, which improves the authenticity of plant physical simulation and the accuracy of dynamic changes. Through this method, not only the accuracy of plant reconstruction is improved, but also more realistic physical effects can be achieved in the process of simulating plant deformation. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a principle block diagram of a 3D Gaussian-based plant physical modeling method. DETAILED DESCRIPTION

[0055] The present invention will be further described below with reference to the accompanying drawings.

[0056] Reference Figure 1 , a 3D Gaussian-based plant physical model modeling method, comprising the following steps:

[0057] Step 1: Use a camera, mobile phone, or drone to take photos around the target plant, ensuring coverage of different angles and details. The number of photos taken should be between 50-150 to ensure the accuracy of subsequent reconstruction.

[0058] Optionally, the group of photos taken in step 1 are pre-processed using a segmentation method based on Mask R-CNN or SAM (Segment Anything Model) to extract pure plant objects, thereby optimizing the subsequent 3D reconstruction results;

[0059] Step 2: Input the photos collected in step 1, reconstruct a sparse point cloud using a method based on SfM (Structure from Motion) and MVS (Multi-View Stereo), and train a 3D Gaussian Splatting model to obtain a Gaussian model and point cloud data;

[0060] In step 2, the purpose is to use the radiation field to perform multi-perspective synthesis of plant scenes captured by multiple photos or videos based on the 3D Gaussian Splatting method. The process is as follows:

[0061] Step 2.1: Use Colmap to perform camera pose estimation and feature point matching to generate sparse point cloud data;

[0062] Step 2.2: Use the PatchMatch algorithm to perform dense point cloud matching to generate a high-density point cloud, providing high-quality data for subsequent Gaussian training;

[0063] Step 2.3: Initialize Gaussian points, including parameters such as position, normal vector, color, and transparency;

[0064] Step 2.4: Optimize Gaussian parameters through volume rendering to adapt to image inputs of different perspectives, and use gradient descent to optimize Gaussian point parameters to improve rendering quality;

[0065] Step 2.5: Complete the training and obtain the Gaussian model data.

[0066] Optionally, in step 2.1, post-processing is performed to delete stray point clouds and segment the point cloud distribution using a mask to improve reconstruction quality.

[0067] Step 3: Extract the skeleton of the point cloud data in step 2, generate a connectivity graph based on Delaunay triangulation, and use the Dijkstra shortest path algorithm to extract the initial skeleton of the plant;

[0068] In step 3, the purpose is to extract the rough skeleton of the point cloud, and the process is as follows:

[0069] Step 3.1: Use the DBScan (Density-Based Spatial Clustering of Applications with Noise) method to remove isolated points and noise;

[0070] Step 3.2: Calculate the point cloud normal vector and use principal component analysis to align the point cloud direction;

[0071] Step 3.3: Use the Delaunay triangulation method to convert the point cloud into a connected graph so that all points in the point cloud have a topological relationship;

[0072] Step 3.4: Use Dijkstra's shortest path search algorithm to find the shortest path from the root to other points in the connected graph and construct the preliminary skeleton of the plant.

[0073] Optionally, in step 3.4, a KD-Tree is used during the neighborhood search process to accelerate the query and improve computational efficiency.

[0074] Step 4: Further optimize the rough plant skeleton obtained in step 3 by using spline interpolation to smooth the skeleton. At the same time, set a minimum branch length threshold and merge or delete branches that are too short to optimize the skeleton structure and make it more consistent with the actual structure of the plant.

[0075] In step 4, the purpose is to extract a fine plant skeleton including radius parameters, and the process is as follows:

[0076] Step 4.1: Use spline interpolation to smooth the image, eliminate noise and mutation points, and improve the continuity of the skeleton;

[0077] Step 4.2: Set a minimum branch length threshold and merge or delete branches that are shorter than the threshold to reduce redundant structures and improve the rationality of the skeleton.

[0078] Step 4.3: Calculate the branch radius by the average distance of the point cloud around the skeleton point to ensure that the skeleton can accurately describe the thickness variation of the plant branch;

[0079] Optionally, in step 4.3, a cylinder fitting method is used to improve the accuracy of radius calculation, and combined with curvature analysis to optimize the morphology of the skeleton.

[0080] Step 5: Building on the skeleton refinement from Step 4, add a physical mechanism to calculate parameters such as elastic modulus, density, and moment of inertia based on the branch radius, length, and plant material properties. Add a cantilever beam model to the skeleton to describe the branch's bending behavior under gravity or external forces. This allows for large deformation and nonlinear elasticity, thereby more realistically simulating plant bending deformation.

[0081] In step 5, the purpose is to add a physical model to the skeleton so that it can conform to the physical mechanism and bend. The process is as follows:

[0082] Step 5.1: Using the branch radius and length obtained in step 4, calculate the branch's elastic modulus, density, moment of inertia, and other parameters based on plant material properties and the biomechanical model;

[0083] Step 5.2: Use the cantilever beam model to describe the bending behavior of the branch under gravity or external force and simulate its stress state;

[0084] Step 5.3: Calculate the tempering stiffness segment by segment so that the skeleton can simulate the real plant bending and recovery characteristics.

[0085] Optionally, in step 5.2, a Cosserat Rod model is used to enable the skeleton to support large deformation and nonlinear elasticity, thereby improving the realism of the simulation;

[0086] Furthermore, in the step 5.2, finite element analysis is combined to further improve the simulation accuracy, accurately calculate the deformation of the plant under the action of external force, and improve the simulation accuracy.

[0087] Step 6: Using the physical skeleton in step 5, calculate the nearest neighbor relationship between each Gaussian point and the skeleton, and establish a mapping relationship between the Gaussian points and the skeleton nodes. When external force is applied, the skeleton is deformed and the positions of the Gaussian points are adjusted accordingly to achieve global deformation. At the same time, to enhance the realism of the deformation, the Gaussian ellipsoid performs local rotation to make the branches bend more naturally.

[0088] In step 6, the purpose is to use the skeleton to drive the point cloud so that the Gaussian model has a physical mechanism. The process is as follows:

[0089] Step 6.1: Calculate the nearest neighbor relationship between each Gaussian point and the skeleton, and bind it to the corresponding skeleton node to establish the mapping relationship between the Gaussian point and the skeleton;

[0090] Step 6.2: Apply external force, including its magnitude, direction, and location, to calculate the rotation and deformation of the skeleton and simulate the plant's actual response to the external force.

[0091] Step 6.3: Use the cantilever beam model to simulate branch deformation and calculate the displacement and angle changes of the skeleton nodes;

[0092] Step 6.4: When the skeleton nodes rotate or move, the Gaussian points rotate and translate accordingly, so that the entire model presents a consistent deformation effect;

[0093] Step 6.5: The Gaussian ellipsoid can be locally rotated to simulate higher-quality deformation effects and improve detail expression.

[0094] This embodiment proposes a plant structure reconstruction method based on 3D Gaussian Splatting, combining segmentation masks, geometric features, and topological information to achieve high-precision reconstruction. The segmentation results are used to extract plant regions, and the Gaussian distribution and covariance matrix are optimized to ensure that the reconstruction is more consistent with the biological characteristics of the plant, improving accuracy and enhancing the applicability of 3D Gaussian Splatting in plant modeling.

[0095] This embodiment proposes a physical deformation simulation method based on the coupling drive of a cantilever beam and a plant skeleton. By combining the physical properties of the cantilever beam with the skeleton topology, accurate branch bending simulation is achieved. In view of the problem that the current Gaussian physical simulation is difficult to meet the requirements of non-rigid deformation, a mechanical constraint model is constructed to make the branches show natural deformation that conforms to biological characteristics when subjected to force. In addition, this method guides the deformation through the skeleton, allowing users to select specific branches for bending control, thereby improving the operability and accuracy of the physical simulation and realistically reproducing the dynamic deformation of plants under external forces.

[0096] The embodiments of this specification are merely examples of implementations of the invention and are provided for illustrative purposes only. The scope of protection of the present invention should not be considered limited to the specific embodiments described in these embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by a person of ordinary skill in the art based on the invention.

Claims

1. A 3D Gaussian-based plant physical modeling method, characterized in that: The method comprises the following steps: Step 1: Use the camera to take photos around the target plant object, ensuring coverage of different angles and details; Step 2: Input the photo set collected in step 1, reconstruct the sparse point cloud using the SfM and MVS-based methods, and train the 3D Gaussian Splatting model to obtain the Gaussian model and point cloud data; Step 3: Extract the skeleton of the point cloud data in step 2, generate a connectivity graph based on Delaunay triangulation, and use the Dijkstra shortest path algorithm to extract the initial skeleton of the plant; Step 4: Further optimize the rough plant skeleton obtained in step 3 by using spline interpolation to smooth the skeleton. At the same time, set a minimum branch length threshold and merge or delete branches that are too short to optimize the skeleton structure and make it more consistent with the actual structure of the plant. Step 5: Based on the skeleton refinement in Step 4, add a physical mechanism to calculate the elastic modulus, density, and moment of inertia parameters based on the branch radius, length, and plant material properties. Add a cantilever beam model to the skeleton to describe the bending behavior of the branch under gravity or external forces. This allows for large deformation and nonlinear elasticity, thereby more realistically simulating the bending deformation of the plant. Step 6: Using the physical skeleton in step 5, calculate the nearest neighbor relationship between each Gaussian point and the skeleton, and establish a mapping relationship between the Gaussian points and the skeleton nodes. When external force is applied, the skeleton is deformed and the positions of the Gaussian points are adjusted accordingly to achieve global deformation. At the same time, to enhance the realism of the deformation, the Gaussian ellipsoid performs local rotation to make the branches bend more naturally.

2. A 3D Gaussian-based plant physical modeling method according to claim 1, characterized in that: In step 1, the number of photos taken is between 50 and 150 to ensure the accuracy of subsequent reconstruction; the group of photos taken is pre-processed using a segmentation method based on Mask R-CNN or SAM to extract pure plant objects, thereby optimizing the subsequent 3D reconstruction results.

3. A 3D Gaussian-based plant physical modeling method according to claim 1 or 2, characterized in that: The process of step 2 is as follows: Step 2.1: Use Colmap to perform camera pose estimation and feature point matching to generate sparse point cloud data; Step 2.2: Use the PatchMatch algorithm to perform dense point cloud matching to generate a high-density point cloud, providing high-quality data for subsequent Gaussian training; Step 2.3: Initialize Gaussian points, including parameters such as position, normal vector, color, and transparency; Step 2.4: Optimize Gaussian parameters through volume rendering to adapt to image inputs of different perspectives, and use gradient descent to optimize Gaussian point parameters to improve rendering quality; Step 2.5: Complete the training and obtain the Gaussian model data.

4. A 3D Gaussian-based plant physical modeling method according to claim 3, characterized in that: In step 2.1, post-processing is performed to delete the outlying point clouds and segment the mask to constrain the point cloud distribution to improve the reconstruction quality.

5. A 3D Gaussian-based plant physical modeling method according to claim 1 or 2, characterized in that: The process of step 3 is as follows: Step 3.1: Use DBScan method to remove isolated points and noise; Step 3.2: Calculate the point cloud normal vector and use principal component analysis to align the point cloud direction; Step 3.3: Use the Delaunay triangulation method to convert the point cloud into a connected graph so that all points in the point cloud have a topological relationship; Step 3.4: Use Dijkstra's shortest path search algorithm to find the shortest path from the root to other points in the connected graph and construct the preliminary skeleton of the plant.

6. A 3D Gaussian-based plant physical modeling method according to claim 5, characterized in that: In step 3.4, KD-Tree is used to accelerate the query and improve the computational efficiency during the neighborhood search process.

7. A 3D Gaussian-based plant physical modeling method according to claim 1 or 2, characterized in that: The process of step 4 is as follows: Step 4.1: Use spline interpolation to smooth the image, eliminate noise and mutation points, and improve the continuity of the skeleton; Step 4.2: Set a minimum branch length threshold and merge or delete branches that are shorter than the threshold to reduce redundant structures and improve the rationality of the skeleton. Step 4.3: Calculate the branch radius by the average distance of the point cloud around the skeleton point to ensure that the skeleton can accurately describe the thickness variation of the plant branches.

8. A 3D Gaussian-based plant physical modeling method according to claim 7, characterized in that: In step 4.3, a cylinder fitting method is used to improve the accuracy of radius calculation and combined with curvature analysis to optimize the morphology of the skeleton.

9. A 3D Gaussian-based plant physical modeling method according to claim 1 or 2, characterized in that: The process of step 5 is as follows: Step 5.1: Using the branch radius and length obtained in step 4, calculate the branch's elastic modulus, density, moment of inertia, and other parameters based on plant material properties and the biomechanical model; Step 5.2: Use the cantilever beam model to describe the bending behavior of the branch under gravity or external force and simulate its stress state; Step 5.3: Calculate the tempering stiffness segment by segment so that the skeleton can simulate the real plant bending and recovery characteristics.

10. A 3D Gaussian-based plant physical modeling method according to claim 1 or 2, characterized in that: The process of step 6 is as follows: Step 6.1: Calculate the nearest neighbor relationship between each Gaussian point and the skeleton, and bind it to the corresponding skeleton node to establish the mapping relationship between the Gaussian point and the skeleton; Step 6.2: Apply external force, including its magnitude, direction, and location, to calculate the rotation and deformation of the skeleton and simulate the plant's actual response to the external force. Step 6.3: Use the cantilever beam model to simulate branch deformation and calculate the displacement and angle changes of the skeleton nodes; Step 6.4: When the skeleton nodes rotate or move, the Gaussian points rotate and translate accordingly, so that the entire model presents a consistent deformation effect; Step 6.5: The Gaussian ellipsoid can be locally rotated to simulate higher-quality deformation effects and improve detail expression.

Images