Nursery stock three-dimensional reconstruction method and application

Abstract

The invention discloses a nursery stock three-dimensional reconstruction method and application, and the method comprises the following steps: building an image collection device; accurately segmenting the multi-view image sequence by adopting an SAM algorithm, and effectively removing background interference in the multi-view sequence; obtaining camera parameters by using an SFM algorithm, combining the sparse point cloud generated by the SFM algorithm and the estimated camera pose as input, and initializing Gaussian distribution; 3D Gaussian distribution is used as scene representation, and a nursery stock three-dimensional model is generated after high-fidelity reconstruction of a nursery stock scene is achieved through a 3DGS algorithm. The method has the beneficial effects that the data after SAM segmentation is used for reconstruction, the reconstruction time is shortened by 40.79%, the PSNR and SSIM mean values are respectively improved by 37.93% and 8.25%, and the LPIPS mean value is reduced by 93.06%; under the same number of iterations, compared with the NeRF model, the PSNR and SSIM mean values of the 3DGS model are respectively improved by 70.49% and 48.64%, and the LPIPS mean value is reduced by 93.55%; the nursery stock three-dimensional model generated by the 3DGS is uniform in point cloud distribution and complete in structure.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction of seedlings, and in particular to a three-dimensional reconstruction method and application of seedlings. Background Art

[0002] 3D reconstruction technology is an important digital tool to promote the development of precision agriculture. This technology can fully present the spatial structure and morphological characteristics of plants, thus laying the foundation for plant phenotypic analysis, yield prediction and breeding selection. There are many types of plants, and existing 3D reconstruction methods generally have problems such as high cost, complex process and insufficient precision. Therefore, efficient and low-cost plant 3D modeling is particularly important for realizing the refinement and intelligence of agricultural production. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a method for three-dimensional reconstruction of seedlings.

[0004] To solve the above technical problems, the present invention provides the following technical solutions: a method for three-dimensional reconstruction of seedlings, comprising the following steps: building an image acquisition device to obtain a multi-view image sequence of seedlings; using a SAM algorithm to accurately segment the multi-view image sequence to effectively remove background interference in the multi-view sequence; then using an SFM algorithm to obtain camera parameters, and combining the sparse point cloud generated by the SFM algorithm and the estimated camera pose as input to initialize the Gaussian distribution; using a 3D Gaussian distribution as a scene representation, and generating a three-dimensional model of the seedling after achieving high-fidelity reconstruction of the seedling scene through a 3DGS algorithm.

[0005] Preferably, the use of the SAM algorithm for precise segmentation includes the following steps: first, the seedling image to be segmented is taken as input, and its features are extracted through an image encoder; the position of the seedling is determined using point prompts and bounding box position information; the input information of the prompt encoder is processed through position encoding to generate corresponding prompt embeddings; by analyzing the relationship between the image features and the prompt embeddings, the mask decoder can identify the outline of the seedling, and the generated segmentation mask is superimposed on the original image to form a segmented image centered on the seedling.

[0006] Preferably, the initialization of Gaussian distribution includes the following steps: the SFM algorithm iteratively processes the disordered image set, uses the feature matching algorithm to extract key points from images taken from different perspectives and find the matching positions of the key points; estimates the position and direction of the camera when the image was taken by calculating the positions of the matching points in different perspectives; then uses the triangulation method to restore the points in the three-dimensional space and generate a sparse 3D point cloud; finally, the sparse point cloud is initialized as a three-dimensional Gaussian distribution defined by the mean, covariance matrix and opacity to perform scene optimization.

[0007] Preferably, during the training process, the 3DGS algorithm inputs an image sequence of a static scene, generates an initialized Gaussian distribution according to the SFM algorithm, optimizes the initialized Gaussian distribution, and adaptively controls its density to enhance the detail representation of the scene.

[0008] Preferably, the Gaussian transformation of the 3DGS algorithm comprises the following steps: using a Gaussian function to represent points in space, each point is represented as a Gaussian distribution, and the 3D Gaussian G is characterized by a center point μ and Σ, which is defined as follows at a given position x,

[0009]

[0010] Where μ is the center position of the 3D Gaussian distribution, and Σ is the covariance matrix, which defines the shape and size of the Gaussian. During the optimization process, Σ is decomposed to maintain its positive definiteness, as shown in the following formula:

[0011] Σ=RSS T R T

[0012] Where R represents the rotation matrix of the Gaussian ellipsoid, and S is the scaling matrix of each axis.

[0013] Preferably, the Gaussian transformation further includes the following steps: in the world coordinate system, a 3D Gaussian is defined by a three-dimensional covariance matrix Σ; a Jacobian matrix is ​​introduced to perform local linear approximation of the nonlinear transformation and converted into a covariance matrix Σ′ in the camera coordinate system, as shown in the following formula:

[0014] Σ'=JWΣW T J T

[0015] Where W is the transformation matrix from the world coordinate system to the camera coordinate system, and J is the Jacobian matrix of the affine approximation of the projective transformation.

[0016] Preferably, the image rendering of the 3DGS algorithm includes the following steps: each 3D Gaussian is described by opacity α and C defined by spherical harmonic function coefficients; the color C of a pixel is calculated by N ordered points in a certain radius range in the point cloud that can affect the pixel, as shown in the following formula:

[0017]

[0018] In the formula, α i Represents the opacity of the current point i, α j Represents the opacity of the point before i.

[0019] Preferably, the loss calculation of the 3DGS algorithm includes the following steps: after the image is synthesized, the loss is calculated according to the difference between the rendered image and the input real image; the loss function is composed of L1 and structural similarity loss D-SSIM indicators, as shown in the following formula:

[0020] L=(1-λ)L1+λL D-SSIM

[0021] Where λ is the weight factor, L1 loss is used to measure the absolute difference between pixel colors, and LD-SSIM loss is used to measure the similarity between pixel windows.

[0022] Preferably, the 3DGS algorithm also includes the following steps: the 3DGS algorithm adaptively changes the distribution of point clouds based on gradients to generate new Gaussians; in each training iteration, if the opacity is lower than a set threshold or there are points close to the camera, they will be deleted; if the gradient of back propagation from the rendering loss to the current Gaussian exceeds a certain threshold, it is considered insufficient to represent the corresponding area and will also be deleted; the number of 3D Gaussian distributions is controlled by periodic cloning and deletion, and adaptively converges to the optimal 3D Gaussian density distribution.

[0023] The present invention also provides an application of a seedling three-dimensional reconstruction method, including the above-mentioned seedling three-dimensional reconstruction method, which is applied to the reconstruction of a seedling three-dimensional model, and the generated seedling three-dimensional model is used for research and analysis of the seedling growth process.

[0024] The beneficial effects of the present invention are as follows: by using the data after SAM segmentation for reconstruction, the reconstruction time is shortened by 40.79%, the PSNR and SSIM means are increased by 37.93% and 8.25% respectively, and the LPIPS mean is reduced by 93.06%; under the same number of iterations, compared with the NeRF model, the PSNR and SSIM means of the 3DGS model are increased by 70.49% and 48.64% respectively, and the LPIPS mean is reduced by 93.55%; the point cloud of the seedling three-dimensional model generated by 3DGS is evenly distributed and has a complete structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the seedling data of the present invention;

[0026] Figure 2 is a schematic diagram of the image acquisition device of the present invention;

[0027] Figure 3 A schematic diagram of the image acquisition scheme of the present invention;

[0028] Figure 4 It is a schematic diagram of the algorithm flow of 3D reconstruction of seedlings of the present invention;

[0029] Figure 5 Schematic diagram of seedling image segmentation based on SAM according to the present invention;

[0030] Figure 6 is a schematic diagram of the image segmentation process of the present invention;

[0031] Figure 7 It is a schematic diagram of the 3DGS training process of the present invention;

[0032] Figure 8 It is a schematic diagram of the seedling image segmentation result of the present invention;

[0033] Fig. 9 It is a schematic diagram of the reconstruction results of the mountain banyan tree under different viewing angles according to the present invention;

[0034] Fig.10 A schematic diagram of the convergence of the training loss of the reconstruction according to the present invention;

[0035] Fig.11 A schematic diagram showing comparison of reconstruction results at different iteration times according to the present invention;

[0036] Fig.12 A schematic diagram showing a comparison of rendering results of different algorithms according to the present invention;

[0037] Fig.13 A schematic diagram showing comparison of point cloud reconstruction results according to the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention, not all of them.

[0039] Example 1

[0040] When reconstructing seedlings, there are problems such as high data collection cost, low reconstruction efficiency and insufficient accuracy. In this embodiment, a single seedling is used as the test object, a set of image acquisition devices suitable for small seedlings is designed and built, and a seedling three-dimensional reconstruction method based on 3DGS is proposed, which provides an effective solution for achieving low-cost and high-precision three-dimensional reconstruction of seedlings. The generated seedling three-dimensional model paves the way for calculating plant phenotypic characteristic parameters and analyzing the changes in phenotypic parameters during plant growth, providing data support and theoretical guidance for plant phenotypic analysis and precise intelligent management.

[0041] Specifically, the following steps are included:

[0042] S1: Build an image acquisition device to obtain multi-view image sequences of seedlings;

[0043] S2: Using the SAM algorithm to accurately segment the multi-view image sequence and effectively remove background interference in the multi-view sequence;

[0044] S3: Use the SFM algorithm to obtain camera parameters, and combine the sparse point cloud generated by the SFM algorithm and the estimated camera pose as input to initialize the Gaussian distribution;

[0045] S4: Using 3D Gaussian distribution as the scene representation, the 3DGS algorithm is used to achieve high-fidelity reconstruction of the seedling scene and generate a seedling three-dimensional model.

[0046] In this embodiment, a set of image acquisition devices suitable for small seedlings is first built, and a multi-view image sequence of the seedlings is obtained through a consumer-grade smartphone; secondly, the SAM (Segment Anything Model) algorithm is used to accurately segment the seedling images to effectively remove the interference of the background in the multi-view sequence; the sparse point cloud generated by the SFM (Structure From Motion) algorithm is initialized into a set of three-dimensional Gaussian distributions for scene optimization; finally, 3D Gaussian is used as the scene representation, and high-fidelity reconstruction of the seedling scene is achieved through 3DGS technology.

[0047] Example 2

[0048] This embodiment proposes an application of a seedling three-dimensional reconstruction method, which is actually applied to the reconstruction of a seedling three-dimensional model. The generated seedling three-dimensional model is used for research and analysis of the seedling growth process.

[0049] The details are as follows:

[0050] Reference Figure 1 , which is a schematic diagram of seedling data. This application takes seedlings with a height of less than 1m as the application object, and randomly selects 6 seedlings including Indian banyan, mandarin orange and perfume lemon for reconstruction experiments. Figure 1 From left to right in the middle are Indian banyan, mountain banyan, bergamot, lemongrass, mandarin orange and pomelo. According to the growth morphology, leaf shape and degree of occlusion between leaves, the seedlings are divided into three levels: simple, medium and complex, so as to more effectively evaluate the impact of seedlings of different complexity on the reconstruction effect. Indian banyan and mountain banyan seedlings have large leaves, and the leaves and stems are clearly visible, with almost no occlusion, and are classified as simple; bergamot and mandarin orange seedlings have small leaves, closely arranged, slightly occluded branches and leaves, and overlap each other, and are classified as medium; lemongrass and pomelo seedlings have dense leaves, chaotic distribution of stems and leaves, and staggered growth of branches, forming a highly occluded environment, and are classified as complex.

[0051] Reference Figure 2As shown in the figure, a stable seedling image acquisition device is proposed for the embodiment, which consists of a tripod, an electric turntable, a mobile device and a black background cloth. In order to reduce motion blur and increase the acquisition speed of multi-view images, a tripod is used to fix the mobile device, and the target seedlings are photographed by rotating the object with a turntable. The mobile device uses a consumer-grade smartphone (iPhone 13Pro) to collect image data with a resolution of 1080x1920 pixels at a rate of 60 frames per second. In order to avoid reflection interference and simplify subsequent reconstruction experiments, a uniform black background is used. During the data acquisition process, a total of 6 scenes were recorded, each containing a different seedling. Figure 2 In the figure, 1 is the seedling, 2 is the electric turntable, 3 is the mobile device, 4 is the black background cloth, and 5 is the tripod.

[0052] Reference Figure 3 , which is a schematic diagram of the image acquisition scheme of this embodiment. The data required for training is a multi-perspective image sequence of a given scene. In view of the fact that it takes a long time to obtain a large number of static images, multi-perspective images are obtained by collecting videos. Before video shooting, the seedlings are placed on a turntable, and after the turntable is started, the camera starts recording. Driven by the turntable, the seedlings rotate along the dotted path to ensure that 360° video is captured. Depending on the complexity of the seedlings, it is necessary to shoot from different heights and angles. 3-5 camera positions need to be collected for each seedling, and the recording time at each position is 60 seconds. During the shooting process, the seedlings are always located in the center of the camera's field of view to ensure that the acquired image is complete.

[0053] For the specific implementation of the data, image acquisition device and acquisition scheme in the actual application process, further reference is made to Figure 4 The diagram is a flow diagram of the 3D reconstruction algorithm for seedlings in this embodiment. Through the frame extraction operation, images are extracted from the video data at a specified frame rate, and these images show minimal scene movement and motion blur. Multi-view images are used as input, and 150 images are used for training for each seedling. The image background and irrelevant information are removed based on the SAM segmentation model to improve the efficiency of the entire reconstruction process and the accuracy of the final seedling model; the structure from motion (SFM) algorithm in COLMAP is used to obtain camera parameters, and the sparse point cloud generated by the algorithm and the estimated camera pose are combined as input to initialize the Gaussian distribution; the 3DGS model is trained through a series of 3D Gaussian parameter optimization steps to create a radiation field representation and effectively generate a 3D model of the seedling.

[0054] More specifically, the seedling 3D reconstruction method in this embodiment includes seedling image segmentation, sparse reconstruction, and 3DGS reconstruction algorithm, which are as follows:

[0055] (1) Seedling image segmentation

[0056] Reference Figure 5 The diagram is a diagram of seedling image segmentation based on SAM. In order to effectively remove the interference of the background in the multi-view sequence and avoid the presence of outliers and noise in the final generated seedling geometric model, this embodiment uses the SegmentAnything Model (SAM) image segmentation algorithm to process the seedling image. Compared with traditional image segmentation models U-Net, YOLOv8 and Mask R-CNN, which have problems such as high dependence on a large amount of annotated data in a specific field, limited generalization ability in new scenes, and low stability of segmentation results, SAM has zero-sample segmentation capabilities, does not require model training or annotated data, and can process objects that have never been seen before and quickly generate accurate segmentation masks only through the guidance of user prompts.

[0057] SAM generates a segmentation mask from an input image through an image encoder, a hint encoder, and a mask decoder. It consists of three main modules: Figure 5 As shown in the dashed box, the image encoder is based on the pre-trained Vision Transformer (ViT) model, which divides the image into blocks and extracts high-dimensional feature representations from the input image; the prompt encoder is responsible for processing the input prompt information, encoding the input sparse prompts (points, boxes, text) and dense prompts (masks), and generating corresponding feature vectors; the mask decoder is used for the final mask prediction.

[0058] Further references Figure 6 The schematic diagram shows the specific process of using SAM to achieve seedling image segmentation. Figure 6 From left to right in the figure are the original image, point prompts and box prompts, mask and background removal. The process is as follows:

[0059] Firstly, the seedling image to be segmented is taken as input and its features are extracted through the image encoder;

[0060] In the point prompt, the point prompt and the bounding box position information are used to determine the position of the seedling. The target box annotation range is the whole seedling. There are many feature points in the twigs, leaves and potted parts. Black points are used to mark the segmented objects of the model. Since the electric turntable and the background will interfere with the reconstruction, they are not suitable to be selected as feature points. White points are used to mark the background.

[0061] Secondly, the input information of the prompt encoder is processed through position encoding to generate the corresponding prompt embedding;

[0062] Finally, by analyzing the relationship between image features and cue embeddings, the mask decoder is able to identify the outline of the seedlings and superimpose the generated segmentation mask with the original image to form a segmented image centered on the seedlings.

[0063] (2) Sparse reconstruction

[0064] The Structure from Motion (SFM) algorithm was used to perform sparse reconstruction of seedlings. SFM is a 3D reconstruction technique based on image sequences. It can reconstruct 3D models from unordered image sets for 3D space visualization. During the image analysis process, the SFM algorithm iteratively processes unordered image sets and uses feature matching algorithms to extract key points from images taken from different perspectives and find the matching positions of these points. By calculating the positions of matching points in different perspectives, the position and direction of the camera when the image was taken (i.e., the camera pose) are estimated. The points in the 3D space are then restored using the triangulation method to generate a sparse 3D point cloud. Finally, the sparse point cloud is initialized as a 3D Gaussian distribution defined by the mean, covariance matrix, and opacity for scene optimization.

[0065] (3) 3DGS reconstruction algorithm

[0066] Reference Figure 7 The diagram shows the 3DGS training process. 3D Gaussian Splatting (3DGS) uses a trainable 3D Gaussian distribution as the basic representation of the scene for reconstruction and rendering of three-dimensional scenes. 3DGS uses an ellipsoid to construct the scene, which can achieve high-fidelity real-time rendering. In the 3DGS training process, by inputting a sequence of images of a static scene, the Gaussian distribution is initialized according to the sparse point cloud generated by SFM, and then this set of Gaussians is optimized, and its density is adaptively controlled to enhance the detailed representation of the scene. In the optimization process, a sorting algorithm is used to achieve fast and accurate backpropagation. The 3DGS training process mainly includes Gaussian transformation, image rendering, loss calculation, and backpropagation.

[0067] The details are as follows:

[0068] 3DGS uses a set of 3D Gaussian distributions to represent the geometry and appearance of the entire scene. These Gaussian distributions describe the scene through parameters such as position, shape, color, and transparency. Points in space are represented by Gaussian functions, and each point is represented by a Gaussian distribution. The 3D Gaussian G is characterized by a central point μ and Σ, and is defined at a given position x as:

[0069]

[0070] Where μ is the center position of the 3D Gaussian distribution, and Σ is the covariance matrix, which defines the shape and size of the Gaussian. It is physically meaningful only when Σ is semi-positive definite, which cannot always be guaranteed during the optimization process. To solve this problem, Σ is decomposed to maintain its positive definiteness during the optimization process, as shown in Equation 2:

[0071] Σ=RSS T RT (2)

[0072] Where R represents the rotation matrix of the Gaussian ellipsoid, and S is the scaling matrix of each axis.

[0073] In the world coordinate system, the 3D Gaussian is defined by a complete three-dimensional covariance matrix Σ. However, when the three-dimensional Gaussian distribution is projected onto the 2D imaging plane, the transformation from perspective projection to orthogonal projection is a nonlinear transformation. In order to prevent the introduction of non-affine transformations in the projection transformation, the Jacobian matrix is ​​introduced in this embodiment to perform local linear approximation of the nonlinear transformation and convert it into the covariance matrix Σ′ in the camera coordinates, as shown in formula (3):

[0074] Σ'=JWΣW T J T (3)

[0075] Where W is the transformation matrix from the world coordinate system to the camera coordinate system, and J is the Jacobian matrix of the affine approximation of the projective transformation.

[0076] In order to obtain the rendered image, in addition to equations (1) and (2) that define the basic properties of 3D Gaussians, color properties are also combined to achieve realistic rendering. Each Gaussian is described by opacity α and C defined by the coefficients of spherical harmonics. The color C of a pixel is calculated by N ordered points in the point cloud that can affect the pixel within a certain radius. The calculation method is shown in the following formula:

[0077]

[0078] In the formula, α i Represents the opacity of the current point i, α j Represents the opacity of the point before i.

[0079] After the image is synthesized, the loss is calculated based on the difference between the rendered image and the input real image. The loss function consists of the L1 and structural similarity loss D-SSIM indicators, which is as follows (4):

[0080] L=(1-λ)L1+λL D-SSIM (4)

[0081] Where λ is the weight factor, L1 loss is used to measure the absolute difference between pixel colors, and LD-SSIM loss is used to measure the similarity between pixel windows.

[0082] In order to accurately represent the geometry of the scene, 3DGS adaptively changes the distribution of the point cloud based on the gradient to generate a new Gaussian. Since the quality of the initial point cloud is not high, it needs to be continuously optimized. In each training iteration, if the opacity is lower than the set threshold or there are some points close to the camera, they will be deleted. If the gradient of the back propagation from the rendering loss to the current Gaussian exceeds a certain threshold, it is considered insufficient to represent the corresponding area. By periodically cloning and deleting to control the number of 3D Gaussian distributions, it adaptively converges to an optimal 3D Gaussian density distribution to maintain a balance between computational efficiency and scene details, thereby better representing the three-dimensional scene.

[0083] Example 3

[0084] In order to verify the results of the seedling three-dimensional reconstruction method in practical applications, this embodiment conducts comparative analysis through experiments, as follows:

[0085] (1) Test environment

[0086] All experiments in this embodiment are based on Ubuntu 22.04 operating system, Intel Core i7-12700KF processor and Nvidia GeForce RTX3090 graphics card to process deep learning tasks and graphics calculations, use PyTorch1.13.0 for model training and reasoning, and use CUDA 11.7 to accelerate the speed of neural network training and reasoning.

[0087] (2) Evaluation indicators

[0088] The Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS) are used to quantitatively evaluate the performance of the model.

[0089] PSNR measures the quality of the reconstructed image by calculating the mean squared error (MSE) between the original image and the reconstructed image. The calculation formula is:

[0090]

[0091] SSIM is an indicator that measures the similarity between two images, taking into account features such as brightness, contrast, and structure. Its calculation formula is:

[0092]

[0093] LPIPS, also known as "perceptual loss", is an image similarity measurement method based on deep learning. It measures the perceptual differences of images in the deep feature space. Its calculation formula is:

[0094]

[0095] The higher the PSNR and SSIM values ​​and the lower the LPIPS, the better the reconstruction quality. Otherwise, it means that the dataset is difficult to converge well.

[0096] (3) Impact of background removal on reconstruction quality

[0097] Reference Figure 8 The diagram shows the segmentation results of different varieties of seedlings. Figure 8 From top to bottom, the first row is the original image, the second row is the predicted mask after SAM processing, and the third row is the final segmentation result. From left to right, each column (i.e., AE column) is Indian banyan, alpine banyan, bergamot, mandarin orange, perfumed lemon and red honey pomelo.

[0098] In the segmentation of Indian banyan and mountain banyan data, SAM can accurately identify branches and leaves and extract their main features. The comparison between the original image and the mask image shows that the edges of the leaves are processed in detail and the segmentation results are highly consistent with the shape of the real seedlings. For medium-complexity bergamot and mandarin oranges, the algorithm also shows high accuracy in the segmentation of leaves and branches. For example, when dealing with the tiny gaps between mandarin orange leaves, the algorithm can effectively identify and avoid mis-segmentation due to complex structures. When dealing with perfumed lemons and red honey pomelo, due to their complex growth structures, the algorithm does not perform as well as simple and medium-complexity seedlings in dealing with overlaps and occlusions, but can still effectively extract the key parts of the seedlings.

[0099] A scene was selected from each seedling dataset of different grades, and the original image and the segmented image were reconstructed respectively. The impact of background removal by the segmentation algorithm on the model reconstruction quality and reconstruction time was evaluated by four indicators: PSNR, SSIM, LPIPS, and reconstruction time. The results are shown in Table 1.

[0100] Table 1: Comparison of reconstruction quality and reconstruction time before and after segmentation.

[0101]

[0102] As shown in Table 1, after applying the SAM model segmentation, the reconstruction quality and efficiency of the seedlings in different levels of scenes have been greatly improved, especially the reduction of perceptual loss, indicating that the generated image is clearer and the structure is better preserved, which is closer to the original image. The average PSNR and SSIM after segmentation are increased by 37.93% and 8.25% respectively, the average LPIPS is reduced by 93.06%, and the reconstruction time is shortened by 40.79%. The results show that the reconstruction of the segmented data is not only significantly better than the original data in visual quality, but also significantly improves the reconstruction efficiency.

[0103] The types of seedlings are complex, and the morphological characteristics and structures of different varieties of seedlings vary significantly. SAM's zero-sample segmentation capability can efficiently handle the segmentation tasks of various types of seedlings without special training, effectively avoiding the limitations and high costs brought by labeling and training data. The background-free data after segmentation has clear scene boundaries, ensuring that the various components of different seedlings, such as leaves and stems, can be clearly identified.

[0104] (4) Reconstruction time analysis

[0105] The reconstruction time varies with different data sets and is related to the number of iterations. In this example, the training time of the model after 30,000 iterations is taken as the final reconstruction time. Taking the alpine banyan seedling as an example, the reconstruction results and training loss convergence of 3DGS at different viewing angles and different numbers of iterations are shown in Figure 2. Fig. 9 and Fig.10 As shown, Fig. 9 The first line is the iteration under perspective 1, the second line is the iteration under perspective 1, and from left to right are 1000, 3000, 5000, 7000, and 30000 iterations respectively. Fig. 9 The middle line box indicates that during the reconstruction process, the missing details are gradually restored as the number of iterations increases.

[0106] Depend on Fig. 9 and Fig.10 It can be seen that in the early stage of 3DGS reconstruction, the training loss is significantly higher, and the details of the leaves at the top of the seedlings and the potted part are blurred. As the training loss decreases, the overall outline of the seedlings and the connection between the leaves and the stems gradually become clear. After 30,000 iterations, the training loss tends to be stable, the details of the leaves in the image, such as texture and gloss, are properly processed, the texture of the potted plants has been significantly improved, and the real scene has been highly restored. The results show that the 3D Gaussian attribute optimization method introduced in the model ensures that the image has higher quality and less distortion when it is generated.

[0107] The iterations were set to 7000 and 30000 to study the effect of the number of iterations on the reconstruction time and quality of 3DGS. The results are shown in Fig.11As shown in Table 2, Fig.11 The first row is 7000 iterations, and the second row is 30000 iterations. From left to right are the seedlings of Banyan Tree, Mandarin Orange, and Red Heart Pomelo.

[0108] Table 2: Comparison of results with different numbers of iterations.

[0109]

[0110] Depend on Fig.11 As shown in Table 2, increasing the number of iterations can improve the reconstruction quality, but it will also prolong the reconstruction time. For the simple seedling scene (ficus indica), 7000 iterations took 2.87 minutes, with a PSNR of 34.36dB, which can clearly capture the morphology and characteristics of the seedlings. The model converges quickly and the rendering effect is better. In the complex seedling scene (red honey pomelo), 7000 iterations took 4 minutes, with a PSNR of 28.47dB. Although 3DGS can reconstruct the overall morphology of the seedlings well, the potted part is still blurred. After 30,000 iterations, which took 14.58 minutes, the model has significantly improved in capturing details. It can not only more accurately restore the shape of the leaves and the details of the branches, but also the reconstruction effect is more refined.

[0111] (5) Comparative analysis of reconstruction quality of different models

[0112] To verify the performance of the 3DGS model, it is compared with NeRF and Instant-ngp. The NeRF method is used as a benchmark for comparison. After training reaches a certain level, images are randomly selected from the seedling dataset for testing, and the corresponding rendered images are generated. The rendering results of the three models on different types of seedling data are shown in Figure 2. Fig.12 As shown, Fig.12 Each column from left to right (i.e., column AE) is Indian banyan, mountain banyan, bergamot, mandarin orange, fragrant lemon and red honey pomelo, and each row from top to bottom is the original image, NeRF method, Instant-ngp method and 3DGS of this method.

[0113] Depend on Fig.12It can be seen that for the simple seedling dataset, 3DGS can clearly present the structure with relatively dispersed distribution of stems and leaves, and accurately restore the large and flat leaf features; NeRF performs the worst, unable to retain the texture and details of the leaves, and the generated image has large errors. For seedlings of medium complexity, both 3DGS and Instant-NGP can accurately reconstruct the spatial structure of the seedlings and maintain a high degree of structural consistency. However, when processing mandarin oranges, Instant-NGP performs poorly in areas with severe occlusion of leaves and stems. The rendered image has low color saturation, and the leaf texture is blurred and the rendering is distorted, resulting in missing image details. For seedlings with complex growth structures, NeRF is blurred in texture details, and the relevant information of the potted part of the red honey pomelo reconstructed using Instant-NGP is missing, affecting the overall quality of the reconstruction. In contrast, even if the stems and leaves are densely distributed and severely occluded, 3DGS can still generate high-quality reconstruction results.

[0114] The PSNR, SSIM and LPIPS values ​​of the three methods, 3DGS, NeRF and Instant-NGP, are calculated and the average value is taken as the final evaluation. The results are shown in Table 3.

[0115] Table 3: Comparison of reconstruction quality of different models.

[0116]

[0117] As shown in Table 3, from the perspective of the image rendering quality indicator PSNR, the rendering quality of NeRF on seedling datasets of different complexities is generally poor, with an average of 20.30dB. This is because NeRF only samples one ray for each pixel when rendering, and rendering a single ray requires hundreds of queries to the MLP, which cannot capture complex details in a short training time, resulting in blurred NeRF rendering effects. In contrast, on seedling datasets of simple and medium complexity, both 3DGS and Instant-NGP are able to generate high-quality images. However, in complex seedling datasets, 3DGS performs better than the Instant-ngp method. In summary, 3DGS presents higher PSNR and SSIM values ​​and lower LPIPS values ​​in the evaluation of seedling datasets of different complexities. Compared with the NeRF model, its PSNR and SSIM averages are increased by 70.49% and 48.64%, respectively, while the LPIPS average is reduced by 93.55%. Compared with the Instant-ngp model, its PSNR and SSIM mean values ​​are improved by 35.30% and 4.47% respectively, while the LPIPS mean value is reduced by 54.72%. Therefore, 3DGS can more accurately capture and express the complex structure of seedlings by introducing 3D Gaussian defined by mean, covariance matrix and opacity as the representation of seedling scenes, thereby achieving higher image quality and visual effects.

[0118] (6) Qualitative analysis of point cloud reconstruction

[0119] To further evaluate the advantages of the proposed method in reconstruction, the point clouds reconstructed by 3DGS and the traditional Colmap method on different seedling scenes were compared. The reconstructed point cloud results were qualitatively analyzed by noise level, detail recovery and geometric structure integrity. Fig.13 As shown, Fig.13 Each column from left to right (i.e., column AE) is respectively Banyan tree, Mountain Banyan tree, Citron tree, Mandarin orange, Lemon tree, and Pomelo tree, and each row from top to bottom is respectively the Colmap method and the present method 3DGS.

[0120] Depend on Fig.13It can be seen that the reconstruction effect of 3DGS in different seedling scenes is significantly better than that of the Colmap method. Although the seedling point cloud generated by Colmap has a high density, due to the complex texture and geometric details in the data set, the reconstruction result of the point cloud model is difficult to accurately capture the fine structural features of the seedling, resulting in blurred local details. The branch details are not as clear as 3DGS, which is difficult to meet the needs of high-precision analysis. At the same time, there are holes and discontinuities on the surface of some seedling pots. This is because the multi-view stereo reconstruction (MVS) algorithm used in Colmap has poor effect when processing reflective materials and low-texture areas, resulting in the loss of some details. In contrast, 3DGS can more accurately reconstruct the overall morphology of seedlings, and the generated point cloud is evenly distributed and richer in details. When dealing with complex seedling structures, 3DGS can more clearly show the outlines of leaves, branches and pots, avoiding the common holes and noise problems in the point cloud generated by Colmap. In short, 3DGS has obvious advantages in fine reconstruction and complex structure processing, and is more suitable for scenes that require high precision and detail integrity.

[0121] Due to the complex growth structure of seedlings in the natural environment and the serious occlusion between leaves, the application of traditional 3D reconstruction technology is limited, and accurate and efficient seedling reconstruction is still challenging. In view of the above problems, this embodiment proposes a seedling 3D reconstruction method based on 3D Gaussian Splatting, which has the following beneficial effects:

[0122] First, a set of image acquisition devices suitable for small seedlings was designed and built, which can complete the low-cost and efficient acquisition of seedling images using consumer-grade smartphones and electric turntables. Based on the SAM image segmentation algorithm, the seedling images were accurately segmented through point and box prompts, effectively removing the interference of the background in the multi-view sequence, and then a data set for seedling 3D reconstruction was constructed, covering 6 different types of seedlings and 900 segmented images; second, a seedling 3D reconstruction method based on 3DGS was proposed. By matching and fusing multi-view 2D segmented images, the sparse point cloud generated by the SFM algorithm was initialized as a Gaussian distribution, and 3D Gaussian was introduced as a radiation field representation; the model was rendered using a differentiable rendering method, so as to quickly generate a high-resolution seedling 3D model; third, compared with the use of original data, the reconstruction time was shortened by 40.79% by using the data after SAM segmentation, the PSNR and SSIM mean values ​​were increased by 37.93% and 8.25% respectively, and the LPIPS mean value was reduced by 93.06%. Under the same number of iterations, compared with the NeRF model, the PSNR and SSIM averages of the 3DGS model increased by 70.49% and 48.64% respectively, and the LPIPS average decreased by 93.55%. Compared with the traditional Colmap 3D reconstruction method, the 3D model point cloud of the seedlings generated by 3DGS is evenly distributed and has a complete structure. Therefore, the reconstruction method proposed in this embodiment can meet the requirements of low-cost and high-precision 3D reconstruction of seedlings in agricultural scenes.

[0123] As the morphology and structure of seedlings continue to change and their complexity increases, the performance of different models in visual quality indicators has shown a certain downward trend. This shows that a single technology or model cannot meet the needs of high-precision 3D reconstruction of seedlings in different scenarios. In the future, it is planned to combine different acquisition technologies and reconstruction methods to explore accurate and fast 3D reconstruction methods for seedlings in complex scenarios.

[0124] Through the description of the above implementation, it is easy for a person skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present invention.

[0125] It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the protection scope of the present invention. For ordinary technicians in this field, other different forms of changes or modifications can be made based on the above description and ideas. It is not necessary and impossible to list all the implementation methods here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the technical solution of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for three-dimensional reconstruction of seedlings, characterized in that: The following steps are included: Build an image acquisition device to obtain multi-view image sequences of seedlings; The SAM algorithm is used to accurately segment the multi-view image sequence to effectively remove background interference in the multi-view sequence; Then use the SFM algorithm to obtain camera parameters, and combine the sparse point cloud generated by the SFM algorithm and the estimated camera pose as input to initialize the Gaussian distribution; 3D Gaussian distribution is used as the scene representation, and the 3DGS algorithm is used to achieve high-fidelity reconstruction of the seedling scene and generate a three-dimensional model of the seedling.

2. The seedling three-dimensional reconstruction method according to claim 1, characterized in that: The precise segmentation using the SAM algorithm includes the following steps: Firstly, the seedling image to be segmented is taken as input, and its features are extracted through the image encoder; Use point hints and bounding box position information to determine the location of the seedlings; Process the input information of the prompt encoder through position encoding to generate the corresponding prompt embedding; By analyzing the relationship between image features and cue embeddings, the mask decoder is able to identify the outline of the seedlings and superimpose the generated segmentation mask with the original image to form a segmented image centered on the seedlings.

3. The seedling three-dimensional reconstruction method according to claim 1, characterized in that: The initialization of Gaussian distribution includes the following steps: The SFM algorithm iteratively processes an unordered image set, extracts key points from images taken from different perspectives using a feature matching algorithm, and finds matching positions of the key points; By calculating the positions of matching points in different perspectives, the position and orientation of the camera when the image was taken are estimated; Then use the triangulation method to recover the points in the three-dimensional space and generate a sparse 3D point cloud; Finally, the sparse point cloud is initialized as a three-dimensional Gaussian distribution defined by mean, covariance matrix and opacity for scene optimization.

4. The seedling three-dimensional reconstruction method according to claim 1, characterized in that: During the training process, the 3DGS algorithm inputs an image sequence of a static scene, generates an initial Gaussian distribution according to the SFM algorithm, optimizes the initial Gaussian distribution, and adaptively controls its density to enhance the detail representation of the scene.

5. The seedling three-dimensional reconstruction method according to claim 1, characterized in that: The Gaussian transformation of the 3DGS algorithm includes the following steps: Use Gaussian function to represent points in space. Each point is represented as Gaussian distribution. 3D Gaussian G is characterized by a center point μ and Σ. At a given position x, it is defined as follows: Where μ is the center position of the 3D Gaussian distribution, and Σ is the covariance matrix, which defines the shape and size of the Gaussian. During the optimization process, Σ is decomposed to maintain its positive definiteness, as shown in the following formula: S = RSS T R T Where R represents the rotation matrix of the Gaussian ellipsoid, and S is the scaling matrix of each axis.

6. The method for three-dimensional reconstruction of seedlings according to claim 5, characterized in that: The Gaussian transformation further comprises the following steps: In the world coordinate system, the 3D Gaussian is defined by the 3D covariance matrix Σ; The Jacobian matrix is ​​introduced to perform local linear approximation of the nonlinear transformation and convert it into the covariance matrix Σ′ in the camera coordinates, as shown below: Σ'=JWΣW T J T Where W is the transformation matrix from the world coordinate system to the camera coordinate system, and J is the Jacobian matrix of the affine approximation of the projective transformation.

7. The seedling three-dimensional reconstruction method according to claim 1, characterized in that: The image rendering of the 3DGS algorithm includes the following steps: Each 3D Gaussian is described by opacity α and C defined by the coefficients of spherical harmonics; The color C of a pixel is calculated by N ordered points in a certain radius range in the point cloud that can affect the pixel, as shown in the following formula: In the formula, α i Represents the opacity of the current point i, α j Represents the opacity of the point before i.

8. The seedling three-dimensional reconstruction method according to claim 1, characterized in that: The loss calculation of the 3DGS algorithm includes the following steps: After the image is synthesized, the loss is calculated based on the difference between the rendered image and the input real image; The loss function is composed of L1 and structural similarity loss D-SSIM indicators, as shown below: L=(1-λ)L1+λL D-SSIM Where λ is the weight factor, L1 loss is used to measure the absolute difference between pixel colors, and LD-SSIM loss is used to measure the similarity between pixel windows.

9. The seedling three-dimensional reconstruction method according to claim 1, characterized in that: The 3DGS algorithm also includes the following steps: The 3DGS algorithm adaptively changes the distribution of point clouds based on gradients to generate new Gaussians; In each training iteration, if the opacity is lower than the set threshold or there are points close to the camera, they will be deleted; If the back-propagated gradient from the rendering loss to the current Gaussian exceeds a certain threshold, it is considered insufficient to represent the corresponding area and will also be deleted; The number of 3D Gaussian distributions is controlled by periodically cloning and deleting, and adaptively converges to the optimal 3D Gaussian density distribution.

10. An application of a seedling three-dimensional reconstruction method, characterized in that: The method comprises the three-dimensional reconstruction method of seedlings as described in claims 1-9, which is applied to the reconstruction of the three-dimensional model of seedlings, and the generated three-dimensional model of seedlings is used for the research and analysis of the growth process of seedlings.

Images