Plant three-dimensional reconstruction method and system based on image segmentation and projection guidance

Abstract

The invention provides a plant three-dimensional reconstruction method and system based on image segmentation and projection guidance, and is applied to the technical field of plant three-dimensional reconstruction, and the method comprises the steps: obtaining a multi-view image sequence of a target plant, the multi-view image sequence being plant images of the target plant collected from a plurality of angles; inputting the multi-view image sequence into a target detection model to obtain a plant positioning image set and a plant positioning frame output by the target detection model; taking the plant positioning frame as prompt input, and inputting the plant positioning image set into a preset image segmentation model to obtain a plant mask of the target plant output by the image segmentation model; performing mask processing on the multi-view image sequence based on a plant mask to obtain a pure multi-view image; and performing three-dimensional reconstruction based on the pure multi-view image through a 3D Gaussian scattering framework to obtain an initial three-dimensional plant model corresponding to the target plant. According to the invention, the interpretability and the three-dimensional contour definition of the plant structure can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of plant three-dimensional reconstruction, and in particular to a plant three-dimensional reconstruction method and system based on image segmentation and projection guidance. BACKGROUND

[0002] In modern plant phenotype research, three-dimensional reconstruction not only helps to improve the measurement accuracy of plant structure parameters such as plant height, leaf posture, and canopy volume, but also supports growth modeling across time series and stress adaptability analysis, and plays an irreplaceable role in intelligent breeding and agricultural robot navigation.

[0003] However, despite the continuous progress of related algorithms and perception hardware in recent years, plant three-dimensional reconstruction still faces a series of challenges in practical applications. On the one hand, complex background interference seriously affects the reconstruction accuracy. In the field or natural environment, the background often contains weeds, supports, soil, and other structures, with blurred boundaries with the plant target, making it difficult to extract the foreground. On the other hand, the plant structure itself has high complexity and non-rigid characteristics, such as slender branches and leaves, gaps, overlaps, and self-occlusion, which easily lead to problems such as occlusion loss and geometric degradation in the three-dimensional reconstruction process.

[0004] Therefore, the plant three-dimensional reconstruction method in the related art has the technical problem of low accuracy. SUMMARY

[0005] The present application provides a plant three-dimensional reconstruction method and system based on image segmentation and projection guidance to solve the problem of low accuracy in the prior art and improve the explainability of plant structure and the clarity of three-dimensional contour.

[0006] The present application provides a plant three-dimensional reconstruction method based on image segmentation and projection guidance, comprising the following steps. A multi-view image sequence of a target plant is obtained, wherein the multi-view image sequence is a plant image of the target plant collected from multiple angles; the multi-view image sequence is input into a target detection model to obtain a plant positioning image set and a plant positioning frame output by the target detection model; the plant positioning frame is input as a prompt, and the plant positioning image set is input into a preset image segmentation model to obtain a plant mask of the target plant output by the image segmentation model; the multi-view image sequence is masked based on the plant mask to obtain a pure multi-view image including only the target plant; and a three-dimensional reconstruction is performed based on the pure multi-view image through a 3D Gaussian scattering framework to obtain an initial three-dimensional plant model corresponding to the target plant.

[0007] The plant three-dimensional reconstruction method based on image segmentation and projection guidance provided by the application comprises the following steps: obtaining a plurality of multi-view images of a target plant; inputting the multi-view images into a target detection model to obtain a plant positioning image set and a plant positioning frame output by the target detection model; obtaining a plant class corresponding to the target plant as a specific prompt word; inputting the specific prompt word and the multi-view image sequence into the pre-trained target detection model to obtain a plant positioning image set and a plant positioning frame output by the target detection model corresponding to the target plant; obtaining an initial three-dimensional plant model corresponding to the target plant through a 3D Gaussian scattering framework based on the pure multi-view images; performing two-dimensional projection on each three-dimensional Gaussian point of the initial three-dimensional plant model respectively to obtain a two-dimensional Gaussian elliptical region corresponding to each three-dimensional Gaussian point; determining a position score of each elliptical boundary point of the two-dimensional Gaussian elliptical region, and screening each three-dimensional Gaussian point of the initial three-dimensional plant model based on the position score to obtain an optimized three-dimensional Gaussian point set; and performing three-dimensional reconstruction on the optimized three-dimensional Gaussian point set through the 3D Gaussian scattering framework to obtain a target three-dimensional plant model corresponding to the target plant.

[0008] The plant three-dimensional reconstruction method based on image segmentation and projection guidance provided by the application comprises the following steps: obtaining a plurality of multi-view images of a target plant; inputting the multi-view images into a target detection model to obtain a plant positioning image set and a plant positioning frame output by the target detection model; obtaining a plant class corresponding to the target plant as a specific prompt word; inputting the specific prompt word and the multi-view image sequence into the pre-trained target detection model to obtain a plant positioning image set and a plant positioning frame output by the target detection model corresponding to the target plant; obtaining an initial three-dimensional plant model corresponding to the target plant through a 3D Gaussian scattering framework based on the pure multi-view images; performing two-dimensional projection on each three-dimensional Gaussian point of the initial three-dimensional plant model respectively to obtain a two-dimensional Gaussian elliptical region corresponding to each three-dimensional Gaussian point; determining a position score of each elliptical boundary point of the two-dimensional Gaussian elliptical region, and screening each three-dimensional Gaussian point of the initial three-dimensional plant model based on the position score to obtain an optimized three-dimensional Gaussian point set; and performing three-dimensional reconstruction on the optimized three-dimensional Gaussian point set through the 3D Gaussian scattering framework to obtain a target three-dimensional plant model corresponding to the target plant.

[0009] The plant three-dimensional reconstruction method based on image segmentation and projection guidance provided by the application comprises the following steps: obtaining a plurality of multi-view images of a target plant; inputting the multi-view images into a target detection model to obtain a plant positioning image set and a plant positioning frame output by the target detection model; obtaining a plant class corresponding to the target plant as a specific prompt word; inputting the specific prompt word and the multi-view image sequence into the pre-trained target detection model to obtain a plant positioning image set and a plant positioning frame output by the target detection model corresponding to the target plant; obtaining an initial three-dimensional plant model corresponding to the target plant through a 3D Gaussian scattering framework based on the pure multi-view images; performing two-dimensional projection on each three-dimensional Gaussian point of the initial three-dimensional plant model respectively to obtain a two-dimensional Gaussian elliptical region corresponding to each three-dimensional Gaussian point; determining a position score of each elliptical boundary point of the two-dimensional Gaussian elliptical region, and screening each three-dimensional Gaussian point of the initial three-dimensional plant model based on the position score to obtain an optimized three-dimensional Gaussian point set; and performing three-dimensional reconstruction on the optimized three-dimensional Gaussian point set through the 3D Gaussian scattering framework to obtain a target three-dimensional plant model corresponding to the target plant.

[0010] According to the plant three-dimensional reconstruction method based on image segmentation and projection guidance provided in the application, before the multi-view image sequence is input into a target detection model to obtain a plant positioning image set and a plant positioning frame output by the target detection model, the method further comprises: obtaining a plant image sample data set, wherein the plant image sample data set is pre-labeled with a plant positioning frame of the target plant; and based on the plant image sample data set, performing target detection training on a preset target detection model to obtain a trained target detection model.

[0011] According to the plant three-dimensional reconstruction method based on image segmentation and projection guidance provided in the application, the target detection model is a YOLOv11 model, and the image segmentation model is a FastSAM model.

[0012] The application further provides a plant three-dimensional reconstruction system based on image segmentation and projection guidance, comprising the following modules: an image acquisition module, configured to acquire a multi-view image sequence of a target plant, wherein the multi-view image sequence is a plant image of the target plant collected from multiple angles; a target detection module, configured to input the multi-view image sequence into a target detection model to obtain a plant positioning image set and a plant positioning frame output by the target detection model; an image segmentation module, configured to input the plant positioning frame as a prompt and input the plant positioning image set into a preset image segmentation model to obtain a plant mask of the target plant output by the image segmentation model; a mask processing module, configured to perform mask processing on the multi-view image sequence based on the plant mask to obtain a pure multi-view image including only the target plant; and a three-dimensional reconstruction module, configured to perform three-dimensional reconstruction based on the pure multi-view image through a 3D Gaussian scattering framework to obtain an initial three-dimensional plant model corresponding to the target plant.

[0013] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the plant three-dimensional reconstruction method based on image segmentation and projection guidance according to any of the above when executing the program.

[0014] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the plant three-dimensional reconstruction method based on image segmentation and projection guidance according to any of the above.

[0015] The application further provides a computer program product comprising a computer program, wherein the computer program is executable on a processor to implement the plant three-dimensional reconstruction method based on image segmentation and projection guidance according to any of the above.

[0016] The plant three-dimensional reconstruction method and system based on image segmentation and projection guidance provided by the application, firstly, the multi-view image sequence provides basic visual data covering all angles of the target plant for reconstruction, ensuring the integrity of spatial information; secondly, the target detection model quickly and accurately identifies and locates the plant main body in the image, generates a positioning box to exclude a large amount of irrelevant background interference; then, the image segmentation model based on the positioning box prompt further separates the plant and the background, and generates a high-precision plant mask; subsequently, the mask processing uses the mask to filter the original image to obtain a pure multi-view image containing only the target plant, which significantly eliminates the negative impact of background noise on reconstruction; finally, the 3D Gaussian scattering framework performs three-dimensional reconstruction based on the pure image input to generate an initial plant model with complete structure, clear details and reduced distortion. The efficient and accurate plant three-dimensional reconstruction is realized. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description one by one. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0018] Figure 1 is a flowchart of the plant three-dimensional reconstruction method based on image segmentation and projection guidance provided by the application.

[0019] Figure 2 is a flowchart of the plant fast and high-precision 3DGS reconstruction method based on segmentation and 3D-2D projection guidance provided by the application.

[0020] Figure 3 is a plant foreground extraction flowchart based on pre-trained YOLOv11 provided by the application.

[0021] Figure 4 is a plant foreground extraction flowchart of YOLOv11 trained by a custom dataset provided by the application.

[0022] Figure 5 is a module schematic diagram of the plant three-dimensional reconstruction system based on image segmentation and projection guidance provided by the application.

[0023] Figure 6 is a schematic diagram of the physical structure of the electronic device provided by the application. DETAILED DESCRIPTION

[0024] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in conjunction with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor belong to the scope of protection of the present application.

[0025] In recent years, plant three-dimensional reconstruction, as an important supporting technology for agricultural intelligent perception and plant phenotype analysis, is gradually changing the traditional agricultural observation mode relying on manual experience and two-dimensional images. By digitizing plant structures in high-precision three-dimensional form, researchers can more comprehensively capture the morphological characteristics, dynamic growth process and environmental response behavior of crops, providing key data support for crop breeding, pest control and precision agriculture decision-making. In modern plant phenotype research, three-dimensional reconstruction not only helps to improve the measurement accuracy of plant structure parameters such as plant height, leaf posture, and canopy volume, but also supports growth modeling across time series and stress adaptability analysis, and plays an irreplaceable role in intelligent breeding and agricultural robot navigation. However, despite the continuous progress of related algorithms and perception hardware in recent years, plant three-dimensional reconstruction still faces a series of challenges in practical applications. First, complex background interference seriously affects the reconstruction accuracy. In the field or natural environment, the background often contains weeds, supports, soil and other structures, with blurred boundaries with the plant target, making it difficult to extract the foreground. Second, plant structures themselves have high complexity and non-rigid characteristics, such as slender branches and leaves, gaps, overlaps and self-occlusion, which easily lead to problems such as occlusion loss and geometric degradation in the three-dimensional reconstruction process. In addition, the mainstream methods often face the problem of low reconstruction efficiency and long optimization time, which limits their application in large-scale germplasm phenotype measurement or real-time monitoring scenarios.

[0026] In view of the above challenges, existing research has attempted to introduce deep learning models to enhance the extraction ability of plant structure features. For example, the PointNet series network learns local and global morphological information by directly processing point cloud data, and the multi-view fusion network combines multiple angle images to improve the geometric consistency and integrity of the reconstruction. These methods generally ignore the important role of plant foreground and background separation in three-dimensional modeling, and are prone to problems such as foreground and background overlap and blurred geometric boundaries in natural scenes, which seriously affect the structure fidelity of the model.

[0027] More importantly, the current mainstream methods lack a mechanism for identifying and removing redundant structures, resulting in a large amount of invalid information when generating high-density point clouds or explicit representations, significantly increasing the computational cost of subsequent optimization processes. At the same time, these methods generally rely on iterative optimization or volume rendering processes, resulting in long overall reconstruction times, especially in high-resolution modeling or large-scale plant phenotype collection tasks, which often take several hours or even longer to complete a complete reconstruction, making it difficult to meet the real-time or high-throughput requirements. These problems are particularly pronounced in actual agricultural application scenarios, and an efficient reconstruction strategy that balances modeling accuracy, speed, and system robustness is urgently needed to break through.

[0028] In recent years, 3D Gaussian Splatting (3DGS) has received widespread attention in order to improve reconstruction efficiency and controllability. Unlike traditional voxel-based or neural implicit field-based reconstruction methods, the core idea of 3DGS is to initialize a sparse point cloud as a set of Gaussian distributions with attributes such as position, shape, color, and opacity, and to perform differentiable rendering through an efficient rasterization mechanism. This explicit representation based on "point-Gaussian-projection" not only has excellent rendering quality and geometric fidelity, but also significantly improves the training and inference speed of the model, making it particularly suitable for efficient multi-view synthesis and editable reconstruction.

[0029] Unlike methods such as Neural Radiance Fields (NeRF) that require a large number of iterations to learn the volume distribution, the Gaussian representation of 3DGS has natural spatial distribution and can be trimmed, supporting direct removal and editing of redundant structures, making it easy to remove background interference and focus on target plant structures. In addition, its rendering process uses a forward projection and depth sorting fusion strategy to maintain high image quality while significantly reducing computational resource consumption, achieving a good balance between image clarity, structural consistency, and system efficiency, and demonstrating its potential advantages in plant three-dimensional modeling. Although 3DGS provides a new approach to fast and high-quality three-dimensional reconstruction, there are still some problems in complex plant scenes, such as unclear foreground and background separation leading to the retention of redundant point Gaussians, cluttered initial point clouds affecting rendering speed, and unclear structure boundaries. Therefore, we introduce two key enhancement modules based on the 3DGS reconstruction framework to address these issues specifically designed for plant three-dimensional reconstruction scenarios.

[0030] First, we combine the positioning ability of YOLOv11 and the strong generalization segmentation ability of FastSAM to build a plant foreground extraction module, which removes background interference before reconstruction and only retains clean plant area images as input, significantly reducing the number of redundant Gaussians.

[0031] Secondly, a 3D-2D projection guidance optimization mechanism is proposed. In the reconstruction process, a three-dimensional Gaussian is projected onto a two-dimensional image plane. Whether the projected elliptical points belong to the target plant is determined. The inconsistent points are removed to further optimize the structure boundary. The two modules work together to not only effectively compress the reconstruction time, but also significantly improve the interpretability of the plant structure and the clarity of the three-dimensional contour.

[0032] Alternatively, the plant three-dimensional reconstruction method based on image segmentation and projection guidance according to the embodiments of the application can be executed by a server, or by a terminal device, or by both the server and the terminal device. Taking the plant three-dimensional reconstruction method based on image segmentation and projection guidance in the embodiments executed by the server as an example.

[0033] Figure 1 The flowchart of the plant three-dimensional reconstruction method based on image segmentation and projection guidance provided by the application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the method comprises the following steps.

[0034] Step 101, acquiring a multi-view image sequence of a target plant, wherein the multi-view image sequence is a plant image of the target plant collected from multiple angles.

[0035] In the embodiments of the application, the target plant is photographed by a preset full-automatic rotating device for 360 degrees, and multi-view image sequences are collected from high, medium and low angles to ensure complete coverage of the spatial structure and detail information of the plant, thereby providing sufficient visual input for subsequent three-dimensional reconstruction.

[0036] Step 102, inputting the multi-view image sequence into a target detection model to obtain a plant positioning image set and a plant positioning frame output by the target detection model.

[0037] In the embodiments of the application, the target detection model (for example, a YOLOv11 model) is used to detect the target plant in each image in the multi-view image sequence to obtain the plant positioning frame of the target plant.

[0038] This stage can adopt two methods: one is to directly use the trained YOLOv11 model to input the target plant category (such as “plant”, “leafy plant” and the like) for rapid detection; the other is to collect custom plant image data, construct and fine-tune the YOLOv11 model suitable for specific plant varieties to improve the recognition accuracy.

[0039] Step 103, inputting the plant positioning frame as a prompt, and inputting the plant positioning image set into a preset image segmentation model to obtain a plant mask of the target plant output by the image segmentation model.

[0040] In the embodiment of the present application, the plant position frame output by the target detection model is taken as prior prompt information (i.e. prompt input), which provides regional guidance for subsequent fine segmentation, ensures that only the plant region is processed at high resolution, and reduces invalid calculation.

[0041] The plant position frame output by the target detection model is subjected to high-resolution mask generation using an image segmentation model (e.g. FastSAM model).

[0042] The image segmentation model combines the prompt input to perform fine segmentation in the plant boundary region, effectively preserving complex structures such as leaf contours and flower details, achieving high-quality foreground extraction, and obtaining a plant mask of the target plant.

[0043] In step 104, the mask processing is performed on the multi-view image sequence based on the plant mask to obtain a pure multi-view image containing only the target plant.

[0044] In the embodiment of the present application, the plant mask output by the image segmentation model is applied to the original image (i.e. multi-view image sequence) to remove the background and retain a pure multi-view image containing only the plant, which is used as a key input for 3D reconstruction.

[0045] In step 105, the 3D Gaussian scattering framework is used to perform three-dimensional reconstruction based on the pure multi-view image to obtain an initial three-dimensional plant model corresponding to the target plant.

[0046] In the embodiment of the present application, the pure multi-view image obtained after mask processing is input into the 3D Gaussian scattering framework, which uses the explicit Gaussian representation and forward projection mechanism of the 3D Gaussian scattering framework to achieve fast and high-fidelity plant three-dimensional point cloud fitting and model reconstruction, thereby obtaining an initial three-dimensional plant model corresponding to the target plant.

[0047] Through the embodiment of the present application, firstly, the multi-view image sequence provides basic visual data covering all angles of the target plant for reconstruction, ensuring the integrity of spatial information; secondly, the target detection model quickly and accurately identifies and locates the plant subject in the image, generates a positioning frame to exclude a large amount of irrelevant background interference; then, the image segmentation model based on the positioning frame prompt further separates the plant from the background to generate a high-precision plant mask; subsequently, the mask processing uses the mask to filter the original image to obtain a pure multi-view image containing only the target plant, which significantly reduces the negative impact of background noise on reconstruction; finally, the 3D Gaussian scattering framework performs three-dimensional reconstruction based on the pure image input to generate an initial plant model with complete structure, clear details, and reduced distortion. Efficient and accurate plant three-dimensional reconstruction is achieved.

[0048] According to the plant three-dimensional reconstruction method based on image segmentation and projection guidance provided by the application, after three-dimensional reconstruction is performed on the pure multi-view image based on the 3D Gaussian scattering framework to obtain an initial three-dimensional plant model corresponding to the target plant, the method further comprises the following steps of: Projecting each three-dimensional Gaussian point of the initial three-dimensional plant model based on the initial three-dimensional plant model to obtain a two-dimensional Gaussian elliptical region corresponding to each three-dimensional Gaussian point; Determining the position score of each elliptical boundary point of the two-dimensional Gaussian elliptical region, and screening each three-dimensional Gaussian point of the initial three-dimensional plant model based on the position score to obtain an optimized three-dimensional Gaussian point set; Performing three-dimensional reconstruction on the optimized three-dimensional Gaussian point set based on the 3D Gaussian scattering framework to obtain a target three-dimensional plant model corresponding to the target plant.

[0049] In the embodiment of the application, a 3D-2D projection consistency optimization mechanism is introduced based on the preliminary three-dimensional model. Each three-dimensional Gaussian point of the initial three-dimensional plant model is projected to a two-dimensional image plane, and the consistency score is calculated through the elliptical points in the two-dimensional image plane, and the redundant points are screened, optimized or removed according to the consistency score, so as to realize the structure boundary optimization and model precision improvement.

[0050] The high-precision plant three-dimensional model (i.e. the target three-dimensional plant model) after optimization is output, which has the real geometric shape and detail restoration capability and can be applied to digital agriculture, plant phenotype analysis, virtual simulation and various scenes.

[0051] In the embodiment of the application, after the pure multi-view image is obtained, 3DGS reconstruction is performed on the pure multi-view image. In the 3D plant reconstruction process, if only YOLOv11 and FastSAM are applied without subsequent optimization, the overlap of the foreground plant and the background structure in the two-dimensional image will cause the segmentation boundary to be blurred, resulting in a rough reconstruction result. In addition, the complex structure of the plant, such as the leaf shielding, the hollow and the overlapping area, often causes the problem of Gaussian distribution mismatch, thereby reducing the geometric fidelity and visual quality of the three-dimensional model.

[0052] To overcome the above problems, the application provides an optimization mechanism based on 3D-2D projection guidance: each three-dimensional Gaussian point is projected onto the corresponding two-dimensional image plane to form a two-dimensional Gaussian elliptical region, and an elliptical boundary point voting mechanism is adopted, that is, whether each pixel point along the elliptical edge falls into the plant area defined by the segmentation mask is judged one by one.

[0053] By counting the proportion of the points in the foreground area in these boundary sampling points, the consistency score between the Gaussian point and the real segmentation can be calculated. The Gaussian points with high scores are retained or enhanced in weight, and the Gaussian points with low scores are removed or reduced in weight.

[0054] The mechanism not only improves the discrimination ability of the edge area of the structure, but also significantly reduces the redundant background points introduced by the segmentation error, thereby enhancing the boundary clarity and structural consistency of the final model. In addition, the mechanism is inserted in a parallel processable manner in the 3DGS process, without significantly increasing the system computing burden, and is suitable for large-scale plant reconstruction or embedded deployment scenarios.

[0055] Reference Figure 2 , Figure 2 is a plant fast high-precision 3DGS reconstruction method based on segmentation and 3D-2D projection guidance provided by the application. Specifically, it includes: a full-automatic device shoots 360-degree images of plants, trains / tunes YOLOv11 to locate plants in the images, plant target detection positioning information, FastSAM generates high-resolution mask according to input prompts, pure plant multi-view images, 3DGS plant three-dimensional reconstruction, 3D-2D projection guidance optimization mechanism (based on 3D-2D projection consistency score), and plant three-dimensional reconstruction model.

[0056] Through the embodiments of the application, a plant fast high-precision 3DGS reconstruction method based on segmentation and 3D-2D projection guidance is provided, aiming to solve the problems of insufficient segmentation accuracy, multiple background interference and low reconstruction efficiency in existing plant three-dimensional reconstruction methods. By combining the fast target detection of YOLOv11 and the local partition fine boundary extraction of FastSAM, and introducing the 3D-2D projection guidance consistency optimization mechanism, the accuracy of plant target extraction is effectively improved, error transmission is suppressed, and the structural consistency of the reconstructed point cloud is enhanced, thereby realizing the dual improvement of speed and quality of the three-dimensional reconstruction result, and finally obtaining a plant three-dimensional model with complete structure, clear edge and visual reality.

[0057] According to the plant three-dimensional reconstruction method based on image segmentation and projection guidance provided by the application, the position score of each elliptical boundary point of the two-dimensional Gaussian ellipse region is determined, and each three-dimensional Gaussian point of the initial three-dimensional plant model is screened based on the position score to obtain an optimized three-dimensional Gaussian point set, including: Uniformly sampling on the boundary of the two-dimensional Gaussian ellipse region to obtain a plurality of elliptical boundary points; Based on the segmentation mask function, the position score of each elliptical boundary point in the plurality of elliptical boundary points is determined; Based on the position score of each elliptical boundary point, a consistency score of the three-dimensional Gaussian point is determined; Retain the three-dimensional Gaussian points with consistency scores greater than or equal to a preset score threshold, and eliminate or weight down the three-dimensional Gaussian points with consistency scores lower than the preset score threshold to obtain the optimized three-dimensional Gaussian point set.

[0058] In the embodiments of the application, let a three-dimensional Gaussian point The projection on the two-dimensional image plane is a two-dimensional Gaussian ellipse region A set of points are uniformly sampled on the boundary of the two-dimensional Gaussian ellipse region wherein, denotes a set of points obtained by uniformly sampling the boundary of the projection ellipse of the th three-dimensional Gaussian point on the image plane, denotes the th sampling point, denotes that the sampling points are all located on the edge of the two-dimensional Gaussian ellipse region (not the entire ellipse region, but the boundary of the ellipse).

[0059] Let the segmentation mask function denote whether the ellipse boundary point belongs to the plant region (1 represents foreground, 0 represents background), then the projection consistency score of the three-dimensional Gaussian point is defined as: wherein, denotes the brightness or color value of the mask image at the ellipse boundary point . Generally speaking: there is color (such as the plant region): then it is determined as foreground (the value is 1). There is no color (black, i.e. RGB = 0,0,0): then it is determined as background (the value is 0).

[0060] A set of points are sampled on the boundary of the two-dimensional ellipse region obtained by projecting the three-dimensional Gaussian point, and whether these points are in the plant region is determined one by one. The proportion of how many points belong to the foreground is counted, and the reliability score of the Gaussian point is obtained. If most of the points are in the plant region, it means that the three-dimensional Gaussian point belongs to the plant, then tends to 1. If most of the points are in the background region, it means that the three-dimensional Gaussian point is a false detection or background noise, then tends to 0. A score threshold

[0061] is set. When the consistency score is lower than the score threshold, it is considered that the three-dimensional Gaussian point is projected in the background region or the segmentation error region, and should be removed or given a low weight: ​​​After the consistency optimization of 3D-2D projection is completed, the optimized three-dimensional Gaussian point set is updated and reconstructed. By eliminating redundant, boundary ambiguous or poor projection consistent Gaussian points, and recalculating the spatial distribution and attribute parameters thereof, the structural compactness and geometric accuracy of the model are further improved.

[0062] Subsequently, the optimized Gaussian points are input into the 3D Gaussian Splatting framework to perform the final three-dimensional reconstruction process.

[0063] Through the embodiment of the present application, the high-definition fitting and texture restoration of the plant structure are realized by using the forward rendering characteristics of the Gaussian voxel in the image sequence, and a high-quality three-dimensional plant model with real shape, continuous edge and rich details is generated, thereby laying a solid foundation for subsequent plant shape analysis and virtual display.

[0064] According to the plant three-dimensional reconstruction method based on image segmentation and projection guidance provided by the present application, the target detection model is a pre-trained target detection model, and the multi-view image sequence is input into the target detection model to obtain the plant positioning image set and the plant positioning frame output by the target detection model, which comprises: The plant category corresponding to the target plant is obtained as a specific prompt word; The specific prompt word and the multi-view image sequence are input into the pre-trained target detection model to obtain the plant positioning image set and the plant positioning frame corresponding to the target plant output by the target detection model.

[0065] In the embodiment of the present application, in order to obtain pure plant foreground multi-view images (i.e. pure multi-view images), a YOLOv11 model and a FastSAM model are used to obtain image data containing only plants.

[0066] The method can achieve two different requirements. The first method is to directly use the pre-trained YOLOv11 model for plant detection.

[0067] Reference Figure 3 , Figure 3 is the plant foreground extraction flowchart based on the pre-trained YOLOv11 provided by the present application. It includes: a pre-trained YOLOv11 model, a plant data set I specific prompt word P, traversing the use cases in I, for each i k According to the specific prompt word, the pre-trained YOLOv11 model is executed, the plant image set D containing the positioning and the corresponding positioning frame B are output, the use cases in D are traversed by the FastSAM plant foreground fast segmentation model, for each d k According to the positioning frame b k The FastSAM model is executed, and the foreground plant image set C={c1, c2, c3…} is output.

[0068] The method uses a pre-trained YOLOv11 model to detect plants in each image, and quickly locates plants by inputting specific plant categories (such as "plant" or "leafy plant"). The model outputs the location box information of the plant, and performs subsequent processing according to the detected area, using the FastSAM model to segment the plant foreground and remove background noise. The advantage of this method is simple and fast operation, suitable for use in scenarios where plant species are included in the pre-trained model, but may have accuracy problems in specific plant species and complex backgrounds. Specifically, the following steps are included.

[0069] Step 1: initial state, input pre-trained YOLOv11 target detection model, plant image set I = {i1, i2, i3,...}, and specific plant prompt word set P = {"plant", "leafy plant"} in the model; Step 2: select an image i from the plant image set k , input it into the YOLOv11 target detection model together with the specific plant prompt word P, and obtain the output image set D = {d1, d2, d3,...} and the corresponding image set location box B = {b1, b2, b3,...}; Step 3: input the obtained output image set D and location box B data into the FastSAM plant foreground rapid segmentation model, and iterate through each data in D; Step 4: for each d k , according to the location box b k , execute the FastSAM model to obtain the foreground plant image set C = {c1, c2, c3,...} containing only the target plant.

[0070] Through the embodiment of the invention, the plant category corresponding to the target plant is obtained as a specific prompt word, and it is input into the target detection model together with the multi-view image sequence. The model can focus more on feature extraction and recognition of the target plant. Different plants have different shapes and characteristics. The introduction of specific prompt words enables the model to be optimized for specific plant categories, improving the recognition ability in complex backgrounds or similar plant interference, thereby enhancing the adaptability and robustness of the model.

[0071] According to the plant three-dimensional reconstruction method based on image segmentation and projection guidance provided by the invention, before inputting the multi-view image sequence into the target detection model to obtain the plant location image set and plant location box output by the target detection model, the above method further includes: obtain a plant image sample data set, wherein the plant image sample data set is pre-labeled with a plant location box of a target plant; Based on the plant image sample dataset, the target detection model is trained to obtain the trained target detection model.

[0072] Since there is only a specific trained model for detection, the freedom is not very high, so the second method is to fine-tune the YOLOv11 model to detect specific plant varieties.

[0073] First, manually collect custom image datasets containing target plant species, which include images from multiple angles and different lighting conditions and as much data as possible. At the same time, manually annotate the dataset, and send the annotated dataset and positioning data to YOLOv11. Train the YOLOv11 model on this custom dataset, optimize the model to detect target plants more accurately. The trained model can perform more accurate plant positioning on custom plants and perform better in complex environments. After using the FastSAM model to segment the image, only the plant part is retained for subsequent 3D reconstruction. Although this method can provide higher detection accuracy, it requires additional training time and dataset preparation.

[0074] Reference Figure 4 , Figure 4 is the YOLOv11 plant foreground extraction flowchart trained by the custom dataset provided by the application, which includes: YOLOv11 model, plant dataset I, manual annotation and screening of each plant image set i k , YOLOv11 model target detection training on custom dataset, trained custom dataset YOLOv11 model, input custom plant image set S that needs to be segmented, traverse use cases in S, output plant image set D including positioning and corresponding positioning box B, traverse use cases in D by FastSAM plant foreground fast segmentation model, for each d k According to the positioning box b k , execute the FastSAM model, and output the foreground plant image set C={c1, c2, c3…}.

[0075] The specific steps are as follows: Step 1: initial state, ready YOLOv11 target detection model, custom plant dataset manually photographed I ; Step 2: manually annotate and screen each image i k in the plant image set, do not annotate pictures of target plants that are not clear, and at the same time ensure that the plants in the custom dataset are multi-angle and multi-scene, and contain a variety of conditions to ensure that the results of subsequent YOLOv11 target detection training are widely applicable; Step 3: Train the labeled dataset as the input of the YOLOv11 target detection model to obtain a model of the self-defined dataset, and test the trained model with the input data as the test set to determine whether the model can be used; Step 4: Additional plant image experiments are performed on images different from the training dataset, and a self-defined plant image set that needs to be segmented S Input the trained YOLOv11 target detection model; Step 5: Obtain a plant image set containing positioning through the self-defined pre-trained YOLOv11 target detection model D ={d1,d2,d3....} and the corresponding positioning frame B ={b1,b2,b3....}; Step 6: Obtain the output image set D and the positioning frame B Data as the input of the FastSAM plant foreground rapid segmentation model, and each data in D is traversed. Step 7: For each d k According to the positioning frame b k , the FastSAM model is executed to obtain a foreground plant image set C={c1,c2,c3....} containing only target plants.

[0076] Through the embodiment of the present application, by using a plant image sample dataset pre-labeled with target plant positioning frames for training, the model can learn the feature representation of the target plant under different perspectives, lighting, background conditions, etc., so as to more accurately identify and position the target plant in subsequent applications. Avoids the positioning deviation caused by the use of a general target detection model due to the diverse plant morphology and complex background, and improves the positioning accuracy and reliability.

[0077] According to the plant three-dimensional reconstruction method based on image segmentation and projection guidance provided by the present application, the target detection model is a YOLOv11 model, and the image segmentation model is a FastSAM model.

[0078] The present application adopts YOLOv11 for rapid coarse segmentation, realizes rapid positioning of the plant region, combines FastSAM to perform local fine segmentation in the target region, effectively reduces the global processing burden of the whole image, significantly improves the segmentation efficiency, reduces the calculation resource consumption, and is suitable for edge devices or large-scale plant batch reconstruction scenes.

[0079] The two-stage structure combining coarse segmentation and fine segmentation can more accurately extract complex plant edges, leaf occlusion areas, hole structures and other details, has higher segmentation precision and lower error, and provides clearer and more complete foreground information for three-dimensional reconstruction.

[0080] The application filters, enhances or removes the Gaussian points by projecting the three-dimensional Gaussian point cloud to a two-dimensional image space and calculating a projection consistency score according to the share of the elliptical boundary points, so as to realize dynamic correction and geometric consistency optimization of the segmentation result in the three-dimensional reconstruction process, effectively suppress the background noise points introduced by the segmentation error, and improve the structure fidelity of the model reconstruction.

[0081] Thanks to the accuracy of the foreground extraction and the introduction of the point cloud optimization mechanism, the application greatly shortens the reconstruction time without reducing the model reconstruction accuracy. The application can improve the 3DGS reconstruction speed by 20% to 30%, the PSNR of the reconstructed image changes less than ±1dB, and the reconstruction accuracy remains basically unchanged.

[0082] The application uses the fast rough segmentation module YOLOv11 to preliminarily extract the target plant region, and combines the lightweight fine segmentation model FastSAM to perform high-precision boundary segmentation in the target region, which significantly improves the segmentation efficiency and accuracy and reduces the overall system inference burden, providing high-quality input for subsequent three-dimensional reconstruction.

[0083] The application introduces a 3D-2D projection guidance mechanism to optimize the segmentation quality through geometric projection feedback in the three-dimensional Gaussian reconstruction process. The current three-dimensional Gaussian point cloud is projected onto the corresponding image plane to form a two-dimensional Gaussian elliptical region, and the projection consistency score is calculated according to the proportion of the elliptical boundary points inside and outside the object. According to the score, each Gaussian point is filtered, enhanced or removed, effectively suppressing the background interference points caused by segmentation errors, improving the semantic consistency and boundary accuracy of the Gaussian point distribution. This mechanism realizes dynamic supervision and structural optimization of the two-dimensional mask quality in the three-dimensional reconstruction process, thereby further improving the structure fidelity and visual clarity of the final model without significantly increasing the computational burden.

[0084] The rough segmentation module based on YOLOv11 is used to preliminarily extract the plant region in the input multi-view image, and the FastSAM model is used to perform fine boundary segmentation in the rough extraction region to obtain a high-quality foreground mask for input selection in subsequent three-dimensional reconstruction. The mask is generated by the FastSAM model and only performs local fine segmentation in the target region provided by YOLOv11, thereby reducing irrelevant area calculation, improving segmentation efficiency and reducing overall inference burden.

[0085] In the 3D Gaussian Splatting reconstruction process, a 3D-2D projection guiding mechanism is introduced, a two-dimensional Gaussian ellipse region is formed by projecting the current three-dimensional Gaussian point cloud to the image plane, and the two-dimensional ellipse of each Gaussian point is calculated and analyzed in turn. The projection consistency score of each Gaussian point is calculated, and the Gaussian points are screened, enhanced or removed according to the score, so as to dynamically optimize the Gaussian point cloud structure, suppress background interference and enhance boundary fidelity.

[0086] Finally, a plant three-dimensional model with accurate structure and clear boundary is generated, the joint improvement of segmentation optimization and structure reconstruction is realized, the calculation resource consumption is significantly reduced, and the reconstruction speed and accuracy are improved.

[0087] The plant three-dimensional reconstruction system based on image segmentation and projection guidance provided by the present application is described below. The plant three-dimensional reconstruction system based on image segmentation and projection guidance described below can be referred to each other corresponding to the plant three-dimensional reconstruction method based on image segmentation and projection guidance described above.

[0088] Reference Figure 5 , Figure 5 is a module schematic diagram of the plant three-dimensional reconstruction system based on image segmentation and projection guidance provided by the present application.

[0089] The image acquisition module 501 is configured to acquire a multi-view image sequence of a target plant, wherein the multi-view image sequence is a plant image of the target plant collected from multiple angles; The target detection module 502 is configured to input the multi-view image sequence into a target detection model to obtain a plant positioning image set and a plant positioning frame output by the target detection model; The image segmentation module 503 is configured to input the plant positioning frame as a prompt and input the plant positioning image set into a preset image segmentation model to obtain a plant mask of the target plant output by the image segmentation model; The mask processing module 504 is configured to perform mask processing on the multi-view image sequence based on the plant mask to obtain a pure multi-view image including only the target plant; The three-dimensional reconstruction module 505 is configured to perform three-dimensional reconstruction based on the pure multi-view image through a 3D Gaussian scattering framework to obtain an initial three-dimensional plant model corresponding to the target plant.

[0090] Specifically, the plant three-dimensional reconstruction system based on image segmentation and projection guidance provided by the present application can realize all method steps realized by the plant three-dimensional reconstruction method based on image segmentation and projection guidance, and can achieve the same technical effects. The same parts and beneficial effects in this embodiment and the method embodiment will not be described in detail.

[0091] Figure 6This is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a plant 3D reconstruction method based on image segmentation and projection guidance. This method includes: acquiring a multi-view image sequence of the target plant, wherein the multi-view image sequence is plant images of the target plant acquired from multiple angles; inputting the multi-view image sequence into a target detection model to obtain a plant localization image set and plant localization bounding boxes output by the target detection model; using the plant localization bounding boxes as prompt input and inputting the plant localization image set into a preset image segmentation model to obtain a plant mask of the target plant output by the image segmentation model; performing masking processing on the multi-view image sequence based on the plant mask to obtain a pure multi-view image containing only the target plant; and performing 3D reconstruction based on the pure multi-view image using a 3D Gaussian scattering framework to obtain an initial 3D plant model corresponding to the target plant.

[0092] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0093] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the plant three-dimensional reconstruction method based on image segmentation and projection guidance provided by the above-mentioned methods, which comprises: acquiring a multi-view image sequence of a target plant, wherein the multi-view image sequence is a plant image of the target plant collected from multiple angles; inputting the multi-view image sequence into a target detection model to obtain a plant positioning image set and a plant positioning frame output by the target detection model; inputting the plant positioning frame as a prompt and inputting the plant positioning image set into a preset image segmentation model to obtain a plant mask of the target plant output by the image segmentation model; performing mask processing on the multi-view image sequence based on the plant mask to obtain a pure multi-view image including only the target plant; and performing three-dimensional reconstruction based on the pure multi-view image through a 3D Gaussian scattering framework to obtain an initial three-dimensional plant model corresponding to the target plant.

[0094] In another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, which is executed by a processor to implement the plant three-dimensional reconstruction method based on image segmentation and projection guidance provided by the above-mentioned methods, which comprises: acquiring a multi-view image sequence of a target plant, wherein the multi-view image sequence is a plant image of the target plant collected from multiple angles; inputting the multi-view image sequence into a target detection model to obtain a plant positioning image set and a plant positioning frame output by the target detection model; inputting the plant positioning frame as a prompt and inputting the plant positioning image set into a preset image segmentation model to obtain a plant mask of the target plant output by the image segmentation model; performing mask processing on the multi-view image sequence based on the plant mask to obtain a pure multi-view image including only the target plant; and performing three-dimensional reconstruction based on the pure multi-view image through a 3D Gaussian scattering framework to obtain an initial three-dimensional plant model corresponding to the target plant.

[0095] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0096] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0097] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for three-dimensional reconstruction of plants based on image segmentation and projection guidance, characterized in that, The method comprises: acquiring a multi-view image sequence of a target plant, wherein the multi-view image sequence is a plant image of the target plant collected from multiple angles; inputting the multi-view image sequence into a target detection model to obtain a plant positioning image set and a plant positioning frame output by the target detection model; inputting the plant positioning frame as a prompt and inputting the plant positioning image set into a preset image segmentation model to obtain a plant mask of the target plant output by the image segmentation model; masking the multi-view image sequence based on the plant mask to obtain a pure multi-view image including only the target plant; performing three-dimensional reconstruction based on the pure multi-view image through a 3D Gaussian scattering framework to obtain an initial three-dimensional plant model corresponding to the target plant.

2. The method for 3D reconstruction of plants based on image segmentation and projection guidance according to claim 1, characterized in that, After the three-dimensional reconstruction based on the pure multi-view image through the 3D Gaussian scattering framework to obtain the initial three-dimensional plant model corresponding to the target plant, the method further comprises: performing two-dimensional projection on each three-dimensional Gaussian point of the initial three-dimensional plant model respectively to obtain a two-dimensional Gaussian elliptical region corresponding to each three-dimensional Gaussian point; determining a position score of each elliptical boundary point of the two-dimensional Gaussian elliptical region, and screening each three-dimensional Gaussian point of the initial three-dimensional plant model based on the position score to obtain an optimized three-dimensional Gaussian point set; performing three-dimensional reconstruction based on the optimized three-dimensional Gaussian point set through the 3D Gaussian scattering framework to obtain a target three-dimensional plant model corresponding to the target plant.

3. The method for 3D reconstruction of plants based on image segmentation and projection guidance according to claim 2, characterized in that, The determination of the position score of each elliptical boundary point of the two-dimensional Gaussian elliptical region and the screening of each three-dimensional Gaussian point of the initial three-dimensional plant model based on the position score to obtain the optimized three-dimensional Gaussian point set comprises: uniformly sampling on the boundary of the two-dimensional Gaussian elliptical region to obtain a plurality of elliptical boundary points; determining a position score of each elliptical boundary point in the plurality of elliptical boundary points based on a segmentation mask function; determining a consistency score of the three-dimensional Gaussian point based on the position score of each elliptical boundary point; retaining the three-dimensional Gaussian point with the consistency score greater than or equal to a preset score threshold, and eliminating or down-weighting the three-dimensional Gaussian point with the consistency score lower than the preset score threshold to obtain the optimized three-dimensional Gaussian point set.

4. The method for 3D reconstruction of plants based on image segmentation and projection guidance according to claim 1, characterized in that, The target detection model is a pre-trained target detection model, and the inputting of the multi-view image sequence into the target detection model to obtain the plant positioning image set and the plant positioning frame output by the target detection model comprises: acquiring a plant category corresponding to the target plant as a specific prompt word; inputting the specific prompt word and the multi-view image sequence into the pre-trained target detection model to obtain the plant positioning image set and the plant positioning frame corresponding to the target plant output by the target detection model.

5. The method for 3D reconstruction of plants based on image segmentation and projection guidance according to claim 1, characterized in that, Before the inputting of the multi-view image sequence into the target detection model to obtain the plant positioning image set and the plant positioning frame output by the target detection model, the method further comprises: acquire a plant image sample dataset, wherein the plant image sample dataset is pre-labeled with a plant positioning frame of the target plant; perform target detection training on a preset target detection model based on the plant image sample dataset to obtain a trained target detection model.

6. The method for 3D reconstruction of plants based on image segmentation and projection guidance according to claim 1, characterized in that, The target detection model is a YOLOv11 model, and the image segmentation model is a FastSAM model.

7. A plant three-dimensional reconstruction system based on image segmentation and projection guidance, characterized in that, The method comprises the following steps: an image acquisition module configured to acquire a multi-view image sequence of a target plant, wherein the multi-view image sequence is a plant image of the target plant collected from multiple angles; a target detection module configured to input the multi-view image sequence into a target detection model to obtain a plant positioning image set and a plant positioning frame output by the target detection model; an image segmentation module configured to input the plant positioning frame as a prompt and input the plant positioning image set into a preset image segmentation model to obtain a plant mask of the target plant output by the image segmentation model; a mask processing module configured to perform mask processing on the multi-view image sequence based on the plant mask to obtain a pure multi-view image including only the target plant; a three-dimensional reconstruction module configured to perform three-dimensional reconstruction based on the pure multi-view image through a 3D Gaussian scattering framework to obtain an initial three-dimensional plant model corresponding to the target plant.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the plant three-dimensional reconstruction method based on image segmentation and projection guidance according to any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the plant three-dimensional reconstruction method based on image segmentation and projection guidance according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the plant three-dimensional reconstruction method based on image segmentation and projection guidance according to any one of claims 1 to 6.

Images