A method, equipment, medium, and product for crop plant architecture reconstruction and segmentation.

Abstract

This application discloses a method, device, medium, and product for crop plant reconstruction and segmentation, relating to the field of crop reconstruction and organ segmentation. The method includes: generating automated foreground masks for each viewpoint based on multi-view two-dimensional images of the target crop using morphological operations and color consistency detection methods; constructing an initial sparse three-dimensional point cloud based on the automated foreground mask using feature detection algorithms and motion recovery algorithms; iteratively training and optimizing the initial three-dimensional Gaussian representation formed from the initial sparse three-dimensional point cloud using the automated foreground mask to obtain a three-dimensional Gaussian model; during the training and optimization process, performing splitting, copying, and pruning operations on Gaussian points in the three-dimensional Gaussian representation based on reprojection error, geometric regularization terms, and the average gradient of the view space; and performing three-dimensional segmentation of the target crop leaves using a symmetric reciprocal consistency mapping based on the three-dimensional Gaussian model. This application can accurately achieve three-dimensional reconstruction of crop plant structure and leaf segmentation.

Description

Technical Field

[0001] This application relates to the field of crop reconstruction and organ segmentation, and in particular to a method, equipment, medium, and product for crop plant reconstruction and segmentation. Background Technology

[0002] 3D Gaussian Splatting (3DGS) provides an efficient neural representation method for fine reconstruction of crop plant architecture. However, due to the high similarity of crop leaf textures and complex topological structures, related technologies still face three major bottlenecks in agricultural scenarios: traditional reconstruction struggles to automatically remove complex acquisition backgrounds, and noisy point clouds in non-target areas severely interfere with the fine modeling of the crop subject; during 3DGS training, the slender stems and thin leaf edges of crops often cause extreme stretching (needle artifacts) or expansion of Gaussian units, resulting in a loss of physical realism and failing to meet the requirements for high-precision phenotypic measurement; and the lack of robust algorithms to associate 2D instance annotations with 3D Gaussian units results in reconstructed models that only possess visual characteristics, making it difficult to achieve leaf-level instance segmentation and plant architecture parameter analysis.

[0003] Therefore, how to accurately achieve three-dimensional reconstruction of crop plant structure and leaf segmentation has become an urgent problem to be solved. Summary of the Invention

[0004] The purpose of this application is to provide a method, equipment, medium, and product for crop plant reconstruction and segmentation, which can accurately realize the three-dimensional reconstruction of crop plant structure and leaf segmentation.

[0005] To achieve the above objectives, this application provides the following solution.

[0006] In a first aspect, this application provides a method for crop plant architecture reconstruction and segmentation, comprising: acquiring multi-view two-dimensional images of a target crop; the multi-view two-dimensional images comprising: two-dimensional static images from multiple different perspectives; generating automated foreground masks for each perspective based on the multi-view two-dimensional images using morphological operations and color consistency detection methods; constructing an initial sparse three-dimensional point cloud of the target crop based on the automated foreground masks for each perspective using feature detection algorithms and motion recovery algorithms; initializing the initial sparse three-dimensional point cloud with a three-dimensional Gaussian representation to obtain an initial three-dimensional Gaussian representation, and based on... An automated foreground mask from each viewpoint iteratively trains and optimizes the initial 3D Gaussian representation, and the trained 3D Gaussian representation is determined as the 3D Gaussian model for reconstructing the target crop. During the training and optimization process, adaptive splitting, copying, and pruning operations are performed on the Gaussian points in the 3D Gaussian representation based on reprojection error, geometric regularization term, and view space average gradient. The geometric regularization term is used to constrain the morphology of the 3D Gaussian primitives in the 3D Gaussian model. Based on the 3D Gaussian model, a symmetric reciprocal consistency mapping is used to perform 3D segmentation of the leaves of the target crop to obtain the segmentation result.

[0007] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to implement the crop plant type reconstruction and segmentation method described in any one of the above.

[0008] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the crop plant type reconstruction and segmentation method described above.

[0009] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the crop plant architecture reconstruction and segmentation method described above.

[0010] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a method, device, medium, and product for crop plant reconstruction and segmentation. It uses morphological operations and color consistency detection to generate automated foreground masks from various viewpoints. Based on the automated foreground masks, it employs feature detection algorithms and motion recovery algorithms to construct an initial sparse 3D point cloud, effectively removing interference from noise point clouds in non-target areas. Based on the automated foreground masks, iterative training and optimization are performed on the initial 3D Gaussian representation formed by the initial sparse 3D point cloud to obtain a 3D Gaussian model. During the training and optimization process, adaptive splitting, copying, and pruning operations are performed on the Gaussian points in the 3D Gaussian representation based on reprojection error, geometric regularization terms, and the average gradient of the view space. A geometric regularization term is introduced into the standard loss function to constrain the morphology of 3D Gaussian primitives with extreme stretching or over-expansion, improving the geometric fidelity of crop detail reconstruction. Based on the 3D Gaussian model, symmetric reciprocal consistency mapping is used to perform 3D segmentation of the target crop leaves, achieving leaf-level instance segmentation. Therefore, this application can accurately achieve 3D reconstruction of crop plant structure and leaf segmentation. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating a method for crop plant architecture reconstruction and segmentation provided in an embodiment of this application.

[0013] Figure 2 This is a schematic diagram of the iterative training and optimization process provided in an embodiment of this application.

[0014] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] The purpose of this application is to provide a method, device, medium, and product for crop plant reconstruction and segmentation, aiming to solve problems such as background interference, geometric artifacts, and difficulties in 3D segmentation in related technologies. This application, based on multi-view imaging and 3D Gaussian splashing, achieves automated geometric regularization reconstruction under mask constraints and establishes a consistent 3D instance mapping mechanism across multiple views, thus achieving the goal of fine reconstruction of crop plant structure and leaf segmentation based on multi-view imaging.

[0017] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] In one exemplary embodiment, such as Figure 1 As shown, a method for crop plant architecture reconstruction and segmentation is provided, including the following steps.

[0019] Step 101: Obtain multi-view two-dimensional images of the target crop.

[0020] The multi-view two-dimensional image includes: two-dimensional static images from multiple different viewpoints.

[0021] Step 102: Based on the multi-view 2D images, generate automated foreground masks for each view using morphological operations and color consistency detection methods.

[0022] Step 103: Based on the automated foreground mask from each viewpoint, the initial sparse 3D point cloud of the target crop is constructed using feature detection algorithms and motion recovery algorithms.

[0023] Step 104: Initialize the initial sparse 3D point cloud with a 3D Gaussian representation to obtain an initial 3D Gaussian representation. Then, iteratively train and optimize the initial 3D Gaussian representation based on the automated foreground mask under each viewpoint, and determine the trained 3D Gaussian representation as the 3D Gaussian model for the reconstruction of the target crop.

[0024] During the training and optimization process, adaptive splitting, copying, and pruning operations are performed on Gaussian points in the 3D Gaussian representation based on reprojection error, geometric regularization, and view-space average gradient. View-space average gradient is introduced to achieve adaptive density control, and a geometric regularization term is introduced into the standard loss function to constrain the shape of the 3D Gaussian primitives in the 3D Gaussian model.

[0025] The geometric regularization term includes a scale ratio regularization term and an absolute scale regularization term. The scale ratio regularization term achieves scale ratio consistency optimization, and the absolute scale regularization term achieves absolute size envelope constraint.

[0026] The purpose of introducing a geometric regularization term into the standard loss function is specifically to improve the geometric fidelity of crop detail reconstruction by constraining the morphology of three-dimensional Gaussian elements that are extremely stretched or over-expanded.

[0027] Step 105: Based on the three-dimensional Gaussian model, the leaves of the target crop are segmented in three dimensions using a symmetric reciprocal consistency mapping to obtain the segmentation result.

[0028] In another exemplary embodiment of this application, step 101 specifically includes: using video stream extraction technology (extracting keyframes through video stream) or image acquisition device to obtain two-dimensional static images of the target crop from different perspectives to obtain multi-view two-dimensional images; wherein, the number of perspectives is determined according to the physical size of the target crop to achieve all-round coverage of plant surface information.

[0029] In another exemplary embodiment of this application, step 102 specifically includes the following steps.

[0030] (1) Convert the multi-view two-dimensional image to HSV space to obtain HSV images under each view, and use the preset crop green threshold range to binarize each HSV image to obtain the original mask under each view.

[0031] (2) Perform morphological operations on the original mask under each viewpoint to obtain the automated foreground mask under each viewpoint. The process of performing morphological operations on the original mask under any viewpoint includes the following steps.

[0032] Using a convolution kernel of a preset size, the original mask under this viewpoint is sequentially closed and opened to fill the voids inside the blades and eliminate isolated noise points, thus obtaining a preliminary mask.

[0033] The connected regions in the preliminary mask are labeled, and the connected region with the largest area is determined as the main plant region. The area ratio and centroid Euclidean distance of each candidate sub-region relative to the main plant region are calculated. The candidate sub-regions are the connected regions in the preliminary mask other than the main plant region.

[0034] For any candidate sub-region, when the centroid Euclidean distance of the candidate sub-region satisfies the distance constraint condition under the corresponding area ratio, or when the area ratio exceeds the preset area protection threshold, the candidate sub-region is retained and merged with the main plant region. After the candidate sub-region is traversed, the refined plant mask is obtained.

[0035] The refined plant mask is subjected to single or multiple dilation operations using a dilation operator of a preset size. The dilated refined plant mask is then translated and mapped to spatial coordinates in combination with the growth pose characteristics of the target crop, thereby generating an automated foreground mask for 3D Gaussian splash initialization from that perspective.

[0036] In another exemplary embodiment of this application, step 103 specifically includes: performing scale-invariant feature transformation (SIFT) feature detection on the automated foreground mask under each viewpoint to obtain image feature points (key points) and corresponding descriptors under each viewpoint; using an exhaustive matching strategy, determining the correspondence between image feature points under different viewpoints based on the descriptors; constructing a feature matching database based on the correspondence by using geometric constraints to filter out mismatches; and using the feature matching database to perform triangulation calculation using the incremental Structure from Motion (SfM) algorithm to solve the camera intrinsic and extrinsic pose parameters of each viewpoint, and generating an initial sparse 3D point cloud representing the preliminary outline of the target crop based on the camera intrinsic and extrinsic pose parameters.

[0037] In another exemplary embodiment of this application, step 104 involves training and geometrically optimizing the sparse point cloud and the automated foreground mask using a 3D Gaussian representation. The overall approach is as follows: each spatial point of the sparse 3D point cloud is initialized as a 3D Gaussian primitive with a center point, anisotropic scaling scale, rotation quaternion, opacity, and spherical harmonic function color attributes; the 3D Gaussian primitives from each viewpoint are projected onto a 2D image plane, and pixel interference from non-target areas is removed using the automated foreground mask. The reprojection error between the predicted image and the real image is calculated, and a training iteration process is initiated to update the attribute parameters of each 3D Gaussian primitive; during the training iteration process, adaptive splitting, copying, and pruning operations are performed based on the average gradient density of the Gaussian primitives in the view space to achieve intensive processing and density balance of the fine structure of crop leaves; within the preset iteration interval corresponding to the intensive processing, a geometric regularization term is introduced into the standard loss function.

[0038] Step 104 specifically includes the following steps.

[0039] (1) Initialize the attributes of each spatial point of the initial sparse three-dimensional point cloud to obtain three-dimensional Gaussian primitives with different initial attribute parameters; the attribute parameters include: position, scaling scale, rotation quaternion, opacity and spherical harmonic function color attribute.

[0040] (2) For the nth iteration, each three-dimensional Gaussian element in the three-dimensional Gaussian element set corresponding to the nth iteration is projected onto the two-dimensional image plane to obtain the projected image. The pixels of the non-target region in the projected image are removed according to the automatic foreground mask under each view to obtain the predicted image. The reprojection error between the predicted image and the real multi-view two-dimensional image is calculated to obtain the reprojection error of the nth iteration. If n=1, the three-dimensional Gaussian element set corresponding to the nth iteration is determined according to the three-dimensional Gaussian elements with different initial attribute parameters.

[0041] (3) Determine whether the nth iteration is in the dense iteration interval and obtain the first judgment result.

[0042] If the first judgment result is yes, then it is determined whether there is a three-dimensional Gaussian primitive (i.e., an abnormal primitive) in the three-dimensional Gaussian primitive set corresponding to the nth iteration whose scaling vector component ratio exceeds the set stretching threshold, and the second judgment result is obtained; the scaling vector component ratio is the ratio of the maximum component to the minimum component of the scaling vector component of the three-dimensional Gaussian primitive in different axes.

[0043] If the first judgment result is negative, the total loss function for the nth iteration is determined based on the reprojection error, and a 3D Gaussian model reconstruction operation is performed. The 3D Gaussian model reconstruction operation includes: performing adaptive splitting, copying, and pruning operations based on the total loss function for the nth iteration and the average gradient density of the 3D Gaussian elements in the view space of the 3D Gaussian elements in the 3D Gaussian element set corresponding to the nth iteration, to obtain the updated 3D Gaussian element set for the nth iteration. If n reaches the set number of iterations, the updated 3D Gaussian element set is determined as the trained 3D Gaussian representation to obtain the 3D Gaussian model of the target crop reconstruction. Otherwise, the updated 3D Gaussian element sets of the nth iteration are used as the 3D Gaussian element sets corresponding to the (n+1)th iteration for the next iteration until the 3D Gaussian model of the target crop reconstruction is obtained.

[0044] If the second judgment result is yes, then a scale ratio regularization term is introduced. The algorithm then determines whether there exists a 3D Gaussian element in the set of 3D Gaussian elements corresponding to the nth iteration whose spatial norm of the scaling vector exceeds a set size threshold, and obtains the third judgment result. The purpose of introducing the scaling ratio regularization term is to penalize and correct the needle-like Gaussian points of extreme stretching.

[0045] If the second judgment result is negative, then determine whether there is a three-dimensional Gaussian element in the set of three-dimensional Gaussian elements corresponding to the nth iteration whose spatial norm of the scaling vector (the spatial norm represents the absolute physical size of the three-dimensional Gaussian element) exceeds the set size threshold, and obtain the fourth judgment result.

[0046] If the third result is yes, then an absolute scale regularization term is introduced. The total loss function for the nth iteration is determined based on the reprojection error, the scale ratio regularization term, and the absolute scale regularization term, and the 3D Gaussian model reconstruction operation is then performed. The purpose of introducing the absolute scale regularization term is to limit the excessive expansion of the 3D Gaussian primitives and ensure the geometric fidelity of the reconstruction of crop edges and fine leaves.

[0047] If the third judgment result is negative, then the total loss function for the nth iteration is determined based on the reprojection error and the scale ratio regularization term, and the three-dimensional Gaussian model reconstruction operation is performed.

[0048] If the fourth result is yes, then an absolute scale regularization term is introduced, the total loss function for the nth iteration is determined based on the reprojection error and the absolute scale regularization term, and the three-dimensional Gaussian model reconstruction operation is performed.

[0049] If the fourth result is negative, the total loss function for the nth iteration is determined based on the reprojection error, and the three-dimensional Gaussian model reconstruction operation is performed.

[0050] The iterative training and optimization process is as follows: Figure 2 As shown.

[0051] In another exemplary embodiment of this application, step 105 specifically includes the following steps.

[0052] (1) Based on the camera intrinsic parameters and external pose parameters of each viewpoint, the three-dimensional Gaussian model is rendered from multiple viewpoints to obtain a rendered image with the same pose as the original acquired image.

[0053] (2) Perform instance segmentation and annotation on each leaf in the rendered image to obtain a two-dimensional annotated image carrying instance labels. The two-dimensional annotated image is a full-size image in which the background (non-leaf) pixel value is 0, and each different leaf is assigned a unique integer number (e.g., all pixels in leaf A region are 1, leaf B is 2, and so on).

[0054] (3) Based on the camera intrinsic parameters and external pose parameters of each viewpoint, the three-dimensional Gaussian primitives in the three-dimensional Gaussian model are back-projected to the plane where the two-dimensional labeled image is located, and the mapping index relationship between the three-dimensional Gaussian primitives and the two-dimensional instance labels is established.

[0055] (4) Traverse the mapping index relationship of each 3D Gaussian primitive under all visible views, count the number of times each 3D Gaussian primitive falls into the 2D blade instance region, calculate the multi-view voting ratio of each 3D Gaussian primitive, and when the multi-view voting ratio exceeds the preset consistency threshold, determine the corresponding 3D Gaussian primitive as a 3D blade candidate primitive. The 2D blade instance region is the set of pixels in the 2D labeled image that belong to a specific blade instance.

[0056] (5) Use the density-based spatial clustering of applications with noise (DBSCAN) algorithm to remove noise from all three-dimensional leaf candidate primitives, and use the spatial location constraint and color distribution of the target crop center to achieve the spatial location constraint and color distribution filtering of the target crop center, so as to obtain a set of three-dimensional leaf instance points with unique instance identifiers.

[0057] (6) Establish the spatial correspondence between the three-dimensional blade instance point set and the three-dimensional Gaussian model, and pass the corresponding unique instance identifier to the attributes of the three-dimensional Gaussian primitive to generate a three-dimensional Gaussian model carrying a unique identifier. Specifically: construct the spatial index structure of the three-dimensional blade instance point set; based on the spatial index structure, use the nearest neighbor search algorithm, with the geometric center coordinates of the three-dimensional Gaussian primitive in the three-dimensional Gaussian model as the query target, to retrieve the matching instance point closest to each three-dimensional Gaussian primitive in the three-dimensional blade instance point set; map and assign the instance identifier (instance ID) carried by the matching instance point to the corresponding three-dimensional Gaussian primitive to obtain a three-dimensional Gaussian model carrying a unique identifier, thereby realizing the cross-dimensional transfer of semantic labels from discrete point cloud to dense Gaussian representation.

[0058] (7) Generate a visual color instance model based on the three-dimensional Gaussian model with a unique identifier, and use the visual color instance model as the segmentation result. The segmentation result is used to analyze the plant structure of the target crop. Specifically: assign each three-dimensional Gaussian element in the three-dimensional Gaussian model with a unique identifier a non-conflicting high-contrast color, and map the assigned color to the zero-order coefficient of the spherical harmonic function of the three-dimensional Gaussian element as the DC component representing the basic color; set the higher-order spherical harmonic function coefficients of the three-dimensional Gaussian element to zero to eliminate the effect of color changing with the observation angle, and ensure the uniqueness and stability of the instance color under different observation angles; retain the original color attributes of the three-dimensional Gaussian elements in the background area, and export the updated instance color information, position information and geometric attributes as a PLY format file with instance tags as a visual color instance model for crop plant structure analysis.

[0059] The following provides a more specific embodiment to further describe the implementation process of the above-mentioned crop plant type reconstruction and segmentation method.

[0060] The crop plant architecture reconstruction and segmentation method in this embodiment includes the following steps.

[0061] Step 1: Obtain multi-view 2D images of the crop to be reconstructed.

[0062] In this step, two-dimensional static images of the crop from multiple spatial orientations are acquired using camera equipment. Specifically, this can be achieved by moving the camera around the plant to acquire a video stream and extracting keyframes, or by using a multi-camera array to acquire images simultaneously. The number of viewing angles is determined based on the physical size and structural complexity of the crop to be reconstructed (for example, for corn seedlings, it is recommended to have no fewer than 32 viewing angles) to ensure comprehensive coverage of the plant surface information.

[0063] Step 2: Generate an automated foreground mask.

[0064] Based on morphological processing and color consistency detection, the background is automatically stripped away to extract high-quality crop foreground areas.

[0065] Step 3: Initialization and geometric optimization training of 3D Gaussian representation.

[0066] An initial sparse point cloud is constructed using SIFT features and the SfM algorithm, and then 3DGS training with geometric regularization constraints is performed using the automated foreground mask obtained in step 2.

[0067] Step 4: 3D instance segmentation and color model generation.

[0068] By using a symmetric reciprocal consistency mapping mechanism, a semantic mapping from two-dimensional instance annotations to three-dimensional Gaussian units is achieved, ultimately deriving a finely segmented crop model.

[0069] The specific logic for generating an automated foreground mask is as follows.

[0070] Step 21: Color Space Conversion and Preliminary Extraction. The acquired RGB image is converted to the more robust HSV color space to lighting conditions, using a preset crop green threshold range (e.g., a preset crop green threshold range). Binarization is performed to obtain the original mask.

[0071] Step 22: Morphological purification. Utilize preset dimensions (e.g., ...) The convolution kernels are sequentially closed and opened to fill the voids inside the blades and eliminate isolated noise points, thus obtaining a preliminary mask.

[0072] Step 23: Region Association Analysis. Connectivity regions are labeled on the initial mask to identify the largest region as the main plant region. The area ratio and centroid Euclidean distance of each of the other candidate sub-regions relative to the main plant region are calculated.

[0073] Step 24: Region Adaptive Merging and Refinement. Based on the area proportion of each sub-region, a corresponding distance threshold is matched. When a sub-region meets the distance constraint condition under its corresponding proportion, or its area proportion exceeds a preset area protection threshold, the sub-region is retained and merged with the main plant region to obtain a refined plant mask.

[0074] Step 25: Spatial Mapping and Dilation. The refined plant mask is dilated once or multiple times using a dilation operator of a preset size, and spatial coordinate translation mapping is performed in combination with the growth pose features of the crop to be reconstructed, thereby generating an automated foreground mask for the final 3D Gaussian splash initialization.

[0075] The initial sparse point cloud construction process is as follows.

[0076] Step 31: Feature Detection. Perform Scale Invariant Feature Transform (SIFT) feature detection on the crop images from each viewpoint after masking to obtain the key points of each image and their corresponding descriptors.

[0077] Step 32: Feature Matching. An exhaustive matching strategy is executed to establish the correspondence between feature points in images from different viewpoints based on descriptors. Geometric constraints are used to filter out false matches, and a feature matching database is constructed.

[0078] Step 33: Incremental SfM. Based on the feature matching database, the incremental Structure from Motion (SfM) algorithm is used to perform triangulation calculation, iteratively solve the intrinsic and extrinsic pose parameters of each camera viewpoint, and generate a sparse 3D point cloud representing the preliminary outline of the crop.

[0079] In the 3DGS training phase under geometric regularization constraints, the artifact problem in crop leaf reconstruction is solved by introducing specific geometric constraints.

[0080] Initialization: Initialize each spatial point of the sparse 3D point cloud as a 3D Gaussian primitive with a location center, anisotropic scaling scale, rotation quaternion, opacity, and spherical harmonic color attribute.

[0081] Iterative training: The three-dimensional Gaussian primitives are projected onto the two-dimensional image plane, and pixel interference in non-target areas is removed by automatic foreground masking. During the training iteration, splitting and pruning are performed based on the average gradient density of the view space.

[0082] Within the preset iteration interval corresponding to the intensive processing, a geometric regularization term is introduced into the loss function. The geometric regularization term includes scale ratio consistency optimization and absolute size envelope constraint.

[0083] Scale ratio consistency optimization: Monitor the ratio of scaling ratios of different axes of the Gaussian element. When the ratio exceeds a set stretching threshold... (like When introducing a scale ratio regularization term, .

[0084] .

[0085] Among them, the first weight factor It can be set to 0.001. This setting can effectively suppress extreme needle-like Gaussian points.

[0086] Absolute size envelope constraint: Monitors the spatial norm of Gaussian elements. When the physical size exceeds a set size threshold... When this is done, an absolute scale regularization term is introduced. .

[0087] .

[0088] For example, when setting the size threshold to 0.01 units, an absolute scale regularization term is introduced. 0.01 units represents the value in a normalized space or relative coordinate system.

[0089] Among them, the second weighting factor It can be set to 0.001. This setting ensures the geometric fidelity of crop edges.

[0090] The expression for the total loss function is as follows.

[0091] .

[0092] Among them, the weight balancing factor It can be set to 0.2. This represents the mean absolute error between the rendered image and the real image. This is a metric used to evaluate the perceptual similarity between a rendered image and a real image. It is the core loss function of standard 3D Gaussian splashing, which represents the reconstruction error. It is the total loss function that incorporates scale ratio regularization and absolute scale regularization terms.

[0093] The construction process for using the reconstructed model to achieve leaf-level instance segmentation in step 4 is as follows.

[0094] Step S41: Multi-view rendering. Using the trained 3D Gaussian representation and combining the camera pose parameters of each viewpoint, multi-view image rendering is performed to obtain a rendered image with the same pose as the original acquired image.

[0095] Step S42: Two-dimensional image annotation. Perform instance segmentation and annotation on each leaf target in the rendered image to obtain a two-dimensional annotated image carrying instance labels.

[0096] Step S43: Index Relationship Mapping. Based on the camera pose parameters, the reconstructed 3D Gaussian primitives are back-projected onto the corresponding 2D labeled image planes from each viewpoint, establishing a mapping index relationship between the 3D Gaussian primitives and the 2D instance labels.

[0097] Step S44: Candidate primitive determination. Traverse the mapping results of each 3D Gaussian primitive under all visible views, count the number of times the point falls into the 2D blade instance region, calculate its multi-view voting ratio, set a preset consistency threshold, and when the multi-view voting ratio exceeds the threshold, the corresponding 3D Gaussian primitive is determined as a 3D blade candidate primitive.

[0098] Step S45: Instance ID Transmission. A clean set of 3D leaf instance points is obtained using DBSCAN clustering and spatial location constraints. Then, a spatial mapping between the point cloud and dense Gaussian units is established using the nearest neighbor search algorithm (KNN), and the instance ID is transmitted to each Gaussian unit.

[0099] The process of transferring instance identifiers from the 3D blade instance point set to the 3D Gaussian primitive includes: Step S451: Constructing a spatial index structure for the 3D blade instance point set. Step S452: Using a nearest neighbor search algorithm, with the geometric center coordinates of the 3D Gaussian primitive as the query target, retrieving the nearest matching instance point to each 3D Gaussian primitive in the 3D blade instance point set. Step S453: Mapping and assigning the instance ID carried by the matching instance point to the corresponding 3D Gaussian primitive, realizing the cross-dimensional transfer of semantic labels from discrete point clouds to dense Gaussian representation.

[0100] Step S46: Visualization of the color model. Assign high-contrast colors to Gaussian elements with different IDs and update their spherical harmonic function DC components. Preserve the original color attributes of the 3D Gaussian elements in the background region. Export the updated instance color information, position information, and geometric attributes as PLY format files with instance labels for use in crop plant structure analysis.

[0101] The crop plant reconstruction and segmentation method of this embodiment first acquires multi-view 2D images of the crop to be reconstructed; then, a high-quality automated foreground mask is generated based on morphological and color consistency detection; the camera pose is estimated using SIFT feature matching, and an initial sparse 3D point cloud is constructed using a motion recovery method; the sparse point cloud is initialized as a 3D Gaussian representation, and during training, adaptive splitting, copying, and pruning operations are performed on the Gaussian points according to the reprojection error and density distribution; geometric regularization constraints are introduced in the densification stage to optimize the scale and aspect ratio of the Gaussian points; based on the reconstructed 3D Gaussian representation, refined 3D segmentation of crop leaves is achieved through symmetric reciprocal consistency mapping. This application can quickly and accurately achieve 3D reconstruction, rendering, and leaf segmentation of crops.

[0102] Based on the same inventive concept, this application also provides a crop plant type reconstruction and segmentation device for implementing the crop plant type reconstruction and segmentation method described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more crop plant type reconstruction and segmentation device embodiments provided below can be found in the limitations of the crop plant type reconstruction and segmentation method above, and will not be repeated here.

[0103] In one exemplary embodiment, a crop plant structure reconstruction and segmentation device is provided, comprising: an image acquisition module for acquiring multi-view two-dimensional images of a target crop; the multi-view two-dimensional images comprising: two-dimensional static images from multiple different perspectives.

[0104] The foreground mask generation module is used to generate automated foreground masks for each viewpoint based on the multi-view 2D image using morphological operations and color consistency detection methods.

[0105] An initial sparse 3D point cloud construction module is used to construct the initial sparse 3D point cloud of the target crop based on an automated foreground mask from various viewpoints, employing feature detection algorithms and motion recovery algorithms.

[0106] The model reconstruction module is used to initialize the initial sparse 3D point cloud with a 3D Gaussian representation to obtain an initial 3D Gaussian representation. It then iteratively trains and optimizes the initial 3D Gaussian representation based on an automated foreground mask from each viewpoint, and determines the trained 3D Gaussian representation as the 3D Gaussian model for reconstructing the target crop. During the training and optimization process, adaptive splitting, copying, and pruning operations are performed on the Gaussian points in the 3D Gaussian representation based on reprojection error, geometric regularization, and the average gradient in the view space. The geometric regularization term is used to constrain the morphology of the 3D Gaussian primitives in the 3D Gaussian model.

[0107] The three-dimensional segmentation module is used to perform three-dimensional segmentation of the leaves of the target crop based on the three-dimensional Gaussian model and using symmetric reciprocal consistency mapping to obtain the segmentation results.

[0108] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 3As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores the three-dimensional Gaussian model and segmentation results of the target crop reconstruction. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a crop plant architecture reconstruction and segmentation method.

[0109] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment to which the present application is applied. Specific computer equipment may include, for example, [the following is a list of possible additional structures]. Figure 3 The embodiments show more or fewer components, combinations of certain components, or different component arrangements. In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, which the processor executes to implement the steps in the above-described method embodiments.

[0110] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0111] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0112] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0113] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0114] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, etc., and are not limited to these.

[0115] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0116] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for reconstructing and segmenting crop plant architecture, characterized in that, include: Acquire multi-view two-dimensional images of the target crop; The multi-view two-dimensional image includes: two-dimensional static images from multiple different viewpoints; Based on the multi-view 2D images, morphological operations and color consistency detection methods are used to generate automated foreground masks for each view. Based on automated foreground masks from various perspectives, feature detection algorithms and motion recovery algorithms are used to construct the initial sparse 3D point cloud of the target crop. The initial sparse 3D point cloud is initialized with a 3D Gaussian representation to obtain an initial 3D Gaussian representation. This initial 3D Gaussian representation is then iteratively trained and optimized based on an automated foreground mask from each viewpoint. The trained 3D Gaussian representation is then determined as the 3D Gaussian model for reconstructing the target crop. During the training and optimization process, adaptive splitting, copying, and pruning operations are performed on the Gaussian points in the 3D Gaussian representation based on reprojection error, geometric regularization, and the view space average gradient. The geometric regularization term is used to constrain the morphology of the 3D Gaussian primitives in the 3D Gaussian model. Based on the aforementioned three-dimensional Gaussian model, the leaves of the target crop are segmented in three dimensions using a symmetric reciprocal consistency mapping to obtain the segmentation results. The initial sparse 3D point cloud is initialized with a 3D Gaussian representation to obtain an initial 3D Gaussian representation. This initial 3D Gaussian representation is then iteratively trained and optimized based on an automated foreground mask from each viewpoint. The trained 3D Gaussian representation is then determined as the 3D Gaussian model for reconstructing the target crop. Specifically, this includes: The spatial points of the initial sparse 3D point cloud are initialized with attributes to obtain 3D Gaussian primitives with different initial attribute parameters; the attribute parameters include: position, scaling scale, rotation quaternion, opacity, and spherical harmonic function color attribute; For the nth iteration, each 3D Gaussian element in the 3D Gaussian element set corresponding to the nth iteration is projected onto the 2D image plane to obtain a projected image. Pixels in non-target regions of the projected image are removed using an automated foreground mask under each viewpoint to obtain a predicted image. The reprojection error between the predicted image and the real multi-view 2D image is calculated to obtain the reprojection error for the nth iteration. If n=1, the 3D Gaussian element set corresponding to the nth iteration is determined based on 3D Gaussian elements with different initial attribute parameters.

2. The method for crop plant architecture reconstruction and segmentation according to claim 1, characterized in that, Obtaining multi-view two-dimensional images of the target crop, specifically including: Two-dimensional static images of the target crop from different perspectives are obtained by using video stream extraction technology or image acquisition equipment to obtain multi-view two-dimensional images; the number of perspectives is determined according to the physical size of the target crop.

3. The method for crop plant architecture reconstruction and segmentation according to claim 1, characterized in that, Based on the multi-view 2D images, an automated foreground mask is generated for each viewpoint using morphological operations and color consistency detection methods, specifically including: The multi-view two-dimensional images are converted to HSV space to obtain HSV images under each view. The HSV images are then binarized using a preset crop green threshold range to obtain the original mask under each view. Morphological operations are performed on the original mask from each viewpoint to obtain an automated foreground mask for each viewpoint. The process of performing morphological operations on the original mask from any given viewpoint includes: Using a convolution kernel of a preset size, the original mask under this viewpoint is sequentially closed and opened to fill the internal holes of the blades and eliminate isolated noise points, thus obtaining a preliminary mask. The connected regions in the preliminary mask are labeled, and the connected region with the largest area is determined as the main plant region. The area ratio and centroid Euclidean distance of each candidate sub-region relative to the main plant region are calculated. The candidate sub-regions are the connected regions in the preliminary mask other than the main plant region. For any candidate sub-region, when the centroid Euclidean distance of the candidate sub-region satisfies the distance constraint condition under the corresponding area ratio, or when the area ratio exceeds the preset area protection threshold, the candidate sub-region is retained and merged with the main plant region. After the candidate sub-region is traversed, the refined plant mask is obtained. The refined plant mask is dilated using a pre-defined dilation operator, and the dilated refined plant mask is spatially translated and mapped based on the growth pose characteristics of the target crop, thereby generating an automated foreground mask for 3D Gaussian splash initialization from this perspective.

4. The method for crop plant architecture reconstruction and segmentation according to claim 1, characterized in that, Based on automated foreground masks from various perspectives, an initial sparse 3D point cloud of the target crop is constructed using feature detection algorithms and motion recovery algorithms. Specifically, this includes: Feature detection is performed on the automated foreground mask under various viewpoints using scale-invariant feature transformation to obtain image feature points and corresponding descriptors under each viewpoint. An exhaustive matching strategy is adopted to determine the correspondence between image feature points under different viewpoints based on the descriptor. Based on the correspondence, a geometric constraint is used to filter out mismatches and construct a feature matching database. Based on the feature matching database, an incremental algorithm for recovering structure from motion is used to perform triangulation calculations, solve the camera intrinsic and extrinsic pose parameters of each viewpoint, and generate an initial sparse 3D point cloud representing the preliminary outline of the target crop based on the camera intrinsic and extrinsic pose parameters.

5. The method for crop plant architecture reconstruction and segmentation according to claim 1, characterized in that, The process includes initializing the initial sparse 3D point cloud with a 3D Gaussian representation to obtain an initial 3D Gaussian representation, iteratively training and optimizing the initial 3D Gaussian representation based on an automated foreground mask from each viewpoint, and determining the trained 3D Gaussian representation as the 3D Gaussian model for reconstructing the target crop. The method further includes: Determine whether the nth iteration is within the dense iteration interval to obtain the first determination result; If the first judgment result is yes, then determine whether there is a three-dimensional Gaussian element in the set of three-dimensional Gaussian elements corresponding to the nth iteration whose scaling vector component ratio exceeds the set stretching threshold, and obtain the second judgment result; the scaling vector component ratio is the ratio of the maximum component to the minimum component of the scaling vector component of the three-dimensional Gaussian element in different axes. If the first judgment result is negative, the total loss function for the nth iteration is determined based on the reprojection error, and a three-dimensional Gaussian model reconstruction operation is performed. The three-dimensional Gaussian model reconstruction operation includes: performing adaptive splitting, copying, and pruning operations based on the total loss function for the nth iteration and the average gradient density of the three-dimensional Gaussian elements in the view space of the three-dimensional Gaussian elements in the set corresponding to the nth iteration, to obtain the three-dimensional Gaussian elements updated for the nth iteration. If n reaches the set number of iterations, the updated three-dimensional Gaussian elements are determined as the trained three-dimensional Gaussian representation to obtain the three-dimensional Gaussian model of the target crop reconstruction. Otherwise, the three-dimensional Gaussian elements updated for the nth iteration are used as the three-dimensional Gaussian elements corresponding to the (n+1)th iteration for the next iteration until the three-dimensional Gaussian model of the target crop reconstruction is obtained. If the second judgment result is yes, then a scale ratio regularization term is introduced, and it is determined whether there are three-dimensional Gaussian elements in the set of three-dimensional Gaussian elements corresponding to the nth iteration whose spatial norm of the scaling vector exceeds the set size threshold, thus obtaining the third judgment result. If the second judgment result is negative, then determine whether there is a three-dimensional Gaussian element in the set of three-dimensional Gaussian elements corresponding to the nth iteration whose spatial norm of the scaling vector exceeds the set size threshold, and obtain the fourth judgment result; If the third judgment result is yes, then an absolute scale regularization term is introduced, and the total loss function of the nth iteration is determined based on the reprojection error, the scale ratio regularization term, and the absolute scale regularization term, and the three-dimensional Gaussian model reconstruction operation is performed. If the third judgment result is negative, then the total loss function for the nth iteration is determined based on the reprojection error and the scale ratio regularization term, and the three-dimensional Gaussian model reconstruction operation is performed. If the fourth judgment result is yes, then an absolute scale regularization term is introduced, the total loss function of the nth iteration is determined based on the reprojection error and the absolute scale regularization term, and the three-dimensional Gaussian model reconstruction operation is performed. If the fourth judgment result is negative, then the total loss function for the nth iteration is determined based on the reprojection error, and the three-dimensional Gaussian model reconstruction operation is performed.

6. The method for crop plant architecture reconstruction and segmentation according to claim 1, characterized in that, Based on the aforementioned three-dimensional Gaussian model, a symmetric reciprocal consistency mapping is used to perform three-dimensional segmentation of the target crop leaves, yielding the segmentation results, specifically including: Based on the camera intrinsic parameters and external pose parameters from each viewpoint, the three-dimensional Gaussian model is rendered from multiple viewpoints to obtain the rendered image. Each leaf in the rendered image is segmented and labeled to obtain a two-dimensional labeled image carrying instance labels; Based on the camera intrinsic parameters and external pose parameters of each viewpoint, the three-dimensional Gaussian primitives in the three-dimensional Gaussian model are back-projected onto the plane where the two-dimensional labeled image is located, respectively, to establish a mapping index relationship between the three-dimensional Gaussian primitives and the two-dimensional instance labels. Traverse the mapping index relationship of each 3D Gaussian primitive under all visible views, count the number of times each 3D Gaussian primitive falls into the 2D blade instance region, calculate the multi-view voting ratio of each 3D Gaussian primitive, and when the multi-view voting ratio exceeds the preset consistency threshold, determine the corresponding 3D Gaussian primitive as a 3D blade candidate primitive. A density-based clustering algorithm was used to remove noise from all 3D leaf candidate primitives, and a set of 3D leaf instance points carrying unique instance identifiers was obtained based on the spatial location and color distribution of the target crop center. Establish a spatial correspondence between the three-dimensional blade instance point set and the three-dimensional Gaussian model, and pass the corresponding unique instance identifier to the attribute of the three-dimensional Gaussian primitive to generate a three-dimensional Gaussian model carrying a unique identifier. A visual color instance model is generated based on a three-dimensional Gaussian model carrying a unique identifier. The visual color instance model is used as the segmentation result, and the segmentation result is used to analyze the plant structure of the target crop.

7. The method for crop plant architecture reconstruction and segmentation according to claim 6, characterized in that, Establishing a spatial correspondence between the set of 3D blade instance points and the 3D Gaussian model, passing the corresponding unique instance identifier to the attributes of the 3D Gaussian primitives, and generating a 3D Gaussian model carrying a unique identifier, specifically includes: Construct the spatial index structure of the three-dimensional blade instance point set; Based on the spatial index structure, using the nearest neighbor search algorithm, with the geometric center coordinates of the three-dimensional Gaussian elements in the three-dimensional Gaussian model as the query target, the matching instance point closest to each three-dimensional Gaussian element is retrieved from the set of three-dimensional leaf instance points; The instance identifiers carried by the matching instance points are mapped and assigned to the corresponding 3D Gaussian elements to obtain a 3D Gaussian model with a unique identifier.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the crop plant architecture reconstruction and segmentation method according to any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the crop plant architecture reconstruction and segmentation method as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the crop plant architecture reconstruction and segmentation method as described in any one of claims 1-7.

Images