3D GS fruit instance counting method based on class center perception

Abstract

The application discloses a 3DGS fruit instance counting method based on class center perception, and specifically comprises the following steps: step 1, collecting multi-view images of fruit trees and performing two-dimensional segmentation and vector field pretreatment; step 2, reconstructing and rendering a scene 3DGS model with multi-attribute decoupling according to the pretreated images and semantic information; step 3, generating a 3DGS three-dimensional relative offset of the instance center of the fruit surface layer by using a two-dimensional center vector prior guide; and step 4, generating a physical collapse point cloud according to the offset, and separating the adhesion cluster by combining hierarchical clustering to output the final fruit counting result. The application breaks the traditional paradigm of three-dimensional point cloud segmentation first and tracking later, effectively solves the technical problems that adjacent fruits are easily merged and connected and counting is inaccurate in a dense and occluded scene by decoupling the rendering coordinates and the clustering coordinates, and greatly improves the precision and stability of three-dimensional fruit instance counting.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of agricultural information technology and computer vision, specifically involving a 3DGS fruit instance counting method based on class center perception. Background Technology

[0002] With the continuous growth of the global population and the increasing frequency of extreme weather events, ensuring food security and improving agricultural production efficiency have become core challenges for modern agricultural development. Precision agriculture and smart agriculture utilize advanced information technology to achieve quantitative monitoring and scientific decision-making throughout the entire agricultural production process. Plant phenotypic analysis, as a bridge connecting the interaction between plant genotype and environment, plays a crucial role in crop breeding, growth monitoring, and yield prediction. However, because plants exhibit extremely complex morphological and structural changes in their natural growth environment and are highly susceptible to interference from light, wind, and dense foliage, traditional plant phenotypic parameter measurements are heavily reliant on manual labor. This is not only time-consuming and labor-intensive but also suffers from strong subjectivity and destructive sampling, encountering insurmountable bottlenecks in accuracy and robustness.

[0003] In recent years, high-throughput phenotypic analysis techniques based on two-dimensional images have made significant progress. However, due to the lack of depth information in two-dimensional images, it is often difficult to extract accurate structural parameters and yield information when faced with complex intersecting occlusion of plant branches and leaves, drastic changes in light, and dense fruit clusters. Extending phenotypic analysis from two-dimensional to three-dimensional to obtain high-precision three-dimensional geometric structure and spatial distribution characteristics of plants, and even extending it to continuous analysis in four-dimensional space including the time dimension, has become an important development trend in the field of agricultural informatics.

[0004] In the assessment phase of agricultural production, instance counting and yield prediction of key organs such as fruits have irreplaceable practical application value for harvest planning, market pricing, and guiding the operation of automated harvesting robots. Traditional 3D fruit perception methods mostly rely on direct scanning with depth cameras and point cloud clustering, but structural breaks and loss of texture details caused by canopy occlusion easily lead to counting errors. In recent years, the paradigm of 3D scene representation technology has shifted from implicit neural radiation fields (NeRF) to explicit 3D Gaussian sputtering (3DGS). 3DGS, with its explicit Gaussian primitives and excellent scene representation capabilities, effectively improves the shortcomings of traditional point clouds in fruit detail reconstruction. However, how to break away from the paradigm of segmentation followed by tracking and achieve accurate 3D fruit separation and counting under complex occlusion remains a pressing technical challenge. Summary of the Invention

[0005] The purpose of this invention is to provide a 3DGS fruit instance counting method based on class center perception, which solves the problems of instance merging and inaccurate counting that are easily generated in existing 3D fruit counting methods under densely clustered and complex occlusion environments.

[0006] The technical solution adopted in this invention is a 3DGS fruit instance counting method based on class center perception, which specifically includes the following steps: Step 1: Collect multi-view image sequences of fruit trees and perform preprocessing to extract two-dimensional prior information; Step 2: Based on the image preprocessed in Step 1, initialize and render a multi-attribute scene 3DGS model by combining semantic information. Step 3: Using two-dimensional prior information, calculate and optimize the 3DGS three-dimensional relative offset from the center of the fruit in the generated scene; Step 4: Based on the 3DGS three-dimensional relative offset obtained in Step 3, perform physical collapse and hierarchical clustering, and output the final fruit instance count result.

[0007] The invention is further characterized by: The specific process of step 1 is as follows: The specific process of step 1 is as follows: Step 1.1: Input a sequence of N fruit tree images taken from different perspectives. ; Step 1.2, image sequence As input, the Structure for Motion Recovery (SFM) algorithm is called to perform feature matching and bundle adjustment calculation, and the camera intrinsic parameters, camera extrinsic parameters, and sparse 3D point cloud used to initialize the 3DGS scene are output for each image. Step 1.3, extract a single image As input, the data is fed into a trained 2D image segmentation model for instance segmentation, and the output is a single image. Target 2D segmentation results With the corresponding semantic mask ,in Representing an image The number of fruit instances identified; Step 1.4: Calculate the two-dimensional geometric center for each two-dimensional fruit instance region obtained in Step 1.3, and calculate the displacement vector of each foreground pixel in the instance region pointing to the corresponding two-dimensional geometric center, and output a two-dimensional displacement vector field with the same size as the original image. Step 1.5: Integrate the processing results and output the image sequence. The corresponding full-view 2D segmentation map set 2D semantic mask set With 2D displacement vector field set .

[0008] In step 1.2, the output specifically includes the set of camera intrinsic parameters. Camera pose extrinsic parameter set and the initial sparse point cloud set .

[0009] In step 1.4, for the k-th fruit instance region in the n-th image... The formula for calculating its two-dimensional geometric center is: (1) in, Represents the pixel coordinates within the instance region. This indicates the number of pixels contained in the instance. For any pixel u within the instance region, the formula for calculating the corresponding two-dimensional displacement vector is: (2) Each foreground pixel stores a horizontal displacement and a vertical displacement; the displacement vectors of the background pixels are all set to zero vectors, and the final output is the 2D displacement vector field corresponding to the nth image. .

[0010] The specific process of step 2 is as follows: Step 2.1, Define the initial multi-attribute Gaussian set. Each Gaussian unit Including three-dimensional coordinates Covariance and scaling / rotation parameters Opacity spherical harmonic coefficient Semantic probability attributes and 3D relative offset attribute And define the optimized Gaussian set as Each Gaussian unit With Same attribute type, This represents the total number of Gaussian cells after adaptive optimization. Step 2.2: Initialize all Gaussian properties, set the maximum iteration threshold, and perform model training and property optimization; Step 2.3: Determine whether the optimization convergence condition is met. If it is met, output the optimized 3DGS scene model.

[0011] The specific process of step 2.2 is as follows: Step 2.2.1, convert the three-dimensional spatial coordinates Initialize to the initial sparse point cloud 3D coordinates; Step 2.2.2, set the rotation parameters Initialize to a unit quaternion Scaling parameters Adaptive initialization based on the average physical distance of the three nearest neighbors; covariance Initialize to ; Step 2.2.3, adjust the opacity. Initialize to ; Step 2.2.4, determine the spherical harmonic coefficients. The zero-order coefficients are initialized by the original color of the point cloud, and all higher-order coefficients are set to 0; Step 2.2.5, transfer the semantic probability attribute Initialize to a neutral scale; Step 2.2.6, perform three-dimensional relative offset Initialize as zero vector ; Step 2.2.7, Set the current iteration counter And preset the maximum iteration threshold. ; Step 2.2.8, from the multi-attribute Gaussian set Extract a pair of visual semantic co-occurring Gaussians that share a geometric space but are functionally decoupled, namely the set of appearance radiating Gaussians. With semantic location sub-Gaussian set ; Step 2.2.9, will and camera internal parameters With camera external parameters The data is fed into the color branch rasterization projection renderer to obtain and... Output predicted color map set from each perspective ; Step 2.2.10, will and camera internal parameters With camera external parameters The data is fed into the semantic branch rasterization projection renderer, and then output to... A set of semantic masks predicted from each perspective ; Step 2.2.11, predict the color map set Predictive semantic mask set With real image set True semantic mask set The data is fed into the loss function calculation unit to calculate the total loss function for dual-channel supervision multi-objectives. ; Step 2.2.12, convert the total loss function The error is backpropagated in the dynamic iterative optimizer, and the properties of the existing Gaussian are adaptively updated according to the gradient, resulting in the intermediate Gaussian set after a single iteration. .

[0012] The specific process of step 2.3 is as follows: Step 2.3.1: Check if the current iteration counter satisfies the condition. If not satisfied, then let , the intermediate Gaussian set As a new multi-attribute Gaussian set Return to step 2.2.8 for the next iteration; if the condition is met, terminate the loop. Step 2.3.2: Select the intermediate Gaussian set that currently meets the conditions. Assign to And output the final optimized 3DGS scene model. Proceed to step 3.

[0013] The specific process of step 3 is as follows: Step 3.1, from the scene Gaussian model Extracting the fruit Gaussian set: A semantic threshold Thr is set, and Gaussian units in the scene Gaussian model are filtered based on whether the Gaussian semantic probability attribute is greater than the threshold Thr to obtain the fruit Gaussian set. The selected fruit Gaussian units are ; Step 3.2, Set the current iteration counter And preset the maximum iteration threshold. ; Step 3.3, transfer the fruit Gaussian Basic 3D coordinates Camera internal parameters Camera external parameters and 2D displacement vector field set As input, the Gaussian points are projected onto the normalized device coordinate system using a full perspective projection matrix transformation, and bilinear mesh sampling is performed to obtain the set of target rays for the 3D virtual camera. ; (3) in, Indicates based on camera intrinsic parameters and external references The projection matrix transformation function, three-dimensional Gaussian coordinates Map to the normalized device coordinate system.

[0014] Next, normalized device coordinates are used. Calling the bilinear sampling function from the 2D displacement vector field Extract the two-dimensional offset vector towards the center of the fruit and combine it with the camera intrinsic inverse matrix. Calculate 3D target rays : (4) in, This represents the two-dimensional centripetal displacement obtained from sampling. Represents the bilinear grid sampling function. This represents the vector normalization operator.

[0015] Step 3.4, transfer the fruit Gaussian 3D coordinates Project the view transformation matrix onto the camera coordinate system, calculate and extract... The absolute value of the axis is used to obtain the absolute line-of-sight depth. ; (5) in, This indicates that the x, y, and z spatial coordinate components in a three-dimensional coordinate system are taken, and the transformed coordinates are... Absolute line-of-sight depth is ; Step 3.5, set the target rays 3D relative offset and absolute depth of vision As input, calculate the total loss function of topological deformation. ; Step 3.6, calculate the total loss function of topological deformation. The data is fed into the error backpropagation optimizer, which only updates the Gaussian of the fruit. Three-dimensional relative offset ; Step 3.7, check if the current iteration counter satisfies the condition. If not satisfied, then let Then return to step 3.3 for the next round of loop optimization; if satisfied, terminate the loop and proceed to step 3.8. Step 3.8, output the optimized Gaussian set of fruits. .

[0016] In step 3.5, the total loss function of topological deformation It is a weighted sum consisting of target ray traction loss, scale-adaptive depth loss, and elastic constraints. The specific calculation formula is as follows: (6) in, Indicates the target's ray-driven loss; This represents scale-adaptive depth loss; This represents the loss due to elastic constraints; These are the corresponding hyperparameter weights; The specific formula for calculating the target traction loss is as follows: (7) (8) (9) in, Indicates target traction loss; For the fruit Gaussian set The total number of Gaussian units in the array; For the j-th fruit Gauss Adaptive weights; For the j-th fruit Gauss The three-dimensional coordinates projected onto the camera coordinate system after physical collapse. ; For the j-th fruit Gauss The corresponding 3D virtual camera target ray; For the j-th fruit Gauss semantic attributes, For the fruit Gauss The magnitude of the corresponding two-dimensional offset vector towards the center of the fruit, , and It is the j-th fruit Gauss Horizontal and vertical displacement components in the image coordinate system The magnitude of the maximum two-dimensional displacement vector. .

[0017] The formula for calculating scale-adaptive depth loss is: (10) (11) in, This represents scale-adaptive depth loss; For the fruit Gaussian set The total number of Gaussian units in the array; For the j-th fruit Gauss Adaptive weights; , represents the actual collapse distance generated inward along the line of sight of the j-th Gaussian point; This represents the absolute depth of the point in the camera coordinate system after it collapses. This represents the absolute line-of-sight depth before optimization. For the j-th fruit Gauss The estimated lateral physical span mapped to three-dimensional physical space; and These represent the horizontal and vertical pixel components of the two-dimensional offset vector on the image plane, respectively. and These represent the camera's horizontal and vertical focal lengths, respectively, and are considered internal camera parameters. The parameters in.

[0018] The formula for calculating elastic constraints is: (12) in, This represents the loss due to elastic constraints; For the fruit Gaussian set The total number of Gaussian units in the array; For the j-th fruit Gauss Adaptive weights; For the j-th fruit Gauss The three-dimensional relative offset; This indicates the calculation of the square value of the L2 norm.

[0019] The specific process of step 4 is as follows: Step 4.1, transfer the fruit Gaussian Basic 3D coordinates relative offset with the optimized 3D By performing vector superposition calculations, the coordinates of the Gaussian center point of the collapsed fruit are obtained. , Output a set of physically collapsed point clouds. ; Step 4.2, assemble the physical collapse point cloud. Together with the set neighborhood radius parameter Input the DBSCAN algorithm and output the initial set of point clusters. ; Step 4.3, Calculate the initial set of point clusters. Each point cluster Maximum three-dimensional physical span : (13) Will Compared with the preset maximum size threshold for a single fruit By comparing, the span was separated. single fruit point cluster set and span Multi-fruit adhesion clusters ; Step 4.4, assembling multi-fruit adhesion clusters Use a smaller neighborhood radius , The DBSCAN algorithm is executed again for clustering, thereby breaking down the originally clustered fruit clusters into independent sets of fruit point clusters. ; Step 4.5: Set the single fruit clusters together. With the split clustered fruit point cluster set Take the union of the sets and merge them to obtain the final set of independent fruit point clusters. traversal Each independent point cluster in Calculate all physical collapse points within the cluster. Spatial arithmetic mean coordinates : (14) Output the set of coordinates of the center point of each fruit. Each of them The three-dimensional physical center of mass represents a real, independent fruit; Step 4.6: Calculate the set of coordinates of the center points of independent fruits. The number of elements is used to output the total number of fruits. .

[0020] The beneficial effects of this invention are as follows: (1) This invention uses the two-dimensional visual segmentation results to generate a two-dimensional displacement vector field pointing to the center of the fruit instance, and combines the camera intrinsic and extrinsic parameters to introduce the two-dimensional prior into the three-dimensional Gaussian space. It does not require additional three-dimensional instance center or three-dimensional instance mask annotation, which greatly reduces the annotation cost of three-dimensional fruit data.

[0021] (2) This invention sets independent semantic attributes and three-dimensional relative offsets in the scene Gaussian attributes, decoupling the basic three-dimensional coordinates used for image rendering from the offset coordinates used for instance clustering. During the offset optimization process, only the position of the virtual collapse point is changed, without changing the basic coordinates, color, scaling, rotation, and opacity of the original scene Gaussian. Therefore, it is possible to obtain a collapsed point cloud suitable for instance separation while maintaining the stability of the original 3DGS scene appearance and geometric expression.

[0022] (3) The present invention optimizes the three-dimensional relative offset by using target ray traction loss, scale adaptive depth loss and elastic constraint to make the Gaussian points on the same fruit surface gather towards its internal central region, so as to create clearer spatial gaps between adjacent adhered fruits and improve the problem that traditional three-dimensional semantic clustering easily merges multiple fruits into one instance.

[0023] (4) The present invention adopts a hierarchical clustering method dominated by semantic connectivity and assisted by collapse points, and uses the maximum size threshold of a single fruit for anti-fragmentation judgment. It can not only split multi-fruit clusters with large spans, but also prevent a single fruit from being divided into multiple instances due to incomplete reconstruction or uneven distribution. This significantly improves the accuracy and stability of fruit counting in natural shading and dense fruit-bearing scenarios. Attached Figure Description

[0024] Figure 1 This is a flowchart of the 3DGS fruit instance counting method based on class center perception of the present invention; Figure 2 This is a 2D prior extraction result diagram of the 3DGS fruit instance counting method based on class center perception of the present invention on a public dataset; Figure 3 This is a comparison of the original point cloud (a) and the topologically collapsed point cloud (b) from the publicly available dataset. Figure 4 This is a diagram showing the 2D prior extraction results from three different angles on a self-built dataset using the 3DGS fruit instance counting method based on class center perception, as described in this invention. Figure 5 This is a comparison chart of subjective experimental counting results of the class-center-aware 3DGS fruit instance counting method of this invention on a self-built dataset. Detailed Implementation

[0025] The following detailed description is provided in conjunction with specific implementation methods.

[0026] Example 1 This invention is based on a 3DGS fruit instance counting method with class center perception, such as... Figure 1 As shown, the specific steps include the following: Phase 1: Preprocessing Phase; Phase Two: Scene 3DGS Generation Process; Third stage: 3DGS offset generation process from the center of the fruit; Phase 4: Post-relocation 3DGS center point clustering process; Example 2 The specific process of the first stage is as follows: Step 1: Input N sequences of fruit tree images. ; Step 2, image sequence As input, the Structure of Motion (SFM) algorithm is invoked for feature matching and bundle adjustment. Step 2 uses the existing SFM algorithm to process the multi-view fruit tree image sequence. The input consists of N fruit tree images with overlapping views. The processing includes image feature extraction, inter-image feature matching, mismatch removal, camera pose estimation, 3D feature point triangulation, and bundle adjustment optimization. The final output includes the camera intrinsic and extrinsic parameters for each image, as well as a sparse 3D point cloud used to initialize the 3DGS scene.

[0027] The motion reconstruction method is an existing multi-view geometric processing method, and this invention does not improve upon it. The purpose of this step is to provide camera geometric parameters and initial spatial coordinates for subsequent 3D Gaussian scene initialization, and for the projection correspondence between the 2D semantic mask, the 2D displacement vector field, and the 3D Gaussian points.

[0028] Output camera intrinsic parameter set pose set and the initial sparse point cloud set ; Step 3, extract a single image As input, the image is fed into a pre-trained SAM model for two-dimensional segmentation and outputs a single image. Target 2D segmentation results semantic mask of the image ,in Representing an image The number of fruit instances identified; Step 4: For each two-dimensional fruit instance region obtained in Step 3, calculate its two-dimensional geometric center, and calculate the displacement vector of each foreground pixel within the instance region pointing to the corresponding two-dimensional geometric center. Output a two-dimensional displacement vector field with the same size as the original image. For the k-th fruit instance region in the n-th image... Its two-dimensional geometric center is: (1) in, Represents the pixel coordinates within the instance region. This indicates the number of pixels contained in the instance. For any pixel u within the instance region, its corresponding two-dimensional displacement vector is: (2) Therefore, the output of step 4 is not a single fruit center coordinate, but a two-dimensional dual-channel displacement vector field, where each foreground pixel stores a horizontal displacement and a vertical displacement; the displacement vector of the background pixels is set to zero. Output Corresponding 2D displacement vector field Finally, the 2D displacement vector field corresponding to the nth image is output. ; The two-dimensional displacement vector field corresponds one-to-one with the pixel positions in the original image. For any foreground pixel position u, the displacement vector stored at that position... The starting point is pixel u, and the ending point is the two-dimensional geometric center c of the fruit instance to which that pixel belongs. The two satisfy: Within the same fruit instance region, the displacement vectors of different pixels have different starting points, but their ending points all point to the same instance center; pixels of different fruit instances point to their respective centers. Thus, the two-dimensional displacement vector field forms a centripetal distribution within each fruit region, providing supervision for subsequently determining which instance center a three-dimensional Gaussian point should shift towards from the current viewpoint.

[0029] Step 5, final output image sequence The corresponding full-view 2D segmentation map set 2D semantic mask set With 2D displacement vector field set .

[0030] Example 3 The specific process of the second stage is as follows: Step 6, Define the initial multi-attribute Gaussian set Each Gaussian unit .

[0031] Step 7, initialize Gaussian properties and set the iteration threshold: 1. Initialize as a point cloud Three-dimensional spatial coordinates; 2. Rotation parameters Initialize to a unit quaternion Scaling parameters Adaptive initialization based on the average physical distance of the three nearest neighbors; Initialize to ; 3. Initialize to ; 4. The zero-order coefficients are initialized with the original color of the point cloud and the higher-order coefficients are set to 0; 5. Initialize to a neutral scale of 0.5; 6. Initialize as zero vector ; 7. Set the current iteration counter. And preset the maximum iteration threshold.

[0032] Step 8, Optimization process: 1. From Extract a pair of visual semantic co-occurring Gaussians that share a geometric space but are functionally decoupled, namely the set of appearance radiating Gaussians. (Among them, the external radiating sub-Gauss) ) and semantic location sub-Gaussian sets (Among them, semantic localization sub-Gaussian) ); 2. , and The data is fed into the color path rasterization projection rendering process to obtain and output a set of predicted color maps from N viewpoints. ; 3. , and The data is fed into the semantic path rasterization projection rendering process to obtain and output a set of predicted semantic masks from N viewpoints. ; 4. , and , The data is fed into the loss function calculation unit to calculate the total loss function for dual-channel supervision multi-objectives. ; 5. The error is fed into the dynamic iterative optimizer to perform backpropagation, and the properties of the existing Gaussian are adaptively updated according to the gradient to obtain a new set of Gaussians. .

[0033] Step 9, determine the optimization conditions: Check if it meets the requirements If not satisfied, Return to step 8 for rendering, iteration counter If satisfied, terminate the loop and proceed to step 10. Step 10, output the optimized scene Gaussian. (Among them, the optimized scene Gaussian unit) ).

[0034] Example 4 The specific process of the third stage is as follows: Step 11, from the scene Gaussian set Extracting the Gaussian set of fruits: setting a semantic threshold According to Gaussian semantic properties Is it greater than the threshold? The Gaussians in the scene optimization Gaussian set are filtered to obtain the fruit Gaussian set. The selected fruit Gaussian units are ; Step 12, Set the current iteration counter And preset the maximum iteration threshold. ; Step 13, Optimization process: 1) Place the fruit Gauss Basic 3D coordinates Camera internal parameters Camera external parameters and 2D displacement vector field set As input, the Gaussian points are projected onto the Normalized Device Coordinates (NDC) system using a full perspective projection matrix transformation, and a bilinear mesh sampling function is executed to output a set of 3D virtual camera target rays. ; (3) in, Indicates based on camera intrinsic parameters and external references The projection matrix transformation function transforms the three-dimensional Gaussian coordinates. Map to the normalized device coordinate system.

[0035] Next, normalized device coordinates are used. Calling the bilinear sampling function from the 2D displacement vector field Extract the two-dimensional offset vector towards the center of the fruit and combine it with the camera intrinsic inverse matrix. Calculate 3D target rays : (4) in, This represents the two-dimensional centripetal displacement obtained from sampling. Represents the bilinear grid sampling function. This represents the vector normalization operator.

[0036] 2) Project the Gaussian coordinates onto the camera coordinate system using the view transformation matrix, and calculate and extract their values. The absolute value of the axis is used to obtain the absolute line-of-sight depth. ; (5) in, This indicates that the x, y, and z spatial coordinate components in a three-dimensional coordinate system are taken, and the transformed coordinates are... Absolute line-of-sight depth is ; 3) , and As input, directly calculate the total loss function of topological deformation. ; Total loss function of topological deformation It is a weighted sum consisting of target ray traction loss, scale-adaptive depth loss, and elastic constraints. The specific calculation formula is as follows: (6) in, Indicates the target's ray-driven loss; This represents scale-adaptive depth loss; This represents the loss due to elastic constraints; These are the corresponding hyperparameter weights.

[0037] The specific formula for calculating the target traction loss is as follows: (7) (8) (9) in, Indicates target traction loss; For the fruit Gaussian set The total number of Gaussian units in the array; For the j-th fruit Gauss Adaptive weights; For the j-th fruit Gauss The three-dimensional coordinates projected onto the camera coordinate system after physical collapse. ; For the j-th fruit Gauss The corresponding 3D virtual camera target ray; For the j-th fruit Gauss semantic attributes, For the fruit Gauss The magnitude of the corresponding two-dimensional offset vector towards the center of the fruit, , and It is the j-th fruit Gauss Horizontal and vertical displacement components in the image coordinate system The magnitude of the maximum two-dimensional displacement vector. .

[0038] The formula for calculating scale-adaptive depth loss is: (10) (11) in, This represents scale-adaptive depth loss; For the fruit Gaussian set The total number of Gaussian units in the array; For the j-th fruit Gauss Adaptive weights; , represents the actual collapse distance generated inward along the line of sight of the j-th Gaussian point; This represents the absolute depth of the point in the camera coordinate system after it collapses. This represents the absolute line-of-sight depth before optimization. For the j-th fruit Gauss The estimated lateral physical span mapped to three-dimensional physical space; and These represent the horizontal and vertical pixel components of the two-dimensional offset vector on the image plane, respectively. and These represent the camera's horizontal and vertical focal lengths, respectively, and are considered internal camera parameters. The parameters in.

[0039] The formula for calculating elastic constraints is: (12) in, This represents the loss due to elastic constraints; For the fruit Gaussian set The total number of Gaussian units in the array; For the j-th fruit Gauss Adaptive weights; For the j-th fruit Gauss The three-dimensional relative offset; This indicates the calculation of the square value of the L2 norm.

[0040] 4) The input error backpropagation optimizer only updates the three-dimensional relative offset of the fruit Gaussian. ; Step 14, determine the optimization conditions: Check if it meets the requirements If the condition is not met, then set the iteration counter. Return to step 3 for iterative optimization; if satisfied, terminate the loop and proceed to step 15. Step 15, output the optimized Gaussian fruit. (Among them, the optimized fruit Gaussian unit) ).

[0041] Example 5 The specific process of the fourth stage is as follows: Step 16, and The calculation is performed by superposition, and the coordinates of the Gaussian center point of the new fruit are output. ,in ; Obtain the set of physically collapsed point clouds

[0042] Step 17, Together with the set neighborhood radius parameter The input is fed into the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm, and the output is the initial set of point clusters. ; Step 18, Calculate the initial set of point clusters. Each point cluster Maximum three-dimensional physical span : (13) Will Compared with the preset maximum size threshold for a single fruit By comparing, the span was separated. single fruit point cluster set and span Multi-fruit adhesion clusters ; Step 19, aggregate multiple fruit clusters Use a smaller neighborhood radius ( The DBSCAN algorithm is executed again for clustering, thereby splitting the originally clustered fruit clusters into independent sets of fruit point clusters. ; Step 20: Set the single fruit clusters. With the split clustered fruit point cluster set Take the union of the sets and merge them to obtain the final set of independent fruit point clusters. traversal Each independent point cluster in Calculate all physical collapse points within the cluster. Spatial arithmetic mean coordinates : (14) Output the set of coordinates of the center point of each fruit. Each of them The three-dimensional physical center of mass represents a real, independent fruit; Step 21, Input the set of independent center point coordinates Perform fruit counting and statistics, and finally output the total number of fruits. .

[0043] In a natural orchard environment, fruits often overlap and are densely packed together. Furthermore, the continuous boundaries of the 3D Gaussian reconstruction make it difficult for traditional semantic filtering and density clustering to accurately separate adjacent fruit instances. This can lead to problems such as multiple fruits being merged and counted, or a single fruit being excessively fragmented and counted.

[0044] This invention, without the need for manual labeling of the center of three-dimensional fruit instances and three-dimensional instance labels, utilizes multi-view two-dimensional semantic masks and two-dimensional centripetal displacement vector fields to guide the three-dimensional Gaussian units on the surface of the fruit to shift relative to the internal center region of their respective fruits. This creates spatial gaps between the originally adhered fruits that can be clustered and identified. By combining semantic connectivity, prior fruit size, and hierarchical clustering, it achieves accurate separation and counting of dense fruit instances.

[0045] Example 6 like Figure 2The figure shows the 2D prior extraction results of the method of the present invention on a public dataset. Figure 2 (I) is the input fruit tree image. Figure 2 (II) is the semantic mask result of the fruit obtained through the two-dimensional semantic segmentation model. Figure 2 (III) is the two-dimensional displacement vector field calculated based on the two-dimensional fruit instance region. Figure 2 (Ⅳ) is a magnified view of a local area of ​​the two-dimensional displacement vector field. From Figure 2 It can be seen that two-dimensional semantic segmentation can effectively extract different fruit instance regions, while the two-dimensional displacement vector field can describe the directional relationship between pixels in the fruit region and the corresponding instance center, providing constraints for subsequent mapping of two-dimensional prior information to three-dimensional Gaussian space.

[0046] like Figure 3 As shown, this is a comparison of the original point cloud and the point cloud after topological collapse on a public dataset. Figure 3 (I) shows the original surface point cloud result. Figure 3 (II) The topologically collapsed point cloud result generated using the present invention with three-dimensional relative offset. It can be seen that adjacent fruits in the original point cloud are prone to spatial connectivity problems; after optimization, the Gaussian points of the same fruit cluster towards the center, forming obvious spatial intervals.

[0047] like Figure 4 As shown, this is the 2D prior extraction result from three different angles on a self-built dataset. Even in a natural orchard where branches and leaves occlude and overlap, the 2D semantic information can still effectively locate the fruit region and generate stable center direction constraints.

[0048] like Figure 5 The results shown are subjective experimental results of the method of this invention on a self-built dataset. From left to right: RGB scene rendered by 3D Gaussian, point cloud with semantic information fused, instance segmentation result based on the original surface point cloud (19 fruits), and instance segmentation result based on the topologically collapsed point cloud of this invention (24 fruits). This method enables effective separation of fruits under dense and occluded conditions, significantly improving the counting accuracy.

Claims

1. A 3DGS fruit instance counting method based on class center perception, characterized in that: Specifically, the steps include the following: Step 1: Collect multi-view image sequences of fruit trees and perform preprocessing to extract two-dimensional prior information; Step 2: Based on the image preprocessed in Step 1, initialize and render a multi-attribute scene 3DGS model by combining semantic information. Step 3: Using two-dimensional prior information, calculate and optimize the 3DGS three-dimensional relative offset from the center of the fruit in the generated scene; Step 4: Based on the 3DGS three-dimensional relative offset obtained in Step 3, perform physical collapse and hierarchical clustering, and output the final fruit instance count result.

2. The 3DGS fruit instance counting method based on class center perception according to claim 1, characterized in that: The specific process of step 1 is as follows: Step 1.1: Input a sequence of N fruit tree images taken from different perspectives. ; Step 1.2, image sequence As input, the Structure for Motion Restoration (SFM) algorithm is called to perform feature matching and bundle adjustment calculation, and the camera intrinsic parameters, camera extrinsic parameters, and sparse 3D point cloud used to initialize the 3DGS scene are output for each image. Step 1.3, extract a single image As input, the data is fed into a trained 2D image segmentation model for instance segmentation, and the output is a single image. Target 2D segmentation results With the corresponding semantic mask ,in Representing an image The number of fruit instances identified; Step 1.4: Calculate the two-dimensional geometric center for each two-dimensional fruit instance region obtained in Step 1.3, and calculate the displacement vector of each foreground pixel in the instance region pointing to the corresponding two-dimensional geometric center, and output a two-dimensional displacement vector field with the same size as the original image. Step 1.5: Integrate the processing results and output the image sequence. The corresponding full-view 2D segmentation map set 2D semantic mask set With 2D displacement vector field set .

3. The 3DGS fruit instance counting method based on class center perception according to claim 2, characterized in that: In step 1.2, the output specifically includes a set of camera intrinsic parameters. Camera pose extrinsic parameter set and the initial sparse point cloud set .

4. The 3DGS fruit instance counting method based on class center perception according to claim 3, characterized in that: In step 1.4, for the k-th fruit instance region in the n-th image... The formula for calculating its two-dimensional geometric center is: （1） in, Represents the pixel coordinates within the instance region. This indicates the number of pixels contained in the instance. For any pixel u within the instance region, the formula for calculating the corresponding two-dimensional displacement vector is: （2） Each foreground pixel stores a horizontal displacement and a vertical displacement; the displacement vectors of the background pixels are all set to zero vectors. The final output is the 2D displacement vector field corresponding to the nth image. .

5. The 3DGS fruit instance counting method based on class center perception according to claim 4, characterized in that: The specific process of step 2 is as follows: Step 2.1, Define the initial multi-attribute Gaussian set. Each Gaussian unit Including three-dimensional coordinates Covariance and scaling / rotation parameters Opacity spherical harmonic coefficient Semantic probability attributes and 3D relative offset attribute And define the optimized Gaussian set as Each Gaussian unit With Same attribute type, This represents the total number of Gaussian cells after adaptive optimization. Step 2.2: Initialize all Gaussian properties, set the maximum iteration threshold, and perform model training and property optimization; Step 2.3: Determine whether the optimization convergence condition is met. If it is met, output the optimized 3DGS scene model.

6. The 3DGS fruit instance counting method based on class center perception according to claim 5, characterized in that: The specific process of step 2.2 is as follows: Step 2.2.1, convert the three-dimensional spatial coordinates Initialize to the initial sparse point cloud 3D coordinates; Step 2.2.2, set the rotation parameters Initialize to a unit quaternion ; Scaling parameters Adaptive initialization based on the average physical distance of the three nearest neighbors; covariance Initialize to ; Step 2.2.3, adjust the opacity. Initialize to ; Step 2.2.4, determine the spherical harmonic coefficients. The zero-order coefficients are initialized by the original color of the point cloud, and all higher-order coefficients are set to 0; Step 2.2.5, transfer the semantic probability attribute Initialize to a neutral scale; Step 2.2.6, perform three-dimensional relative offset Initialize as zero vector ; Step 2.2.7, Set the current iteration counter And preset the maximum iteration threshold. ; Step 2.2.8, from the multi-attribute Gaussian set Extract a pair of visual semantic co-occurring Gaussians that share a geometric space but are functionally decoupled, namely the set of appearance radiating Gaussians. With semantic location sub-Gaussian set ; Step 2.2.9, will and camera internal parameters With camera external parameters The data is fed into the color branch rasterization projection renderer to obtain and... Output predicted color map set from each perspective ; Step 2.2.10, will and camera internal parameters With camera external parameters The data is fed into the semantic branch rasterization projection renderer, and then output to... A set of semantic masks predicted from each perspective ; Step 2.2.11, predict the color map set Predictive semantic mask set With real image set True semantic mask set The data is fed into the loss function calculation unit to calculate the total loss function for dual-channel supervision multi-objectives. ; Step 2.2.12, convert the total loss function The error is backpropagated in the dynamic iterative optimizer, and the properties of the existing Gaussian are adaptively updated according to the gradient, resulting in the intermediate Gaussian set after a single iteration. .

7. The 3DGS fruit instance counting method based on class center perception according to claim 6, characterized in that: The specific process of step 2.3 is as follows: Step 2.3.1: Check if the current iteration counter satisfies the condition. If not satisfied, then let , the intermediate Gaussian set As a new multi-attribute Gaussian set Return to step 2.2.8 for the next iteration; if the condition is met, terminate the loop. Step 2.3.2: Select the intermediate Gaussian set that currently meets the conditions. Assign to And output the final optimized 3DGS scene model. Proceed to step 3.

8. The 3DGS fruit instance counting method based on class center perception according to claim 7, characterized in that: The specific process of step 3 is as follows: Step 3.1, from the scene Gaussian model Extracting the fruit Gaussian set: A semantic threshold Thr is set, and Gaussian units in the scene Gaussian model are filtered based on whether the Gaussian semantic probability attribute is greater than the threshold Thr to obtain the fruit Gaussian set. The selected fruit Gaussian units are ; Step 3.2, Set the current iteration counter And preset the maximum iteration threshold. ; Step 3.3, transfer the fruit Gaussian Basic 3D coordinates Camera internal parameters Camera external parameters and 2D displacement vector field set As input, the Gaussian points are projected onto the normalized device coordinate system using a full perspective projection matrix transformation, and bilinear mesh sampling is performed to obtain the set of target rays for the 3D virtual camera. ; （3） in, Indicates based on camera intrinsic parameters and external references The projection matrix transformation function transforms the three-dimensional Gaussian coordinates. Map to the normalized device coordinate system; Next, normalized device coordinates are used. Calling the bilinear sampling function from the 2D displacement vector field Extract the two-dimensional offset vector towards the center of the fruit and combine it with the camera intrinsic inverse matrix. Calculate 3D target rays : （4） in, This represents the two-dimensional centripetal displacement obtained from sampling. Represents the bilinear grid sampling function. Represents the vector normalization operator; Step 3.4, transfer the fruit Gaussian 3D coordinates Project the view transformation matrix onto the camera coordinate system, calculate and extract... The absolute value of the axis is used to obtain the absolute line-of-sight depth. ; （5） in, This indicates that the x, y, and z spatial coordinate components in a three-dimensional coordinate system are taken, and the transformed coordinates are... Absolute line-of-sight depth is ; Step 3.5, set the target rays 3D relative offset and absolute depth of vision As input, calculate the total loss function of topological deformation. ; Step 3.6, calculate the total loss function of topological deformation. The data is fed into the error backpropagation optimizer, which only updates the three-dimensional relative offset of the fruit Gaussian. ; Step 3.7, check if the current iteration counter satisfies the condition. If not satisfied, then let Then return to step 3.3 for the next round of loop optimization; if satisfied, terminate the loop and proceed to step 3.

8. Step 3.8, output the optimized Gaussian set of fruits. .

9. The 3DGS fruit instance counting method based on class center perception according to claim 8, characterized in that, In step 3.5, the total loss function of topological deformation It is a weighted sum consisting of target ray traction loss, scale-adaptive depth loss, and elastic constraints. The specific calculation formula is as follows: （6） in, Indicates the target's ray-driven loss; This represents scale-adaptive depth loss; This represents the loss due to elastic constraints; These are the corresponding hyperparameter weights; The specific formula for calculating the target traction loss is as follows: （7） （8） （9） in, Indicates target traction loss; For the fruit Gaussian set The total number of Gaussian units in the array; For the j-th fruit Gauss Adaptive weights; For the j-th fruit Gauss The three-dimensional coordinates projected onto the camera coordinate system after physical collapse. ; For the j-th fruit Gauss The corresponding 3D virtual camera target ray; For the j-th fruit Gauss semantic attributes, For the fruit Gauss The magnitude of the corresponding two-dimensional offset vector towards the center of the fruit, , and It is the j-th fruit Gauss Horizontal and vertical displacement components in the image coordinate system The magnitude of the maximum two-dimensional displacement vector. ; The formula for calculating scale-adaptive depth loss is: （10） （11） in, This represents scale-adaptive depth loss; For the fruit Gaussian set The total number of Gaussian units in the array; For the j-th fruit Gauss Adaptive weights; , represents the actual collapse distance generated inward along the line of sight of the j-th Gaussian point; This represents the absolute depth of the point in the camera coordinate system after it collapses. This represents the absolute line-of-sight depth before optimization. For the j-th fruit Gauss The estimated lateral physical span mapped to three-dimensional physical space; and These represent the horizontal and vertical pixel components of the two-dimensional offset vector on the image plane, respectively. and These represent the camera's horizontal and vertical focal lengths, respectively, and are considered internal camera parameters. Parameters in; The formula for calculating elastic constraints is: （12） in, This represents the loss due to elastic constraints; For the fruit Gaussian set The total number of Gaussian units in the array; For the j-th fruit Gauss Adaptive weights; For the j-th fruit Gauss The three-dimensional relative offset; This indicates the calculation of the square value of the L2 norm.

10. The 3DGS fruit instance counting method based on class center perception according to claim 9, characterized in that: The specific process of step 4 is as follows: Step 4.1, transfer the fruit Gaussian Basic 3D coordinates relative offset with the optimized 3D By performing vector superposition calculations, the coordinates of the Gaussian center point of the collapsed fruit are obtained. , Output a set of physically collapsed point clouds. ; Step 4.2, assemble the physical collapse point cloud. Together with the set neighborhood radius parameter Input the DBSCAN algorithm and output the initial set of point clusters. ; Step 4.3, Calculate the initial set of point clusters. Each point cluster Maximum three-dimensional physical span : （13） Will Compared with the preset maximum size threshold for a single fruit By comparing, the span was separated. single fruit point cluster set and span Multi-fruit adhesion clusters ; Step 4.4, assembling multi-fruit adhesion clusters Use a smaller neighborhood radius , The DBSCAN algorithm is executed again for clustering, thereby breaking down the originally clustered fruit clusters into independent sets of fruit point clusters. ; Step 4.5: Set the single fruit clusters together. With the split clustered fruit point cluster set Take the union of the sets and merge them to obtain the final set of independent fruit point clusters. traversal Each independent point cluster in Calculate all physical collapse points within the cluster. Spatial arithmetic mean coordinates : （14） Output the set of coordinates of the center point of each fruit. Each of them The three-dimensional physical center of mass represents a real, independent fruit; Step 4.6: Calculate the set of coordinates of the center points of independent fruits. The number of elements is used to output the total number of fruits. .

Images