Interactive referential semantic segmentation and volume measurement method for complex geometry

Abstract

This invention provides an interactive semantic segmentation and volume measurement method for complex geometric structures, belonging to the field of image processing technology. To address the problem of excessive reliance on manual annotation for text input in current semantic segmentation methods, this invention introduces the SAM model and the Describe Anything model to provide a preliminary segmentation foundation for subsequent processing and generate complex and detailed natural language descriptions of regions of interest. Based on this, a feature extraction module is designed, using a text encoder to extract word features and utilizing MLP to obtain Gaussian positional features, laying the foundation for subsequent feature fusion steps. To address the lack of spatial logical reasoning capabilities in previous methods, this invention designs a feature fusion module for describing complex spatial relationships. It utilizes positional and word features to optimize the semantic feature parameters in the Gaussian field, thereby obtaining spatially aware semantic features to support higher-level natural language semantic parsing.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an interactive semantic segmentation and volume measurement method for complex geometric structures. Background Technology

[0002] Currently, two existing technologies with similar objectives to this invention are as follows: 1. Point cloud segmentation method based on geometric feature clustering The core idea of ​​this method is to use the continuous curved surface features of an object's surface and the geometric differences between the ground and other backgrounds to cluster and segment 3D point cloud data. First, noise in the point cloud is removed using a filtering algorithm. Then, geometric features such as the normal vector and curvature of the point cloud are extracted. After setting feature thresholds, Euclidean clustering or region growing algorithms are used to divide point cloud clusters with similar geometric features into "target regions," while the remaining point clouds are classified as background regions. Finally, the segmented target point clouds are reconstructed and their volume is calculated. This method requires no manual intervention and can achieve end-to-end automated measurement.

[0003] 2. A point cloud semantic segmentation method based on supervised learning This method relies on deep learning technology to construct a semantic segmentation model based on classic point cloud segmentation networks such as PointNet and PointCNN. First, a large dataset of labeled point clouds containing different targets is collected. The coordinate and color information of the point clouds are used as input to the model, and supervised training enables the model to learn feature representations for different targets. In practical applications, the point cloud of the grain pile to be processed is input into the trained model, which can automatically output the category label for each point, thereby achieving segmentation between the target region and the background region.

[0004] In the segmentation technology system related to target object volume measurement, point cloud segmentation methods based on geometric feature clustering, as a traditional and classic approach, have many significant shortcomings in practical applications, as follows: 1. This method has extremely high requirements for point cloud quality, often requiring a high-precision scanner to obtain clear edges, resulting in high hardware costs; and there are often no suitable deployment locations, making deployment difficult.

[0005] 2. When there are local depressions or obstacles on the surface of an object, the consistency of geometric features will be disrupted, which can easily lead to missed or false detections in segmentation, resulting in a decrease in the accuracy of volume measurement.

[0006] 3. The transition between the target area and the background area often has a blurred boundary. It is difficult to accurately define the segmentation boundary by relying solely on geometric feature thresholds, resulting in boundary offset in the segmented target point cloud, which directly affects the accuracy of subsequent volume measurement.

[0007] 4. Lacking semantic understanding of "what is equipment" and "what is background", it cannot automatically distinguish between target objects and non-target objects. All subsequent cleaning and judgment work relies on human experience, which greatly reduces the degree of automation.

[0008] Supervised learning-based point cloud semantic segmentation methods, as the mainstream intelligent segmentation schemes, have improved their adaptability to complex scenes by relying on deep learning, but they still have many significant shortcomings in practical applications, as follows: 1. The performance of depth sensing devices is easily affected by the environment. For example, dust floating in the air can scatter laser light or interfere with optical measurements; the surface texture of the target object is simple and has low reflectivity, which may result in sparse feature points.

[0009] 2. A large amount of point cloud data covering target objects in different environments needs to be collected in advance and accurately labeled. The labeling process requires professional personnel, which is time-consuming and labor-intensive, with extremely high labor and time costs.

[0010] 3. The performance of the segmentation model directly depends on the accuracy of the labeled data. If there are problems such as boundary offset or category confusion during the labeling process, the model will learn the wrong features, which will lead to missed detections and false detections in segmentation, and the accuracy of subsequent volume measurement cannot be guaranteed.

[0011] 4. Deep learning-based segmentation models typically have a large number of parameters and high computational complexity, requiring high-performance hardware to meet real-time segmentation needs. Their deployment and maintenance costs are high, making them difficult to widely promote and apply. Summary of the Invention

[0012] To address the shortcomings of existing technologies, the present invention aims to propose an interactive semantic segmentation and volume calculation method for complex geometric structures. This method targets unstructured and irregular complex geometric structures and aims to achieve accurate semantic segmentation of the target region through a small amount of user-interactive annotation, thereby quickly and accurately calculating the volume of the target object based on 3D point clouds.

[0013] Interactive semantic segmentation and volume measurement methods for complex geometric structures include: Step 1: Acquire multiple multi-view images, process them using a 3D Gaussian sputtering algorithm, and obtain the 3D Gaussian distribution parameters; Step 2: Select one image from multiple multi-view images as the image to be tested, receive the prompt information drawn by the user on the image to be tested, input the image to be tested and the prompt information into the SAM segmentation model to obtain the segmentation mask of the region of interest; input the segmentation mask and the image to be tested into the DAM model to obtain the corresponding natural language description information. Step 3: Process the natural language description information and the three-dimensional Gaussian distribution parameters to obtain word feature vectors and the positional features of each three-dimensional Gaussian ellipsoid; Step 4: Introduce referential features for each 3D Gaussian ellipsoid. Optimize the referential features based on the word feature vector and the positional features of each 3D Gaussian ellipsoid to obtain the optimized referential features for each 3D Gaussian ellipsoid. Step 5: Render the target segmentation mask based on the optimized referential features and word feature vectors of each 3D Gaussian ellipsoid; Step 6: Calculate the volume of the target object within the region of interest based on the target segmentation mask.

[0014] Optionally, in step 2, the image to be measured and the prompt information are input into the SAM segmentation model to obtain the segmentation mask of the region of interest, including: The image to be measured is scaled, pixel normalized, and pixel filled to obtain a preprocessed image. The preprocessed image is then input into an image encoder to obtain visual features. The prompt information drawn by the user on the image to be measured is encoded to obtain prompt features; Visual features and cue features are input into the mask decoder to obtain the segmentation mask of the region of interest.

[0015] Optionally, step 3 specifically includes: Step 3.1: Input the natural language description information into the pre-trained BERT text encoder to obtain the word feature vector W. , For the first t Word features for Dimensions T Indicates the number of word features; Step 3.2: Encode the center position of each 3D Gaussian ellipsoid to obtain the position code of each 3D Gaussian ellipsoid. This is achieved through the following formula: ; in, Indicates the first i The center position of a three-dimensional Gaussian ellipsoid Indicates the first i The positional encoding of a three-dimensional Gaussian ellipsoid, where L is the frequency order of the encoding; Step 3.3: Input the position code into the multilayer perceptron to obtain the position features of each 3D Gaussian ellipsoid. .

[0016] Optionally, step 4 specifically includes: Step 4.1: Introduce referential features for each 3D Gaussian ellipsoid f r,i and will f r,iMapping to a high-dimensional space, the mapped features are fused with their corresponding location features to obtain location-aware features. f p,r,i ; Step 4.2: Group all three-dimensional Gaussian ellipsoids to obtain multiple instances and the set of three-dimensional Gaussian ellipsoids corresponding to each instance; Step 4.3: Use an attention mechanism to generate weight coefficients for each 3D Gaussian ellipsoid. Perform weighted aggregation on all 3D Gaussian ellipsoids in the set corresponding to the same instance to obtain the location-aware instance-level features. Specifically, this is achieved through the following formula: ; in, Representation of instances j The number of three-dimensional Gaussian ellipsoids in the corresponding set of three-dimensional Gaussian ellipsoids. Representation of instances j The corresponding set of three-dimensional Gaussian ellipsoids n The weighting coefficients, Representation of instances j The corresponding set of three-dimensional Gaussian ellipsoids n Location-aware features; Step 4.4: Calculate location-aware instance-level features and features of each word correlation coefficient Specifically, it is calculated using the following formula: ; Step 4.5: Based on the correlation coefficient Location-aware instance-level features Features of words By fusing the data, enhanced instance features are obtained. Specifically, this is achieved through the following formula: ; Step 4.6: Broadcast enhanced instance features through a feature broadcasting mechanism. Back-projecting onto each 3D Gaussian ellipsoid yields the optimized referential features for each 3D Gaussian ellipsoid. Specifically, this is achieved through the following formula: ; in, Representing a three-dimensional Gaussian ellipsoid n The referential characteristics, Represents a combination operation. This represents a learnable transformation layer.

[0017] Optionally, step 4.2 specifically includes: Based on the characteristics of reference f r,i and location features Calculate the joint characteristics of a three-dimensional Gaussian ellipsoid. Specifically, this is achieved through the following formula: ; in, β and λ The weighting coefficient, symbol This refers to the splicing operation of features; The similarity distance between two 3D Gaussian ellipsoids is calculated based on joint features, specifically using the following formula: ; in, For a three-dimensional Gaussian ellipsoid i and the three-dimensional Gaussian ellipsoid l Similarity distance, Representing a three-dimensional Gaussian ellipsoid l The combined features; All 3D Gaussian ellipsoids are grouped according to similarity distance to obtain multiple instances and a set of 3D Gaussian ellipsoids corresponding to each instance.

[0018] Optionally, step 5 specifically includes: Calculate the similarity score between the reference feature and the word feature vector after optimization for each 3D Gaussian ellipsoid. Specifically, this is achieved through the following formula: ; in, Representing a three-dimensional Gaussian ellipsoid i Optimized referential features; Similarity score Perform rasterization to obtain the target segmentation mask. M ( v Specifically, this is achieved through the following formula: ; in, v This represents a pixel on a 2D image plane, and N represents the total number of three-dimensional Gaussian ellipsoids. Indicates the first i A three-dimensional Gaussian ellipsoid in pixels v Opacity at the location, Indicates the first u A three-dimensional Gaussian ellipsoid in pixels v Opacity at that location.

[0019] Optionally, step 6 specifically includes: Step 6.1: Back-project the pixels covered by the target segmentation mask into three-dimensional space to obtain the three-dimensional point set P; Step 6.2: Scan the bottom reference height of the target object within the region of interest drawn by the user. Based on the bottom reference height and the three-dimensional point set, calculate the volume of the target object using the integration method.

[0020] The beneficial effects of adopting the above technical solution are as follows: To address the problem of excessive reliance on manual annotation in current semantic segmentation methods, this invention introduces the SAM model and the Describe Anything model to provide a preliminary segmentation foundation for subsequent processing and generate complex and detailed natural language descriptions of regions of interest. Based on this, a feature extraction module is designed, which uses a text encoder to extract word features and utilizes MLP to obtain positional features, laying the foundation for subsequent feature fusion steps.

[0021] To address the problem that previous methods lacked spatial logical reasoning capabilities, this invention designs a feature fusion module for complex spatial relationship descriptions in referential segmentation, such as "the grain pile closest to...", "the cabinet to the right of the door", and "the ore pile behind the ore cart". This module optimizes the referential features in the Gaussian field using positional and word features, thereby obtaining referential features with spatial awareness to support higher-level natural language referential parsing.

[0022] This invention proposes to apply referential semantic segmentation to the volume measurement of complex geometric structures, breaking through the limitations of previous segmentation methods that could only be achieved through simple semantic categories. By introducing referential semantic information, the segmentation model can more accurately identify and distinguish different geometric structures. Even in complex scenarios with tightly stacked structures, irregular shapes, or blurred boundaries, it can still achieve high-precision segmentation results and improve measurement efficiency. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the interactive semantic segmentation and volume measurement method for complex geometric structures in an embodiment of the present invention. Figure 2 This is a schematic diagram of the data input and description generation process in an embodiment of the present invention; Figure 3 This is a schematic diagram of the feature extraction process in an embodiment of the present invention; Figure 4 This is a schematic diagram of the feature fusion process in an embodiment of the present invention; Figure 5 This is a schematic diagram of the segmentation mask generation process in an embodiment of the present invention; Figure 6 This is a schematic diagram of the volume calculation process in an embodiment of the present invention; Figure 7 This is a schematic diagram of the grain pile inside the grain warehouse in an embodiment of the present invention; Figure 8 This is a schematic diagram of the grain pile segmentation result in an embodiment of the present invention; Figure 9 This is a schematic diagram of the segmentation performance prediction results in an embodiment of the present invention; Figure 10 This is a comparison chart of the estimated volume result and the actual volume result in an embodiment of the present invention. Detailed Implementation

[0024] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0025] To address the problems of existing technologies, this invention provides an interactive semantic segmentation and volume calculation method for complex geometric structures. It aims to segment target objects in static scenes through referential descriptions, while simultaneously solving the problem of high reliance on manual natural language descriptions in existing semantic segmentation techniques. Finally, the volume of the target object is calculated using the segmentation results. This method has applications in multiple fields. For example, in large-scale smart warehousing, it can be used to measure the volume of various irregular piles such as grain piles, ore piles, and sand and gravel piles, enabling efficient statistics on resource reserves. In agricultural planting, it can be used to calculate the volume of silage piles, straw recycling piles, and fertilizer raw material piles, aiding in the management of agricultural production materials.

[0026] This invention takes an image as input and generates a segmentation mask and detailed natural language description of the user's region of interest based on simple user interaction prompts (such as dots and boxes). It also extracts word features using a text encoder. Taking multi-view images as input, Gaussian parameters are initialized using 3D Gaussian sputtering technology, and positional parameters are input to an MLP to extract positional features. Subsequently, the positional and word features are embedded sequentially into the referential features of a 3D Gaussian field, optimizing the referential feature parameters to obtain more accurate semantic information. Then, a segmentation mask is rendered using the word features and the optimized referential features to segment the user's region of interest, achieving accurate segmentation of the target region in the image. Finally, the corresponding 3D point set is extracted using the segmented mask, and the volume of the target object is calculated using an integral method.

[0027] This invention is based on 3D Gaussian sputtering technology. In standard 3DGS, each Gaussian point is defined by its center position x, rotation r, scaling s, opacity α, and color c. The center position defines the center point of the ellipsoid in three-dimensional space, represented as (x, y, z) coordinates; the rotation parameter determines the orientation of the ellipsoid in space, using quaternions (q...). w ,q x ,q y,q z The scaling parameter is expressed as the scaling factor (s) in the three axes. x ,s y ,s z Opacity determines the degree to which light is blocked when passing through a point; it is a scalar between 0 and 1. The color parameter stores a set of coefficients for spherical harmonic functions, controlling how color changes with the viewing angle. Based on this, an additional denotative characteristic parameter f is introduced. r,i It is represented by a 16-dimensional eigenvector.

[0028] To achieve the goal of obtaining the segmentation mask and calculating the volume, this invention also introduces the SegmentAnything Model (SAM) to obtain the instance-level segmentation mask of the image, and utilizes the DescribeAnything Model to provide a detailed description of the instance. The technical approach is as follows: First, a 3D Gaussian semantic field is constructed by introducing referential feature parameters to describe various semantic information within a 3D Gaussian field. Second, a data input and description generation module is designed to autonomously generate detailed natural language descriptions using multi-view images as input and initialize the 3D Gaussian parameters. Third, a feature extraction module is designed to extract word features from the generated natural language descriptions and input the Gaussian position parameters into an MLP to obtain the position features of each Gaussian. Fourth, a feature fusion module is designed to embed position features and word features into the referential features of the semantic field to obtain optimized referential feature parameters. Fifth, a segmentation mask generation module is designed to generate a Gaussian-language similarity score map using word features and optimized referential features. The score is then rasterized to obtain the segmentation mask for a given instance. Finally, the segmentation mask is projected onto a 3D space, and its volume is calculated using an integral method.

[0029] Specifically, in combination Figure 1 This may include the following steps: Step 1: Combining Figure 2 Multiple multi-view images are acquired, and the 3D Gaussian Splatting algorithm is used to process the multiple multi-view images to obtain 3D Gaussian distribution parameters. The 3D Gaussian distribution parameters include the parameters of multiple 3D Gaussian ellipsoids, and the parameters of the 3D Gaussian ellipsoids include center position, rotation parameters, scaling parameters, opacity and color parameters. This invention acquires and inputs 20–50 multi-view images with a resolution of 1080P (1920×1080 pixels).

[0030] Step 2: Combining Figure 2The system selects one image from multiple multi-view images as the image to be measured, and receives prompts drawn by the user on the image to be measured. The user can select a bounding box on the image to be measured, and needs to provide two coordinates, the upper left and lower right. After prompting and encoding, the feature size is 2×256 (2 represents two coordinates, that is, one coordinate corresponds to 1×256). Alternatively, the user can click on the image to be measured. After prompting and encoding, each click will result in a feature size of 1×256. N points will form N×256.

[0031] The image to be measured and the prompt information are input into the SAM segmentation model to obtain the segmentation mask of the region of interest, specifically including: The image to be measured is scaled, pixel normalized, and pixel padded to obtain a preprocessed image. Specifically, the 1080P input image is scaled to 1024 on the longest side while maintaining the aspect ratio. Color normalization is performed by subtracting the average value from each pixel and dividing by the standard deviation. Finally, padding is applied to ensure that the image size is the input size of the model, i.e., 1024*1024.

[0032] The preprocessed image is input into a Vision Transformer-based image encoder to obtain visual features; The prompt information drawn by the user on the image to be measured is encoded to obtain prompt features; Visual features and cue features are input into the mask decoder to obtain the segmentation mask of the region of interest.

[0033] The segmentation mask M and the image I to be tested are input into the DAM (Describe Anything) model to obtain detailed natural language description information that includes spatial relationships and has context-aware capabilities. ; ; Step 3: Process the natural language description information and the 3D Gaussian distribution parameters to obtain word feature vectors and the positional features of each 3D Gaussian ellipsoid, and combine them with... Figure 3 Specifically, it includes the following steps: Step 3.1: Input the natural language description information into the pre-trained BERT text encoder to obtain the word feature vector W. , For the first t Word features for Dimensions (768 dimensions). T Indicates the number of word features; Step 3.2: Encode the center position of each 3D Gaussian ellipsoid to obtain the position code of each 3D Gaussian ellipsoid. This is achieved through the following formula: ; in, Indicates the first i The center position of a three-dimensional Gaussian ellipsoid Indicates the first i The positional encoding of a three-dimensional Gaussian ellipsoid, where L is the frequency order of the encoding; Step 3.3: Input the position encoding into a lightweight multilayer perceptron to learn the implicit association between spatial coordinates and semantic features, thereby obtaining the position features of each 3D Gaussian ellipsoid. , is represented as: ; in, These are the learnable parameters of the MLP.

[0034] Step 4: Introduce referential features for each 3D Gaussian ellipsoid. Based on the word feature vector and the positional features of each 3D Gaussian ellipsoid, optimize the referential features to obtain the optimized referential features for each 3D Gaussian ellipsoid. Combine this with... Figure 4 Specifically, it includes the following steps: Step 4.1: Introduce referential features for each 3D Gaussian ellipsoid f r,i and will f r,i Mapping to a high-dimensional space, the mapped features and their corresponding location features are fused to achieve spatial-semantic alignment, resulting in location-aware features. f p,r,i ; Step 4.2: Group all three-dimensional Gaussian ellipsoids to obtain multiple instances and the set of three-dimensional Gaussian ellipsoids corresponding to each instance; To address the difficulty in accurately classifying instances that are geographically close but semantically different, or semantically identical but spatially separated, this invention addresses the issue based on referential features. f r,i and location features Calculate the joint characteristics of a three-dimensional Gaussian ellipsoid. Specifically, this is achieved through the following formula: ; in, β and λ These are weighting coefficients used to balance the contributions of semantic and spatial features; their symbols are... This refers to the splicing operation of features; The similarity distance between two 3D Gaussian ellipsoids is calculated based on joint features, specifically using the following formula: ; in, For a three-dimensional Gaussian ellipsoid iand the three-dimensional Gaussian ellipsoid l The smaller the similarity distance, the more likely they belong to the same instance. Representing a three-dimensional Gaussian ellipsoid l The combined features; The DBSCAN clustering method is used to group all 3D Gaussian ellipsoids according to their similarity distance, resulting in multiple instances and a set of 3D Gaussian ellipsoids for each instance, i.e., the instance partitioning result set. Where z j Denotes the Gaussian set contained in the j-th instance; Step 4.3: Since not all Gaussian points are equally important in describing an object, for example, for a "chair" instance, the Gaussian points of the seat cushion and backrest are more semantically representative of the "chair" than the small screws at the bottom. Therefore, this invention uses an attention mechanism to generate weight coefficients for each 3D Gaussian ellipsoid, suppressing background or edge noise, and weighted aggregation of all 3D Gaussian ellipsoids in the set corresponding to the same instance to obtain position-aware instance-level features. Specifically, this is achieved through the following formula: ; in, Representation of instances j The number of three-dimensional Gaussian ellipsoids in the corresponding set of three-dimensional Gaussian ellipsoids. Representation of instances j The corresponding set of three-dimensional Gaussian ellipsoids n The weighting coefficients, Representation of instances j The corresponding set of three-dimensional Gaussian ellipsoids n Location-aware features; Step 4.4: Calculate location-aware instance-level features and features of each word correlation coefficient Specifically, it is calculated using the following formula: ; Step 4.5: Based on the correlation coefficient Location-aware instance-level features Features of words By fusing the data, enhanced instance features are obtained. Specifically, this is achieved through the following formula: ; Through the above fusion, each instance feature not only contains its own visual information such as geometric position and appearance attributes, but also explicitly encodes semantic constraints related to words in the natural language description and spatial constraints related to location.

[0035] Step 4.6: Broadcast enhanced instance features through a feature broadcasting mechanism. Back-projecting onto each 3D Gaussian ellipsoid yields the optimized referential features for each 3D Gaussian ellipsoid. Specifically, this is achieved through the following formula: ; in, Representing a three-dimensional Gaussian ellipsoid n The referential characteristics, Represents a combination operation. Represents a learnable transformation layer; The referential language field formed by the referential feature representation in the optimized 3D Gaussian field enables the 3D Gaussian to have language position awareness, allowing the model to better understand referential relationships such as "the grain pile in front of the funnel cart" and "the red cup on the table".

[0036] Step 5: Render the target segmentation mask based on the optimized referential features and word feature vectors of each 3D Gaussian ellipsoid; After the 3D Gaussian semantic field is constructed and referential features with language and location awareness are obtained, combined with... Figure 5 Calculate the similarity score between the reference features and word feature vectors after optimization for each three-dimensional Gaussian ellipsoid. Specifically, this is achieved through the following formula: ; in, Representing a three-dimensional Gaussian ellipsoid i Optimized referential features; Similarity score Perform rasterization to obtain the target segmentation mask. M ( v Specifically, this is achieved through the following formula: ; in, v This represents a pixel on a 2D image plane, and N represents the total number of three-dimensional Gaussian ellipsoids. Indicates the first i A three-dimensional Gaussian ellipsoid in pixels v Opacity at the location, Indicates the first u A three-dimensional Gaussian ellipsoid in pixels v Opacity at the location; This represents the cumulative transparency value rendered from front to back.

[0037] Step 6: Calculate the volume of the target object within the user's region of interest based on the target segmentation mask, and combine it with... Figure 6 Specifically, the following steps are included: Step 6.1: Using the depth information acquired by the depth camera, combined with the camera intrinsic parameter matrix, backproject the pixels covered by the target segmentation mask into 3D space, restoring the information from the image space to the physical coordinate system, and obtaining the 3D point set P, represented as: ; in, This represents the distance (depth) from the point to the camera. ) represents its projected position on the horizontal reference plane; Step 6.2: Scan the bottom reference height of the target object within the region of interest drawn by the user. Based on the bottom reference height and the 3D point set, calculate the volume of the target object using the integral method. Specifically, the horizontal plane (x, y) is divided into countless tiny square grids using a grid method, with the area of ​​each grid being... For each grid l, the height h of the object above it is... l Subtract the base height from the average height of all point clouds within the region: ; The total volume of the entire geometry is obtained by summing the geometric volumes within each grid. in, Let l be the average height of the point cloud in the l-th grid. The area is the unit grid area.

[0038] This invention proposes an interactive semantic segmentation and volume measurement method for complex geometries based on 3D Gaussian sputtering. This technique has the following main advantages: First, this invention addresses the problem of excessive reliance on manual annotation in current semantic segmentation methods for text input. It introduces the SAM model and the Describe Anything model to provide a preliminary segmentation foundation for subsequent processing and generate complex and detailed natural language descriptions of regions of interest. Based on this, a feature extraction module is designed, which uses a text encoder to extract word features and utilizes MLP to obtain positional features, laying the foundation for subsequent feature fusion steps.

[0039] Second, this invention addresses the problem that previous methods lacked spatial logical reasoning capabilities. For complex spatial relationship descriptions in referential segmentation, such as "the grain pile closest to...", "the cabinet to the right of the door", and "the ore pile behind the ore cart", a feature fusion module is designed. This module optimizes the referential features in the Gaussian field using positional and word features, thereby obtaining referential features with spatial awareness, supporting higher-level natural language referential parsing.

[0040] Third, this invention proposes to apply referential semantic segmentation to the volume measurement of complex geometric structures, breaking through the limitations of previous segmentation methods that could only be achieved through simple semantic categories. By introducing referential semantic information, the segmentation model can more accurately identify and distinguish different geometric structures. Even in complex scenarios with tight stacking, irregular shapes, or blurred boundaries, it can still achieve high-precision segmentation results and improve measurement efficiency.

[0041] This invention constructs an interactive referential semantic segmentation method, the framework of which is as follows: Figure 1 As shown, this invention introduces the referential segmentation method into the field of object volume measurement, providing a new approach to solving the problem of target object volume measurement.

[0042] In practice, the first step is to acquire multi-view images of the target object, which serve as the foundation for subsequent segmentation processing. The following experiment uses the calculation of grain pile volume as an example. Figure 7 One of the input grain pile images is shown. Next, the segmentation method proposed in this invention is used to process the input grain pile image, and the final segmentation result of the grain pile is as follows. Figure 8 As shown in the image, the segmentation results demonstrate that the method of this invention can accurately separate the grain pile from the background.

[0043] To comprehensively and objectively evaluate the performance of this segmentation model, we will conduct test experiments in real-world grain storage scenarios of varying sizes and grain types. These test scenarios cover different scales, from small warehouses to large grain depots, and include common grain types such as wheat, corn, and rice, ensuring the test results have broad representativeness and reliability. During the testing process, we evaluate the performance of the segmentation model based on several key metrics, including pixel accuracy, intersection-over-union ratio (IoU), and segmentation efficiency. The prediction results are as follows: Figure 9 As shown in the figure. Simultaneously, the volume results calculated by the model and the actual volume results were compared. The experimental results for a small warehouse are shown in the figure. Figure 10 As shown, the accuracy and reliability of the method of the present invention in grain volume measurement can be intuitively evaluated.

[0044] The results show that the model can achieve a high level of segmentation for complex and irregular geometric structures with small volume errors. It can accurately capture the actual shape and range of the target object and maintain high-quality segmentation results even in complex environments such as uneven lighting and dust.

[0045] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. An interactive semantic segmentation and volume measurement method for complex geometric structures, characterized in that, include: Step 1: Acquire multiple multi-view images, process them using a 3D Gaussian sputtering algorithm, and obtain the 3D Gaussian distribution parameters; Step 2: Select one image from multiple multi-view images as the image to be tested, receive the prompt information drawn by the user on the image to be tested, input the image to be tested and the prompt information into the SAM segmentation model to obtain the segmentation mask of the region of interest; input the segmentation mask and the image to be tested into the DAM model to obtain the corresponding natural language description information. Step 3: Process the natural language description information and the three-dimensional Gaussian distribution parameters to obtain word feature vectors and the positional features of each three-dimensional Gaussian ellipsoid; Step 4: Introduce referential features for each 3D Gaussian ellipsoid. Optimize the referential features based on the word feature vector and the positional features of each 3D Gaussian ellipsoid to obtain the optimized referential features for each 3D Gaussian ellipsoid. Step 5: Render the target segmentation mask based on the optimized referential features and word feature vectors of each 3D Gaussian ellipsoid; Step 6: Calculate the volume of the target object within the region of interest based on the target segmentation mask.

2. The interactive semantic segmentation and volume measurement method for complex geometric structures according to claim 1, characterized in that, In step 2, the image to be measured and the prompt information are input into the SAM segmentation model to obtain the segmentation mask of the region of interest, including: The image to be measured is scaled, pixel normalized, and pixel filled to obtain a preprocessed image. The preprocessed image is then input into an image encoder to obtain visual features. The prompt information drawn by the user on the image to be measured is encoded to obtain prompt features; Visual features and cue features are input into the mask decoder to obtain the segmentation mask of the region of interest.

3. The interactive semantic segmentation and volume measurement method for complex geometric structures according to claim 1, characterized in that, Step 3 specifically includes: Step 3.1: Input the natural language description information into the pre-trained BERT text encoder to obtain the word feature vector W. , For the first t Word features for Dimensions T Indicates the number of word features; Step 3.2: Encode the center position of each 3D Gaussian ellipsoid to obtain the position code of each 3D Gaussian ellipsoid. This is achieved through the following formula: ； in, Indicates the first i The center position of a three-dimensional Gaussian ellipsoid Indicates the first i The positional encoding of a three-dimensional Gaussian ellipsoid, where L is the frequency order of the encoding; Step 3.3: Input the position code into the multilayer perceptron to obtain the position features of each 3D Gaussian ellipsoid. .

4. The interactive semantic segmentation and volume measurement method for complex geometric structures according to claim 1, characterized in that, Step 4 specifically includes: Step 4.1: Introduce referential features for each 3D Gaussian ellipsoid f r,i and will f r,i Mapping to a high-dimensional space, the mapped features are fused with their corresponding location features to obtain location-aware features. f p,r,i ; Step 4.2: Group all three-dimensional Gaussian ellipsoids to obtain multiple instances and the set of three-dimensional Gaussian ellipsoids corresponding to each instance; Step 4.3: Use an attention mechanism to generate weight coefficients for each 3D Gaussian ellipsoid. Perform weighted aggregation on all 3D Gaussian ellipsoids in the set corresponding to the same instance to obtain the location-aware instance-level features. Specifically, this is achieved through the following formula: ； in, Representation of instances j The number of three-dimensional Gaussian ellipsoids in the corresponding set of three-dimensional Gaussian ellipsoids. Representation of instances j The corresponding set of three-dimensional Gaussian ellipsoids n The weighting coefficients, Representation of instances j The corresponding set of three-dimensional Gaussian ellipsoids n Location-aware features; Step 4.4: Calculate location-aware instance-level features and features of each word correlation coefficient Specifically, it is calculated using the following formula: ； Step 4.5: Based on the correlation coefficient Location-aware instance-level features Features of words By fusing the data, enhanced instance features are obtained. Specifically, this is achieved through the following formula: ； Step 4.6: Broadcast enhanced instance features through a feature broadcasting mechanism. Back-projecting onto each 3D Gaussian ellipsoid yields the optimized referential features for each 3D Gaussian ellipsoid. Specifically, this is achieved through the following formula: ； in, Representing a three-dimensional Gaussian ellipsoid n The referential characteristics, Represents a combination operation. This represents a learnable transformation layer.

5. The interactive semantic segmentation and volume measurement method for complex geometric structures according to claim 4, characterized in that, Step 4.2 specifically includes: Based on the characteristics of reference f r,i and location features Calculate the joint characteristics of a three-dimensional Gaussian ellipsoid. Specifically, this is achieved through the following formula: ； in, β and λ The weighting coefficient, symbol This refers to the splicing operation of features; The similarity distance between two 3D Gaussian ellipsoids is calculated based on joint features, specifically using the following formula: ； in, For a three-dimensional Gaussian ellipsoid i and the three-dimensional Gaussian ellipsoid l Similarity distance, Representing a three-dimensional Gaussian ellipsoid l The combined features; All 3D Gaussian ellipsoids are grouped according to similarity distance to obtain multiple instances and a set of 3D Gaussian ellipsoids corresponding to each instance.

6. The interactive semantic segmentation and volume measurement method for complex geometric structures according to claim 1, characterized in that, Step 5 specifically includes: Calculate the similarity score between the reference feature and the word feature vector after optimization for each 3D Gaussian ellipsoid. Specifically, this is achieved through the following formula: ； in, Representing a three-dimensional Gaussian ellipsoid i Optimized referential features; Similarity score Perform rasterization to obtain the target segmentation mask. M ( v Specifically, this is achieved through the following formula: ； in, v This represents a pixel on a 2D image plane, and N represents the total number of three-dimensional Gaussian ellipsoids. Indicates the first i A three-dimensional Gaussian ellipsoid in pixels v Opacity at the location, Indicates the first u A three-dimensional Gaussian ellipsoid in pixels v Opacity at that location.

7. The interactive semantic segmentation and volume measurement method for complex geometric structures according to claim 1, characterized in that, Step 6 specifically includes: Step 6.1: Back-project the pixels covered by the target segmentation mask into three-dimensional space to obtain the three-dimensional point set P; Step 6.2: Scan the bottom reference height of the target object within the region of interest drawn by the user. Based on the bottom reference height and the three-dimensional point set, calculate the volume of the target object using the integration method.

Images