Open vocabulary 3d instance segmentation method with geometrically stabilized multi-source semantic selection

Abstract

The application provides a geometrically stable multi-source semantic selection open vocabulary three-dimensional instance segmentation method, and relates to the technical field of three-dimensional instance segmentation. The method comprises the following steps: acquiring multi-view RGB-D images and three-dimensional point clouds of a target scene, and acquiring three-dimensional instance units with point-level support relations; generating candidate semantic lines corresponding to each semantic source on each three-dimensional instance unit, and determining semantic candidate packages of the corresponding semantic sources; extracting geometric support features and cross-source consistency features of the three-dimensional instance units corresponding to each candidate semantic line in the semantic candidate package, and determining the calibration confidence of each candidate semantic line; and segmenting the subject semantic labels of each three-dimensional instance unit from the corresponding semantic candidate package according to the observable feature vectors of the three-dimensional instance units and the calibration confidence of each candidate semantic line. The technical scheme provided by the application can realize the segmentation of open vocabulary three-dimensional instances under the influence of multi-source prediction-induced semantic fragmentation.

Description

Technical Field

[0001] This application relates to the field of 3D instance segmentation technology, and more specifically, to an open-vocabulary 3D instance segmentation method with geometrically stabilized multi-source semantic selection. Background Technology

[0002] 3D instance segmentation refers to the process of semantic classification and individual target differentiation in a 3D scene. Unlike semantic segmentation, which only labels point clouds as categories, or object detection, which only outputs bounding boxes, instance segmentation requires accurately outputting point-level masks for each independent object in 3D space. This is crucial for applications such as autonomous driving, intelligent robots, and augmented reality. With the development of sensors such as LiDAR and depth cameras, the cost of acquiring 3D data has decreased. However, the inherent sparsity, unstructured nature, and occlusion of data pose significant challenges to algorithm design. In recent years, deep learning methods have advanced this field through various paradigms, including point-based, mesh-based, and projection-based approaches. However, in complex open scenes, the model's fine perception of target contours, its generalization ability to unknown categories, and the trade-off between computational efficiency and accuracy remain key challenges that urgently need to be overcome.

[0003] Chinese patent CN119963566B discloses an open-vocabulary 3D panoramic segmentation method, device, storage medium, and program product. The method includes: acquiring camera parameter information of the desired viewpoint in a target 3D scene; inputting the camera parameter information of the desired viewpoint into a pre-defined instance segmentation network model to obtain the predicted instance segmentation result for the desired viewpoint; determining the object category corresponding to each instance number in the predicted instance segmentation result based on a pre-acquired voting matrix, where the voting matrix includes the object categories and instance numbers of multiple objects from different viewpoints; determining the predicted semantic segmentation result for the desired viewpoint based on each object category; and determining the predicted panoramic segmentation result for the desired viewpoint based on the predicted instance segmentation result and the predicted semantic segmentation result. This scheme mainly uses a multi-view information aggregation and voting mechanism to match the predicted result output by the 2D instance segmentation network with a pre-constructed voting matrix that integrates multi-view object categories and instance numbers, accurately assigning an object category to each instance number, and simultaneously generating a semantic segmentation result based on this. Finally, the panoramic segmentation result is obtained by combining instance and semantic information. In lexical 3D instance segmentation tasks, it is typically necessary to fuse multiple semantic prediction branches (such as 2D visual language models and 3D point cloud language models) to generate candidate semantic labels for each 3D instance unit. However, multi-source prediction introduces severe semantic fragmentation and conflict, meaning that candidate labels from different models for the same instance unit are discretely distributed in the text space, and their original confidence scores are incomparable due to model heterogeneity. This makes simple score aggregation prone to selecting incorrect semantic labels. Therefore, how to achieve open-vocabulary 3D instance segmentation under the influence of semantic fragmentation introduced by multi-source prediction has become a challenge for the industry. Summary of the Invention

[0004] This application provides a geometrically stabilized multi-source semantic selection method for 3D instance segmentation of open vocabularies, which can achieve 3D instance segmentation of open vocabularies under the influence of semantic fragmentation introduced by multi-source prediction.

[0005] In a first aspect, this application provides a geometrically stabilized multi-source semantic selection open-vocabulary 3D instance segmentation method, comprising the following steps: Acquire multi-view RGB-D images and 3D point clouds of the target scene, generate 2D image masks and 3D instance masks respectively, establish point-level support relationships between 2D image masks and 3D instance masks, and use 3D instance masks with point-level support relationships as 3D instance units. On each 3D instance cell, candidate semantic lines corresponding to each semantic source are generated through multiple complementary semantic prediction branches. All candidate semantic lines originating from the same semantic source on the same 3D instance cell are combined into a semantic candidate package corresponding to the semantic source. Extract the geometric support features and cross-source consistency features of the 3D instance units corresponding to each candidate semantic row in the semantic candidate package, and calibrate the original confidence of the candidate semantic row based on the geometric support features and the cross-source consistency features to obtain the calibrated confidence of each candidate semantic row; The observable feature vectors of each 3D instance unit are extracted, and the main semantic labels of each 3D instance unit are segmented from the corresponding semantic candidate package based on the observable feature vectors and the calibration confidence of each candidate semantic row.

[0006] Preferably, acquiring multi-view RGB-D images and 3D point clouds of the target scene, and generating 2D image masks and 3D instance masks respectively, specifically includes: A text-driven open vocabulary detection model is used to generate two-dimensional candidate regions for each frame of RGB-D images of the target scene. The two-dimensional candidate regions are filtered for validity and non-maximum suppression is performed to obtain a two-dimensional image mask; A 3D instance segmentation model is used to generate 3D instance candidate masks from the 3D point cloud of the target scene. The geometric validity of the 3D instance candidate mask is verified to obtain the 3D instance mask.

[0007] Preferably, establishing a point-level support relationship between the two-dimensional image mask and the three-dimensional instance mask, and using the three-dimensional instance mask with the point-level support relationship as a three-dimensional instance unit specifically includes: Based on the camera intrinsic parameters, pose and depth information corresponding to the two-dimensional image mask of each frame, the visible three-dimensional point set covered by the two-dimensional image mask of each frame is obtained by projecting and mapping the three-dimensional point cloud onto the image plane. Calculate the number of intersection points between the visible 3D point set and the 3D instance mask. When the number of intersection points reaches a preset support threshold, establish a point-level support relationship between the 2D image mask and the 3D instance mask. A geometric identifier vector is constructed for each 3D instance mask with point-level support relationship, and a 3D instance unit is obtained by combining the corresponding 3D instance mask, the 2D image mask associated with the point-level support relationship, and the geometric identifier vector.

[0008] Preferably, generating candidate semantic rows corresponding to each semantic source on each 3D instance unit through complementary multiple semantic prediction branches specifically includes: Obtain the 3D instance mask corresponding to each 3D instance unit; The first semantic source performs semantic scoring based on the point-level open vocabulary category response inside the 3D instance mask, and generates candidate rows for the first semantic source. The second semantic source projects a 3D instance mask onto a multi-view RGB-D image, constructs an object center clipping block with foreground constraints, generates category responses for each viewpoint through a visual-language encoder, and aggregates them by weight according to viewpoint quality to generate candidate rows for the second semantic source. The third semantic source orthogonally aligns the visual features within the local clipping block corresponding to the 3D instance mask, and maps the aligned visual features back to the visible 3D points of the 3D instance mask to generate candidate rows for the third semantic source.

[0009] Preferably, combining all candidate semantic rows originating from the same semantic source on the same 3D instance unit into a semantic candidate package corresponding to the semantic source specifically includes: For each 3D instance cell, obtain all candidate semantic rows from the same semantic source. Each candidate semantic row contains candidate category, candidate confidence and source ranking attribute. The candidate semantic lines are grouped according to their semantic sources to form semantic candidate packages corresponding to the semantic sources.

[0010] Preferably, extracting the geometric support features and cross-source consistency features of the 3D instance units corresponding to each candidate semantic row in the semantic candidate package specifically includes: For each candidate semantic line in the semantic candidate package, obtain the number of effective viewpoints, the number of two-dimensional support masks, and the number of three-dimensional support points that support its corresponding three-dimensional instance mask; The number of effective viewpoints, the number of two-dimensional support masks, and the number of three-dimensional support points are combined to form the geometric support features of the corresponding three-dimensional instance unit; Based on the representative labels of each semantic source, the predicted labels of the current candidate semantic row are judged for cross-source consistency, and the cross-source consistency features of the corresponding three-dimensional instance unit are obtained.

[0011] Preferably, the original confidence of the candidate semantic line is calibrated based on the geometric support feature and the cross-source consistency feature to obtain the calibrated confidence of each candidate semantic line, specifically including: Based on the neutral confidence level, the comprehensive geometric gain term and penalty term are determined according to the geometric support features, and the consistency gain term is determined according to the cross-source consistency features; The neutral confidence, the comprehensive geometric gain, and the consistency gain are superimposed, and the penalty term is subtracted. The result is then cropped to a preset interval to obtain the calibration confidence of the corresponding candidate semantic row.

[0012] Preferably, the main semantic label of each 3D instance unit is segmented from the corresponding semantic candidate bag based on the observable feature vector and the calibration confidence of each candidate semantic row using the following steps: For each candidate semantic line in each semantic candidate package, the observable feature vector is concatenated with the candidate ranking feature vector corresponding to the candidate semantic line to form a joint feature vector, and the joint feature vector is semantically classified by a classifier to obtain the predicted semantic label and classification score of the candidate semantic line. Select the candidate semantic row with the highest classification score from each semantic candidate bag, and use its predicted semantic label as the candidate label of the corresponding semantic source; Using the calibration confidence level corresponding to each semantic source as the weight, a weighted vote is performed on the candidate labels of all semantic sources to obtain the fusion score of each candidate label; The candidate label with the highest fusion score is used as the main semantic label of the corresponding 3D instance unit.

[0013] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: The open-vocabulary 3D instance segmentation method with geometrically stabilized multi-source semantic selection provided in this application first acquires multi-view RGB-D images and 3D point clouds of the target scene, generates 2D image masks and 3D instance masks respectively, and establishes point-level support relationships between the 2D image masks and 3D instance masks. The 3D instance masks with point-level support relationships are used as 3D instance units. Second, candidate semantic rows corresponding to each semantic source are generated on each 3D instance unit through complementary multiple semantic prediction branches. All candidate semantic rows originating from the same semantic source on the same 3D instance unit are combined into a semantic candidate package corresponding to the semantic source. Further, the geometric support features and cross-source consistency features of the 3D instance unit corresponding to each candidate semantic row in the semantic candidate package are extracted, and the original confidence of the candidate semantic row is calibrated based on the geometric support features and the cross-source consistency features to obtain the calibrated confidence of each candidate semantic row. Finally, the observable feature vector of each 3D instance unit is extracted, and the main semantic label of each 3D instance unit is segmented from the corresponding semantic candidate package according to the observable feature vector and the calibrated confidence of each candidate semantic row.

[0014] Therefore, this application can achieve the segmentation of open-vocabulary 3D instances under the influence of semantic fragmentation introduced by multi-source prediction. First, multi-view RGB-D images and 3D point clouds of the target scene are acquired, and 2D image masks and 3D instance masks are generated respectively. Then, a point-level support relationship is established between the 2D image mask and the 3D instance mask. The 3D instance mask with the point-level support relationship is used as the 3D instance unit. The point-level support relationship is used to correlate the pixel-level segmentation accuracy of the 2D image mask with the spatial integrity of the 3D instance mask, so that each 3D instance unit has both fine boundary information at the 2D observation level and spatial attribution information at the 3D geometric level. This provides a processing object with clear spatial domain constraints and reliable multi-view observation support for subsequent multi-source semantic prediction, avoiding semantic confusion caused by boundary ambiguity and insufficient observation when performing semantic prediction directly on unconstrained 3D point clouds. Secondly, candidate semantic rows corresponding to each semantic source are generated on each 3D instance unit through multiple complementary semantic prediction branches. All candidate semantic rows originating from the same semantic source on the same 3D instance unit are combined into a semantic candidate package corresponding to the semantic source. This is achieved by using three semantic prediction branches that are complementary in feature granularity and modality to predict from three different angles: 3D point-level features, multi-view 2D visual-linguistic features, and orthogonal alignment mapping features. Semantic interpretation is performed on the same 3D instance unit, enabling candidate semantic rows to cover semantic information that cannot be fully represented by a single modality. Simultaneously, constructing semantic candidate packages at the semantic source granularity achieves ordered isolation and structured organization of candidate results from different sources, providing a clear data foundation for subsequent cross-source comparison and fusion decision-making. Then, geometric support features and cross-source consistency features of the 3D instance unit corresponding to each candidate semantic row in the semantic candidate package are extracted. The original confidence of the candidate semantic row is calibrated based on these geometric support features and cross-source consistency features to obtain the calibrated confidence of each candidate semantic row. This quantifies the confidence of each candidate semantic row using geometric support features. The degree of multi-view observation received by the selected semantic row in three-dimensional space is measured by cross-source consistency features to measure the degree of consensus reached between multiple complementary semantic sources for the predicted label of the candidate semantic row. The two dimensions are independent yet complementary, and together they adjust the gain and penalty of the original confidence score. This effectively suppresses the problem of inflated confidence scores caused by modal bias, perspective limitation, or accidental high scores from a single semantic source. The calibrated confidence score reflects the credibility of the classification score itself, the external reliability of geometric observation, and the degree of consistency of multi-source semantics. This avoids the situation where models are incomparable due to heterogeneity, which makes it easy for simple score aggregation to select incorrect semantic labels.Finally, the observable feature vectors of each 3D instance unit are extracted, and the main semantic label of each 3D instance unit is segmented from the corresponding semantic candidate package based on the observable feature vectors and the calibration confidence of each candidate semantic row. This unifies the information of the 3D instance unit in five dimensions: ranking significance, cross-source consistency, candidate distribution state, projection reliability, and geometric morphology. This ensures that the selection of the main semantic label no longer relies solely on a single calibration confidence value, but comprehensively considers the distribution pattern of multi-source candidates, geometric observation conditions, and 3D shape characteristics. Through a two-level decision-making mechanism of joint feature classification and cross-source weighted voting, robust main semantic label determination is made among the candidate results of multiple semantic sources, improving the semantic accuracy and stability of 3D instance segmentation under open vocabulary conditions. In summary, the technical solution provided in this application can achieve the segmentation of open vocabulary 3D instances under the influence of semantic fragmentation introduced by multi-source prediction. Attached Figure Description

[0015] Figure 1 This is a simplified flowchart of the geometrically stabilized multi-source semantic selection open-vocabulary 3D instance segmentation method shown in this application; Figure 2 This is an exemplary flowchart of the geometrically stabilized multi-source semantic selection open-vocabulary 3D instance segmentation method shown in this application; Figure 3 This is an exemplary flowchart illustrating the determination of candidate semantic lines as shown in this application. Detailed Implementation

[0016] To better understand the technical solution of this application, the technical solution of this application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] refer to Figure 1 The figure is a simplified flowchart of the open-vocabulary 3D instance segmentation method of geometrically stabilized multi-source semantic selection proposed in this application. In the figure, a multi-view RGB-D image is first input. On the one hand, a 3D instance mask is generated from the multi-view RGB-D image and 3D instance units are constructed. On the other hand, a 3D point cloud is obtained by combining the multi-view RGB-D image. Point-level support relationship is established based on the 3D point cloud and the 2D image mask. Based on the point-level support relationship, a semantic candidate package is constructed. The semantic candidate package contains 3D instance unit features corresponding to multiple candidate semantic rows generated by different semantic sources. By combining the 3D instance unit information with the candidate semantic information in the semantic candidate package after the confidence is calibrated by geometric support features and cross-source consistency features, the main semantic label corresponding to each 3D instance unit is finally output, realizing 3D instance segmentation under open vocabulary.

[0018] refer to Figure 2This figure is an exemplary flowchart of an open-vocabulary 3D instance segmentation method for geometrically stabilized multi-source semantic selection according to some embodiments of this application. The figure mainly includes the following steps: In step S101, a multi-view RGB-D image and a 3D point cloud of the target scene are acquired, a 2D image mask and a 3D instance mask are generated respectively, and a point-level support relationship is established between the 2D image mask and the 3D instance mask. The 3D instance mask with the point-level support relationship is used as a 3D instance unit.

[0019] It should be noted that, in this application, the target scene refers to an indoor or outdoor 3D environment requiring open-vocabulary 3D instance segmentation. Specifically, the target scene is a physical space jointly represented by multi-view RGB-D images and 3D point clouds, such as an entire room, multiple rooms within a building, or an outdoor street block, containing various object instances to be identified and segmented. The 3D point cloud of this scene and the corresponding multi-view RGB-D images constitute the raw data input for the entire method. The 3D point cloud provides the geometric structure information of the scene, while the multi-view RGB-D images provide appearance and color evidence for subsequent 2D candidate generation, semantic feature extraction, and multi-source semantic prediction. In subsequent steps, the two are interconnected through RGB-D geometric projection relationships, jointly serving the construction of 3D instance units and the generation of open-vocabulary semantic labels.

[0020] The following steps are used to acquire multi-view RGB-D images and 3D point clouds of the target scene, and generate 2D image masks and 3D instance masks respectively: A text-driven open vocabulary detection model is used to generate two-dimensional candidate regions for each frame of RGB-D images of the target scene. The two-dimensional candidate regions are filtered for validity and non-maximum suppression is performed to obtain a two-dimensional image mask; A 3D instance segmentation model is used to generate 3D instance candidate masks from the 3D point cloud of the target scene. The geometric validity of the 3D instance candidate mask is verified to obtain the 3D instance mask.

[0021] Specifically, RGB-D images of the target scene captured by N cameras in different poses are acquired and formed into a multi-view image set. Each frame of the RGB-D image contains a color image and a depth image registered with the camera's intrinsic and extrinsic parameters. The pixel value of each pixel in the depth image represents the distance from the corresponding 3D spatial point along the camera's optical axis to the camera's imaging plane. C preset open vocabulary category names are then input into the pre-trained language encoder CLIP Text. The encoder encodes C category word embedding vectors, each with a dimension of 512. L2 normalization is applied to each of the C category word embedding vectors to make each vector have a magnitude of 1. The normalized C category word embedding vectors are then arranged column-wise to form a 512×C classification weight matrix. Each column of the classification weight matrix corresponds to a normalized semantic feature representation of an open vocabulary category. For each frame of color image, a feature extractor consisting of a ResNet-50 backbone network and a feature pyramid network is used to extract a multi-scale feature pyramid. Specifically, the four residual convolution stages of the ResNet-50 backbone network output three-level feature maps with strides of 8, 16, and 32, respectively. The feature pyramid network maps the three-level feature maps to a unified 256-channel format through top-down paths and lateral connections, resulting in five feature layers with different resolutions. The lowest-level feature layer, i.e., the highest-resolution feature layer, is then input into the region proposal network. Multiple candidate anchor boxes are generated. For each candidate anchor box, a 7×7×256 region feature tensor is extracted from the feature layers of the five feature layers that match the scale of the anchor box through a region of interest alignment operation. This tensor is then compressed into a 1024-dimensional region feature vector by global average pooling. The region feature vector is multiplied by the classification weight matrix to obtain the original classification score vector of dimension C. This vector is then normalized by Softmax to obtain the category confidence of the candidate anchor box for the corresponding C open vocabulary categories. Simultaneously, the mask prediction branch uses a fully convolutional decoder composed of four transposed convolutional layers to upsample and decode the 7×7×256 region feature tensor, outputting a pixel-level binary mask of the same size as the bounding rectangle of the candidate anchor box. In the mask, each element with a value of 1 indicates that the corresponding pixel belongs to the foreground target, and 0 indicates that it belongs to the background. The pixel-level binary mask, bounding rectangle, and category confidence of each candidate anchor box are encapsulated into a two-dimensional candidate region.

[0022] It should be noted that the two-dimensional candidate region is a data structure consisting of a pixel-level binary mask, the coordinates of the bounding rectangle, and a class confidence triplet.

[0023] Specifically, candidate anchor boxes with a category confidence score greater than a preset confidence threshold of 0.3 are selected and their corresponding two-dimensional candidate regions are retained. The intersection-union ratio (IUR) between each pair of bounding rectangles of all retained two-dimensional candidate regions is calculated. When the IUR is greater than a preset overlap threshold of 0.5, the one with the lower category confidence score is removed. After non-maximum suppression, a two-dimensional image mask without redundancy is obtained. The two-dimensional image mask is a Boolean matrix with the same size as the corresponding color image, and its element values ​​indicate whether the corresponding pixels belong to the same instance target.

[0024] Specifically, the 3D point cloud of the target scene is input into the 3D instance segmentation model. Each point in the 3D point cloud contains a six-dimensional attribute vector composed of 3D spatial coordinates and RGB color values. First, the 3D point cloud is voxelized, dividing the 3D space into a cubic mesh with a side length of 0.02 meters. For each non-empty voxel containing at least one point, the mean of the 3D coordinates and the mean of the RGB color values ​​of all points within it are calculated and concatenated to form a 12-dimensional voxel feature vector. A PointNet encoder, consisting of two fully connected layers sequentially connected to ReLU activation functions, encodes the 12-dimensional voxel feature vector of each non-empty voxel into a 6-dimensional attribute vector. A 4D voxel embedding vector is generated by arranging the 64D voxel embedding vectors of all non-empty voxels according to spatial coordinates to form a sparse 3D tensor. This tensor is then input into a feature extraction network containing four 3D sparse convolutional layers. The kernel size of each layer is 3×3×3, and the number of channels is 64, 128, 192, and 256, respectively. After four layers of downsampling, each non-empty voxel obtains a 256-dimensional high-level semantic feature vector. Simultaneously, the last sparse convolutional layer of the feature extraction network predicts a 3D offset vector pointing to the center of the ground truth instance for each non-empty voxel and a 64D instance embedding vector. During the decoding stage, the process proceeds from each non-empty voxel along its offset vector. After eight iterations of center drift, non-empty voxels converging to the same drift endpoint are grouped into the same instance group. Then, a secondary clustering algorithm, DBSCAN, is used in the 64-dimensional instance embedding space with cosine distance as the metric. The DBSCAN parameter MinPts is set to 4, and the cosine distance threshold is set to 0.5 to remove noisy voxels and merge semantically consistent instance groups, resulting in a voxel set corresponding to each 3D instance candidate mask. This set is then mapped back to the original 3D point cloud to obtain a point-level binary mask covering all points belonging to the same instance. Geometric validity is verified for each 3D instance candidate mask, and the values ​​of all points within the mask are calculated. The centroid coordinates are obtained by taking the arithmetic mean of the three-dimensional coordinates. The Euclidean distance from each point in the mask to the centroid coordinates is calculated, and the arithmetic mean and standard deviation of all Euclidean distances are obtained. Points whose Euclidean distance exceeds 3 times the standard deviation are marked as outliers and removed from the mask. Then, the minimum bounding box volume of the mask-covered point set is calculated. The point density value is obtained by calculating the ratio of the number of remaining points in the mask after removing outliers to the bounding box volume. When the point density value is lower than the preset density threshold of 50 points per cubic meter, the mask-covered point distribution is determined to be too sparse and is removed. The three-dimensional instance candidate mask retained after geometric validity verification is the three-dimensional instance mask.

[0025] It should be noted that the 3D instance mask in this application is a 3D instance candidate mask that has undergone geometric validity verification. Determining the 3D instance mask involves classifying the points in the 3D point cloud according to their respective physical entities, so that each 3D instance mask uniquely corresponds to an independent object in the scene, thereby transforming the unstructured point cloud into a 3D instance unit with a clear individual boundary.

[0026] To establish a point-level support relationship between a 2D image mask and a 3D instance mask, the following steps are used to implement the 3D instance mask with the point-level support relationship as a 3D instance unit: Based on the camera intrinsic parameters, pose and depth information corresponding to the two-dimensional image mask of each frame, the visible three-dimensional point set covered by the two-dimensional image mask of each frame is obtained by projecting and mapping the three-dimensional point cloud onto the image plane. Calculate the number of intersection points between the visible 3D point set and the 3D instance mask. When the number of intersection points reaches a preset support threshold, establish a point-level support relationship between the 2D image mask and the 3D instance mask. A geometric identifier vector is constructed for each 3D instance mask with point-level support relationship, and a 3D instance unit is obtained by combining the corresponding 3D instance mask, the 2D image mask associated with the point-level support relationship, and the geometric identifier vector.

[0027] Specifically, firstly, for each 3D spatial point in the 3D point cloud of the target scene, projection calculation is performed using the camera extrinsic and intrinsic parameter matrices of the frame to which the point belongs. Specifically, the world coordinates of the 3D spatial point are first multiplied left by the camera extrinsic parameter matrix of that frame to obtain the 3D coordinates in the camera coordinate system. Then, the 3D coordinates in the camera coordinate system are multiplied left by the camera intrinsic parameter matrix of that frame to obtain the pixel coordinates of the point on the image plane of the corresponding frame. Simultaneously, the depth component in the camera coordinate system is used as the projected depth value. The pixel coordinates are compared with the depth value at the same location in the depth image of the corresponding frame. When the absolute value of the difference between the two is less than a preset depth tolerance, it is determined that the 3D spatial point is visible in that frame. If the image is not visible, it is considered occluded or unreliable and is excluded. For each 2D image mask in each frame, its pixel coverage area on the image plane is traversed. 3D spatial points whose projected pixel coordinates fall within the pixel coverage area of ​​the 2D image mask and pass the depth consistency check are included in the visible 3D point set of the 2D image mask. The visible 3D point set is the set of all 3D points in the 3D point cloud covered by the 2D image mask on the image plane. When the same 3D spatial point is covered by multiple 2D image masks in the same frame, it is assigned to the 2D image mask with the smallest projected depth value to avoid duplicate counting. Secondly, for each 2D image mask, the visible 3D points... The system calculates the intersection of each 2D image mask with the set of 3D points covered by each 3D instance mask, and counts the number of points in the intersection. The intersection is the set of 3D points that simultaneously belong to both the visible 3D point set and the 3D instance mask. When the number of intersection points reaches a preset support threshold, it is determined that there is a point-level support relationship between the 2D image mask and the 3D instance mask. This point-level support relationship means that there is a sufficient number of spatially overlapping points between the visible 3D point set and the 3D instance mask. The preset support threshold is used to ensure that a sufficient number of 3D points are simultaneously covered by both masks to eliminate false associations caused by accidental overlap. The specific threshold can be set according to actual needs or expert knowledge. Finally... For each 3D instance mask, the mask identifiers and frame indices of all 2D image masks with which it has point-level support relationships are collected. These are arranged in ascending order of frame index to form an identifier sequence. A geometric identifier vector is generated by splicing the centroid coordinates of the 3D instance mask and the coordinates of the eight vertices of the smallest bounding box in a fixed order. The geometric identifier vector is a reversible geometric code generated based on the instance spatial location and geometric range. It is used to identify and distinguish different 3D instance units in multi-frame data. Each 3D instance mask, the support records of all 2D image masks associated with its point-level support relationships, and the geometric identifier vector are combined into a 3D instance unit.

[0028] It should be noted that the 3D instance unit in this application is a data structure containing the 3D instance mask and all its supporting 2D image mask association information. Each support record contains the mask identifier of the associated 2D image mask, the frame index of the frame it belongs to, and the number of intersection points. By establishing the point-level support relationship between the 2D image mask and the 3D instance mask and constructing the 3D instance unit, the point-level support relationship uses the spatial intersection of 3D points as the basis for the association between 2D observation and 3D geometry. The depth consistency check eliminates false correspondences caused by occlusion or noise during the projection process, ensuring the geometric reliability of the association relationship between the 2D image mask and the 3D instance mask. The 3D instance unit encapsulates each 3D instance mask and all its supporting 2D image mask mask identifiers, frame indexes, and the number of intersection points into a unified data structure, providing complete spatial domain and observation domain information input for the subsequent semantic prediction branch. In addition, the geometric identifier vector generated based on the centroid coordinates and bounding box vertex coordinates enables the same 3D instance unit to be quickly retrieved and compared through geometric feature comparison in different processing stages, significantly reducing the computational overhead of instance matching and deduplication when the scale of multi-frame and multi-view data is large.

[0029] In step S102, candidate semantic lines corresponding to each semantic source are generated on each three-dimensional instance unit through multiple complementary semantic prediction branches, and all candidate semantic lines originating from the same semantic source on the same three-dimensional instance unit are combined into a semantic candidate package of the corresponding semantic source.

[0030] refer to Figure 3 The figure is an exemplary flowchart of determining candidate semantic lines according to this application. In this application, the generation of candidate semantic lines corresponding to each semantic source on each three-dimensional instance unit through multiple complementary semantic prediction branches is achieved by the following steps: In step S1021, the 3D instance mask corresponding to each 3D instance unit is obtained; In step S1022, the first semantic source performs semantic scoring based on the point-level open vocabulary category response inside the three-dimensional instance mask to generate candidate rows of the first semantic source; In step S1023, the second semantic source projects a 3D instance mask onto a multi-view RGB-D image, constructs an object center clipping block with foreground constraints, generates category responses for each viewpoint through a visual-language encoder, and performs weighted aggregation according to viewpoint quality to generate candidate rows for the second semantic source. In step S1024, the third semantic source orthogonally aligns the visual features within the local clipping block corresponding to the 3D instance mask, and maps the aligned visual features back to the visible 3D points of the 3D instance mask to generate candidate rows of the third semantic source.

[0031] Specifically, for each 3D instance unit, the 3D instance mask it contains is obtained, and three complementary semantic prediction branches are performed on all 3D points covered by the 3D instance mask to generate candidate semantic lines for three semantic sources respectively.

[0032] Specifically, the processing of the first semantic source is as follows: For each 3D point within the 3D instance mask, the 32D point-level geometric feature vector of the point output by the 3D instance segmentation network is used to calculate the cosine similarity between the point and the 512D text embedding vectors of each of the C open vocabulary categories, respectively, to obtain the category response value of the point in each category. The text embedding vector is obtained by encoding the category name by a pre-trained language encoder and mapping it to a space of the same dimension as the point-level feature through a linear projection layer. Then, the category response values ​​of all visible points within the 3D instance mask are aggregated by average pooling to obtain the global semantic score of each category. The global semantic score is normalized by Softmax and used as the confidence of the category. The category identifier and its confidence are organized into the first semantic source candidate row. The first semantic source candidate row is a semantic score record containing the category identifier and confidence of each category, calculated by the first semantic source on the 3D point cloud.

[0033] Specifically, the processing of the second semantic source is as follows: traverse all 2D image masks associated with the established point-level support relationships in each 3D instance unit. For each associated 2D image mask, project the 3D points of the 3D instance mask onto the image plane of the corresponding frame using the camera intrinsic and extrinsic matrices of the corresponding frame to determine the projected pixel region of the 3D instance mask on the image of that frame. Based on this projected pixel region, extract the bounding rectangle to construct a clipping block. Simultaneously, use the 2D image mask as a foreground constraint to apply mask filtering to the pixels within the clipping block. Set the pixels within the clipping block that are outside the coverage area of ​​the 2D image mask to zero to remove background pixels, resulting in an object center clipping block with foreground constraints. The object center clipping block with foreground constraints is a local image block with the bounding rectangle of the 2D image mask projection area as its boundary and the internal background pixels set to zero. Use a pre-trained visual-language encoder for the visual encoder. Feature extraction is performed on the center clipping block of the object with foreground constraints to obtain a 768-dimensional visual feature vector. The cosine similarity between this visual feature vector and the text embedding vector of each of the C open vocabulary categories is calculated to obtain the category response value of each category under that viewpoint. At the same time, a viewpoint quality score is calculated for each viewpoint. The viewpoint quality score is the product of the ratio of the projection area of ​​the 3D instance mask on the image frame to the total surface area of ​​the mask and the cosine value of the angle between the viewpoint direction and the surface normal. The viewpoint quality scores of all effective viewpoints are normalized by Softmax and used as the weighted aggregation weights for each viewpoint. The category response values ​​of each viewpoint are weighted and summed according to the corresponding viewpoint weights to obtain the aggregated category confidence, which is organized into the second semantic source candidate row. The second semantic source candidate row is a semantic score record containing category identifiers and confidence scores obtained by extracting features from multi-view 2D images through visual-language encoding and weighted aggregation.

[0034] Specifically, the processing of the third semantic source is as follows: Based on the object-centered clipping blocks with foreground constraints constructed by the second semantic source, the same visual-language encoder is used to extract the two-dimensional visual feature matrix of each clipping block. The two-dimensional visual feature matrix is ​​a feature matrix of dimension N×768 obtained by flattening the feature map containing spatial location information output by the visual encoder of the clipping block, where N is the number of effective feature points in the clipping block. At the same time, a text feature matrix of dimension C×768 composed of C open vocabulary category text embeddings is obtained. An orthogonal alignment transformation matrix is ​​calculated for each clipping block. Specifically, the product matrix of the visual feature matrix and the text feature matrix is ​​calculated. Singular value decomposition is performed on the product matrix to obtain the left singular vector matrix U, the singular value diagonal matrix S, and the right singular vector matrix V. The orthogonal transformation matrix R is obtained by multiplying the left singular vector matrix U by the transpose of the right singular vector matrix V. To obtain the optimal orthogonal rotation matrix that aligns the visual feature space with the text embedding space, calculated using orthogonal Protodyakonov analysis, this orthogonal alignment transformation matrix is ​​applied to the visual feature matrix of the cropped block to obtain the aligned visual feature matrix. This eliminates the inter-domain distribution difference between the pre-trained image domain and the 3D rendered image domain of the visual-language encoder. Subsequently, based on the correspondence between 2D pixels and 3D points recorded during projection mapping, the feature values ​​of each pixel position in the aligned visual feature matrix are mapped back to the corresponding visible 3D point of the 3D instance mask. The features mapped to the same 3D point from multiple viewpoints are weighted and averaged to obtain the point-level alignment features of each visible 3D point. The cosine similarity between the point-level alignment features of each visible 3D point and the text embeddings of each category is calculated. Average pooling aggregation is performed on all visible points of the 3D instance mask by category to obtain the confidence of each category, which is then organized into the third semantic source candidate row.

[0035] It should be noted that the third semantic source candidate behavior, after orthogonal alignment to eliminate inter-domain differences, is obtained by three-dimensional-two-dimensional feature mapping, which includes the semantic score record containing the category identifier and confidence level. The first semantic source directly performs open-vocabulary semantic response in three-dimensional space based on three-dimensional point-level features, providing fine-grained geometric semantic information. The second semantic source is based on two-dimensional visual-linguistic features with multi-view projection, and uses the discrimination ability learned by the pre-trained encoder on large-scale image and text data to provide highly discriminative semantic information. The third semantic source establishes a spatial mapping relationship between three-dimensional features and two-dimensional features through orthogonal alignment, making up for the inter-domain differences between direct three-dimensional encoding and pre-trained two-dimensional encoder. The three semantic sources complement each other in terms of feature granularity and modality.

[0036] The following steps are used to combine all candidate semantic rows from the same semantic source on the same 3D instance cell into a semantic candidate package corresponding to the semantic source: For each 3D instance cell, obtain all candidate semantic rows from the same semantic source. Each candidate semantic row contains candidate category, candidate confidence and source ranking attribute. The candidate semantic lines are grouped according to their semantic sources to form semantic candidate packages corresponding to the semantic sources.

[0037] Specifically, firstly, for each 3D instance cell, from all candidate semantic rows generated by the three semantic prediction branches of that cell, the category identifier, confidence score, and in-source ranking attribute of each candidate semantic row are extracted one by one. The category identifier is the open vocabulary category name code obtained by the corresponding semantic source prediction in the candidate semantic row. The confidence score is the normalized probability value of the category belonging to the current 3D instance cell output by the corresponding semantic source in the candidate semantic row. The in-source ranking attribute is the position index of all candidate semantic rows within the same semantic source after sorting them in descending order of candidate confidence scores. Subsequently, all first semantic source candidate rows from the first semantic source on the same 3D instance cell are grouped into the first semantic candidate bag, all second semantic source candidate rows from the second semantic source are grouped into the second semantic candidate bag, and all third semantic source candidate rows from the third semantic source are grouped into the third semantic candidate bag. Source candidate lines are categorized into the third semantic candidate package. Within each semantic candidate package, all candidate semantic lines are sorted in descending order of confidence. The source ranking attribute of each candidate semantic line is then reassigned based on its position number. Taking a 3D instance cell containing 3 semantic sources and each source outputting K candidate semantic lines as an example, after grouping, 3 semantic candidate packages are obtained. Each semantic candidate package contains K candidate semantic lines. The candidate category identifier for each candidate semantic line is the corresponding open vocabulary category name. The confidence level is the category confidence level output by the corresponding semantic source. The source ranking attribute is the index of the candidate semantic line in the current semantic candidate package after sorting by confidence in descending order, with a value range from 1 to K. The candidate semantic line with a source ranking attribute of 1 is the candidate semantic line with the highest candidate confidence in that semantic candidate package.

[0038] It should be noted that in this application, the semantic candidate package is an ordered set composed of all candidate semantic rows generated from the same semantic source, sorted by confidence level. The number of elements in each semantic candidate package is the number of candidate categories output by the corresponding semantic source. By grouping all candidate semantic rows from the same semantic source on the same 3D instance unit according to the semantic source and constructing the semantic candidate package, the semantic candidate package provides a structured isolation of candidate semantic rows at the semantic source granularity. This allows for the clear tracing of the semantic source to which each candidate semantic row belongs when extracting geometric support features and cross-source consistency features. This avoids attribute confusion or cross-interference of candidate semantic rows generated from different semantic sources during the subsequent calibration process. In addition, the semantic candidate package is arranged in descending order of candidate confidence and assigned a source ranking attribute with a position number. This provides an ordered data foundation for the subsequent confidence calibration stage to use the source ranking attribute to measure the relative reliability between candidate semantic rows within the same semantic source. This allows the calibration process to make a comprehensive judgment by combining the two dimensions of absolute confidence and relative ranking.

[0039] In step S103, the geometric support features and cross-source consistency features of the three-dimensional instance units corresponding to each candidate semantic line in the semantic candidate package are extracted, and the original confidence of the candidate semantic line is calibrated based on the geometric support features and the cross-source consistency features to obtain the calibrated confidence of each candidate semantic line.

[0040] The geometric support features and cross-source consistency features of the 3D instance units corresponding to each candidate semantic row in the semantic candidate package are extracted using the following steps: For each candidate semantic line in the semantic candidate package, obtain the number of effective viewpoints, the number of two-dimensional support masks, and the number of three-dimensional support points that support its corresponding three-dimensional instance mask; The number of effective viewpoints, the number of two-dimensional support masks, and the number of three-dimensional support points are combined to form the geometric support features of the corresponding three-dimensional instance unit; Based on the representative labels of each semantic source, the predicted labels of the current candidate semantic row are judged for cross-source consistency, and the cross-source consistency features of the corresponding three-dimensional instance unit are obtained.

[0041] Specifically, firstly, for each candidate semantic row in the semantic candidate package, all 2D image masks that have point-level support relationships with the 3D instance mask are queried from the support records of the 3D instance unit. The number of valid viewpoints is obtained by deduplicating the frame indices of the 2D image masks from different frames. The number of valid viewpoints is the total number of different camera viewpoints from which the 2D image mask supporting the current 3D instance mask originates. The total number of 2D image masks that have point-level support relationships with the 3D instance mask is obtained by counting the total number of 2D support masks that have established point-level support relationships with the 3D instance mask. The total number of 2D support masks is the total number of 2D image masks that have established point-level support relationships with the 3D instance mask. The sum of the intersection points in all support records is taken as the number of 3D support points. The quantity, the number of 3D support points, is the sum of the number of intersection points between the visible 3D point sets of all 2D image masks supporting the 3D instance mask and the 3D instance mask; then, the number of effective viewpoints, the number of 2D support masks, and the number of 3D support points are concatenated in a fixed order into a 3D column vector as geometric support features, which are feature vectors composed of three geometric statistics describing the degree of support of the 3D instance mask by multi-view 2D observations; subsequently, for the candidate category of the candidate semantic row, all semantic sources are traversed, and for each semantic source, the candidate category of the candidate semantic row with the highest candidate confidence is selected from its semantic candidate bag as the representative label of the semantic source, the representative label being the semantic source in the current 3D instance single... The predicted category with the highest confidence level is used as the candidate label for the current candidate semantic row. Each semantic source's representative label is compared with this predicted label to determine if they are consistent. The proportion of consistent representative labels to all semantic sources is used as the cross-source consistency ratio. This cross-source consistency ratio is the proportion of semantic sources whose representative labels are the same as the current predicted label, ranging from 0 to 1. A cross-source consistency ratio of 1 indicates that all semantic sources have reached a consensus on this predicted label, while a cross-source consistency ratio of 0 indicates that no other semantic sources support this predicted label. A binary consistency vector is constructed with the dimension of the number of semantic sources, where each dimension corresponds to a semantic source. When the representative label of the corresponding semantic source... When the label matches the predicted label, the dimension is assigned a value of 1; otherwise, it is assigned a value of 0. The binary consistency vector is a vector that uses binary encoding to represent whether each semantic source is consistent with the current predicted label. The cross-source consistency ratio and the binary consistency vector are concatenated to form the cross-source consistency feature. Taking three semantic sources as an example, when the predicted category of a candidate semantic row is "chair", the representative label of the first semantic source is "chair", the representative label of the second semantic source is "seat", and the representative label of the third semantic source is "chair", the cross-source consistency ratio is 2 / 3, the binary consistency vector is [1,0,1], and the cross-source consistency feature is the 4-dimensional vector [2 / 3,1,0,1] obtained by concatenating the scalar 2 / 3 with the vector [1,0,1].The geometric support feature and the cross-source consistency feature are combined as support features for the candidate semantic row, and used to subsequently calibrate the candidate confidence of the candidate semantic row.

[0042] It should be noted that the cross-source consistency feature in this application describes the degree of consistency between the current predicted label and multiple semantic sources. The cross-source consistency feature consists of a cross-source consistency ratio and a binary consistency vector. The cross-source consistency ratio reflects the degree of consensus reached between the current predicted label and multiple semantic sources in a quantitative value. When multiple semantic sources based on different feature granularities and different modalities all point to the same category, the reliability of the predicted label is significantly higher than the result of relying on only a single semantic source. This provides an external reliability metric independent of the original classification score for subsequent confidence calibration. The binary consistency vector accurately records whether the representative label of each semantic source is consistent with the current predicted label in a source-by-source encoding manner. This enables the subsequent confidence calibration process to not only utilize the overall consistency ratio but also distinguish the source composition of consistency, providing an information basis for applying differentiated calibration weights to different semantic sources.

[0043] The original confidence of the candidate semantic row is calibrated based on the geometric support features and the cross-source consistency features. The calibrated confidence of each candidate semantic row is obtained by the following steps: Based on the neutral confidence level, the comprehensive geometric gain term and penalty term are determined according to the geometric support features, and the consistency gain term is determined according to the cross-source consistency features; The neutral confidence, the comprehensive geometric gain, and the consistency gain are superimposed, and the penalty term is subtracted. The result is then cropped to a preset interval to obtain the calibration confidence of the corresponding candidate semantic row.

[0044] Specifically, for each candidate semantic line, a neutral confidence level is first set as the calibration baseline value. This neutral confidence level is the prior uniform probability of the candidate category of the candidate semantic line across all open vocabulary categories, and its value is equal to the reciprocal of the total number of open vocabulary categories C. It represents the baseline probability estimate of the candidate category without introducing any geometric or multi-source calibration information. Then, a comprehensive geometric gain term is jointly calculated based on the three components of the geometric support features of the candidate semantic line. Specifically, the number of effective viewpoints, the number of two-dimensional support masks, and the number of three-dimensional support points are divided by their corresponding preset reference values ​​(preset reference values ​​for the number of viewpoints, preset reference values ​​for the number of masks, and preset reference values ​​for the number of support points, respectively). (This can be set according to actual needs), resulting in three normalized ratios. The arithmetic mean of these three normalized ratios is multiplied by the first learnable weight coefficient to obtain the comprehensive geometric gain term. This comprehensive geometric gain term describes the comprehensive gain contribution of the current 3D instance mask to the sufficiency of multi-view geometric observation. The averaging method avoids the problem of duplicate rewards caused by the inherent correlation between geometric components. Simultaneously, a penalty term is calculated based on the number of 3D support points in the geometric support features. The calculation method is as follows: when the number of 3D support points is lower than the preset minimum reference point number, the difference between the preset minimum reference point number and the number of 3D support points is multiplied by the penalty weight coefficient to obtain the penalty term; otherwise, a penalty is applied. The term takes a value of zero. The penalty term is a suppression measure that lowers the confidence level when the number of 3D support points is severely insufficient, used to suppress the artificially high confidence level caused by insufficient geometric observation coverage. Subsequently, a consistency gain term is determined based on the cross-source consistency features of the candidate semantic row. The cross-source consistency ratio in the cross-source consistency features is multiplied by the second learnable weight coefficient to obtain the consistency gain term. The consistency gain term describes the gain contribution of multiple semantic sources to the degree of consensus on the current predicted label. When the cross-source consistency ratio is 1, the consistency gain term takes the maximum value, indicating that all semantic sources are consistent. When the cross-source consistency ratio is 0, the consistency gain term is 0, indicating no consensus and no gain. Among them, the first The first and second learnable weight coefficients are automatically determined during the training phase through end-to-end gradient optimization: 3D scene samples containing known real semantic labels are used as training data. The cross-entropy loss between the calibration confidence of all candidate semantic rows and the real labels is used as the optimization objective. The initial values ​​of the two weight coefficients are iteratively updated through the backpropagation algorithm. In each iteration, the values ​​of each weight coefficient are adjusted along the negative gradient direction of the loss function until the loss converges to a preset threshold or the preset number of iterations is reached, at which point training stops. The two weight coefficient values ​​after training convergence are fixed for the inference phase. During initialization, the first learnable weight coefficient is set to 0.1, and the second learnable weight coefficient is set to 0.05. To ensure that the gain terms are at a relatively small level during the initial training phase, unlearned weights are prevented from excessively perturbing the original confidence level in the early calibration stages. As training progresses, the coefficients gradually converge to their optimal values. Then, the neutral confidence level, the comprehensive geometric gain term, and the consistency gain term are added together, and a penalty term is subtracted to obtain the uncropped calibration value. This uncropped calibration value is then compared with the lower and upper bounds of a pre-set confidence interval. If the uncropped calibration value is less than the lower bound of 0, it is truncated to 0; if it is greater than the upper bound of 1, it is truncated to 1. The truncated value is used as the calibration confidence level for the candidate semantic row. This calibration confidence level is a confidence value calibrated by both geometric support and cross-source consistency, and its value always lies within a closed interval between 0 and 1.

[0045] It should be noted that, in this application, the calibration confidence refers to the confidence of class assignment after dual verification by geometric observation sufficiency and multi-source semantic consistency. The original confidence only reflects the classification score of a single semantic source for a candidate class and cannot measure the degree of observational support in the three-dimensional geometric space or the degree of consensus among multiple complementary semantic sources. The calibration confidence, by introducing geometric support features, gives a gain to candidate semantic rows with sufficient multi-view observations and support points, and imposes a penalty on candidate semantic rows with sparse observations. At the same time, it gives a consistency gain to the predicted labels jointly supported by multiple semantic sources through cross-source consistency features. This makes the final calibration confidence reflect the comprehensive information of three dimensions: the intrinsic confidence of the classification score, the external observation reliability in the three-dimensional geometric space, and the degree of consistency among multi-source semantics. This provides a more accurate and robust decision basis than the original confidence for the subsequent step of segmenting the main semantic label from each semantic candidate bag, effectively reducing the risk of misclassification caused by modal bias or perspective limitation of a single semantic source.

[0046] In step S104, the observable feature vectors of each three-dimensional instance unit are extracted, and the main semantic labels of each three-dimensional instance unit are segmented from the corresponding semantic candidate package based on the observable feature vectors and the calibration confidence of each candidate semantic row.

[0047] The following steps are used to extract the observable feature vectors of each 3D instance unit: Based on the ordering position of the candidate semantic lines of each semantic source in the semantic candidate bag and the representative labels of each semantic source, the candidate ordering features and cross-source semantic consistency features are calculated respectively. The candidate distribution status of the semantic candidate packets of each semantic source is statistically analyzed to obtain the candidate packet statistical features. The visibility support of the 3D instance mask in multi-view images is evaluated to obtain the RGB-D projection reliability features. The geometric properties of the 3D instance mask are extracted to obtain the 3D shape features. The candidate ranking features, the cross-source semantic consistency features, the candidate packet statistical features, the RGB-D projection reliability features, and the three-dimensional shape features are concatenated to obtain an observable feature vector.

[0048] Specifically, firstly, for each 3D instance unit, traverse each of its subordinate semantic candidate bags. Within each semantic candidate bag, find the candidate semantic row with the highest candidate confidence. Calculate the source-internal ranking attribute value of this candidate semantic row (a value of 1 indicates first place), calculate the difference between its candidate confidence and the confidence of the second-ranked candidate in the same bag to obtain the confidence interval, and calculate the proportion of this candidate confidence to the total confidence of all candidates in the same bag to obtain the confidence percentage. Concatenate the source-internal ranking attribute value, confidence interval, and confidence percentage to form the candidate ranking feature corresponding to this semantic source. This candidate ranking feature describes the support strength and discriminative power of the current semantic source for the first-ranked candidate semantic row. Then, compare the rankings of all semantic sources... The first step is to determine whether the candidate categories of the first and last candidate semantic rows are the same. If all are the same, the cross-source semantic consistency feature is assigned a value of 1. If some are the same, the proportion of semantic sources with the same category to the total number of semantic sources is calculated as the consistency value. If all are different, the value is assigned a value of 0. The cross-source semantic consistency feature is a scalar value describing whether multiple semantic sources have reached a consensus on the same candidate category. Next, statistics are performed on each semantic candidate package. The total number of candidate semantic rows, the maximum candidate confidence score, the standard deviation of the candidate confidence score, and the difference between the confidence scores of the first and last candidates in each package are calculated as the candidate package distribution dispersion. The statistical values ​​of each package are concatenated according to the semantic source order to form the candidate package statistical features. To describe the distribution of the number of candidate semantic rows and the degree of confidence dispersion of each semantic source, feature vectors are generated. Then, based on the 3D instance mask of the 3D instance unit and all its associated 2D image masks, for the viewpoint corresponding to each associated 2D image mask, the 3D points of the 3D instance mask are projected onto the image plane of that viewpoint. The proportion of the number of pixels covered by the projection to the total number of pixels in the image of that viewpoint is calculated to obtain the projection area proportion. At the same time, the proportion of the number of pixels with effective depth readings in the depth image of the projection region to the total number of pixels covered by the projection is calculated to obtain the effective depth coverage rate. The projection area proportion and the effective depth coverage rate are normalized and weighted according to preset weights to obtain the reliability weight of that viewpoint. The reliability weights of all associated viewpoints are calculated. The RGB-D projection reliability feature is obtained by means of the technical average. The RGB-D projection reliability feature is a scalar value describing the degree to which the 3D instance mask can be reliably observed in multi-view images. Then, the length, width and height of the minimum bounding box are calculated for all 3D points covered by the 3D instance mask. The standard deviation of the Euclidean distance from all points to the centroid of the point set is calculated as the shape dispersion. Principal component analysis is performed on all points to obtain the three principal component directions and their corresponding eigenvalues. The ratio of the minimum eigenvalue to the maximum eigenvalue is used as the shape flatness. The length, width and height of the bounding box, the shape dispersion and the shape flatness are concatenated to form a 3D shape feature. The 3D shape feature is a five-dimensional feature vector describing the spatial geometry of the 3D instance mask.Finally, the candidate ranking features, cross-source semantic consistency features, candidate bag statistical features, RGB-D projection reliability features, and 3D shape features are concatenated into an observable feature vector in a predetermined order.

[0049] It should be noted that the observable feature vector in this application is a multi-dimensional feature vector that comprehensively reflects the five dimensions of ranking significance, cross-source consistency, candidate distribution, projection reliability, and geometric morphology of the three-dimensional instance unit. The observable feature vector encodes the information of the three-dimensional instance unit in the five dimensions of ranking significance, cross-source semantic consistency, candidate distribution status, RGB-D projection reliability, and geometric morphology into a fixed-length multi-dimensional vector, which serves as the input feature for the subsequent main semantic label selection step. This makes the label selection process no longer rely solely on a single calibration confidence value, but can comprehensively consider the multi-source candidate distribution pattern, geometric observation conditions, and three-dimensional shape characteristics of the three-dimensional instance unit, thereby making a more accurate and robust main semantic label decision among the candidate results of multiple semantic sources.

[0050] The main semantic labels of each 3D instance unit are segmented from the corresponding semantic candidate bag based on the observable feature vector and the calibration confidence of each candidate semantic row using the following steps: For each candidate semantic line in each semantic candidate package, the observable feature vector is concatenated with the candidate ranking feature vector corresponding to the candidate semantic line to form a joint feature vector, and the joint feature vector is semantically classified by a classifier to obtain the predicted semantic label and classification score of the candidate semantic line. Select the candidate semantic row with the highest classification score from each semantic candidate bag, and use its predicted semantic label as the candidate label of the corresponding semantic source; Using the calibration confidence level corresponding to each semantic source as the weight, a weighted vote is performed on the candidate labels of all semantic sources to obtain the fusion score of each candidate label; The candidate label with the highest fusion score is used as the main semantic label of the corresponding 3D instance unit.

[0051] Specifically, firstly, for each candidate semantic row in each semantic candidate bag of each 3D instance unit, the observable feature vector of the 3D instance unit and the candidate ranking feature vector of the candidate semantic row are concatenated in a fixed order to form a joint feature vector. The joint feature vector is a multi-dimensional feature vector composed of candidate ranking feature sub-vectors from the observable feature vector, cross-source semantic consistency features, candidate bag statistical features, RGB-D projection reliability features, 3D shape features, and the candidate ranking features of the candidate semantic row itself, concatenated in a predetermined order. The joint feature vector is then input into a multilayer sensing layer consisting of three fully connected layers cascaded sequentially, with ReLU activation functions inserted between adjacent layers. The knowledge machine classifier, wherein the input layer dimension of the multilayer perceptron classifier is equal to the dimension of the joint feature vector, the dimension of both hidden layers is 128, and the output layer dimension is equal to the total number of open vocabulary categories C. After softmax normalization, the output layer outputs a category probability distribution of dimension C. The category with the highest probability value in the category probability distribution is taken as the predicted semantic label of the candidate semantic row. The predicted semantic label is the final semantic category name corresponding to the candidate semantic row obtained after classification by joint features. The maximum probability value is taken as the classification score of the candidate semantic row. The classification score is the normalized probability value output by the multilayer perceptron classifier for the predicted semantic label. Next, from each semantic candidate bag... The process involves identifying the candidate semantic row with the highest classification score and using its predicted semantic label as the candidate label for the corresponding semantic source. The candidate label represents the optimal semantic category determined by the joint feature classification of each semantic source on the 3D instance unit. Then, using the calibration confidence score of the first-ranked candidate semantic row corresponding to each semantic source as the voting weight for that semantic source, a weighted voting process is performed on the candidate labels of all semantic sources. Specifically, this involves iterating through all M semantic sources' candidate labels, using the calibration confidence score for each candidate label as the initial vote count, and summing the calibration confidence scores of multiple semantic sources when they output the same candidate label as the fusion score for that candidate label. The sum of the calibration confidence scores of all semantic sources for the selected label is used to measure the overall support strength of the candidate label among multiple semantic sources. Finally, the candidate label with the highest fusion score is taken as the main semantic label of the corresponding 3D instance unit. Taking three semantic sources as an example, the first semantic source outputs the candidate label "chair" with a calibration confidence score of 0.8, the second semantic source outputs the candidate label "chair" with a calibration confidence score of 0.6, and the third semantic source outputs the candidate label "seat" with a calibration confidence score of 0.7. Then, the fusion score of "chair" is 0.8 + 0.6 = 1.4, and the fusion score of "seat" is 0.7. Finally, the "chair" with the highest fusion score is selected as the main semantic label of the 3D instance unit.

[0052] It should be noted that the main semantic label in this application is the final open vocabulary semantic category annotation result of the three-dimensional instance unit.

[0053] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0054] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A geometrically stabilized multi-source semantic selection open-vocabulary 3D instance segmentation method, characterized in that, Includes the following steps: Acquire multi-view RGB-D images and 3D point clouds of the target scene, generate 2D image masks and 3D instance masks respectively, establish point-level support relationships between 2D image masks and 3D instance masks, and use 3D instance masks with point-level support relationships as 3D instance units. On each 3D instance cell, candidate semantic lines corresponding to each semantic source are generated through multiple complementary semantic prediction branches. All candidate semantic lines originating from the same semantic source on the same 3D instance cell are combined into a semantic candidate package corresponding to the semantic source. Extract the geometric support features and cross-source consistency features of the 3D instance units corresponding to each candidate semantic row in the semantic candidate package, and calibrate the original confidence of the candidate semantic row based on the geometric support features and the cross-source consistency features to obtain the calibrated confidence of each candidate semantic row; The observable feature vectors of each 3D instance unit are extracted, and the main semantic labels of each 3D instance unit are segmented from the corresponding semantic candidate package based on the observable feature vectors and the calibration confidence of each candidate semantic row.

2. The geometrically stabilized multi-source semantic selection open-vocabulary 3D instance segmentation method as described in claim 1, characterized in that, The process of acquiring multi-view RGB-D images and 3D point clouds of the target scene, and generating 2D image masks and 3D instance masks respectively, specifically includes: A text-driven open vocabulary detection model is used to generate two-dimensional candidate regions for each frame of RGB-D images of the target scene. The two-dimensional candidate regions are filtered for validity and non-maximum suppression is performed to obtain a two-dimensional image mask; A 3D instance segmentation model is used to generate 3D instance candidate masks from the 3D point cloud of the target scene. The geometric validity of the 3D instance candidate mask is verified to obtain the 3D instance mask.

3. The geometrically stabilized multi-source semantic selection open-vocabulary 3D instance segmentation method as described in claim 1, characterized in that, Establishing a point-level support relationship between a 2D image mask and a 3D instance mask, and using the 3D instance mask with the point-level support relationship as a 3D instance unit specifically includes: Based on the camera intrinsic parameters, pose and depth information corresponding to the two-dimensional image mask of each frame, the visible three-dimensional point set covered by the two-dimensional image mask of each frame is obtained by projecting and mapping the three-dimensional point cloud onto the image plane. Calculate the number of intersection points between the visible 3D point set and the 3D instance mask. When the number of intersection points reaches a preset support threshold, establish a point-level support relationship between the 2D image mask and the 3D instance mask. A geometric identifier vector is constructed for each 3D instance mask with point-level support relationship, and a 3D instance unit is obtained by combining the corresponding 3D instance mask, the 2D image mask associated with the point-level support relationship, and the geometric identifier vector.

4. The geometrically stabilized multi-source semantic selection open-vocabulary 3D instance segmentation method as described in claim 1, characterized in that, The generation of candidate semantic rows for each semantic source on each 3D instance unit through complementary multiple semantic prediction branches specifically includes: Obtain the 3D instance mask corresponding to each 3D instance unit; The first semantic source performs semantic scoring based on the point-level open vocabulary category response inside the 3D instance mask, and generates candidate rows for the first semantic source. The second semantic source projects a 3D instance mask onto a multi-view RGB-D image, constructs an object center clipping block with foreground constraints, generates category responses for each viewpoint through a visual-language encoder, and aggregates them by weight according to viewpoint quality to generate candidate rows for the second semantic source. The third semantic source orthogonally aligns the visual features within the local clipping block corresponding to the 3D instance mask, and maps the aligned visual features back to the visible 3D points of the 3D instance mask to generate candidate rows for the third semantic source.

5. The geometrically stabilized multi-source semantic selection open-vocabulary 3D instance segmentation method as described in claim 1, characterized in that, Combining all candidate semantic rows originating from the same semantic source on the same 3D instance unit into a semantic candidate package corresponding to the semantic source specifically includes: For each 3D instance cell, obtain all candidate semantic rows from the same semantic source. Each candidate semantic row contains candidate category, candidate confidence and source ranking attribute. The candidate semantic lines are grouped according to their semantic sources to form semantic candidate packages corresponding to the semantic sources.

6. The geometrically stabilized multi-source semantic selection open-vocabulary 3D instance segmentation method as described in claim 1, characterized in that, The extraction of geometric support features and cross-source consistency features of the 3D instance units corresponding to each candidate semantic row in the semantic candidate package specifically includes: For each candidate semantic line in the semantic candidate package, obtain the number of effective viewpoints, the number of two-dimensional support masks, and the number of three-dimensional support points that support its corresponding three-dimensional instance mask; The number of effective viewpoints, the number of two-dimensional support masks, and the number of three-dimensional support points are combined to form the geometric support features of the corresponding three-dimensional instance unit; Based on the representative labels of each semantic source, the predicted labels of the current candidate semantic row are judged for cross-source consistency, and the cross-source consistency features of the corresponding three-dimensional instance unit are obtained.

7. The geometrically stabilized multi-source semantic selection open-vocabulary 3D instance segmentation method as described in claim 1, characterized in that, The original confidence of the candidate semantic line is calibrated based on the geometric support features and the cross-source consistency features to obtain the calibrated confidence of each candidate semantic line, specifically including: Based on the neutral confidence level, the comprehensive geometric gain term and penalty term are determined according to the geometric support features, and the consistency gain term is determined according to the cross-source consistency features; The neutral confidence, the comprehensive geometric gain, and the consistency gain are superimposed, and the penalty term is subtracted. The result is then cropped to a preset interval to obtain the calibration confidence of the corresponding candidate semantic row.

8. The geometrically stabilized multi-source semantic selection open-vocabulary 3D instance segmentation method as described in claim 1, characterized in that, The main semantic labels of each 3D instance unit are segmented from the corresponding semantic candidate bag based on the observable feature vector and the calibration confidence of each candidate semantic row using the following steps: For each candidate semantic line in each semantic candidate package, the observable feature vector is concatenated with the candidate ranking feature vector corresponding to the candidate semantic line to form a joint feature vector, and the joint feature vector is semantically classified by a classifier to obtain the predicted semantic label and classification score of the candidate semantic line. Select the candidate semantic row with the highest classification score from each semantic candidate bag, and use its predicted semantic label as the candidate label of the corresponding semantic source; Using the calibration confidence level corresponding to each semantic source as the weight, a weighted vote is performed on the candidate labels of all semantic sources to obtain the fusion score of each candidate label; The candidate label with the highest fusion score is used as the main semantic label of the corresponding 3D instance unit.

Images