Three-dimensional-textual retrieval dataset construction method and retrieval method based on component-level semantic alignment

Abstract

The present application belongs to the technical field of natural language processing, and relates to a three-dimensional-text retrieval dataset construction method and retrieval method based on component-level semantic alignment. The construction method comprises: obtaining point cloud and component segmentation annotation; mapping the segmentation region to a pre-defined component category to generate component-level semantic labels; assigning colors to the components to generate colored point cloud and rendered images; generating structured text description in combination with component attributes; and constructing a three-modal component-level corresponding dataset. The segmentation annotation and semantic labels are only used for construction and do not participate in training. The retrieval method trains a model using the dataset: through a modality-specific slot attention module, geometry encoding is introduced for point cloud, position embedding is introduced for images, and semantic weighting is introduced for text, so that local features are decomposed into component-level slot representations; through a prototype-guided component matching module, the local features are mapped to a unified semantic space to realize fine-grained cross-modal retrieval without manual component annotation.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, specifically relating to a method for constructing and retrieving a 3D-text retrieval dataset based on component-level semantic alignment. Background Technology

[0002] Cross-modal 3D-text retrieval aims to establish semantic correspondences between 3D shapes and natural language descriptions, with wide applications in scenarios such as autonomous driving, embodied intelligence, and virtual reality. Existing methods are mainly divided into two categories: global semantic alignment and part-level alignment. Global alignment methods, represented by Text2Shape, encode 3D shapes and text descriptions into global feature vectors respectively, and construct a joint embedding space through metric learning to achieve cross-modal matching. Building on this, methods such as ULIP and OpenShape utilize the image-text alignment space of CLIP to establish semantic associations between point clouds and text using images as mediators. To compensate for the shortcomings of global representation in modeling local structures, part-level alignment methods such as Parts2Words and SCA3D attempt to establish correspondences between local regions of point clouds and text words, improving fine-grained part matching capabilities through optimal transmission or part-level data augmentation. Furthermore, to support part-level 3D understanding, datasets such as PartNet provide hierarchical part segmentation annotations, laying the foundation for part-level shape analysis.

[0003] However, existing methods still suffer from the following problems in 3D-text retrieval: First, global semantic alignment methods compress 3D shapes into a single global feature, making it difficult to retain fine-grained information at the component level within a class, resulting in insufficient ability to distinguish subtle structural differences among similar objects; Second, component-level alignment methods rely on externally provided component segmentation annotations or predefined component partitioning methods, making it difficult to learn stable component representations in the absence of explicit component segmentation annotations, while obtaining high-precision component segmentation annotations is costly, limiting the generalization ability of the method; Third, there is an information asymmetry between 3D shapes and text descriptions, as text usually only describes salient components while 3D shapes contain complete geometric structures, and a large number of 3D regions without semantic correspondence to the text are prone to noise interference in the feature space; Fourth, existing datasets exhibit inconsistencies in semantic annotations at the component level, with the same physical component potentially corresponding to different semantic labels in different samples, and there is a lack of direct correspondence between the annotation semantics and the natural language description, making it difficult to support the training and evaluation of fine-grained cross-modal alignment tasks. Summary of the Invention

[0004] The purpose of this invention is to provide a method for constructing and retrieving 3D-text retrieval datasets based on component-level semantic alignment, in order to solve the problems in existing 3D-text cross-modal retrieval, such as the difficulty in capturing component-level differences due to global features, reliance on expensive component annotations, and matching noise caused by asymmetric modal information.

[0005] The present invention achieves the above objectives through the following technical solutions: Firstly, this invention proposes a method for constructing a 3D-text retrieval dataset based on component-level semantic alignment, characterized in that the method includes: Obtain point cloud coordinate information and component segmentation annotations of a 3D object. The component segmentation annotations are used to identify the component region to which each point in the point cloud belongs. Semantic mapping is performed on the component segmentation annotations. Based on the point cloud geometric features, each segmented region is determined as a semantic component in a predefined component category system, and component-level semantic labels are generated. Assign color identifiers to each semantic component, generate a colored point cloud, and render the colored point cloud to obtain a multi-view rendered image; Extract the component-level structural attributes and appearance attributes from the multi-view rendering image, and generate a structured text description by combining the component-level semantic tags and color identifiers; The tinted point cloud, multi-view rendered image, and structured text description are stored in a cross-modal association to obtain the retrieval dataset corresponding to the 3D object.

[0006] Furthermore, the three-dimensional object is a furniture-type object; the step of determining each segmented region as a semantic component in a predefined component category system based on point cloud geometric features includes: Calculate the geometric features of each segmented region, including the position and height of the segmented region relative to the whole object, the horizontal extension range, the length-to-width ratio, and the spatial offset. The geometric features are matched with the standard geometric templates of each semantic component, and the component category to which the segmented region belongs is determined based on the matching degree. The predefined component category system includes at least: chair back, chair seat, armrests, chair legs, chair base, tabletop, table legs, and table base.

[0007] Secondly, this invention proposes a three-dimensional text-based cross-modal retrieval method, comprising the following steps: The retrieval model is trained using the retrieval dataset described above, and the global contrast loss and component alignment loss are jointly optimized during the training process. The three-dimensional point cloud, rendered image, and text query to be retrieved are used as the modality to be processed. Local features and global features of the modality to be processed are extracted respectively. Local features include point-by-point features of the point cloud modality, block features of the image modality, and word features of the text modality. The global and local features of the modality to be processed are input into the trained retrieval model, which includes a modality-specific slot attention module and a component matching module. The local features of the modality to be processed are input into the modality-specific slot attention module, which introduces geometric encoding for the point cloud, position embedding for the image, and semantic weighting for the text. After iterative updates, the local features of each modality are decomposed into slot representations of each component in the corresponding predefined component category system. The slot representations of each modality are input into the prototype-guided component matching module, and the slot representations of each modality are mapped to a unified semantic space to obtain the component-level embeddings of each modality. The similarity is calculated based on the component-level embedding, and the retrieval results are output.

[0008] Furthermore, the local features of each modality are decomposed into slot representations of each component in the corresponding predefined component category system, including: Determine the encoding feature matrix of the local features of the modality to be processed; Based on the encoded feature matrix and the preset slot initialization matrix, the preset slot initialization matrix is ​​used as the initial value of the slot state, and the slot state is updated multiple times through an iterative slot attention mechanism. A single iteration of the iterative slot attention mechanism includes: using the encoded feature matrix as input features, calculating attention weights through Softmax normalization, and weighted aggregation of the input features into an update vector. The current slot state is updated via a gated loop unit and further refined through a feedforward network with residual connections. After the second iteration, a stable component-level slot representation is obtained. Including the following formulas: ; in, For the first Each slot indicates that in In the state of this iteration, FFN represents the feedforward network, LN represents layer normalization, and GRU represents the gated recurrent unit. This represents the intermediate state output by the GRU.

[0009] Furthermore, the modality-specific slot attention module obtains the encoding feature matrix of each modality's local features in the following way: For point cloud modalities, based on the three-dimensional coordinates of each point in the point cloud and their... The relative geometric features are calculated from the relative displacement vectors of the nearest neighbor points. The relative geometric features include the relative displacement vector, Euclidean distance, squared distance, unit direction vector, azimuth angle, and elevation angle. The relative geometric features are then incorporated into the key calculation of the slot attention through a dual-path approach. The first path fuses the geometric features with the point-by-point features before key mapping, while the second path projects the geometric features independently and then directly injects the key representation to obtain the encoded feature matrix of the point cloud mode. For image modalities, a learnable two-dimensional position embedding matrix is ​​generated based on the spatial position coordinates of image patch features. The two-dimensional position embedding matrix is ​​then explicitly superimposed onto the image patch features to obtain the encoding feature matrix of the image modality. For the text modality, based on the original feature matrix of text lexical units, the importance score of each lexical unit is predicted by a multilayer perceptron. The original lexical unit features are then adaptively enhanced based on the importance scores to obtain the encoding feature matrix of the text modality.

[0010] Furthermore, the modality-specific slot attention module also utilizes image modality to generate slot activation priors, and the slot activation probability of each modality is obtained in the following way: Using the global features of the image modality as input, the data is mapped by a multilayer perceptron and activated by a sigmoid to generate prior activation probabilities for each slot. These prior activation probabilities are then propagated to the point cloud modality and the text modality to constrain the number of effective slots that remain consistent across different modalities for the same 3D object. Determine the self-activation probability of each modal slot feature, and map the slot features to scalars based on the modal's own activation predictor and activate them through the Sigmoid function; The prior activation probability is weighted and fused with the self-activation probability to obtain the final slot activation probability for each mode. As shown in the following formula: ; in, Representing modes The activation predictor is used to map slot features, The value is a scalar, and the slot activation estimate for this mode is obtained by passing it through the Sigmoid function. The first image modality derived from the image modality Prior probability of each slot This is the fusion coefficient.

[0011] Furthermore, the component matching module maps each modal slot representation to a unified semantic space in the following manner: The semantic description of each component category in the predefined component category system is determined, and the semantic description is encoded into a text feature vector by the CLIP text encoder. The initial anchor point of the shared semantic prototype is obtained by linear projection and L2 normalization. Learnable offsets are superimposed on the initial anchor points to obtain an optimizable shared semantic prototype, and the Dustbin prototype is randomly initialized. The cosine similarity between each modal slot feature and the shared semantic prototype and Dustbin prototype is calculated, and the soft-assigned weights are obtained by temperature scaling and Softmax normalization. Based on the soft-assigned weights, the shared semantic prototypes are weighted and summed to obtain the prototype features corresponding to each slot. The original slot features and the prototype features are then adaptively fused through a gating mechanism to obtain the component-level embeddings of each modality.

[0012] Furthermore, the Dustbin prototype is used to absorb noisy slots that do not match any of the shared semantic prototypes; when calculating the soft allocation weights, logarithmic bias terms in different directions are added to the shared semantic prototypes and the Dustbin prototypes respectively, so that slots with lower activation probabilities are more likely to be allocated to the Dustbin prototypes.

[0013] Furthermore, the global contrast loss is calculated in the following manner: Global features of three modalities—text, image, and point cloud—are determined, and L2 normalization is performed on the global features. Based on the symmetric InfoNCE loss function, the bidirectional contrast loss of text-image pairs, text-point cloud pairs, and image-point cloud pairs is calculated respectively; The global contrastive loss is obtained by averaging the contrastive losses of the three modality pairs.

[0014] Furthermore, the component alignment loss is calculated in the following manner: Determine the component-level embedding of each mode and the slot activation probability and soft activation mask of each mode; The component-level embeddings of each modality are weighted and aggregated based on the slot activation probability and soft activation mask to obtain the component-aware sample-level representation. InfoNCE loss is then applied to the sample-level representation to obtain the inter-sample contrast loss. By comparing the weights assigned to shared semantic prototypes by point cloud component-level embeddings and text component-level embeddings, the cross-modal component correspondence strength is calculated, a soft correspondence matrix is ​​generated, and the similarity between cross-modal corresponding component embeddings within the same sample is maximized based on the soft correspondence matrix to obtain the intra-sample alignment loss. The component alignment loss is obtained by weighted summing of the inter-sample contrast loss and the intra-sample alignment loss.

[0015] The beneficial effects of this invention are as follows: 1. This invention constructs the FG3D fine-grained 3D-text benchmark dataset, maps segmented regions to a unified component category system through geometric features, and generates colored point clouds and multi-view rendered images using a deterministic color matching strategy. It also automatically generates structured text descriptions using a visual language model, achieving strict semantic correspondence between point clouds, images, and text at the component level. This solves the problems of inconsistent component annotation and high annotation costs in existing datasets.

[0016] 2. This invention constructs a modality-specific slot attention mechanism, which introduces geometric encoding, position embedding and semantic weighting for point clouds, images and text respectively, enabling the model to adaptively decompose input features into component-level slot representations without the need for manual component annotation, thereby reducing the dependence on data annotation.

[0017] 3. The present invention constructs a prototype-guided component matching module, which maps the slot representations of heterogeneous modalities to a unified semantic space by sharing semantic prototypes, and uses Dustbin prototypes to absorb noisy slots that are unrelated to semantic components, effectively mitigating matching interference caused by information asymmetry between modalities.

[0018] 4. This invention jointly optimizes the global contrast loss and the component alignment loss, enabling the model to establish a stable cross-modal correspondence at the component granularity, and supporting bidirectional fine-grained retrieval from shape to text and from text to shape. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a method for constructing a three-dimensional text retrieval dataset in Embodiment 1 of the present invention; Figure 2 This is a flowchart illustrating a three-dimensional text cross-modal retrieval method in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the text length distribution in the FG3D dataset in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram showing the comparison of semantic coverage between the FG3D dataset and Text2Shape in Embodiment 1 of the present invention; Figure 5 This is an overall architecture diagram of the fine-grained three-dimensional-text cross-modal retrieval framework proposed in Embodiment 2 of the present invention; Figure 6 This is a schematic diagram of the text-to-shape retrieval results provided in Embodiment 2 of the present invention; Figure 7 This is a schematic diagram of the visualization results of slot decomposition and cross-modal component alignment provided in Embodiment 2 of the present invention. Detailed Implementation

[0020] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content.

[0021] Example 1

[0022] This embodiment first constructs a fine-grained 3D-text benchmark dataset, denoted as FG3D (Fine-Grained3D). This dataset selects furniture category samples from ShapeNet, uses the original point cloud and segmentation results provided by PartNet as the geometric basis, and is constructed through an automated process that combines visual language models with geometric rules.

[0023] Unlike existing public datasets such as Text2Shape and PartNet, the FG3D dataset constructed in this embodiment is the first fine-grained benchmark that models 3D-text consistency at the component semantic level. Each sample consists of a shading point cloud with component-level semantic information, a multi-view rendered image, and a text description. The three modalities have a consistent semantic correspondence at the component level.

[0024] Please see Figure 1 This embodiment proposes a method for constructing a 3D-text retrieval dataset based on component-level semantic alignment. The method includes the following steps: Step 101: Obtain the point cloud coordinate information and component segmentation annotation of the 3D object. The component segmentation annotation is used to identify the component region to which each point in the point cloud belongs.

[0025] Specifically, this embodiment selects furniture category samples from ShapeNet and uses the original point cloud and segmentation results provided by PartNet as the geometric basis. PartNet provides hierarchical component segmentation annotations covering 24 categories, with each point in the point cloud labeled as belonging to a certain component region.

[0026] Step 102: Perform semantic mapping on the component segmentation annotations, determine each segmented region as a semantic component in the predefined component category system based on the point cloud geometric features, and generate component-level semantic labels.

[0027] Specifically, the FG3D dataset is constructed through an automated process combining visual language models and geometric rules. First, semantic determination of the original PartNet segmented regions is performed based on point cloud geometric features. In this embodiment, the geometric features of each segmented region are calculated, including the region's position and height relative to the overall object, its horizontal extension range, aspect ratio, and spatial offset. Then, these geometric features are matched with standard geometric templates for each semantic component. The component category of the segmented region is determined based on the matching degree and uniformly mapped to a predefined component category system.

[0028] The predefined component category system includes at least the following component types: chair back, chair seat, armrests, chair legs, chair base, tabletop, table legs, and table base. This geometric inference strategy ensures that the same functional component has consistent semantic labels across different samples, resolving the inconsistency issue caused by differences in annotator interpretation in the original PartNet annotations.

[0029] It should be noted that this component category system is predefined for furniture objects. For other categories of 3D objects, the corresponding component types can be adjusted according to the actual application scenario.

[0030] Step 103: Assign color identifiers to each semantic component, generate a colored point cloud, and render the colored point cloud to obtain a multi-view rendered image.

[0031] Specifically, this embodiment employs a deterministic color matching strategy to assign color identifiers to each semantic component. For example, the chair back is assigned brown, the seat is assigned black, and the chair legs are assigned gray. This ensures that the point cloud coloring results are strictly consistent with the color vocabulary in the text.

[0032] Perform multi-view rendering on the colored point cloud, setting multiple rendering views at uniform intervals around the vertical axis of the 3D object.

[0033] Step 104: Extract the component-level structural attributes and appearance attributes from the multi-view rendering image, and generate a structured text description by combining the component-level semantic tags and color identifiers.

[0034] Specifically, multi-view rendered images are input into a visual language model (Qwen-VL-Max is used in this embodiment) to extract key attributes such as object type, supporting structure, and appearance, including at least one of the following: component shape description, relative positional relationship of components, surface color of components, material of components, and geometric size ratio of components. Then, a structured text description is generated by integrating component semantics, color information, and structural attributes. The text description is generated using a preset template and includes an object type field, a component list field, a color field for each component, and a structural attribute field for each component.

[0035] Step 105: Perform cross-modal association storage of the shading point cloud, multi-view rendered image and structured text description to obtain the retrieval dataset corresponding to the 3D object.

[0036] Specifically, the data of the above three modalities are stored in a cross-modal association. Each sample contains a shading point cloud of a 3D object, a multi-view rendered image, and a structured text description. The three modalities have a consistent semantic correspondence at the component level.

[0037] It should be noted that the component segmentation annotations and component-level semantic labels in the dataset are only used for component category mapping, color assignment, and text generation during the construction phase. They are not stored externally as part of the dataset, nor are they included in the loss function calculation for model training. Therefore, this dataset supports the training of cross-modal retrieval models without explicit component annotation supervision.

[0038] After completing the above dataset construction, this embodiment performs statistical analysis and quality assessment on the constructed FG3D dataset.

[0039] I. Statistical Results of the Dataset The FG3D dataset constructed in this embodiment contains 14,538 samples, which are divided into training set, validation set and test set in a ratio of 8:1:1. The training set is used for model training, the validation set is used for hyperparameter tuning, and the test set is used for performance evaluation.

[0040] The specific composition of each sample is as follows: (1) Point cloud data: Each sample contains 10,000 color-labeled 3D points. Each point records 3D coordinate information and color label. The color label in the point cloud is consistent with the component color assigned in step 103.

[0041] (2) Rendered images: Each sample is equipped with four rendered images from different perspectives, which are evenly set at 90-degree intervals around the vertical axis of the three-dimensional object, and the output resolution is 224×224.

[0042] (3) Text description: Each sample is accompanied by a structured text description with an average length of 12.4 words. The text length is mainly concentrated in the range of 10-16 words, and the overall length is moderate.

[0043] The core statistical metrics of the dataset are shown in Table 1 below: Table 1 Core Statistical Indicators of FG3D Dataset II. Text Description Quality Analysis This embodiment performs quality analysis on the text descriptions of the FG3D dataset and compares them with the Text2Shape dataset.

[0044] like Figure 3 As shown, the text description length of the FG3D dataset is mainly concentrated in the range of 10-16 words, with an overall moderate length. This length distribution ensures that the text description contains sufficient component-level semantic information (such as component type, color, shape, positional relationship, etc.) while avoiding redundant information caused by excessively long texts.

[0045] like Figure 4 As shown, in terms of semantic coverage, the FG3D dataset has higher completeness in component-level semantic representation. Specifically: (1) Component word coverage: FG3D reaches 100%, while Text2Shape has a large amount of text descriptions that do not contain component words and only describe overall attributes.

[0046] (2) Color word coverage: FG3D reaches 95%, while Text2Shape has a lower color word coverage and inconsistent color annotation.

[0047] (3) Geometric attribute description: FG3D's text description includes fine-grained information such as part shape description, relative position relationship of parts, and geometric size ratio of parts, while Text2Shape mainly focuses on overall geometric attributes.

[0048] The above analysis results show that the FG3D dataset provides richer component-level supervision signals in the text description dimension compared to Text2Shape, making it more suitable for training and evaluating fine-grained cross-modal retrieval tasks. This is also an important reason why the FG3D dataset was chosen as the training data for the retrieval model in this embodiment.

[0049] Example 2

[0050] This embodiment uses the FG3D dataset constructed in Embodiment 1 to train the retrieval model. The structured text descriptions in this dataset are generated using preset templates and include object type fields, part list fields, color fields for each part, and structural attribute fields for each part. For example, for a chair sample, the generated text description could be "a chair with a brown square back, a black round seat, and four gray cylindrical legs." This high-coverage part-level semantic description provides sufficient supervision for training the fine-grained cross-modal retrieval model.

[0051] Please see Figure 5This embodiment proposes a fine-grained 3D-text cross-modal retrieval framework, whose input includes three modalities: point cloud, multi-view images, and text descriptions. In the feature extraction stage, a Dynamic Graph Convolutional Neural Network (DGCNN) extracts point-by-point features and global features from the point cloud, while a frozen contrastive language-image pre-trained encoder (CLIP encoder) extracts image patch features, lexical features, and global features from the image and text, respectively. Subsequently, the local features of each modality are input into a modality-specific slot attention module to generate part-level slot representations and their activation probabilities. Here, the slot is a feature vector learned from the input features through a slot attention mechanism, representing the semantic information at the part level of an object. A prototype-guided part matching module further maps the slot features to a shared semantic prototype space, using Dustbin prototypes (also known as discarded prototypes, used to absorb noisy slots that do not match any shared semantic prototypes) to suppress noise interference from slots without semantic correspondence, and obtains part-level embedding representations for each modality. Finally, fine-grained 3D-text retrieval is achieved through joint optimization of global contrastive loss and part alignment loss.

[0052] Please see Figure 2 This embodiment proposes a three-dimensional text cross-modal retrieval method, including the following steps: Step 201: Train the retrieval model using the retrieval dataset as described in Example 1, and jointly optimize the global contrast loss and component alignment loss during the training process.

[0053] Specifically, training samples from the dataset of Example 1 are obtained. The training samples include 3D point clouds, multi-view rendered images, and structured text descriptions, and the three modalities have strict semantic correspondence at the component level.

[0054] The training samples are input into the retrieval model, and the global contrast loss and component alignment loss are calculated. Based on the weighted sum of the global contrast loss and component alignment loss, the parameters of the retrieval model are updated using the backpropagation algorithm. The model is iteratively trained until the loss converges, resulting in a well-trained retrieval model.

[0055] Step 202: Obtain the 3D point cloud, rendered image, and text query to be retrieved as the modality to be processed. Extract the local features and global features of the modality to be processed. The local features include the point-by-point features of the point cloud modality, the block features of the image modality, and the lexical features of the text modality.

[0056] Specifically, the point cloud modality employs a learnable DGCNN encoder to extract point-by-point and global features. The input consists of 2048 points randomly downsampled from each sample, with an embedding dimension of 512. All parameters are used in training. The image and text modalities extract image patch features, word features, and corresponding global features using a frozen CLIP encoder, respectively. The input image resolution is 224×224, and each sample provides four rendering perspectives. During training, one perspective is randomly sampled as input. The image serves as an intermediary modality between 3D geometry and linguistic semantics within the framework.

[0057] Step 203: Input the global and local features of the modality to be processed into the trained retrieval model, which includes a modality-specific slot attention module and a component matching module.

[0058] Step 204: Input the local features of the modality to be processed into the modality-specific slot attention module, introduce geometric encoding for the point cloud, introduce position embedding for the image, and introduce semantic weighting for the text. After iterative updates, the local features of each modality are decomposed into slot representations of each component in the corresponding predefined component category system.

[0059] Specifically, the modality-specific slot attention module maintains independent parameters for point clouds, images, and text modalities, while maintaining a consistent slot iteration update framework across all modalities. The differences between the three modality branches mainly lie in the preprocessing methods of the input features: geometric encoding is introduced for point clouds, position embedding is introduced for images, and semantic weighting is introduced for text.

[0060] (1) Geometric encoding of point cloud modes For point cloud modalities, local geometric encoding is introduced. Specifically, this includes encoding each point in the point cloud and its corresponding geometric modalities. Calculate the relative geometric features of the nearest neighbors, including the relative displacement vector, Euclidean distance, squared distance, unit direction vector, azimuth angle, and elevation angle.

[0061] As an example, given the pointwise features extracted by DGCNN and three-dimensional coordinates Relative geometric encoding is introduced into the point cloud branch. For each point... and Neighbor Calculate the relative geometric features: ; in It is a two-layer, multi-layer sensor. For point and neighbors The relative displacement vector between them The Euclidean distance between the two points is... Let be the unit direction vector of the relative displacement. and These are the azimuth and elevation angles in spherical coordinates, respectively.

[0062] The aforementioned features characterize local neighborhood relationships from multiple aspects, including location, orientation, and scale. These relative geometric features are incorporated into the key calculation of slot attention using a dual-path approach, as shown in the following equation: ; in, , and All are learnable linear projection matrices. and To control the contribution coefficients of the two paths, the first path fuses geometric features with point-by-point features before key mapping, while the second path projects geometric features independently and directly injects the key representation. This dual-path fusion approach allows the point cloud slot attention to simultaneously consider local geometric consistency and component boundary sensitivity. The advantage of this design is its enhanced ability to distinguish spatially adjacent points that belong to different semantically components, such as distinguishing between adjacent chair backs and armrests.

[0063] (2) Position embedding of image modalities For image modalities, spatial location embedding is introduced. Specifically, this involves explicitly overlaying learnable 2D location embeddings into image patch features, enabling spatially adjacent image patches to maintain higher consistency in the feature space. The advantage of this design is that it helps slots preferentially aggregate spatially adjacent and semantically related image patches into representations of the same component during iterative aggregation, such as aggregating multiple image patches belonging to the same chair back region together.

[0064] As an example, given the image patch features extracted by the CLIP visual encoder Introducing learnable 2D location embeddings into the image branch This information is then explicitly overlaid into image patch features to supplement task-related spatial location information. ; This design enables spatially adjacent image patches to maintain higher consistency in the feature space, thereby helping the slot to preferentially aggregate spatially adjacent and semantically related image patches into the same part representation during iterative aggregation. Enhanced image features As input for slot attention.

[0065] (3) Semantic weighting of text modalities For the text modality, semantic importance weighting is introduced. Specifically, this includes predicting an importance score for each text lexical unit, and adaptively enhancing the original lexical unit features based on the importance scores. The enhanced lexical unit features are then used as input for slot attention.

[0066] As an example, given the lexical features output by the CLIP text encoder Different lexical units contribute significantly differently to component semantic modeling. Indiscriminately aggregating low-information lexical units such as function words and conjunctions with high-semantic-information lexical units can easily introduce additional noise and weaken the discriminative power of component-level representations. Therefore, this embodiment introduces a semantic importance module into the text branch to predict the importance score of each lexical unit and adaptively enhance the original features accordingly. ; in, For the first The original features of each word element It is a two-layer, multi-layer sensor. It is the Sigmoid activation function. Scaling factor. Importance score. The higher the value, the greater the weight of the corresponding lexical feature after enhancement, thus having a more significant impact on slot updates in subsequent attention calculations. The enhanced lexical features are then linearly projected to obtain keys and values, which are then incorporated into the subsequent slot update process. After several iterations, each slot gradually converges into a stable component-level text representation.

[0067] (4) Slot attention iterative update mechanism The modality-specific slot attention module decomposes the local features of each modality into slot representations of each component in the corresponding predefined component category system as follows: determine the encoding feature matrix of the local features of the modality to be processed; based on the encoding feature matrix and the preset slot initialization matrix, use the preset slot initialization matrix as the initial value of the slot state, and update the slot state multiple times through an iterative slot attention mechanism.

[0068] In this embodiment, the maximum number of slots is set to 6, the minimum number of active slots is set to 2, and the number of slot attention iterations is set to 3. After multiple iterations, each slot gradually converges into a stable component-level slot representation.

[0069] A single iteration of the iterative slot attention mechanism includes: using the encoded feature matrix as input features, calculating attention weights through Softmax normalization, and weighted aggregation of the input features into an update vector. The current slot state is updated via a gated loop unit and further refined through a feedforward network with residual connections. After several iterations, a stable component-level slot representation is obtained, including the following formula: ; in, For the first Each slot indicates that in In the state of this iteration, FFN represents the feedforward network (consisting of two fully connected layers and an activation function), LN represents layer normalization, and GRU represents the gated recurrent unit. This represents the intermediate state output by the GRU.

[0070] (5) Determination of slot activation probability The modality-specific slot attention module also utilizes image modalities to generate slot activation priors. The slot activation probabilities of each modality are obtained as follows: using the global features of the image modality as input, mapping them through a multilayer perceptron and activating them with a Sigmoid function to generate slot-by-slot prior activation probabilities; determining the intrinsic activation probability of each modality's slot features, mapping the slot features to scalars based on the modality's own activation predictor and activating them with a Sigmoid function; and weighting and fusing the prior activation probabilities with the intrinsic activation probabilities to obtain the final slot activation probabilities of each modality. As shown in the following formula: ; in, Representing modes The activation predictor is used to convert slot features The mapping is performed as a scalar, and the slot activation estimate for this mode is obtained via the Sigmoid function. The first image modality derived from the image modality Prior probability of each slot The fusion coefficient; The prior activation probabilities are propagated to the point cloud modality and the text modality, thereby constraining the number of effective slots for the same 3D object across different modalities through the global structural prior of the image modality, thus avoiding matching noise caused by modal information asymmetry. For example, for a chair sample with 3 parts, all modalities should activate 3 slots.

[0071] Step 205: Input the slot representation of each modality into the prototype-guided component matching module, map the slot representation of each modality to a unified semantic space, and obtain the component-level embedding of each modality.

[0072] Specifically, the component matching module maps each modal slot representation to a unified semantic space in the following manner: (1) Determine the semantic description of each component category in the predefined component category system, encode the semantic description into a text feature vector using the CLIP text encoder, and obtain the initial anchor point of the shared semantic prototype through linear projection and L2 normalization. For example, the semantic description of the chair back is denoted as... The initial anchor point is obtained after encoding, projection, and normalization. (i.e., semantic anchors).

[0073] (2) Based on the initial anchor point, a learnable offset is superimposed to obtain an optimizable shared semantic prototype, and the Dustbin prototype is randomly initialized; the relevant formulas are as follows: ; in, This indicates the CLIP text encoder. Indicates a linear projection layer. It represents a fixed semantic anchor point, serving as a stable semantic reference; This represents a learnable offset with a limited magnitude, used for local adjustments to the prototype position near the anchor point. The Dustbin prototype is randomly initialized and jointly learned during training.

[0074] (3) Calculate the cosine similarity between the features of each modal slot and the shared semantic prototype and Dustbin prototype, and obtain the soft assignment weights by Softmax normalization after temperature scaling. Unlike hard assignment, soft assignment allows each slot to be associated with multiple prototypes with different weights at the same time, rather than being forced to be assigned to a single prototype. The advantage of this design is that it can more naturally model the progressive transition relationship between parts. For example, a slot may have some features of both "chair back" and "armrest".

[0075] When calculating the allocation weights, logarithmic bias terms in different directions are added to the shared semantic prototype and the Dustbin prototype, making it easier to assign slots with lower activation probabilities to the Dustbin prototype. The Dustbin prototype is used to absorb noisy slots that do not match any of the shared semantic prototypes, such as background regions in the point cloud that are unrelated to the text description.

[0076] In practice, the assigned weight of each slot relative to all prototypes is calculated. Given L2-normalized slot features... and each prototype First, the cosine similarity between the two is calculated, and then the temperature coefficient is used to determine the similarity. Scaling is then applied. Subsequently, slot activation probabilities are introduced. Before Softmax normalization, logarithmic bias terms in different directions are added to the semantic prototype and Dustbin prototype respectively to obtain the adjusted pre-normalized scores. The final allocation weights are then obtained through Softmax normalization. : ; in, Represents the first mode in mode m. The first sample The slot was assigned to the first The weights of each prototype are determined, and normalization constraints are satisfied. The activation probability of the slot is... When the probability is high, the allocation weight tends to be concentrated on the semantic prototype; conversely, slots with low activation probability are more likely to be assigned to the Dustbin prototype.

[0077] In the above weight allocation, the first The weights corresponding to each prototype represent the degree of association between the slot and the semantic prototype, denoted as . (j≤N) c Based on this, define the soft activation slot mask: ; This mask is equal to the sum of the weights assigned to the corresponding slot across all semantic prototypes. When the weights assigned to a slot are primarily concentrated on the Dustbin prototype, The value will approach zero, thus naturally suppressing the contribution of the slot in subsequent alignments. Since the mask is continuously differentiable with respect to the assigned weights, it is more conducive to maintaining the stability of gradient propagation.

[0078] (4) Based on the soft allocation weight, the shared semantic prototype is weighted and summed to obtain the prototype feature corresponding to each slot, and the original slot feature and the prototype feature are adaptively fused through the gating mechanism to obtain the component-level embedding of each modality.

[0079] In practice, prototype semantic information is injected into the slot representation and mapped to a unified feature space. First, the prototypes are weighted and summed according to semantic weights to obtain the prototype-weighted features corresponding to each slot: ; Subsequently, the prototype semantics and the original slot features are adaptively fused using a gating fusion mechanism: ; in, For learnable parameters, This represents the Sigmoid activation function. This represents concatenation along the feature dimension, with element-wise multiplication denoted as... Gating vector It can adaptively adjust the fusion ratio of prototype semantics and original slot features along the feature dimension. The fused features are further mapped to shared spatial part embeddings through a projection network with residual connections. ; in, It is a two-layer, multi-layer perceptron. Soft activation mask. The embedding of non-active slots is set to zero, and only semantically valid part representations are retained to participate in subsequent cross-modal matching.

[0080] In a shared semantic prototype space, cross-modal component correspondences are established by assigning weights to semantic prototypes through comparison slots. Given a modality... The individual slots and modes The For each slot, the corresponding strength between them is defined as the inner product of their respective semantic assignment weight vectors: ; This correspondence strength considers only the semantic prototype portion and excludes the Dustbin prototype, thus more directly reflecting the semantic consistency of the two slots in the component. When two slots have higher assigned weights on the same semantic prototype, their correspondence strength increases accordingly, indicating that they are more likely to correspond to the same physical component. Based on this, this embodiment further obtains the soft correspondence matrix through masked Softmax normalization: ; in, This represents the temperature coefficient. The mask term is used to suppress the interference of inactive slots on the matching process, thereby improving the stability of the correspondence. The resulting soft correspondence matrix will be used in the above component alignment loss to construct cross-modal component alignment constraints.

[0081] Step 206: Calculate similarity based on the component-level embedding and output the retrieval results.

[0082] Specifically, for the shape-to-text retrieval task, the comprehensive similarity between the query point cloud and the candidate text in the global feature space and the component-level embedding space is calculated. The candidate samples are then sorted from high to low similarity, and the sorted results are returned as the retrieval output. In this embodiment, cosine similarity is used as the similarity metric.

[0083] In this embodiment, global contrast loss Alignment loss with components The specific calculation method is as follows: (1) Global contrast loss The global comparison loss A symmetric InfoNCE loss is employed to bridge the gap between global features across text, image, and point cloud modalities. Specifically, global features for the three modalities are determined and L2 normalized. Based on the symmetric InfoNCE loss function, bidirectional contrast losses for text-image pairs, text-point cloud pairs, and image-point cloud pairs are calculated separately. The average contrast loss of the three modal pairs is then used to obtain the global contrast loss.

[0084] As an example, for each modal pair ,in , , Representing text, image, and point cloud modalities respectively, a symmetric InfoNCE loss is applied to the L2-normalized global features in the shared embedding space: ; in, For batch size, and They represent the first Each sample in modality and L2 normalized global features For temperature coefficient, This represents the inner product operation. The final global loss is obtained by averaging the bidirectional contrastive losses of all mode pairs: ; (2) Component alignment loss The component alignment loss Including inter-sample contrast loss and in-sample alignment loss Specifically, the component-level embedding of each modality and the slot activation probability and soft activation mask of each modality are determined. The soft activation mask is obtained by summing the weights of the slots assigned to all semantic prototypes. When the assigned weights of the slots are mainly concentrated on the Dustbin prototype, the soft activation mask approaches zero, thus naturally suppressing the contribution of the slots in subsequent alignment.

[0085] Based on the slot activation probability and soft activation mask, the component-level embeddings of each modality are weighted and aggregated to obtain the component-aware sample-level representation, as shown in the following equation: ; in, This indicates the embedding of shared space components after L2 normalization. Indicates the soft-active slot mask. This represents the probability of slot activation.

[0086] Applying the InfoNCE loss to the sample-level representation yields the inter-sample contrast loss. The advantage of this design is that the aggregated representation retains fine-grained structural information, and samples with similar semantics within a batch are more likely to form difficult negative samples, thus prompting the model to further distinguish component-level differences. The specific formula is as follows: ; By comparing the weights assigned to shared semantic prototypes by point cloud component-level embeddings and text component-level embeddings, cross-modal component correspondence strength is calculated, generating a soft correspondence matrix. Specifically, given a modality... The individual slots and modes The The correspondence strength between two slots is defined as the inner product of their respective semantic assignment weight vectors, considering only the semantic prototype part and excluding the Dustbin prototype, thus more directly reflecting the semantic consistency of the two slots in the component. When two slots have high assignment weights on the same semantic prototype, such as both corresponding to the "chair back" prototype, their correspondence strength increases accordingly. Based on the soft correspondence matrix, the similarity between cross-modal corresponding component embeddings within the same sample is maximized to obtain the intra-sample alignment loss.

[0087] The intra-sample alignment loss is based on the soft correspondence matrix constructed above, maximizing the feature similarity between cross-modal corresponding slots within the same sample: ; in, and Representing modes The individual slots and modes The L2-normalized component embedding for each slot. Soft correspondence matrix. As weights, slot pairs with higher correspondence strengths are given a larger proportion in the loss. Since this correspondence matrix is ​​in soft correspondence form, a single slot can establish correspondences of varying degrees with multiple cross-modal slots.

[0088] The final component alignment loss is averaged over all mode pairs: ; (3) Total loss function The total loss of the model is obtained by weighted summing of the global contrast loss and the component alignment loss: ; in, This represents the balance coefficient. The two losses are jointly optimized from the start of training, without employing a phased training strategy.

[0089] Experimental verification To verify the effectiveness of the present invention, this embodiment was experimentally evaluated on two datasets: Text2Shape and FG3D.

[0090] I. Experimental Setup (1) Dataset The Text2Shape dataset is a standard benchmark in the field of cross-modal 3D-text retrieval, containing 6,591 chair and 8,447 table 3D models from ShapeNet, each with approximately 5 manually annotated natural language descriptions. This embodiment follows the standard training, validation, and testing split of Text2Shape to ensure a fair comparison with existing work.

[0091] The FG3D dataset, the fine-grained benchmark dataset constructed in Example 1, is based on PartNet point cloud data. Each sample includes a colored point cloud with part-level semantic annotations, a multi-view rendered image, and a text description that integrates part semantics, color, and structural attributes. Compared to Text2Shape, FG3D's text descriptions provide a more complete characterization of the object's type, structure, and appearance features at the part level, thus placing higher demands on the model's fine-grained cross-modal retrieval capabilities.

[0092] (2) Evaluation indicators Following existing work, this embodiment employs standard evaluation metrics for cross-modal retrieval: Recall at K (RR@K, K=1,5) and Normalized Discounted Cumulative Gain (NDCG@5), and evaluates them in two directions: Shape-to-Text (S2T) and Text-to-Shape (T2S). All metrics are reported as percentages.

[0093] (3) Implementation details The point cloud encoder uses DGCNN, with input consisting of 2048 points randomly downsampled from each sample. The embedding dimension is set to 512, and all parameters are used in training. Both the image and text encoders use pre-trained CLIP encoders, with their parameters frozen during training. The input image resolution is 224×224, and each sample provides four rendering perspectives. One of these perspectives is randomly sampled as input during the training phase.

[0094] The AdamW optimizer was used during training, with a learning rate of 1×10⁻⁶. -4 The weight decay was set to 0.05, the batch size to 32, and cosine annealing was used for training for 100 epochs. The maximum number of slots was set to 6, the minimum number of active slots to 2, the number of semantic prototypes to 12, the number of Dustbin prototypes to 2, the unified feature dimension to 256, and the number of slot attention iterations to 3. The contrastive learning temperature was set to 0.07, the prototype assignment temperature to 0.10, and the part alignment loss weight to 0.5. The global contrastive loss and the part alignment loss were jointly optimized from the start of training.

[0095] All experiments were performed on a single NVIDIA A100-SXM4-40GB GPU.

[0096] II. Comparison with existing methods (1) Results on the Text2Shape dataset This embodiment compares the method of the present invention with existing representative methods, including Text2Shape and Y... 2 Seq2Seq, TriCoLo, Parts2Words, COM3D, Uni3D-g, Tri-modal Recon., SCA3D, and 3DAlign-DAER are detailed in Table 2.

[0097] Table 2 Comparison of retrieval performance on the Text2Shape dataset Experimental results show that the proposed method (FGR3D) significantly outperforms existing methods in all metrics of Text2Shape. In the shape-to-text direction, RR@1 reaches 51.49%, an improvement of 23.38 percentage points compared to the previous best 3DAlign-DAER. In the text-to-shape direction, RR@1 reaches 39.85%, an improvement of 22.32 percentage points.

[0098] The performance improvement primarily stems from the effective synergy between the proposed component-level modeling mechanism and the frozen CLIP backbone network. Differences in similar furniture in Text2Shape mainly lie in key components such as backrests, armrests, and table legs. Alignment methods relying on global features often struggle to model these local differences. In this invention, the modality-specific slot attention module achieves stable component-level decomposition without explicit component annotation, enabling the model to focus more naturally on key components. The prototype-guided component matching module achieves cross-modal component alignment by sharing a semantic prototype space and effectively suppresses noise interference caused by modal information asymmetry using Dustbin prototypes. The synergistic effect of these two mechanisms allows the general semantic prior provided by the CLIP backbone network to be more effectively mapped to fine-grained component-level representations, thereby providing stronger discriminative power among candidate samples with high intra-class similarity.

[0099] (2) Results on the FG3D dataset Table 3 presents the experimental results on FG3D. Since FG3D is a newly constructed dataset in this application, all baseline methods included in the comparison were retrained under uniform input conditions. Furthermore, this embodiment introduces ULIP-2 and OpenShape to evaluate the transferability of globally pre-trained features in fine-grained retrieval tasks. It should be noted that 3DAlign-DAER and Tri-modal Recon. have not publicly released source code or pre-trained models, making rigorous reproduction on FG3D impossible; therefore, they were not included in this comparison.

[0100] Table 3 compares retrieval performance on the FG3D dataset. Best results are highlighted in bold.

[0101] On the FG3D dataset, the advantages of the method of this invention are even more significant. The RR@1 of the shape-to-text direction reaches 68.78%, and the RR@1 of the text-to-shape direction reaches 61.69%, which are improvements of 34.19 and 27.51 percentage points, respectively, compared with the strongest baseline SCA3D.

[0102] Compared to Text2Shape, the method of this invention has a greater improvement in FG3D. This is mainly due to the fact that the text description of FG3D contains richer component-level details, which enables the retrieval task to shift from global semantic matching to fine-grained structural matching, thereby giving fuller play to the advantages of component-level representation learning.

[0103] Furthermore, while ULIP-2 and OpenShape possess strong general-purpose 3D representation capabilities, they still lag significantly behind the method of this invention in FG3D. This indicates that global semantic alignment methods are insufficient for retrieval tasks centered on fine-grained matching at the component level, and further verifies the necessity of explicit component modeling mechanisms.

[0104] III. Ablation Experiment In this embodiment, a system ablation experiment was conducted on the FG3D dataset to verify the effectiveness of each key design.

[0105] (1) Ablation of core modules Table 4 shows the ablation results of the core module.

[0106] Table 4 Ablation Experiment Results of Core Module The experiment removed the modality-specific slot attention module, the prototype-guided part matching module, and the image-guided mechanism, respectively. The results show that removing the modality-specific slot attention module reduced the shape-to-text direction RR@1 to 37.55%; removing the prototype-guided part matching module reduced RR@1 to 32.39%; and removing the image-guided mechanism further reduced RR@1 to 26.07%. This indicates that the modality-specific slot attention module, the prototype-guided part matching module, and the image-guided mechanism are significantly complementary in part-level matching, and their synergy is essential to fully leverage the advantages of fine-grained modeling.

[0107] (2) Modal-Specific Groove Attention (MSSA) module ablation Table 5 analyzes the effects of different modality inductive biases in the modality-specific slot attention module.

[0108] Table 5. Results of MSSA module ablation experiments Experiments compared the uniform slot attention with the modality-specific slot attention proposed in this invention. Using uniform slot attention resulted in a 11.70 percentage point decrease in the shape-to-text direction RR@1, indicating that differentiated structural induction biases for different modalities are necessary. Further comparison of the components revealed that the text semantic importance weighting had the most significant impact, reducing RR@1 by 2.89 percentage points after removal; point cloud relative geometric encoding was the second most significant, decreasing by 2.76 percentage points; and image spatial location embedding had a relatively smaller impact, decreasing by 1.17 percentage points.

[0109] (3) Prototype-guided part matching (PGPM) module ablation Table 6 illustrates the impact of each design element in the component matching module.

[0110] Table 6. Ablation Experiment Results of PGPM Module Experiments analyzed the impact of various design elements in the prototype matching module. After hard assignment, the model performance almost degraded to a stochastic level, indicating that continuous soft assignment is crucial for maintaining training stability. Removing the Dustbin prototype reduced the shape-to-text RR@1 by 36.66 percentage points, demonstrating the decisive role of noise filtering in cross-modal part matching. Removing the gated residual mechanism reduced RR@1 to 46.77%, indicating that adaptive fusion of prototype semantics and original slot features is equally important for maintaining the discriminative power of part representations.

[0111] (4) Loss function ablation Table 7 shows the performance of global contrast loss and component alignment loss under different combinations.

[0112] Table 7 Ablation experiments using loss functions. ✓ Indicates the use of the corresponding loss term.

[0113] The experiments compared the results using only global contrastive loss, only part alignment loss, and a combination of both losses. When using only global contrastive loss, the shape-to-text RR@1 was 33.29%, indicating that global semantic constraints are a necessary foundation but insufficient to support fine-grained retrieval. When using only part alignment loss, the RR@1 reached 59.77%, directly demonstrating the core role of part-level alignment. After jointly optimizing both types of losses, the model achieved the best performance, indicating a clear complementarity between global semantic constraints and part-level alignment.

[0114] IV. Computational Efficiency Analysis Table 8 compares the computational cost of FGR3D with representative methods.

[0115] Table 8. Computational overhead comparison. All measurements were performed on a single NVIDIA A100-SXM4-40GB GPU in a batch size of 32.

[0116] This embodiment analyzes the computational overhead of the method of the present invention. The total number of model parameters is 156.54M, of which only 5.24M are trainable. The training time is 2.21 minutes / cycle, the inference latency is 4.61 milliseconds / sample, and the peak training memory usage is 13.09GB. Compared with existing CLIP-based methods, the method of the present invention achieves significantly better retrieval performance with only a small increase in trainable parameters, indicating that the method maintains a good balance between performance and computational cost.

[0117] V. Qualitative Analysis (1) Text-to-shape retrieval results like Figure 6 As shown, for each text query, the method of this invention outputs the top 5 candidate shapes sorted by cosine similarity, with true matches marked with green ground truth (GT) labels. In all five query examples, the true match ranked first, indicating that the model can accurately model fine-grained semantics at the component level. While the remaining high-ranking results are not true matches, they maintain a high degree of consistency with the query in terms of overall structure and main component attributes, differing only in local details. This reflects that the method of this invention has learned a component-level semantic space with strong discriminative power.

[0118] (2) Visualization of slot decomposition and component alignment like Figure 7 As shown, the slot decomposition learned by the method of this invention has high consistency with the actual component partitioning and can adaptively adjust the number of active slots according to structural complexity. For example, a simple table may only activate 2 slots, while a complex chair may activate 3 slots. The slight off-diagonal response appearing in individual samples reflects the geometric continuity between adjacent components, and also shows that the soft assignment mechanism can more naturally model the gradual transition between components. Overall, the method of this invention can learn a structured representation that is relatively consistent with human semantic components without the need for explicit component annotation.

[0119] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0120] In addition, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0121] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for constructing a 3D-text retrieval dataset based on component-level semantic alignment, characterized in that, The method includes: Obtain the point cloud coordinate information and component segmentation annotations of a 3D object. The component segmentation annotations are used to identify the component region to which each point in the point cloud belongs. Semantic mapping is performed on the component segmentation annotations. Based on the point cloud geometric features, each segmented region is determined as a semantic component in a predefined component category system, and component-level semantic labels are generated. Assign color identifiers to each semantic component, generate a colored point cloud, and render the colored point cloud to obtain a multi-view rendered image; Extract the component-level structural attributes and appearance attributes from the multi-view rendering image, and generate a structured text description by combining the component-level semantic tags and color identifiers; The tinted point cloud, multi-view rendered image, and structured text description are stored in a cross-modal association to obtain the retrieval dataset corresponding to the 3D object.

2. The method for constructing a 3D-text retrieval dataset based on component-level semantic alignment according to claim 1, characterized in that, The three-dimensional object is a furniture-type object; the process of determining each segmented region as a semantic component in a predefined component category system based on point cloud geometric features includes: Calculate the geometric features of each segmented region, including the position and height of the segmented region relative to the whole object, the horizontal extension range, the length-to-width ratio, and the spatial offset. The geometric features are matched with the standard geometric templates of each semantic component, and the component category to which the segmented region belongs is determined based on the matching degree. The predefined component category system includes at least the following: chair back, chair seat, armrests, chair legs, chair base, tabletop, table legs, and table base.

3. A three-dimensional text-based cross-modal retrieval method, characterized in that, Includes the following steps: The retrieval model is trained using the retrieval dataset as described in claim 1, and the global contrast loss and component alignment loss are jointly optimized during the training process; The three-dimensional point cloud, rendered image, and text query to be retrieved are used as the modality to be processed. Local features and global features of the modality to be processed are extracted respectively. Local features include point-by-point features of the point cloud modality, block features of the image modality, and word features of the text modality. The global and local features of the modality to be processed are input into the trained retrieval model, which includes a modality-specific slot attention module and a component matching module. The local features of the modality to be processed are input into the modality-specific slot attention module, which introduces geometric encoding for the point cloud, position embedding for the image, and semantic weighting for the text. After iterative updates, the local features of each modality are decomposed into slot representations of each component in the corresponding predefined component category system. The slot representations of each modality are input into the prototype-guided component matching module, and the slot representations of each modality are mapped to a unified semantic space to obtain the component-level embeddings of each modality. The similarity is calculated based on the component-level embedding, and the retrieval results are output.

4. The three-dimensional text cross-modal retrieval method according to claim 3, characterized in that, The local feature decomposition of each modality is divided into slot representations of each component in the corresponding predefined component category system, including: Determine the encoding feature matrix of the local features of the modality to be processed; Based on the encoded feature matrix and the preset slot initialization matrix, the preset slot initialization matrix is ​​used as the initial value of the slot state, and the slot state is updated multiple times through an iterative slot attention mechanism. A single iteration of the iterative slot attention mechanism includes: using the encoded feature matrix as input features, calculating attention weights through Softmax normalization, and weighted aggregation of the input features into an update vector. The current slot state is updated via a gated loop unit and further refined through a feedforward network with residual connections. After the second iteration, a stable component-level slot representation is obtained. Including the following formulas: ； in, For the first Each slot indicates in In the state of this iteration, FFN represents the feedforward network, LN represents layer normalization, and GRU represents the gated recurrent unit. This represents the intermediate state output by the GRU.

5. The three-dimensional text cross-modal retrieval method according to claim 4, characterized in that, The modality-specific slot attention module obtains the encoding feature matrix of each modality's local features in the following way: For point cloud modalities, based on the three-dimensional coordinates of each point in the point cloud and their... The relative geometric features are calculated from the relative displacement vectors of the nearest neighbor points. The relative geometric features include the relative displacement vector, Euclidean distance, squared distance, unit direction vector, azimuth angle, and elevation angle. The relative geometric features are then incorporated into the key calculation of the slot attention through a dual-path approach. The first path fuses the geometric features with the point-by-point features before key mapping, while the second path projects the geometric features independently and then directly injects the key representation to obtain the encoded feature matrix of the point cloud mode. For image modalities, a learnable two-dimensional position embedding matrix is ​​generated based on the spatial position coordinates of image patch features. The two-dimensional position embedding matrix is ​​then explicitly superimposed onto the image patch features to obtain the encoding feature matrix of the image modality. For the text modality, based on the original feature matrix of text lexical units, the importance score of each lexical unit is predicted by a multilayer perceptron. The original lexical unit features are then adaptively enhanced based on the importance scores to obtain the encoding feature matrix of the text modality.

6. The three-dimensional text cross-modal retrieval method according to claim 3, characterized in that, The modality-specific slot attention module also utilizes image modality to generate slot activation priors, and the slot activation probabilities of each modality are obtained in the following way: Using the global features of the image modality as input, the data is mapped by a multilayer perceptron and activated by a sigmoid to generate prior activation probabilities for each slot. These prior activation probabilities are then propagated to the point cloud modality and the text modality to constrain the number of effective slots that remain consistent across different modalities for the same 3D object. Determine the self-activation probability of each modality slot feature, and map the slot features to scalars based on the modality's own activation predictor and activate them through the Sigmoid function; The prior activation probability is weighted and fused with the self-activation probability to obtain the final slot activation probability for each mode. As shown in the following formula: ； in, Representing modes The activation predictor is used to convert slot features The mapping is converted to a scalar, and the slot activation estimate of this mode is obtained through the Sigmoid function. The first image modality derived from the image modality Prior probability of each slot This is the fusion coefficient.

7. The three-dimensional text cross-modal retrieval method according to claim 3, characterized in that, The component matching module maps each modal slot representation to a unified semantic space in the following way: The semantic description of each component category in the predefined component category system is determined, and the semantic description is encoded into a text feature vector by the CLIP text encoder. The initial anchor point of the shared semantic prototype is obtained by linear projection and L2 normalization. Learnable offsets are superimposed on the initial anchor points to obtain an optimizable shared semantic prototype, and the Dustbin prototype is randomly initialized. The cosine similarity between each modal slot feature and the shared semantic prototype and Dustbin prototype is calculated, and the soft-assigned weights are obtained by temperature scaling and Softmax normalization. Based on the soft-assigned weights, the shared semantic prototypes are weighted and summed to obtain the prototype features corresponding to each slot. The original slot features and the prototype features are then adaptively fused through a gating mechanism to obtain the component-level embeddings of each modality.

8. The three-dimensional text cross-modal retrieval method according to claim 7, characterized in that, The Dustbin prototype is used to absorb noisy slots that do not match any of the shared semantic prototypes. When calculating the soft allocation weights, logarithmic bias terms in different directions are added to the shared semantic prototypes and the Dustbin prototypes respectively, so that slots with lower activation probabilities are more likely to be allocated to the Dustbin prototype.

9. The three-dimensional text cross-modal retrieval method according to claim 3, characterized in that, The global contrast loss is calculated in the following way: Global features of three modalities—text, image, and point cloud—are determined, and L2 normalization is performed on the global features. Based on the symmetric InfoNCE loss function, the bidirectional contrast loss of text-image pairs, text-point cloud pairs, and image-point cloud pairs is calculated respectively; The global contrastive loss is obtained by averaging the contrastive losses of the three modality pairs.

10. The three-dimensional text cross-modal retrieval method according to claim 3, characterized in that, The component alignment loss is calculated in the following way: Determine the component-level embedding of each mode and the slot activation probability and soft activation mask of each mode; The component-level embeddings of each modality are weighted and aggregated based on the slot activation probability and soft activation mask to obtain the component-aware sample-level representation. InfoNCE loss is then applied to the sample-level representation to obtain the inter-sample contrast loss. By comparing the weights assigned to shared semantic prototypes by point cloud component-level embeddings and text component-level embeddings, the cross-modal component correspondence strength is calculated, a soft correspondence matrix is ​​generated, and the similarity between cross-modal corresponding component embeddings within the same sample is maximized based on the soft correspondence matrix to obtain the intra-sample alignment loss. The component alignment loss is obtained by weighted summing of the inter-sample contrast loss and the intra-sample alignment loss.

Images