3D task processing method and device based on image text cross-modal migration

Abstract

The invention relates to the technical field of computer vision, and discloses a 3D task processing method and device based on image text cross-modal migration, and the method comprises the steps: obtaining to-be-processed 3D point cloud data, and carrying out the structural processing, and obtaining an ordered point cloud block set; obtaining two-dimensional image data and / or text description data associated with the 3D point cloud data, extracting an image feature vector and / or a text feature vector, performing unified serialization processing on the ordered point cloud block set, and generating a 3D point cloud feature sequence fused with multi-modal information; and inputting the 3D point cloud feature sequence, the image feature vector and the text feature vector into a gating fusion module, dynamically adjusting the weight of each modal feature, performing fusion, generating an enhanced 3D feature representation, and executing a 3D downstream task based on the enhanced 3D feature representation. According to the method, the robustness, the accuracy and the generalization ability of the 3D downstream task in a data scarce scene can be improved, and efficient and robust 3D scene understanding is realized.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to a 3D task processing method and apparatus based on cross-modal transfer of image text. Background Technology

[0002] Point cloud data plays a crucial role in the widespread application of 3D perception technologies, such as autonomous driving, robot interaction, and metaverse construction. However, the 3D domain faces a data scarcity dilemma, with its data scale and modal maturity significantly lagging behind 2D vision and natural language processing. 2D vision relies on image libraries with hundreds of millions of images, such as ImageNet, and natural language processing is based on corpora with hundreds of billions of tokens, both having established complete pre-training paradigms. In contrast, the high cost of sensor acquisition and complex data annotation in the 3D domain result in a limited number of samples in mainstream point cloud datasets. This data scarcity severely restricts the improvement of the geometric representation capabilities of 3D models, making it difficult to meet the needs of deep model training.

[0003] Currently, the commonly used single-modal training methods for handling 3D tasks suffer from significant deficiencies in generalization ability and robustness due to their reliance on limited 3D point cloud data for model training. In practical applications, such as the identification of occluded pedestrians in autonomous driving, the accuracy of the model is often difficult to guarantee, greatly impacting the overall performance and safety of the system. Furthermore, single-modal training methods cannot fully utilize the rich information contained in other modalities, such as the structured semantic information of image data and the sequential semantic information of text data. This not only wastes data resources but also further limits the ability of 3D models to understand and represent complex scenes. In summary, existing technologies for handling 3D tasks mainly face the problems of scarce 3D data and insufficient model generalization ability, robustness, and understanding of complex scenes due to single-modal training methods.

[0004] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the Invention

[0005] This application provides a 3D task processing method and apparatus based on cross-modal transfer of images and text. By fusing prior information from image and text modalities, the method processes and enhances the features of 3D point cloud data, thereby improving the robustness, accuracy, and generalization ability of downstream 3D tasks in data-scarce scenarios and achieving efficient and robust 3D scene understanding.

[0006] In a first aspect, embodiments of this application provide a 3D task processing method based on cross-modal transfer of image text, including:

[0007] Acquire 3D point cloud data to be processed, perform structured processing on the 3D point cloud data to be processed, and obtain an ordered set of point cloud blocks;

[0008] Acquire at least one two-dimensional image data and / or at least one text description data associated with the 3D point cloud data;

[0009] The two-dimensional image data is processed using a pre-trained image feature extraction model to obtain an image feature vector; and / or, the text description data is processed using a pre-trained text feature extraction model to obtain a text feature vector;

[0010] Based on the image feature vector and / or the text feature vector, the ordered set of point cloud blocks is subjected to unified serialization processing to generate a 3D point cloud feature sequence that integrates multimodal information.

[0011] The 3D point cloud feature sequence with fused multimodal information, the image feature vector, and the text feature vector are input into the gated fusion module. The gated fusion module dynamically adjusts the weights of each modal feature and fuses them to generate an enhanced 3D feature representation.

[0012] Based on the enhanced 3D feature representation, the specified 3D downstream task is executed.

[0013] Furthermore, in some embodiments of this application, the step of performing structured processing on the 3D point cloud data to obtain an ordered set of point cloud blocks includes:

[0014] The 3D point cloud data to be processed is divided into multiple point cloud blocks;

[0015] A sorting operation is performed on the plurality of point cloud blocks to form an ordered set of point cloud blocks; wherein the sorting operation is based on the spatial location information and / or geometric feature information of the point cloud blocks.

[0016] Furthermore, in some embodiments of this application, the step of processing the two-dimensional image data using a pre-trained image feature extraction model to obtain an image feature vector; and / or, processing the text description data using a pre-trained text feature extraction model to obtain a text feature vector, includes:

[0017] The image feature vector of the two-dimensional image data is extracted using a visual model based on the Transformer architecture;

[0018] The text feature vectors of the text description data are extracted using a language model based on the Transformer architecture.

[0019] Furthermore, in some embodiments of this application, before performing unified serialization processing on the ordered set of point cloud blocks based on the image feature vector and / or the text feature vector to generate a 3D point cloud feature sequence fused with multimodal information, the method further includes:

[0020] The image feature vector and / or the text feature vector are input into the modality adapter for processing;

[0021] The modal adapter adapts image features and / or text features to the 3D point cloud feature space.

[0022] Furthermore, in some embodiments of this application, the step of performing unified serialization processing on the ordered set of point cloud blocks based on the image feature vector and / or the text feature vector to generate a 3D point cloud feature sequence that fuses multimodal information includes:

[0023] Obtain a fused feature vector for guiding the sorting, the fused feature vector being generated based on the image feature vector and / or the text feature vector;

[0024] Using the fused feature vector, the point cloud blocks in the ordered point cloud block set are reordered or their sequence positions are adjusted to generate the 3D point cloud feature sequence that fuses multimodal information.

[0025] Furthermore, in some embodiments of this application, when both image feature vectors and text feature vectors exist simultaneously, obtaining the fused feature vector used to guide the ranking includes:

[0026] The image feature vector and the text feature vector are concatenated or weighted and fused.

[0027] The concatenated or weighted fused vectors are encoded using a neural network to generate a fused feature vector for guiding the ranking.

[0028] Furthermore, in some embodiments of this application, the step of dynamically adjusting the weights of each modal feature through the gating fusion module includes:

[0029] The fusion weights of image features, text features, and 3D point cloud features are dynamically adjusted based on the attributes of the object represented by the 3D point cloud data, the requirements of the 3D downstream task, and at least one of the image feature vector and the text feature vector.

[0030] Furthermore, in some embodiments of this application, performing the specified 3D downstream task based on the enhanced 3D feature representation includes:

[0031] Select the corresponding task adaptation network based on the type of the 3D downstream task;

[0032] The enhanced 3D feature representation is input into the task adaptation network to generate corresponding task result data.

[0033] Furthermore, in some embodiments of this application, selecting the corresponding task adaptation network based on the type of the 3D downstream task includes:

[0034] When the 3D downstream task is an object classification task, the task adaptation network includes a fully connected layer and a classifier to output the category probability distribution.

[0035] When the 3D downstream task is a semantic segmentation task, the task adaptation network includes a pointwise feature decoder for outputting the semantic label of each point.

[0036] When the 3D downstream task is an object detection task, the task adaptation network includes a bounding box regression module, which is used to output the object position and category;

[0037] When the 3D downstream task is a 3D visual question answering task, the task adaptation network includes a multimodal reasoning module for receiving question text and outputting answers.

[0038] Secondly, embodiments of this application provide a 3D task processing apparatus based on cross-modal transfer of image text, comprising:

[0039] The data acquisition module is used to acquire 3D point cloud data to be processed, and to perform structured processing on the 3D point cloud data to be processed to obtain an ordered set of point cloud blocks.

[0040] The data association module is used to acquire at least one two-dimensional image data and / or at least one text description data associated with the 3D point cloud data;

[0041] The feature extraction module is used to process the two-dimensional image data using a pre-trained image feature extraction model to obtain an image feature vector; and / or to process the text description data using a pre-trained text feature extraction model to obtain a text feature vector.

[0042] The sequence generation module is used to perform unified serialization processing on the ordered set of point cloud blocks based on the image feature vector and / or the text feature vector to generate a 3D point cloud feature sequence that integrates multimodal information.

[0043] The feature fusion module is used to input the 3D point cloud feature sequence of the fused multimodal information, the image feature vector and the text feature vector into the gated fusion module, and dynamically adjust the weights of each modal feature and fuse them to generate an enhanced 3D feature representation.

[0044] The task execution module is used to execute specified 3D downstream tasks based on the enhanced 3D feature representation.

[0045] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the 3D task processing method based on image-text cross-modal transfer as described in the first aspect.

[0046] Fourthly, embodiments of this application provide a storage medium storing a computer program capable of being loaded by a processor and executing the 3D task processing method based on image-text cross-modal transfer as described in the first aspect.

[0047] This application provides a 3D task processing method and apparatus based on image-text cross-modal transfer. First, the 3D point cloud data to be processed is acquired and structured, and divided into an ordered set of point cloud blocks. At the same time, combined with the associated two-dimensional image data and / or text description data, image feature vectors and text feature vectors are extracted using a pre-trained image feature extraction model and a text feature extraction model, respectively. These feature vectors provide multimodal prior information for 3D point cloud data. Then, based on these feature vectors, the ordered set of point cloud blocks is uniformly serialized to generate a 3D point cloud feature sequence that integrates multimodal information, achieving effective fusion between different modal data. Next, the fused feature sequence is input into a gated fusion module, which dynamically adjusts the weights of each modality feature according to task requirements and data characteristics and fuses them to generate an enhanced 3D feature representation. This module can flexibly adjust the contribution of each modality data according to specific tasks, improving the model's understanding and representation capabilities of 3D data. Finally, the specified 3D downstream tasks are executed based on the enhanced 3D feature representation, thereby improving the robustness, accuracy, and generalization ability of 3D downstream tasks in data-scarce scenarios, achieving efficient and robust 3D scene understanding. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is an application environment diagram of the 3D task processing method based on cross-modal transfer of image text provided in the embodiments of this application;

[0050] Figure 2This is a flowchart illustrating the 3D task processing method based on cross-modal transfer of image text provided in an embodiment of this application.

[0051] Figure 3 This is a flowchart illustrating the overall technical approach provided in the embodiments of this application;

[0052] Figure 4 This is a schematic diagram of the structure of the 3D task processing device based on cross-modal transfer of image and text provided in the embodiments of this application;

[0053] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0054] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of systems and methods consistent with those detailed in the appended claims or with some aspects of this application.

[0055] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover descriptions such as non-exclusive inclusion, so that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0056] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0057] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0058] To address the aforementioned technical problems and overcome the shortcomings of existing technologies, this application provides a 3D task processing method and apparatus based on cross-modal transfer of images and text, which improves the robustness, accuracy, and generalization ability of downstream 3D tasks in data-scarce scenarios, and achieves efficient and robust 3D scene understanding.

[0059] Figure 1 This is an application environment diagram of a 3D task processing method based on cross-modal transfer of image text in one embodiment. (Refer to...) Figure 1 This 3D task processing method based on image-text cross-modal transfer is applied to a 3D task processing system based on image-text cross-modal transfer. The system includes a terminal 110 and a server 120. The terminal 110 and server 120 are connected via a network. The terminal 110 can be a desktop terminal or a mobile terminal; the mobile terminal can be at least one of a mobile phone, tablet computer, or laptop computer. The server 120 can be a standalone server or a server cluster consisting of multiple servers. Server 120 is configured to execute the aforementioned 3D task processing method based on image-text cross-modal transfer, including: acquiring 3D point cloud data to be processed; performing structured processing on the 3D point cloud data to obtain an ordered set of point cloud blocks; acquiring at least one two-dimensional image data and / or at least one text description data associated with the 3D point cloud data; processing the two-dimensional image data using a pre-trained image feature extraction model to obtain image feature vectors; and / or processing the text description data using a pre-trained text feature extraction model to obtain text feature vectors; performing unified serialization processing on the ordered set of point cloud blocks based on the image feature vectors and / or text feature vectors to generate a 3D point cloud feature sequence fused with multimodal information; inputting the 3D point cloud feature sequence fused with multimodal information, the image feature vectors, and the text feature vectors into a gating fusion module, dynamically adjusting the weights of each modality feature through the gating fusion module and fusing them to generate an enhanced 3D feature representation; and executing a specified 3D downstream task based on the enhanced 3D feature representation.

[0060] Please see Figure 2 , Figure 2 This is a flowchart illustrating a 3D task processing method based on cross-modal transfer of image and text provided in an embodiment of this application. This embodiment mainly uses the application of this 3D task processing method based on cross-modal transfer of image and text to a computer device as an example for illustration. The 3D task processing method based on cross-modal transfer of image and text provided in an embodiment of this application may specifically include the following steps:

[0061] S1. Obtain the 3D point cloud data to be processed, perform structured processing on the 3D point cloud data to be processed, and obtain an ordered set of point cloud blocks;

[0062] Specifically, for step S1, in practical applications, 3D point cloud data is usually chaotic, with no clear order between points. This poses significant challenges to subsequent processing and analysis. Therefore, the acquired 3D point cloud data needs to be structured first. The goal of structured processing is to organize the disordered point cloud data into an ordered form to facilitate better feature extraction and analysis. Specifically, techniques such as micro-point cloud grouping networks can be used to divide the original point cloud data into multiple smaller point cloud blocks, which exhibit a certain degree of spatial locality. Then, a point serialization module is used to sort these point cloud blocks, forming an ordered set of point cloud blocks. The sorting operation can be based on the spatial location information and / or geometric feature information of the point cloud blocks, ensuring that the sorted point cloud blocks exhibit a certain regularity in spatial layout or geometric characteristics. For example, in autonomous driving scenarios, the point cloud blocks can be sorted according to their projection positions on the road plane, allowing the sorted point cloud blocks to better reflect the spatial structure of the vehicle's surrounding environment.

[0063] S2. Acquire at least one two-dimensional image data and / or at least one text description data associated with the 3D point cloud data;

[0064] Specifically, in step S2, 3D point cloud data is often correlated with 2D image data and / or text description data. 2D image data provides rich appearance information, such as color and texture, which is crucial for understanding and identifying the shape and category of objects. Text description data provides semantic information about objects or scenes, such as the object's name, attributes, and functions. Acquiring this correlated data provides more information sources for subsequent feature fusion and analysis. For example, in an industrial robot's 3D target grasping task, a 2D image of the object can be acquired through a camera, combined with text description information from sensors, such as the object's material and weight, to gain a more comprehensive understanding of the object's characteristics and thus better plan the grasping action.

[0065] S3. Process two-dimensional image data using a pre-trained image feature extraction model to obtain image feature vectors; and / or process text description data using a pre-trained text feature extraction model to obtain text feature vectors;

[0066] Specifically, for step S3, the pre-trained image feature extraction model and text feature extraction model are trained on a large amount of image and text data, enabling them to extract feature vectors with representative and semantic information. For two-dimensional image data, visual models based on the Transformer architecture, such as ViT and Swin-Transformer, can be used to extract image feature vectors. These models can segment the image into multiple small patches, then extract and encode features from each patch, ultimately obtaining a vector that represents the overall features of the image. For text description data, language models based on the Transformer architecture, such as BERT and GPT, can be used to extract text feature vectors. These models can encode each word in the text, capturing the semantic relationships and contextual information between words, generating a vector that reflects the semantics of the text. For example, in medical imaging organ reconstruction tasks, pre-trained image feature extraction models can be used to process two-dimensional slice images of organs to obtain image feature vectors, which, combined with the text feature vectors from the doctor's written report, provide more comprehensive information for subsequent organ reconstruction.

[0067] S4. Based on image feature vectors and / or text feature vectors, perform unified serialization processing on the ordered set of point cloud blocks to generate a 3D point cloud feature sequence that integrates multimodal information;

[0068] Specifically, in step S4, the purpose of unified serialization is to effectively fuse the feature information of 2D images and / or text with 3D point cloud data to generate a 3D point cloud feature sequence that comprehensively reflects multimodal information. Image feature vectors and / or text feature vectors can be used as guiding information to reorder or adjust the sequence position of the ordered set of point cloud blocks. For example, based on information such as object shape and texture contained in the image feature vectors, the set of point cloud blocks can be sorted so that the sorted point cloud blocks more closely match the actual shape of the object in spatial structure. The resulting 3D point cloud feature sequence, which fuses multimodal information, not only contains the spatial geometric information of the 3D point cloud data but also integrates the semantic information of the image and / or text.

[0069] S5. Input the 3D point cloud feature sequence, image feature vector and text feature vector that integrate multimodal information into the gated fusion module. The gated fusion module dynamically adjusts the weights of each modal feature and fuses them to generate an enhanced 3D feature representation.

[0070] Specifically, in step S5, the core function of the gated fusion module is to dynamically adjust and fuse the weights of image features, text features, and 3D point cloud features based on different task requirements and data characteristics, generating an enhanced 3D feature representation. The gated fusion module can employ techniques such as gating units to achieve dynamic weight adjustment. The gating unit can automatically learn the importance of each modal feature based on the input feature vector and task information, and generate corresponding weight coefficients. For example, when performing a 3D object classification task, if the object appearance information contained in the image feature vector is crucial for distinguishing different categories, the gated fusion module will automatically assign higher weights to the image features. In this way, the advantages of different modal features can be fully utilized to generate a more comprehensive and accurate enhanced 3D feature representation.

[0071] S6. Based on the enhanced 3D feature representation, execute the specified 3D downstream task;

[0072] Specifically, for step S6, depending on the specific application requirements, the enhanced 3D feature representation can be used to perform various downstream 3D tasks, such as object classification, semantic segmentation, object detection, and 3D visual question answering. In object classification, the enhanced 3D feature representation can be input into a classifier, and classification can be performed through network structures such as fully connected layers, outputting the probability distribution of each category. In semantic segmentation, network structures such as pointwise feature decoders can be used to predict the semantic label of each point, thereby segmenting and labeling different objects in the 3D scene. In object detection, techniques such as bounding box regression modules can be used to predict the position and category of objects based on the enhanced 3D feature representation. In 3D visual question answering, a multimodal reasoning module can be designed to interact with the user-input text question and the enhanced 3D feature representation, performing feature fusion and reasoning through techniques such as attention mechanisms, ultimately outputting the answer.

[0073] In summary, this embodiment performs structured processing on 3D point cloud data, combines the features of relevant 2D images and / or text description data, integrates multimodal information into the 3D point cloud feature sequence through unified serialization, and then dynamically adjusts and fuses the feature weights of each modality through a gating fusion module to generate an enhanced 3D feature representation. This effectively utilizes the rich prior knowledge of 2D image and text modalities, overcomes the limitation of scarce 3D data, improves the processing effect of specified 3D downstream tasks, and enhances the robustness and accuracy of the model.

[0074] Furthermore, in some embodiments, step S1, "performing structured processing on the 3D point cloud data to be processed to obtain an ordered set of point cloud blocks," may specifically include:

[0075] S11. Divide the 3D point cloud data to be processed into multiple point cloud blocks;

[0076] Specifically, 3D point cloud data is a collection of numerous point coordinates, randomly distributed in space. To facilitate subsequent processing, this unordered point cloud data needs to be divided into multiple smaller point cloud blocks. The division can be based on spatial location, such as using a fixed voxel grid to divide the space into small cubic cells with equal side lengths, with the point clouds within each cell forming a point cloud block. Alternatively, it can be based on the geometric features of the object, such as using clustering algorithms to group points with similar geometric features together to form point cloud blocks. For example, in industrial parts inspection scenarios, point cloud data can be divided into point cloud blocks corresponding to different components based on the part's contour and structural characteristics, allowing for separate analysis and processing.

[0077] S12. Perform a sorting operation on multiple point cloud blocks to form an ordered set of point cloud blocks; wherein the sorting operation is based on the spatial location information and / or geometric feature information of the point cloud blocks.

[0078] Specifically, the segmented point cloud blocks still lack a clear sequential relationship, thus requiring a sorting operation to form an ordered set. The sorting can be based on the spatial location information of the point cloud blocks, such as sorting according to the projection of the center coordinates of the point cloud blocks onto a specific direction (e.g., the X, Y, or Z axis), or topological sorting according to the overall spatial position of the point cloud blocks. Alternatively, it can be based on the geometric features of the point cloud blocks, such as their size, shape, or normal direction. For example, in autonomous driving scenarios, the point cloud blocks can be sorted according to their relative positions in the vehicle's direction of travel, allowing the sorted blocks to reflect the sequential order of the vehicle's surroundings, facilitating subsequent tasks such as road recognition and obstacle detection.

[0079] This embodiment divides 3D point cloud data into multiple point cloud blocks and sorts them based on spatial location and / or geometric features to obtain an ordered set of point cloud blocks. This solves the problem of the disorder of point clouds and makes the point cloud data exhibit structured features, laying the foundation for subsequent fusion with modal features such as images and text. This helps to improve the effectiveness of multimodal information fusion and thus optimizes the processing performance of downstream 3D tasks.

[0080] Furthermore, in some embodiments, step S3, "processing two-dimensional image data using a pre-trained image feature extraction model to obtain image feature vectors; and / or processing text description data using a pre-trained text feature extraction model to obtain text feature vectors," may specifically include:

[0081] S31. Use a visual model based on the Transformer architecture to extract image feature vectors from two-dimensional image data;

[0082] Specifically, visual models based on the Transformer architecture, such as ViT (VisionTransformer) and Swin-Transformer, can segment an image into multiple small patches, and then extract and encode features for each patch. A self-attention mechanism captures the global dependencies between image patches, thereby generating image feature vectors with rich semantic information. For example, the image is first segmented into fixed-size patches, then these patches are flattened and embedded into a high-dimensional space. Next, a multi-layer Transformer encoder processes the embedded vectors to obtain the feature representation of each patch. Finally, the global feature vector of the image can be obtained through global average pooling or by extracting feature vectors from specific locations.

[0083] S32. Use a language model based on the Transformer architecture to extract text feature vectors from text description data.

[0084] Specifically, language models based on the Transformer architecture can capture semantic relationships and contextual information between words by encoding text, generating text feature vectors with semantic information. Taking BERT as an example, each word in the text is first converted into a word embedding vector, and positional embeddings and segment embeddings are added to represent the word's position and its sentence part. Then, the embedded vectors are processed through a multi-layer Transformer encoder to obtain the feature representation of each word. Finally, the global feature vector of the text can be obtained by taking the feature vector corresponding to the label or by performing global average pooling on the feature vectors of all words.

[0085] This embodiment utilizes a pre-trained visual model and a language model based on the Transformer architecture to extract feature vectors from two-dimensional images and text descriptions, respectively. Leveraging the powerful processing capabilities of the Transformer architecture for sequential data, it fully mines the structured semantic information of images and the sequential semantic information of text, providing high-quality cross-modal feature support for the fusion of 3D point cloud feature sequences, enhancing the representational ability of multimodal fusion features, and better serving downstream 3D tasks.

[0086] Furthermore, in some embodiments, after step S3 and before step S4, the 3D task processing method provided in this embodiment may further include:

[0087] Input the image feature vector and / or text feature vector into the modality adapter for processing;

[0088] Image features and / or text features are adapted to the 3D point cloud feature space using modal adapters.

[0089] Specifically, the role of a modal adapter is to adjust and transform image and / or text feature vectors to enable effective fusion with the 3D point cloud feature space. Since data from different modalities have different feature distributions and semantic information, direct fusion may lead to feature mismatch or information loss. The modal adapter maps image and text feature vectors to a feature space compatible with the 3D point cloud feature space through a series of linear transformations or nonlinear mappings. For example, a multilayer perceptron (MLP) can be used as a modal adapter to transform image and text feature vectors layer by layer, giving them similar dimensions and semantic characteristics to 3D point cloud features. The modal adapter learns the mapping relationship between image, text, and 3D point cloud features, transforming image and text features into feature representations consistent with the 3D point cloud feature space. For example, when processing 3D point cloud data in autonomous driving scenarios, information such as color and texture in image features and semantic description information in text features can be transformed by the modal adapter into feature representations matching the geometric features (such as point coordinates and normal directions) of the 3D point cloud, thereby achieving effective fusion of multimodal features in the same feature space.

[0090] This embodiment adapts image and / or text features to the 3D point cloud feature space through a modal adapter, solving the problem of heterogeneity in the spatial distribution of different modal features. This places image, text features, and 3D point cloud features in a unified feature space, providing an adapted feature foundation for subsequent unified serialization and fusion processing, improving the compatibility and effectiveness of multimodal feature fusion, and thus improving the performance of downstream 3D tasks.

[0091] Furthermore, in some embodiments, step S4, "based on image feature vectors and / or text feature vectors, performing unified serialization processing on the ordered set of point cloud blocks to generate a 3D point cloud feature sequence that integrates multimodal information," may specifically include:

[0092] S41. Obtain a fused feature vector for guiding the sorting, the fused feature vector being generated based on image feature vectors and / or text feature vectors;

[0093] Specifically, the generation of the fused feature vector is achieved by combining image feature vectors and / or text feature vectors. The purpose of this step is to integrate feature information from different modalities into a unified feature vector to guide the sorting of point cloud blocks. For example, image feature vectors and text feature vectors can be concatenated or weighted and then encoded through a neural network to generate a fused feature vector for guiding the sorting.

[0094] S42. Using the fused feature vector, the point cloud blocks in the ordered set of point cloud blocks are reordered or their sequence positions are adjusted to generate a 3D point cloud feature sequence that fuses multimodal information.

[0095] Specifically, by using fused feature vectors to guide the sorting of point cloud blocks, the sorted set of point cloud blocks can better reflect the structural and semantic information of objects or scenes. The fused feature vectors can be used as the basis for sorting to rearrange the point cloud blocks. For example, the similarity between each point cloud block and the fused feature vector can be calculated, and then the blocks can be sorted according to the similarity scores.

[0096] This embodiment generates a fused feature vector based on image and / or text feature vectors, and uses it to guide the sorting or position adjustment of point cloud blocks to generate a 3D point cloud feature sequence that integrates multimodal information. This allows the point cloud feature sequence to simultaneously contain 3D spatial information and cross-modal information from images and text, achieving deep fusion of multimodal information at the sequence level. This enhances the semantic and structural representation capabilities of 3D point cloud features and provides richer feature support for downstream 3D tasks.

[0097] Furthermore, in some embodiments, when both image feature vectors and text feature vectors exist simultaneously, step S41, "obtaining a fused feature vector for guiding sorting," may specifically include:

[0098] S411. Concatenate or weightedly fuse image feature vectors and text feature vectors;

[0099] S412. Encode the concatenated or weighted fused vectors using a neural network to generate a fused feature vector for guiding sorting.

[0100] Specifically, when both image and text feature vectors exist, they need to be integrated into a fused feature vector. Concatenation is the most direct method, simply joining the two feature vectors end-to-end to form a longer vector. Weighted fusion, on the other hand, linearly combines the two feature vectors according to pre-defined weights to generate a new feature vector. For example, assuming the image feature vector has a weight of α and the text feature vector has a weight of β, the fused feature vector can be represented as α × image feature vector + β × text feature vector. The vector resulting from concatenation or weighted fusion contains comprehensive information from both the image and text, but may have high dimensionality and complex relationships between the information. Encoding through a neural network can map the high-dimensional vector to a more suitable space, extracting more representative and discriminative features, thereby generating a fused feature vector to guide ranking.

[0101] In this embodiment, when image and text feature vectors exist simultaneously, a fused feature vector is generated by splicing or weighting and encoding through a neural network to guide the sorting. This fully integrates the structured information of the image and the semantic information of the text to form a more comprehensive guidance signal. This ensures that the sorting of point cloud blocks can make balanced use of the advantages of the two modalities, improves the fusion effect of 3D point cloud feature sequences on multimodal information, and thus optimizes the processing accuracy of downstream 3D tasks.

[0102] Furthermore, in some embodiments, the step S5 of "dynamically adjusting the weights of each modal feature through the gating fusion module" may specifically include:

[0103] The fusion weights of image features, text features, and 3D point cloud features are dynamically adjusted based on the attributes of the object represented by the 3D point cloud data, the requirements of the 3D downstream task, and at least one of image feature vectors and text feature vectors.

[0104] Specifically, adjusting the weights of each modal feature includes:

[0105] The fusion weights of image features, text features, and 3D point cloud features are dynamically adjusted based on the attributes of the objects represented by 3D point cloud data. Different types of 3D objects have different attributes, such as shape, material, and color. These attributes affect the model's feature requirements. For example, for complex mechanical parts, geometric features may be more important; while for decorative items, color and texture features may be more critical. Therefore, the fusion weights can be adjusted according to the object's attributes, allowing the model to better focus on key features when processing objects with different attributes.

[0106] The fusion weights of image features, text features, and 3D point cloud features can be dynamically adjusted based on the requirements of downstream 3D tasks. Different downstream 3D tasks have different feature requirements. For example, in 3D object classification tasks, more attention may be needed to focus on the overall shape and appearance features of the object; while in semantic segmentation tasks, more refined local geometric and texture features are required. Therefore, the weights of each modality feature can be dynamically adjusted according to the specific task requirements to optimize the model's performance on that task.

[0107] The fusion weights of image features, text features, and 3D point cloud features are dynamically adjusted based on at least one of the image feature vectors and text feature vectors. The image and text feature vectors contain rich information that can be used to guide the weight adjustment. For example, if a region in the image feature vector has very significant features, it may mean that the region is important to the current task, and the weight of the image feature can be appropriately increased; similarly, if a semantic concept in the text feature vector is highly relevant to the current task, the weight of the text feature can be increased.

[0108] This embodiment uses a gated fusion module to dynamically adjust the feature weights of each modality based on the attributes of 3D point cloud objects, downstream task requirements, and image and text features. This enables the fusion process to adapt to different scenarios and task characteristics, highlighting key modal information while also taking into account the supplementary role of other modalities. This generates enhanced 3D feature representations that better meet task requirements, effectively improving the adaptability and processing effect of downstream 3D tasks in different scenarios.

[0109] Furthermore, in some embodiments, step S6, "based on the enhanced 3D feature representation, performing the specified 3D downstream task," may specifically include:

[0110] S61. Select the appropriate task adaptation network based on the type of the 3D downstream task;

[0111] S62. Input the enhanced 3D feature representation into the task adaptation network to generate the corresponding task result data.

[0112] Specifically, different downstream 3D tasks have different goals and requirements, thus necessitating the selection of network structures suitable for specific tasks. For example, for object classification tasks, fully connected layers and classifiers can be used to output category probability distributions; for semantic segmentation tasks, pointwise feature decoders can be used to predict the semantic label of each point; for object detection tasks, bounding box regression modules can be used to predict the object's location and category; and for 3D visual question answering tasks, multimodal inference modules can be used to receive question text and output answers. The enhanced 3D feature representations are input into the selected task-adaptive network to generate the final task results. For instance, in object classification tasks, the enhanced 3D feature representations are input into fully connected layers and classifiers, undergoing a series of linear transformations and nonlinear activation functions to output the probability distribution of each category, thus completing the classification task. In semantic segmentation tasks, the enhanced 3D feature representations are input into pointwise feature decoders, which process and decode the features of each point, ultimately outputting the semantic label of each point, achieving the segmentation and annotation of different objects in a 3D scene.

[0113] This embodiment selects the corresponding task adaptation network according to the type of 3D downstream task, and transforms the enhanced 3D feature representation into the output form required by the task in a targeted manner. This enables the fused multimodal features to accurately match the processing requirements of different tasks, giving full play to their enhancement role in various 3D downstream tasks and improving the execution efficiency and result accuracy of different tasks.

[0114] Furthermore, in some embodiments, step S61, "selecting the corresponding task adaptation network according to the type of the 3D downstream task," may specifically include:

[0115] When the downstream 3D task is an object classification task, the task adaptation network includes a fully connected layer and a classifier to output the class probability distribution.

[0116] Specifically, the goal of object classification is to determine the category of objects in 3D point cloud data. The task adaptation network maps the enhanced 3D feature representations to a category space through fully connected layers, and then outputs the probability distribution for each category through a classifier (such as Softmax). For example, assuming there are N categories, the fully connected layer transforms the feature vector into an N-dimensional vector, with each dimension corresponding to a score for a category. The Softmax function then converts these scores into probability values, representing the likelihood that the input data belongs to the corresponding category.

[0117] When the downstream 3D task is a semantic segmentation task, the task adaptation network includes a pointwise feature decoder to output the semantic label for each point.

[0118] Specifically, the goal of semantic segmentation is to assign a semantic label to each point in a 3D point cloud. The task adaptation network processes the enhanced 3D feature representation through a pointwise feature decoder to generate a feature representation for each point, and then outputs a semantic label for each point through a classifier. The pointwise feature decoder typically employs a lightweight network structure to ensure that the features of each point are effectively processed.

[0119] When the downstream 3D task is object detection, the task adaptation network includes a bounding box regression module, which is used to output the object's position and category.

[0120] Specifically, the goal of object detection is to determine the location and category of objects in a 3D point cloud. The task adaptation network processes the enhanced 3D feature representation through a bounding box regression module to generate the bounding box coordinates and category information of the objects. The bounding box regression module typically employs a regression network structure, which predicts the coordinates and category probabilities of the bounding boxes by learning the geometric features and location information of the objects.

[0121] When the downstream 3D task is a 3D visual question answering task, the task adaptation network includes a multimodal reasoning module, which is used to receive the question text and output the answer;

[0122] Specifically, the goal of the 3D visual question answering task is to generate corresponding answers based on 3D point cloud data and user-inputted question text. The task adaptation network uses a multimodal inference module to fuse and infer the enhanced 3D feature representation and question text features to generate the answer. The multimodal inference module typically employs attention mechanisms and a sequence-to-sequence (Seq2Seq) model structure to achieve effective fusion and inference of multimodal information.

[0123] This embodiment configures corresponding task-adaptive networks for different 3D downstream tasks such as object classification, semantic segmentation, object detection, and 3D visual question answering. This enables the enhanced 3D features to be transformed into specific outputs for each task through dedicated networks, such as category probabilities, semantic labels, bounding boxes, and answers. This achieves the accurate application of multimodal fusion features in various specific tasks and comprehensively improves the processing performance of different 3D downstream tasks.

[0124] Furthermore, in some embodiments, the multimodal inference module is configured to perform:

[0125] The user-input question text is encoded to generate a question feature vector;

[0126] The problem feature vector is interacted with the enhanced 3D feature representation through attention;

[0127] Predict the answer to the question text based on the features after interaction.

[0128] Specifically, in 3D visual question answering tasks, user questions are input in text form, which needs to be converted into feature vectors that the model can process. Pre-trained language models (such as BERT, GPT, etc.) can be used to encode the question text. These models can capture semantic information and contextual relationships in the text, generating question feature vectors with rich semantics. For example, the question text is segmented into words or sub-word units, converted into high-dimensional vectors through word embedding layers, and then processed through multi-layer Transformer encoders to obtain feature representations for each word. Finally, the feature vector corresponding to the CLS tag is taken as the overall feature vector of the question. The fusion of the question feature vector with the enhanced 3D feature representation is achieved through attention interaction. The attention mechanism can automatically learn the correlation between two feature spaces and find the 3D feature parts most relevant to the question. Specifically, a bilinear attention network or a graph attention network can be used to calculate the attention weights between the question feature vector and the 3D feature representation, and then the 3D features are weighted and summed according to these weights to generate a 3D feature representation relevant to the question. After obtaining the 3D feature representations related to the question, answer prediction can be performed using a Multilayer Perceptron (MLP) or other sequence-to-sequence (Seq2Seq) models. MLPs can map the interacted features to the answer space, generating the final answer text. Seq2Seq models can generate more complex text answers, suitable for question-answering tasks requiring detailed explanation. For example, a decoder network can be used, starting with the interacted features as the initial state, to progressively generate each word of the answer text.

[0129] This embodiment encodes the text question through a multimodal reasoning module, and predicts the answer after attention interaction with enhanced 3D features. This achieves a deep association between question features and 3D features, enabling the model to accurately understand and answer text questions based on 3D scene information. This improves the ability of the 3D visual question answering task to handle complex semantic interactions, and enhances the accuracy of the answer and the depth of semantic understanding of the task.

[0130] Furthermore, in some embodiments, the 3D task processing method provided in this embodiment may further include a model training step. In the model training stage, a training dataset containing 3D point cloud data and its associated two-dimensional image data and / or text description data is used to train a pre-trained image feature extraction model and / or a pre-trained text feature extraction model, and a gated fusion module that performs cross-modal feature fusion processing is trained.

[0131] Specifically, the training dataset forms the foundation for model learning, containing 3D point cloud data along with associated 2D image data and / or text description data. This data needs to be preprocessed and labeled for training. For example, 3D point cloud data can come from LiDAR scans, 2D image data from camera captures, and text description data can be manually written object descriptions or scene descriptions. The training dataset is used to further train the pre-trained image feature extraction model and / or text feature extraction model, aiming to better adapt the model to the data characteristics and task requirements of a specific domain. For example, supervised learning methods can be used to guide the model in learning the mapping relationship between image and text features and 3D point cloud data using labeled data. Training the gated fusion module allows it to learn how to dynamically adjust the weights of each modality feature to achieve the best fusion effect. During training, the gated fusion module continuously optimizes its internal parameters and weight adjustment strategies based on the input image, text, and 3D point cloud features, as well as the corresponding annotation information.

[0132] This embodiment uses a training set containing 3D point cloud and associated image and text data to train a pre-trained image and text feature extraction model and a gated fusion module. This enables the model to learn effective cross-modal transfer and fusion rules from multimodal data, enhances the model's ability to utilize multimodal information, ensures stable output of high-quality enhanced features when processing downstream 3D tasks, and improves the model's generalization ability and practical application effect.

[0133] To facilitate understanding of the 3D task processing method based on cross-modal transfer of image and text provided in this embodiment, such as Figure 3 As shown, this embodiment also provides an overall technical roadmap for a 3D task processing method based on cross-modal transfer of image text, which is mainly divided into three branches:

[0134] 2D to 3D knowledge transfer and adaptation branch: After passing through the 2D-3D serialization module and the 2D-3D adapter, the 2D-3D fusion features are obtained.

[0135] Text-to-3D knowledge transfer and adaptation branch: Through text-3D serialization module and text-3D adapter processing, text-3D fusion features are generated.

[0136] Multimodal fusion branch: The fusion features from the above two branches enter the unified serialization module, and after processing by the gating device, form 2D-text-3D fusion features, which are ultimately applied to downstream 3D tasks. The overall technical route reflects the progressive process from unimodal migration to multimodal fusion, providing clear step-by-step guidance for the implementation of the framework.

[0137] In implementing this technical approach, the first step is knowledge transfer from 2D to 3D. We first explore using 2D pre-trained models, employing point cloud block-level sorting operations and introducing modal adapters to transfer these models to the 3D domain. The aim is to fully utilize mature models and knowledge in the 2D domain, overcoming the limitations imposed by insufficient 3D data. Secondly, we leverage prior knowledge from text modalities. Text modalities provide crucial discrete and sequential information for 3D models, playing an indispensable role in their training. Introducing prior knowledge from text modalities effectively supplements the deficiencies of 3D data, providing more comprehensive information for model training. Finally, we propose a unified multimodal transfer framework. In this framework, prior information from 2D and text is organically integrated through a gating module. This gating module dynamically adjusts the fusion method of the two prior information types based on different task requirements and data characteristics, thereby providing the model with rich and accurate prior knowledge and improving its robustness and accuracy.

[0138] The following section details the overall technical solution for a 3D task processing method based on cross-modal transfer of text from images. The specific process is as follows:

[0139] (1) Knowledge transfer and adaptation from 2D to 3D

[0140] Addressing the challenge of scarce 3D data, this paper first explores the use of 2D pre-trained models, such as ViT, Swin-Transformer, and DINO, both self-supervised and supervised pre-trained, to handle downstream tasks of 3D point clouds, including object classification and component segmentation. This embodiment primarily proposes three modules: a micro-point cloud grouping network, a point sequencer module, and a fine-tuning module. The main challenge is to address the disordered and sparse nature of point clouds. Micro-point cloud grouping aggregates sparse point clouds into point cloud block sets. The point sequencer module then sorts these blocks, creating ordered point cloud sets. Finally, the fine-tuning module integrates prior 2D knowledge with 3D information. Through these operations, projection-free processing and the combination of 3D point cloud characteristics and 2D pre-trained knowledge enable better adaptation to downstream point cloud tasks.

[0141] (2) Knowledge transfer and adaptation from text modality to 3D

[0142] After completing the research in the first part, this embodiment will shift its focus to exploring how to achieve deep integration of textual modal prior knowledge and 3D knowledge. Through careful observation, this embodiment has discovered a significant commonality between textual information and point cloud information—both are discrete data formats. Textual information is presented word by word, while point cloud information is represented by independent points in space. However, these discrete data are not randomly arranged, but rather follow specific order rules. Taking textual information as an example, although the characters are discretely distributed in form, they are strictly organized according to semantic logical order. Once the order is deviated, the model will encode the textual information incorrectly, thus failing to accurately understand the meaning expressed by the text. The same applies to point cloud information. Although point clouds are discrete in space, there is an inherent topological sequence relationship between the individual points. This topological sequence determines the coherence and logic of the point cloud's spatial structure, which is crucial for the 3D model to understand the shape, position, and spatial relationships of objects. Based on the above analysis, the first step planned for this embodiment is to accurately align the topological order of the point cloud with the semantic order of the text. By establishing a mapping relationship between the two, we can ensure that their respective information advantages are fully utilized during the fusion process, thereby improving the model's ability to process complex information and its accuracy of understanding.

[0143] 2.1 Text-3D Serialization Module

[0144] This embodiment first performs visualization work on some features of the point cloud, as shown in the following figure. Figure 2As shown on the left, when unsorted, point cloud features are scattered in one-dimensional space. This chaotic distribution makes it difficult to establish an effective correspondence between point cloud features and one-dimensional semantic sequences of text, and alignment operations are also impossible. To solve this problem, this embodiment employs a topological sorting operation. Through this operation, the chaotic point cloud features are organized, achieving an ordered arrangement in one-dimensional space. After topological sorting, the point cloud features present a complete and ordered sequence in one-dimensional space. At this point, the point cloud possesses sequence features similar to text, laying the foundation for subsequent operations. This embodiment can then accurately align the topological order structure of the point cloud with the semantic order structure of the text, and on this basis, achieve deep fusion of the two.

[0145] 2.2 Constructing a Text-3D Model

[0146] In this current stage, this embodiment will elaborate on how to construct a text-3D model based on the shared characteristics of point clouds and text. The overall model construction approach is similar to the 2D-3D framework. The main difference lies in the sorting module. In the text-3D model constructed in this embodiment, the semantic information of the text is used to guide the topological sorting of the 3D point cloud, thereby achieving a deep fusion of text position information and point cloud spatial information. Specifically, the text information is first converted into tokens, and then a multilayer perceptron is used to convert the text information described from different perspectives into unified semantic information, which can mitigate the bias caused by the text information described from different perspectives. Finally, the unified semantic information is used to guide the topological sorting of the 3D point cloud. Through this process, the logicality of text semantics and the structure of point cloud space can be fully utilized to provide the model with more comprehensive and effective information. After completing the above information fusion, this embodiment will carry out adjustment and fine-tuning work on the text pre-trained model. Through the text-3D adapter, the text pre-trained model is finely adjusted to better adapt to the characteristics of point cloud data, further optimizing the model's performance when processing text-3D data and improving the overall performance of the model.

[0147] (3) Establish a 3D multimodal transfer framework

[0148] After completing the research in Part Two, this embodiment discovered an inconsistency in transfer methods between image and text modalities. Taking the sorting module as an example, when using a 2D image pre-trained model, this embodiment relies on the prior features of the image to guide the point cloud sorting; while when using a text pre-trained model, it uses the semantic features of the text to guide the point cloud sorting. These two different sorting methods not only hinder the full utilization of prior knowledge from different modalities but also bring difficulties to the information fusion of images, text, and point clouds. To solve this problem, this embodiment decides to explore the establishment of a unified multimodal transfer framework. This embodiment will deeply analyze the image and text modal models to find common features between them. These common features will serve as key information to guide the point cloud sorting operation, thereby eliminating the sorting differences caused by different modal features and ensuring that the point cloud can evenly integrate the information of images and text during the sorting process. Meanwhile, by designing a gating module, the mined common features are effectively integrated with 3D data. The gating module can adapt and transform data from different modalities, so that common features can be smoothly integrated into 3D data. This successfully builds a unified multimodal transfer framework, providing strong support for more efficient processing of multimodal information in the future.

[0149] 3.1 Unified Serialization Module

[0150] With the widespread application of the Transformer architecture, pre-trained models based on this architecture have emerged for various modalities, such as ViT and DINO in the 2D image domain, and BERT and GPT in the text domain. When using the Transformer architecture, the raw modal data needs to be processed into tokens. Therefore, this embodiment will focus on token-level work. Specifically, this embodiment first processes the 3D data, converting it into token form. To effectively sort these tokens, this embodiment needs to utilize the common features of images and text for guidance. The steps to obtain this common feature are as follows: First, the image is converted into a one-dimensional vector using an encoder, and the same operation is performed on the text, encoding it into a one-dimensional vector; then, these two vectors are concatenated to obtain a new combined vector; finally, a multilayer perceptron is used to encode this combined vector, thereby obtaining a vector that represents the common features of the image and text. After obtaining this common feature vector, this embodiment can use it to guide the sorting of point cloud tokens, thereby achieving effective fusion and collaboration of multimodal data during processing.

[0151] 3.2 Constructing a Multimodal Transfer Framework

[0152] After processing by the unified serialization module, the 3D data generates an ordered set of 3D point cloud blocks. Subsequently, this embodiment uses a gating module to deeply fuse the 3D data, images, and text. The gating module can dynamically adjust the weights of each modality of data according to different data requirements. For example, when 3D data is used to represent a regular cube, due to the cube's simple structure and obvious discrete features, this embodiment increases the weight of text features. The semantic information contained in the text can provide a more accurate description of the discrete features of the 3D data, helping the model better understand and represent the structural characteristics of the cube. When 3D data is used to represent objects with complex texture features, such as leaves, the continuity and richness of texture are crucial. In this case, this embodiment increases the weight of image features. Images can intuitively show the continuous changes in texture, supplementing the 3D data with rich continuous feature details and improving the model's ability to depict leaf textures.

[0153] After completing the aforementioned feature adjustments, this embodiment applies the fused features to downstream 3D tasks, including item classification, semantic segmentation, and 3D-VAQ. In item classification, the model, leveraging the fused features, can more accurately identify the feature differences between different items, achieving efficient classification. In semantic segmentation, these features help the model accurately delineate the boundaries and categories of different objects in a 3D scene. In 3D-VAQ, the multimodal fusion features provide strong support for the model to understand the semantic information of the 3D scene and answer related questions, effectively improving the model's performance in various downstream 3D tasks.

[0154] In summary, the cross-modal transfer learning framework proposed in this embodiment effectively addresses the problem of insufficient model generalization ability caused by the scarcity of 3D data through a multimodal knowledge collaborative transfer mechanism, demonstrating significant innovative value in both academic and applied fields. In academic research, it innovatively constructs a joint mapping space of visual-linguistic-geometric representations. Through modal-shared feature projection and an adaptive attention mechanism, it achieves the organic integration of structured semantic priors of images and serialized semantic priors of text in 3D representation. This not only provides a scalable technical path for the unification of multimodal pre-trained models but also lays a new theoretical foundation for cross-modal geometric semantic understanding. In engineering applications, this framework, through its plug-and-play modular design, can be seamlessly integrated into mainstream 3D network architectures (such as PointNet++ and PointTransformer), enabling rapid deployment in practical scenarios such as autonomous driving environment perception, industrial robot 3D target grasping, and AR / VR scene reconstruction without complex model reconstruction. Compared to traditional methods, its advantages are reflected in three aspects: First, cross-modal knowledge distillation significantly improves the model's robustness to detection under complex conditions such as sparse point clouds and noise interference, such as improving the accuracy of recognizing occluded pedestrians in autonomous driving systems; second, it utilizes textual semantic guidance to enhance fine-grained classification capabilities, improving defect classification accuracy in industrial quality inspection scenarios; third, relying on the strong generalization characteristics of the pre-trained model, only 10% of the labeled data is needed to achieve the performance of the baseline model trained on the full dataset, greatly reducing data labeling costs. This application can also be extended to multiple fields such as smart city 3D modeling, medical image organ reconstruction, and digital protection of cultural heritage, promoting the evolution of 3D vision technology towards higher levels of scene understanding and interaction capabilities.

[0155] It should be understood that, although Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0156] To facilitate better implementation of the 3D task processing method based on image-text cross-modal transfer according to the embodiments of this application, this application also provides a 3D task processing device based on image-text cross-modal transfer, which is based on the above-described 3D task processing method based on image-text cross-modal transfer. The meanings of the terms used are the same as in the above-described 3D task processing method based on image-text cross-modal transfer, and specific implementation details can be found in the description of the method embodiments.

[0157] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a 3D task processing device based on cross-modal transfer of image and text provided in an embodiment of this application. Specifically, the 3D task processing device may include a data acquisition module 201, a data association module 202, a feature extraction module 203, a sequence generation module 204, a feature fusion module 205, and a task execution module 206, as follows:

[0158] The data acquisition module 201 is used to acquire 3D point cloud data to be processed, perform structured processing on the 3D point cloud data to be processed, and obtain an ordered set of point cloud blocks.

[0159] The data association module 202 is used to acquire at least one two-dimensional image data and / or at least one text description data associated with the 3D point cloud data;

[0160] The feature extraction module 203 is used to process two-dimensional image data using a pre-trained image feature extraction model to obtain image feature vectors; and / or to process text description data using a pre-trained text feature extraction model to obtain text feature vectors.

[0161] The sequence generation module 204 is used to perform unified serialization processing on an ordered set of point cloud blocks based on image feature vectors and / or text feature vectors to generate a 3D point cloud feature sequence that integrates multimodal information.

[0162] The feature fusion module 205 is used to input the 3D point cloud feature sequence, image feature vector and text feature vector of fused multimodal information into the gated fusion module, and dynamically adjust the weight of each modal feature and fuse them to generate an enhanced 3D feature representation.

[0163] Task execution module 206 is used to execute specified 3D downstream tasks based on the enhanced 3D feature representation.

[0164] Specific limitations regarding the 3D task processing device based on image-text cross-modal transfer can be found in the limitations of the 3D task processing method based on image-text cross-modal transfer described above, and will not be repeated here. Each module in the aforementioned 3D task processing device based on image-text cross-modal transfer can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0165] The 3D task processing device based on image-text cross-modal transfer provided in this embodiment acquires 3D point cloud data to be processed through a data acquisition module 201, performs structured processing on the 3D point cloud data to be processed, and obtains an ordered set of point cloud blocks; acquires at least one two-dimensional image data and / or at least one text description data associated with the 3D point cloud data through a data association module 202; processes the two-dimensional image data using a pre-trained image feature extraction model through a feature extraction module 203 to obtain image feature vectors; and / or processes the text description data using a pre-trained text feature extraction model to obtain text feature vectors; and, based on the image feature vectors, processes the text description data through a sequence generation module 204. The system uses quantitative and / or textual feature vectors to perform unified serialization processing on an ordered set of point cloud blocks, generating a 3D point cloud feature sequence that integrates multimodal information. The feature fusion module 205 inputs the 3D point cloud feature sequence, image feature vectors, and textual feature vectors into a gating fusion module. The gating fusion module dynamically adjusts the weights of each modality feature and fuses them to generate an enhanced 3D feature representation. The task execution module 206 executes a specified 3D downstream task based on the enhanced 3D feature representation. Therefore, this embodiment can improve the robustness, accuracy, and generalization ability of 3D downstream tasks in data-scarce scenarios, achieving efficient and robust 3D scene understanding.

[0166] Furthermore, embodiments of this application also provide an electronic device, such as... Figure 5 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0167] The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will understand that... Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0168] The processor 301 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 302, and by calling data stored in the memory 302, thereby providing overall monitoring of the electronic device. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 301.

[0169] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and 3D task processing methods based on image-text cross-modal transfer by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0170] The electronic device also includes a power supply 303 that supplies power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 303 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0171] The electronic device may also include an input unit 304, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0172] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 301 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 302 according to the following instructions, and the processor 301 runs the applications stored in the memory 302 to realize various functions, as follows:

[0173] The system acquires 3D point cloud data to be processed, performs structured processing on the 3D point cloud data to obtain an ordered set of point cloud blocks; acquires at least one 2D image data and / or at least one text description data associated with the 3D point cloud data; processes the 2D image data using a pre-trained image feature extraction model to obtain image feature vectors; and / or processes the text description data using a pre-trained text feature extraction model to obtain text feature vectors; performs unified serialization processing on the ordered set of point cloud blocks based on the image feature vectors and / or text feature vectors to generate a 3D point cloud feature sequence that integrates multimodal information; inputs the 3D point cloud feature sequence, image feature vectors, and text feature vectors that integrate multimodal information into a gating fusion module, dynamically adjusts the weights of each modality feature through the gating fusion module, and fuses them to generate an enhanced 3D feature representation; and executes the specified 3D downstream task based on the enhanced 3D feature representation.

[0174] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0175] This application embodiment acquires and structures 3D point cloud data, dividing it into an ordered set of point cloud blocks. Simultaneously, it combines associated 2D image data and / or text description data, utilizing pre-trained image feature extraction and text feature extraction models to extract image feature vectors and text feature vectors, respectively. These feature vectors provide multimodal prior information for the 3D point cloud data. Based on these feature vectors, the ordered set of point cloud blocks is uniformly serialized to generate a 3D point cloud feature sequence that fuses multimodal information, achieving effective fusion between different modalities. The fused feature sequence is input into a gated fusion module, which dynamically adjusts the weights of each modality feature according to task requirements and data characteristics, and then fuses them to generate an enhanced 3D feature representation. This allows for flexible adjustment of the contribution of each modality data according to specific tasks, improving the model's understanding and representation capabilities of 3D data. Based on the enhanced 3D feature representation, a specified 3D downstream task is executed, thereby improving the robustness, accuracy, and generalization ability of the 3D downstream task in data-scarce scenarios, achieving efficient and robust 3D scene understanding.

[0176] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0177] To this end, embodiments of this application provide a storage medium storing multiple instructions that can be loaded by a processor to execute steps in any of the 3D task processing methods based on image-text cross-modal transfer provided in embodiments of this application. For example, the instructions can execute the following steps:

[0178] The system acquires 3D point cloud data to be processed, performs structured processing on the 3D point cloud data to obtain an ordered set of point cloud blocks; acquires at least one 2D image data and / or at least one text description data associated with the 3D point cloud data; processes the 2D image data using a pre-trained image feature extraction model to obtain image feature vectors; and / or processes the text description data using a pre-trained text feature extraction model to obtain text feature vectors; performs unified serialization processing on the ordered set of point cloud blocks based on the image feature vectors and / or text feature vectors to generate a 3D point cloud feature sequence that integrates multimodal information; inputs the 3D point cloud feature sequence, image feature vectors, and text feature vectors that integrate multimodal information into a gating fusion module, dynamically adjusts the weights of each modality feature through the gating fusion module, and fuses them to generate an enhanced 3D feature representation; and executes the specified 3D downstream task based on the enhanced 3D feature representation.

[0179] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0180] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0181] Since the instructions stored in the storage medium can execute the steps in any of the 3D task processing methods based on image-text cross-modal transfer provided in the embodiments of this application, the beneficial effects that any of the 3D task processing methods based on image-text cross-modal transfer provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0182] The foregoing has provided a detailed description of a 3D task processing method and apparatus based on cross-modal transfer of image and text provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A 3D task processing method based on cross-modal transfer of text in images, characterized in that, Includes the following steps: Acquire 3D point cloud data to be processed, perform structured processing on the 3D point cloud data to be processed, and obtain an ordered set of point cloud blocks; Acquire at least one two-dimensional image data and / or at least one text description data associated with the 3D point cloud data; The two-dimensional image data is processed using a pre-trained image feature extraction model to obtain an image feature vector; and / or, the text description data is processed using a pre-trained text feature extraction model to obtain a text feature vector; Based on the image feature vector and / or the text feature vector, the ordered set of point cloud blocks is subjected to unified serialization processing to generate a 3D point cloud feature sequence that integrates multimodal information. The 3D point cloud feature sequence with fused multimodal information, the image feature vector, and the text feature vector are input into the gated fusion module. The gated fusion module dynamically adjusts the weights of each modal feature and fuses them to generate an enhanced 3D feature representation. Based on the enhanced 3D feature representation, the specified 3D downstream task is executed.

2. The 3D task processing method based on cross-modal transfer of image and text according to claim 1, characterized in that, The structuring of the 3D point cloud data to be processed yields an ordered set of point cloud blocks, including: The 3D point cloud data to be processed is divided into multiple point cloud blocks; A sorting operation is performed on the plurality of point cloud blocks to form an ordered set of point cloud blocks; wherein the sorting operation is based on the spatial location information and / or geometric feature information of the point cloud blocks.

3. The 3D task processing method based on cross-modal transfer of image and text according to claim 1, characterized in that, The step of processing the two-dimensional image data using a pre-trained image feature extraction model to obtain an image feature vector; and / or processing the text description data using a pre-trained text feature extraction model to obtain a text feature vector includes: The image feature vector of the two-dimensional image data is extracted using a visual model based on the Transformer architecture; The text feature vectors of the text description data are extracted using a language model based on the Transformer architecture.

4. The 3D task processing method based on cross-modal transfer of image text according to claim 1, characterized in that, Before performing unified serialization processing on the ordered set of point cloud blocks based on the image feature vector and / or the text feature vector to generate a 3D point cloud feature sequence that fuses multimodal information, the method further includes: The image feature vector and / or the text feature vector are input into the modality adapter for processing; The modal adapter adapts image features and / or text features to the 3D point cloud feature space.

5. The 3D task processing method based on cross-modal transfer of image and text according to claim 1, characterized in that, The step of performing unified serialization processing on the ordered set of point cloud blocks based on the image feature vector and / or the text feature vector to generate a 3D point cloud feature sequence that integrates multimodal information includes: Obtain a fused feature vector for guiding the sorting, the fused feature vector being generated based on the image feature vector and / or the text feature vector; Using the fused feature vector, the point cloud blocks in the ordered point cloud block set are reordered or their sequence positions are adjusted to generate the 3D point cloud feature sequence that fuses multimodal information.

6. The 3D task processing method based on cross-modal transfer of image and text according to claim 5, characterized in that, When both image feature vectors and text feature vectors exist, obtaining the fused feature vector used to guide the sorting includes: The image feature vector and the text feature vector are concatenated or weighted and fused. The concatenated or weighted fused vectors are encoded using a neural network to generate a fused feature vector for guiding the ranking.

7. The 3D task processing method based on cross-modal transfer of image and text according to claim 1, characterized in that, The dynamic adjustment of the weights of each modal feature through the gating fusion module includes: The fusion weights of image features, text features, and 3D point cloud features are dynamically adjusted based on the attributes of the object represented by the 3D point cloud data, the requirements of the 3D downstream task, and at least one of the image feature vector and the text feature vector.

8. The 3D task processing method based on cross-modal transfer of image and text according to claim 1, characterized in that, The execution of specified 3D downstream tasks based on the enhanced 3D feature representation includes: Select the corresponding task adaptation network based on the type of the 3D downstream task; The enhanced 3D feature representation is input into the task adaptation network to generate corresponding task result data.

9. The 3D task processing method based on cross-modal transfer of image and text according to claim 8, characterized in that, The step of selecting the corresponding task adaptation network based on the type of the 3D downstream task includes: When the 3D downstream task is an object classification task, the task adaptation network includes a fully connected layer and a classifier to output the category probability distribution. When the 3D downstream task is a semantic segmentation task, the task adaptation network includes a pointwise feature decoder for outputting the semantic label of each point. When the 3D downstream task is an object detection task, the task adaptation network includes a bounding box regression module, which is used to output the object position and category; When the 3D downstream task is a 3D visual question answering task, the task adaptation network includes a multimodal reasoning module for receiving question text and outputting answers.

10. A 3D task processing device based on cross-modal transfer of image and text, characterized in that, include: The data acquisition module is used to acquire 3D point cloud data to be processed, and to perform structured processing on the 3D point cloud data to be processed to obtain an ordered set of point cloud blocks. The data association module is used to acquire at least one two-dimensional image data and / or at least one text description data associated with the 3D point cloud data; The feature extraction module is used to process the two-dimensional image data using a pre-trained image feature extraction model to obtain an image feature vector; and / or to process the text description data using a pre-trained text feature extraction model to obtain a text feature vector. The sequence generation module is used to perform unified serialization processing on the ordered set of point cloud blocks based on the image feature vector and / or the text feature vector to generate a 3D point cloud feature sequence that integrates multimodal information. The feature fusion module is used to input the 3D point cloud feature sequence of the fused multimodal information, the image feature vector and the text feature vector into the gated fusion module, and dynamically adjust the weights of each modal feature and fuse them to generate an enhanced 3D feature representation. The task execution module is used to execute specified 3D downstream tasks based on the enhanced 3D feature representation.

Images