Three-dimensional point cloud scene understanding method based on large language model and related products

Abstract

The invention discloses a three-dimensional point cloud scene understanding method based on a large language model and a related product. The method comprises the following steps: acquiring a three-dimensional scene to be understood; collecting a plurality of two-dimensional multi-view images of a three-dimensional scene to be understood; obtaining view scene features based on the plurality of two-dimensional multi-view images; performing interactive learning on the three-dimensional learnable query vector by utilizing the view scene features and three-dimensional point cloud data corresponding to a three-dimensional scene to be understood to obtain three-dimensional perception scene embedding representation; and understanding the three-dimensional perception scene embedding representation by using the large language model, and generating an understanding result corresponding to the three-dimensional scene to be understood. The understanding accuracy of the large language model is improved by utilizing the view scene features. And performing interactive learning on a three-dimensional learnable query vector by utilizing view scene features and three-dimensional point cloud data, so that information from different modals is embedded, represented and fused in a three-dimensional perception scene, and complex information in the three-dimensional scene can be more accurately understood by a large language model.

Description

Technical Field

[0001] This application relates to the technical field of three-dimensional scene understanding, and in particular to a three-dimensional point cloud scene understanding method and related products based on a large language model. Background Art

[0002] With the booming development of natural language processing technology and computer vision technology, scene understanding has become a new hot spot. Among them, models such as CLIP and Uni3D have explored alignment methods for multimodal data. Given the significant advantages of large language models in processing natural language, researchers are exploring how to use ordinary large language models to solve a wide range of multimodal tasks. Two-dimensional vision-language tasks have received sufficient attention, and these tasks attempt to endow large language models with the ability to understand two-dimensional images. The mainstream two-dimensional vision-language model technologies can be divided into two paradigms: one is to try to train the model with a large amount of text-image instruction adjustment data (such as CLIP and LLaVA), and the other paradigm focuses on connecting the pre-trained vision encoder and the large language model through additional learnable modules (such as the perceptron in Flamingo and the Q-Former in BLIP-2) to align the latent space of visual embeddings with the text embeddings of the large language model.

[0003] Similar to two-dimensional vision-language, researchers have also introduced large language models to understand the three-dimensional world. Existing methods aim to align the three-dimensional modality with the text modality to build a three-dimensional multimodal large language model. This alignment enables the large language model to understand three-dimensional visual representations, laying the foundation for solving downstream tasks in three-dimensional scenes. Integrating three-dimensional representations into large language models usually requires adding special structures that can effectively process spatial information. The 3D Data and Large Language Model (3DLLM) was the first to inject three-dimensional spatial information into the large language model, marking a major advancement in this field. LL3DA introduced visual prompts based on the Q-Former architecture, opening up new opportunities for large language models to solve tasks in the three-dimensional world. By applying three-dimensional prompts, 3DMIT eliminated the alignment stage between the three-dimensional scene and natural language, but it involves the fine-tuning stage of the large language model.

[0004] Although some research on three-dimensional multimodal large language models has emerged, the three-dimensional scene understanding method based on large language models is still not fully mature. Three-dimensional point clouds depict the scene from an overall perspective, providing the structure and spatial information of the three-dimensional world. Therefore, current methods highly rely on three-dimensional point clouds in scene understanding.

[0005] However, the three-dimensional point cloud reconstruction process of indoor scenes often leads to information loss. One problem is that areas with textureless planes or repetitive patterns (such as uniform walls, homogeneous floor tiles, and glass surfaces) lack unique features in the captured images. This defect seriously hinders the effectiveness of stereo matching algorithms, as the algorithm relies on establishing robust correspondences between image pairs. As a result, these homogeneous regions are easily missed and appear as holes in the reconstructed three-dimensional point cloud.

[0006] Another problem is that when dealing with objects with complex structures, the misalignment between the captured images and the densely reconstructed point cloud easily leads to detail distortion. This distortion hinders the complete retention of detailed information such as surface texture, boundary contours, and microscopic features. These elements, which are often crucial for a comprehensive scene representation, may appear blurred or indistinguishable in the reconstructed three-dimensional point cloud. The degradation of the overall quality and utility of the reconstructed three-dimensional point cloud weakens the ability to represent three-dimensional scenes and may lead to a significant decline in the performance of existing methods.

[0007] In addition, some three-dimensional scene understanding methods based on large language models fail to fully achieve the effective fusion of multi-modal information in three-dimensional scenes, making it difficult to generate a unified feature representation that conforms to the directivity of text instructions. It is still necessary to segment objects in three-dimensional scenes and fine-tune the backbone network of large language models, etc., which will significantly consume processing time and memory. Summary of the Invention

[0008] Based on the above problems, the present application provides a three-dimensional point cloud scene understanding method and related products based on a large language model, aiming to improve the accuracy of the large language model in understanding three-dimensional point cloud scenes.

[0009] The embodiments of the present application disclose the following technical solutions:

[0010] The first aspect of the present application provides a three-dimensional point cloud scene understanding method based on a large language model, including:

[0011] Obtain a three-dimensional scene to be understood;

[0012] Collect multiple two-dimensional multi-view images of the three-dimensional scene to be understood;

[0013] Obtain view scene features based on the multiple two-dimensional multi-view images;

[0014] Use the view scene features and the three-dimensional point cloud data corresponding to the three-dimensional scene to be understood to perform interactive learning on three-dimensional learnable query vectors, and obtain a three-dimensional perception scene embedding representation;

[0015] Use the large language model to understand the three-dimensional perception scene embedding representation and generate an understanding result corresponding to the three-dimensional scene to be understood.

[0016] Optionally, obtaining the view scene features based on the multiple two-dimensional multi-view images specifically includes:

[0017] Performing feature extraction on the multiple two-dimensional multi-view images to obtain visual features corresponding to each two-dimensional multi-view image;

[0018] Performing alignment processing on the visual features corresponding to each two-dimensional multi-view image and the text instructions to obtain the aligned visual features corresponding to each two-dimensional multi-view image;

[0019] Determining the camera pose corresponding to each two-dimensional multi-view image;

[0020] Based on the aligned visual features and camera poses corresponding to each two-dimensional multi-view image, obtaining the view scene features.

[0021] Optionally, obtaining the view scene features based on the aligned visual features and camera poses corresponding to each two-dimensional multi-view image specifically includes:

[0022] Performing transformation processing on the camera poses corresponding to each two-dimensional multi-view image to obtain the position embedding representations corresponding to each two-dimensional multi-view image;

[0023] Performing fusion processing on the aligned visual features and position embedding representations corresponding to each two-dimensional multi-view image to obtain the scene features corresponding to each two-dimensional multi-view image;

[0024] Performing aggregation processing on the scene features corresponding to the multiple two-dimensional multi-view images to obtain the view scene features.

[0025] Optionally, the interactive learning of the three-dimensional learnable query vector using the view scene features and the three-dimensional point cloud data corresponding to the three-dimensional scene to be understood to obtain the three-dimensional perception scene embedding representation specifically includes:

[0026] Using a self-attention layer with shared weights to perform interactive learning on the three-dimensional learnable query vector and the view scene features, and the vector after the first interactive learning;

[0027] Using a cross-attention layer to perform interaction on the vector after the first interactive learning and the three-dimensional point cloud data corresponding to the three-dimensional scene to be understood to obtain the three-dimensional perception scene embedding representation.

[0028] Optionally, collecting multiple two-dimensional multi-view images of the three-dimensional scene to be understood specifically includes:

[0029] Using a camera to collect multiple two-dimensional multi-view images of the three-dimensional scene to be understood.

[0030] The second aspect of this application provides a three-dimensional point cloud scene understanding device based on a large language model, including:

[0031] An acquisition module, configured to acquire a three-dimensional scene to be understood;

[0032] An acquisition model, configured to acquire multiple two-dimensional multi-view images of the three-dimensional scene to be understood;

[0033] A view scene feature determination module, configured to obtain view scene features based on the multiple two-dimensional multi-view images;

[0034] An interaction module, configured to perform interactive learning on a three-dimensional learnable query vector by using the view scene features and three-dimensional point cloud data corresponding to the three-dimensional scene to be understood, so as to obtain a three-dimensional perception scene embedding representation;

[0035] An understanding result determination module, configured to use a large language model to understand the three-dimensional perception scene embedding representation and generate an understanding result corresponding to the three-dimensional scene to be understood.

[0036] Optionally, the view scene feature determination module includes:

[0037] A feature extraction unit, configured to perform feature extraction on the multiple two-dimensional multi-view images to obtain visual features corresponding to each two-dimensional multi-view image;

[0038] A preprocessing unit, configured to perform alignment processing on the visual features corresponding to each two-dimensional multi-view image and a text instruction to obtain aligned visual features corresponding to each two-dimensional multi-view image;

[0039] A camera pose determination unit, configured to determine the camera pose corresponding to each two-dimensional multi-view image;

[0040] A view scene determination unit, configured to obtain view scene features based on the aligned visual features and camera poses corresponding to each two-dimensional multi-view image.

[0041] A third aspect of the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the three-dimensional point cloud scene understanding method provided in the first aspect.

[0042] A fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the three-dimensional point cloud scene understanding method provided in the first aspect.

[0043] A fifth aspect of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the three-dimensional point cloud scene understanding method provided in the first aspect.

[0044] Compared with the prior art, the present application has the following beneficial effects:

[0045] The present application obtains a three-dimensional scene to be understood; acquires multiple two-dimensional multi-view images of the three-dimensional scene to be understood; obtains view scene features based on the multiple two-dimensional multi-view images; performs interactive learning on a three-dimensional learnable query vector by using the view scene features and the three-dimensional point cloud data corresponding to the three-dimensional scene to be understood, to obtain a three-dimensional perception scene embedding representation; and uses a large language model to understand the three-dimensional perception scene embedding representation and generate an understanding result corresponding to the three-dimensional scene to be understood. The present application first uses the view scene features to solve the problems of missing three-dimensional point cloud data and insufficient object details in the existing three-dimensional scene understanding solutions, significantly improves the accuracy of the three-dimensional perception scene embedding representation, and further improves the understanding accuracy of the large language model. Moreover, the present application performs interactive learning on the three-dimensional learnable query vector by using the view scene features and the three-dimensional point cloud data, so that the three-dimensional perception scene embedding representation integrates information from different modalities, effectively solves the problem of insufficient fusion of different modality information in the prior art, and enables the large language model to more accurately understand the complex information in the three-dimensional scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following described drawings are only some embodiments of the present application, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0047] Figure 1 It is a flowchart of a three-dimensional point cloud scene understanding method based on a large language model provided by an embodiment of the present application;

[0048] Figure 2 It is a first flowchart for determining view scene features provided by an embodiment of the present application;

[0049] Figure 3 It is a second flowchart for determining view scene features provided by an embodiment of the present application;

[0050] Figure 4 It is a structural diagram of a three-dimensional point cloud scene understanding device based on a large language model provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0051] As described above, there are problems of missing three-dimensional point cloud data, insufficient object details, and insufficient fusion of same modality information in the existing three-dimensional scene understanding solutions.

[0052] In view of the above problems, the present application provides a three-dimensional point cloud scene understanding method and related products based on a large language model. The method includes: obtaining a three-dimensional scene to be understood; collecting multiple two-dimensional multi-view images of the three-dimensional scene to be understood; obtaining view scene features based on the multiple two-dimensional multi-view images; using the view scene features and the three-dimensional point cloud data corresponding to the three-dimensional scene to be understood to perform interactive learning on three-dimensional learnable query vectors to obtain a three-dimensional perceptual scene embedding representation; using the large language model to understand the three-dimensional perceptual scene embedding representation and generate an understanding result corresponding to the three-dimensional scene to be understood.

[0053] By integrating view scene features into the three-dimensional perceptual scene embedding representation in the present application, the problems that existing methods highly rely on three-dimensional point cloud representations, resulting in an inability to give accurate answers when facing information loss in the three-dimensional point cloud reconstruction process, and that existing methods ignore the detailed information that two-dimensional multi-view images can provide and their visual consistency with three-dimensional point clouds, restricting the large language model's understanding of complex object details in three-dimensional scenes are solved.

[0054] By combining view scene features and three-dimensional point cloud data in the present application, the problem that existing methods fail to fully achieve the effective fusion of multi-modal information in three-dimensional scenes and are difficult to generate a unified feature representation that conforms to the text instruction directivity is solved, significantly improving the accuracy of the three-dimensional perceptual scene embedding representation and the understanding ability of the large language model.

[0055] To enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0056] Figure 1 The following is a flowchart of a three-dimensional point cloud scene understanding method provided by an embodiment of the present application. As Figure 1 shown, a three-dimensional point cloud scene understanding method based on a large language model includes:

[0057] S101: Obtain a three-dimensional scene to be understood.

[0058] The present application does not limit the three-dimensional scene. It is mainly to clarify the target scene and lay a foundation for subsequent data collection and processing.

[0059] S102: Collect multiple two-dimensional multi-view images of the three-dimensional scene to be understood.

[0060] This application does not limit the device for collecting two-dimensional multi-view images. For example, cameras, mobile phones, digital cameras, etc.

[0061] As an alternative embodiment, a camera is used to collect multiple two-dimensional multi-view images of the three-dimensional scene to be understood.

[0062] S102 mainly provides two-dimensional learning materials for subsequent three-dimensional learnable query vectors. By capturing the three-dimensional scene from different angles, rich visual information is provided, which helps to comprehensively understand the details of the three-dimensional scene. This enables the three-dimensional learnable query vectors to fully learn the three-dimensional scene to be understood, further ensuring that the large language model fully understands the three-dimensional scene.

[0063] S103: Obtain view scene features based on the multiple two-dimensional multi-view images.

[0064] This application does not specifically limit the method for determining view scene features, as long as it is ensured that the view scene features contain rich scene details.

[0065] Since two-dimensional multi-view images generally display objects visually and spatially, this application provides an alternative embodiment to extract visual features from each two-dimensional multi-view image to obtain the corresponding visual features of each two-dimensional multi-view image.

[0066] The visual features are the visual semantic features of the two-dimensional multi-view images.

[0067] Extract spatial features from each two-dimensional multi-view image to obtain the corresponding spatial features of each two-dimensional multi-view image.

[0068] This application does not limit the method for spatial extraction. For example, determine the camera pose of each two-dimensional multi-view image, and determine the corresponding spatial features of each two-dimensional multi-view image based on the camera pose of each two-dimensional multi-view image.

[0069] This application does not limit the processing method of the camera pose. For example, using Flatten+MLP, or extracting the plucker features of the camera pose corresponding to each two-dimensional multi-view image.

[0070] Fuse the visual features and spatial features corresponding to each two-dimensional multi-view image to obtain view scene features.

[0071] Since the visual features and spatial features corresponding to each two-dimensional multi-view image can only represent the information of the three-dimensional scene involved in that image, there are certain limitations. Therefore, the visual features and spatial features corresponding to each two-dimensional multi-view image can be fused to obtain more comprehensive information about the three-dimensional scene.

[0072] The above embodiments only illustrate relatively specifically the method for determining view scene features, and do not represent that the method for determining view scene features in S103 is limited to the above one. The method for determining view scene features can be defined according to the actual situation. By using the view scene features extracted from multiple two-dimensional multi-view images, the deficiencies of three-dimensional point cloud data in describing the details of complex objects are supplemented. This combination not only makes up for the possible information loss in traditional three-dimensional point cloud data, but also increases the capture of detailed features such as the surface texture and color of the object. In this way, the generated three-dimensional perception scene embedding representation is more accurate and comprehensive, thus improving the accuracy of the subsequent large language model's understanding of three-dimensional scenes.

[0073] S104: Use the view scene features and the three-dimensional point cloud data corresponding to the three-dimensional scene to be understood to perform interactive learning on the three-dimensional learnable query vector, and obtain a three-dimensional perception scene embedding representation.

[0074] This application does not limit the method of interactive learning. For example, an interactive model is used to perform interactive learning on the three-dimensional learnable query vector to obtain a three-dimensional perception scene embedding representation. The interactive model includes but is not limited to models such as PerceiverResampler and Q-Former.

[0075] As an alternative embodiment, based on the Q-Former model, use the view scene features and the three-dimensional point cloud data corresponding to the three-dimensional scene to be understood to perform interactive learning on the three-dimensional learnable query vector, and obtain a three-dimensional perception scene embedding representation, specifically including:

[0076] Use the self-attention or multi-head attention mechanism to perform interactive learning on the three-dimensional learnable query vector based on the view scene features and the three-dimensional point cloud data corresponding to the three-dimensional scene to be understood, and obtain a three-dimensional perception scene embedding representation.

[0077] S105: Use the large language model to understand the three-dimensional perception scene embedding representation and generate an understanding result corresponding to the three-dimensional scene to be understood.

[0078] This application directly uses the large language model to understand the three-dimensional perception scene embedding representation, without the need for an efficient training strategy to fine-tune the backbone network of the large language model. This application adopts a specific process, including S103 obtaining view scene features based on the multiple two-dimensional multi-view images and S104 using the view scene features and the three-dimensional point cloud data corresponding to the three-dimensional scene to be understood to perform interactive learning on the three-dimensional learnable query vector to obtain a three-dimensional perception scene embedding representation; so that there is no need to fine-tune the backbone network of the large language model, ensuring the performance of the model in the three-dimensional scene understanding task.

[0079] This application can effectively solve the problems in the prior art, such as the inability to handle information loss during 3D point cloud reconstruction and the insufficient understanding of the details of complex objects.

[0080] The above is the main technical solution of this application. Now, the extended implementation of the main technical solution of this application will be introduced. Specifically as follows:

[0081] Different from 3D point clouds that tend to depict the overall scene, 2D multi-view images can provide more detailed information about scene components. Specifically, this application designs a novel method for aggregating multi-view images for subsequent enhancement, including extracting multi-view features with spatial information and then aggregating these features to obtain view scene features. That is, S103 obtains view scene features based on the multiple 2D multi-view images. This application provides an optional embodiment:

[0082] Figure 2 The first flowchart for determining view scene features provided by an embodiment of this application is as Figure 2 shown. S201: Extract features from the multiple 2D multi-view images to obtain the visual features corresponding to each 2D multi-view image.

[0083] This application does not limit the method of feature extraction. For example:

[0084] All 2D multi-view images of the 3D scene to be understood are processed by a pre-trained image encoder to obtain the corresponding to the 2D multi-view image, where n is the 2D multi-view image. The pre-trained image encoder here includes but is not limited to various models such as ViT, ResNet, DenseNet, etc. Taking ViT as an example, for the i-th 2D multi-view image I i , the visual feature f i corresponding to the i-th 2D multi-view image is obtained in the following way:

[0085] f i = ViT(I i ).

[0086] S202: Align the visual feature corresponding to each 2D multi-view image with the text instruction to obtain the aligned visual feature corresponding to each 2D multi-view image.

[0087] This application does not limit the method of alignment processing. As an optional embodiment:

[0088] Apply a two-dimensional image-text alignment module to derive visual features related to text instructions. Here, the two-dimensional image-text alignment module can select models such as Q-Former, Oscar, Uniter, etc. Taking Q-Former as an example, the aligned visual feature q corresponding to the i-th two-dimensional multi-view image i can be obtained by the following method:

[0089] q i = Q-Former(f i ).

[0090] S203: Determine the camera pose corresponding to each two-dimensional multi-view image.

[0091] Since the visual features corresponding to each two-dimensional multi-view image extracted above lack spatial information, the present application also needs to determine the spatial features of each two-dimensional multi-view image based on the camera pose corresponding to each two-dimensional multi-view image.

[0092] S204: Based on the aligned visual features and camera poses corresponding to each two-dimensional multi-view image, obtain view scene features.

[0093] The present application does not limit the method for determining the visual scene features. As an optional embodiment:

[0094] Figure 3 This is the second flowchart for determining the view scene features provided by the embodiments of the present application. As Figure 3 shown, S301: Perform conversion processing on the camera pose corresponding to each two-dimensional multi-view image to obtain the position embedding representation corresponding to each two-dimensional multi-view image.

[0095] Essentially, each two-dimensional multi-view image has a camera pose, which refers to the position and orientation of the two-dimensional multi-view image relative to the three-dimensional scene, and thus indirectly reflects some spatial information. Therefore, the present application uses a flatten layer, followed by a two-layer multi-layer perceptron (MLP), to convert the camera pose p corresponding to the i-th two-dimensional multi-view image i into the position embedding representation PosEmb corresponding to each two-dimensional multi-view image i :

[0096] PosEmb i = MLP(Flatten(p i ))).

[0097] S302: Perform fusion processing on the aligned visual features and position embedding representations corresponding to each two-dimensional multi-view image to obtain the scene features corresponding to each two-dimensional multi-view image.

[0098] Through the above conversion, the spatial information is directly encoded into the position embedding representation. Then, these position embedding representations are added to the scene features to ensure that the spatial context of each 2D multi-view image is retained in the scene features corresponding to each 2D multi-view image.

[0099] Taking the i-th 2D multi-view image as an example, the scene feature F corresponding to the i-th 2D multi-view image i view :

[0100] F i view = q i + PosEmb i .

[0101] S303: Aggregate the scene features corresponding to multiple 2D multi-view images to obtain the view scene features.

[0102] This application does not limit the aggregation method. As an optional embodiment:

[0103] Aggregate the scene features corresponding to multiple 2D multi-view images into the view scene features. Since F i view only represents partial information of the 3D scene to be understood. Therefore, this application explores using multiple stacked Transformer layers to fuse the scene features corresponding to multiple 2D multi-view images into the view scene features.

[0104] Specifically, after performing the self-attention operation, the multi-head cross-attention (MHCA) mechanism is adopted to perform attention interaction between the scene features corresponding to multiple 2D multi-view images and a set of learnable query vectors Q.

[0105] The aggregation process of the view scene features can be described as follows:

[0106]

[0107] Among them, q, k, and v represent query, key, and value respectively in the attention mechanism, and Proj() represents a simple linear layer. This mechanism can perform attention interaction between the scene features of different views, effectively integrating spatial context information from multiple perspectives. Integrate the spatial information into the scene features and fuse these features into the view scene features, thereby providing detailed information about the scene.

[0108] Considering that the constructed 3D point cloud features may face information loss, and the view scene features contain detailed information, the present application extracts comprehensive and detailed clues through the interaction between the 3D point cloud features and the view scene features. The implementation methods include but are not limited to models such as Perceiver Resampler and Q-Former. For example, in S104, the 3D learnable query vectors are interactively learned using the view scene features and the 3D point cloud data corresponding to the 3D scene to be understood, obtaining a 3D perceptual scene embedding representation. The present application provides an alternative embodiment:

[0109] Use a self-attention layer with shared weights to interactively learn the 3D learnable query vectors and the view scene features, and the vectors after the first interactive learning.

[0110] Use a cross-attention layer to interact the vectors after the first interactive learning and the 3D point cloud data corresponding to the 3D scene to be understood, obtaining the 3D perceptual scene embedding representation.

[0111] Specifically, S104 includes a set of trainable vectors, namely 3D learnable query vectors, and the model performs a two-step interaction process on the 3D learnable query vectors. First, the 3D learnable query vectors interact with the view scene features through a self-attention layer with shared weights. The self-attention mechanism ensures that the 3D learnable query vectors can obtain rich and detailed visual context information, which is crucial for eliminating ambiguity and compensating for information loss caused by alignment problems. It effectively helps the 3D learnable query vectors capture detailed information that may be lost during the point cloud construction process from the multi-view visual representation.

[0112] Subsequently, the vectors after the first interactive learning interact with the input 3D point cloud data through a cross-attention layer. The cross-attention mechanism enables the vectors after the first interactive learning to integrate into the overall information of the scene. Through the combined action of the self-attention and cross-attention mechanisms, the vectors after the first interactive learning can learn a unified representation that conforms to the directivity of the multi-view images and the 3D point cloud text instructions, capturing comprehensive and detailed information of the 3D scene. The comprehensive information enables the model to understand the scene from an overall perspective, while the detailed information helps to supplement ambiguous and missing information. The 3D perceptual scene embedding representation is projected into the input embedding space of the large language model through a linear layer.

[0113] This step effectively bridges the gap between the 3D point cloud modality and the text modality. It is achieved by leveraging the visual consistency between 3D point cloud features and multi-view features. Additionally, interactive generation is used to create meaningful 3D perceptual scene embedding representations that encapsulate comprehensive and detailed information about the scene. Thus, the 3D perceptual scene embedding representations can improve the accuracy of the large language model's understanding of 3D scenes and enhance the large language model's context-related understanding ability.

[0114] In this application, both S103 and S104 can be completed using a model. If a model is used, the model needs to be trained. This application adopts a three-stage training method to improve the 3D scene understanding performance.

[0115] The first stage: Use a large amount of training data to pre-train the model involved in S104. The training data covers tasks such as 3D visual question answering, scene description, embodied dialogue, and planning.

[0116] The second stage: Use the same dataset to pre-train the models involved in S103 and S104 respectively. These two pre-training stages enable the large language model to deeply understand 3D concepts and be proficient in handling various tasks.

[0117] The third stage: Use task-specific data to fine-tune the models involved in S103 and S104 to further enhance their ability to handle each specific task.

[0118] In all training stages, the structure of the model in this application remains unchanged. In the first stage, the model involved in S103 still provides view scene features, but its initial parameters are frozen. Additionally, standard cross-entropy loss is used to model language differences, which helps the model learn to generate context-appropriate and coherent responses. It should be noted that all parameters of the large language model backbone network are frozen during the training and fine-tuning stages. The large language model here can be selected from models such as FlanT5, Opt, Vicuna, etc.

[0119] This application adopts a novel method to interactively learn 3D learnable query vectors using view scene features and 3D point cloud data. This method allows information from different modalities (i.e., 2D visual features and 3D spatial information) to be fused at a deep level. This cross-modal interactive learning mechanism ensures that data from 2D images and 3D point clouds can fully complement each other, effectively solving the problem of insufficient fusion of different modality information in the prior art.

[0120] This application proposes a method of aggregating two-dimensional multi-view images and camera poses into view scene features, combining the detailed information of multi-view images with the spatial information of camera poses to generate view scene features containing rich scene details. It effectively solves the problems of lost three-dimensional point cloud reconstruction information and insufficient object details in existing three-dimensional scene understanding methods, significantly improves the representation ability of three-dimensional scenes, and provides a more accurate information basis for subsequent understanding and reasoning.

[0121] The present invention performs interactive learning on a set of learnable query vectors with three-dimensional point cloud data and view scene features. By using self-attention and cross-attention mechanisms, it effectively fuses information from different modalities to generate a unified feature representation that conforms to the directivity of text instructions, namely, a three-dimensional perception embedding representation. This design solves the problem of insufficient fusion of information from different modalities in existing methods, enables the model to better understand complex information in three-dimensional scenes, and improves the processing ability for three-dimensional scene understanding tasks.

[0122] During the training process of the present invention, there is no need to fine-tune the backbone network of the large language model. Through pre-training and task-specific fine-tuning, the model is optimized using the standard cross-entropy loss, which reduces training time and memory consumption while ensuring the performance of the model in three-dimensional scene understanding tasks.

[0123] Figure 4 It is a structural diagram of a three-dimensional point cloud scene understanding device provided by an embodiment of this application. As Figure 4 shown, based on the three-dimensional point cloud scene understanding method provided by the foregoing embodiment, this application also correspondingly provides a three-dimensional point cloud scene understanding device based on a large language model, including:

[0124] An acquisition module for acquiring the three-dimensional scene to be understood.

[0125] An acquisition model for acquiring multiple two-dimensional multi-view images of the three-dimensional scene to be understood.

[0126] A view scene feature determination module for obtaining view scene features based on the multiple two-dimensional multi-view images.

[0127] An interaction module for performing interactive learning on three-dimensional learnable query vectors using the view scene features and the three-dimensional point cloud data corresponding to the three-dimensional scene to be understood, to obtain a three-dimensional perception scene embedding representation.

[0128] An understanding result determination module for using the large language model to understand the three-dimensional perception scene embedding representation and generating an understanding result corresponding to the three-dimensional scene to be understood.

[0129] As an alternative embodiment, the view scene feature determination module includes a feature extraction unit, a camera pose determination unit, and a view scene determination unit. Now, the functions of each unit in the view scene feature determination module will be described in detail:

[0130] The feature extraction unit is configured to extract features from the multiple two-dimensional multi-view images to obtain visual features corresponding to each two-dimensional multi-view image.

[0131] The preprocessing unit is configured to perform alignment processing on the visual features corresponding to each two-dimensional multi-view image and the text instructions to obtain the aligned visual features corresponding to each two-dimensional multi-view image.

[0132] The camera pose determination unit is configured to determine the camera pose corresponding to each two-dimensional multi-view image.

[0133] The view scene determination unit is configured to obtain view scene features based on the aligned visual features and the camera poses corresponding to each two-dimensional multi-view image.

[0134] As an alternative embodiment, the view scene determination unit includes a transformation subunit, a scene feature determination subunit, and a view scene determination subunit. Now, the functions of each subunit in the view scene determination unit will be described:

[0135] The transformation subunit is configured to perform transformation processing on the camera pose corresponding to each two-dimensional multi-view image to obtain a position embedding representation corresponding to each two-dimensional multi-view image.

[0136] The scene feature determination subunit is configured to fuse the aligned visual features and the position embedding representations corresponding to each two-dimensional multi-view image to obtain scene features corresponding to each two-dimensional multi-view image.

[0137] The view scene determination subunit is configured to aggregate the scene features corresponding to the multiple two-dimensional multi-view images to obtain view scene features.

[0138] As an alternative embodiment, the interaction module includes

[0139] The self-attention layer unit with weight sharing is configured to perform interactive learning on the three-dimensional learnable query vector and the view scene features using the self-attention layer with weight sharing to obtain a vector after the first interactive learning.

[0140] The cross-attention layer unit is configured to perform interaction on the vector after the first interactive learning and the three-dimensional point cloud data corresponding to the three-dimensional scene to be understood using the cross-attention layer to obtain the three-dimensional perceptual scene embedding representation.

[0141] As an alternative embodiment, the acquisition module is specifically configured to:

[0142] Use a camera to collect multiple two-dimensional multi-view images of the three-dimensional scene to be understood.

[0143] An embodiment of the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement a three-dimensional point cloud scene understanding method based on a large language model.

[0144] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements a three-dimensional point cloud scene understanding method based on a large language model.

[0145] An embodiment of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements a three-dimensional point cloud scene understanding method based on a large language model.

[0146] It should be noted that the various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other, and the key points of each embodiment are the differences from other embodiments. In particular, for the device and apparatus embodiments, since they are basically similar to the method embodiments, they are described relatively simply, and the relevant parts can be referred to the partial description of the method embodiments. The device and apparatus embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0147] As described above, it is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A three-dimensional point cloud scene understanding method based on a large language model, characterized in that The three-dimensional point cloud scene understanding method based on a large language model includes: Obtain the three-dimensional scene to be understood; Collect multiple two-dimensional multi-view images of the three-dimensional scene to be understood; Obtain view scene features based on the multiple two-dimensional multi-view images; Perform interactive learning on the three-dimensional learnable query vector using the view scene features and the three-dimensional point cloud data corresponding to the three-dimensional scene to be understood, to obtain a three-dimensional perception scene embedding representation; Use a large language model to understand the three-dimensional perception scene embedding representation and generate an understanding result corresponding to the three-dimensional scene to be understood.

2. The method for three-dimensional point cloud scene understanding based on a large language model according to claim 1, wherein, The obtaining of view scene features based on the multiple two-dimensional multi-view images specifically includes: Extract features from the multiple two-dimensional multi-view images to obtain visual features corresponding to each two-dimensional multi-view image; Perform alignment processing on the visual features corresponding to each two-dimensional multi-view image and the text instructions to obtain the aligned visual features corresponding to each two-dimensional multi-view image; Determine the camera pose corresponding to each two-dimensional multi-view image; Obtain view scene features based on the aligned visual features and camera poses corresponding to each two-dimensional multi-view image.

3. The method for three-dimensional point cloud scene understanding based on a large language model according to claim 2, wherein, The obtaining of view scene features based on the aligned visual features and camera poses corresponding to each two-dimensional multi-view image specifically includes: Perform transformation processing on the camera poses corresponding to each two-dimensional multi-view image to obtain a position embedding representation corresponding to each two-dimensional multi-view image; Perform fusion processing on the aligned visual features and position embedding representations corresponding to each two-dimensional multi-view image to obtain a scene feature corresponding to each two-dimensional multi-view image; Perform aggregation processing on the scene features corresponding to the multiple two-dimensional multi-view images to obtain view scene features.

4. The three-dimensional point cloud scene understanding method based on a large language model according to claim 1, wherein, The performing of interactive learning on the three-dimensional learnable query vector using the view scene features and the three-dimensional point cloud data corresponding to the three-dimensional scene to be understood to obtain a three-dimensional perception scene embedding representation specifically includes: Perform interactive learning on the three-dimensional learnable query vector and the view scene features using a self-attention layer with shared weights to obtain a vector after the first interactive learning; Perform interaction on the vector after the first interactive learning and the three-dimensional point cloud data corresponding to the three-dimensional scene to be understood using a cross-attention layer to obtain the three-dimensional perception scene embedding representation.

5. The 3D point cloud scene understanding method based on the large language model according to claim 1, characterized in that The collecting of multiple two-dimensional multi-view images of the three-dimensional scene to be understood specifically includes: Use a camera to collect multiple two-dimensional multi-view images of the three-dimensional scene to be understood.

6. A three-dimensional point cloud scene understanding device based on a large language model, characterized in that, The three-dimensional point cloud scene understanding device based on a large language model includes: An obtaining module for obtaining the three-dimensional scene to be understood; A collecting model for collecting multiple two-dimensional multi-view images of the three-dimensional scene to be understood; A view scene feature determination module for obtaining view scene features based on the multiple two-dimensional multi-view images; An interaction module for performing interactive learning on the three-dimensional learnable query vector using the view scene features and the three-dimensional point cloud data corresponding to the three-dimensional scene to be understood to obtain a three-dimensional perception scene embedding representation; An understanding result determination module for using a large language model to understand the three-dimensional perception scene embedding representation and generate an understanding result corresponding to the three-dimensional scene to be understood.

7. The three-dimensional point cloud scene understanding device based on a large language model according to claim 6, wherein The view scene feature determination module includes: A feature extraction unit for extracting features from the multiple two-dimensional multi-view images to obtain visual features corresponding to each two-dimensional multi-view image; A preprocessing unit for aligning the visual features corresponding to each two-dimensional multi-view image with the text instructions to obtain the aligned visual features corresponding to each two-dimensional multi-view image; A camera pose determination unit for determining the camera pose corresponding to each two-dimensional multi-view image; A view scene determination unit for obtaining view scene features based on the aligned visual features and camera poses corresponding to each two-dimensional multi-view image.

8. A computer device, comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the large language model-based three-dimensional point cloud scene understanding method according to any one of claims 1-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the large language model-based three-dimensional point cloud scene understanding method according to any one of claims 1-5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the large language model-based three-dimensional point cloud scene understanding method according to any one of claims 1-5.

Images