A three-dimensional point cloud scene understanding method based on a large language model and related products

Abstract

The application discloses a three-dimensional point cloud scene understanding method based on a large language model and related products, and the method comprises the following steps: obtaining a three-dimensional scene to be understood; collecting multiple two-dimensional multi-view images of the three-dimensional scene to be understood; obtaining view scene features based on the multiple two-dimensional multi-view images; performing interactive learning on a three-dimensional learnable query vector by using the view scene features and three-dimensional point cloud data corresponding to the three-dimensional scene to be understood, so as to obtain a three-dimensional perception scene embedding representation; and understanding the three-dimensional perception scene embedding representation by using a large language model, and generating an understanding result corresponding to the three-dimensional scene to be understood. The application improves the understanding accuracy of the large language model by using the view scene features. The three-dimensional learnable query vector is subjected to interactive learning by using the view scene features and the three-dimensional point cloud data, so that the three-dimensional perception scene embedding representation fuses information from different modalities, and the large language model can more accurately understand complex information in the three-dimensional scene.

Description

Technical Field

[0001] This application relates to the field of 3D scene understanding technology, and in particular to a 3D point cloud scene understanding method and related products based on a large language model. Background Technology

[0002] With the rapid development of natural language processing and computer vision technologies, scene understanding has become a new hot topic. Models like CLIP and Uni3D have explored alignment methods for multimodal data. Given the significant advantages of large language models in processing natural language, researchers are exploring how to leverage ordinary large language models to solve a wide range of multimodal tasks. Two-dimensional vision-language tasks have received considerable attention, attempting to endow large language models with the ability to understand two-dimensional images. Mainstream two-dimensional vision-language model techniques can be divided into two paradigms: one attempts to train the model with a large amount of text-image instruction-adjusted data (such as CLIP and LLaVA), while the other focuses on connecting the pre-trained visual encoder and the large language model through additional learnable modules (such as the perceptron in Flamingo and the Q-Former in BLIP-2), aligning the latent space of visual embeddings with the text embeddings of the large language model.

[0003] Similar to 2D vision-language, researchers have also introduced large language models to understand the 3D world. Existing methods aim to align 3D modalities with text modalities to construct 3D multimodal large language models. This alignment enables large language models to understand 3D visual representations, laying the foundation for solving downstream tasks in 3D scenes. Integrating 3D representations into large language models typically requires adding special structures that can effectively process spatial information. 3D Data with Large Language Models (3DLLM) pioneered the injection of 3D spatial information into large language models, marking a significant advancement in the field. LL3DA introduces visual cues based on the Q-Former architecture, opening new opportunities for large language models to solve tasks in the 3D world. By applying 3D cues, 3DMIT eliminates the alignment phase between 3D scenes and natural language, but involves a fine-tuning phase for the large language model.

[0004] Although some research has emerged on 3D multimodal large language models, methods for 3D scene understanding based on large language models are still not fully mature. 3D point clouds depict a scene from a holistic perspective, providing structural and spatial information about the 3D world. Therefore, current methods heavily rely on 3D point clouds for scene understanding.

[0005] However, the reconstruction of 3D point clouds from indoor scenes often results in information loss. One problem is that regions with textureless planes or repetitive patterns (such as uniform walls, homogeneous floor tiles, and glass surfaces) lack unique features in the captured images. This deficiency severely hinders the effectiveness of stereo matching algorithms, which rely on establishing robust correspondences between image pairs. Consequently, these homogeneous regions are easily missed, appearing as holes in the reconstructed 3D point cloud.

[0006] Another issue is that misalignment between the captured image and the densely reconstructed point cloud can easily lead to detail distortion when dealing with structurally complex objects. This distortion hinders the complete preservation of detailed information such as surface texture, boundary contours, and microscopic features. These elements, which are often crucial for comprehensive scene representation, may appear blurry or indistinguishable in the reconstructed 3D point cloud. The overall decline in the quality and utility of the reconstructed 3D point cloud weakens the ability to represent 3D scenes and may lead to a significant performance degradation of existing methods.

[0007] Furthermore, some 3D scene understanding methods based on large language models have failed to fully realize the effective fusion of multimodal information in 3D scenes, making it difficult to generate unified feature representations that conform to the directionality of text instructions. They still require segmentation of objects in 3D scenes and fine-tuning of the backbone network of large language models, which significantly consumes processing time and memory. Summary of the Invention

[0008] To address the aforementioned issues, this application provides a method and related products for understanding 3D point cloud scenes based on a large language model, with the aim of improving the accuracy of understanding 3D point cloud scenes using a large language model.

[0009] The embodiments of this application disclose the following technical solutions:

[0010] The first aspect of this application provides a method for understanding 3D point cloud scenes based on a large language model, including:

[0011] Obtain the 3D scene to be understood;

[0012] Acquire multiple 2D, multi-view images of the 3D scene to be understood;

[0013] View scene features are obtained based on the multiple two-dimensional, multi-view images;

[0014] The 3D learnable query vector is interactively learned using the view scene features and the 3D point cloud data corresponding to the 3D scene to be understood, to obtain the 3D perceptual scene embedding representation.

[0015] We utilize large language models to understand the embedded representation of 3D perceived scenes and generate the understanding results corresponding to the 3D scenes to be understood.

[0016] Optionally, obtaining the view scene features based on the multiple two-dimensional multi-view images specifically includes:

[0017] Feature extraction is performed on the multiple two-dimensional multi-view images to obtain the visual features corresponding to each two-dimensional multi-view image;

[0018] Align the visual features and text instructions corresponding to each two-dimensional multi-view image to obtain the aligned visual features corresponding to each two-dimensional multi-view image.

[0019] Determine the camera pose for each 2D multi-view image;

[0020] Based on the aligned visual features and camera pose corresponding to each 2D multi-view image, the view scene features are obtained.

[0021] Optionally, obtaining view scene features based on the aligned visual features and camera pose corresponding to each two-dimensional multi-view image specifically includes:

[0022] The camera pose of each 2D multi-view image is transformed to obtain the position embedding representation of each 2D multi-view image.

[0023] The aligned visual features and position embedding representations corresponding to each 2D multi-view image are fused to obtain the scene features corresponding to each 2D multi-view image.

[0024] The scene features corresponding to multiple two-dimensional multi-view images are aggregated to obtain view scene features.

[0025] Optionally, the step of interactively learning the 3D learnable query vector using the view scene features and the 3D point cloud data corresponding to the 3D scene to be understood to obtain a 3D perceptual scene embedding representation specifically includes:

[0026] The three-dimensional learnable query vector and the view scene features are interactively learned using a weight-sharing self-attention layer, resulting in the vector after the first interactive learning.

[0027] By using a cross-attention layer to interact with the vectors learned from the first interaction and the 3D point cloud data corresponding to the 3D scene to be understood, the 3D perception scene embedding representation is obtained.

[0028] Optionally, the acquisition of multiple two-dimensional, multi-view images of the three-dimensional scene to be understood specifically includes:

[0029] Multiple two-dimensional, multi-view images of the three-dimensional scene to be understood are captured using a camera.

[0030] The second aspect of this application provides a 3D point cloud scene understanding device based on a large language model, comprising:

[0031] The acquisition module is used to acquire the 3D scene to be understood;

[0032] The acquisition model is used to acquire multiple two-dimensional, multi-view images of the three-dimensional scene to be understood;

[0033] The view scene feature determination module is used to obtain view scene features based on the multiple two-dimensional multi-view images;

[0034] The interaction module is used to interactively learn the 3D learnable query vector by utilizing the view scene features and the 3D point cloud data corresponding to the 3D scene to be understood, so as to obtain the 3D perception scene embedding representation.

[0035] The understanding result determination module is used to understand the embedded representation of the 3D perceived scene using a large language model and generate the understanding result corresponding to the 3D scene to be understood.

[0036] Optionally, the view scene feature determination module includes:

[0037] The feature extraction unit is used to extract features from the multiple two-dimensional multi-view images to obtain the visual features corresponding to each two-dimensional multi-view image.

[0038] The preprocessing unit is used to align the visual features and text instructions corresponding to each two-dimensional multi-view image to obtain the aligned visual features corresponding to each two-dimensional multi-view image.

[0039] The camera pose determination unit is used to determine the camera pose corresponding to each two-dimensional multi-view image.

[0040] The view scene determination unit is used to obtain view scene features based on the aligned visual features and camera pose corresponding to each two-dimensional multi-view image.

[0041] A third aspect of this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the three-dimensional point cloud scene understanding method based on a large language model provided in the first aspect.

[0042] The fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the three-dimensional point cloud scene understanding method based on a large language model provided in the first aspect.

[0043] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the three-dimensional point cloud scene understanding method based on a large language model provided in the first aspect.

[0044] Compared with the prior art, this application has the following advantages:

[0045] This application acquires a 3D scene to be understood; collects multiple 2D multi-view images of the 3D scene to be understood; obtains view scene features based on the multiple 2D multi-view images; uses the view scene features and the corresponding 3D point cloud data of the 3D scene to be understood to interactively learn 3D learnable query vectors, obtaining a 3D perceptual scene embedding representation; and uses a large language model to understand the 3D perceptual scene embedding representation and generates the understanding result corresponding to the 3D scene to be understood. This application first uses view scene features to solve the problems of missing 3D point cloud data and insufficient object details in existing 3D scene understanding schemes, significantly improving the accuracy of the 3D perceptual scene embedding representation, thereby improving the understanding accuracy of the large language model. Furthermore, this application uses view scene features and 3D point cloud data to interactively learn 3D learnable query vectors, enabling the 3D perceptual scene embedding representation to integrate information from different modalities, effectively solving the problem of insufficient fusion of different modal information in existing technologies, allowing the large language model to more accurately understand the complex information in the 3D scene. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A flowchart illustrating a 3D point cloud scene understanding method based on a large language model, provided for embodiments of this application;

[0048] Figure 2 A first flowchart for determining view scene features provided in an embodiment of this application;

[0049] Figure 3 A second flowchart for determining view scene features provided in an embodiment of this application;

[0050] Figure 4 This is a structural diagram of a 3D point cloud scene understanding device based on a large language model, provided in an embodiment of this application. Detailed Implementation

[0051] As described above, existing 3D scene understanding solutions suffer from problems such as missing 3D point cloud data, insufficient object details, and inadequate fusion of same-modal information.

[0052] In view of the above problems, this application provides a method and related products for understanding three-dimensional point cloud scenes based on a large language model. The method includes: acquiring a three-dimensional scene to be understood; acquiring multiple two-dimensional multi-view images of the three-dimensional scene to be understood; obtaining view scene features based on the multiple two-dimensional multi-view images; using the view scene features and the three-dimensional point cloud data corresponding to the three-dimensional scene to be understood to interactively learn three-dimensional learnable query vectors to obtain a three-dimensional perceptual scene embedding representation; using a large language model to understand the three-dimensional perceptual scene embedding representation and generating the understanding result corresponding to the three-dimensional scene to be understood.

[0053] This application addresses the problem that existing methods heavily rely on 3D point cloud representations, leading to an inability to provide accurate answers when faced with information loss during 3D point cloud reconstruction. It also solves the problem that existing methods ignore the detailed information provided by 2D multi-view images and their visual consistency with 3D point clouds, thus limiting the understanding of complex object details in 3D scenes by large language models.

[0054] This application addresses the problem that existing methods fail to fully integrate multimodal information in 3D scenes and struggle to generate unified feature representations that conform to the directionality of text commands by combining view scene features and 3D point cloud data. This significantly improves the accuracy of 3D perception scene embedding representation and the understanding ability of large language models.

[0055] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0056] Figure 1 A flowchart illustrating a 3D point cloud scene understanding method based on a large language model, as provided in this application embodiment, is shown below. Figure 1 The method for understanding 3D point cloud scenes based on a large language model, as shown, includes:

[0057] S101: Obtain the 3D scene to be understood.

[0058] This application does not limit the 3D scene. The main purpose is to clarify the target scene and lay the foundation for subsequent data acquisition and processing.

[0059] S102: Acquire multiple 2D multi-view images of the 3D scene to be understood.

[0060] This application does not limit the devices used to acquire two-dimensional multi-view images, such as cameras, mobile phones, digital cameras, etc.

[0061] As an alternative embodiment, a camera is used to acquire multiple two-dimensional, multi-view images of the three-dimensional scene to be understood.

[0062] S102 primarily provides two-dimensional learning material for subsequent 3D learnable query vectors. By capturing the 3D scene from different angles, it provides rich visual information, which helps to comprehensively understand the details of the 3D scene. This allows the 3D learnable query vectors to fully learn the 3D scene to be understood, further ensuring that the large language model fully understands the 3D scene.

[0063] S103: Obtain view scene features based on the multiple two-dimensional multi-view images.

[0064] This application does not impose specific limitations on the method for determining view scene features, as long as the view scene features contain rich scene details.

[0065] Since two-dimensional multi-view images generally display objects visually and spatially, this application provides an optional embodiment to extract visual features from each two-dimensional multi-view image to obtain the visual features corresponding to each two-dimensional multi-view image.

[0066] The visual features are the visual semantic features of two-dimensional multi-view images.

[0067] Spatial features are extracted from each two-dimensional multi-view image to obtain the spatial features corresponding to each two-dimensional multi-view image.

[0068] This application does not limit the spatial extraction method. For example, it can determine the camera pose of each two-dimensional multi-view image and determine the spatial features corresponding to each two-dimensional multi-view image based on the camera pose of each two-dimensional multi-view image.

[0069] This application does not limit the method of processing camera pose, such as using Flatten+MLP, or extracting the Plucker features of the camera pose corresponding to each two-dimensional multi-view image.

[0070] The visual and spatial features corresponding to each two-dimensional multi-view image are fused to obtain the view scene features.

[0071] Because the visual and spatial features corresponding to each two-dimensional multi-view image can only represent information about the three-dimensional scene involved in that image, they have certain limitations. Therefore, the visual and spatial features corresponding to each two-dimensional multi-view image can be fused to obtain more comprehensive information about the three-dimensional scene.

[0072] The above embodiments only illustrate the method for determining view scene features in a more specific way, and do not imply that the method for determining view scene features in S103 is limited to the above one. The method for determining view scene features can be limited according to the actual situation. Utilizing view scene features extracted from multiple two-dimensional, multi-view images supplements the shortcomings of 3D point cloud data in describing the details of complex objects. This combination not only compensates for the information gaps that may exist in traditional 3D point cloud data, but also increases the capture of detailed features such as object surface texture and color. In this way, the generated 3D perceptual scene embedding representation is more accurate and comprehensive, thereby improving the accuracy of subsequent large language models in understanding 3D scenes.

[0073] S104: Using the view scene features and the 3D point cloud data corresponding to the 3D scene to be understood, the 3D learnable query vector is interactively learned to obtain the 3D perception scene embedding representation.

[0074] This application does not limit the methods of interactive learning. For example, it may utilize interactive models to interactively learn 3D learnable query vectors to obtain 3D perceptual scene embedding representations. Interactive models include, but are not limited to, PerceiverResampler, Q-Former, and other models.

[0075] As an optional embodiment, based on the Q-Former model, the 3D learnable query vector is interactively learned using the view scene features and the 3D point cloud data corresponding to the 3D scene to be understood, to obtain a 3D perceptual scene embedding representation, specifically including:

[0076] By utilizing self-attention or multi-head attention mechanisms, the 3D learnable query vectors are interactively learned based on the view scene features and the 3D point cloud data corresponding to the 3D scene to be understood, thereby obtaining the 3D perception scene embedding representation.

[0077] S105: Use a large language model to understand the embedded representation of a 3D perceived scene and generate the understanding results corresponding to the 3D scene to be understood.

[0078] This application directly utilizes a large language model to understand the embedding representation of 3D perceptual scenes, eliminating the need for fine-tuning the efficient training strategy of the large language model backbone network. This application employs a specific process, including S103 obtaining view scene features based on multiple 2D multi-view images, and S104 using the view scene features and the 3D point cloud data corresponding to the 3D scene to be understood to interactively learn the 3D learnable query vector, thereby obtaining the 3D perceptual scene embedding representation. This eliminates the need for fine-tuning the large language model backbone network, ensuring the model's performance in 3D scene understanding tasks.

[0079] This application can effectively solve the problems in the existing technology, such as the inability to handle information loss during the reconstruction of 3D point clouds and insufficient understanding of the details of complex objects.

[0080] The above describes the main technical solution of this application. Further implementations of the main technical solution are now introduced. Details are as follows:

[0081] Unlike 3D point clouds, which tend to depict the overall scene, 2D multi-view images can provide more detailed information about scene components. Specifically, this application designs a novel method for aggregating multi-view images for subsequent enhancement, including extracting multi-view features with spatial information and then aggregating these features to obtain view scene features. That is, in S103, view scene features are obtained based on the multiple 2D multi-view images. This application provides an optional embodiment:

[0082] Figure 2 A first flowchart for determining view scene features provided in an embodiment of this application, such as... Figure 2 As shown, S201: Feature extraction is performed on the multiple two-dimensional multi-view images to obtain the visual features corresponding to each two-dimensional multi-view image.

[0083] This application does not limit the method of feature extraction, for example:

[0084] All 2D multi-view images of the 3D scene to be understood Image encoders trained by pretraining Processing is performed to obtain the corresponding two-dimensional multi-view images. Where n represents a two-dimensional multi-view image. The pre-trained image encoders here include, but are not limited to, various models such as ViT, ResNet, and DenseNet. Taking ViT as an example, for the i-th two-dimensional multi-view image I... i The visual features f corresponding to the i-th two-dimensional multi-view image are obtained as follows: i :

[0085] f i =ViT(I i ).

[0086] S202: Align the visual features and text instructions corresponding to each two-dimensional multi-view image to obtain the aligned visual features corresponding to each two-dimensional multi-view image.

[0087] This application does not limit the method of alignment processing; as one optional embodiment:

[0088] A 2D image-to-text alignment module is applied to derive the visual features associated with text commands. The 2D image-to-text alignment module can use models such as Q-Former, Oscar, or Uniter. Taking Q-Former as an example, the aligned visual features q corresponding to the i-th 2D multi-view image are... i It can be obtained in the following ways:

[0089] q i =Q-Former(f i ).

[0090] S203: Determine the camera pose corresponding to each 2D multi-view image.

[0091] Since the visual features corresponding to each two-dimensional multi-view image extracted above lack spatial information, this application also needs to determine the spatial features of each two-dimensional multi-view image based on the camera pose corresponding to each two-dimensional multi-view image.

[0092] S204: Based on the aligned visual features and camera pose corresponding to each two-dimensional multi-view image, the view scene features are obtained.

[0093] This application does not limit the method for determining visual scene features; as one optional embodiment:

[0094] Figure 3 A second flowchart for determining view scene features provided in embodiments of this application, such as... Figure 3 As shown, S301: The camera pose corresponding to each two-dimensional multi-view image is transformed to obtain the position embedding representation corresponding to each two-dimensional multi-view image.

[0095] Essentially, each 2D multi-view image has a camera pose, which refers to the position and orientation of the 2D multi-view image relative to the 3D scene, thus indirectly reflecting some spatial information. Therefore, this application uses a flatten layer, followed by a two-layer multilayer perceptron (MLP), to process the camera pose p corresponding to the i-th 2D multi-view image. i Converted into a location embedding representation PosEmb for each 2D multi-view image i :

[0096] PosEmb i =MLP(Flatten(p i )).

[0097] S302: The aligned visual features and position embedding representations corresponding to each two-dimensional multi-view image are fused to obtain the scene features corresponding to each two-dimensional multi-view image.

[0098] Through the above transformation, spatial information is directly encoded into location embedding representations. These location embedding representations are then added to the scene features, ensuring that the spatial context of each 2D multi-view image is preserved in the scene features corresponding to each 2D multi-view image.

[0099] Taking the i-th two-dimensional multi-view image as an example, the scene features F corresponding to the i-th two-dimensional multi-view image are... i view :

[0100] F i view =q i +PosEmb i .

[0101] S303: Aggregate the scene features corresponding to multiple two-dimensional multi-view images to obtain view scene features.

[0102] This application does not limit the method of aggregation; as one optional embodiment:

[0103] The scene features corresponding to multiple two-dimensional, multi-view images are aggregated into view scene features. Because F i view The images represent only a portion of the 3D scene to be understood. Therefore, this application explores using multiple stacked Transformer layers to fuse scene features from multiple 2D multi-view images into view scene features.

[0104] Specifically, after performing the self-attention operation, a multi-head cross-attention (MHCA) mechanism is adopted to enable attention interaction between scene features corresponding to multiple two-dimensional multi-view images and a set of learnable query vectors Q.

[0105] The aggregation process of view scene features can be described as follows:

[0106]

[0107] In this attention mechanism, q, k, and v represent the query, key, and value, respectively, and Proj() represents a simple linear layer. This mechanism enables attentional interactions between scene features in different views, effectively integrating spatial contextual information from multiple perspectives. It integrates spatial information into scene features and fuses these features into view scene features, thereby providing detailed information about the scene.

[0108] Considering that the constructed 3D point cloud features may suffer from information loss, and that the view scene features contain detailed information, this application extracts comprehensive and detailed clues through the interaction between 3D point cloud features and view scene features. The implementation methods include, but are not limited to, models such as Perceiver Resampler and Q-Former. For example, in S104, the view scene features and the 3D point cloud data corresponding to the 3D scene to be understood are used to interactively learn the 3D learnable query vector to obtain a 3D perceptual scene embedding representation. This application provides an optional embodiment:

[0109] The three-dimensional learnable query vector and the view scene features are interactively learned using a weight-sharing self-attention layer, resulting in the vector after the first interactive learning.

[0110] By using a cross-attention layer to interact with the vectors learned from the first interaction and the 3D point cloud data corresponding to the 3D scene to be understood, the 3D perception scene embedding representation is obtained.

[0111] Specifically, S104 contains a set of trainable vectors, namely 3D learnable query vectors, and utilizes the model to perform a two-step interaction process on these 3D learnable query vectors. First, the 3D learnable query vectors interact with the view scene features through a weight-shared self-attention layer. The self-attention mechanism ensures that the 3D learnable query vectors acquire rich and detailed visual contextual information, which is crucial for disambiguation and compensating for information loss caused by alignment issues. This effectively helps the 3D learnable query vectors capture detailed information that might be lost during point cloud construction from multi-view visual representations.

[0112] Subsequently, the vectors learned through the first interaction interact with the input 3D point cloud data through a cross-attention layer. The cross-attention mechanism allows the vectors to incorporate overall scene information. Through the combined effect of self-attention and cross-attention mechanisms, the vectors learned through the first interaction learn a unified representation that conforms to the directionality of multi-view images and 3D point cloud text instructions, capturing comprehensive and detailed information about the 3D scene. Comprehensive information allows the model to understand the scene from a holistic perspective, while detailed information helps to fill in ambiguous and missing information. The 3D-perceptual scene embedding representation is projected onto the input embedding space of a large language model through a linear layer.

[0113] This step effectively bridges the gap between the 3D point cloud modality and the text modality. It leverages the visual consistency between 3D point cloud features and multi-view features. Furthermore, meaningful 3D-aware scene embedding representations are generated through interaction, encapsulating comprehensive and detailed information about the scene. Therefore, 3D-aware scene embedding representations can improve the accuracy of large language models' understanding of 3D scenes and enhance their ability to understand context-sensitive information.

[0114] In this application, S103 and S104 can both be completed using a model. If a model is used, it needs to be trained. This application adopts a three-stage training method to improve the performance of 3D scene understanding.

[0115] Phase 1: Pre-training the models involved in S104 using a large amount of training data, which covers tasks such as 3D visual question answering, scene description, embodied dialogue, and planning.

[0116] The second stage involves pre-training the models involved in S103 and S104 using the same dataset. These two pre-training stages enable the large language model to deeply understand three-dimensional concepts, allowing it to skillfully handle various tasks.

[0117] Phase 3: Fine-tune the models involved in S103 and S104 using data from specific tasks to further improve their ability to handle each specific task.

[0118] Throughout all training phases, the model structure remains unchanged. In the first phase, the model involved in S103 still provides view-scene features, but its initialization parameters are frozen. Furthermore, standard cross-entropy loss is used to model language differences, which helps the model learn to generate context-appropriate and coherent responses. Notably, all parameters of the large language model backbone network are frozen during both the training and fine-tuning phases. The large language model used here can be FlanT5, Opt, Vicuna, or similar models.

[0119] This application employs a novel method to interactively learn 3D learnable query vectors using view scene features and 3D point cloud data. This method allows for deep fusion of information from different modalities (i.e., 2D visual features and 3D spatial information). This cross-modal interactive learning mechanism ensures that data from 2D images and 3D point clouds are fully complementary, effectively solving the problem of insufficient fusion of information from different modalities in existing technologies.

[0120] This application proposes a method for aggregating two-dimensional multi-view images and camera pose into view scene features. By combining detailed information from the multi-view images with spatial information from the camera pose, view scene features rich in scene details are generated. This effectively solves the problems of information loss and insufficient object detail in existing 3D scene understanding methods, significantly improving the representation ability of 3D scenes and providing a more accurate information foundation for subsequent understanding and reasoning.

[0121] This invention employs a set of learnable query vectors to interactively learn from 3D point cloud data and view scene features. Utilizing self-attention and cross-attention mechanisms, it effectively fuses information from different modalities to generate a unified feature representation that conforms to the directionality of text instructions—that is, an embedded representation for 3D perception. This design addresses the problem of insufficient fusion of information from different modalities in existing methods, enabling the model to better understand complex information in 3D scenes and improving its ability to handle 3D scene understanding tasks.

[0122] This invention eliminates the need for fine-tuning the backbone network of a large language model during training. By pre-training and task-specific fine-tuning, and optimizing the model using standard cross-entropy loss, it reduces training time and memory consumption while ensuring the model's performance in 3D scene understanding tasks.

[0123] Figure 4 A structural diagram of a 3D point cloud scene understanding device based on a large language model provided in this application embodiment is shown below. Figure 4 As shown, based on the 3D point cloud scene understanding method based on a large language model provided in the preceding embodiments, this application also provides a 3D point cloud scene understanding device based on a large language model, including:

[0124] The acquisition module is used to acquire the 3D scene to be understood.

[0125] The acquisition model is used to acquire multiple 2D, multi-view images of the 3D scene to be understood.

[0126] The view scene feature determination module is used to obtain view scene features based on the multiple two-dimensional multi-view images.

[0127] The interaction module is used to interactively learn the 3D learnable query vector by utilizing the view scene features and the 3D point cloud data corresponding to the 3D scene to be understood, so as to obtain the 3D perception scene embedding representation.

[0128] The understanding result determination module is used to understand the embedded representation of the 3D perceived scene using a large language model and generate the understanding result corresponding to the 3D scene to be understood.

[0129] As an optional embodiment, the view scene feature determination module includes a feature extraction unit, a camera pose determination unit, and a view scene determination unit. The function of each unit in the view scene feature determination module is now described in detail:

[0130] The feature extraction unit is used to extract features from the multiple two-dimensional multi-view images to obtain the visual features corresponding to each two-dimensional multi-view image.

[0131] The preprocessing unit is used to align the visual features and text instructions corresponding to each two-dimensional multi-view image to obtain the aligned visual features corresponding to each two-dimensional multi-view image.

[0132] The camera pose determination unit is used to determine the camera pose corresponding to each two-dimensional multi-view image.

[0133] The view scene determination unit is used to obtain view scene features based on the aligned visual features and camera pose corresponding to each two-dimensional multi-view image.

[0134] As an optional embodiment, the view scene determination unit includes a transformation subunit, a scene feature determination subunit, and a view scene determination subunit. The function of each subunit in the view scene determination unit is now described:

[0135] The transformation subunit is used to transform the camera pose corresponding to each 2D multi-view image to obtain the position embedding representation corresponding to each 2D multi-view image.

[0136] The scene feature determination subunit is used to fuse the aligned visual features and position embedding representations corresponding to each two-dimensional multi-view image to obtain the scene features corresponding to each two-dimensional multi-view image.

[0137] The view scene determination subunit is used to aggregate scene features corresponding to multiple two-dimensional multi-view images to obtain view scene features.

[0138] As an optional embodiment, the interaction module includes

[0139] The weight-shared self-attention layer unit is used to interactively learn the 3D learnable query vector and the view scene features using the weight-shared self-attention layer, resulting in the vector after the first interactive learning.

[0140] The cross-attention layer unit is used to interact with the vector learned after the first interaction and the 3D point cloud data corresponding to the 3D scene to be understood, so as to obtain the 3D perception scene embedding representation.

[0141] As an optional embodiment, the acquisition module is specifically used for:

[0142] Multiple two-dimensional, multi-view images of the three-dimensional scene to be understood are captured using a camera.

[0143] This application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement a method for understanding three-dimensional point cloud scenes based on a large language model.

[0144] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for understanding 3D point cloud scenes based on a large language model.

[0145] This application provides a computer program product, including a computer program that, when executed by a processor, implements a method for understanding 3D point cloud scenes based on a large language model.

[0146] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and apparatus embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0147] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for understanding 3D point cloud scenes based on a large language model, characterized in that, The method for understanding 3D point cloud scenes based on a large language model includes: Obtain the 3D scene to be understood; Acquire multiple 2D, multi-view images of the 3D scene to be understood; View scene features are obtained based on the multiple two-dimensional, multi-view images; The 3D learnable query vector is interactively learned using the view scene features and the 3D point cloud data corresponding to the 3D scene to be understood, to obtain the 3D perceptual scene embedding representation. We utilize large language models to understand the embedded representation of 3D perceived scenes and generate the understanding results corresponding to the 3D scenes to be understood. The step of interactively learning the 3D learnable query vector using the view scene features and the 3D point cloud data corresponding to the 3D scene to be understood to obtain a 3D perceptual scene embedding representation specifically includes: The self-attention layer with weight sharing is used to interactively learn the 3D learnable query vector and the view scene features to obtain the vector after the first interactive learning. By using a cross-attention layer to interact with the vectors learned from the first interaction and the 3D point cloud data corresponding to the 3D scene to be understood, the 3D perception scene embedding representation is obtained.

2. The method for understanding 3D point cloud scenes based on a large language model according to claim 1, characterized in that, The process of obtaining view scene features based on the multiple two-dimensional multi-view images specifically includes: Feature extraction is performed on the multiple two-dimensional multi-view images to obtain the visual features corresponding to each two-dimensional multi-view image; Align the visual features and text instructions corresponding to each two-dimensional multi-view image to obtain the aligned visual features corresponding to each two-dimensional multi-view image. Determine the camera pose for each 2D multi-view image; Based on the aligned visual features and camera pose corresponding to each 2D multi-view image, the view scene features are obtained.

3. The method for understanding 3D point cloud scenes based on a large language model according to claim 2, characterized in that, The view scene features obtained based on the aligned visual features and camera pose corresponding to each two-dimensional multi-view image specifically include: The camera pose of each 2D multi-view image is transformed to obtain the position embedding representation of each 2D multi-view image. The aligned visual features and position embedding representations corresponding to each 2D multi-view image are fused to obtain the scene features corresponding to each 2D multi-view image. The scene features corresponding to multiple two-dimensional multi-view images are aggregated to obtain view scene features.

4. The method for understanding 3D point cloud scenes based on a large language model according to claim 1, characterized in that, The acquisition of multiple two-dimensional, multi-view images of the three-dimensional scene to be understood specifically includes: Multiple two-dimensional, multi-view images of the three-dimensional scene to be understood are captured using a camera.

5. A 3D point cloud scene understanding device based on a large language model, characterized in that, The 3D point cloud scene understanding device based on a large language model includes: The acquisition module is used to acquire the 3D scene to be understood; The acquisition model is used to acquire multiple two-dimensional, multi-view images of the three-dimensional scene to be understood; The view scene feature determination module is used to obtain view scene features based on the multiple two-dimensional multi-view images; The interaction module is used to interactively learn the 3D learnable query vector by utilizing the view scene features and the 3D point cloud data corresponding to the 3D scene to be understood, so as to obtain the 3D perception scene embedding representation. The understanding result determination module is used to understand the embedded representation of the 3D perceived scene using a large language model and generate the understanding result corresponding to the 3D scene to be understood. The interactive modules include: The weight-shared self-attention layer unit is used to interactively learn the 3D learnable query vector and the view scene features using the weight-shared self-attention layer to obtain the vector after the first interactive learning. The cross-attention layer unit is used to interact with the vector learned after the first interaction and the 3D point cloud data corresponding to the 3D scene to be understood, so as to obtain the 3D perception scene embedding representation.

6. The 3D point cloud scene understanding device based on a large language model according to claim 5, characterized in that, The view scene feature determination module includes: The feature extraction unit is used to extract features from the multiple two-dimensional multi-view images to obtain the visual features corresponding to each two-dimensional multi-view image. The preprocessing unit is used to align the visual features and text instructions corresponding to each two-dimensional multi-view image to obtain the aligned visual features corresponding to each two-dimensional multi-view image. The camera pose determination unit is used to determine the camera pose corresponding to each two-dimensional multi-view image. The view scene determination unit is used to obtain view scene features based on the aligned visual features and camera pose corresponding to each two-dimensional multi-view image.

7. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the three-dimensional point cloud scene understanding method based on a large language model as described in any one of claims 1-4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the three-dimensional point cloud scene understanding method based on a large language model as described in any one of claims 1-4.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the three-dimensional point cloud scene understanding method based on a large language model as described in any one of claims 1-4.

Images