Three-dimensional scene understanding method based on multi-modal large model

Abstract

The invention relates to the technical field of multi-modal large models, solves the technical problem that a traditional large language model does not have the capability of processing multi-aspect input information, and particularly relates to a three-dimensional scene understanding method based on a multi-modal large model, and the method comprises the steps: building a data set containing a point cloud-image-text pair; inputting the data set into a coding module to extract point cloud features, image features and text features; establishing a mask object reconstruction task, a mask text reconstruction task, a point cloud-image alignment task and a text-image alignment task to pre-train the coding module; and performing joint fine tuning on the pre-trained coding module and the voice-text multi-modal large model LLASM on a target task. According to the invention, the pre-trained multi-modal coding module is combined with the large language model, various visual language tasks on 3D and 2D can be completed, text and voice input is supported at the same time, and a universal, comprehensive and robust multi-modal large model is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal large models, and in particular to a three-dimensional scene understanding method based on a multimodal large model. Background Art

[0002] In recent years, the powerful logical reasoning and semantic understanding capabilities demonstrated by large language models (LLMs) have attracted the attention of both academia and industry. Some current work explores combining images with large language models to form a new generation of multimodal large language models that can understand and reason about the two-dimensional world. Although these models are already very powerful in understanding two-dimensional images, they are not trained based on the real three-dimensional physical world, and there is still a gap between them and true strong artificial intelligence.

[0003] The real three-dimensional physical world involves richer concepts, such as spatial relationships, textures, physical properties, and interactions. Therefore, some works propose to inject information about the three-dimensional world into large language models, leading to a new series of 3D-LLMs, which take three-dimensional representations as input and perform a series of three-dimensional related tasks, such as 3D question answering, 3D situational reasoning, and visual language navigation. However, a powerful intelligent agent needs to be able to process multiple input information, such as images, text, speech, point clouds, etc., to better understand the scene and realize a robust, comprehensive, and general multimodal large model. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a three-dimensional scene understanding method based on a multimodal large model, which solves the technical problem that traditional large language models are not capable of processing multi-faceted input information.

[0005] To solve the above technical problems, the present invention provides the following technical solutions: a three-dimensional scene understanding method based on a multimodal large model, the method comprising the following steps: S1. Create a dataset containing point cloud-image-text pairs ,in, Point cloud data for indoor scenes. For indoor scene point cloud data The corresponding projected image, For text description; S2. Dataset Extract point cloud features in the input encoding module , Image features and text features ,The encoding module includes a point cloud encoder, an image encoder, and a text encoder; S3, based on point cloud features , Image features and text features , establish mask object reconstruction task, mask text reconstruction task, point cloud-image alignment task and text-image alignment task to pre-train the encoding module; S4. Jointly fine-tune the pre-trained encoding module and the speech-text multimodal large model LLASM on the target task.

[0006] Furthermore, in step S2, the specific process includes the following steps: S21. Indoor scene point cloud data Input to the point cloud encoder to extract point cloud features ; S22, projecting the image Send it to the pre-trained image encoder to obtain the initial image features; S23, flatten the last two dimensions of the initial image features and send the result to the four-layer Transformer block to generate image features ; S24. Describe the text Input the pre-trained BERT Tokenizer to generate word embeddings; S25: Send word embeddings into the four-layer Transformer block to generate text features .

[0007] Furthermore, in step S21, the specific process includes the following steps: S211, use the 3D segmentation mask labels in the ScanNet dataset for indoor scene point cloud data Cutting is performed to obtain a series of point clouds of objects in the scene; S212. For each object point cloud, sample 1024 points and normalize the coordinates, send them to PointNet++ for encoding, retain the classification head of PointNet++, and take the output features of the previous layer of the classification head as the point cloud features of the object. , the result of the classification head is the category vector ; S213, point cloud features for each object , category vector and the location of the object , and combine to generate the features of each object , the expression is: ; in, and is the projection matrix, which transforms the category vector and location Projection to point cloud features Same dimensions; S214, all the features of the object Input together into the four-layer Transformer block to generate the final point cloud features .

[0008] Furthermore, in step S3, the specific process includes the following steps: S31, point cloud features and text features Randomly mask 15% of the features to obtain the masked point cloud features and text features ; S32, point cloud features after using mask , Image features and text features Model masked objects and use cross entropy loss with class prediction supervision only for masked objects ; S33. Using point cloud features , Image features And the masked text features Perform masked text modeling and use cross entropy loss that only supervises the masked words in the text ; S34. Align point cloud features using L1 smooth loss and image features , denoted as , the formula is: ; in, is the projection matrix, which is used to transform the point cloud features Projection to and image features Same dimension; S35. Align text features using L1 smooth loss and image features , denoted as , the formula is: ; in, is the projection matrix, which is used to transform text features Projection to and image features Same dimension; S36, comprehensive cross entropy loss , And the loss function , Train the encoding module and freeze the parameters of the pre-trained image encoder and Tokenizer during training.

[0009] Furthermore, in step S32, the specific process includes the following steps: S321, point cloud features after masking Add learnable tokens to restore the size before masking, denoted as ; S322, will and image features Send it to a layer of Transformer block to generate point cloud features after image enhancement ; S323, will and text features Send it to a layer of Transformer block to generate point cloud features after text enhancement ; S324, the result is sent to the decoder containing four layers of MLP, where the last layer outputs point cloud features The semantic category corresponding to each object in; S325. Use cross entropy loss , only the masked objects are supervised for category prediction.

[0010] Furthermore, in step S33, the specific process includes the following steps: S331, masked text features Add learnable tokens to restore the size before masking, denoted as ; S332, will and image features Send it to a layer of Transformer block to generate text features after image enhancement ; S333, will and point cloud features Send it to a layer of Transformer block to generate text features after point cloud enhancement ; S334, the result is sent to the decoder containing four layers of MLP, where the last layer outputs text features The true value Token ID corresponding to each word in; S335. Use cross entropy loss , only the masked words in the text are supervised.

[0011] Furthermore, in step S4, it specifically includes: For different fine-tuning tasks, the corresponding encoder is selected to extract features according to the input modal information, and any decoding network related to the current target task is used to generate prediction results in the format required by the task, and the loss function corresponding to the target task is used for supervision.

[0012] Furthermore, in step S4, the specific process includes the following steps: S41, inputting the point cloud into a pre-trained point cloud encoder to extract point cloud features; S42, inputting the voice or text of the problem description into the speech-text multimodal large model LLASM, and using LLAMA-Adapter to fuse the point cloud features encoded in step S41 with the problem description features encoded by the speech-text multimodal large model LLASM; S43, using the original prediction head of the speech-text multimodal large model LLASM to generate answers to the questions; S44, sending the point cloud features encoded in step S41 and the last layer of features before the prediction head of the speech-text multimodal large model LLASM to the decoding network of the 3D question answering task, predicting the correlation between each object in the point cloud features and the answer generated in step S43, selecting the object with the highest correlation and selecting its position with a bounding box; S45. Use cross entropy loss to supervise the outputs of steps S43 and S44.

[0013] Further, in step S44, the decoding network includes four Transformer blocks and two layers of MLP, wherein the first Transformer block uses cross attention, point cloud features as query Q, the last layer of features before the speech-text multimodal model LLASM prediction head as key K and value V, and the last three Transformer blocks use conventional self-attention.

[0014] By means of the above technical solution, the present invention provides a three-dimensional scene understanding method based on a multimodal large model, which has at least the following beneficial effects: 1. The encoding module training strategy proposed in the present invention utilizes information from multiple modalities to enhance the encoding module's ability to understand the scene, and aligns multiple modalities to the same feature space, so that the trained encoder can adapt to a variety of multimodal related tasks.

[0015] 2. The fine-tuning strategy proposed in the present invention combines the pre-trained multimodal encoding module with the large language model to realize a universal, comprehensive and robust multimodal large model, which can complete various visual language tasks in 3D and 2D and support text and voice input at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a flow chart of the three-dimensional scene understanding method of the present invention; Figure 2 It is a structural schematic diagram of the training encoding module of the present invention; Figure 3 Schematic diagram of the structure of the present invention for joint fine-tuning of multi-modal large models on specific tasks. DETAILED DESCRIPTION

[0017] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods, so that the implementation process of how the present application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0018] In order to better understand the scene and implement a robust, comprehensive, and universal multimodal model, please refer to Figure 1-Figure 3 This embodiment proposes a 3D scene understanding method based on a multimodal large model. By proposing a coding module training strategy and utilizing information from multiple modalities, and combining a fine-tuning strategy, the pre-trained multimodal coding module is combined with a large language model, which can complete multiple visual language tasks in 3D and 2D, and support text and voice input at the same time. The method includes the following steps: S1. Create a dataset containing point cloud-image-text pairs ,in, Point cloud data for indoor scenes. For indoor scene point cloud data The corresponding projected image, This example uses the ScanNet dataset as the data source to establish a dataset. , the specific contents are as follows: By obtaining indoor scene point cloud data from the ScanNet dataset , and randomly select a camera viewpoint to project the point cloud onto a two-dimensional plane to obtain the corresponding projection image ; Then convert the text in the ScanNet dataset into a text description of the scene , including question-answer pairs in the ScanQA dataset and text descriptions in the ScanRefer and ReferIt3D datasets; finally, a dataset containing point cloud-image-text pairs was established .

[0019] S2. Dataset Extract point cloud features in the input encoding module , Image features and text features , the encoding module includes a point cloud encoder, an image encoder and a text encoder; in step S2, the specific process includes the following steps: S21. Indoor scene point cloud data Input to the point cloud encoder to extract point cloud features ; In step S21, the specific process includes the following steps: S211, use the 3D segmentation mask labels in the ScanNet dataset for indoor scene point cloud data Cutting is performed to obtain a series of point clouds of objects in the scene; S212. For each object point cloud, sample 1024 points and normalize the coordinates, send them to PointNet++ for encoding, retain the classification head of PointNet++, and take the output features of the previous layer of the classification head as the point cloud features of the object. , the result of the classification head is the category vector ; S213, point cloud features for each object , category vector and the location of the object , and combine to generate the features of each object , the expression is: ; in, and is the projection matrix, which transforms the category vector and location Projection to point cloud features Same dimensions; S214, all the features of the object Input together into the four-layer Transformer block to generate the final point cloud features .

[0020] S22, projecting the image The projected image is sent to the pre-trained image encoder to obtain the initial image features. Specifically, the projected image is sent to the pre-trained CLIP (Contrastive Language–Image Pre-training) image encoder to obtain the initial image features. This is a process involving image processing and deep learning model reasoning. The CLIP model is trained on a large number of text-image pairs through contrastive learning, so that it can understand and associate image and text content.

[0021] S23, flatten the last two dimensions of the initial image features, change the feature dimension to 3 to conform to the input format of the Transformer block, and send the result to the four-layer Transformer block to generate image features ; S24. Describe the text Input into the pre-trained BERT Tokenizer to generate word embeddings; In this embodiment, it is a relatively straightforward process to input the text description into the pre-trained BERT Tokenizer to generate word embeddings. BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained deep bidirectional representation model whose Tokenizer is used to segment text into sub-word units (such as words or sub-word fragments) and convert these units into numeric IDs that the model can understand. These IDs are then used as input to the BERT model, and the model outputs the corresponding word embeddings (i.e., vector representations of the text).

[0022] S25: Send word embeddings into the four-layer Transformer block to generate text features , where the parameters of the four-layer Transformer block are initialized by the first four layers of the pre-trained BERT.

[0023] S3, based on point cloud features , Image features and text features , establish mask object reconstruction tasks, mask text reconstruction tasks, point cloud-image alignment tasks, and text-image alignment tasks to pre-train the encoding module; Figure 2 As shown, the encoding module training strategy in this embodiment utilizes information from multiple modalities to enhance the encoding module's ability to understand the scene, and aligns multiple modalities to the same feature space, so that the trained encoder can adapt to a variety of multimodal related tasks. In step S3, the specific process includes the following steps: S31, point cloud features and text features Randomly mask 15% of the features to obtain the masked point cloud features and text features ; S32, point cloud features after using mask , Image features and text features Model masked objects and use cross entropy loss with class prediction supervision only for masked objects ; In step S32, the specific process includes the following steps: S321, point cloud features after masking Add learnable tokens to restore the size before masking, denoted as ; S322, will and image features Send it to a layer of Transformer block to generate point cloud features after image enhancement , where the multi-head attention part of the Transformer block uses cross attention, As the query Q of the cross attention, the image features As the key K and value V of cross attention; S323, will and text features Send it to a layer of Transformer block to generate point cloud features after text enhancement , where the multi-head attention part of the Transformer block uses cross attention, Query Q as cross-attention, text features As the key K and value V of cross attention; S324, the result is sent to the decoder containing four layers of MLP, where the last layer outputs point cloud features The semantic category corresponding to each object in; S325. Use cross entropy loss , only the masked objects are supervised for category prediction.

[0024] S33. Using point cloud features , Image features And the masked text features Perform masked text modeling and use cross entropy loss that only supervises the masked words in the text In step S33, the specific process includes the following steps: S331, masked text features Add learnable tokens to restore the size before masking, denoted as ; S332, will and image features Send it to a layer of Transformer block to generate text features after image enhancement , where the multi-head attention part of the Transformer block uses cross attention, As the query Q of the cross attention, the image features As the key K and value V of cross attention; S333, will and point cloud features Send it to a layer of Transformer block to generate text features after point cloud enhancement , where the multi-head attention part of the Transformer block uses cross attention, As the query Q of cross attention, point cloud features As the key K and value V of cross attention; S334, the result is sent to the decoder containing four layers of MLP, where the last layer outputs text features The true value Token ID corresponding to each word in; S335. Use cross entropy loss , only the masked words in the text are supervised, and the true value Token ID is obtained by the original text through the Tokenizer in step S24.

[0025] S34. Align point cloud features using L1 smooth loss and image features , denoted as , the formula is: ; in, is the projection matrix, which is used to transform the point cloud features Projection to and image features Same dimension; S35. Align text features using L1 smooth loss and image features , denoted as , the formula is: ; in, is the projection matrix, which is used to transform text features Projection to and image features Same dimension; S36, comprehensive cross entropy loss , And the loss function , The encoding module is trained, and during the training, the parameters of the pre-trained image encoder and Tokenizer are frozen. After the training is completed, all network modules involved in step S3 are discarded, and only all encoding modules involved in step S2 are retained, and these encoding modules are used for the fine-tuning operation described in the next step.

[0026] S4, the pre-trained encoding module and the speech-text multimodal large model LLASM are jointly fine-tuned on the target task. In this embodiment, for different fine-tuning tasks, the corresponding encoder is selected according to the input modal information to extract features, and any decoding network related to the current target task is used to generate prediction results in the format required by the task, and the loss function corresponding to the target task is used for supervision. That is, this step will be different in the selection of encoding module, decoding network and loss function according to the different fine-tuning tasks, such as Figure 3 As shown, this embodiment takes the 3D question answering task as an example. In step S4, the specific process includes the following steps: S41, inputting the point cloud into a pre-trained point cloud encoder to extract point cloud features; S42, inputting the voice or text of the problem description into the speech-text multimodal large model LLASM, and using LLAMA-Adapter to fuse the point cloud features encoded in step S41 with the problem description features encoded by the speech-text multimodal large model LLASM; S43, using the original prediction head of the speech-text multimodal large model LLASM to generate answers to the questions; S44, the point cloud features encoded in step S41 and the last layer of features before the prediction head of the speech-text multimodal large model LLASM are sent to the decoding network of the 3D question answering task, and the correlation between each object in the point cloud features and the answer generated in step S43 is predicted, and the object with the highest correlation is selected and its position is selected with a bounding box. A specific design example of the decoding network is given below, namely: The decoding network contains four Transformer blocks and two layers of MLP. The first Transformer block uses cross attention, point cloud features as query Q, and the last layer of features before the speech-text multimodal model LLASM prediction head as key K and value V. The last three Transformer blocks use regular self-attention. S45. Use cross entropy loss to supervise the outputs of steps S43 and S44.

[0027] The fine-tuning strategy proposed in this embodiment combines the pre-trained multimodal encoding module with the large language model to realize a universal, comprehensive and robust multimodal large model that can complete a variety of visual language tasks in 3D and 2D and support text and voice input at the same time.

[0028] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, so the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0029] The above implementation methods have been described in detail. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A three-dimensional scene understanding method based on a multimodal large model, characterized in that: The method comprises the following steps: S1. Create a dataset containing point cloud-image-text pairs ,in, Point cloud data for indoor scenes. For indoor scene point cloud data The corresponding projected image, For text description; S2. Dataset Extract point cloud features in the input encoding module , Image features and text features ,The encoding module includes a point cloud encoder, an image encoder, and a text encoder; S3, based on point cloud features , Image features and text features , establish mask object reconstruction task, mask text reconstruction task, point cloud-image alignment task and text-image alignment task to pre-train the encoding module; S4. Jointly fine-tune the pre-trained encoding module and the speech-text multimodal large model LLASM on the target task.

2. The three-dimensional scene understanding method according to claim 1, characterized in that: In step S2, the specific process includes the following steps: S21. Indoor scene point cloud data Input to the point cloud encoder to extract point cloud features ; S22, projecting the image Send it to the pre-trained image encoder to obtain the initial image features; S23, flatten the last two dimensions of the initial image features and send the result to the four-layer Transformer block to generate image features ; S24. Describe the text Input the pre-trained BERT Tokenizer to generate word embeddings; S25: Send word embeddings into the four-layer Transformer block to generate text features .

3. The three-dimensional scene understanding method according to claim 2, characterized in that: In step S21, the specific process includes the following steps: S211, use the 3D segmentation mask labels in the ScanNet dataset for indoor scene point cloud data Cutting is performed to obtain a series of point clouds of objects in the scene; S212. For each object point cloud, sample 1024 points and normalize the coordinates, send them to PointNet++ for encoding, retain the classification head of PointNet++, and take the output features of the previous layer of the classification head as the point cloud features of the object. , the result of the classification head is the category vector ; S213, point cloud features for each object , category vector and the location of the object , and combine to generate the features of each object , the expression is: ； in, and is the projection matrix, which transforms the category vector and location Projection to point cloud features Same dimensions; S214, all the features of the object Input together into the four-layer Transformer block to generate the final point cloud features .

4. The three-dimensional scene understanding method according to claim 1, characterized in that: In step S3, the specific process includes the following steps: S31, point cloud features and text features Randomly mask 15% of the features to obtain the masked point cloud features and text features ; S32, point cloud features after using mask , Image features and text features Model masked objects and use cross entropy loss with class prediction supervision only for masked objects ; S33. Using point cloud features , Image features And the masked text features Perform masked text modeling and use cross entropy loss that only supervises the masked words in the text ; S34. Align point cloud features using L1 smooth loss and image features , denoted as , the formula is: ； in, is the projection matrix, used to transform the point cloud features Projection to and image features Same dimension; S35. Align text features using L1 smooth loss and image features , denoted as , the formula is: ； in, is the projection matrix, which is used to transform text features Projection to and image features Same dimension; S36, comprehensive cross entropy loss , And the loss function , Train the encoding module and freeze the parameters of the pre-trained image encoder and Tokenizer during training.

5. The three-dimensional scene understanding method according to claim 4, characterized in that: In step S32, the specific process includes the following steps: S321, point cloud features after masking Add learnable tokens to restore the size before masking, denoted as ; S322, will and image features Send it to a layer of Transformer block to generate point cloud features after image enhancement ; S323, will and text features Send it to a layer of Transformer block to generate point cloud features after text enhancement ; S324, the result is sent to the decoder containing four layers of MLP, where the last layer outputs point cloud features The semantic category corresponding to each object in; S325. Use cross entropy loss , only the masked objects are supervised for category prediction.

6. The three-dimensional scene understanding method according to claim 4, characterized in that: In step S33, the specific process includes the following steps: S331, masked text features Add learnable tokens to restore the size before masking, denoted as ; S332, will and image features Send it to a layer of Transformer block to generate text features after image enhancement ; S333, will and point cloud features Send it to a layer of Transformer block to generate text features after point cloud enhancement ; S334, the result is sent to the decoder containing four layers of MLP, where the last layer outputs text features The true value Token ID corresponding to each word in; S335. Use cross entropy loss , only the masked words in the text are supervised.

7. The three-dimensional scene understanding method according to claim 1, characterized in that: In step S4, it specifically includes: For different fine-tuning tasks, the corresponding encoder is selected to extract features according to the input modal information, and any decoding network related to the current target task is used to generate prediction results in the format required by the task, and the loss function corresponding to the target task is used for supervision.

8. The three-dimensional scene understanding method according to claim 7, characterized in that: In step S4, the specific process includes the following steps: S41, inputting the point cloud into a pre-trained point cloud encoder to extract point cloud features; S42, inputting the voice or text of the problem description into the speech-text multimodal large model LLASM, and using LLAMA-Adapter to fuse the point cloud features encoded in step S41 with the problem description features encoded by the speech-text multimodal large model LLASM; S43, using the original prediction head of the speech-text multimodal large model LLASM to generate answers to the questions; S44, sending the point cloud features encoded in step S41 and the last layer of features before the prediction head of the speech-text multimodal large model LLASM to the decoding network of the 3D question answering task, predicting the correlation between each object in the point cloud features and the answer generated in step S43, selecting the object with the highest correlation and selecting its position with a bounding box; S45. Use cross entropy loss to supervise the outputs of steps S43 and S44.

9. The three-dimensional scene understanding method according to claim 8, characterized in that: In step S44, the decoding network includes four Transformer blocks and two layers of MLP, where the first Transformer block uses cross attention, point cloud features as query Q, the last layer of features before the speech-text multimodal model LLASM prediction head as key K and value V, and the last three Transformer blocks use conventional self-attention.

Images