A 3D scene understanding and interaction method and device based on multimodal fusion

Abstract

The present invention discloses a 3D scene understanding and interaction method and device based on multimodal fusion, comprising the following steps: data input and preprocessing, space-based self-attention mechanism processing, multi-view cross-modal fusion, output and interaction. The beneficial effects of the present invention are: 1. Improved versatility: Through the end-to-end multi-task tuning framework, the model can handle multiple tasks at the same time without separate fine-tuning; 2. Enhanced spatial understanding ability: The 3D-ISR module significantly improves the model's perception of the spatial relationship of objects in complex scenes; 3. Optimized fusion effect: The MCMF module realizes efficient fusion of 3D geometric and 2D semantic features at the feature level.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a 3D scene understanding and interaction method and device based on multimodal fusion. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology in recent years, 3D scene understanding has gradually become one of the core technologies in fields such as robotics and autonomous driving. Current research can be roughly divided into the following categories: 1.3D Visual Grounding: Locating the target object based on language description requires the artificial intelligence model to accurately analyze the geometric and semantic characteristics of objects in the 3D environment.

[0003] 2.3D Question Answering: Answering semantic and geometric questions about the scene, which requires the AI ​​model to have spatial reasoning capabilities.

[0004] 3D Dense Captioning: Generates detailed natural language descriptions of scenes or objects, which requires the AI ​​model to have rich language generation capabilities.

[0005] Traditional methods are usually optimized for a single task, for example, using point cloud data combined with a vision-language-model to complete scene description, or aligning point cloud and text through a pre-trained model.

[0006] Existing models often lack a deep fusion mechanism when fusing 2D semantic features with 3D geometric features. For example, simply superimposing point cloud and image features may lead to redundancy and inconsistency of feature information. Second, the shortcomings of existing models also include: 1. Insufficient spatial relationship modeling capabilities: Existing methods have difficulty capturing the spatial relationship of objects in complex 3D scenes. For example, in an indoor scene, the positional relationship between a table and a chair may require more precise geometric modeling, while traditional methods rely on simple distance calculations and lack semantic support.

[0007] 2. Finetuning is highly dependent on: Most models need to be fine-tuned for different tasks, which increases the complexity of development and deployment and limits the practicality of the models in multi-task settings. Summary of the invention

[0008] The purpose of the present invention is to provide a 3D scene understanding and interaction method based on multimodal fusion to solve the problems raised in the above background technology.

[0009] To achieve the above object, the present invention provides the following technical solutions: In a first aspect, an embodiment of the present invention provides a 3D scene understanding and interaction method based on multimodal fusion, comprising the following steps: Data input and preprocessing: obtain depth camera image sequences and 3D color point cloud data, select, segment, extract, and multi-scale crop the depth camera image sequences, generate CLIP features through the CLIP visual encoder, and perform 3D instance segmentation and 3D object encoding on the 3D color point cloud to obtain object features; Spatial self-attention mechanism processing: perform position encoding and linear transformation on the object coordinate information, build a spatial relationship matrix, calculate the spatial self-attention map based on the self-attention mechanism based on the original weight, and obtain the weight or probability of the relationship between objects through Sigmoid function processing; Multi-view cross-modal fusion: 3D object features and multi-view 2D features are mapped and converted through the mapping network respectively, and then processed by the self-attention mechanism respectively, and then interacted through the cross-modal cross-attention mechanism and added with the instance-level token to obtain the fused features; Output and interaction: The fused features are input into a large language model, which responds to the user in natural language based on the input information and user questions, thus achieving intelligent understanding and interaction of 3D scenes.

[0010] Preferably, in the data input and preprocessing, K images are selected from the depth camera image sequence, where K is a positive integer, and the selection method includes random selection, selection based on image quality, or selection based on content relevance.

[0011] Preferably, in the space-based self-attention mechanism processing, the position encoding adopts sine-cosine position encoding, and the calculation method of the elements of the spatial relationship matrix includes Euclidean distance or cosine similarity.

[0012] Preferably, the position encoding and linear transformation of the object coordinate information to construct the spatial relationship matrix refers to position encoding the object coordinate information, and then obtaining relevant outputs and spatial features through a linear layer to further construct the spatial relationship matrix.

[0013] Preferably, calculating a space-based self-attention map by a self-attention mechanism in combination with the original weights, and obtaining the weight or probability of the relationship between objects by Sigmoid function processing means combining the original weights with the linear layer output, performing an operation, and calculating a space-based self-attention map by a self-attention mechanism; the self-attention map is processed by a Sigmoid function to map the value between 0 and 1, which is used to represent the weight or probability of the relationship between objects.

[0014] Preferably, the large language model is GPT-3, BERT or a model customized and trained according to specific needs, and a large amount of text data and annotated data related to the 3D scene are used during training.

[0015] In a second aspect, an embodiment of the present invention provides a 3D scene understanding and interaction device based on multimodal fusion, which is used to implement the method described in any of the above embodiments, and the device includes: Data input and preprocessing module: The data input and preprocessing module obtains the depth camera image sequence and 3D color point cloud data, selects, segments, extracts, and multi-scales the depth camera image sequence, generates CLIP features through the CLIP visual encoder, and performs 3D instance segmentation and 3D object encoding on the 3D color point cloud to obtain object features; Self-attention mechanism processing module: The self-attention mechanism processing module is used to perform position encoding and linear transformation on the object coordinate information, construct a spatial relationship matrix, calculate a space-based self-attention map through the self-attention mechanism in combination with the original weight, and obtain the weight or probability of the relationship between objects through Sigmoid function processing; Multi-view cross-modal fusion module: The multi-view cross-modal fusion module is used to map and transform 3D object features and multi-view 2D features through a mapping network, and then process them through a self-attention mechanism, and then interact through a cross-modal cross-attention mechanism and add them to instance-level tokens to obtain fused features; Output and interaction module: The output and interaction module is used to input the fused features into a large language model. The large language model responds to the user in natural language based on the input information and user questions, thereby achieving intelligent understanding and interaction of the 3D scene.

[0016] In a third aspect, an embodiment of the present invention provides an electronic device, comprising at least one processor, which is communicatively connected to at least one memory, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any of the above embodiments.

[0017] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method described in any one of the above embodiments when executed.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. Improved versatility: Through an end-to-end multi-task tuning framework, this model can handle multiple tasks simultaneously without the need for separate fine-tuning.

[0019] 2. Enhanced spatial understanding ability: The 3D-ISR module significantly improves the model's ability to perceive the spatial relationships of objects in complex scenes.

[0020] 3. Optimization of fusion effect: The MCMF module realizes efficient fusion of 3D geometric and 2D semantic features at the feature level. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of a 3D scene understanding and interaction method based on multimodal fusion according to an embodiment of the present invention; Figure 2 This is a diagram of the architecture of a 3D scene understanding and interaction device based on multimodal fusion according to an embodiment of the present invention; Figure 3 This is a processing flow chart of a 3D scene understanding and interaction device based on multimodal fusion according to an embodiment of the present invention; Figure 4 This is a diagram of the architecture of a multi-view cross-modal fusion (MCMF) module according to an embodiment of the present invention; Figure 5 Schematic diagram of a 3D instance spatial relationship (3D-ISR) module according to an embodiment of the present invention; Figure 6 FIG. 2 is a diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] See also Figure 1 The present invention provides a technical solution: a 3D scene understanding and interaction method based on multimodal fusion, comprising the following steps: S100: Data input and preprocessing: Obtain depth camera image sequence and 3D color point cloud data, select, segment, extract, and multi-scale crop the depth camera image sequence, generate CLIP features through the CLIP visual encoder, and perform 3D instance segmentation and 3D object encoding on the 3D color point cloud to obtain object features; S200: Processing based on space self-attention mechanism: Position encoding and linear transformation of object coordinate information are performed to construct a spatial relationship matrix, and the space-based self-attention map is calculated through the self-attention mechanism in combination with the original weights, and the weight or probability of the relationship between objects is obtained through Sigmoid function processing; S300: Multi-view cross-modal fusion: 3D object features and multi-view 2D features are mapped and converted through the mapping network respectively, and then processed by the self-attention mechanism respectively, and then interacted through the cross-modal cross-attention mechanism and added with the instance-level token to obtain the fused features; S400: Output and interaction: The fused features are input into a large language model, and the large language model responds to the user in natural language based on the input information and user questions, thereby achieving intelligent understanding and interaction of the 3D scene.

[0024] In one embodiment of the present invention, the specific processing process of step S100 includes: 1. Obtain multi-source data: Obtain time-series depth camera image sequences and corresponding 3D color point cloud data through a depth camera.

[0025] Specifically, in this embodiment, a depth camera is used to collect a depth camera image sequence and 3D color point cloud data of a specific scene. For example, in an indoor environment, data is obtained from different angles and time points to ensure data diversity and integrity.

[0026] 2. Image data processing: 2.1. Select K images from the depth camera image sequence, where K is a preset positive integer.

[0027] 2.2. Perform segmentation and extraction operations on the selected K images to separate the objects in the images.

[0028] 2.3. Perform multi-scale cropping on the segmented objects to obtain image object data suitable for subsequent processing.

[0029] 2.4. Pass the cropped object through the CLIP visual encoder to generate CLIP features.

[0030] Specifically, in this embodiment, for the depth camera image sequence, K images are selected according to a preset rule or algorithm. The selection can be performed by random selection, image quality or content relevance, etc. Then, the selected images are segmented and extracted using an existing image segmentation algorithm (such as a segmentation model based on deep learning) to accurately separate the objects in the image. For the segmented objects, multi-scale cropping is performed according to different scale requirements. For example, multiple scale ranges can be set and the objects are cropped separately to obtain object information at different scales. Finally, the cropped objects are input into the CLIP visual encoder to generate corresponding CLIP features. The CLIP visual encoder can use an existing pre-trained model, such as the CLIP model of OpenAI, through which image objects are converted into feature vectors with semantic information.

[0031] 3. 3D point cloud data processing: 3.1. Use the 3D instance segmentation network to process the 3D color point cloud, segment different object instances, and obtain object coordinate information.

[0032] 3.2. Encode the segmented objects through the 3D object encoder to extract the relevant information and features of the objects.

[0033] Specifically, in this embodiment, a dedicated 3D instance segmentation network is used to process 3D color point cloud data. The network can be a network architecture based on deep learning, such as a network based on PointNet or its variants. By training the network, it can accurately segment different object instances in the 3D point cloud and obtain the coordinate information of each object. Then, the segmented objects are encoded by a 3D object encoder, which can be a simple multi-layer neural network for extracting relevant information and features such as geometric features and color features of the objects to obtain a 3D feature representation of each object.

[0034] In one embodiment of the present invention, the specific processing process of step S200 based on the spatial self-attention mechanism processing (3D instance spatial relationship module - 3D-ISR) includes: 1. Position encoding and linear transformation: The object coordinate information is encoded, and then the output P1 to P is obtained through the linear layer. N and spatial features C1 to C N , construct the spatial relationship matrix , where N is the number of objects.

[0035] In the position encoding stage, common position encoding methods, such as sine-cosine position encoding, are used to encode the coordinate information of the object and introduce the position information for subsequent processing. Then, the position encoded information is transformed through a linear layer to obtain the output P1 to P N and spatial features C1 to C N These outputs and features will be used to construct a spatial relationship matrix, where the matrix elements represent the spatial relationship measure between object i and object j, which can be calculated using methods such as Euclidean distance and cosine similarity according to the specific design.

[0036] 2. Self-attention calculation and weight adjustment: Combined with the original weight W P The output of the linear layer is operated and processed by the self-attention mechanism to calculate the space-based self-attention map. The self-attention map is processed by the Sigmoid function, and the output value is mapped between 0 and 1 to obtain the weight or probability representing the relationship between objects.

[0037] In the self-attention calculation and weight adjustment part, the original weight W P The output of the linear layer is added and then input into the self-attention mechanism. The self-attention mechanism can adopt the self-attention module in the common Transformer architecture to obtain a space-based self-attention map by calculating the similarity and weights between the query, key, and value. The self-attention map reflects the degree of association and importance between different objects. Finally, the self-attention map is processed by the Sigmoid function to map the values ​​between 0 and 1 to obtain the weights or probabilities representing the relationship between objects. These weights or probabilities will be used in subsequent processing to highlight important object relationships and information.

[0038] In one embodiment of the present invention, the specific processing process of step S300 includes: 1. Feature mapping and conversion: 1.1. Map the 3D object features through the 3D object mapping network to obtain flattened features with a dimension of , where D is the feature dimension.

[0039] 1.2. Multi-view 2D features are mapped through a 2D object mapping network to obtain features of dimension , where K is the number of viewpoints.

[0040] 2. Modal processing within the self-attention mechanism: 2.1. The flattened 3D features are processed through the 3D self-attention mechanism to obtain new features, which involves the calculation of query, key, and value.

[0041] 2.2. The multi-view 2D features are processed by the 2D self-attention mechanism to obtain the processed features, which also involves the calculation of Q, K, and V.

[0042] 3. Cross-modal interaction and fusion: 3.1. The 3D features processed by the 3D self-attention mechanism (only the Q part) and the 2D features processed by the 2D self-attention mechanism (the K and V parts) are interacted through the cross-modal cross-attention mechanism.

[0043] 3.2. Add the result after interaction to the instance-level token to obtain the final fused features.

[0044] In this embodiment, specifically, in the feature mapping and conversion stage, the 3D object features are mapped through a 3D object mapping network. The 3D object mapping network can be a convolutional neural network or other network architecture suitable for processing 3D features, and its purpose is to convert 3D object features into flattened features suitable for subsequent processing, with a dimension of . At the same time, multi-view 2D features are mapped through a 2D object mapping network, which can be a multi-layer perceptron or other network capable of processing 2D features, mapping multi-view 2D features into features of dimension , where K is the number of views, N is the number of objects, and D is the feature dimension.

[0045] In this embodiment, specifically, in the mode processing part of the self-attention mechanism, the flattened 3D features are processed by the 3D self-attention mechanism. The 3D self-attention mechanism can refer to the self-attention mechanism in the Transformer, and obtain new features by calculating the similarity and weight between 3D features, thereby enhancing the expression ability of 3D features and the modeling of internal relationships between objects. For multi-view 2D features, after being processed by the 2D self-attention mechanism, the similarity and weight between the features of objects under different perspectives are also calculated to obtain the processed features, highlighting the important features and information of objects under different perspectives.

[0046] In this embodiment, specifically, in the cross-modal interaction and fusion stage, the 3D features processed by the 3D self-attention mechanism (only the Q part) and the 2D features processed by the 2D self-attention mechanism (the K and V parts) are interacted through the cross-modal cross-attention mechanism. The cross-modal cross-attention mechanism can be designed to be similar to the cross-attention mechanism in Transformer, but it is necessary to consider the differences between the 3D and 2D modalities and the way of information fusion. By calculating the similarity and weight between the query of the 3D feature and the key and value of the 2D feature, the information interaction and fusion between the 3D and 2D modalities are realized. Finally, the result after the interaction is added to the instance-level token to obtain the final fused feature. The instance-level token can be a learnable vector used to add additional information to the fused feature or guide the fusion process.

[0047] In one embodiment of the present invention, the specific processing process of step S400 includes: 1. Input into a large language model: Input the fused feature information into a pre-trained large language model.

[0048] 2. User interaction and answers: The large language model responds to users in detail in natural language based on the input fusion feature information and various questions raised by users about the scene, providing specific information such as the positional relationship and attributes of objects in the scene, thereby achieving intelligent understanding and interaction of 3D scenes.

[0049] In this embodiment, specifically, the fused feature information is input into a pre-trained large language model. The large language model can adopt existing models such as GPT-3, BERT, etc., or can be customized and trained according to specific needs. When training a large language model, a large amount of text data and annotated data related to 3D scenes can be used to enable it to understand and generate natural language descriptions and answers about 3D scenes.

[0050] In this embodiment, specifically, when a user asks a question about a scene, the large language model responds to the user in detail in natural language based on the input fusion feature information and the user's question through the internal neural network and language generation mechanism. For example, if the user asks "what items are on the table in the scene", the large language model will generate an answer like "there is a red cup and a blue book on the table" based on the information about the location and category of the objects in the scene in the fusion features, thereby realizing intelligent understanding and interaction of 3D scenes.

[0051] See also Figure 2 and Figure 3 , Figure 2It is shown that an embodiment of the present invention provides a 3D scene understanding and interaction device based on multimodal fusion, which is used to implement the method described in the above embodiment, and the device includes: Data input and preprocessing module 100: The data input and preprocessing module obtains a depth camera image sequence and 3D color point cloud data, selects, segments, extracts, and multi-scales the depth camera image sequence, generates CLIP features through a CLIP visual encoder, and performs 3D instance segmentation and 3D object encoding on the 3D color point cloud to obtain object features; Self-attention mechanism processing module 200: The self-attention mechanism processing module is used to perform position encoding and linear transformation on the object coordinate information, construct a spatial relationship matrix, calculate a space-based self-attention map through the self-attention mechanism in combination with the original weight, and obtain the weight or probability of the relationship between objects through Sigmoid function processing; Multi-view cross-modal fusion module 300: The multi-view cross-modal fusion module is used to map and transform the 3D object features and the multi-view 2D features through the mapping network respectively, and then process them respectively through the self-attention mechanism, and then interact through the cross-modal cross-attention mechanism and add them with the instance-level token to obtain the fused features; Output and interaction module 400: The output and interaction module is used to input the fused features into a large language model, and the large language model responds to the user in a natural language according to the input information and user questions, thereby realizing intelligent understanding and interaction of the 3D scene.

[0052] In this embodiment, a 3D scene understanding and interaction device based on multimodal fusion also has a built-in 3D instance segmentation network, a visual encoder (such as SAM and CLIP), a 3D instance spatial relationship (3D-ISR) module, LLM, etc. The components are interconnected through data transmission and processing to jointly complete the functions of 3D scene understanding and multi-task instruction adjustment.

[0053] Figure 3It is a specific process of scene processing by the device and method provided by the above embodiment. At the beginning, a depth camera image sequence (time 1, time 2... time T) and a 3D color point cloud are input, and then the input image is preprocessed and feature extracted, including: selecting K images from the image sequence; segmenting and extracting the selected K images, and then performing multi-scale cropping, inputting the cropped images into the CLIP visual encoder to extract CLIP features; using the 3D instance segmentation network to process the 3D color point cloud and segment object instances (object 1, object 2...). Then, object encoding and feature fusion are performed, including: encoding the segmented 3D object instances from different perspectives (perspective 1, perspective 2... perspective K) through the 3D object encoder; using the multi-view cross-modal fusion module (MCMF) to fuse CLIP features and 3D object encoding features, combining scene set tokens, text tokens and instance-level tokens; processing the spatial relationship between objects through the 3D instance spatial relationship module (3D-ISR). Next is user interaction and text processing, including: the user inputs a description or question about the scene, and the text information is encoded through a text encoder; the multimodal information (visual features, object relationship features, etc.) and the encoded text information are input into a large language model; the large language model generates responses (such as descriptions of objects in the scene).

[0054] For details, please refer to Figure 4 , Multi-view cross-modal fusion module (MCMF) 300 architecture and workflow: 3D object mapping network and 2D object mapping network 301: used to map and transform the original 3D and 2D features so that they can better adapt to the subsequent self-attention mechanism and cross-modal interaction.

[0055] Self-attention mechanism 302: Captures the internal relationships and importance between objects in 3D and 2D feature spaces respectively, enhancing the expressiveness of features.

[0056] Cross-modal attention mechanism 303: It realizes information interaction and fusion between 3D and 2D modalities, allowing the model to comprehensively utilize information from different perspectives and modalities to improve its understanding of objects and scenes.

[0057] Specifically, the multi-view cross-modal fusion module first obtains input data, including the dimensions of 3D object features and multi-view 2D features, where the 3D object feature dimension is , represents the 3D features of N objects, and the dimension of each feature is D 3d .

[0058] The multi-view 2D feature dimension is , where K represents the number of viewing angles and N represents the number of objects. Represents the dimension of each 2D feature.

[0059] Specifically, the processing flow of the multi-view cross-modal fusion module 300 on data is as follows: 1. Feature Mapping: The 3D object features are flattened through the 3D object mapping network, and the dimension is still .

[0060] The multi-view 2D features are mapped through the 2D object mapping network to obtain a dimension of characteristics.

[0061] 2. Self-attention mechanism processing: The flattened 3D features are processed through the 3D self-attention mechanism to obtain a new Features, where Q, K, and V are the query, key, and value in the self-attention mechanism.

[0062] The multi-view 2D features are processed by the 2D self-attention mechanism to obtain Features also involve the calculation of Q, K, and V.

[0063] 3. Cross-modal interaction: The 3D features processed by the 3D self-attention mechanism (only the Q part) and the 2D features processed by the 2D self-attention mechanism (the K and V parts) interact through the cross-modal cross-attention mechanism.

[0064] The result of the interaction is added to the instance-level token to obtain the final fused features.

[0065] For details, please refer to Figure 5 , which shows a detailed architecture and processing flow of a 3D instance spatial relationship module (3D-ISR) in this embodiment, mainly involving a space-based self-attention mechanism and subsequent processing steps. The self-attention mechanism processing module 200 is configured as within the module, and the self-attention mechanism processing module 200 adopts a space-based self-attention mechanism.

[0066] First, the 3D instance spatial relationship module obtains object coordinate information: including the coordinate information of object 1, object 2, object 3, etc., which is input into the subsequent processing flow after position encoding.

[0067] Spatial-based self-attention mechanism: Secondly, the processing flow of the self-attention mechanism processing module 200 is: 1. Position encoding and linear layer: After the object coordinate information is position-encoded, it is obtained from P1 to P through the linear layer. N etc. output.

[0068] At the same time, you will also get C1 to C N These features are combined into a spatial relationship matrix , where N represents the number of objects.

[0069] 2. Original weight and self-attention mechanism: Original weight W P It is then added to the output of the linear layer and processed by the self-attention mechanism.

[0070] The self-attention mechanism calculates a space-based self-attention map based on information such as the spatial relationship between objects. This self-attention map can reflect the degree of correlation and importance between different objects.

[0071] Finally, the follow-up processing: 1. Sigmoid function: The self-attention map is processed by the Sigmoid function, and the output value is mapped to between 0 and 1, which is used to represent the weight or probability of the relationship between different objects.

[0072] 2. Scene-level tokens: The information after the above processing is finally generated into scene-level tokens through operations such as scene mapping network and pooling layer. These tokens can be used for subsequent multi-layer perceptron processing to achieve modeling and analysis of the spatial relationship of objects in 3D scenes.

[0073] In this embodiment, the core of the 3D instance spatial relationship module is to effectively capture the spatial relationship between different objects in the 3D scene through a space-based self-attention mechanism, and integrate this relationship information into subsequent processing, so as to better understand and analyze the structure of the 3D scene and the interaction between objects. This mechanism can improve the model's ability and accuracy in understanding the scene when processing 3D vision tasks, and has important application value in the fields of object recognition, scene understanding, robot navigation, etc.

[0074] Table 1: Experimental indicators of the Inst3D-LMM method provided by this application and the current best method on the ScanQA and Scan2Cap datasets

[0075] In the table, B-1 and B-24 are indicators in the field of machine translation and text generation. B-1 focuses on the matching degree at the word level, and B-24 is the matching degree at the sentence level. METEOR is also an indicator in the field of machine translation and text generation, which adds the evaluation of part of speech changes and synonym matching. ROUGE is an indicator of the text summarization task, which evaluates the overlap rate between the summary and the manual reference summary. CIDER is a commonly used indicator in the field of image description, which compares the similarity between the generated image description and the reference description. EM is an indicator in the field of text generation, which evaluates whether the predicted result and the reference result are consistent at the text level.

[0076] See also Figure 6 , Figure 6 A schematic diagram of an electronic device 20 that can implement an embodiment of the present invention is shown, and the electronic device is intended to represent various forms of control devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0077] The electronic device 20 includes at least one processor 21, and a memory connected to the at least one processor 21, such as a read-only memory (ROM) 22, a random access memory (RAM) 23, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 21 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 22 or the computer program loaded from the storage unit 28 to the random access memory (RAM) 13. In the RAM 23, various programs and data required for the operation of the electronic device 20 can also be stored. The processor 21, ROM 22 and RAM 23 are connected to each other through a bus 24. An input / output (I / O) interface 25 is also connected to the bus 24.

[0078] A number of components in the electronic device 20 are connected to the I / O interface 25, including: an input unit 26, such as a keyboard, a mouse, etc.; an output unit 27, such as various types of displays, speakers, etc.; a storage unit 28, such as a disk, an optical disk, etc.; and a communication unit 29, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 29 allows the electronic device 20 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0079] The processor 21 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 21 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 21 performs the various methods and processes described above.

[0080] In some embodiments, the method of the above embodiment may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 28. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 20 via the ROM 22 and / or the communication unit 29. When the computer program is loaded into the RAM 23 and executed by the processor 21, one or more steps in the method described above may be performed. Alternatively, in other embodiments, the processor 21 may be configured to perform the method of the above embodiment in any other appropriate manner (e.g., by means of firmware).

[0081] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0082] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0083] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0084] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0085] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0086] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0087] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A 3D scene understanding and interaction method based on multimodal fusion, characterized in that: The steps include: Data input and preprocessing: Obtain depth camera image sequences and 3D color point cloud data, select, segment, extract, and multi-scale crop the depth camera image sequences, generate CLIP features through the CLIP visual encoder, and perform 3D instance segmentation and 3D object encoding on the 3D color point cloud to obtain object features; Spatial self-attention mechanism processing: perform position encoding and linear transformation on the object coordinate information, build a spatial relationship matrix, calculate the spatial self-attention map based on the self-attention mechanism based on the original weight, and obtain the weight or probability of the relationship between objects through Sigmoid function processing; Multi-view cross-modal fusion: 3D object features and multi-view 2D features are mapped and converted through the mapping network respectively, and then processed by the self-attention mechanism respectively, and then interacted through the cross-modal cross-attention mechanism and added with the instance-level token to obtain the fused features; Output and interaction: The fused features are input into a large language model, which responds to the user in natural language based on the input information and user questions, thus achieving intelligent understanding and interaction of 3D scenes.

2. The 3D scene understanding and interaction method based on multimodal fusion according to claim 1, characterized in that: In the data input and preprocessing, K images are selected from the depth camera image sequence, where K is a positive integer, and the selection method includes random selection, selection based on image quality, or selection based on content relevance.

3. The 3D scene understanding and interaction method based on multimodal fusion according to claim 2, characterized in that: In the space-based self-attention mechanism processing, position encoding adopts sine-cosine position encoding, and the calculation method of the elements of the spatial relationship matrix includes Euclidean distance or cosine similarity.

4. The 3D scene understanding and interaction method based on multimodal fusion according to claim 3, characterized in that: The position encoding and linear transformation of the object coordinate information to construct the spatial relationship matrix refers to position encoding the object coordinate information, and then obtaining the relevant output and spatial features through the linear layer, and then constructing the spatial relationship matrix .

5. The 3D scene understanding and interaction method based on multimodal fusion according to claim 4, characterized in that: Combining the original weights with the self-attention mechanism to calculate the space-based self-attention map, and processing it with the Sigmoid function to obtain the weight or probability of the relationship between objects means combining the original weights with the output of the linear layer, processing it with the self-attention mechanism, and calculating the space-based self-attention map; the self-attention map is processed by the Sigmoid function to map the value between 0 and 1, which is used to represent the weight or probability of the relationship between objects.

6. The 3D scene understanding and interaction method based on multimodal fusion according to claim 4, characterized in that: The large language model is GPT-3, BERT, or a model customized and trained according to specific needs, and a large amount of text data and labeled data related to 3D scenes are used during training.

7. A 3D scene understanding and interaction device based on multimodal fusion, used to implement the method described in any one of claims 1 to 5, characterized in that: The device includes: Data input and preprocessing module: The data input and preprocessing module obtains the depth camera image sequence and 3D color point cloud data, selects, segments, extracts, and multi-scales the depth camera image sequence, generates CLIP features through the CLIP visual encoder, and performs 3D instance segmentation and 3D object encoding on the 3D color point cloud to obtain object features; Self-attention mechanism processing module: The self-attention mechanism processing module is used to perform position encoding and linear transformation on the object coordinate information, construct a spatial relationship matrix, calculate a space-based self-attention map through the self-attention mechanism in combination with the original weight, and obtain the weight or probability of the relationship between objects through Sigmoid function processing; Multi-view cross-modal fusion module: The multi-view cross-modal fusion module is used to map and transform 3D object features and multi-view 2D features through a mapping network, and then process them through a self-attention mechanism, and then interact through a cross-modal cross-attention mechanism and add them to instance-level tokens to obtain fused features; Output and interaction module: The output and interaction module is used to input the fused features into a large language model. The large language model responds to the user in natural language based on the input information and user questions, thereby achieving intelligent understanding and interaction of the 3D scene.

8. An electronic device, characterized in that: The method comprises at least one processor, wherein the processor is communicatively connected to at least one memory, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method according to any one of claims 1 to 6 when the computer instructions are executed.

Images