Three-dimensional visual question and answer method and system

Abstract

The invention relates to a three-dimensional visual question and answer method and system. The method comprises the following steps: extracting features from a two-dimensional image, a three-dimensional scene and a problem text by a multi-modal feature backbone network; according to the problem-oriented two-dimensional view selection mechanism, a two-dimensional view most related to a problem is extracted by utilizing an image-text retrieval model, and the quality of visual clues contained in the two-dimensional view is improved; a two-dimensional-three-dimensional fused visual question and answer model is adopted, a double-branch Transform structure is utilized, a cross attention mechanism is adopted to carry out a visual multi-modal implicit alignment mechanism, and mutually enhanced two-dimensional-three-dimensional-language joint features are obtained; and the answer decoder is used for generating three-dimensional visual questions and answers based on the multi-modal joint features. According to the method, the knowledge from two-dimensional vision-language pre-training can be reserved, implicit alignment can be carried out among the visual modes, the detailed corresponding relation can be captured, and the three-dimensional visual question-answering performance of the system can be improved by combining the semantic related scene view provided by the question-oriented related view.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional visual question answering systems, and particularly relates to a method and system for answering questions by fusing multi-modal information according to a three-dimensional scene. Background Art

[0002] Three-dimensional visual question answering is a cutting-edge cross-field in the fields of computer vision and natural language processing, aiming to enable computer systems to understand and answer questions about three-dimensional scenes. Different from traditional two-dimensional visual question answering, three-dimensional visual question answering involves analyzing objects and environments in three-dimensional space, requiring the model to not only identify the objects in the scene, but also understand their positions, orientations, and mutual relationships in three-dimensional space, and perform reasoning and question answering accordingly. Three-dimensional visual question answering has broad application potential in multiple fields such as virtual reality, robot navigation, and smart home control, and particularly shows important value in providing a more natural human-computer interaction method and improving the machine's understanding ability of complex spatial environments.

[0003] The three-dimensional visual question answering task was first proposed by Azuma et al. in "ScanQA: 3D question answering for spatial scene understanding" in 2022. Existing representative methods include: the three-dimensional vision-language fusion module based on the Transformer-like structure proposed by Azuma et al., which combines visual features obtained by a three-dimensional object detector for visual question answering; the method proposed by Jin et al. in "Context-aware Alignment and Mutual Masking for 3D-Language Pre-training" and Zhu et al. in "3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment", which obtains self-supervised vision-language representations based on three-dimensional vision-language pre-training tasks and applies them to three-dimensional visual question answering; the CLIP-Guided method proposed by Parelli et al., which uses a pre-trained two-dimensional vision-language model and combines scene bird's-eye view features for graph-scene and text-scene multi-modal alignment to obtain better vision-language representations for three-dimensional visual question answering, and the Multi-CLIP method further proposed by Delitzas et al., which is an improved multi-modal alignment based on multiple bird's-eye views and contrastive alignment loss, etc.

[0004] Although existing 3D visual question answering systems have made progress in incorporating 3D modalities into visual question answering, the performance of current 3D visual question answering tasks is still limited, especially in generalizing to new 3D visual concepts. This limitation is mainly attributed to the scarcity of visual content in existing datasets. In popular 3D visual question answering datasets like ScanQA and SQA, only about 800 indoor scenes are used to address 3D visual question answering tasks. This challenge may be addressed by incorporating 2D vision-language models that are familiar with diverse visual concepts. Some previous 3D visual question answering methods enhance 3D scene representation by using scene-level feature alignment losses of 2D top-down images. However, two problems have emerged in these methods: First, the complexity and incompleteness of top-down views hinder the normal 2D vision-language model's understanding of 2D views, limiting the integration of 2D vision-language knowledge into the model; Second, by using image / scene-level rather than region / object-level representations, this global-level modality alignment method ignores the detailed 2D-3D correspondences; only using the computed features and ignoring the detailed vision-language relationships captured in the 2D vision-language model leads to insufficient utilization of pre-trained 2D vision-language models. Summary of the Invention

[0005] In view of the deficiencies of the prior art, the present invention proposes a 3D visual question answering method and system based on multimodal fusion.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A 3D visual question answering method for understanding and answering questions about 3D scenes, comprising the following steps:

[0008] (1) Question-oriented 2D view selection mechanism: For data samples of 3D scene, question, and answer triples, use a text-image retrieval model to select relevant 2D views that match the semantics of the question.

[0009] (2) 2D-3D fusion visual question answering model: Through a two-branch Transformer visual question answering model initialized based on a pre-trained 2D vision-language model, fuse 2D visual features, 3D visual features, and language features.

[0010] (3) Twin-Transformer 2D-3D modality fusion network: Adopt a cross-attention mechanism for implicit alignment of 2D-3D features to ensure tight fusion of intermediate features of the two visual modalities, obtaining multimodal vision-language fusion features.

[0011] (4) Answer decoder: Based on the visual - language fusion features obtained in (2) and (3), use a Transformer answer decoder initialized with a pre - trained two - dimensional visual - language model to generate a free - form answer.

[0012] Furthermore, in step (1), use the pre - trained image - text retrieval model BLIP to screen the two - dimensional views of the three - dimensional scene. This screening process is guided by the question, or use the GPT - 4 model to generate corresponding declarative sentences based on the question for guidance.

[0013] Furthermore, in step (2), use the pre - trained two - dimensional visual - language model BLIP to initialize the two - dimensional and three - dimensional visual question - answering models. Use their visual encoders and text encoders to encode the features of the two - dimensional views and question texts obtained in step (1) to obtain corresponding two - dimensional feature sequences and text feature sequences, and use a three - dimensional object detector to encode the three - dimensional scene into a three - dimensional object feature sequence within the scene; then introduce a two - branch Transformer visual question - answering model to fuse the two - dimensional visual - language features and three - dimensional visual - language features respectively. Here, the "two - dimensional visual - language features" refer to two - dimensional visual features and language features, and the "three - dimensional visual - language features" refer to three - dimensional visual features and language features.

[0014] Furthermore, in step (3), based on the two - branch Transformer visual question - answering model in step (2), mix the intermediate - layer encoded features of the two - dimensional or three - dimensional Transformer branches with the three - dimensional or two - dimensional visual features of each branch to enhance the two - dimensional - language features and three - dimensional - language features obtained in the previous steps, and obtain the fused two - dimensional - three - dimensional - language joint features of each branch. Here, the "two - dimensional - three - dimensional - language joint features" include the fused two - dimensional visual features, three - dimensional visual features, and language features.

[0015] Furthermore, in step (4), use the fused two - dimensional - three - dimensional - language joint features obtained in steps (2) and (3), generate an answer word sequence autoregressively through a Transformer language decoder, and use a language modeling loss function to optimize the three - dimensional visual question - answering model.

[0016] A three - dimensional visual question - answering system adopting the above - mentioned method, which includes:

[0017] A multi - modal feature backbone network for extracting features from two - dimensional images, three - dimensional scenes, and question text sequences, using a visual Transformer network, a VoteNet network, and a Transformer network structure respectively;

[0018] The question-oriented 2D view selection module is used to extract the 2D view most relevant to the question using the image-text retrieval model, and to increase the quality of the effective visual clues contained in the 2D view;

[0019] The 2D-3D fusion visual question answering module is used to use a dual-branch Transformer structure and a cross-attention mechanism to perform an implicit alignment mechanism of 2D-3D features to obtain 2D-3D-language joint features that enhance each other.

[0020] Answer decoder for generating free-form answers based on joint 2D-3D-language features for 3D visual question answering.

[0021] On the one hand, the present invention uses a question-oriented 2D view selection mechanism to select scene views that are semantically relevant to the question for model input to effectively integrate 2D information; on the other hand, a dual-branch Transformer model architecture that accepts 2D and 3D inputs is used to implement a simple and effective 2D-3D fusion strategy that compactly combines 2D and 3D representations. This dual-branch architecture retains as much knowledge from 2D vision-language pre-training as possible; in addition, the Twin-Transformer method is introduced to mix 2D and 3D visual representation sequences in two Transformers through cross-attention, allowing implicit alignment between the two modalities and capturing fine-grained 2D-3D correspondences, so that 2D and 3D representation sequences can enhance each other and improve the system's 3D visual question-answering performance.

[0022] The reason why the present invention has the above-mentioned inventive effects is that: on the one hand, through the question-oriented two-dimensional view selection mechanism, the 2D visual clues most relevant to the question can be captured, helping the three-dimensional visual question-answering system to capture effective visual clues; on the other hand, through the dual-branch Transformer architecture, the knowledge from the two-dimensional vision-language pre-training is retained as much as possible, and the cross-attention mechanism is used to implicitly align the two-dimensional-three-dimensional features, ensuring that the intermediate features of the two modalities are closely integrated, allowing implicit alignment between the visual modalities, and capturing detailed correspondences, so that the two-dimensional-three-dimensional visual features can enhance each other. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of the three-dimensional visual question answering method of the present invention. DETAILED DESCRIPTION

[0024] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] The present invention is a 3D visual question answering system based on multimodal fusion, and its process is as follows: Figure 1 As shown, the following steps are included:

[0026] (1) Problem-oriented 2D view selection

[0027] The present invention selects the two-dimensional views most relevant to the problem to provide more effective visual cues for subsequent processing. This method aims to effectively integrate the visual concepts required by the problem while filtering out irrelevant visual cues to improve the model's ability to interpret and respond to 3D scenes. Specifically, the present invention uses the BLIP image-text retrieval model to select the most suitable two-dimensional view from the view set, and this process is guided by the question or a question-based declarative sentence. The question-based declarative sentence is generated from the question using GPT-4.

[0028] (2) 2D-3D fusion visual question answering model

[0029] The visual question answering model structure of this method is as Figure 1 shown. Using the semantically relevant two-dimensional views obtained in (1), the present invention proposes a 2D-3D visual question answering model using a two-branch Transformer, which accepts two-dimensional images and 3D scenes and compactly utilizes two-dimensional vision language models. The two-branch 2D and 3D vision-language encoding backbone networks encode the two-dimensional image and 3D scene into per-block and per-object visual feature sequences, respectively, which are fused with the question text feature sequence to obtain 2D-vision language and 3D-vision language features. These two branches utilize the structure and initial parameters of the two-dimensional vision language model to obtain two-dimensional vision language knowledge from two-dimensional pre-training. The 3D branch is also initialized from the two-dimensional vision language, which can reduce the domain shift between the two-dimensional and three-dimensional features of the two branches, bringing the advantage of 2D-3D feature alignment.

[0030] Specifically, for each data sample (S, I, q, a), including a 3D scene, a relevant 2D image, a question, and an answer, the present invention first uses a 3D object detector to extract 3D object-level features h 3D . At the same time, the visual encoder in the two-dimensional VLM (vision language model) is also encoded to extract two-dimensional image patch-level features h 2D :

[0031] h 2D = Transformer VLM，I (I)

[0032] h 3D = VoteNet(S)

[0033] where Transformer VLM，I is the image encoder backbone initialized by the two-dimensional vision language model, and VoteNet is the 3D object detector backbone. After that, the question text features h q can be encoded:

[0034] h q = Transformer VLM，q (q)

[0035] where Transformer VLM，q is a text encoder for initializing a two - dimensional vision - language model.

[0036] Previous methods aligned image - level two - dimensional features with scene - level representations, thus ignoring the fine - grained relationships between image regions and scene objects. For example, scene - level representations globally pooled from all objects may lose the detailed semantics describing specific object attributes (e.g., red apple), but focus on higher - level semantics (e.g., living room). These methods also discard the structure and weights of the pre - trained two - dimensional vision - language model, thus losing the valuable two - dimensional vision - language relationships learned from two - dimensional pre - training.

[0037] In contrast, the method of the present invention maximally utilizes two - dimensional pre - training to identify diverse visual concepts. To fully exploit the capabilities of the pre - trained two - dimensional vision - language model, the aligned vision - language fusion encoder and language decoder from the two - dimensional vision - language model are directly selected instead of relying on pooled scene or image - level representations. This method allows the two - dimensional picture - patch - level feature h 2D and the three - dimensional object h 3D to be fused with the text feature h q :

[0038] h' 2D = Transformer VLM，2D (h q , h 2D )

[0039] h' 3D = Transformer VLM，3D (h q , h 3D )

[0040] where Transformer VLM，2D , Transformer VLM，3D is a two - dimensional / three - dimensional vision - language fusion encoder of the vision - language model, which fuses the features into a joint vision - language feature h' 2D , h' 3D . The three - dimensional branch has the same structure as the two - dimensional branch and is initialized with the parameters of the two - dimensional branch.

[0041] (3) Twin - Transformer Two - Dimensional - Three - Dimensional Modal Fusion Network

[0042] While a simple two-branch Transformer structure can capture two-dimensional and three-dimensional visual inputs, it may only allow limited interaction between two-dimensional and three-dimensional visual contexts. However, randomly adding additional modules to the two-dimensional vision-language fusion encoder may damage the pre-trained two-dimensional vision-language knowledge. Therefore, a compact but lightweight modality fusion mechanism between the two-dimensional and three-dimensional branches may be crucial for successful knowledge transfer and integration from two dimensions to three dimensions.

[0043] The present invention proposes a Twin-Transformer (cross-linked Transformer) structure that mixes two-dimensional / three-dimensional intermediate vision-language feature sequences with two-dimensional / three-dimensional visual feature sequences for cross-attention in the other branch, allowing for compact two-dimensional-three-dimensional interaction without changing the basic model architecture. In addition, in this way, each branch of each visual modality can perceive complementary visual cues from the other modality.

[0044] Specifically, after each layer of the Transformer in the two branches, the cross-attention input on one branch is augmented with the intermediate representation from the other branch, thereby facilitating a more refined fusion of two-dimensional and three-dimensional visual information without changing the basic model structure. Specifically, this method represents the two-dimensional or three-dimensional backbone Transformer VLM，2D , Transformer VLM，3D as an L-layer Transformer block where the modality x ∈ {2D, 3D}, and the Twin-Transformer structure is represented as follows:

[0045]

[0046]

[0047]

[0048] where || is the sequence concatenation operation, is the intermediate encoded feature sequence from the l-th layer of the two-dimensional and three-dimensional branches, Transformer TWIN，Xis a Twin-Transformer backbone network in two-dimensional or three-dimensional modalities. In Twin-Transformer, the intermediate features of two-dimensional and three-dimensional modalities are compactly integrated without changing the existing model architecture, thereby retaining two-dimensional visual language knowledge through the use of feature connections and an additional cross-attention mechanism. In addition, in order to retain the initial cross-attention between the two-dimensional visual and language feature sequences in the three-dimensional branch, the present invention initializes the three-dimensional branch with the same parameters from the two-dimensional branch. By end-to-end micro-adjusting the entire model and using three-dimensional-visual question answering supervision, key two-dimensional visual language knowledge can be retained and refined, similar to fine-tuning a two-dimensional visual language model for two-dimensional visual question answering.

[0049] (4) Answer decoder

[0050] After obtaining the two-dimensional-three-dimensional fusion features, the present invention uses the language decoder of BLIP to generate free-form answers. This method is particularly advantageous for generalization to unseen answers and is also applicable to datasets that require long text outputs, such as ScanQA. The shared language decoder Transformer VLM，D After receiving the fused visual-language features, the word sequence probabilities are calculated as follows:

[0051] p 2D = Transformer VLM，D (h q , h′ 2D )

[0052] p 3D = Transformer VLM，D (h q , h′ 3D )

[0053] where p 2D ∈ [0, 1] N×C , p 3D ∈ [0, 1] N×C represent the predicted probability distributions of the C candidate text words at each position of the N-length answers from the two-dimensional and three-dimensional branches, respectively. The entire visual question answering model (including the two-branch Transformer, Twin-Transformer, answer decoder) is optimized using a perplexity-based loss:

[0054]

[0055] where λ is a loss function balance factor, a1,..., a N is the answer word index sequence, is the probability that the predicted word index at the i-th position is a i for modality X ∈ {2D, 3D}.

[0056] This method enables the co-optimization of 2D and 3D branches. During inference, the predicted probabilities of the two branches are simply added together to generate the final answer. In addition, this method narrows down the prediction scope by evaluating the total perplexity of each answer candidate, restricting it within the answer vocabulary of the dataset.

[0057] The following experimental results show that, compared with existing methods, the 3D visual question answering method of the present invention can achieve higher question answering accuracy.

[0058] This embodiment uses two publicly available datasets, ScanQA and SQA. The ScanQA dataset contains 41,000 question-answer pairs, covering 800 indoor 3D scenes in the ScanNet 3D scene dataset (including two test sets, one with real object annotations and one without real object annotations). The SQA dataset, based on 650 scenes in ScanNet, contains 33,000 question-answer annotations.

[0059] Existing method 1: The method proposed by Azuma et al. in the literature "ScanQA: 3D Question Answering for Spatial Scene Understanding".

[0060] Existing method 2: The method proposed by Jin et al. in the literature "Context-aware Alignment and Mutual Masking for 3D-Language Pre-training".

[0061] Existing method 3: The method proposed by Zhu et al. in the literature "3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment".

[0062] The present invention: The method of this embodiment.

[0063] Answer accuracy is used in the experiment to evaluate the visual question answering performance. This metric measures the percentage of exact matches between the model's predicted answers and the annotated answers. For the ScanQA dataset, due to the answers usually being long texts and in free format, text similarity metrics (BLEU-1, BLEU-4, ROUGE, METEOR, and CIDEr scores) are also used, which measure the similarity of the word sequences between the model's predicted answers and the annotated answers.

[0064] As can be seen from Table 1, the present invention has surpassed the previous methods in terms of performance and achieved significant improvements. Without using the 3D vision-language pre-training adopted in the second and third existing methods, the present invention has surpassed these methods based on 3D vision-language pre-training by introducing an efficient 2D-to-3D knowledge transfer and fusion scheme. On the one hand, the present invention proposes a problem-oriented 2D view selection mechanism to capture the 2D visual cues most relevant to the problem, helping the 3D visual question answering system capture effective visual cues; on the other hand, the present invention proposes a dual-branch Transformer architecture to retain as much knowledge as possible from 2D vision-language pre-training, while adopting a cross-attention mechanism for implicit alignment of 2D-3D features, ensuring the tight fusion of intermediate features of the two modalities, allowing implicit alignment between different visual modalities, enabling the 2D-3D visual features to enhance each other, and improving the performance of 3D visual question answering.

[0065] Table 1. Comparative experimental results with existing methods

[0066]

[0067] Based on the same inventive concept, another embodiment of the present invention provides a video temporal localization system adopting the above method, which includes:

[0068] A multi-modal feature backbone network for extracting features from 2D images, 3D scenes, and question text sequences, adopting a vision Transformer network, a VoteNet network, and a Transformer network structure respectively;

[0069] A problem-oriented 2D view selection module for using an image-text retrieval model to extract the 2D view most relevant to the question and increasing the quality of effective visual cues contained in the 2D view;

[0070] A 2D-3D fusion visual question answering module for using a dual-branch Transformer structure and adopting a cross-attention mechanism for implicit alignment of 2D-3D features to obtain 2D-3D-language joint features that enhance each other in 2D and 3D;

[0071] An answer decoder for generating a free-form answer based on the 2D-3D-language joint features to perform 3D visual question answering.

[0072] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (such as a computer, a server, a smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for performing the steps in the method of the present invention.

[0073] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disc), and the computer-readable storage medium stores a computer program, which when executed by a computer, implements each step of the method of the present invention.

[0074] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these changes and modifications.

Claims

1. A three-dimensional visual question answering method, characterized in that, Including the following steps: For the data samples of three-dimensional scenes, questions, and answer triples, use an image-text retrieval model to select relevant two-dimensional views that match the semantics of the questions; Through a dual-branch Transformer visual question answering model initialized based on a pre-trained two-dimensional vision-language model, fuse two-dimensional visual features, three-dimensional visual features, and language features; Adopt a cross-attention mechanism for implicit alignment of two-dimensional and three-dimensional features to ensure the tight fusion of intermediate features of the two modalities and obtain multi-modal vision-language fusion features; Based on the obtained vision-language fusion features, use a Transformer answer decoder initialized based on a pre-trained two-dimensional vision-language model to generate free-form answers.

2. The method according to claim 1, characterized in that, The step of using an image-text retrieval model to select relevant two-dimensional views that match the semantics of the questions includes: using the pre-trained image-text retrieval model BLIP to screen the two-dimensional views of the three-dimensional scene, and this screening process is guided by the question, or using the corresponding declarative sentence generated by the GPT-4 model based on the question for guidance.

3. The method according to claim 1, wherein The step of fusing two-dimensional visual features, three-dimensional visual features, and language features through a dual-branch Transformer visual question answering model initialized based on a pre-trained two-dimensional vision-language model includes: using the pre-trained two-dimensional vision-language model BLIP to initialize the two-dimensional and three-dimensional visual question answering branches, using its visual encoder and text encoder to encode the features of the two-dimensional views and texts obtained, to obtain the corresponding two-dimensional feature sequences and text feature sequences, and using a three-dimensional object detector to encode the three-dimensional scene into an object feature sequence within the scene; subsequently, introduce a dual-branch Transformer visual question answering model to fuse two-dimensional-language features and three-dimensional-language features respectively.

4. The method according to claim 1, characterized in that, The implicit alignment of 2D-3D features through the cross-attention mechanism includes: adopting a Twin-Transformer structure to mix the 2D / 3D intermediate VL tokens with the 2D / 3D visual tokens for cross-attention in the other branch, allowing for compact 2D-3D interaction without changing the basic model architecture; in this way, each branch of the visual modality perceives complementary visual cues from the other modality; after each layer of the Transformer in both branches, the cross-attention input on one branch is augmented with the intermediate representation from the other branch, thus facilitating a more refined fusion of 2D and 3D visual information without changing the basic model structure; initializing the 2D / 3D visual-language fusion encoder backbone Transformer with a 2D visual-language model VLM，2D , Transformer VLM，3D is denoted as an L-layer Transformer block where modality X ∈ {2D, 3D}, and the Twin-Transformer structure is represented as follows: where || is the sequence concatenation operation, is the intermediate encoded feature sequence of the l-th layer from the two-dimensional and three-dimensional branches, and Transformer TWIN，X is the Twin-Transformer backbone network of the two-dimensional or three-dimensional modality, h q , h 2D , h 3D are the encoded problem text feature sequence, two-dimensional visual feature sequence, and three-dimensional visual feature sequence, respectively.

5. The method according to claim 1, characterized in that, The answer decoder uses the obtained fused 2D-3D-language joint features and a shared language decoder Transformer VLM，D After receiving the fused vision-language features, it calculates the word sequence probabilities as follows: where p 2D ∈ [0, 1] N×C , p 3D ∈ [0, 1] N×C respectively represent the predicted probability distributions of the C candidate text words at each position of the answers of the N lengths from the two-dimensional and three-dimensional branches.

6. The method according to claim 5, wherein The entire visual question answering model is optimized using a perplexity-based loss function, and the adopted perplexity-based loss function is: where λ is a loss function balance factor, a1, ..., a N is the sequence of answer word indices, is the probability that the model predicts the word index as a i at the i-th position for modality X ∈ {2D, 3D}.

7. The method according to claim 1, wherein The knowledge transfer from the pre-trained two-dimensional vision-language model includes: using the two-dimensional visual encoder and language encoder of the pre-trained two-dimensional vision-language model to initialize the two-dimensional view encoder and question text encoder of the three-dimensional visual question answering system, using the vision-language fusion encoder of the pre-trained two-dimensional vision-language model to initialize the two-dimensional-language / three-dimensional-language fusion encoder of the three-dimensional visual question answering system, and using the language decoder of the pre-trained two-dimensional vision-language model to initialize the answer decoder.

8. A three-dimensional visual question answering system, characterized in that Including: A multi-modal feature backbone network for extracting features from two-dimensional images, three-dimensional scenes, and question text sequences, adopting visual Transformer network, VoteNet network, and Transformer network structures respectively; A question-oriented two-dimensional view selection module for using an image-text retrieval model to extract the two-dimensional views most relevant to the question and improve the quality of effective visual cues contained in the two-dimensional views; A two-dimensional-three-dimensional fusion visual question answering module for using a dual-branch Transformer structure and adopting a cross-attention mechanism for implicit alignment of two-dimensional and three-dimensional features to obtain two-dimensional-three-dimensional mutually enhanced two-dimensional-three-dimensional-language joint features; Answer decoder, generating free-form answers based on two-dimensional-three-dimensional-language joint features for three-dimensional visual question answering.

9. A computer device, characterized in that, Comprising a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 7 is implemented.

Images