Visual question and answer method based on multi-modal perception and related equipment thereof

Abstract

The invention belongs to the field of artificial intelligence, is applied to the field of financial question answering, and relates to a visual question answering method based on multi-modal perception, which comprises the following steps: receiving data to be answered, the data to be answered comprising a target text question and target visual data; performing coding operation on the target visual data based on a preset coding model to obtain a target feature; inputting the target feature into a preset causal transformation model, and performing causal transformation operation to obtain an output target causal feature; and obtaining a target answer based on the target causal feature, the target text question and a preset target autoregressive visual question and answer model. The invention further provides a visual question answering device based on multi-modal perception, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the target autoregression visual question and answer model can be stored in a block chain. The accuracy of visual answering is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a visual question answering method based on multimodal perception and related equipment. Background Art

[0002] Visual Question Answering (VQA), as a cutting-edge research direction in the field of artificial intelligence, aims to build intelligent systems that can understand visual content and communicate in natural language. This task plays a key role in simulating human cognitive processes and provides a powerful tool for processing and interpreting multimedia data in the real world. VQA models have made significant progress in feature fusion efficiency and interpretability. However, the VQA field still faces some challenging issues:

[0003] Although existing VQA models have made many innovations in feature fusion technology, the semantic gap between visual content and natural language description is still difficult to bridge effectively. This semantic mismatch not only affects the model's ability to comprehensively understand visual and language information, but also restricts the model's robustness and the interpretability of generated answers. This means that in some cases, the model may not be able to accurately understand the key information in the visual content, resulting in inaccurate generated answers. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a visual question answering method based on multimodal perception and related equipment to improve the accuracy of visual answers.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a visual question answering method based on multimodal perception, which adopts the following technical solution:

[0006] Receiving data to be answered, wherein the data to be answered includes a target text question and target visual data;

[0007] Performing encoding operation on the target visual data based on a preset encoding model to obtain target features;

[0008] Inputting the target feature into a preset causal conversion model, performing a causal conversion operation, and obtaining an output target causal feature;

[0009] A target answer is obtained based on the target causal feature, the target text question and a preset target autoregressive visual question answering model.

[0010] Furthermore, the step of obtaining a target answer based on the target causal feature, the target text question and a preset target autoregressive visual question answering model includes:

[0011] Combining the target causal feature with the target text question to obtain a target multimodal sequence;

[0012] Inputting the target multimodal sequence into a preset target autoregressive visual question answering model to obtain an output answer sequence;

[0013] The answer sequence is input into a preset decoder to obtain the target answer.

[0014] Furthermore, the step of combining the target causal feature with the target text question to obtain a target multimodal sequence includes:

[0015] Inputting the target text question into a preset encoding model to obtain output target text features;

[0016] The target causal feature is combined with the text feature to obtain the target multimodal sequence.

[0017] Furthermore, the step of combining the target causal feature with the text feature to obtain the target multimodal sequence includes:

[0018] Performing a splicing operation on the target causal feature and the text feature to obtain a spliced ​​feature;

[0019] A dimensionality reduction operation is performed on the concatenated features to obtain the target multimodal sequence.

[0020] Furthermore, before the step of obtaining a target answer based on the target causal feature, the target text question and a preset target autoregressive visual question answering model, the method further includes:

[0021] receiving training data, the training data comprising arbitrarily interwoven image data, text questions, and video data;

[0022] Inputting the arbitrarily interwoven image data, text questions, and video data into a pre-trained coding model, obtaining visual features corresponding to the image data, text features corresponding to the text, and video causal embedding corresponding to the video data output by the coding model;

[0023] Inputting the visual features and the video causal embedding into a preset causal transformation model respectively, performing a causal transformation operation, and obtaining output visual causal features and video causal features respectively;

[0024] Combining the visual causal feature, the video causal feature, and the text feature to obtain a multimodal sequence;

[0025] Based on the multimodal sequence, an autoregressive modeling operation is performed on a preset visual question answering model to obtain the target autoregressive visual question answering model.

[0026] Furthermore, the causal conversion model includes a causal self-attention layer, a cross attention layer and a feedforward layer, and the steps of inputting the visual features and the video causal embedding into the causal conversion model respectively, performing a causal conversion operation, and obtaining output visual causal features and video causal features respectively include:

[0027] Inputting the visual features and the video causal embedding into the causal self-attention layer to obtain an output visual representation and the video representation;

[0028] The visual representation and the video representation are both regarded as queries, and the visual features and the video causal embedding are both regarded as keys and values, which are input into the cross attention layer to obtain output aggregated visual features and aggregated video features;

[0029] The aggregated visual features and the aggregated video features are input into the feed-forward layer to obtain the outputted visual causal features and the video causal features.

[0030] Furthermore, the step of combining the visual causal feature, the video causal feature and the text feature to obtain a multimodal sequence includes:

[0031] Setting a start mark and an end mark for each of the visual causal features and the video causal features to obtain marked visual causal features and video causal features;

[0032] The labeled visual causal features and video causal features are combined with the text question to obtain the multimodal sequence.

[0033] In order to solve the above technical problems, the embodiment of the present application further provides a visual question-answering device based on multimodal perception, which adopts the following technical solution:

[0034] A receiving module, used for receiving data to be answered, wherein the data to be answered includes a target text question and target visual data;

[0035] An encoding module, used for performing an encoding operation on the target visual data based on a preset encoding model to obtain target features;

[0036] A conversion module, used to input the target feature into a preset causal conversion model, perform a causal conversion operation, and obtain an output target causal feature;

[0037] An output module is used to obtain a target answer based on the target causal feature, the target text question and a preset target autoregressive visual question answering model.

[0038] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0039] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned visual question answering method based on multimodal perception when executing the computer-readable instructions.

[0040] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0041] A computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by a processor, implement the steps of the above-mentioned visual question answering method based on multimodal perception.

[0042] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0043] This application uses a preset encoding model to encode visual data, which can accurately extract key features in visual data and enhance the understanding of visual data. The target features are input into the preset causal transformation model, and the causal transformation operation is performed to obtain the output target causal features, thereby converting 2D visual data into a 1D causal sequence. The causal transformation model can more naturally capture the spatial and semantic relationships in visual data, reduce the interference of redundant information, and improve the performance and efficiency of the model in visual question-answering tasks. By combining the target causal features, text questions, and the preset target autoregressive visual question-answering model, the target answer can be generated accurately and quickly to meet the user's real-time and accuracy requirements for the visual question-answering system. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0046] Figure 2 A flowchart of an embodiment of a visual question answering method based on multimodal perception according to the present application;

[0047] Figure 3 is a structural schematic diagram of an embodiment of a visual question-answering device based on multimodal perception according to the present application;

[0048] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0050] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0051] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0052] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0053] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0054] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0055] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0056] It should be noted that the visual question answering method based on multimodal perception provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the visual question answering device based on multimodal perception is generally arranged in the server / terminal device.

[0057] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0058] Continue to refer Figure 2 , shows a flow chart of an embodiment of a visual question answering method based on multimodal perception according to the present application. The visual question answering method based on multimodal perception comprises the following steps:

[0059] Step S201, receiving data to be answered, wherein the data to be answered includes a target text question and target visual data.

[0060] In this embodiment, the visual data may be an image or a video. The data to be answered includes a text question and visual data for visual question answering.

[0061] In this embodiment, the electronic device (e.g. Figure 1 shown Server / Terminal Equipment ) can receive the data to be answered via a wired connection or a wireless connection. It should be noted that the above wireless connection methods may include but are not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0062] Step S202: encoding the target visual data based on a preset encoding model to obtain target features.

[0063] In this embodiment, the coding model can accurately capture the core elements in the visual data, convert complex visual information into concise and efficient feature representations, and improve the efficiency of subsequent processing. The coding model preset in this application can be an EVA-CLIP model. The EVA-CLIP model is a model improved based on the contrastive language image pre-training (CLIP) technology. It significantly improves the training efficiency and effect of CLIP by introducing new representation learning, optimization and enhancement technologies. As a pre-trained coding model, the coding model can extract general visual representations containing high-level semantic information from visual signals. The coding model is not only good at processing visual data, but also has cross-modal understanding capabilities. The coding model has a strong generalization ability. Even in the face of unseen visual data, the model can perform effective feature extraction and reasoning based on the knowledge it has learned, and can output accurate target features.

[0064] Step S203: input the target feature into a preset causal conversion model, perform a causal conversion operation, and obtain an output target causal feature.

[0065] In this embodiment, the target feature is input into a preset causal transformation model, a causal transformation operation is performed, and the output target causal feature is obtained, so as to realize the conversion of 2D visual data into a 1D causal sequence. The causal transformation model can use the Causal Transformer, which is not specifically limited in this application. The causal transformation model can more naturally capture the spatial and semantic relationships in the visual data, reduce the interference of redundant information, and improve the performance and efficiency of the model in the visual question answering task.

[0066] Step S204, obtaining a target answer based on the target causal feature, the target text question and a preset target autoregressive visual question answering model.

[0067] In this embodiment, the target autoregressive visual question answering model can be obtained by modeling based on LLM (Large Language Model). For an introduction to the specific target autoregressive visual question answering model, please refer to the following training process of the target autoregressive visual question answering model. This application can accurately and quickly generate the target answer by combining the target causal features, text questions and the preset target autoregressive visual question answering model.

[0068] This application uses a preset encoding model to encode visual data, which can accurately extract key features in visual data and enhance the understanding of visual data. The target features are input into the preset causal transformation model, and the causal transformation operation is performed to obtain the output target causal features, thereby converting 2D visual data into a 1D causal sequence. The causal transformation model can more naturally capture the spatial and semantic relationships in visual data, reduce the interference of redundant information, and improve the performance and efficiency of the model in visual question-answering tasks. By combining the target causal features, text questions, and the preset target autoregressive visual question-answering model, the target answer can be generated accurately and quickly to meet the user's real-time and accuracy requirements for the visual question-answering system.

[0069] In some optional implementations of this embodiment, the step of combining the target causal feature with the text feature to obtain the target multimodal sequence includes:

[0070] The target causal feature and the text feature are concatenated to obtain a concatenated

[0071] feature;

[0072] A dimensionality reduction operation is performed on the concatenated features to obtain the target multimodal sequence.

[0073] · In this embodiment, the target causal features are combined with the text features through a splicing operation, thereby realizing the integration of cross-modal information. This integration can make full use of the complementary information in different modal data and improve the comprehensiveness and accuracy of the data. The dimensionality reduction operation aims to retain the main information and key features in the data while removing redundancy and noise. When performing a dimensionality reduction operation, a dimensionality reduction method can be selected according to actual needs, such as a feature extraction method, which represents the original data by constructing new features, thereby realizing dimensionality reduction. More representative features can be extracted, thereby improving the performance of subsequent models. The present application performs a dimensionality reduction operation on the splicing features, which can significantly reduce the dimension of the data, thereby reducing computational complexity and storage requirements.

[0074] In some optional implementations of this embodiment, in step S204, the step of obtaining a target answer based on the target causal feature, the target text question and the preset target autoregressive visual question answering model includes:

[0075] Combining the target causal feature with the target text question to obtain a target multimodal sequence;

[0076] Inputting the target multimodal sequence into a preset target autoregressive visual question answering model to obtain an output answer sequence;

[0077] The answer sequence is input into a preset decoder to obtain the target answer.

[0078] In this embodiment, the target causal features capture the key causal relationships in the visual data, which help the model understand the visual content and answer related questions. Combining these causal relationships with the target text questions forms a more complete and accurate target multimodal sequence, thereby improving the question-answering system's ability to understand the target text questions and the accuracy of analyzing the target causal features.

[0079] Specifically, the step of combining the target causal feature with the target text question to obtain a target multimodal sequence includes:

[0080] The target text question is input into the preset encoding model to obtain the output target text.

[0081] Mark text features;

[0082] The target causal feature is combined with the text feature to obtain the target multimodal sequence.

[0083] · In this embodiment, the encoding model converts the target text question into a vector representation, namely the target text feature. By combining the target causal feature with the text feature, the target multimodal sequence is obtained, forming a target multimodal sequence containing richer information, which is helpful for the question answering of the subsequent model.

[0084] · In some optional implementations of this embodiment, before step S204, that is, before the step of obtaining the target answer based on the target causal feature, the target text question and the preset target autoregressive visual question answering model, the following is further included:

[0085] receiving training data, the training data comprising arbitrarily interwoven image data, text questions, and video data;

[0086] Input the arbitrarily interwoven image data, text questions and video data into a pre-trained coding model to obtain visual features corresponding to the image data output by the coding model, text features corresponding to the text, and video causal embedding corresponding to the video data; input the visual features and the video causal embedding into a preset causal transformation model, respectively, perform causal transformation operations, and respectively obtain output visual causal features and video causal features;

[0087] Combining the visual causal feature, the video causal feature, and the text feature to obtain a multimodal sequence;

[0088] Based on the multimodal sequence, an autoregressive modeling operation is performed on a preset visual question answering model to obtain the target autoregressive visual question answering model.

[0089] In this embodiment, the training data includes data of multiple modalities such as images, texts and videos, which enables the model to recognize and process different multimodal data, enrich the training data, and enable the model to have a more powerful multimodal understanding and generation capability, so that the model can handle a wider range of application scenarios, such as video question answering, social media content analysis, etc. Through the encoding model, the visual features of the image data, the text features of the text questions, and the video causal embedding of the video data can be extracted respectively, providing a rich and accurate information basis for the subsequent multimodal sequence construction and visual question answering model training. By performing causal conversion operations on the visual features and video causal embeddings through the causal conversion model, the causal relationship in the data can be further mined and utilized. Using the causal conversion model to convert 2D image data or video data into a 1D causal sequence is more in line with the human perception of images / videos, which helps the model to better understand and process the content of the image / video. The introduction of this causal relationship helps the model to better understand the inherent logic and connection between the data, thereby improving the accuracy and robustness of the visual question answering model. Combining visual causal features, video causal features and text features to obtain a multimodal sequence. This way of constructing multimodal sequences not only retains the rich information in the original data, but also enhances the correlation and consistency between data by introducing causal relationships. This multimodal sequence is of great significance for the training of visual question answering models and helps to improve the generalization ability and performance of the model. The multimodal sequence is input into the LLM (Large Language Model) of multimodal modeling for unified autoregressive modeling to obtain the target autoregressive visual question answering model. Taking LLaMA as an example, although it is a language model focused on text processing, the multimodal LLM inspired by it can be extended to process data in other modalities such as images and audio. Compared with traditional grid sequence autoregressive modeling, the causal transformation model can capture the spatial and semantic relationships of images more naturally, so the multimodal data formed is of higher quality, which can improve the performance of the target autoregressive visual question answering model in image / video generation and understanding tasks. In summary, through the above processing, the target autoregressive visual question answering model obtained can make full use of the information in multimodal data and mine and utilize the causal relationships in the data. This enables the model to more accurately understand questions, analyze image and video content, and generate more accurate answers when handling visual question answering tasks, thereby improving the accuracy of the visual question answering model.

[0090] In some optional implementations of this embodiment, the causal conversion model includes

[0091] Causal self-attention layer, cross-attention layer and feed-forward layer, the visual features and

[0092] The video causal embedding is respectively input into the causal conversion model, and the causal conversion operation is performed to obtain the output visual causal features and video causal features respectively, including:

[0093] Inputting the visual features and the video causal embedding into the causal self-attention layer to obtain an output visual representation and the video representation;

[0094] The visual representation and the video representation are both regarded as queries, and the visual features and the video causal embedding are both regarded as keys and values, which are input into the cross attention layer to obtain output aggregated visual features and aggregated video features;

[0095] The aggregated visual features and the aggregated video features are input into the feed-forward layer to obtain the outputted visual causal features and the video causal features.

[0096] · In this embodiment, the causal self-attention layer is mainly used to process the causal relationship in the sequence data. In the causal conversion model, it receives visual features and video causal embedding as input, and calculates the correlation between these features through the self-attention mechanism. Through the self-attention mechanism, the causal self-attention layer can capture the causal relationship between visual features and video causal embedding, which helps the model better understand the dynamic changes and internal logic in the data. Unlike the recurrent neural network (RNN), the causal self-attention layer can process various positions in the sequence data in parallel, thereby improving the computational efficiency. The cross attention layer is mainly used to realize information exchange and fusion between different modalities. In the causal conversion model, it takes the visual representation and video representation as queries, and the visual features and video causal embedding as keys and values ​​for cross attention calculation. In this way, the model can integrate complementary information from different modalities to form a more comprehensive and accurate feature representation. The feedforward layer is a fully connected neural network that receives the output from the cross attention layer and further processes and transforms it. In the causal conversion model, the feedforward layer can perform linear transformation and nonlinear activation on the input features to generate more flexible and diverse output features.

[0097] In some optional implementations of this embodiment, the step of combining the visual causal feature, the video causal feature, and the text feature to obtain a multimodal sequence includes:

[0098] Setting a start mark and an end mark for each of the visual causal features and the video causal features to obtain marked visual causal features and video causal features;

[0099] The labeled visual causal features and video causal features are combined with the text question to obtain the multimodal sequence.

[0100] In this embodiment, the existence of the start marker and the end marker ensures that each visual causal feature and video causal feature has a clear boundary in the sequence. This helps the model to more accurately identify and utilize information of different features when processing multimodal sequences, thereby improving the distinguishability and utilization of features.

[0101] This application uses a preset encoding model to encode visual data, which can accurately extract key features in visual data and enhance the understanding of visual data. The target features are input into the preset causal transformation model, and the causal transformation operation is performed to obtain the output target causal features, thereby converting 2D visual data into a 1D causal sequence. The causal transformation model can more naturally capture the spatial and semantic relationships in visual data, reduce the interference of redundant information, and improve the performance and efficiency of the model in visual question-answering tasks. By combining the target causal features, text questions, and the preset target autoregressive visual question-answering model, the target answer can be generated accurately and quickly to meet the user's real-time and accuracy requirements for the visual question-answering system.

[0102] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned target autoregressive visual question answering model, the above-mentioned target autoregressive visual question answering model can also be stored in a node of a blockchain.

[0103] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0104] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0105] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning. This application can be applied to the field of smart government affairs, thereby promoting intelligence.

[0106] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0107] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0108] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a visual question-answering device based on multimodal perception, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0109] like Figure 3 As shown, the visual question answering device 300 based on multimodal perception described in this embodiment includes: a receiving module 301, an encoding module 302, a conversion module 303 and an output module 304. Among them:

[0110] The receiving module 301 is used to receive data to be answered, wherein the data to be answered includes a target text question and target visual data;

[0111] The encoding module 302 is used to perform an encoding operation on the target visual data based on a preset encoding model to obtain target features;

[0112] The conversion module 303 is used to input the target feature into a preset causal conversion model, perform a causal conversion operation, and obtain an output target causal feature;

[0113] The output module 304 is used to obtain a target answer based on the target causal feature, the target text question and a preset target autoregressive visual question answering model.

[0114] In this embodiment, the present application uses a preset encoding model to encode visual data, which can accurately extract key features in visual data and enhance the understanding of visual data. The target features are input into the preset causal transformation model, and the causal transformation operation is performed to obtain the output target causal features, so as to realize the conversion of 2D visual data into a 1D causal sequence. The causal transformation model can more naturally capture the spatial and semantic relationships in visual data, reduce the interference of redundant information, and improve the performance and efficiency of the model in visual question-answering tasks. By combining the target causal features, text questions and the preset target autoregressive visual question-answering model, the target answer can be generated accurately and quickly to meet the user's real-time and accuracy requirements for the visual question-answering system.

[0115] In some optional implementations of this embodiment, the encoding module 302 is further used to input the target visual data into a pre-trained encoding model to obtain target features output by the encoding model.

[0116] In some optional implementations of this embodiment, the output module 304 includes a combining submodule, a first input submodule, and a second input submodule, wherein:

[0117] A combining submodule, used for combining the target causal feature with the target text question to obtain a target multimodal sequence;

[0118] A first input submodule, used for inputting the target multimodal sequence into a preset target autoregressive visual question answering model to obtain an output answer sequence;

[0119] The second input submodule is used to input the answer sequence into a preset decoder to obtain a target answer.

[0120] Specifically, the combination submodule also includes an input unit and a combination unit, wherein: the input unit is used to input the target text problem into a preset encoding model to obtain the output target text features; the combination unit is used to combine the target causal features with the text features to obtain the target multimodal sequence.

[0121] The combining unit is further used to perform a splicing operation on the target causal feature and the text feature to obtain a spliced ​​feature; and perform a dimensionality reduction operation on the spliced ​​feature to obtain the target multimodal sequence.

[0122] In some optional implementations of this embodiment, the visual question-answering device based on multimodal perception further includes a training module, which includes a receiving submodule, a third input submodule, a causal conversion submodule, a data combination submodule, and a modeling submodule. Among them:

[0123] A receiving submodule, configured to receive training data, wherein the training data includes arbitrarily interwoven image data, text questions, and video data;

[0124] A third input submodule, configured to input the arbitrarily interwoven image data, text questions, and video data into a pre-trained coding model, and obtain visual features corresponding to the image data, text features corresponding to the text, and video causal embedding corresponding to the video data output by the coding model;

[0125] A causal conversion submodule, used to input the visual features and the video causal embedding into a preset causal conversion model, perform a causal conversion operation, and obtain output visual causal features and video causal features respectively;

[0126] A data combining submodule, used for combining the visual causal features, the video causal features and the text features to obtain a multimodal sequence;

[0127] The modeling submodule is used to perform an autoregressive modeling operation on a preset visual question answering model based on the multimodal sequence to obtain the target autoregressive visual question answering model.

[0128] In some optional implementations of this embodiment, the causal conversion model includes a causal self-attention layer, a cross-attention layer and a feedforward layer, and the causal conversion submodule includes an input unit, an aggregation unit and a feedforward unit, wherein:

[0129] An input unit, configured to input the visual features and the video causal embedding into the causal self-attention layer to obtain an output visual representation and the video representation;

[0130] an aggregation unit, configured to regard both the visual representation and the video representation as queries, regard both the visual features and the video causal embedding as keys and values, input the results into the cross attention layer, and obtain output aggregated visual features and aggregated video features;

[0131] A feedforward unit is used to input the aggregated visual features and the aggregated video features into the feedforward layer to obtain the output visual causal features and video causal features.

[0132] In some optional implementations of this embodiment, the data combining submodule includes a marking unit and a combining unit, wherein:

[0133] A marking unit, used for setting a start mark and an end mark for each of the visual causal features and the video causal features to obtain marked visual causal features and video causal features;

[0134] A combining unit is used to combine the visual causal features and the video causal features of the mark with the text question to obtain the multimodal sequence.

[0135] This application uses a preset encoding model to encode visual data, which can accurately extract key features in visual data and enhance the understanding of visual data. The target features are input into the preset causal transformation model, and the causal transformation operation is performed to obtain the output target causal features, thereby converting 2D visual data into a 1D causal sequence. The causal transformation model can more naturally capture the spatial and semantic relationships in visual data, reduce the interference of redundant information, and improve the performance and efficiency of the model in visual question-answering tasks. By combining the target causal features, text questions, and the preset target autoregressive visual question-answering model, the target answer can be generated accurately and quickly to meet the user's real-time and accuracy requirements for the visual question-answering system.

[0136] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0137] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0138] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0139] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of a visual question answering method based on multimodal perception, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0140] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the visual question answering method based on multimodal perception.

[0141] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0142] This application uses a preset encoding model to encode visual data, which can accurately extract key features in visual data and enhance the understanding of visual data. The target features are input into the preset causal transformation model, and the causal transformation operation is performed to obtain the output target causal features, thereby converting 2D visual data into a 1D causal sequence. The causal transformation model can more naturally capture the spatial and semantic relationships in visual data, reduce the interference of redundant information, and improve the performance and efficiency of the model in visual question-answering tasks. By combining the target causal features, text questions, and the preset target autoregressive visual question-answering model, the target answer can be generated accurately and quickly to meet the user's real-time and accuracy requirements for the visual question-answering system.

[0143] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the visual question answering method based on multimodal perception as described above.

[0144] This application uses a preset encoding model to encode visual data, which can accurately extract key features in visual data and enhance the understanding of visual data. The target features are input into the preset causal transformation model, and the causal transformation operation is performed to obtain the output target causal features, thereby converting 2D visual data into a 1D causal sequence. The causal transformation model can more naturally capture the spatial and semantic relationships in visual data, reduce the interference of redundant information, and improve the performance and efficiency of the model in visual question-answering tasks. By combining the target causal features, text questions, and the preset target autoregressive visual question-answering model, the target answer can be generated accurately and quickly to meet the user's real-time and accuracy requirements for the visual question-answering system.

[0145] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0147] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A visual question answering method based on multimodal perception, characterized in that: The steps include: Receiving data to be answered, wherein the data to be answered includes a target text question and target visual data; Performing encoding operation on the target visual data based on a preset encoding model to obtain target features; Inputting the target feature into a preset causal conversion model, performing a causal conversion operation, and obtaining an output target causal feature; A target answer is obtained based on the target causal feature, the target text question and a preset target autoregressive visual question answering model.

2. The visual question answering method based on multimodal perception according to claim 1, characterized in that: The step of obtaining a target answer based on the target causal feature, the target text question and a preset target autoregressive visual question answering model comprises: Combining the target causal feature with the target text question to obtain a target multimodal sequence; Inputting the target multimodal sequence into a preset target autoregressive visual question answering model to obtain an output answer sequence; The answer sequence is input into a preset decoder to obtain the target answer.

3. The visual question answering method based on multimodal perception according to claim 2, characterized in that: The step of combining the target causal feature with the target text question to obtain a target multimodal sequence includes: Inputting the target text question into a preset encoding model to obtain output target text features; The target causal feature is combined with the text feature to obtain the target multimodal sequence.

4. The visual question answering method based on multimodal perception according to claim 3, characterized in that: The step of combining the target causal feature with the text feature to obtain the target multimodal sequence includes: Performing a splicing operation on the target causal feature and the text feature to obtain a spliced ​​feature; A dimensionality reduction operation is performed on the concatenated features to obtain the target multimodal sequence.

5. The visual question answering method based on multimodal perception according to claim 1, characterized in that: Before the step of obtaining a target answer based on the target causal feature, the target text question and a preset target autoregressive visual question answering model, the step further includes: receiving training data, the training data comprising arbitrarily interwoven image data, text questions, and video data; Inputting the arbitrarily interwoven image data, text questions, and video data into a pre-trained coding model, obtaining visual features corresponding to the image data, text features corresponding to the text, and video causal embedding corresponding to the video data output by the coding model; Inputting the visual features and the video causal embedding into a preset causal transformation model respectively, performing a causal transformation operation, and obtaining output visual causal features and video causal features respectively; Combining the visual causal feature, the video causal feature, and the text feature to obtain a multimodal sequence; Based on the multimodal sequence, an autoregressive modeling operation is performed on a preset visual question answering model to obtain the target autoregressive visual question answering model.

6. The visual question answering method based on multimodal perception according to claim 5, characterized in that: The causal conversion model includes a causal self-attention layer, a cross-attention layer and a feedforward layer. The steps of inputting the visual features and the video causal embedding into the causal conversion model respectively, performing a causal conversion operation, and obtaining output visual causal features and video causal features respectively include: Inputting the visual features and the video causal embedding into the causal self-attention layer to obtain an output visual representation and the video representation; The visual representation and the video representation are both regarded as queries, and the visual features and the video causal embedding are both regarded as keys and values, which are input into the cross attention layer to obtain output aggregated visual features and aggregated video features; The aggregated visual features and the aggregated video features are input into the feed-forward layer to obtain the outputted visual causal features and the video causal features.

7. The visual question answering method based on multimodal perception according to claim 5, characterized in that: The step of combining the visual causal feature, the video causal feature and the text feature to obtain a multimodal sequence comprises: Setting a start mark and an end mark for each of the visual causal features and the video causal features to obtain marked visual causal features and video causal features; The labeled visual causal features and video causal features are combined with the text question to obtain the multimodal sequence.

8. A visual question-answering device based on multimodal perception, characterized in that: include: A receiving module, used for receiving data to be answered, wherein the data to be answered includes a target text question and target visual data; An encoding module, used for performing an encoding operation on the target visual data based on a preset encoding model to obtain target features; A conversion module, used to input the target feature into a preset causal conversion model, perform a causal conversion operation, and obtain an output target causal feature; An output module is used to obtain a target answer based on the target causal feature, the target text question and a preset target autoregressive visual question answering model.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the visual question answering method based on multimodal perception as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the visual question answering method based on multimodal perception as described in any one of claims 1 to 7 are implemented.

Images