Visual question and answer generation method and device and electronic equipment

Abstract

The invention provides an answer generation method and device for visual questions and answers and electronic equipment. The method comprises the steps of obtaining a question image of a question to be answered and question text information of the question to be answered; and inputting the question image and the question text information into a multi-modal model obtained by pre-training, and obtaining an answer image which can answer the question to be answered and / or answer text information which can answer the question to be answered and is output by the multi-modal model, the multi-modal model can learn high-dimensional information of the question image based on the question image and the question text information, so that the matching degree between the answer image output by the multi-modal model and the question to be answered is increased; and or the matching degree between the answer text information output by the multi-modal model and the question to be answered exceeds a matching degree threshold value. The answering accuracy of visual questions and answers can be improved.

Description

Technical Field

[0001] The present invention relates to the field of visual question answering technology, and in particular to a method, device, and electronic device for generating answers to visual question answers. Background Art

[0002] In the field of artificial intelligence, visual question answering technology based on multimodal models has become a key bridge connecting vision and language understanding. Traditional multimodal models jointly process image and text information, attempting to achieve cross-modal semantic alignment and thus answer natural language questions related to the image content.

[0003] However, related technologies generally have a core flaw: the tagger design in native multimodal models adopts a decoupling paradigm to encode images into discrete visual tag sequences. Its optimization goal only focuses on the low-level reconstruction accuracy of the original image, causing key semantic features to be diluted or lost during the encoding process, ultimately leading to inaccurate answers to visual question and answering based on multimodal models.

[0004] Therefore, finding a visual question answering answer generation method that can improve the accuracy of visual question answering has become a current research hotspot. Summary of the Invention

[0005] The present invention provides a method, device and electronic device for generating answers to visual questions and answers, which can improve the accuracy of answers to visual questions and answers.

[0006] The present invention provides a method for generating answers to visual question-answering, the method comprising: obtaining a question image of a question to be answered, and question text information of the question to be answered; inputting the question image and the question text information into a pre-trained multimodal model, and obtaining an answer image and / or answer text information output by the multimodal model that can answer the question to be answered, wherein the multimodal model can learn high-dimensional information of the question image based on the question image and the question text information, so that the matching degree between the answer image output by the multimodal model and the question to be answered, and / or the matching degree between the answer text information output by the multimodal model and the question to be answered exceeds a matching threshold.

[0007] According to a method for generating answers to visual question and answering provided by the present invention, the multimodal model includes a large language model and a visual discrete word segmenter, wherein the large language model is connected to the visual discrete word segmenter, and the visual discrete word segmenter includes an encoder, a quantizer, and a decoder; the step of inputting the question image and the question text information into a pre-trained multimodal model to obtain an answer image and / or answer text information output by the multimodal model that can answer the question to be answered specifically includes: inputting the question image into the encoder to obtain a feature map corresponding to the question image output by the encoder; inputting the feature map into the quantizer to obtain a quantized embedding corresponding to the question image output by the quantizer; and inputting the quantized embedding and the question text information into the large language model connected to the decoder to obtain an answer image and / or answer text information that can answer the question to be answered output by the multimodal model.

[0008] According to a method for generating answers to visual questions and answers provided by the present invention, the feature map is input into the quantizer to obtain a quantized embedding corresponding to the question image output by the quantizer, specifically including: inputting the feature map into the quantizer, and mapping each feature in the feature map to a matching codeword in a codebook through the quantizer; based on the matching codewords in the codebook, obtaining a quantized embedding corresponding to the question image output by the quantizer.

[0009] According to a method for generating answers to visual question-answering answers provided by the present invention, the quantized embedding and the question text information are input into the large language model connected to the decoder to obtain an answer image and / or answer text information output by the multimodal model that can answer the question to be answered. The method specifically includes: inputting the question text information into the large language model, and converting the question text information into a text embedding corresponding to the question text information based on the text embedding layer in the large language model; inputting the quantized embedding into the large language model, and mapping the first dimension of the quantized embedding to a second dimension having the same dimension as the text embedding through a multilayer perceptron to obtain a quantized embedding with the second dimension; based on the text embedding with the second dimension and the quantized embedding with the second dimension, sequentially passing through the large language model and the decoder, obtaining an answer image and / or answer text information output by the multimodal model that can answer the question to be answered.

[0010] According to a method for generating answers to visual question and answering provided by the present invention, the multimodal model includes a large language model and a visual discrete word segmenter, wherein the large language model is connected to the visual discrete word segmenter, and the visual discrete word segmenter includes an encoder, a quantizer, and a decoder; the multimodal model is pre-trained in the following manner: obtaining question image training samples and question text information training samples; based on the question image training samples and the question text information training samples, combined with the total supervision loss function, the large language model and the visual discrete word segmenter in the multimodal model are end-to-end trained to obtain a trained multimodal model, wherein the total supervision loss function is determined based on a reconstruction loss function used to optimize the reconstruction of the visual discrete word segmenter, and a text loss function used to optimize the accuracy of the answer text information output by the large language model.

[0011] According to a method for generating answers to visual questions and answers provided by the present invention, the multimodal model also includes a multilayer perceptron; the large language model and the visual discrete word segmenter in the multimodal model are end-to-end trained based on the question image training samples and the question text information training samples in combination with the total supervision loss function to obtain a trained multimodal model, specifically including: based on the question image training samples and the question text information training samples in combination with the total supervision loss function, by freezing the large language model, the multilayer perceptron and the visual discrete word segmenter in the multimodal model are end-to-end trained to obtain a trained multilayer perceptron and a trained visual discrete word segmenter; unfreezing the large language model, and based on the question image training samples and the question text information training samples in combination with the total supervision loss function, the large language model in the multimodal model is end-to-end trained to obtain a trained large language model; and obtaining a trained multimodal model based on the trained multilayer perceptron, the trained visual discrete word segmenter, and the trained large language model.

[0012] The present invention also provides an answer generation device for visual question answering, the device comprising: an acquisition module for acquiring a question image of a question to be answered, and question text information of the question to be answered; a processing module for inputting the question image and the question text information into a pre-trained multimodal model to obtain an answer image output by the multimodal model that can answer the question to be answered and\or answer text information that can answer the question to be answered, wherein the multimodal model can learn high-dimensional information of the question image based on the question image and the question text information, so that the matching degree between the answer image output by the multimodal model and the question to be answered, and\or the matching degree between the answer text information output by the multimodal model and the question to be answered exceeds a matching threshold.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for generating an answer to a visual question and answer as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the answer generation method for visual question answering as described in any of the above.

[0015] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described methods for generating answers to visual question answers.

[0016] The present invention provides a method, device, and electronic device for generating answers to visual question-answering (VQA). The method comprises: obtaining a question image and text information of a question to be answered; inputting the question image and text information into a pre-trained multimodal model, and obtaining an answer image and / or text information output by the multimodal model that can answer the question to be answered. The multimodal model can learn high-dimensional information about the question image based on the question image and text information, so that the matching degree between the answer image output by the multimodal model and the question to be answered, and / or the matching degree between the answer text information output by the multimodal model and the question to be answered exceeds a matching threshold. The present invention enables the multimodal model to learn high-dimensional information about the question image based on the question image and text information, thereby ensuring that the matching degree between the answer image output by the multimodal model and the question to be answered, and / or the matching degree between the answer text information output by the multimodal model and the question to be answered exceeds a matching threshold, thereby improving the accuracy of VQA answers. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 4 is a flow chart of the method for generating answers to visual question and answer provided by the present invention.

[0019] Figure 2The present invention provides a flowchart of inputting question images and question text information into a pre-trained multimodal model to obtain an answer image and / or answer text information that can answer the question to be answered as output by the multimodal model.

[0020] Figure 3 This is a flowchart of the present invention for inputting quantized embedding and question text information into a large language model accessed by a decoder, and obtaining an answer image and / or answer text information that can answer the question to be answered as output by a multimodal model.

[0021] Figure 4 It is a structural diagram of the answer generation device for visual question answering provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0024] The answer generation method for visual question answering provided by this invention adjusts the visual discrete word segmenter in a multimodal model in an end-to-end manner and utilizes a large language model as a visual assistant for the visual tagger. Specifically, by additionally introducing a codebook embedding (corresponding to a quantized embedding) of the tagger in the large language model, rather than using only discrete indices, and applying a tag-level caption loss (corresponding to a text loss function) to optimize the representation of the multimodal model, the performance of the multimodal model in visual understanding and generation tasks is significantly improved, thereby improving the accuracy of answers for visual question answering based on the multimodal model.

[0025] Figure 1 4 is a flow chart of the method for generating answers to visual question and answer provided by the present invention.

[0026] The following will be combined Figure 1 The process of the answer generation method for visual question answering provided by the present invention is described.

[0027] In an exemplary embodiment of the present invention, Figure 1 It can be seen that the answer generation method of visual question answering can include step 110 and step 120, and each step will be introduced below.

[0028] In step 110 , a question image of a question to be answered and question text information of the question to be answered are obtained.

[0029] In step 120, the question image and question text information are input into a pre-trained multimodal model, and an answer image and / or answer text information that can answer the question to be answered are output by the multimodal model, wherein the multimodal model can learn high-dimensional information of the question image based on the question image and question text information, so that the matching degree between the answer image output by the multimodal model and the question to be answered, and / or the matching degree between the answer text information output by the multimodal model and the question to be answered exceeds a matching threshold.

[0030] In one embodiment, a question image of a question to be answered and question text information of the question to be answered can be obtained. For example, the question image of the question to be answered may be about a sealed package box; and the question text information of the question to be answered may be textual description information of how to open the sealed package box as provided by the user.

[0031] In another embodiment, the question image and question text information can be input together into a pre-trained multimodal model. Since the multimodal model can learn high-dimensional information of the question image based on the question image and question text information, wherein the high-dimensional information can be considered as information that characterizes the detailed features of the picture of the question image, the detailed feature information about the question image can be effectively captured based on the multimodal model, thereby ensuring that the answer image output by the multimodal model that can answer the question to be answered has a matching degree exceeding a matching degree threshold with the question to be answered, and\or the answer text information that can answer the question to be answered has a matching degree exceeding a matching degree threshold with the question to be answered. In other words, the answer image output by the multimodal model that can answer the question to be answered and\or the answer text information that can answer the question to be answered can answer the question to be answered more accurately.

[0032] Continuing with the example of the question image of the question to be answered in the previous article, which may be about a sealed packaging box; the question text information of the question to be answered may be the text description information proposed by the user on how to open the sealed packaging box; the answer image output by the multimodal model may be an instruction image guiding the user on how to open the sealed packaging box; the answer text information output by the multimodal model is the instruction text information that matches the answer image and guides the user on how to open the sealed packaging box.

[0033] The present invention provides a method for generating answers to visual question-answering (VQA) questions, including: obtaining a question image and question text information of a question to be answered; inputting the question image and question text information into a pre-trained multimodal model, and obtaining an answer image and / or answer text information output by the multimodal model that can answer the question to be answered, wherein the multimodal model can learn high-dimensional information about the question image based on the question image and question text information, so that the matching degree between the answer image output by the multimodal model and the question to be answered, and / or the matching degree between the answer text information output by the multimodal model and the question to be answered exceeds a matching threshold. The present invention enables the multimodal model to learn high-dimensional information about the question image based on the question image and question text information, thereby enabling the matching degree between the answer image output by the multimodal model and the question to be answered, and / or the matching degree between the answer text information output by the multimodal model and the question to be answered, to exceed a matching threshold, thereby improving the accuracy of VQA answers.

[0034] Figure 2 The present invention provides a flowchart of inputting question images and question text information into a pre-trained multimodal model to obtain an answer image and / or answer text information that can answer the question to be answered as output by the multimodal model.

[0035] The following will be combined Figure 2 The present invention provides a process for inputting question images and question text information into a pre-trained multimodal model to obtain an answer image and / or answer text information that can answer the question to be answered, which are output by the multimodal model.

[0036] In an exemplary embodiment of the present invention, the multimodal model may include a large language model and a visual discrete word segmenter, wherein the large language model is connected to the visual discrete word segmenter, and the visual discrete word segmenter includes an encoder, a quantizer, and a decoder; Figure 2 It can be seen that inputting the question image and question text information into the pre-trained multimodal model to obtain the answer image and\or answer text information that can answer the question to be answered output by the multimodal model can include steps 210 to 230, and each step will be introduced below.

[0037] In step 210, the problem image is input to the encoder, and a feature map corresponding to the problem image is output by the encoder.

[0038] In one embodiment, the problem image can be input to the encoder to obtain the feature map corresponding to the problem image output by the encoder. Projection to feature map On, among them, D is the characteristic dimension; h=H / s; w=W / s , s is the downsampling factor.

[0039] In step 220, the feature map is input to a quantizer to obtain a quantized embedding corresponding to the problem image output by the quantizer.

[0040] In another embodiment, the feature map can also be input into a quantizer to obtain a quantized embedding corresponding to the problem image output by the quantizer. Each feature is mapped to the codebook The nearest code in the codebook, that is, the matching codeword in the codebook, to obtain the quantized embedding ,in, is the codebook size. In another example, the decoder can reconstruct the quantized embedding back to the original image, thereby realizing the self-supervised learning training of the entire visual discrete word segmenter.

[0041] In step 230, the quantized embedding and the question text information are input into the large language model connected to the decoder, and the answer image and / or the answer text information that can answer the question to be answered are output by the multimodal model.

[0042] In another embodiment, the quantized embedding and question text information can be input into a large language model connected to a decoder, thereby obtaining an answer image that can answer the question to be answered and\or answer text information that can answer the question to be answered, which is output by the multimodal model. It can be understood that connecting the codebook embedding (corresponding to the quantized embedding) from the visual discrete word segmenter to the large language model can effectively utilize the richer feature representation encoded in the visual discrete word segmenter, while achieving end-to-end training, so that the multimodal model can learn the high-dimensional information of the question image (corresponding to the richer feature representation encoded in the visual discrete word segmenter) based on the question image and question text information. This can ensure that the matching degree between the answer image output by the multimodal model and the question to be answered, and\or the matching degree between the answer text information output by the multimodal model and the question to be answered exceeds a matching threshold, thereby improving the accuracy of the visual question answering.

[0043] It should be noted that the large language model can output the mapping codebook index (also called codeword) of the answer image, then pass the mapping codebook index through the quantizer to obtain a quantized embedding, and then input the quantized embedding into the decoder to obtain the answer image output by the decoder that can answer the question to be answered.

[0044] In another exemplary embodiment of the present invention, continuing with the above-mentioned embodiment as an example, the feature map is input to the quantizer to obtain the quantized embedding corresponding to the problem image output by the quantizer, which can be implemented in the following manner: The feature map is input to the quantizer, and the quantizer maps each feature in the feature map to a matching codeword in the codebook; Based on the matching codewords in the codebook, the quantized embedding corresponding to the problem image output by the quantizer is obtained.

[0045] In one embodiment, the feature map can be input to a quantizer, and the quantizer maps each feature in the feature map to a matching codeword in the codebook. Further, based on the matching codeword in the codebook, the quantizer outputs a quantized embedding corresponding to the problem image. That is, the quantizer converts Each feature is mapped to the codebook The nearest code in the codebook, that is, the matching codeword in the codebook, to obtain the quantized embedding ,in, is the codebook size. Through this embodiment, the codebook embedding (corresponding to the quantized embedding) can be connected from the visual discrete word segmenter to the large language model, thereby effectively utilizing the richer feature representation encoded in the visual discrete word segmenter. This allows the multimodal model to learn the high-dimensional information of the question image based on the question image and question text information (corresponding to the richer feature representation encoded in the visual discrete word segmenter). This allows the matching degree between the answer image output by the multimodal model and the question to be answered, and / or the matching degree between the answer text information output by the multimodal model and the question to be answered to exceed a matching degree threshold, thereby improving the accuracy of visual question answering.

[0046] Figure 3 This is a flowchart of the present invention for inputting quantized embedding and question text information into a large language model accessed by a decoder, and obtaining an answer image and / or answer text information that can answer the question to be answered as output by a multimodal model.

[0047] The following will be combined Figure 3 The process of inputting quantized embeddings and question text information into a large language model connected to a decoder to obtain an answer image and / or answer text information that can answer the question as output by a multimodal model is described.

[0048] In an exemplary embodiment of the present invention, Figure 3 It can be seen that inputting the quantized embedding and question text information into the large language model connected to the decoder to obtain the answer image and\or answer text information that can answer the question to be answered output by the multimodal model can include steps 310 to 330, and each step will be introduced below.

[0049] In step 310 , the question text information is input into the large language model, and the question text information is converted into a text embedding corresponding to the question text information based on the text embedding layer in the large language model.

[0050] In one embodiment, the question text information can be input into the large language model, and the question text information can be converted into a text embedding corresponding to the question text information based on the text embedding layer in the large language model. That is, for the text input (question text information) T , you can use the tokenizer and text embedding layer in the pre-trained large language model to convert it into text token embedding (corresponding to text embedding) .

[0051] In step 320 , the quantized embedding is input into the large language model, and the first dimension of the quantized embedding is mapped to a second dimension having the same dimension as the text embedding through a multilayer perceptron, thereby obtaining a quantized embedding having the second dimension.

[0052] In step 330, based on the text embedding with the second dimension and the quantized embedding with the second dimension, the large language model and the decoder are sequentially passed to obtain the answer image and\or the answer text information that can answer the question to be answered output by the multimodal model.

[0053] In another embodiment, the quantized embedding can also be input into the large language model, and through the multi-layer perceptron, the first dimension of the quantized embedding is mapped to the second dimension with the same dimension as the text embedding, so that the quantized embedding with the second dimension can be obtained; it can be understood that the text embedding has a second dimension. Furthermore, based on the text embedding with the second dimension and the quantized embedding with the second dimension, the large language model and the decoder are sequentially passed to obtain the answer image that can answer the question to be answered and\or the answer text information that can answer the question to be answered output by the multimodal model. In this embodiment, mapping the first dimension of the quantized embedding to the second dimension with the same dimension as the text embedding can effectively ensure the compatibility of the visual discrete word segmenter with the large language model. During the application process, for a given image , first obtain its quantized embedding from the codebook of the visual discrete word segmenter 。 To ensure compatibility of the visual discrete word segmenter with large language models, a multi-layer perceptron can be used, where the multi-layer perceptron acts as a lightweight projection layer that quantizes the embedding Map to , where C represents the hidden dimension size of the large language model; since the entire computational graph, including the pre-trained large language model and the visual discrete word segmenter, is still differentiable, the entire structure can be trained end-to-end using gradient-based optimization.

[0054] In yet another embodiment, based on the text embedding having a second dimension and the quantized embedding having a second dimension, a decoder can be used to obtain an answer image and / or answer text information output by the large language model that can answer the unanswered question. In this embodiment, compatibility with the large language model can be achieved without adjusting the large language model, reducing processing complexity.

[0055] In another exemplary embodiment of the present invention, the multimodal model may include a large language model and a visual discrete word segmenter, wherein the large language model is connected to the visual discrete word segmenter, and the visual discrete word segmenter includes an encoder, a quantizer, and a decoder; Among them, the multimodal model can be pre-trained in the following ways: Obtain question image training samples and question text information training samples; Based on question image training samples and question text information training samples, combined with the total supervision loss function, the large language model and visual discrete word segmenter in the multimodal model are end-to-end trained to obtain a trained multimodal model. Among them, the total supervision loss function is determined according to the reconstruction loss function used to optimize the visual discrete word segmenter for reconstruction, and the text loss function used to optimize the accuracy of the answer text information output by the large language model.

[0056] In one embodiment, the total supervision loss function can be expressed as follows: L=L cap +m×L vq (1) Among them, L represents the total supervision loss function; L cap represents the text loss function used to optimize the accuracy of the answer text information output by the large language model; L vq represents the reconstruction loss function used to optimize the visual discrete word segmenter for reconstruction; m represents the weight coefficient, which can be adjusted according to actual conditions.

[0057] In this embodiment, the multimodal model is trained based on the total supervision loss function constructed by the reconstruction loss function for optimizing the visual discrete word segmenter for reconstruction and the text loss function for optimizing the accuracy of the answer text information output by the large language model, which can achieve end-to-end training, thereby improving the matching degree between the answer image output by the multimodal model and the question to be answered, and\or the matching degree between the answer text information output by the multimodal model and the question to be answered exceeds the matching degree threshold, thereby improving the answer accuracy of the visual question answering.

[0058] In another exemplary embodiment of the present invention, the multimodal model may further include a multilayer perceptron; wherein, based on question image training samples and question text information training samples, combined with a total supervision loss function, the large language model and the visual discrete word segmenter in the multimodal model are end-to-end trained to obtain a trained multimodal model, which can be implemented in the following manner: Based on the problem image training samples and the problem text information training samples, combined with the total supervision loss function, by freezing the large language model, the multi-layer perceptron and the visual discrete word segmenter in the multimodal model are trained end-to-end to obtain the trained multi-layer perceptron and the trained visual discrete word segmenter; Unfreeze the large language model and perform end-to-end training on the large language model in the multimodal model based on the question image training samples and the question text information training samples, combined with the total supervision loss function, to obtain the trained large language model; Based on the trained multi-layer perceptron, the trained visual discrete word segmenter, and the trained large language model, a trained multimodal model is obtained.

[0059] In one embodiment, alignment learning can be performed. During the application process, a pre-trained large language model and a visual discrete word segmenter can be used, the large language model can be kept frozen, and the visual projection layer and the visual discrete word segmenter with subtitle loss can be trained. This setting enables the visual discrete word segmenter to effectively perceive the semantic information in the large language model, thereby effectively connecting vision and language. That is, based on the problem image training samples and the problem text information training samples, combined with the total supervision loss function, by freezing the large language model, the multi-layer perceptron and the visual discrete word segmenter in the multimodal model are end-to-end trained to obtain a trained multi-layer perceptron and a trained visual discrete word segmenter.

[0060] In another embodiment, semantic learning can be performed. At this stage, the weights of the large language model, the projection layer, and the visual discrete word segmenter can be unfrozen, and the subtitle loss and the reconstruction loss can be used together to optimize them, that is, the large language model is unfrozen, and based on the question image training samples and the question text information training samples, combined with the total supervision loss function, the large language model in the multimodal model is end-to-end trained to obtain a trained large language model. Furthermore, based on the trained multi-layer perceptron, the trained visual discrete word segmenter, and the trained large language model, a trained multimodal model can be obtained. Through this embodiment, a multimodal model can be trained efficiently and accurately, thereby laying the foundation for answering visual questions based on a multimodal model.

[0061] As described above, the method for generating answers to visual question-answering provided by the present invention includes: obtaining a question image of a question to be answered and question text information of the question to be answered; inputting the question image and the question text information into a pre-trained multimodal model, and obtaining an answer image and / or answer text information output by the multimodal model that can answer the question to be answered, wherein the multimodal model can learn high-dimensional information about the question image based on the question image and the question text information, so that the matching degree between the answer image output by the multimodal model and the question to be answered, and / or the matching degree between the answer text information output by the multimodal model and the question to be answered exceeds a matching threshold. The present invention can learn high-dimensional information about the question image based on the question image and the question text information, so that the matching degree between the answer image output by the multimodal model and the question to be answered, and / or the matching degree between the answer text information output by the multimodal model and the question to be answered exceeds a matching threshold, thereby improving the accuracy of answers in visual question-answering.

[0062] The following describes the answer generation device for visual question answering provided by the present invention. The answer generation device for visual question answering described below and the answer generation method for visual question answering described above can refer to each other.

[0063] Figure 4 It is a structural diagram of the answer generation device for visual question answering provided by the present invention.

[0064] In an exemplary embodiment of the present invention, Figure 4 It can be seen that the answer generation device for visual question answering can include an acquisition module 410 and a processing module 420. Each module will be introduced below.

[0065] The acquisition module 410 may be configured to acquire a question image of a question to be answered and question text information of the question to be answered; The processing module 420 may be configured to input the question image and the question text information into a pre-trained multimodal model, and obtain an answer image and / or answer text information output by the multimodal model that can answer the question to be answered, wherein: The multimodal model can learn high-dimensional information of the question image based on the question image and the question text information, so that the matching degree between the answer image output by the multimodal model and the question to be answered, and\or the matching degree between the answer text information output by the multimodal model and the question to be answered exceeds a matching threshold.

[0066] In an exemplary embodiment of the present invention, the multimodal model includes a large language model and a visual discrete word segmenter, wherein the large language model is connected to the visual discrete word segmenter, and the visual discrete word segmenter includes an encoder, a quantizer, and a decoder; The processing module 420 may input the question image and the question text information into a pre-trained multimodal model to obtain an answer image and / or answer text information that can answer the question to be answered, output by the multimodal model, in the following manner: Inputting the problem image into the encoder, and obtaining a feature map corresponding to the problem image output by the encoder; Inputting the feature map into the quantizer to obtain a quantized embedding corresponding to the problem image output by the quantizer; The quantized embedding and the question text information are input into the large language model connected to the decoder, and the answer image and / or answer text information that can answer the question to be answered are output by the multimodal model.

[0067] In an exemplary embodiment of the present invention, the processing module 420 may implement inputting the feature map into the quantizer to obtain the quantized embedding corresponding to the problem image output by the quantizer in the following manner: Inputting the feature map into the quantizer, and mapping each feature in the feature map to a matching codeword in a codebook through the quantizer; Based on the matching codewords in the codebook, a quantized embedding corresponding to the problem image output by the quantizer is obtained.

[0068] In an exemplary embodiment of the present invention, the processing module 420 may implement inputting the quantized embedding and the question text information into the large language model connected to the decoder, thereby obtaining an answer image and / or answer text information that can answer the question to be answered, output by the multimodal model, in the following manner: Inputting the question text information into the large language model, and converting the question text information into a text embedding corresponding to the question text information based on a text embedding layer in the large language model; Inputting the quantized embedding into the large language model, and mapping the first dimension of the quantized embedding to a second dimension having the same dimension as the text embedding through a multilayer perceptron to obtain a quantized embedding having the second dimension; Based on the text embedding with the second dimension and the quantized embedding with the second dimension, the large language model and the decoder are sequentially passed to obtain an answer image and / or answer text information that can answer the question to be answered, which are output by the multimodal model.

[0069] In an exemplary embodiment of the present invention, the multimodal model includes a large language model and a visual discrete word segmenter, wherein the large language model is connected to the visual discrete word segmenter, and the visual discrete word segmenter includes an encoder, a quantizer, and a decoder; the processing module 420 can implement pre-training to obtain the multimodal model in the following manner: Obtain question image training samples and question text information training samples; Based on the problem image training samples and the problem text information training samples, combined with the total supervision loss function, the large language model and the visual discrete word segmenter in the multimodal model are trained end-to-end to obtain a trained multimodal model, wherein, The total supervision loss function is determined based on a reconstruction loss function for optimizing the reconstruction of the visual discrete word segmenter and a text loss function for optimizing the accuracy of the answer text information output by the large language model.

[0070] In an exemplary embodiment of the present invention, the multimodal model further includes a multilayer perceptron; the processing module 420 can implement end-to-end training of the large language model and the visual discrete word segmenter in the multimodal model based on the question image training samples and the question text information training samples in combination with the total supervision loss function in the following manner to obtain a trained multimodal model: Based on the question image training samples and the question text information training samples, combined with the total supervision loss function, by freezing the large language model, the multi-layer perceptron and the visual discrete word segmenter in the multimodal model are end-to-end trained to obtain the trained multi-layer perceptron and the trained visual discrete word segmenter; Unfreeze the large language model, and perform end-to-end training on the large language model in the multimodal model based on the question image training sample and the question text information training sample in combination with the total supervision loss function to obtain a trained large language model; Based on the trained multi-layer perceptron, the trained visual discrete word segmenter, and the trained large language model, a trained multimodal model is obtained.

[0071] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communications bus 540. The processor 510 may call logic instructions in the memory 530 to execute a method for generating answers to visual question answers, the method comprising: obtaining a question image of a question to be answered and question text information of the question to be answered; inputting the question image and the question text information into a pre-trained multimodal model, and obtaining an answer image and / or answer text information output by the multimodal model that can answer the question to be answered, wherein the multimodal model can learn high-dimensional information of the question image based on the question image and the question text information, so that the matching degree between the answer image output by the multimodal model and the question to be answered, and / or the matching degree between the answer text information output by the multimodal model and the question to be answered exceeds a matching threshold.

[0072] Furthermore, the logic instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0073] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the answer generation method for visual question answering provided by the above methods, which method includes: obtaining a question image of a question to be answered, and question text information of the question to be answered; inputting the question image and the question text information into a pre-trained multimodal model to obtain an answer image output by the multimodal model that can answer the question to be answered and\or answer text information that can answer the question to be answered, wherein the multimodal model can learn high-dimensional information of the question image based on the question image and the question text information, so that the matching degree between the answer image output by the multimodal model and the question to be answered, and\or the matching degree between the answer text information output by the multimodal model and the question to be answered exceeds a matching threshold.

[0074] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the answer generation method for visual question answering provided by the above-mentioned methods, the method comprising: obtaining a question image of a question to be answered, and question text information of the question to be answered; inputting the question image and the question text information into a pre-trained multimodal model to obtain an answer image output by the multimodal model that can answer the question to be answered and\or answer text information that can answer the question to be answered, wherein the multimodal model can learn high-dimensional information of the question image based on the question image and the question text information, so that the matching degree between the answer image output by the multimodal model and the question to be answered, and\or the matching degree between the answer text information output by the multimodal model and the question to be answered exceeds a matching threshold.

[0075] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0076] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for generating answers to visual question answering, characterized in that: The method comprises: Obtaining a question image of the question to be answered and question text information of the question to be answered; The question image and the question text information are input into a pre-trained multimodal model, and an answer image and / or answer text information that can answer the question to be answered are output by the multimodal model, wherein: The multimodal model can learn high-dimensional information of the question image based on the question image and the question text information, so that the matching degree between the answer image output by the multimodal model and the question to be answered, and\or the matching degree between the answer text information output by the multimodal model and the question to be answered exceeds a matching threshold.

2. The method for generating answers to visual question-answering questions according to claim 1, wherein: The multimodal model includes a large language model and a visual discrete word segmenter, wherein the large language model is connected to the visual discrete word segmenter, and the visual discrete word segmenter includes an encoder, a quantizer, and a decoder; Inputting the question image and the question text information into a pre-trained multimodal model to obtain an answer image and / or answer text information that can answer the question to be answered, which are output by the multimodal model, specifically includes: Inputting the problem image into the encoder, and obtaining a feature map corresponding to the problem image output by the encoder; Inputting the feature map into the quantizer to obtain a quantized embedding corresponding to the problem image output by the quantizer; The quantized embedding and the question text information are input into the large language model connected to the decoder, and the answer image and / or answer text information that can answer the question to be answered are output by the multimodal model.

3. The method for generating answers to visual question-answering questions according to claim 2, wherein: Inputting the feature map into the quantizer to obtain a quantized embedding output by the quantizer corresponding to the problem image specifically includes: Inputting the feature map into the quantizer, and mapping each feature in the feature map to a matching codeword in a codebook through the quantizer; Based on the matching codewords in the codebook, a quantized embedding corresponding to the problem image output by the quantizer is obtained.

4. The method for generating answers to visual question-answering questions according to claim 2, wherein: Inputting the quantized embedding and the question text information into the large language model connected to the decoder, and obtaining an answer image and / or answer text information output by the multimodal model that can answer the question to be answered, specifically includes: Inputting the question text information into the large language model, and converting the question text information into a text embedding corresponding to the question text information based on a text embedding layer in the large language model; Inputting the quantized embedding into the large language model, and mapping the first dimension of the quantized embedding to a second dimension having the same dimension as the text embedding through a multilayer perceptron to obtain a quantized embedding having the second dimension; Based on the text embedding with the second dimension and the quantized embedding with the second dimension, the large language model and the decoder are sequentially passed to obtain an answer image and / or answer text information that can answer the question to be answered, which are output by the multimodal model.

5. The method for generating answers to visual question-answering according to claim 1, wherein: The multimodal model includes a large language model and a visual discrete word segmenter, wherein the large language model is connected to the visual discrete word segmenter, and the visual discrete word segmenter includes an encoder, a quantizer, and a decoder; the multimodal model is pre-trained in the following manner: Obtain question image training samples and question text information training samples; Based on the problem image training samples and the problem text information training samples, combined with the total supervision loss function, the large language model and the visual discrete word segmenter in the multimodal model are trained end-to-end to obtain a trained multimodal model, wherein, The total supervision loss function is determined based on a reconstruction loss function for optimizing the reconstruction of the visual discrete word segmenter and a text loss function for optimizing the accuracy of the answer text information output by the large language model.

6. The method for generating answers to visual question-answering according to claim 5, wherein: The multimodal model further includes a multilayer perceptron; based on the question image training samples and the question text information training samples, combined with the total supervision loss function, the large language model and the visual discrete word segmenter in the multimodal model are end-to-end trained to obtain a trained multimodal model, specifically including: Based on the question image training samples and the question text information training samples, combined with the total supervision loss function, by freezing the large language model, the multi-layer perceptron and the visual discrete word segmenter in the multimodal model are end-to-end trained to obtain the trained multi-layer perceptron and the trained visual discrete word segmenter; Unfreeze the large language model, and perform end-to-end training on the large language model in the multimodal model based on the question image training sample and the question text information training sample in combination with the total supervision loss function to obtain a trained large language model; Based on the trained multi-layer perceptron, the trained visual discrete word segmenter, and the trained large language model, a trained multimodal model is obtained.

7. A visual question-answering answer generation device, characterized in that: The device comprises: An acquisition module is used to acquire a question image of a question to be answered and question text information of the question to be answered; A processing module is configured to input the question image and the question text information into a pre-trained multimodal model, and obtain an answer image and / or answer text information output by the multimodal model that can answer the question to be answered, wherein: The multimodal model can learn high-dimensional information of the question image based on the question image and the question text information, so that the matching degree between the answer image output by the multimodal model and the question to be answered, and\or the matching degree between the answer text information output by the multimodal model and the question to be answered exceeds a matching threshold.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for generating an answer to a visual question answering according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating an answer to a visual question answering as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating an answer to a visual question answering as claimed in any one of claims 1 to 6 is implemented.

Images