Visual question answering method, device, equipment, storage medium and product

Abstract

The present application relates to the field of visual question answering technology, and discloses a visual question answering method, apparatus, device, storage medium and product, including: extracting features from visual question data to generate multimodal feature data corresponding to the visual question data; searching in a preset template library based on the multimodal feature data to obtain target example question answering data that matches the multimodal feature data; constructing model prompt words based on the target example question answering data; inputting the model prompt words and the visual question data into a large language model, so that the large language model generates a visual question answering result based on the visual question data under the guidance of the model prompt words. Since the model prompt words can be searched for data to construct the multimodal feature data corresponding to the visual question data, the large language model is guided by the model prompt words, ensuring that even if data containing images such as visual documents are input, the large language model can still perform question answering normally.

Description

Technical Field

[0001] The present application relates to the field of visual question answering technology, and in particular to visual question answering methods, devices, equipment, storage media and products. Background Art

[0002] A Large Language Model (LLM) is a deep learning-based model that can understand and generate human language and is commonly used in natural language processing tasks such as text generation, translation, and question answering.

[0003] Document understanding systems based on large models are often used in various document scenarios. They can understand the input documents accordingly and perform tasks proposed by users based on the understanding, such as giving corresponding answers to questions raised by users based on the input documents, or understanding the input documents according to the needs proposed by users, and outputting key field information required by users.

[0004] However, such applications can generally only be applied to text documents, while documents in real applications are not all text documents. There are also a large number of visual documents, such as documents containing scans or documents containing handheld photos. In addition, document understanding tasks in practical applications are diverse, including various information extraction problems and generative problems covering logical computational reasoning. Summary of the invention

[0005] The main purpose of this application is to provide a visual question answering method, device, equipment, storage medium and product, aiming to solve the technical problem that the related technology large language model cannot support the input of visual documents when used.

[0006] To achieve the above objectives, this application proposes a visual question answering method, which includes:

[0007] Extracting features from the visual problem data to generate multimodal feature data corresponding to the visual problem data;

[0008] Searching in a preset template library according to the multimodal feature data to obtain at least one target example question and answer data matching the multimodal feature data, the preset template library including a plurality of example question and answer data and standard multimodal feature data corresponding to each example question and answer data;

[0009] Constructing a model prompt word according to the at least one target example question-answering data;

[0010] The model prompt words and the visual question data are input into a large language model, so that the large language model generates a visual question answering result according to the visual question data under the guidance of the model prompt words.

[0011] In a possible implementation of the present application, the visual problem data includes image data and problem data;

[0012] The extracting features of the visual problem data to generate multimodal feature data corresponding to the visual problem data includes:

[0013] Performing text recognition on the image data to obtain image text data corresponding to the image data;

[0014] Performing feature extraction on the image data to generate visual feature data, performing feature extraction on the image text data to generate text feature data, and performing feature extraction on the question data to generate question feature data;

[0015] The visual feature data, the text feature data and the question feature data are fused to generate multimodal feature data corresponding to the visual question data.

[0016] In a possible implementation of the present application, performing text recognition on the image data to obtain image text data corresponding to the image data includes:

[0017] Acquire at least one text area in the image data;

[0018] Performing text recognition on each text area respectively to obtain regional text data corresponding to each text area;

[0019] Based on a preset order and the regional position of each text region, the regional text data corresponding to each text region is assembled to generate image text data corresponding to the image data, wherein preset separators are set between the regional text data of different text regions in the image text data.

[0020] In a possible implementation of the present application, the fusing of the visual feature data, the text feature data and the question feature data to generate multimodal feature data corresponding to the visual question data includes:

[0021] Obtaining a preset proportionality coefficient, wherein the preset proportionality coefficient is used to characterize the weight ratio of each type of feature data in the multimodal feature data;

[0022] Based on the preset scale factor, the visual feature data, the text feature data and the question feature data are fused to generate multimodal feature data corresponding to the visual question data.

[0023] In a possible implementation of the present application, the preset scale factor includes a visual scale factor, a text scale factor, and a question scale factor;

[0024] The step of fusing the visual feature data, the text feature data and the question feature data based on the preset scale factor to generate multimodal feature data corresponding to the visual question data includes:

[0025] Obtaining the image size and text area size of the image data;

[0026] Calculate the text ratio according to the text area size and the image size;

[0027] Adjusting the visual proportion coefficient and the text proportion coefficient in the preset proportion coefficient according to the text proportion coefficient;

[0028] Based on the adjusted preset scale factor, the visual feature data, the text feature data and the question feature data are fused to generate multimodal feature data corresponding to the visual question data.

[0029] In a possible implementation of the present application, searching in a preset template library according to the multimodal feature data to obtain at least one target example question and answer data matching the multimodal feature data includes:

[0030] Comparing the multimodal feature data with the standard multimodal feature data corresponding to each example question and answer data in the preset template library to obtain the feature similarity corresponding to each example question and answer data;

[0031] At least one target example question and answer data matching the multimodal feature data is selected from the preset template library based on the feature similarity.

[0032] In a possible implementation of the present application, selecting at least one target example question and answer data matching the multimodal feature data from the preset template library based on the feature similarity includes:

[0033] Sort the example question and answer data in the preset template library from large to small based on the corresponding feature similarity to generate a sorting result;

[0034] The top N example question and answer data in the sorting result are selected as target example question and answer data matching the multimodal feature data, where N is a positive integer.

[0035] In addition, to achieve the above purpose, the present application also proposes a visual question-answering device, the device comprising:

[0036] An extraction module, used to extract features from the visual problem data and generate multimodal feature data corresponding to the visual problem data;

[0037] A search module, configured to search in a preset template library according to the multimodal feature data to obtain at least one target example question and answer data matching the multimodal feature data, wherein the preset template library includes a plurality of example question and answer data and standard multimodal feature data corresponding to each example question and answer data;

[0038] A construction module, configured to construct a model prompt word according to the at least one target example question and answer data;

[0039] A generation module is used to input the model prompt word and the visual question data into a large language model, so that the large language model generates a visual question answering result according to the visual question data under the guidance of the model prompt word.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a visual question answering device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the visual question answering method as described above.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the visual question answering method as described above are implemented.

[0042] In addition, to achieve the above-mentioned purpose, the present application also proposes a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the visual question answering method described above are implemented.

[0043] One or more technical solutions proposed in this application have at least the following technical effects:

[0044] Since the data can be searched based on the multimodal feature data corresponding to the visual question data to build the model prompt words, the large language model can be guided by the model prompt words to quickly determine the relationship between the question and the answer, thereby correctly generating the visual question and answer results. This ensures that even if data containing images such as visual documents are input, the large language model can still perform question and answer normally. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 A flowchart diagram of the first embodiment of the visual question answering method of the present application;

[0048] Figure 2 A flowchart diagram of the second embodiment of the visual question answering method of this application;

[0049] Figure 3 This is a document example diagram of densely arranged visual documents according to an embodiment of the present application;

[0050] Figure 4 A flowchart diagram of the third embodiment of the visual question answering method of this application;

[0051] Figure 5 This is a schematic diagram of the overall process of visual question answering according to an embodiment of the present application;

[0052] Figure 6 This is a schematic diagram of the module structure of the visual question answering device according to an embodiment of the present application;

[0053] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the visual question answering method in the embodiment of the present application.

[0054] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0055] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0056] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0057] Based on this, the present application embodiment provides a visual question answering method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the visual question answering method of the present application.

[0058] In this embodiment, the visual question answering method includes steps S10 to S40:

[0059] Step S10: extracting features from the visual problem data to generate multimodal feature data corresponding to the visual problem data.

[0060] It should be noted that the executor of this embodiment can be the visual question answering device, and the visual question answering device can be an electronic device such as a personal computer or a server, or other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment and the following embodiments, the visual question answering method of the present invention is explained by taking the visual question answering device as an example.

[0061] It should be noted that Visual Question Answering (VQA) refers to the task of outputting result data based on input image data and question data. The visual question data may include input image data and question data.

[0062] Among them, the image data can be a visual document, which at least includes scans, photographs or images generated by other means, and can also include text data; the image data and problem data can be input separately, and when necessary, the problem data can also be included in the image data and input together. At this time, annotations can be added to the image data to indicate which part of the data is the problem data.

[0063] In actual use, the visual question data may be sent to the visual question answering device by other devices, or may be generated by the visual question answering device itself, which is not limited in this embodiment.

[0064] In practical applications, in order to ensure that the large language model can understand the visual question data, it is necessary to find corresponding examples to prompt the large language model. In order to facilitate the search for similar data as examples, feature extraction can be performed on the visual question data to generate multimodal feature data corresponding to the visual question data.

[0065] Among them, multimodal feature data can be fused by feature data of multiple different modalities, such as visual features, text features, question features, etc.; when extracting features from visual question data, a preset feature extraction algorithm can be used, or a pre-trained feature extraction model can be used, and this embodiment does not limit this.

[0066] Step S20: searching in a preset template library according to the multimodal feature data to obtain at least one target example question and answer data matching the multimodal feature data.

[0067] It should be noted that the preset template library can be a database pre-built by the administrator of the visual question and answer device, such as the faiss database. The preset template library can include multiple example question and answer data and standard multimodal feature data corresponding to each example question and answer data.

[0068] In actual use, the construction of the preset template library can be based on preset samples. Each sample can include image data and question and answer pairs (question & answer, referred to as QA pairs). When constructing the preset template library, example question and answer data can be constructed based on the image data and the question and answer pairs. Then, feature extraction is performed on the image data contained in the example question and answer data and the question data in the question and answer pairs. The generated multimodal feature data is used as the standard multimodal feature data corresponding to the example question and answer data. Then, the example question and answer data and the standard multimodal feature data are associated as a sample and stored in the preset template library.

[0069] When extracting features from the image data contained in the example question-answer data and the question data in the question-result pair, the feature extraction method and process used are consistent with the feature extraction method and process used for extracting features from the visual question data;

[0070] When constructing the preset template library, the database is constructed based on questions, that is, each example question and answer data only contains image data and a question-result pair. If a sample contains image data and multiple question-result pairs, the image data contained therein will be combined with each question-result pair to construct multiple example question and answer data.

[0071] Step S30: constructing a model prompt word according to the at least one target example question and answer data.

[0072] In actual use, when constructing a model prompt based on the target example question and answer data, you can first fill in the preset prompt template according to the question and answer data of each target instance to generate prompt information corresponding to each target instance question and answer data, and then splice the prompt information corresponding to each target instance question and answer data to generate the model prompt.

[0073] Step S40: inputting the model prompt words and the visual question data into a large language model, so that the large language model generates a visual question answering result according to the visual question data under the guidance of the model prompt words.

[0074] It should be noted that the large language model can be an intelligent large language model such as chatGPT. When the model prompt words and visual question data are input into the large language model together, the large language model will determine the correlation between the question and the result according to the guidance of the model prompt words, and then generate the visual question and answer result based on the visual question data based on the correlation.

[0075] When the model prompt words and visual question data are input into the large language model, the model prompt words and the visual question data may be concatenated to generate model input data, and then the model input data may be input into the large language model.

[0076] This embodiment provides a visual question answering method. Since the data can be searched for model prompt words based on the multimodal feature data corresponding to the visual question data, the large language model is guided by the model prompt words to quickly determine the relationship between the question and the answer, thereby correctly generating a visual question answering result, ensuring that even if data containing images such as visual documents are input, the large language model can still perform question answering normally.

[0077] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 2 , the visual question data includes image data and question data, and step S10 of the visual question answering method includes steps S101 to S103:

[0078] Step S101: performing text recognition on the image data to obtain image text data corresponding to the image data.

[0079] It should be noted that the image contained in the image data may be a scanned image or other image containing text information. In order to correctly construct the corresponding multimodal features, text recognition can be performed on the image data first to obtain the image text data corresponding to the image data.

[0080] In actual use, when performing text recognition, optical character recognition (OCR) technology can be used for recognition.

[0081] OCR refers to the process by which an electronic device (such as a scanner or digital camera) examines characters, determines their shape by detecting dark and light patterns, and then uses character recognition methods to translate the shape into computer text.

[0082] Of course, other methods may also be used for text recognition, such as using models, etc., and this embodiment does not limit this.

[0083] In a possible implementation of this embodiment, in order to avoid feature loss as much as possible, step S101 in this embodiment may include:

[0084] Acquire at least one text area in the image data;

[0085] Performing text recognition on each text area respectively to obtain regional text data corresponding to each text area;

[0086] Based on a preset sequence and the regional position of each text region, the regional text data corresponding to each text region is assembled to generate image text data corresponding to the image data.

[0087] It should be noted that the text area in the image data can be divided according to the paragraphs, blank lines, separators, etc. of the text. For example, each text paragraph in the image data is divided into a text area, or the part between paragraphs with a spacing greater than a preset value is divided into a text area.

[0088] In actual use, OCR recognition can be performed on the characters contained in each text area respectively to recognize the characters in the area, thereby obtaining the area text data corresponding to each text area.

[0089] In actual application, the preset order can be pre-set by the administrator of the visual question answering device, for example, the preset order is set from top to bottom and from left to right. The text areas can be sorted according to the preset order and the area position of the text area, and then the area text data corresponding to each text area is assembled according to the sorting order of each text area, so as to generate image text data corresponding to the image data.

[0090] Among them, preset delimiters are set between the regional text data of different text areas in the image text data. The preset delimiters can be set in advance by the administrator of the visual question answering device, such as setting a space "" as a preset delimiter. Of course, other symbols can also be set as preset delimiters, such as setting "|" or "@" as preset delimiters.

[0091] It can be understood that the image text data can be regarded as a text sequence that is segmented by a preset separator and assembled from regional text data corresponding to each text region.

[0092] Step S102: extracting features from the image data to generate visual feature data, extracting features from the image text data to generate text feature data, and extracting features from the question data to generate question feature data.

[0093] It should be noted that since the images contained in the image data may be scanned copies, photos, or even mixed images of scanned copies and photos, in order to ensure that the final multimodal feature data can accurately represent the overall characteristics of the visual question data, feature extraction can be performed on the image data, image text data, and question data to generate visual feature data, text feature data, and question feature data. The question data can be a question represented by text data.

[0094] In actual use, when extracting features from image data, traditional image feature extraction algorithms can be used, such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up RobustFeatures) and other algorithms, or models can be used for image feature extraction, such as AlexNet, VGGNet and other models;

[0095] Since the CLIP (Contrastive Language-Image Pre-training) model can realize the association between images and texts, in order to ensure that there is an association between the features of images and texts, when extracting features from image data, the VIT part of the CLIP model can be used to encode the visual features of the image data and generate visual feature data.

[0096] Similarly, when extracting features from image text data or question data, a traditional text feature extraction algorithm can be used, or a model can be used for text feature extraction. If a model is used for text feature extraction, the model used can be a BGE-large model.

[0097] Among them, the BGE-large (bge-large-zh) model is a Chinese text representation model that can map any text data into a low-dimensional dense vector for tasks such as retrieval, classification, clustering or semantic matching, and can support the call of external knowledge for large models.

[0098] In actual use, image text data can be regarded as a text sequence that is segmented by preset delimiters and assembled from regional text data corresponding to each text area. When extracting features from image text data, text features can be extracted separately for the regional text data corresponding to each text area.

[0099] Step S103: Fusing the visual feature data, the text feature data and the question feature data to generate multimodal feature data corresponding to the visual question data.

[0100] In actual use, the visual feature data, text feature data and question feature data can be fused into a new feature vector by splicing, averaging, multiplication, or other methods with similar functions, and the generated new feature vector can be used as the multimodal feature data corresponding to the visual question data.

[0101] In a possible implementation of this embodiment, in order to ensure that the multimodal feature data can correctly represent the visual problem data, step S103 of this embodiment may include:

[0102] Get the preset scale factor;

[0103] Based on the preset scale factor, the visual feature data, the text feature data and the question feature data are fused to generate multimodal feature data corresponding to the visual question data.

[0104] It should be noted that the preset proportional coefficient is used to characterize the weight ratio of each type of feature data in the multimodal feature data, and the preset proportional coefficient may include a visual proportional coefficient, a text proportional coefficient, and a question proportional coefficient.

[0105] Among them, the visual scale coefficient is used to characterize the weight ratio of visual feature data in multimodal feature data, the text scale coefficient is used to characterize the weight ratio of text feature data in multimodal feature data, and the question scale coefficient is used to characterize the weight ratio of question feature data in multimodal feature data.

[0106] The preset scale factor can be pre-set by the administrator of the visual question answering device according to the importance of various features, for example: setting the visual scale factor to 0.3, the text scale factor to 0.3, and the question scale factor to 0.4.

[0107] In actual use, the visual feature data, text feature data and question feature data can be weighted and summed based on a preset proportional coefficient to generate a new feature vector, and the generated feature vector is used as the multimodal feature data corresponding to the visual question data. At this time, the multimodal feature data can be represented as:

[0108] E=α1*V+α2*T+α3*Q

[0109] In the formula, the preset scale coefficients include α1, α2 and α3, α1 is the visual scale coefficient, α2 is the text scale coefficient, and α3 is the question scale coefficient; E is the multimodal feature data, V is the visual feature data, T is the text feature data, and Q is the question feature data.

[0110] In a possible implementation of this embodiment, in order to ensure the rationality of the preset proportionality coefficient, the step of fusing the visual feature data, the text feature data and the question feature data based on the preset proportionality coefficient to generate the multimodal feature data corresponding to the visual question data may include:

[0111] Obtaining the image size and text area size of the image data;

[0112] Calculate the text ratio according to the text area size and the image size;

[0113] Adjusting the visual proportion coefficient and the text proportion coefficient in the preset proportion coefficient according to the text proportion coefficient;

[0114] Based on the adjusted preset scale factor, the visual feature data, the text feature data and the question feature data are fused to generate multimodal feature data corresponding to the visual question data.

[0115] It should be noted that the image size of the image data may be the size data such as the height and width of the image data, and the text area size of the image data may be the area size of the area containing the text data in the image data.

[0116] In actual use, the image area size of the image data can be calculated according to the image size, and then the ratio of the text area size to the image area size is used as the text ratio coefficient.

[0117] For example: Assuming that the image size of the image data is h, the image width is w, and the text area size is area(text), then the text ratio coefficient β=area(text) / (h*w).

[0118] In actual use, since the problem feature is more important in this task, it is kept consistent for all samples and the problem proportion coefficient is not adjusted. The visual feature data of almost all densely arranged documents are not much different, but the text therein is completely different. For such samples, the weight of their semantic features should be increased and the weight of visual features should be reduced. Therefore, the specific implementation of adjusting the visual proportion coefficient and the text proportion coefficient in the preset proportion coefficient according to the text proportion coefficient can be to reduce the visual proportion coefficient in the preset proportion coefficient according to the text proportion coefficient, and to increase the visual proportion coefficient in the preset proportion coefficient according to the text proportion coefficient.

[0119] In actual use, after the adjustment is completed, the visual feature data, text feature data and question feature data can be weighted and summed according to the adjusted preset proportional coefficient, and the generated new feature vector can be used as the multimodal feature data corresponding to the visual question data.

[0120] For example, assuming that the preset scale factors include α1, α2, and α3, α1 is the visual scale factor, α2 is the text scale factor, α3 is the question scale factor, and the text ratio factor is β, then the visual scale factor α1 in the adjusted preset scale factor ' =α1*(1-β), text scale factor α2 ' =α1*(1+β), and the problem proportional coefficient remains unchanged at α3, then the final multimodal feature data can be represented as:

[0121] E = α1 '*V+α2 ' *T+α3*Q

[0122] In the formula, the adjusted preset proportional coefficients include α1 ' , α2 ' and α3, α1 ' is the adjusted visual scale factor, α2 ' is the adjusted text ratio coefficient, α3 is the question ratio coefficient; E is the multimodal feature data, V is the visual feature data, T is the text feature data, and Q is the question feature data.

[0123] For ease of understanding, now combined Figure 3 This is for illustration only, but not for limitation. Figure 3 This is a document example diagram of densely arranged visual documents in this embodiment.

[0124] like Figure 3 As shown in the figure, for scans of some densely arranged visual documents (such as papers), the documents are arranged very densely. On this basis, even if some of the texts change, the overall visual effect changes little. Two different documents of this type, even if they contain different texts, will not have much difference in their visual feature data. At this time, for such samples, the weight of their semantic features should be increased and the weight of their visual features should be reduced.

[0125] This embodiment provides a visual question answering method, which uses different methods to extract features from multiple aspects and fuses the extracted feature data to ensure that multimodal feature data that can correctly represent the visual question data can be generated, making it easier to find corresponding target example question answering data.

[0126] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. Figure 4 , step S20 of the visual question answering method includes steps S201-S202:

[0127] Step S201: The multimodal feature data is compared with the standard multimodal feature data corresponding to each example question and answer data in the preset template library to obtain the feature similarity corresponding to each example question and answer data.

[0128] In actual use, the feature differences between the multimodal feature data and the standard multimodal feature data corresponding to each example question and answer data in the preset template library can be calculated respectively, and then the feature similarity corresponding to each example question and answer data can be determined based on the feature differences.

[0129] Among them, the feature difference can be a vector distance, and the algorithm used at this time can be a distance algorithm (such as the Euclidean distance algorithm). At this time, if it is assumed that the example question and answer data in the preset template library is S, and its corresponding standard multimodal feature data is E, and the visual question data is S', and its corresponding multimodal feature data is E', then the feature difference can be represented as D=distance (E, E'). If the vector distance is used to represent the feature difference, the larger the vector distance, the lower the similarity between the two vectors. Therefore, at this time, an inverse calculation can be performed based on the feature difference, and the obtained value is used as the feature similarity;

[0130] Of course, the feature difference can be vector similarity, then the algorithm used at this time can be a similarity algorithm (such as a cosine similarity algorithm). At this time, if it is assumed that the example question and answer data in the preset template library is S, and its corresponding standard multimodal feature data is E, and the visual question data is S', and its corresponding multimodal feature data is E', then the feature difference can be represented as C=cosine(E,E'), and the feature difference can be directly used as feature similarity.

[0131] Step S202: selecting at least one target example question and answer data matching the multimodal feature data from the preset template library based on the feature similarity.

[0132] In actual use, in order to ensure the effect of the example, when selecting the target example question and answer data, it is necessary to select the example question and answer data with a higher corresponding feature similarity as the target example question and answer data. In order to distinguish whether the feature similarity is high, the manager of the visual question and answer device can pre-set the similarity threshold and select the example question and answer data in the preset template library whose corresponding feature similarity is greater than the similarity threshold as the target example question and answer data.

[0133] In a possible implementation of this embodiment, in order to select the most appropriate target example question and answer data as much as possible, step S202 of this embodiment may include:

[0134] Sort the example question and answer data in the preset template library from large to small based on the corresponding feature similarity to generate a sorting result;

[0135] The top N example question and answer data in the sorting result are selected as target example question and answer data that match the multimodal feature data.

[0136] It should be noted that N is a positive integer and can be pre-set by the administrator of the visual question answering device according to actual needs, such as setting N to 3.

[0137] It can be understood that since the example question and answer data in the preset template library are sorted from large to small based on the corresponding feature similarity, in the sorting result, the higher the ranked example question and answer data is, the more likely it is to be similar to the visual question data. Therefore, taking the top N (topN) example question and answer data from the sorting result as the target example question and answer data that matches the multimodal feature data corresponding to the visual question data can ensure that the example question and answer data with the highest possibility of being similar to the visual question data is selected as the target example question and answer data, thereby ensuring its guiding effect as an example to guide the large language model as much as possible.

[0138] For ease of understanding, now combined Figure 5 This is for illustration only, but not for limitation. Figure 5 Schematic diagram of the overall process of visual question answering in this embodiment.

[0139] like Figure 5 As shown, when the user needs to conduct visual question answering, he will input the invoice image data and the question data "What is the invoice code?" At this time, the visual question answering device will perform OCR recognition on the invoice image data to generate an OCR recognition result (i.e., image text data), and perform visual feature extraction on the invoice image data to obtain visual feature data. Then, text feature extraction is performed on the question data and the image text data respectively to generate text feature data and question feature data (the two features are features of the embedding vector type). Then, the visual feature data, text feature data, and question feature data are fused to generate multimodal feature data. The multimodal feature data is compared with the standard multimodal feature data corresponding to each example question and answer data in the template database. Through the nearest neighbor algorithm, some example question and answer data with the closest vector distance (i.e., the greatest feature similarity) are selected as the target example question and answer data. Then, a model prompt word (Prompt, for example) is constructed based on the target example question and answer data. Figure 5 ), then the relevant content of the visual question data (question data) and the OCR recognition result are also filled into the Prompt, and the Prompt is sent to the large language model, so that the large language model generates a visual question answering result based on the visual question data under the guidance of the model prompt word.

[0140] The present embodiment provides a visual question-answering method, which obtains feature similarities corresponding to each example question-answering data by comparing multimodal feature data with standard multimodal feature data corresponding to each example question-answering data in a preset template library, and selects target example question-answering data based on the feature similarity, thereby ensuring that the selected target example question-answering data has a high degree of similarity with the visual question data, thereby ensuring that the guiding effect of the constructed model prompt words is better.

[0141] This application also provides a visual question-answering device, please refer to Figure 6 , the visual question answering device comprises:

[0142] An extraction module 10, configured to extract features from the visual problem data and generate multimodal feature data corresponding to the visual problem data;

[0143] A search module 20, configured to search in a preset template library according to the multimodal feature data to obtain at least one target example question and answer data matching the multimodal feature data, wherein the preset template library includes a plurality of example question and answer data and standard multimodal feature data corresponding to each example question and answer data;

[0144] A construction module 30, configured to construct a model prompt word according to the at least one target example question and answer data;

[0145] The generation module 40 is used to input the model prompt words and the visual question data into the large language model, so that the large language model generates a visual question answering result according to the visual question data under the guidance of the model prompt words.

[0146] The visual question answering device provided by the present application adopts the visual question answering method in the above embodiment, which can solve the technical problem that the large language model in the related art cannot support the input of visual documents when used. Compared with the prior art, the beneficial effects of the visual question answering device provided by the present application are the same as the beneficial effects of the visual question answering method provided by the above embodiment, and the other technical features in the visual question answering device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0147] The present application provides a visual question answering device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the visual question answering method in the above-mentioned embodiment one.

[0148] Reference below Figure 7 , which shows a schematic diagram of the structure of a visual question-answering device suitable for implementing an embodiment of the present application. The visual question-answering device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The visual question-answering device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0149] like Figure 7 As shown, the visual question answering device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the visual question answering device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the visual question answering device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a visual question answering device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0150] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0151] The visual question answering device provided by the present application adopts the visual question answering method in the above embodiment, which can solve the technical problem that the large language model in the related technology cannot support the input of visual documents when used. Compared with the prior art, the beneficial effects of the visual question answering device provided by the present application are the same as the beneficial effects of the visual question answering method provided by the above embodiment, and the other technical features in the visual question answering device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0152] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0153] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0154] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the visual question answering method in the above-mentioned embodiment.

[0155] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic or infrared systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0156] The computer-readable storage medium may be included in the visual question-answering device; or may exist independently without being assembled into the visual question-answering device.

[0157] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the visual question answering device, the visual question answering device:

[0158] Perform feature extraction on visual question data to generate multimodal feature data corresponding to the visual question data; search in a preset template library according to the multimodal feature data to obtain at least one target example question and answer data matching the multimodal feature data, wherein the preset template library includes multiple example question and answer data and standard multimodal feature data corresponding to each example question and answer data; construct a model prompt word according to the at least one target example question and answer data; input the model prompt word and the visual question data into a large language model, so that the large language model generates a visual question and answer result according to the visual question data under the guidance of the model prompt word.

[0159] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0160] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0161] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0162] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned visual question answering method, and can solve the technical problem that the large language model of the related art cannot support the input of visual documents when used. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the visual question answering method provided in the above-mentioned embodiment, and will not be repeated here.

[0163] The present application also provides a computer program product, including a computer program, which implements the steps of the visual question answering method as described above when executed by a processor.

[0164] The computer program product provided by this application can solve the technical problem that the large language model of the related art cannot support the input of visual documents when used. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the visual question answering method provided by the above embodiment, which will not be repeated here.

[0165] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A visual question answering method, characterized in that: The method comprises: Extracting features from the visual problem data to generate multimodal feature data corresponding to the visual problem data; Searching in a preset template library according to the multimodal feature data to obtain at least one target example question and answer data matching the multimodal feature data, the preset template library including a plurality of example question and answer data and standard multimodal feature data corresponding to each example question and answer data; Constructing a model prompt word according to the at least one target example question-answering data; Inputting the model prompt word and the visual question data into a large language model, so that the large language model generates a visual question answering result according to the visual question data under the guidance of the model prompt word; The visual problem data includes image data and problem data, the image data includes visual documents, and the visual documents include scanned copies, photographed pictures or generated images.

2. The visual question answering method according to claim 1, wherein: The extracting features of the visual problem data to generate multimodal feature data corresponding to the visual problem data includes: Performing text recognition on the image data to obtain image text data corresponding to the image data; Performing feature extraction on the image data to generate visual feature data, performing feature extraction on the image text data to generate text feature data, and performing feature extraction on the question data to generate question feature data; The visual feature data, the text feature data and the question feature data are fused to generate multimodal feature data corresponding to the visual question data.

3. The visual question answering method according to claim 2, wherein: The performing text recognition on the image data to obtain image text data corresponding to the image data includes: Acquire at least one text area in the image data; Performing text recognition on each text area respectively to obtain regional text data corresponding to each text area; Based on a preset order and the regional position of each text region, the regional text data corresponding to each text region is assembled to generate image text data corresponding to the image data, wherein preset separators are set between the regional text data of different text regions in the image text data.

4. The visual question answering method according to claim 2, wherein: The step of fusing the visual feature data, the text feature data and the question feature data to generate multimodal feature data corresponding to the visual question data includes: Obtaining a preset proportionality coefficient, wherein the preset proportionality coefficient is used to characterize the weight ratio of each type of feature data in the multimodal feature data; Based on the preset scale factor, the visual feature data, the text feature data and the question feature data are fused to generate multimodal feature data corresponding to the visual question data.

5. The visual question answering method according to claim 4, characterized in that: The preset scale factor includes a visual scale factor, a text scale factor and a question scale factor; The step of fusing the visual feature data, the text feature data and the question feature data based on the preset scale factor to generate multimodal feature data corresponding to the visual question data includes: Obtaining the image size and text area size of the image data; Calculate the text ratio according to the text area size and the image size; Adjusting the visual proportion coefficient and the text proportion coefficient in the preset proportion coefficient according to the text proportion coefficient; Based on the adjusted preset scale factor, the visual feature data, the text feature data and the question feature data are fused to generate multimodal feature data corresponding to the visual question data.

6. The visual question answering method according to any one of claims 1 to 5, characterized in that: The step of searching in a preset template library according to the multimodal feature data to obtain at least one target example question and answer data matching the multimodal feature data includes: Comparing the multimodal feature data with the standard multimodal feature data corresponding to each example question and answer data in the preset template library to obtain the feature similarity corresponding to each example question and answer data; At least one target example question and answer data matching the multimodal feature data is selected from the preset template library based on the feature similarity.

7. The visual question answering method according to claim 6, wherein: The selecting at least one target example question-answer data matching the multimodal feature data from the preset template library based on the feature similarity includes: Sort the example question and answer data in the preset template library from large to small based on the corresponding feature similarity to generate a sorting result; The top N example question and answer data in the sorting result are selected as target example question and answer data matching the multimodal feature data, where N is a positive integer.

8. A visual question-answering device, characterized in that: The device comprises: An extraction module, used to extract features from the visual problem data and generate multimodal feature data corresponding to the visual problem data; A search module, configured to search in a preset template library according to the multimodal feature data to obtain at least one target example question and answer data matching the multimodal feature data, wherein the preset template library includes a plurality of example question and answer data and standard multimodal feature data corresponding to each example question and answer data; A construction module, configured to construct a model prompt word according to the at least one target example question and answer data; A generation module, configured to input the model prompt word and the visual question data into a large language model, so that the large language model generates a visual question answering result according to the visual question data under the guidance of the model prompt word; The visual problem data includes image data and problem data, the image data includes visual documents, and the visual documents include scanned copies, photographed pictures or generated images.

9. A visual question-answering device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the visual question answering method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the visual question answering method according to any one of claims 1 to 7 are implemented.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the visual question answering method according to any one of claims 1 to 7 are implemented.

Images