Visual question and answer method and system, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a visual question and answer method and system, electronic equipment and a storage medium. The method comprises the steps that a first question text in a first image is acquired; determining a first hash value of the first question text; retrieving a second question text in a preset question bank according to the first hash value; and determining a first answer text of the first question text according to a retrieval result. On the basis of supporting multi-question answering, the cost is reduced, and the question answering quality is ensured.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular to a visual question answering method, system, electronic device, and storage medium. Background Art

[0002] Visual Question Answering (VQA) is a technology that involves computer vision and natural language processing. It can be applied to scenarios where questions presented in the form of images need to be answered, such as in educational tutoring scenarios.

[0003] Currently, visual question answering methods typically involve: when an image contains a single question, searching an image question bank based on image similarity to retrieve the answer; when an image contains two or more questions, generating the answer based on the image using a multimodal model. However, this method of recalling answers from an image question bank cannot support answer recall for multiple questions; and the method of generating answers using a multimodal model is prone to repeated reasoning, missing questions, and incorrect answers. This not only increases the reasoning cost but also hinders the quality of the answers. Summary of the Invention

[0004] The embodiments of the present disclosure provide a visual question answering method, system, electronic device, and storage medium, which can reduce costs and ensure answer quality while supporting multiple questions.

[0005] In a first aspect, an embodiment of the present disclosure provides a visual question answering method, comprising:

[0006] Obtaining the first title text in the first image;

[0007] Determine a first hash value of the first topic text;

[0008] Retrieving a second question text from a preset question bank according to the first hash value;

[0009] According to the search results, a first answer text of the first question text is determined.

[0010] In a second aspect, the present disclosure also provides a visual question answering system, including:

[0011] An acquisition module, configured to acquire the first title text in the first image;

[0012] A hash value determination module, configured to determine a first hash value of the first question text;

[0013] A retrieval module, configured to retrieve a second question text in a preset question bank according to the first hash value;

[0014] The answer module is used to determine a first answer text for the first question text according to the search results.

[0015] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0016] one or more processors;

[0017] a storage device for storing one or more programs,

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the visual question answering method as described in any one of the embodiments of the present disclosure.

[0019] In a fourth aspect, an embodiment of the present disclosure further provides a storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to execute the visual question answering method as described in any one of the embodiments of the present disclosure.

[0020] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, characterized in that the computer program product includes a computer program, and when the computer program is executed by a processor, it implements the visual question answering method as described in any one of the embodiments of the present disclosure.

[0021] In the technical solution of the embodiment of the present disclosure, the first question text in the first image can be obtained; the first hash value of the first question text can be determined; based on the first hash value, the second question text can be retrieved from the preset question bank; and based on the retrieval results, the first answer text of the first question text can be determined. By searching the preset question bank based on the first hash value of the first question text, not only can the first answer text be recalled for each first question text when the first image contains at least one first question text, thereby supporting the scenario of recalling multiple questions, but the matching accuracy of the question text can also be improved, thereby avoiding the situation of missing questions or incorrect questions. By recalling the answer text based on the preset question bank, the situation of repeated reasoning for repeated questions can be avoided, thereby reducing the reasoning cost. In addition, because the quality of the answer text in the preset question bank can be controlled in advance, the quality of the first answer text determined based on the preset question bank retrieval results can also be guaranteed. That is, it is possible to reduce costs and ensure the quality of answers while supporting the answering of multiple questions. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0023] Figure 1A flowchart of a visual question answering method provided by an embodiment of the present disclosure;

[0024] Figure 2 A schematic block diagram of the data flow of a visual question answering method provided by an embodiment of the present disclosure;

[0025] Figure 3 A schematic block diagram of data flow for determining the sixth question text in a visual question answering method provided by an embodiment of the present disclosure;

[0026] Figure 4 A schematic block diagram of the data flow for updating a preset question bank in a visual question answering method provided by an embodiment of the present disclosure;

[0027] Figure 5 A schematic block diagram of the data flow for question bank generation and consumption in a visual question answering method provided by an embodiment of the present disclosure;

[0028] Figure 6 A schematic diagram of the structure of a visual question answering system provided by an embodiment of the present disclosure;

[0029] Figure 7 A schematic structural diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0031] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0032] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0033] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0034] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0035] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0036] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0037] Figure 1 This is a flow chart of a visual question-answering method provided by an embodiment of the present disclosure. This embodiment of the present disclosure is applicable to situations where questions presented in the form of images are answered, for example, to situations where at least one question contained in an image is answered. This method can be performed by a visual question-answering system, which can be implemented in software and / or hardware. The system can be integrated into application software, such as artificial intelligence-based conversational application software, and can be installed along with the application software in electronic devices, such as mobile phones, computers, and other computer devices.

[0038] like Figure 1 As shown, the visual question answering method provided in this embodiment may include:

[0039] S110: Obtain a first title text in a first image.

[0040] In an embodiment of the present disclosure, the first image can be received through a user interface provided by the visual question answering system. For example, the first image input by the user can be received through a conversation window of a dialogue application software to which the visual question answering system belongs. The first question text may include the question text to be answered in the first image. All question texts can be extracted from the first image through technologies such as optical character recognition (OCR), and at least part of the question text in all question texts can be determined as the first question text to obtain the first question text in the first image. The number of first question texts may include at least one.

[0041] In some optional implementations, obtaining the first question text in the first image may include: in response to receiving the first image, extracting and storing the third question text in the first image; in response to identifying that the processing requirements corresponding to the first image belong to preset requirements, obtaining the first question text from the stored third question text.

[0042] For example, Figure 2 This is a schematic diagram of the data flow of a visual question answering method provided by an embodiment of the present disclosure. Figure 2 , a first image and a first conversation content can be received through a user interface provided by the visual question answering system; wherein, at least one first image can be received, and at least one round of first conversation content can be received.

[0043] The first conversation content and the first image may have a corresponding relationship. The corresponding first image may be determined based on the first conversation content. For example, assuming that the first conversation content includes "What is the answer to question x in the picture above", then the first image corresponding to the first conversation content may include the first image received before the first conversation content; assuming that the first conversation content includes "Please answer all the questions in the picture below", then the first image corresponding to the first conversation content may include the first image received after the first conversation. In addition, if the corresponding first image cannot be determined based on the first conversation content, the corresponding relationship between the first conversation content and the first image may also be determined based on a first preset rule. For example, the first image received most recently in history may be used as the first image corresponding to the first conversation content by default.

[0044] like Figure 2 As shown, after receiving the first image, the OCR technology can be used to cut the question and obtain the entire question text in the first image (i.e., the third question text). In addition, the third question text can be stored in a preset storage space, such as Figure 2 can be stored in the online cache.

[0045] After receiving the first conversation content, the corresponding first image can be determined according to the first conversation content, and the processing requirements corresponding to the first image can be determined. Among them, the corresponding first image and processing requirements can be determined according to the first conversation content through an existing neural network model with natural language processing capabilities. Among them, the processing requirements may include but are not limited to: image editing (such as photo editing, cutouts, partial replacement, etc.), image recognition (such as identifying animals and plants, identifying the same items, etc.), image description (such as describing pictures, etc.) and picture problem solving, etc. Among them, the preset requirements may include picture problem solving.

[0046] When it is determined that the processing requirement corresponding to the first image is image recognition and problem solving, the first problem text corresponding to the processing requirement can be obtained from the stored third problem text. For example, assuming that the third problem text includes three problem texts numbered 3-5, and the processing requirement includes solving the problem texts numbered 4-5, the problem texts numbered 4-5 in the third problem text can be obtained as the first problem text.

[0047] The third question text stored in the preset storage space can be cleared according to a preset clearing policy, which can include, for example, a time policy and / or a frequency policy. If the preset clearing policy includes a time policy, third question texts stored in the preset storage space for a preset time period can be cleared. If the preset clearing policy includes a frequency policy, third question texts used for recalling answers less than a preset number of times can be cleared. Clearing the preset storage space can improve storage space utilization.

[0048] In these optional implementations, by extracting the entire question text upon receiving the first image, the first question text can be quickly obtained when it is identified that the processing requirement belongs to the preset requirement, which can reduce the problem-solving delay to a certain extent and improve the user experience.

[0049] In addition, reference Figure 2 When it is identified that the processing requirement corresponding to the first image does not belong to the preset requirement, other processing flows may be taken, which will not be elaborated in detail here.

[0050] S120: Determine a first hash value of the first question text.

[0051] In the disclosed embodiment, a text normalization hash method can be used to determine the first hash value of the first question text. For example, the first question text can be normalized first, and then a preset hash algorithm can be used to calculate the hash value of the normalized first question text to obtain the first hash value. The normalization of the first question text can include, but is not limited to: removing formatting characters such as spaces and line breaks in the first question text; converting characters in the first question text to lowercase, etc.

[0052] S130. Retrieve the second question text in a preset question bank according to the first hash value.

[0053] In the disclosed embodiments, the preset question bank may include pre-set question texts, answer texts corresponding to the question texts, and question information. The question information may include, but is not limited to, hash values, labels, feature vectors, question identifiers, and the like. The preset question bank may be searched for a hash value identical to a first hash value to retrieve a second question text identical to the first question text.

[0054] Since hash values ​​typically have a unique correspondence with text, the present disclosure enables rapid retrieval of the original question by searching for the second question text based on the first hash value, improving the speed and accuracy of question text matching and avoiding incorrect question recalls. Furthermore, compared to traditional image question bank recall based on image similarity, the present disclosure uses question text recall based on the first hash value corresponding to at least one first question text, which not only supports multi-question recall scenarios but also avoids missed questions.

[0055] In some optional implementations, it may also include: obtaining a first tag of the first question text; correspondingly, searching for the second question text in a preset question bank based on the first hash value, which may include: performing a first-stage search in the preset question bank based on the first tag; and searching for the second question text in the first-stage search results based on the first hash value.

[0056] The dimensions of the first label and the label in the question information of the preset question bank may be the same, for example, they may include but are not limited to grade dimension labels, subject dimension labels, and language dimension labels. The corresponding first label can be determined based on the first question text by a second preset rule or machine learning method. The first stage search may include searching the preset question bank for question texts with the same label as the first label. Afterwards, the question text with the same hash value as the first hash value can be searched again from the first stage search results to obtain the second question text.

[0057] In these optional implementations, by first filtering the preset question text using the first tag and then retrieving the second question text using the first hash value, efficient and accurate question bank retrieval can be achieved.

[0058] S140. Determine a first answer text for the first question text based on the search results.

[0059] In the embodiment of the present disclosure, when there is only one first topic text, the search results may include the second topic text being retrieved and the second topic text not being retrieved. When there are two or more first topic texts, the search results may include the second topic texts being retrieved in their entirety, the second topic texts being partially retrieved, and the second topic texts not being retrieved at all.

[0060] Among them, determining the first answer text of the first question text based on the retrieval results may include: in response to retrieving the second question text, obtaining the second answer text corresponding to the second question text from a preset question bank; determining the first answer text based on the second answer text; in response to not retrieving the second question text, generating the first answer text based on the first question text through a first generation model.

[0061] For first question texts that retrieve a second question text, a high-quality answer text corresponding to the second question text (i.e., the second answer text) can be directly returned from the preset question bank as the first answer text. For first question texts that do not retrieve a second question text, an online first generative model can be used to answer the question in real time, generating a corresponding first answer text based on the first question text. Typically, a neural network model with strong answering capabilities can be selected for the first generative model to ensure the quality of the first answer text.

[0062] For example, if there is only one first question text, if a second question text is retrieved, the second answer text corresponding to the second question text can be used as the first answer text of the first question text. If the second question text is not retrieved, the first answer text corresponding to the first question text can be generated using the first generation model.

[0063] If there are two or more first question texts, if all second question texts are retrieved, the second answer text corresponding to each second question text can be used as the first answer text for the corresponding first question text. If no second question texts are retrieved, the first answer text corresponding to each first question text can be generated using the first generation model. If some second question texts are retrieved, the second answer text corresponding to the retrieved second question text can be used as the first answer text for the first question text for which the second question text is retrieved; if no second question text is retrieved, the first answer text can be generated using the first generation model.

[0064] In some optional implementations, generating a first answer text based on a first question text through a first generation model may include: obtaining a first feature vector of the first question text; retrieving a fourth question text from a preset question bank based on the first feature vector; in response to retrieving the fourth question text, obtaining a third answer text corresponding to the fourth question text from the preset question bank; and generating a first answer text based on the first question text and the third answer text through the first generation model.

[0065] The first feature vector and the feature vector in the question information of the preset question bank can be determined in the same manner. The similarity between the first feature vector and each feature vector in the preset question bank can be determined, and the fourth question text can be determined based on the similarity. For example, a preset number of question texts with a similarity greater than a preset threshold can be determined as the fourth question text. The answer text corresponding to the fourth question text in the preset question bank can be referred to as the third answer text. The fourth question text and the third answer text can be used to guide the first generation model to generate the first answer text corresponding to the first question text, thereby improving the quality and accuracy of the answer.

[0066] In these optional implementations, if the original text of the first question is not retrieved, a similar fourth question text can be retrieved based on the first feature vector. Furthermore, the high-quality third answer text corresponding to the fourth question text can be used as reference information to guide the generation of the first answer text, thereby improving the quality and accuracy of online answers.

[0067] In addition, see again Figure 2 In some implementations, while searching for the second question text in the preset question bank based on the first hash value, the fourth question text can be searched in the preset question bank based on the first feature vector. Subsequently, whether the question text has been retrieved can be determined by determining whether the second question text and / or the fourth question text have been retrieved. If neither the second question text nor the fourth question text has been retrieved, the question text has not been retrieved, and the first answer text can be directly generated using the first generation model. If the second question text and / or the fourth question text has been retrieved, the question text has been retrieved, and further determination can be made as to whether the second question text has been retrieved. If the second question text has been retrieved, the second answer text corresponding to the second question text can be returned. If the second question text has not been retrieved, it can be deemed that the fourth question text has been retrieved, and the first answer text can be generated by referring to the third answer text corresponding to the fourth question text.

[0068] In these implementation methods, by combining hash values ​​and feature vectors to retrieve preset question banks, the recall coverage of preset question banks can be improved, and efficient and accurate question bank answer retrieval can be achieved.

[0069] In the technical solution of the embodiment of the present disclosure, the first question text in the first image can be obtained; the first hash value of the first question text can be determined; based on the first hash value, the second question text can be retrieved from the preset question bank; and based on the retrieval results, the first answer text of the first question text can be determined. By searching the preset question bank based on the first hash value of the first question text, not only can the first answer text be recalled for each first question text when the first image contains at least one first question text, thereby supporting the scenario of recalling multiple questions, but the matching accuracy of the question text can also be improved, thereby avoiding the situation of missing questions or incorrect questions. By recalling the answer text based on the preset question bank, the situation of repeated reasoning for repeated questions can be avoided, thereby reducing the reasoning cost. In addition, because the quality of the answer text in the preset question bank can be controlled in advance, the quality of the first answer text determined based on the preset question bank retrieval results can also be guaranteed. That is, it is possible to reduce costs and ensure the quality of answers while supporting the answering of multiple questions.

[0070] The embodiments of the present disclosure can be combined with the various optional solutions in the visual question answering method provided in the above embodiments. The visual question answering method provided in this embodiment describes in detail the updating process of the preset question bank.

[0071] In the visual question-answering method provided in this embodiment, the updating process of the preset question bank may include: determining a sixth question text based on a fifth question text and a candidate question bank every first preset time period; wherein the fifth question text includes the first question text obtained within a second preset time period; generating a fourth answer text based on the sixth question text through at least two second generation models; and updating the preset question bank based on the sixth question text and the fourth answer text.

[0072] In this embodiment, the preset question bank can be updated after it is built. The first preset time period and the second preset time period can be pre-set based on experimental values ​​or empirical values. For example, the first preset time period can be 1 day, 2 days, etc., and the second preset time period can be 1 week, 2 weeks, etc.

[0073] After each determination of the first answer to the first question, the first question, the first answer, and question information (such as the first hash value, the first label, and the first feature vector) of the first question are recorded. At intervals of a first preset time period, the first question within a second preset time period from the current question is obtained from the record as the fifth question.

[0074] The candidate question bank can be considered a predecessor question bank used to update the preset question bank. It can be considered that the candidate question bank contains not only the data for the preset question bank, but also the predecessor data used to update the preset question bank. By deploying the candidate question bank, read and write operations can be performed on the predecessor data used to update the preset question bank based on the candidate question bank. Furthermore, the preset question bank can be updated using the candidate question bank during idle periods of the image recognition and problem-solving service, thereby ensuring online service performance.

[0075] Determining the sixth question text based on the fifth question text and the candidate question bank may include: removing duplicates from the fifth question text, and removing duplicates from the fifth question text based on the candidate question bank. Because the candidate question bank includes question texts from the preset question bank, the fifth question text (i.e., the sixth question text) for generating the first answer text using the first generated text can be filtered out.

[0076] The at least two second generation models can be used to regenerate the answer text based on the sixth question text, thereby generating at least two fourth answer texts. Furthermore, if the first generation model is a relatively powerful model, the at least two second generation models can be used to generate a fourth answer text based on the first answer text corresponding to the sixth question text, thereby improving the quality of the fourth answer text.

[0077] The preferred final answer text can be determined by combining at least two fourth answer texts to further ensure the quality of the answer text for the sixth question. The sixth question text, its corresponding final answer text, and question information can be stored in a candidate database. Furthermore, during the idle period of the image recognition and problem-solving service, the preset question bank can be updated through the candidate question bank to achieve the update of the preset question bank.

[0078] Because the final answer can be determined by combining at least two fourth-answer texts, the performance requirements for the second generative model are relatively relaxed. A more general neural network model can be used to generate the fourth-answer text, ensuring both quality and cost. Furthermore, if costs are sufficient, a more powerful model can be selected for the second generative model to further improve answer quality.

[0079] In some optional implementations, determining the sixth question text based on the fifth question text and the candidate question bank may include: determining a second hash value of the fifth question text; counting the frequency of occurrence of the fifth question text based on the second hash value; determining the seventh question text from the fifth question text based on the frequency of occurrence; in response to the candidate question bank containing the seventh question text, eliminating the seventh question text; and determining the sixth question text based on the remaining seventh question text after elimination.

[0080] For example, Figure 3 This is a schematic diagram of the data flow for determining the sixth question text in a visual question answering method provided by an embodiment of the present disclosure. Figure 3 The process of determining the text of the sixth question may include:

[0081] First, idle resources can be used to write the historical first question text, first answer text, and question information of the first question text into the original message table.

[0082] Secondly, every first preset time period, the first title text within the current second preset time period can be obtained from the original message table as the fifth title text, and the fifth title text and corresponding title information can be written into the picture title table.

[0083] Next, the first hash value (which can be called the second hash value in this case) corresponding to the fifth topic text can be obtained from the topic information. The fifth topic texts can be aggregated according to the second hash value to count the frequency of occurrence of each fifth topic text. The fifth topic texts with a frequency of occurrence higher than a preset frequency can be used as the seventh topic text to construct a high-frequency topic table containing the seventh topic text. Among them, the original message table, the image topic table, and the high-frequency topic table can be constructed in the form of a preset database table.

[0084] Next, the candidate question bank can be searched based on the seventh question text in the high-frequency question table. For example, the candidate question bank can be searched based on the second hash value corresponding to the seventh question text to see if the original question exists.

[0085] Finally, when the seventh question text exists in the candidate question bank, it can be considered that the high-quality answer text corresponding to the seventh question text already exists in the preset question bank, and the seventh question text can be eliminated. When the seventh question text is not included in the candidate question bank, it can be considered that the preset question bank does not contain a high-quality answer text for the seventh question text, and the seventh question text can be determined as the sixth question text. Figure 3 , the text of the sixth question can be delivered to the answer text generation stage through the message queue (MQ) method.

[0086] In these optional implementations, by determining the frequency of occurrence of the fifth question text based on the hash value, it is possible to quickly and inexpensively determine the repeatability and aggregate the massive question texts. In addition, other methods can be used to aggregate the fifth question text, such as using feature vectors, which are not exhaustive here. By screening out the seventh question text that appears frequently from the fifth question text, it is beneficial to improve the hit rate of the question text in the preset question bank, thereby reducing the cost of reasoning and improving the quality of the answer text. By deduplicating the seventh question text based on the candidate question bank, it is possible to avoid repeated answers to questions for which the answer text already exists in the preset question bank, thereby avoiding waste of resources.

[0087] In some optional implementations, generating a fourth answer text based on the sixth question text through at least two second generation models may include: verifying the sixth question text according to a first preset dimension; and in response to passing the verification, generating a fourth answer text based on the sixth question text through at least two second generation models.

[0088] The sixth question text may be obtained from the message queue and verified against a first preset dimension. The first preset dimension may include, but is not limited to, completeness and clarity. The completeness, clarity, and other dimensions of the sixth question text may be verified using a pre-built neural network model. By verifying the completeness of the sixth question text, incomplete question texts that require additional information to answer may be excluded. By verifying the clarity of the sixth question text, question texts with unclear semantic references may be excluded.

[0089] In addition, while recording the first question text, the first answer text, and the question information of the first question text, other question information of the first question text may also be determined, such as the image category (e.g., photographed, screenshot, etc.), image clarity (e.g., clear, unclear), etc. This question information may be used to subsequently verify the first preset dimension of the sixth question text. For example, based on the image clarity in the question information, the sixth question text that may contain erroneous information may be excluded.

[0090] The verification results may include a passed verification and a failed verification. The sixth question text and its corresponding verification result may be stored as pre-order data in a candidate question bank. The sixth question text that passed the verification may be obtained from the candidate question bank, and a fourth answer text may be generated based on the sixth question text using at least two second generation models. For sixth question texts that failed the verification, generation of the fourth answer text may be stopped.

[0091] In these optional implementations, by performing verification before answering the sixth question text, invalid or incomplete question texts can be filtered out, thereby avoiding the subsequent waste of high-cost computing resources.

[0092] In some optional implementations, updating the preset question bank based on the sixth question text and the fourth answer text may include: evaluating the fourth answer text in a second preset dimension through a referee model; wherein the second preset dimension includes separate evaluation and / or cross-validation; determining the fifth answer text from the fourth answer text based on the evaluation results; and updating the preset question bank based on the sixth question text and the fifth answer text.

[0093] For example, Figure 4 This is a schematic diagram of the data flow for updating the preset question bank in a visual question answering method provided by an embodiment of the present disclosure. Figure 4 , after getting the sixth question text from the message queue and verifying it, it can be generated through n second generation models ( Figure 4 denoted as Model 1-Model n), and respectively generate fourth answer texts 1-fourth answer texts n. The fourth answer texts 1-fourth answer texts n can be individually input into the referee model so that the referee model evaluates each fourth answer text individually; and / or, the fourth answer texts 1-fourth answer texts n can be simultaneously input into the referee model so that the referee model cross-validates each fourth answer text. The referee model can include a pre-built neural network model with answer text evaluation capabilities. Furthermore, a superior final answer text, i.e., the fifth answer text, can be screened out from each fourth answer text.

[0094] In addition, for some special cases, for example, if more than half of the answers in the fourth answer text 1-fourth answer text n are refused or clarified, the sixth question text can be specially processed (such as discarded or manually intervened).

[0095] The updating of the preset question bank based on the sixth question text and the fifth answer text can include: first updating the candidate question bank based on the sixth question text and the fifth answer text; then, when the image recognition and problem solving service is idle, updating the preset question bank based on the updated candidate question bank. This can avoid affecting the performance of the image recognition and problem solving service.

[0096] See again Figure 4 Updating the preset question bank according to the sixth question text and the fifth answer text may include: obtaining question information corresponding to the sixth question text; wherein the question information includes a third hash value; updating the candidate question bank according to the sixth question text, the fifth answer text and the question information; and updating the preset question bank according to the updated candidate question bank.

[0097] The question information corresponding to the sixth question text may also include but is not limited to information such as hash values, labels, and feature vectors. The hash value corresponding to the sixth question text may be referred to as a third hash value. The third hash value corresponding to the sixth question text may be obtained from the recorded first hash value. The sixth question text, the fifth answer text, and the question information may be stored in a candidate question bank to update the candidate question bank. Furthermore, the preset question bank may be updated based on the updated candidate question bank. In addition, see Figure 4 , the text of the seventh question in the subsequent high-frequency question table can be retrieved based on the updated candidate question bank.

[0098] In these optional implementations, by introducing a referee model to individually evaluate and / or cross-validate the fourth answer text, high-quality fifth answer texts can be screened out, thereby facilitating the generation of a preset question bank with higher accuracy.

[0099] In some optional implementations, the question information may further include a question identifier; the method may further include: collecting statistics on usage of question texts in a preset question bank according to the question identifier; and updating the preset question bank based on the question identifier and usage.

[0100] The question identification (ID) information of the sixth question text can be automatically generated, and the question identification information can be written into the preset question bank when the sixth question text and the fifth answer text are written into the preset question bank.

[0101] After determining the first answer text for the first question text, the first answer text may be displayed on the user interface. For example, the first answer text may be displayed in a conversation window of a dialogue-type application software to which the visual question-answering system belongs. In addition, the display control of the first answer text may also be associated with a user feedback control (such as a like control, a dislike control, or a rating control, etc.). In response to the triggering of the user feedback control, feedback information may be received, and the feedback information may belong to usage. In addition, the usage may also include other information, such as the hit ratio of the question text, the recall frequency, and other information.

[0102] In these optional implementation methods, the usage of each question text in the preset question bank can be counted according to the question identifier, and the preset question bank can be updated according to the usage, which helps to clear the question texts in the preset question bank with poor user feedback, low hit rate, and low recall frequency, so as to realize the iterative update of the preset question bank, save question bank resources and optimize the user experience.

[0103] For example, Figure 5 This is a schematic diagram of the data flow for generating and consuming a question bank in a visual question answering method provided by an embodiment of the present disclosure. Figure 5 The question bank generation and consumption process may include three steps: question screening process, question bank production process, and question bank consumption process. The question screening process may include the writing process of the original message table, picture question table, and high-frequency question table described above. The question bank production process may include the process of obtaining the verified sixth question text from the candidate question bank described above, and calling at least two second generation models to generate the fourth answer text based on the sixth question text; screening the fifth answer text through the referee model, and updating the preset question bank based on the sixth question text and the fifth answer text. The question bank consumption process may include the process of processing requirement identification, first question text acquisition, and preset question bank recall described above.

[0104] By generating and filtering high-quality answers offline and storing them in a pre-set question bank, we can provide more accurate answers to frequently encountered secondary questions in the question bank than the model's real-time answers, improving accuracy. For numerous repeated image-based question-solving requests, directly returning answers from the pre-set question bank avoids repeated model invocations, significantly saving computing resources and costs. Providing a more accurate and responsive image-based problem-solving service improves user experience and helps boost user retention.

[0105] The technical solution of the embodiment of the present disclosure describes in detail the updating process of the preset question bank. By screening the fifth question text in the second image received in the historical time period (i.e., the second preset time period) for high-frequency question texts every first preset time period, the recall rate of the question texts in the preset question bank can be improved, thereby improving the utilization rate of the preset question bank. By deduplicating the high-frequency question texts based on the candidate question bank to obtain the sixth question text, it is possible to avoid generating answer texts for repeated high-frequency questions again, thereby reducing resource consumption and avoiding redundancy in the preset question bank. By combining the fourth answer text of the sixth question text generated by multiple second generation models to update the preset question bank, the quality of the answer texts in the preset question bank can be improved.

[0106] In addition, the visual question answering method provided by the embodiment of the present disclosure and the visual question answering method provided by the above embodiment belong to the same public concept. Technical details not fully described in this embodiment can be found in the above embodiment, and the same technical features have the same beneficial effects in this embodiment and the above embodiment.

[0107] Figure 6 The following is a schematic diagram of the structure of a visual question answering system provided by an embodiment of the present disclosure. The visual question answering system provided by this embodiment is applicable to situations where questions in the form of images are answered, for example, situations where at least one question contained in an image is answered.

[0108] like Figure 6 As shown, the visual question answering system provided by the embodiment of the present disclosure may include:

[0109] An acquisition module 610 is configured to acquire a first title text in a first image;

[0110] A hash value determination module 620 is configured to determine a first hash value of the first question text;

[0111] A retrieval module 630 is configured to retrieve a second question text in a preset question bank according to the first hash value;

[0112] Answer module 640, for determining the first answer text of the first question text according to the search results

[0113] In some optional implementations, the acquisition module can be used to:

[0114] In response to receiving the first image, extracting and storing the third topic text in the first image;

[0115] In response to identifying that the processing requirement corresponding to the first image belongs to a preset requirement, the first question text is obtained from the stored third question text.

[0116] In some optional implementations, the answer module can be used to:

[0117] In response to retrieving the second question text, obtaining a second answer text corresponding to the second question text from a preset question bank; and determining the first answer text based on the second answer text;

[0118] In response to not retrieving the second question text, a first answer text is generated based on the first question text by using the first generation model.

[0119] In some optional implementations, the answer module can be used to:

[0120] Obtain the first eigenvector of the first question text;

[0121] Retrieving the text of the fourth question in a preset question bank according to the first feature vector;

[0122] In response to retrieving the fourth question text, obtaining a third answer text corresponding to the fourth question text from a preset question bank;

[0123] A first answer text is generated according to the first question text and the third answer text through the first generation model.

[0124] In some optional implementations, the acquisition module may also be used to:

[0125] Get the first label of the first question text;

[0126] Accordingly, the retrieval module can be used to:

[0127] According to the first tag, the first stage search is performed in the preset question bank;

[0128] According to the first Hash value, the second topic text is retrieved from the first-stage search results.

[0129] In some optional implementations, the visual question answering system may include:

[0130] The question bank update module can be used to update the preset question bank based on the following process:

[0131] Every first preset time period, determining a sixth question text based on the fifth question text and the candidate question bank; wherein the fifth question text includes the first question text obtained within the second preset time period;

[0132] generating a fourth answer text based on the sixth question text by at least two second generation models;

[0133] The preset question bank is updated according to the text of the sixth question and the text of the fourth answer.

[0134] In some optional implementations, the question bank update module can be used to:

[0135] Determine a second hash value of the fifth question text;

[0136] Counting the occurrence frequency of the fifth question text according to the second hash value;

[0137] According to the frequency of occurrence, determine the seventh question text from the fifth question text;

[0138] In response to the candidate question bank including the seventh question text, eliminating the seventh question text;

[0139] Determine the text of question six based on the text of question seven remaining after elimination.

[0140] In some optional implementations, the question bank update module can be used to:

[0141] Verify the text of question six based on the first preset dimension;

[0142] In response to the verification being passed, a fourth answer text is generated according to the sixth question text by using at least two second generation models.

[0143] In some optional implementations, the question bank update module can be used to:

[0144] Evaluating the fourth answer text on a second preset dimension using the referee model; wherein the second preset dimension includes individual evaluation and / or cross-validation;

[0145] determining a fifth answer text from the fourth answer text according to the evaluation result;

[0146] The preset question bank is updated according to the sixth question text and the fifth answer text.

[0147] In some optional implementations, the question bank update module can be used to:

[0148] Obtaining topic information corresponding to the sixth topic text; wherein the topic information includes the third hash value;

[0149] updating the candidate question bank based on the sixth question text, the fifth answer text, and question information;

[0150] The preset question bank is updated according to the updated candidate question bank.

[0151] In some optional implementations, the question information also includes a question identifier; the question bank update module can also be used to:

[0152] According to the question ID, statistics are collected on the usage of the question texts in the preset question bank;

[0153] Update the preset question bank based on question identification and usage.

[0154] The visual question answering system provided by the embodiments of the present disclosure can execute the visual question answering method provided by any embodiment of the present disclosure, and has the functional modules and beneficial effects corresponding to the execution method.

[0155] It is worth noting that the various units and modules included in the above system are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0156] Reference below Figure 7 , which shows an electronic device (eg Figure 7 The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0157] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the electronic device 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0158] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display, a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Figure 7The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0159] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the visual question answering method of the embodiment of the present disclosure are performed.

[0160] The electronic device provided by the embodiment of the present disclosure and the visual question answering method provided by the above embodiment belong to the same disclosed concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0161] The embodiments of the present disclosure provide a storage medium of computer-executable instructions. When the computer-executable instructions are executed by a computer processor, they can be used to execute the visual question answering method provided by the above embodiments.

[0162] It should be noted that the storage medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a flash memory (FLASH), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing executable instructions that can be used by or in conjunction with an instruction execution system, system or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable executable instructions. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit executable instructions for use by or in conjunction with an instruction execution system, system, or device. The executable instructions contained on the storage medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0163] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0164] The storage medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0165] The storage medium carries one or more executable instructions. When the one or more executable instructions are executed by the electronic device, the electronic device:

[0166] Acquire a first question text in a first image; determine a first hash value of the first question text; retrieve a second question text in a preset question bank based on the first hash value; and determine a first answer text for the first question text based on the retrieval result.

[0167] The executable instructions for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The executable instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0168] An embodiment of the present disclosure further provides a computer program product, including a computer program, which, when executed by a processor, can implement the visual question answering method provided in any embodiment of the present disclosure.

[0169] The computer program product includes a computer program carried on a non-transitory computer-readable medium, the computer program including a program code for executing the visual question answering method. The program code can be written in one or more programming languages ​​or a combination thereof, the programming language including an object-oriented programming language such as Java, Smalltalk, C++, and also including a conventional procedural programming language such as "C" language or a similar programming language. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, via the Internet using an Internet service provider).

[0170] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0171] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the names of the units and modules do not, in certain circumstances, limit the units and modules themselves.

[0172] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and the like.

[0173] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, system, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0174] According to one or more embodiments of the present disclosure, a visual question answering method is provided, comprising:

[0175] Obtaining the first title text in the first image;

[0176] Determine a first hash value of the first topic text;

[0177] Retrieving a second question text from a preset question bank according to the first hash value;

[0178] According to the search results, a first answer text of the first question text is determined.

[0179] According to one or more embodiments of the present disclosure, a visual question answering method is provided, further comprising:

[0180] In some optional implementations, obtaining the first title text in the first image includes:

[0181] In response to receiving the first image, extracting and storing a third topic text in the first image;

[0182] In response to identifying that the processing requirement corresponding to the first image belongs to a preset requirement, the first topic text is obtained from the stored third topic text.

[0183] According to one or more embodiments of the present disclosure, a visual question answering method is provided, further comprising:

[0184] In some optional implementations, determining the first answer text to the first question text based on the search results includes:

[0185] In response to retrieving the second question text, obtaining a second answer text corresponding to the second question text from the preset question bank; and determining the first answer text based on the second answer text;

[0186] In response to not retrieving the second question text, the first answer text is generated according to the first question text by using a first generation model.

[0187] According to one or more embodiments of the present disclosure, a visual question answering method is provided, further comprising:

[0188] In some optional implementations, generating the first answer text according to the first question text using a first generation model includes:

[0189] Obtaining a first feature vector of the first question text;

[0190] Retrieving a fourth question text in the preset question bank according to the first feature vector;

[0191] In response to retrieving the fourth question text, obtaining a third answer text corresponding to the fourth question text from the preset question bank;

[0192] The first answer text is generated according to the first question text and the third answer text through the first generation model.

[0193] According to one or more embodiments of the present disclosure, a visual question answering method is provided, further comprising:

[0194] In some optional implementations, obtaining a first tag of the first topic text;

[0195] The step of retrieving the second question text from a preset question bank according to the first Hash value includes:

[0196] Perform a first-stage search in the preset question bank according to the first tag;

[0197] The second title text is retrieved from the first-stage search results according to the first Hash value.

[0198] According to one or more embodiments of the present disclosure, a visual question answering method is provided, further comprising:

[0199] In some optional implementations, the updating process of the preset question bank includes:

[0200] every first preset time period, determining a sixth question text based on the fifth question text and the candidate question bank; wherein the fifth question text includes the first question text obtained within the second preset time period;

[0201] generating a fourth answer text according to the sixth question text by at least two second generation models;

[0202] The preset question bank is updated according to the sixth question text and the fourth answer text.

[0203] According to one or more embodiments of the present disclosure, a visual question answering method is provided, further comprising:

[0204] In some optional implementations, determining the sixth question text based on the fifth question text and the candidate question bank includes:

[0205] Determine a second hash value of the fifth question text;

[0206] Counting the occurrence frequency of the fifth question text according to the second hash value;

[0207] Determining a seventh topic text from the fifth topic text according to the occurrence frequency;

[0208] In response to the candidate question bank including the seventh question text, eliminating the seventh question text;

[0209] The sixth question text is determined based on the seventh question text remaining after elimination.

[0210] According to one or more embodiments of the present disclosure, a visual question answering method is provided, further comprising:

[0211] In some optional implementations, generating a fourth answer text based on the sixth question text using at least two second generation models includes:

[0212] Performing verification of the sixth question text according to a first preset dimension;

[0213] In response to the verification being passed, a fourth answer text is generated according to the sixth question text by using at least two second generation models.

[0214] According to one or more embodiments of the present disclosure, a visual question answering method is provided, further comprising:

[0215] In some optional implementations, updating the preset question bank based on the sixth question text and the fourth answer text includes:

[0216] Performing an evaluation of a second preset dimension on the fourth answer text using the referee model; wherein the second preset dimension includes individual evaluation and / or cross-validation;

[0217] determining a fifth answer text from the fourth answer text according to the evaluation result;

[0218] The preset question bank is updated according to the sixth question text and the fifth answer text.

[0219] According to one or more embodiments of the present disclosure, a visual question answering method is provided, further comprising:

[0220] In some optional implementations, updating the preset question bank based on the sixth question text and the fifth answer text includes:

[0221] Obtaining topic information corresponding to the sixth topic text; wherein the topic information includes a third hash value;

[0222] updating the candidate question bank according to the sixth question text, the fifth answer text, and the question information;

[0223] The preset question bank is updated according to the updated candidate question bank.

[0224] According to one or more embodiments of the present disclosure, a visual question answering method is provided, further comprising:

[0225] In some optional implementations, the topic information further includes a topic identifier; and the method further includes:

[0226] According to the question identifier, statistics are collected on the usage of the question texts in the preset question bank;

[0227] The preset question bank is updated according to the question identifier and the usage situation.

[0228] According to one or more embodiments of the present disclosure, a visual question answering system is provided, comprising:

[0229] An acquisition module, configured to acquire the first title text in the first image;

[0230] A hash value determination module, configured to determine a first hash value of the first question text;

[0231] A retrieval module, configured to retrieve a second question text in a preset question bank according to the first hash value;

[0232] The answer module is used to determine a first answer text for the first question text according to the search results.

[0233] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0234] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0235] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A visual question answering method, characterized in that: include: Obtaining the first title text in the first image; Determine a first hash value of the first topic text; Retrieving a second question text from a preset question bank according to the first hash value; According to the search results, a first answer text of the first question text is determined.

2. The method according to claim 1, characterized in that The obtaining of the first title text in the first image includes: In response to receiving the first image, extracting and storing a third topic text in the first image; In response to identifying that the processing requirement corresponding to the first image belongs to a preset requirement, the first topic text is obtained from the stored third topic text.

3. The method according to claim 1, characterized in that Determining a first answer text to the first question text based on the search results includes: In response to retrieving the second question text, obtaining a second answer text corresponding to the second question text from the preset question bank; and determining the first answer text based on the second answer text; In response to not retrieving the second question text, the first answer text is generated according to the first question text by using a first generation model.

4. The method according to claim 3, characterized in that Generating the first answer text according to the first question text by using the first generation model includes: Obtaining a first feature vector of the first question text; Retrieving a fourth question text in the preset question bank according to the first feature vector; In response to retrieving the fourth question text, obtaining a third answer text corresponding to the fourth question text from the preset question bank; The first answer text is generated according to the first question text and the third answer text through the first generation model.

5. The method according to claim 1, wherein Also includes: Obtain a first label of the first topic text; The step of retrieving the second question text from a preset question bank according to the first Hash value includes: Perform a first-stage search in the preset question bank according to the first tag; The second title text is retrieved from the first-stage search results according to the first Hash value.

6. The method according to claim 1, characterized in that The updating process of the preset question bank includes: every first preset time period, determining a sixth question text based on the fifth question text and the candidate question bank; wherein the fifth question text includes the first question text obtained within the second preset time period; generating a fourth answer text according to the sixth question text by at least two second generation models; The preset question bank is updated according to the sixth question text and the fourth answer text.

7. The method according to claim 6, characterized in that Determining the sixth question text based on the fifth question text and the candidate question bank includes: Determine a second hash value of the fifth question text; Counting the occurrence frequency of the fifth question text according to the second hash value; Determining a seventh topic text from the fifth topic text according to the occurrence frequency; In response to the candidate question bank including the seventh question text, eliminating the seventh question text; The sixth question text is determined based on the seventh question text remaining after elimination.

8. The method according to claim 6, characterized in that The step of generating a fourth answer text based on the sixth question text by using at least two second generation models includes: Performing verification of the sixth question text according to a first preset dimension; In response to the verification being passed, a fourth answer text is generated according to the sixth question text by using at least two second generation models.

9. The method according to claim 6, characterized in that The updating of the preset question bank according to the sixth question text and the fourth answer text includes: Performing an evaluation of a second preset dimension on the fourth answer text using the referee model; wherein the second preset dimension includes individual evaluation and / or cross-validation; determining a fifth answer text from the fourth answer text according to the evaluation result; The preset question bank is updated according to the sixth question text and the fifth answer text.

10. The method according to claim 9, characterized in that The updating of the preset question bank according to the sixth question text and the fifth answer text includes: Obtaining topic information corresponding to the sixth topic text; wherein the topic information includes a third hash value; updating the candidate question bank according to the sixth question text, the fifth answer text, and the question information; The preset question bank is updated according to the updated candidate question bank.

11. The method according to claim 10, characterized in that The topic information also includes a topic identifier; the method further includes: According to the question identifier, statistics are collected on the usage of the question texts in the preset question bank; The preset question bank is updated according to the question identifier and the usage situation.

12. A visual question answering system, characterized in that: include: An acquisition module, configured to acquire the first title text in the first image; A hash value determination module, configured to determine a first hash value of the first question text; A retrieval module, configured to retrieve a second question text in a preset question bank according to the first hash value; The answer module is used to determine a first answer text for the first question text according to the search results.

13. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the visual question answering method as described in any one of claims 1-11.

14. A storage medium comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are used to perform the visual question answering method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the visual question answering method according to any one of claims 1 to 11.

Images