A visual question answering analysis method, device, system and readable storage medium

Abstract

The present invention discloses a visual question-answering analysis method, device, system and computer-readable storage medium. The method comprises: obtaining an image and question text information, and taking the image and question text information as input; judging whether there is OCR information in the image, and if so, using a character recognition network to perform character recognition on the image, and obtaining a plurality of character results and a confidence score corresponding to each character result; obtaining a corresponding related answer space and a corresponding classification probability value in combination with each character according to each character result, the corresponding confidence score and a pre-established answer space set; determining a target answer space from each related answer space according to each classification probability value; the present invention can improve the accuracy of the classification result and the accuracy of the visual question-answering result during use.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a visual question-answering analysis method, device, system and computer-readable storage medium. Background Art

[0002] Visual question answering means that given an image and a natural language question about the image, the computer can generate a correct answer. This field integrates the relevant technologies of CV and NLP. Computers need to learn to understand images and texts. It is an emerging research field in recent years. VizWiz is a project released by the University of Texas at Austin to answer visual questions for the blind. It is a data set collected by real blind users. It can truly reflect the real needs of users and many trivial problems in the implementation of theories in reality. The VizWiz data set collection process is for the blind to take a photo using the VizWiz software and record a verbal question about it. The question is uploaded to a crowdsourcing website and the answer is collected. In the past few years, a variety of VQA data sets have emerged in computer vision. They are all manually created and set up. This technology will bring great convenience to their lives, helping them improve their visual defects and break down social barriers.

[0003] The current mainstream technology is based on two attention mechanisms: top-down and bottom-up. The former means that people will be attracted by prominent objects in the visual sense, from the bottom-level information of the image to the upper-level semantics; while the latter means that when people are doing a task, they pay close attention to the part related to the task, from the upstream task to the image. The existing technology uses the Lxmert network (such as Figure 1 As shown in the figure, the input image and question text are analyzed to obtain a predicted answer space. However, since the answer space in the answer space set adopted by it does not take OCR (Optical Character Recognition) information into consideration, but usually for images containing OCR information, most of the answers are in the OCR information, the methods in the prior art will lead to inaccurate classification results, affecting the accuracy of the visual question answering results.

[0004] In view of this, how to provide a visual question answering analysis method, device, system and computer-readable storage medium that solve the above-mentioned technical problems has become a problem that needs to be solved by technical personnel in this field. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a visual question and answer analysis method, device, system and computer-readable storage medium, which can improve the accuracy of classification results and the accuracy of visual question and answer results during use.

[0006] To solve the above technical problems, an embodiment of the present invention provides a visual question answering analysis method, including:

[0007] Acquire an image and question text information, and use the image and question text information as input;

[0008] Determine whether there is OCR information in the image, and if so, use a character recognition network to perform character recognition on the image to obtain multiple character results and a confidence score corresponding to each character result;

[0009] According to each character result, the corresponding confidence score and the pre-established answer space set, obtain the corresponding answer space and the corresponding classification probability value in combination with each character;

[0010] According to each of the classification probability values, a target answer space is determined from each of the related answer spaces.

[0011] Optionally, the process of obtaining the relevant answer space corresponding to each character and the corresponding classification probability value according to each character result, the corresponding confidence score and the pre-established answer space set is:

[0012] For each of the character results, matching multiple related answer spaces corresponding to the character result and the similarity corresponding to each of the related answer spaces from a pre-established answer space set;

[0013] According to the confidence score of the character result and the similarity of each of the related answer spaces, the classification probability value of each of the related answer spaces is obtained to obtain the classification probability value of each related answer space corresponding to each of the character results.

[0014] Optionally, before determining whether OCR information exists in the image, the method further includes:

[0015] Determine whether the image is deflected, and if so, rotate the image to obtain a forward image;

[0016] Then, the method of determining whether there is OCR information in the image is:

[0017] Determine whether OCR information exists in the forward image.

[0018] Optionally, after obtaining the image and question text information, the method further includes:

[0019] Recognize the image to obtain corresponding object information and attribute information;

[0020] Integrating the object information and the attribute information with the question text information in a text form to obtain text information;

[0021] Then, the image and the question text information are taken as input:

[0022] The image and the text information are taken as input.

[0023] Optionally, the process of determining a target answer space from each of the related answer spaces according to each of the classification probability values ​​is:

[0024] The relevant answer space with the largest classification probability value is used as the target answer space.

[0025] Optionally, also include:

[0026] Using a pre-established Lxmert network to analyze the input image and the question text information, a plurality of predicted answer spaces and a classification probability value corresponding to each of the predicted answer spaces are obtained;

[0027] The process of determining the target answer space from each of the related answer spaces according to each of the classification probability values ​​is as follows:

[0028] The answer space with the largest classification probability value is selected from each of the relevant answer spaces and each of the predicted answer spaces as the target answer space.

[0029] The embodiment of the present invention further provides a visual question-answering analysis device, comprising:

[0030] An acquisition module, used to acquire image and question text information, and use the image and question text information as input;

[0031] A judgment module is used to judge whether there is OCR information in the image, and if so, trigger the recognition module;

[0032] The recognition module is used to perform character recognition on the image using a character recognition network to obtain multiple character results and a confidence score corresponding to each character result;

[0033] An analysis module, for obtaining, based on each character result, the corresponding confidence score and a pre-established answer space set, a corresponding relevant answer space and a corresponding classification probability value in combination with each character;

[0034] The screening module is used to determine a target answer space from each of the related answer spaces according to each of the classification probability values.

[0035] Optionally, the analysis module includes:

[0036] A matching unit, for matching, for each of the character results, a plurality of related answer spaces corresponding to the character result and a similarity corresponding to each of the related answer spaces from a pre-established answer space set;

[0037] A calculation unit is used to obtain the classification probability value of each related answer space according to the confidence score of the character result and the similarity of each related answer space, so as to obtain the classification probability value of each related answer space corresponding to each character result.

[0038] The embodiment of the present invention further provides a visual question answering analysis system, including:

[0039] Memory for storing computer programs;

[0040] A processor is used to implement the steps of the visual question answering analysis method as described above when executing the computer program.

[0041] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the visual question answering analysis method as described above are implemented.

[0042] A visual question-answering analysis method, device, system and computer-readable storage medium are provided in an embodiment of the present invention. The method includes: obtaining an image and question text information, and using the image and question text information as input; determining whether there is OCR information in the image, and if so, using a character recognition network to perform character recognition on the image to obtain multiple character results and a confidence score corresponding to each character result; obtaining a related answer space corresponding to each character and a corresponding classification probability value based on each character result, the corresponding confidence score and a pre-established answer space set; and determining a target answer space from each related answer space based on each classification probability value.

[0043] It can be seen that in the embodiment of the present invention, when performing visual question answering, the acquired image and question text information are used as input, and when OCR information is included in the image, a character recognition network is used to perform character recognition on the image to obtain multiple character recognition results and their corresponding confidence scores, and then, based on each character result, each corresponding confidence score and a pre-established answer space set, the relevant answer space corresponding to each character and the corresponding classification probability value are obtained, and based on each classification probability value, the final target answer space is determined from each relevant answer space; the present invention can improve the accuracy of the classification results and the accuracy of the visual question answering results during use. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the prior art and the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 It is a schematic diagram of the structure of the Lxmert network in the prior art;

[0046] Figure 2 A flowchart of a visual question answering analysis method provided by an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of the structure of a visual question-answering analysis device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] Embodiments of the present invention provide a visual question and answer analysis method, device, system, and computer-readable storage medium, which can improve the accuracy of classification results and the accuracy of visual question and answer results during use.

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Please refer to Figure 2 , Figure 2 A flowchart of a visual question answering analysis method provided by an embodiment of the present invention. The method includes:

[0051] S110: Acquire image and question text information, and use the image and question text information as input;

[0052] It should be noted that when the user needs to conduct visual question answering, he / she inputs the image and question text, and the system uses the acquired image and question text information as input and inputs it into the analysis model.

[0053] S120: Determine whether there is OCR information in the image, if yes, proceed to S130;

[0054] S130: Performing character recognition on the image using a character recognition network to obtain a plurality of character results and a confidence score corresponding to each character result;

[0055] Specifically, the analysis model can first determine whether there is OCR information in the image, and when there is OCR information in the image, a character recognition network (such as a STAGE network) can be used to recognize the characters in the image to obtain multiple character recognition results and a confidence score corresponding to each character recognition result. For example, for an image with Coca-Cola, the character results that may be output may include: "coco cola", "cola" or "coca cola", and a confidence score will be given for each of these character results.

[0056] S140: According to each character result, the corresponding confidence score and the pre-established answer space set, obtain the corresponding answer space and the corresponding classification probability value combined with each character;

[0057] It should be noted that in the embodiment of the present invention, an answer space set is pre-established, and then after each character result is obtained, specifically for each character result, multiple related answer spaces corresponding to the character result and the similarity corresponding to each related answer space can be matched from the pre-established answer space set, and then the classification probability value of each related answer space can be obtained based on the confidence score of the character result and the similarity of each related answer space, so as to obtain the classification probability value of each related answer space corresponding to each character result.

[0058] That is, for each character result, multiple related answer spaces may be matched from the answer space set, and a similarity is obtained for each related answer space, and then the confidence score of the character result is multiplied by the similarity of the corresponding related answer space to obtain the classification probability value of the related answer space, so that the classification probability value of each related answer space corresponding to each character result can be obtained. Among them, the similarity between the character result and the related answer space can be calculated according to S(pred,ans)=LCS(pred,ans) / (LD(pred,ans)+LCS(pred,ans)), where LCS(pred,ans) is the longest common subsequence between the character result and the string in the corresponding related answer space, LD(pred,ans) is the minimum number of edits required to change string A to string B, and S(pred,ans) is the similarity. Among them, when string A is a string in the character result, string B is a string in the related answer space; when string A is a string in the related answer space, string B is a string in the character result.

[0059] S150: Determine a target answer space from each relevant answer space according to each classification probability value.

[0060] Specifically, after obtaining the classification probability value of each relevant answer space, the target answer space is screened out from each relevant answer space according to each classification probability value, and the answer of the target answer space is output.

[0061] Furthermore, the process of determining the target answer space from each relevant answer space according to each classification probability value in the above S150 is:

[0062] The relevant answer space with the largest probability value of each classification is taken as the target answer space.

[0063] That is, the maximum classification probability value can be directly determined from the various classification probability values, and the relevant answer space corresponding to the maximum classification probability value is used as the target answer space.

[0064] Furthermore, in order to avoid the phenomenon that the final result is wrong due to the error in character recognition, the method may also include:

[0065] The pre-established Lxmert network is used to analyze the input image and question text information to obtain multiple predicted answer spaces and the classification probability values ​​corresponding to each predicted answer space;

[0066] Specifically, the Lxmert network is a two-stream BERT network (a classic network in the NLP field). The input of this network structure consists of two parts, image and text. The image features come from the features extracted by the target detection network. In this solution, each image generates 36 target detection frames, and the dimension of each target detection frame is 2048. Therefore, each image will generate a feature vector of 36*2048 dimensions. The text module uses the pre-trained weights of BERT. After the text module is embedded in the word, each sentence will generate a feature vector of N*768 dimensions. N represents the maximum sentence length. In this solution, N=40. After obtaining the image and text features respectively, each is input into the corresponding encoder network structure. The encoder network structure is a combination of the Transformer (the core network module of the BERT network) structure. After passing through the corresponding encoder network structure, the text and network modules are fused and input into the cross-modal network structure. Finally, the obtained feature vector is used for classification to obtain each predicted answer space and the classification probability value corresponding to each predicted answer space.

[0067] In addition, when it is determined that there is no OCR information in the image, traditional methods can be used for analysis. Specifically, the Lxmert network can be used to analyze the input image and question text information to obtain multiple predicted answer spaces and classification probability values ​​corresponding to each predicted answer space. Then, the predicted answer space with the largest classification probability value is selected from each predicted answer space as the final answer space.

[0068] Then, the process of determining the target answer space from each relevant answer space according to each classification probability value in the above S150 may specifically be:

[0069] The answer space with the largest classification probability value is selected from each relevant answer space and each predicted answer space as the target answer space.

[0070] That is, in the embodiment of the present invention, an answer space with the largest classification probability value is selected from each predicted answer space and each related answer space as the final target answer space, thereby improving the accuracy of the output structure.

[0071] Furthermore, before determining whether OCR information exists in the image in S120, the method may further include:

[0072] Determine whether the image is deflected. If so, rotate the image to obtain a positive image. Then, determine whether there is OCR information in the image:

[0073] Determine whether there is OCR information in the positive image.

[0074] That is, since the blind person takes images randomly and does not know whether the shooting angle is in the correct direction, in order to improve the recognition accuracy, the embodiment of the present invention can also first determine whether the image is deflected. When the image is not deflected, it can be directly determined whether there is OCR information in the image. When the image is deflected, the image can be rotated to make it positive, thereby obtaining a positive image, and then further determine whether there is OCR information in the positive image.

[0075] In practical applications, an image rotation recognition model can be established in advance, and then the image rotation recognition model can be used to identify whether the image is rotated, and when the rotation is sent, the image is adjusted to the positive direction. When establishing the image rotation recognition model, a certain amount of pictures can be selected from the training set (for example, 5,000 pictures), and each picture can be calibrated in different directions, such as positive, rotated 90°, rotated 180°, rotated 270°, etc., to obtain a new training set, and then model training is performed to obtain the image rotation recognition model.

[0076] Furthermore, after acquiring the image and question text information in the above S110, the method may further include:

[0077] Recognize the image and obtain the corresponding object information and attribute information;

[0078] Integrate the object information and attribute information with the question text information in the form of text to obtain text information;

[0079] Then, the image and question text information are taken as input:

[0080] Takes image and text information as input.

[0081] It should be noted that in order to further improve the recognition accuracy in the embodiment of the present invention, after obtaining the image and the question text information, the image can be recognized, and then the object information and attribute information in the image can be obtained. For example, the image can be analyzed by the Faster RCNN detection network to obtain the classification results of object classification and attribute classification. For example, the object obtained through image recognition analysis is a dog, and the attribute of the object is yellow. Then, the text information of "Yellow dogs" can be added to the front or back of the question text information "what is this" to form a new text information, and then the text information and the image are used as input and input into the analysis model.

[0082] It can be seen that in the embodiment of the present invention, when performing visual question answering, the acquired image and question text information are used as input, and when OCR information is included in the image, a character recognition network is used to perform character recognition on the image to obtain multiple character recognition results and their corresponding confidence scores, and then, based on each character result, each corresponding confidence score and a pre-established answer space set, the relevant answer space corresponding to each character and the corresponding classification probability value are obtained, and based on each classification probability value, the final target answer space is determined from each relevant answer space; the present invention can improve the accuracy of the classification results and the accuracy of the visual question answering results during use.

[0083] Based on the above embodiment, the present invention also provides a visual question answering analysis device. Figure 3 , the device comprises:

[0084] An acquisition module 21 is used to acquire image and question text information and use the image and question text information as input;

[0085] A judgment module 22 is used to judge whether there is OCR information in the image, and if so, trigger 23 a recognition module;

[0086] The recognition module 23 is used to perform character recognition on the image using a character recognition network to obtain multiple character results and a confidence score corresponding to each character result;

[0087] The analysis module 24 is used to obtain the relevant answer space corresponding to each character and the corresponding classification probability value according to each character result, the corresponding confidence score and the pre-established answer space set;

[0088] The screening module 25 is used to determine a target answer space from various related answer spaces according to various classification probability values.

[0089] Furthermore, the analysis module includes:

[0090] A matching unit, for matching, for each character result, a plurality of related answer spaces corresponding to the character result and a similarity corresponding to each related answer space from a pre-established answer space set;

[0091] The calculation unit is used to obtain the classification probability value of each relevant answer space according to the confidence score of the character result and the similarity of each relevant answer space, so as to obtain the classification probability value of each relevant answer space corresponding to each character result.

[0092] It should be noted that the visual question and answer analysis device provided in the embodiment of the present invention has the same beneficial effects as the visual question and answer analysis method provided in the above embodiment, and for the specific introduction of the visual question and answer analysis method involved in the embodiment of the present invention, please refer to the above embodiment, and the present invention will not go into details here.

[0093] Based on the above embodiment, the embodiment of the present invention further provides a visual question answering analysis system, which includes:

[0094] Memory for storing computer programs;

[0095] A processor is used to implement the steps of the above-mentioned visual question answering analysis method when executing a computer program.

[0096] For example, the processor in the embodiment of the present invention can be specifically used to obtain image and question text information, and use the image and question text information as input; determine whether there is OCR information in the image, and if so, use a character recognition network to perform character recognition on the image to obtain multiple character results and confidence scores corresponding to each character result; based on each character result, the corresponding confidence score and a pre-established answer space set, obtain the relevant answer space corresponding to each character and the corresponding classification probability value; based on each classification probability value, determine the target answer space from each relevant answer space.

[0097] Based on the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned visual question answering analysis method are implemented.

[0098] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0099] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0100] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0101] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A visual question answering analysis method, characterized in that: include: Acquire an image and question text information, and use the image and question text information as input; Determine whether there is OCR information in the image, and if so, use a character recognition network to perform character recognition on the image to obtain multiple character results and a confidence score corresponding to each character result; For each of the character results, multiple related answer spaces corresponding to the character result are matched from a pre-established answer space set, and the similarity between the character result and each of the related answer spaces is obtained, and the confidence score of the character result is multiplied by the similarity of the corresponding related answer space to obtain the classification probability value of the related answer space, so as to obtain the classification probability value of each related answer space corresponding to each of the character results; According to each of the classification probability values, a target answer space is determined from each of the related answer spaces; wherein: Obtaining the similarity between the character result and the corresponding related answer space includes: taking the longest common subsequence between the character result and the character string in the corresponding related answer space, dividing it by the minimum number of edits required to change character string A to character string B, and obtaining the corresponding quotient value; summing the longest common subsequence between the character result and the character string in the corresponding related answer space and the quotient value to obtain the similarity between the character result and the related answer space; wherein, when character string A is the character string in the character result, character string B is the character string in the related answer space; when character string A is the character string in the related answer space, character string B is the character string in the character result.

2. The visual question answering analysis method according to claim 1, characterized in that: Before determining whether OCR information exists in the image, the method further includes: Determine whether the image is deflected, and if so, rotate the image to obtain a forward image; Then, the method of determining whether there is OCR information in the image is: Determine whether OCR information exists in the forward image.

3. The visual question answering analysis method according to claim 1, characterized in that: After obtaining the image and question text information, the method further includes: Recognize the image to obtain corresponding object information and attribute information; Integrating the object information and the attribute information with the question text information in a text form to obtain text information; Then, the image and the question text information are taken as input: The image and the text information are taken as input.

4. The visual question answering analysis method according to any one of claims 1 to 3, characterized in that: The process of determining the target answer space from each of the related answer spaces according to each of the classification probability values ​​is as follows: The relevant answer space with the largest classification probability value is used as the target answer space.

5. The visual question answering analysis method according to any one of claims 1 to 3, characterized in that: Also includes: Using a pre-established Lxmert network to analyze the input image and the question text information, a plurality of predicted answer spaces and a classification probability value corresponding to each of the predicted answer spaces are obtained; The process of determining the target answer space from each of the related answer spaces according to each of the classification probability values ​​is as follows: The answer space with the largest classification probability value is selected from each of the relevant answer spaces and each of the predicted answer spaces as the target answer space.

6. A visual question-answering analysis device, characterized in that: include: An acquisition module, used to acquire image and question text information, and use the image and question text information as input; A judgment module is used to judge whether there is OCR information in the image, and if so, trigger the recognition module; The recognition module is used to perform character recognition on the image using a character recognition network to obtain multiple character results and a confidence score corresponding to each character result; An analysis module is used to match, for each of the character results, a plurality of related answer spaces corresponding to the character result from a pre-established answer space set, and obtain the similarity between the character result and each of the related answer spaces, and multiply the confidence score of the character result by the similarity of the corresponding related answer space to obtain the classification probability value of the related answer space, so as to obtain the classification probability value of each related answer space corresponding to each of the character results; wherein: Obtaining the similarity between the character result and the corresponding related answer space includes: taking the longest common subsequence between the character result and the character string in the corresponding related answer space, dividing by the minimum number of edits required to change character string A into character string B, and obtaining a corresponding quotient; summing the longest common subsequence between the character result and the character string in the corresponding related answer space and the quotient to obtain the similarity between the character result and the related answer space; wherein, when character string A is a character string in the character result, character string B is a character string in the related answer space; when character string A is a character string in the related answer space, character string B is a character string in the character result; The screening module is used to determine a target answer space from each of the related answer spaces according to each of the classification probability values.

7. A visual question answering analysis system, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the visual question answering analysis method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the visual question answering analysis method according to any one of claims 1 to 5 are implemented.

Images