Visual information extraction method and device, computer equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a visual information extraction method and device, computer equipment and a medium, and the method comprises the steps: firstly obtaining question text data and content text data in a target picture; secondly, performing image-text matching based on the target picture, the question text data and the content text data input, and generating a to-be-checked answer corresponding to the question text data; performing logic verification by using the to-be-checked answer, the target picture and the question text data to obtain candidate answers corresponding to the question text data; and finally, performing information correction on the candidate answers according to the content text data and the question text data to obtain target content data corresponding to the specified content item. Compared with a method which only depends on the target picture and the question text data, a more accurate to-be-checked answer can be generated; furthermore, by utilizing logic verification, the accuracy of the candidate answers is remarkably improved; and finally, through information correction, obtaining more accurate target content data, and generating more accurate answer text data.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and particularly to a method, device, computer equipment and medium for visual information extraction. Background Art

[0002] Visual Information Extraction (VIE) can be a process of analyzing and understanding multimodal data in visual rich documents and extracting specified text information. Visual rich documents widely exist in various commercial and administrative activities, such as invoices, identity documents, resumes, insurance documents, purchase orders, industry analysis reports, and business communications. Automated visual information extraction is of great significance for improving data processing efficiency, reducing labor costs, and enhancing the accuracy and reliability of information management.

[0003] However, in related technologies, the accuracy of visual information extraction needs to be improved. Therefore, there is an urgent need to propose a new method for visual information extraction. Summary of the Invention

[0004] This application provides a method, device, computer equipment and medium for visual information extraction, which solves the technical problem that the accuracy of visual information extraction in related technologies needs to be improved, and achieves the technical effect of more accurately extracting visual information.

[0005] To achieve the above object, the main technical solutions adopted in this application include:

[0006] In a first aspect, an embodiment of this application provides a method for visual information extraction, and the method includes:

[0007] Obtain problem text data and content text data in a target picture; wherein, the problem text data is determined for a specified content item in the target picture;

[0008] Based on the target picture, the problem text data, and the content text data, perform graphic-text matching to generate a to-be-verified answer corresponding to the problem text data; and use the to-be-verified answer, the target picture, and the problem text data to perform logical verification to obtain a candidate answer corresponding to the problem text data;

[0009] According to the content text data and the problem text data, correct the candidate answer to obtain target content data corresponding to the specified content item; wherein, the target content data is used to generate answer text data corresponding to the problem text data.

[0010] In a second aspect, an embodiment of this application provides a visual information extraction device, and the device includes:

[0011] A data acquisition module, configured to acquire problem text data and content text data in a target picture; wherein, the problem text data is determined for a specified content item in the target picture;

[0012] A candidate answer generation module, configured to perform image - text matching based on the input of the target picture, the problem text data, and the content text data to generate a to - be - checked answer corresponding to the problem text data; and perform logical verification using the to - be - checked answer, the target picture, and the problem text data to obtain a candidate answer corresponding to the problem text data;

[0013] An information correction module, configured to correct the information of the candidate answer according to the content text data and the problem text data to obtain target content data corresponding to the specified content item; wherein, the target content data is used to generate answer text data corresponding to the problem text data.

[0014] In a third aspect, an embodiment of the present application provides a computer device, including:

[0015] A memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to execute the method described in any of the above embodiments.

[0016] In a fourth aspect, an embodiment of the present application provides a computer - readable storage medium, on which computer instructions are stored, and the computer instructions are used to cause a computer to execute the method described in any of the above embodiments.

[0017] In the embodiments of the present application, first, problem text data and content text data in a target picture are acquired; secondly, image - text matching is performed based on the input of the target picture, the problem text data, and the content text data to generate a to - be - checked answer corresponding to the problem text data; and logical verification is performed using the to - be - checked answer, the target picture, and the problem text data to obtain a candidate answer corresponding to the problem text data; finally, the information of the candidate answer is corrected according to the content text data and the problem text data to obtain target content data corresponding to the specified content item. By combining the target picture, the problem text data, and the content text data to generate a to - be - checked answer, compared with the method that only relies on the target picture and the problem text data, it can utilize information resources more comprehensively, thereby generating a more accurate to - be - checked answer; further, logical verification is performed using the to - be - checked answer, the target picture, and the problem text data, effectively filtering inaccurate or unreasonable answers, significantly improving the accuracy of the candidate answer; finally, by combining the content text data and the problem text data to correct the information of the candidate answer, errors or deviations in the candidate answer are accurately corrected to obtain more accurate target content data, thereby generating more accurate answer text data. Brief Description of the Drawings

[0018] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0019] Figure 1 It is a flowchart of the visual information extraction method provided by the embodiments of this specification;

[0020] Figure 2 It is a flowchart of the visual information extraction method provided by the embodiments of this specification;

[0021] Figure 3 It is a flowchart of the visual information extraction method provided by the embodiments of this specification;

[0022] Figure 4 It is a flowchart of the visual information extraction method provided by the embodiments of this specification;

[0023] Figure 5 It is a schematic diagram of the model framework of the visual information extraction method provided by the embodiments of this specification;

[0024] Figure 6 It is a schematic diagram of the visual information extraction device provided by the embodiments of this specification;

[0025] Figure 7 It is a schematic diagram of a computer provided by the embodiments of this specification. Detailed Embodiments

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present application.

[0027] In related technologies, visual information extraction is based on deep learning OCR (Optical Character Recognition). First, it is necessary to detect text lines in the image and then perform text recognition. However, for many specific visual information extraction tasks, specialized processing often needs to be carried out for each task. For example, during the detection process, the detection model is trained to output specific detection boxes, and each task (such as invoices, enterprise research reports, etc.) requires a corresponding text detection model. If real-time inference is required, deploying these models simultaneously will lead to a significant increase in the cost of graphics cards. If a unified text detection model is used, corresponding parsing rules need to be developed for each task to extract the results of specific tasks, which requires a high development labor cost, and the rule generalization ability is poor, and often needs to be optimized one by one according to the tasks. In addition, this technology can only extract visual information and cannot perform question answering.

[0028] In some other related technologies, visual information extraction is based on multimodal large models. Only by inputting the image and the question into the multimodal large model can the answer be output. However, there are hallucination problems in multimodal large models, and there is a certain gap in recognition accuracy compared with OCR technology.

[0029] Based on this, the present application proposes a visual information extraction method, which includes: first, obtaining the question text data and the content text data in the target picture; secondly, based on the target picture, the question text data, and the content text data input, performing image-text matching to generate the answer to be checked corresponding to the question text data; and using the answer to be checked, the target picture, and the question text data for logical verification to obtain the candidate answer corresponding to the question text data; finally, correcting the candidate answer according to the content text data and the question text data to obtain the target content data corresponding to the specified content item. By combining the target picture, the question text data, and the content text data to generate the answer to be checked, compared with the method that only relies on the target picture and the question text data, it can utilize information resources more comprehensively, thereby generating a more accurate answer to be checked; further, using the answer to be checked, the target picture, and the question text data for logical verification effectively filters inaccurate or unreasonable answers, significantly improving the accuracy of the candidate answer; finally, by combining the content text data and the question text data to correct the candidate answer, the errors or deviations in the candidate answer are accurately corrected, and the more accurate target content data corresponding to the specified content item is obtained, thereby generating more accurate answer text data.

[0030] According to an embodiment of the present application, an embodiment of a visual information extraction method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0031] Please refer to Figure 1 , an embodiment of the present application provides a visual information extraction method, which includes:

[0032] S110. Obtain problem text data and content text data in a target picture.

[0033] Among them, the target picture can be a picture containing visual and text information, such as an invoice, a screenshot of a research report, etc.

[0034] The problem text data is determined for a specified content item in the target picture. For example, the problem text data can be "What is the name of the purchaser of this invoice?".

[0035] The content item can be a field representing specific information in the target picture, such as the name of the purchaser, the amount, the name of the seller, the invoice number, the invoicing date, the address of the purchaser, the phone number, etc. in the invoice. The specified content item can be a content item for which information or data needs to be extracted. For example, in the problem text data "What is the name of the purchaser of this invoice?", the specified content item is "name of the purchaser".

[0036] The content data can be the data corresponding to the content item. For example, for the name of the purchaser on the invoice, the corresponding content data is "xx Company".

[0037] The content text data can be the data composed of all content items in the target picture and their corresponding content data.

[0038] In some embodiments, the visual information extraction method is applied to the visual information extraction module of an enterprise information management platform. The enterprise information management platform can manage information such as invoices and research reports, support users in uploading target images such as invoice pictures and research report screenshots, and then extract the content data corresponding to important content items therein. After extraction, it can be displayed on the page of the enterprise information management platform and stored in the enterprise's database after being confirmed by the user. It can be understood that invoices, research reports, etc. have relatively standardized formats and content items. Therefore, for a certain type of target image, the enterprise information management platform can preset corresponding content items to be extracted. Exemplarily, the content items to be extracted from an invoice include the name of the purchaser, the name of the seller, the amount, and the invoicing date. When the user uploads an invoice picture, the enterprise information management platform can automatically generate the question text data "What are the name of the purchaser, the name of the seller, the amount, and the invoicing date of this invoice?", and then input the question text data into the visual information extraction module for visual information extraction. Exemplarily, the user can also manually extract a certain field from the target image as needed, such as extracting the seller's address. The enterprise information management platform can automatically generate the question text data "What is the seller's address of this invoice?". The enterprise information management platform also includes an OCR visual information extraction module based on deep learning, which can be used to accurately obtain the content text data in the target image.

[0039] S120. Perform graphic-text matching based on the target image, question text data, and content text data input, generate a to-be-checked answer corresponding to the question text data; and perform logical verification using the to-be-checked answer, target image, and question text data to obtain a candidate answer corresponding to the question text data.

[0040] Among them, graphic-text matching can be a process of generating a to-be-checked answer corresponding to the question text data based on the target image, question text data, and content text data input.

[0041] The to-be-checked answer can be a preliminary answer generated during the graphic-text matching process. Specifically, the to-be-checked answer is based on the preliminary matching result of the target image, question text data, and content text data, and its accuracy needs to be further verified.

[0042] Logical verification can be a process of using the to-be-checked answer, target image, and question text data for verification to determine whether the to-be-checked answer is correct.

[0043] The candidate answer can be an answer determined to be correct after logical verification.

[0044] In some embodiments, by analyzing the visual information features in the target image and the content text features of the content text data, and combining the semantics of the question text data, the answer to be checked related to the question text data is preliminarily determined. For example, when the question text data is "What is the name of the purchaser on this invoice?", the graphic-text matching process will combine the content text features, locate the visual area related to the "name of the purchaser" in the target image, and extract the text information in this area as the answer to be checked, generating the answer to be checked "xx Company".

[0045] Furthermore, logical verification will check whether the answer to be checked "xx Company" is consistent with the purchaser name field in the target image and whether it meets the semantic requirements of the question text data. If both checks pass, the answer to be checked is determined as the candidate answer.

[0046] S130. Correct the information of the candidate answer according to the content text data and the question text data to obtain the target content data corresponding to the specified content item.

[0047] Among them, information correction can be a process of correcting or confirming the candidate answer according to the content text data and the question text data. The target content data is used to generate the answer text data corresponding to the question text data.

[0048] The answer text data can be the final answer text generated according to the target content data. Specifically, the answer text data is the output result of the visual information extraction method, presenting the extraction result of the specified content item in text form. For example, the answer text data can be data in JSON format generated according to the question text data and the target content data, which is convenient for the computer to process this data.

[0049] In some embodiments, information correction adjusts and optimizes the candidate answer by comparing the candidate answer features with the content text features and the semantic requirements of the question text data to obtain the target content data corresponding to the specified content item. For example, if the candidate answer is "1008 yuan", but the target content data of the amount in the content text data is "10008 yuan", the information correction process will correct the candidate answer to "10008 yuan", and the generated answer text data in JSON format is:

[0050]

[0051] In the above embodiments, by combining the target picture, the question text data, and the content text data to generate the answer to be checked, compared with the method that only relies on the target picture and the question text data, it can make more comprehensive use of information resources, thereby generating a more accurate answer to be checked. Further, by performing logical verification on the answer to be checked, the target picture, and the question text data, inaccurate or unreasonable answers are effectively filtered, significantly improving the accuracy of the candidate answers. Finally, by combining the content text data and the question text data to correct the information of the candidate answers, errors or deviations in the candidate answers are accurately corrected, and more accurate target content data corresponding to the specified content items are obtained, thereby generating more accurate and reliable answer text data.

[0052] In some embodiments, the content text data in the target picture is obtained in the following manner, including: inputting the target picture into an optical character recognition model for information extraction to obtain the content text data in the target picture.

[0053] Among them, the optical character recognition model can be an algorithm model based on deep learning OCR technology that has been trained and is used to recognize and extract text information from images. Using this optical character recognition model, all the text data can be accurately extracted from the target picture.

[0054] In some embodiments, the target picture is input into the optical character recognition model for information extraction. Exemplarily, a screenshot of a research report is input into the optical character recognition model. The optical character recognition model outputs multiple text blocks in the form of text box coordinates plus text content by analyzing the pixel information and text layout features in the target picture. These text blocks contain all the text information in the research report screenshot and its position information in the picture. Further, in order to integrate these scattered text blocks into complete content text data, predefined rules can be adopted. For example, all the text blocks are concatenated into a long text string in the order of text boxes from left to right and from top to bottom as the content text data. This concatenation method can retain the reading order of the text in the original research report screenshot, thereby generating complete and ordered content text data.

[0055] In the above embodiments, inputting the target picture into the optical character recognition model for information extraction to obtain the content text data in the target picture can provide an accurate data basis for generating the answer to be checked, performing logical verification on the answer to be checked, and correcting the information of the candidate answers.

[0056] It should be emphasized that the optical character recognition model focuses on extracting text from images. Even in complex backgrounds, it can effectively recognize complex text and shows high accuracy in character recognition tasks. Therefore, this application uses the optical character recognition model to more precisely obtain the content text data in the target picture.

[0057] Furthermore, the present application performs image-text matching based on the target image, question text data, and content text data through a multimodal large model to generate an answer to be checked corresponding to the question text data. This process makes full use of the ability of the multimodal large model to understand the connection between images and texts. By analyzing the connection between the target image features and the content text features, the multimodal large model can infer the answer to be checked corresponding to the question text data. The precise content text data extracted by the optical character recognition model helps the multimodal large model to more accurately understand the connection between the target image features and the content text features, thereby generating a more precise answer to be checked.

[0058] Finally, based on the content text data and question text data, this application uses a large language model to correct the information of the candidate answers and obtain the target content data corresponding to the specified content item. Through the understanding, analysis and reasoning of the content text features, question text features and candidate answer text features, the large language model can identify and correct deviations or errors in the candidate answers. The accurate content text data extracted by the optical character recognition model helps the large language model better understand the relationship between the question text data and the content text data, so as to make accurate corrections to the candidate answers.

[0059] See also Figure 2 In some embodiments, image-text matching is performed based on the target image, question text data, and content text data input to generate an answer to be checked corresponding to the question text data; and logical verification is performed using the answer to be checked, the target image, and the question text data to obtain a candidate answer corresponding to the question text data, including:

[0060] S310: calling the target multimodal large model for the first time, inputting the target image, question text data, and content text data into the target multimodal large model for image-text matching, and obtaining the answer to be checked.

[0061] S320, calling the target multimodal large model for the second time, inputting the answer to be checked, the target image and the question text data into the target multimodal large model for logical verification, and obtaining the candidate answer.

[0062] In some embodiments, when the target multimodal large model is called for the first time, the target image, question text data, and content text data are input into the target multimodal large model. The target multimodal large model utilizes its powerful semantic understanding ability and cross-modal reasoning ability to generate an answer to be checked related to the question text data by analyzing the target image features in the target image and the content text features of the content text data, and combining the semantics of the question text data. For example, when the question text data is "What is the name of the seller on this invoice?", the image-text matching process will locate the visual area in the target image related to the "seller name", extract the text information related to the "seller name" from the target image, and generate the answer to be checked "xx Company".

[0063] In some embodiments, when the target multimodal large model is called for the second time, the answer to be checked, the target image, and the question text data are input into the target multimodal large model. The target multimodal large model uses its reflection ability to perform logical verification to improve the accuracy and reliability of the answer. By re-evaluating the generated answer and combining the original input information, logical verification is carried out. Exemplarily, by checking whether the answer to be checked "xx Company" is consistent with the buyer name field in the target image and whether it meets the semantic requirements of the question text data. If both checks pass, the answer to be checked is determined as a candidate answer. It can be understood that if the verification fails, the answer to be checked is determined as a wrong answer, and the answer to be checked can be generated again and verified again.

[0064] In the above embodiments, the answer to be checked is generated through image-text matching in the first call, and the answer to be checked is confirmed through logical verification in the second call. This phased processing method gives full play to the reasoning and reflection abilities of the multimodal large model, effectively reducing the errors and uncertainties that may occur in the single-stage processing using only reasoning.

[0065] It should be emphasized that in this embodiment, the same target multi-modal large model is called twice. When it is called for the first time, the target picture, question text data, and content text data are input into the target multi-modal large model to generate an answer to be checked related to the question text data. The goal of this stage is to utilize the multi-modal understanding ability of the model to quickly infer and generate a preliminary answer. At this time, the task of the model is to match the visual information (target picture) with the language information (question text data and content text data) to find an answer related to the question text data. When it is called for the second time, the answer to be checked generated in the first time, the target picture, and the question text data are input into the target multi-modal large model again for logical verification, and finally a candidate answer is obtained. The goal of this stage is to verify whether the preliminary answer generated in the first time is reasonable and accurate. By re-evaluating the logical relationship between the answer and the original input information, the model can further confirm the reliability of the answer. In the first call, the model generates an answer to be checked through picture-text matching, but the answer at this time may be affected by various factors. For example, the model's extraction of the features of the target picture is inaccurate, and the understanding of the question text data is biased, etc. Therefore, the generated answer may have a certain degree of uncertainty. The second call further confirms whether the answer is accurate through logical verification. This stage can effectively filter out the possible wrong answers generated in the first call, thereby improving the reliability of the final result.

[0066] This mechanism of calling the same target multi-modal large model in stages, firstly, can reduce the errors that may occur in a single inference stage of the model, thereby improving the accuracy of the final answer; secondly, it can make the model more robust when facing complex inputs. Even if the answer generated in the first call is biased, the verification process in the second call can still correct these errors, thereby enhancing the robustness of the model in complex scenarios.

[0067] In some embodiments, inputting the answer to be checked, the target picture, and the question text data into the target multi-modal large model for logical verification to obtain a candidate answer includes one of the following situations:

[0068] In the case where the answer to be checked is verified as a wrong answer, re-perform picture-text matching to generate a new answer to be checked; if it continues to be verified as a wrong answer and the number of wrong times reaches a preset threshold, determine the answer to be checked that appears the most times as the candidate answer.

[0069] In the case where the answer to be checked is verified as a correct answer, determine the answer to be checked as the candidate answer.

[0070] In some embodiments, the target multimodal large model is called a second time, and the answer to be checked, the target picture, and the question text data are input into the target multimodal large model. The target multimodal large model uses its reflection ability to perform logical verification, improve the accuracy and reliability of the answer, re-evaluate the generated answer, and combine the original input information for logical verification. Exemplarily, by checking whether the answer to be checked "XX Company" is consistent with the purchaser name field in the target picture and whether it meets the semantic requirements of the question text data, if both checks pass, the answer to be checked is determined as a candidate answer. Exemplarily, by checking whether the answer to be checked "XX Company" is consistent with the purchaser name field in the target picture and whether it meets the semantic requirements of the question text data, if either check fails, the answer to be checked is determined as a wrong answer. During this process, there is a preset threshold for the number of errors. In the case where the answer to be checked is verified as a wrong answer, the picture-text matching is performed again to generate a new answer to be checked; if it continues to be verified as a wrong answer and the number of errors reaches the preset threshold, the answer to be checked that appears the most times is determined as the candidate answer.

[0071] In the above embodiments, through logical verification and optimization of the error handling mechanism, the accuracy and reliability of the candidate answer are significantly improved, and the adaptive ability and robustness of the visual information extraction method are enhanced.

[0072] In some embodiments, the candidate answer is corrected for information according to the content text data and the question text data to obtain the target content data corresponding to the specified content item, including: calling the large language model, and inputting the candidate answer, the content text data, and the question text data into the large language model to correct the information of the candidate answer to obtain the target content data.

[0073] Specifically, for relatively dense or long texts, the candidate answers output by the target multimodal large model may have missing or misspelled characters. This is because when the multimodal large model processes complex texts, it may be affected by the hallucination problem, resulting in inaccurate generated answers. In contrast, the optical character recognition model has higher accuracy in text recognition and can more accurately identify the text content in the target picture. Therefore, by jointly inputting the candidate answer output by the target multimodal large model, the content text data obtained by information extraction using the optical character recognition model, and the question text data into the large language model, the problem of missing or misspelled characters caused by hallucinations in the target multimodal large model can be further eliminated by leveraging the language understanding and generation capabilities of the large language model. In this way, the information correction of the candidate answer can be achieved, and thus more accurate target content data can be obtained.

[0074] In some embodiments, the scale of the large language model is larger than the scale of the language part in the target multimodal large model, and the large language model has stronger format compliance than the target multimodal large model. Exemplarily, when the target multimodal large model is required to output answer text data in JSON format, due to its weak format compliance, there may be fewer parentheses or there may be additional words such as "The answer is". By using the large language model for information correction, correct-format answer text data can be obtained.

[0075] In the above embodiments, through the collaborative work of the target multimodal large model, the optical character recognition model, and the large language model, the hallucination problem that may occur when the target multimodal large model processes complex text is effectively reduced, and the accuracy of the extracted target content data is significantly improved, providing a more reliable solution for complex multimodal information extraction tasks.

[0076] Please refer to Figure 3 , in some embodiments, the target multimodal large model is obtained in the following manner:

[0077] S610. Construct a training sample set.

[0078] S620. Use the training sample set to perform hybrid training on the initial multimodal large model to fine-tune the initial multimodal large model and obtain the target multimodal large model.

[0079] Among them, the training sample set is generated by mixing based on the first sample and the second sample; the first sample is generated based on the sample picture, the sample question text, and the sample content text corresponding to the sample picture; the second sample is generated based on the sample picture, the sample question text, and the sample answer text corresponding to the sample picture.

[0080] In some embodiments, the target multimodal large model has two uses: the first is that when inputting the target picture, the question text data, and the content text data, it can generate the answer to be checked corresponding to the question text data; the second is that when inputting the answer to be checked, the target picture, and the question text data for logical verification, it determines whether the answer to be checked is correct.

[0081] In some embodiments, the training tasks of the initial multimodal large model correspond to the uses of the target multimodal large model: the first training task is to use the first sample to train the reasoning ability of the initial multimodal large model, that is, the ability to generate answer text based on the sample picture, the sample question text, and the sample content text corresponding to the sample picture; the second training task is to use the second sample to train the reflection ability of the initial multimodal large model, that is, the ability to determine whether the answer to be checked is correct based on the sample picture, the sample question text, and the sample answer text corresponding to the sample picture.

[0082] In some embodiments, the sample images are the images used for information extraction common in actual financial scenarios, such as invoices, screenshots of research reports, etc., and the number of samples between categories is balanced by sampling. Part of the sample question texts can be generated by annotators based on the sample images, and another part can be generated by using a multimodal large model for the sample images. For example, for the same sample image, the annotator generates 5 sample question texts based on the elements in the sample image, and the image is input into the multimodal large model. The multimodal large model generates 3 more sample question texts based on the prompts and its understanding of the sample image. The sample content texts can be obtained by using an optical character recognition model to recognize the sample images. The correct answer texts corresponding to the sample question texts are written by annotators. The sample images, sample question texts, and sample content texts corresponding to the sample images constitute the first sample. The correct answer texts are the label data of the first sample.

[0083] In some embodiments, the generation methods of the sample images, sample question texts, and sample content texts are the same as those in the previous embodiment. The sample answer texts corresponding to the sample images are written by annotators. The sample answer texts can be correct answers or wrong answers; among them, the judgment result corresponding to the correct answer is correct, and the judgment result corresponding to the wrong answer is wrong. The sample images, sample question texts, and sample answer texts corresponding to the sample images constitute the second sample. The judgment result is the label data of the second sample.

[0084] In some embodiments, the first training task and the second training task are carried out in a mixed manner. Specifically, the first sample and the second sample are randomly mixed, 70% of the mixed samples are taken to form a training sample set, 20% of the mixed samples are taken to form a validation sample set, and the remaining 10% of the mixed samples are taken to form a test sample set. Then, the training sample set is used to perform mixed training on the initial multimodal large model, and the initial multimodal large model can be the InternVL2-8B pre-trained model. The fine-tuning process can be to adopt the Adam optimizer, set the learning rate to 5e-6, the batch size to 128. After each epoch, the validation sample set is used to evaluate the performance of the intermediate multimodal large model (such as accuracy, loss value, etc.). According to the results of the validation sample set, it is decided whether to stop fine-tuning early or adjust the hyperparameters. Finally, the test sample set is used to test the intermediate multimodal large model, and the complete matching ratio or complete inclusion ratio of the output result of the intermediate multimodal large model and the label data can be used as the judgment criterion. After passing the test, the intermediate multimodal large model is determined to be the target multimodal large model.

[0085] It should be emphasized that if the inference ability of the initial multi-modal large model is first fine-tuned using the first sample and then its logical judgment ability is fine-tuned using the second sample, it will cause the target multi-modal large model to forget the inference task, thereby reducing the inference ability; conversely, if its logical judgment ability is fine-tuned first and then its inference ability is fine-tuned, it will lead to the target multi-modal large model forgetting the logical judgment task and reducing the logical judgment ability. Therefore, in this application, the initial multi-modal large model is mixedly trained by randomly mixing the first sample and the second sample. Through this method, the target multi-modal large model can simultaneously obtain strong inference ability and logical judgment ability, effectively avoiding the forgetting of the previous task caused by sequential fine-tuning, thereby ensuring that the comprehensive performance of the model is not affected.

[0086] In the above embodiment, the initial multi-modal large model is mixedly trained using the training sample set constructed by mixing the first sample and the second sample to obtain the target multi-modal large model, which can enable the target multi-modal large model to simultaneously obtain strong inference ability and logical judgment ability, and helps to reduce the hallucination of the target multi-modal large model and generate more accurate candidate answers.

[0087] Please refer to Figure 4 , in some embodiments, the target multi-modal large model includes a visual part, a language part, and an adaptation part connecting the visual part and the language part; the initial multi-modal large model is fine-tuned in the following manner:

[0088] S710. Freeze the parameters of the visual part and the parameters of the optical character recognition model.

[0089] S720. Fine-tune the parameters of the language part and the parameters of the adaptation part.

[0090] Among them, the optical character recognition model is used to extract text data from the target picture to obtain the content text data corresponding to the target picture.

[0091] The visual part of the target multi-modal large model is used to extract features from the target picture to obtain the initial target picture features, the language part is used to extract features from the content text data, the question text data, and the answer to be checked, respectively obtaining the content text features, the question text features, and the answer to be checked text features. The adaptation part is used to transform the initial target picture features into target picture features that the language part can understand. In addition, the language part is also used to obtain the answer to be checked according to the understood target picture features, content text features, and question text features, and to judge whether the answer to be checked is correct according to the understood target picture features, question text features, and answer to be checked text features.

[0092] Freezing can be to keep certain parameters unchanged during the fine-tuning process to avoid these parameters being updated during the fine-tuning process.

[0093] In some embodiments, in order to preserve the feature representations learned by the visual part and the optical character recognition model during the fine-tuning process and prevent performance degradation caused by fine-tuning, the parameters of the visual part of the initial multi-modal large model and the parameters of the optical character recognition model are frozen, and only the parameters of the language part and the adaptation part are fine-tuned. Fine-tuning the parameters of the adaptation part can optimize the fusion and alignment effects between the target image features and the content text features. Fine-tuning the parameters of the language part, on the one hand, can optimize its understanding of the target image features, content text features, and question text features, so as to generate more accurate answers to be checked; on the other hand, it can optimize its understanding of the target image features, question text features, and the text features of the answer to be checked, so as to make a more accurate judgment on whether the answer to be checked is correct.

[0094] In the above embodiments, by freezing the parameters of the visual part and the parameters of the optical character recognition model, and at the same time fine-tuning the parameters of the language part and the adaptation part, the stability of visual feature extraction and text extraction can be preserved during the fine-tuning process, and performance degradation caused by fine-tuning can be avoided. At the same time, fine-tuning the parameters of the language part and the adaptation part can optimize the model's processing ability for visual information extraction tasks. This strategy not only makes full use of the advantages of the pre-trained model, but also enhances the adaptability of the multi-modal large model to specific tasks through targeted fine-tuning.

[0095] Please refer to Figure 5 , in this embodiment, the model framework for visual information extraction includes: an optical character recognition model 804, a target multi-modal large model 806, and a large language model 805. Among them, the target multi-modal large model 806 includes a visual part 801, an adaptation part 802, and a language part 803. The target multi-modal large model 806 can handle visual information extraction tasks for different types of target images; in this embodiment, the target multi-modal large model 806 is first called, and based on the target image and the content text data accurately recognized and extracted by the optical character recognition model 804, an answer to be checked is inferred and generated. Compared with relying only on the target image, the errors caused by hallucination of the target multi-modal large model 806 can be significantly reduced; then the target multi-modal large model 806 is called again to logically verify the answer to be checked using its reflection ability, further reducing hallucination errors; finally, the large language model 805 is introduced to correct the candidate answer based on the content text data and question text data accurately recognized and extracted by the above optical character recognition model 804, so as to more effectively reduce the impact of hallucination errors. Through this multi-level collaborative verification and information fusion method, the hallucination problem of the multi-modal large model in the visual information extraction task is significantly reduced, and the accuracy and reliability of the answer text data are effectively improved.

[0096] In some embodiments, before the target multimodal large model is called for the first time and the target picture, question text data, and content text data are input into the target multimodal large model for picture-text matching to obtain the answer to be checked, it further includes: The target multimodal large model is constructed based on the initial multimodal large model by selecting corresponding model parameters to be adjusted according to the complexity of the target picture.

[0097] Among them, the complexity of the target picture can include three levels: low, medium, and high. The initial multimodal large model is adjusted according to the model parameters to be adjusted to obtain the target multimodal large model. Corresponding to the complexity level of the target picture, the model parameters to be adjusted are low-complexity model parameters, medium-complexity model parameters, and high-complexity model parameters.

[0098] In some embodiments, a picture complexity classification model is pre-trained. The target picture is input into the picture complexity classification model for complexity level classification to obtain the complexity of the target picture. Then, based on the initial multimodal large model, corresponding model parameters to be adjusted are selected according to the complexity of the target picture to construct the target multimodal large model. Exemplarily, when the target picture is input into the picture complexity classification model and the complexity of the target picture is obtained as high, based on the initial multimodal large model, high-complexity model parameters are selected to construct the target multimodal large model, which can extract visual information with high accuracy for high-complexity target pictures.

[0099] It should be noted that a training dataset can be pre-constructed. The data samples in the training dataset use pictures with different complexities, and the labels used for the data samples are low, medium, and high picture complexity data. The initial picture complexity classification model is trained using the training dataset until the model training stop condition is met to obtain the picture complexity classification model.

[0100] In some embodiments, pictures of different types (such as invoices, research report screenshots, natural background pictures) can be classified according to text density, layout complexity, color contrast, etc. An example table is shown in Table 1. For text density, 0 indicates low density, and 1 indicates high density; for layout complexity, 0 indicates low complexity, and 1 indicates high complexity; for color contrast, 0 indicates high contrast, and 1 indicates low contrast.

[0101] Table 1 Example Table of Picture Complexity Classification

[0102] Text density Layout complexity Color contrast Complexity 0 0 0 Low 1 0 0 Medium 0 1 0 Medium 1 1 0 Medium 1 0 1 Medium 0 1 1 Medium 1 1 1 High

[0103] It should be emphasized that based on the initial multi-modal large model, this embodiment dynamically selects appropriate model parameters to be adjusted according to the complexity of the target picture, so as to construct a more targeted target multi-modal large model. This method can adaptively adjust the model parameters according to the complexity of the target picture to better adapt to target pictures with different complexities. Compared with the strategy of using the same model to process target pictures of all complexities, this method not only improves the visual information extraction efficiency of low-complexity pictures, takes into account the visual information extraction efficiency and accuracy of medium-complexity pictures, but also enhances the visual information extraction accuracy of high-complexity pictures, thus optimizing the performance of the multi-modal large model in different application scenarios.

[0104] In some embodiments, constructing the training sample set includes: constructing a low-complexity training sample set, a medium-complexity training sample set, and a high-complexity training sample set.

[0105] Among them, the low-complexity training sample set is generated by mixing the low-complexity first sample and the low-complexity second sample; the low-complexity first sample is generated based on the sample low-complexity picture, the sample question text, and the sample content text corresponding to the sample picture; the low-complexity second sample is generated based on the sample low-complexity picture, the sample question text, and the sample answer text corresponding to the sample picture.

[0106] The medium-complexity training sample set is generated by mixing the medium-complexity first sample and the medium-complexity second sample; the medium-complexity first sample is generated based on the sample medium-complexity picture, the sample question text, and the sample content text corresponding to the sample picture; the medium-complexity second sample is generated based on the sample medium-complexity picture, the sample question text, and the sample answer text corresponding to the sample picture.

[0107] The high-complexity training sample set is generated by mixing the high-complexity first sample and the high-complexity second sample; the high-complexity first sample is generated based on the sample high-complexity picture, the sample question text, and the sample content text corresponding to the sample picture; the high-complexity second sample is generated based on the sample high-complexity picture, the sample question text, and the sample answer text corresponding to the sample picture.

[0108] In some embodiments, the target multi-modal large model includes a visual part, a language part, and an adaptation part connecting the visual part and the language part; fine-tuning the initial multi-modal large model includes:

[0109] Based on the low-complexity training sample set, freeze the parameters of the visual part, the language part, and the optical character recognition model, and only fine-tune the parameters of the adaptation part to obtain low-complexity model parameters;

[0110] Based on the medium-complexity training sample set, freeze the parameters of the visual part and the optical character recognition model, and fine-tune the parameters of the language part and the adaptation part to obtain medium-complexity model parameters;

[0111] Based on the high-complexity training sample set, freeze the parameters of the optical character recognition model, and fine-tune the parameters of the visual part, the language part, and the adaptation part to obtain high-complexity model parameters.

[0112] In some embodiments, when training the initial large model using the low-complexity training sample set, freeze the parameters of the visual part, the language part, and the optical character recognition model, and only fine-tune the parameters of the adaptation part. When the fine-tuning is completed, record and save the parameter changes of the adaptation part to obtain low-complexity model parameters; when training the initial large model using the medium-complexity training sample set, freeze the parameters of the visual part and the optical character recognition model, and fine-tune the parameters of the language part and the adaptation part. When the fine-tuning is completed, record and save the parameter changes of the adaptation part and the language part to obtain medium-complexity model parameters; when training the initial large model using the high-complexity training sample set, freeze the parameters of the optical character recognition model, and fine-tune the parameters of the visual part, the language part, and the adaptation part. When the fine-tuning is completed, record and save the parameter changes of the adaptation part, the language part, and the visual part to obtain high-complexity model parameters.

[0113] It should be noted that low-complexity pictures usually have simple structures, clear contents, and simple backgrounds. Since the pre-trained initial multi-modal large model can very effectively understand the pictures and texts in this training dataset, there is no need to change the basic structures of the visual part and the language part. The parameters of the visual part, the language part, and the optical character recognition model of the initial multi-modal large model can be frozen, and only the adaptation part is fine-tuned to help the model specifically adjust for the visual information extraction task of low-complexity pictures.

[0114] Medium-complexity pictures usually have one or two factors among the structure, content, or background that are relatively complex. Since the pre-trained initial multi-modal large model can relatively effectively understand such pictures, there is no need to change the basic structure of the visual part. The parameters of the visual part and the optical character recognition model of the initial multi-modal large model can be frozen, and only the parameters of the language part and the adaptation part are fine-tuned. Fine-tuning the parameters of the adaptation part helps to optimize the fusion and alignment effect between the sample picture features and the sample content text features. Fine-tuning the parameters of the language part can not only improve its understanding of the target picture features, content text features, and question text features, so as to generate more accurate answers to be checked; but also enhance its understanding of the relationship between the target picture features, question text features, and the text features of the answer to be checked, so as to make a more accurate judgment on whether the answer to be checked is correct.

[0115] High-complexity images usually have complex structures, content, and backgrounds. Pre-trained initial multi-modal large models may have difficulty effectively understanding such images. Therefore, only the parameters of the optical character recognition model are frozen, while the parameters of the visual part, language part, and adaptation part are all fine-tuned to improve the model's ability to extract and understand the features of high-complexity images, optimize the fusion and alignment effect between the sample image features and the sample content text features, thereby generating more accurate answers to be checked, and making more precise inferences in logical judgments.

[0116] In this embodiment, for training data sets of different complexities, different parts of the initial multi-modal large model are frozen for fine-tuning, so as to obtain model parameters adapted to target images of different complexities, laying a foundation for adaptively selecting model parameters according to the complexity of the target image during the visual information extraction process and constructing the target multi-modal large model. It should be emphasized that by using this strategy for fine-tuning, first, it can avoid overfitting of the model to low-complexity target images while maintaining its high processing ability; second, it can take into account the overfitting problem of the model to medium-complexity target images and improve the model's ability to understand images and texts, enhancing the accuracy of the model's visual information extraction; finally, it can enhance the model's ability to understand complex scenes, strengthen its feature extraction and logical reasoning abilities, and thus more effectively process complex tasks.

[0117] Please refer to Figure 6 , the embodiment of the present application also provides a visual information extraction device 800, and the visual information extraction device 800 includes:

[0118] A data acquisition module 810, configured to acquire the problem text data and the content text data in the target image; wherein, the problem text data is determined for a specified content item in the target image;

[0119] A candidate answer generation module 820, configured to perform image-text matching based on the input of the target image, the problem text data, and the content text data, generate an answer to be checked corresponding to the problem text data; and perform logical verification using the answer to be checked, the target image, and the problem text data to obtain a candidate answer corresponding to the problem text data;

[0120] An information correction module 830, configured to correct the candidate answer according to the content text data and the problem text data to obtain target content data corresponding to the specified content item; wherein, the target content data is used to generate answer text data corresponding to the problem text data.

[0121] In some embodiments, the data acquisition module 810 further includes:

[0122] An information extraction unit, configured to input the target image into an optical character recognition model for information extraction to obtain the content text data in the target image.

[0123] In some embodiments, the candidate answer generation module 820 further includes:

[0124] A to-be-checked answer acquisition unit, configured to call the target multi-modal large model for the first time, input the target picture, question text data, and content text data into the target multi-modal large model for picture-text matching, and obtain the to-be-checked answer;

[0125] A candidate answer acquisition unit, configured to call the target multi-modal large model for the second time, input the to-be-checked answer, target picture, and question text data into the target multi-modal large model for logical verification, and obtain the candidate answer.

[0126] In some embodiments, the candidate answer generation module 820 further includes:

[0127] A candidate answer determination unit, configured to, in the case where the to-be-checked answer is verified as a wrong answer, re-perform picture-text matching to generate a new to-be-checked answer; if it is continuously verified as a wrong answer and the number of wrong times reaches a preset threshold, determine the to-be-checked answer with the most occurrences as the candidate answer; in the case where the to-be-checked answer is verified as a correct answer, determine the to-be-checked answer as the candidate answer.

[0128] In some embodiments, the information correction module 830 further includes:

[0129] A large language model call unit, configured to call the large language model, input the candidate answer, content text data, and question text data into the large language model to correct the information of the candidate answer and obtain the target content data.

[0130] In some embodiments, the visual information extraction device 800 further includes:

[0131] A training sample set construction module, configured to construct a training sample set, where the training sample set is generated by mixing a first sample and a second sample; the first sample is generated based on a sample picture, sample question text, and sample content text corresponding to the sample picture; the second sample is generated based on a sample picture, sample question text, and sample answer text corresponding to the sample picture;

[0132] A mixed training module, configured to perform mixed training on the initial multi-modal large model by using the training sample set to fine-tune the initial multi-modal large model and obtain the target multi-modal large model.

[0133] In some embodiments, the target multi-modal large model includes a visual part, a language part, and an adaptation part connecting the visual part and the language part; the visual information extraction device 800 further includes:

[0134] A parameter freezing module for freezing the parameters of the vision part and the parameters of the optical character recognition model;

[0135] A fine-tuning module for fine-tuning the parameters of the language part and the parameters of the adaptation part.

[0136] The further function descriptions of the above-mentioned various modules and units are the same as those in the corresponding above-mentioned embodiments, and will not be elaborated here.

[0137] The visual information extraction device in this embodiment is presented in the form of functional units. Here, the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and a memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0138] Please refer to Figure 7 , Figure 7 which is a schematic structural diagram of a computer device provided by an embodiment of the present application. As Figure 7 shown, the computer device includes: one or more processors 10, a memory 20, and an interface for connecting each component, including a high-speed interface and a low-speed interface. Each component communicates with each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in the memory or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some alternative embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a set of blade servers, or a multi-processor system). Figure 7 In

[0139] Processor 10 can be a central processor, a network processor, or a combination thereof. Among them, processor 10 can further include a hardware chip. The above-mentioned hardware chip can be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The above-mentioned programmable logic device can be a complex programmable logic device, a field programmable gate array, a general array logic, or any combination thereof.

[0140] Among them, the memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above-mentioned embodiment.

[0141] The memory 20 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the computer device, etc. In addition, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some alternative embodiments, the memory 20 may optionally include a memory remotely provided with respect to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0142] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, a hard disk, or a solid-state drive; the memory 20 may further include a combination of the above types of memory.

[0143] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30, and the output device 20 may be connected through a bus or other means. Figure 7 Taking connection through a bus as an example.

[0144] The input device 30 can receive input digital or character information and generate key signal inputs related to the user settings and function controls of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (such as an LED), and a haptic feedback device (such as a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light-emitting diode, a display, and a plasma display. In some alternative embodiments, the display device may be a touch screen.

[0145] The embodiments of the present application also provide a computer-readable storage medium. The method according to the embodiments of the present application can be implemented in hardware, firmware, or be implemented as computer code that can be recorded on a storage medium, or be implemented as computer code that is originally stored in a remote storage medium or a non-transitory machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be processed by such software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by the computer, the processor, or the hardware, the method shown in the above embodiments is implemented.

[0146] The embodiments of the present application provide a computer program product, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of any embodiment of the present application.

[0147] Although the embodiments of the present application are described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations all fall within the scope defined by the appended claims.

[0148] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0149] For example, when receiving the user's active request, a prompt message is sent to the user to clearly prompt the user that the operation requested to be executed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application program, a server, or a storage medium that executes the operation of the technical solution of the present disclosure according to the prompt message.

[0150] As an optional but non-limiting implementation manner, the manner of sending a prompt message to the user in response to receiving the user's active request can be, for example, in the form of a pop-up window, and the prompt message can be presented in text in the pop-up window. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0151] It is understandable that the above notification and the process of obtaining user authorization are only illustrative and do not limit the implementation manner of the present disclosure. Other manners that comply with relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0152] It is understandable that the data involved in the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the corresponding laws, regulations and related provisions.

[0153] It is understandable that in the specific implementation manner of the present application, data related to user information, location information, navigation data, etc. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0154] The systems, devices, modules or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0155] For the convenience of description, when describing the above device, it is divided into various units according to functions for description. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0156] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0157] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or a device for implementing the functions specified in one or more of the blocks.

[0158] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or a device for implementing the functions specified in one or more of the blocks.

[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or a device for implementing the functions specified in one or more of the blocks.

[0160] It should also be noted that the term "comprising", "including", or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, commodity, or device including a series of elements not only includes those elements but also includes other elements not explicitly listed, or elements inherent to such process, method, commodity, or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, commodity, or device including the said element.

[0161] Each embodiment in this specification is described in a progressive manner. The same or similar parts among the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, it is described relatively simply, and the relevant parts can be referred to the description of the method embodiment.

[0162] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

[0163] Although the embodiments of the present application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A visual information extraction method, characterized in that: The method comprises: Acquire question text data and content text data in the target image; wherein the question text data is determined for a specified content item in the target image; Perform image-text matching based on the target image, the question text data, and the content text data input to generate an answer to be checked corresponding to the question text data; and perform logical verification using the answer to be checked, the target image, and the question text data to obtain a candidate answer corresponding to the question text data; The candidate answers are corrected according to the content text data and the question text data to obtain target content data corresponding to the designated content item; wherein the target content data is used to generate answer text data corresponding to the question text data.

2. The method according to claim 1, characterized in that The content text data in the target image is obtained by the following method, including: The target image is input into an optical character recognition model to extract information, and the content text data in the target image is obtained.

3. The method according to claim 2, characterized in that The image-text matching is performed based on the target image, the question text data, and the content text data input to generate an answer to be checked corresponding to the question text data; And using the answer to be checked, the target image and the question text data to perform logic verification to obtain a candidate answer corresponding to the question text data, including: The target multimodal large model is called for the first time, and the target image, the question text data, and the content text data are input into the target multimodal large model for image-text matching to obtain the answer to be checked; The target multimodal large model is called for the second time, and the answer to be checked, the target image and the question text data are input into the target multimodal large model for logical verification to obtain the candidate answer.

4. The method according to claim 3, characterized in that The step of inputting the answer to be checked, the target image and the question text data into the target multimodal large model for logic verification to obtain the candidate answer includes one of the following situations: In the case where the answer to be checked is verified as an incorrect answer, re-matching of the image and text is performed to generate a new answer to be checked; if it is continuously verified as an incorrect answer and the number of errors reaches a preset threshold, the answer to be checked with the largest number of occurrences is determined as the candidate answer; In the case where the answer to be checked is verified to be the correct answer, the answer to be checked is determined as the candidate answer.

5. The method according to claim 3, characterized in that: The step of correcting the candidate answer information according to the content text data and the question text data to obtain target content data corresponding to the designated content item includes: The large language model is called, and the candidate answers, the content text data and the question text data are input into the large language model to correct the information of the candidate answers to obtain the target content data.

6. The method according to claim 3, characterized in that The target multimodal large model is obtained by: Constructing a training sample set, wherein the training sample set is generated by mixing a first sample and a second sample; the first sample is generated based on a sample image, a sample question text, and a sample content text corresponding to the sample image; the second sample is generated based on the sample image, the sample question text, and a sample answer text corresponding to the sample image; The initial multimodal large model is mixedly trained using the training sample set to fine-tune the initial multimodal large model to obtain the target multimodal large model.

7. The method according to claim 6, characterized in that The target multimodal large model includes a visual part, a language part, and an adaptation part connecting the visual part and the language part; The initial multimodal large model is fine-tuned by: freezing parameters of the vision component and parameters of the optical character recognition model; The parameters of the language part and the parameters of the adaptation part are fine-tuned.

8. A visual information extraction device, characterized in that: The device comprises: A data acquisition module, used to acquire question text data and content text data in a target image; wherein the question text data is determined for a specified content item in the target image; A candidate answer generation module is used to perform image-text matching based on the target image, the question text data, and the content text data input to generate an answer to be checked corresponding to the question text data; and to perform logical verification using the answer to be checked, the target image, and the question text data to obtain a candidate answer corresponding to the question text data; An information correction module is used to perform information correction on the candidate answers according to the content text data and the question text data to obtain target content data corresponding to the specified content item; wherein the target content data is used to generate answer text data corresponding to the question text data.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images