Document identification method and device, computing equipment and storage medium

Abstract

The invention provides a document recognition method and device, computing equipment and a storage medium, and the method comprises the steps: inputting a to-be-recognized document into an optical character recognition (OCR) engine, and obtaining a first recognition result output by the OCR engine; respectively inputting the to-be-recognized document and the recognition prompt words of the different types of data into a plurality of different multi-modal large language models to obtain a plurality of second recognition results output by the plurality of different multi-modal large language models; aiming at the text data, aligning the first recognition result with the plurality of second recognition results to obtain a first output result; for the graph data, obtaining a second output result based on the plurality of second recognition results; and obtaining an identification result of the to-be-identified document based on the first output result and the second output result. According to the scheme, OCR identification can be accurately and effectively carried out on the document.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a document recognition method, apparatus, computing device and storage medium. Background Art

[0002] With the development of science and technology, optical character recognition (OCR) technology has developed rapidly; however, traditional OCR engines have the following problems:

[0003] First, it can only handle a single problem. For example, the traditional OCR engine can only be used to recognize Chinese, or can only be used to recognize English; it cannot recognize both Chinese and English at the same time. Second, it cannot interpret graphic data. For example, the traditional OCR engine cannot interpret the information contained in the statistical chart, nor can it correctly identify whether there is a QR code.

[0004] A complete document often contains multiple types of data, so how to accurately and effectively perform OCR recognition on the document remains to be solved. Summary of the invention

[0005] The present application provides a document recognition method, apparatus, computing device and storage medium, which can realize accurate and effective OCR recognition of documents.

[0006] In the first aspect, an embodiment of the present application provides a document recognition method, which can be executed by a document recognition device, which can be a terminal device or a module for a terminal device, or a server or a module for a server. The present application does not limit the execution subject of the method. The method includes: inputting the document to be recognized into an optical character recognition (OCR) engine to obtain a first recognition result output by the OCR engine; inputting the document to be recognized and recognition prompt words of different types of data into multiple different multimodal large language models respectively, to obtain multiple second recognition results output by the multiple different multimodal large language models; for text data, aligning the first recognition result and the multiple second recognition results to obtain a first output result; for image data, based on the multiple second recognition results, obtaining a second output result; based on the first output result and the second output result, obtaining the recognition result of the document to be recognized.

[0007] The above scheme, on the one hand, combines the OCR engine and the multimodal large language model, which can solve the single problem of the traditional OCR engine and also realize the interpretation of image data; on the other hand, the document to be recognized is divided into text data and image data, and the recognition results of the OCR engine and multiple multimodal large language models are aligned for the text data and the image data respectively, so that the recognition result of the document to be recognized is more reliable, so that the recognition result of the document to be recognized can be obtained accurately and effectively, thereby improving the accuracy of OCR recognition.

[0008] In a possible implementation method, the document to be recognized is divided into blocks according to different types of data to obtain blocks of text data and blocks of image data; the blocks of text data are input into an optical character recognition (OCR) engine; the blocks of text data and corresponding recognition prompt words, the blocks of image data and corresponding recognition prompt words are respectively input into multiple different multimodal large language models.

[0009] The above scheme, on the one hand, inputs different types of data into different recognition engines, which can obtain accurate and effective recognition results for different types of data; on the other hand, setting different recognition prompt words for different types of data can improve the recognition accuracy of the multimodal large language model, thereby improving the accuracy of the recognition results of the documents to be recognized.

[0010] In a possible implementation method, the first recognition result includes a sub-recognition result for each row; weight information corresponding to the OCR engine and the multiple different multimodal large language models is obtained respectively; for any sub-recognition result in the first recognition result, at least one target recognition result that meets the matching requirements with the sub-recognition result is determined from the multiple second recognition results; based on the weight information corresponding to the OCR engine and the multiple different multimodal large language models, the sub-recognition results and the at least one target recognition result, the weighted weights corresponding to the same recognition results are counted; the recognition result corresponding to the maximum weighted weight sum is used as the first output item; and the first output result is obtained based on each first output item.

[0011] In the above scheme, the first output item is determined by the recognition results of multiple engines, so the confidence of the first output item is relatively reliable and can be directly used by downstream tasks. Therefore, the first output result can be obtained accurately and effectively, thereby improving the accuracy of OCR recognition.

[0012] In one possible implementation method, if at least one target recognition result that meets the matching requirements with the sub-recognition result is not determined, the sub-graph corresponding to the position indication information in the sub-recognition result and the corresponding recognition prompt word are input into the multiple different multimodal large language models to obtain multiple third recognition results; based on the multiple third recognition results, at least one target recognition result that meets the matching requirements with the sub-recognition result is re-determined.

[0013] The above scheme can improve the accuracy of identifying documents to be identified by re-identifying those that do not meet the matching requirements.

[0014] In a possible implementation method, if multiple consecutive rows of sub-recognition results are all in at least one second recognition result, the multiple consecutive rows of sub-recognition results are merged into one sub-recognition result.

[0015] The above solution fully utilizes the semantic analysis integrity advantage of the multimodal large language model engine and solves the pain point of cross-row cell recognition in the traditional OCR engine.

[0016] In one possible implementation method, if the maximum weighted weight sum is greater than a first weight threshold, the recognition result with the largest weighted weight sum is used as the first output item; if the maximum weighted weight sum is not greater than the first weight threshold, the sub-graph corresponding to the position indication information in the sub-recognition result is obtained; a new sub-graph is generated based on the information corresponding to the sub-graph and any second recognition result; and the recognition result corresponding to the new sub-graph whose similarity with the sub-graph is greater than the pixel threshold is used as the first output item.

[0017] In the above scheme, if the maximum weighted sum is not greater than the first weight threshold, the first output item is determined using the pixel similarity between the sub-image and the new sub-image. Since the credibility of the pixel-level matching is high, the first output item can be obtained accurately and effectively.

[0018] In a possible implementation method, the graph data is a statistical graph; any second recognition result is a key-value pair result set;

[0019] Calculate the weighted sum of the same key-value pair result sets; and use the key-value pair result set corresponding to the maximum weighted sum as the second output result.

[0020] In the above scheme, the second output result is determined according to the recognition results of multiple large language models, and the confidence of the second output result is relatively credible, so the second output result can be accurately and effectively determined, thereby improving the accuracy of OCR recognition.

[0021] In one possible implementation method, the key-value pair whose maximum weighted weight sum is greater than a second weight threshold is used as the second output item; the key-value pair whose maximum weighted weight sum is not greater than the second weight threshold and the subgraph where the statistical graph is located are input into at least one multimodal large language model, and the key-value pair belonging to the subgraph where the statistical graph is located is determined as the second output item; and the second output result is obtained based on each second output item.

[0022] In the above scheme, if the maximum weighted sum is not greater than the second weight threshold, the semantic analysis capability of the multimodal large language model is used to determine whether the identified key-value pair is located on the statistical graph, so as to accurately and effectively determine the second output result.

[0023] In a second aspect, an embodiment of the present application provides a document recognition device, comprising: a recognition unit and an output unit. The recognition unit is used to input a document to be recognized into an optical character recognition (OCR) engine to obtain a first recognition result output by the OCR engine; input the document to be recognized and recognition prompt words of different types of data into multiple different multimodal large language models respectively to obtain multiple second recognition results output by the multiple different multimodal large language models; the output unit is used to align the first recognition result with the multiple second recognition results for text data to obtain a first output result; for image data, based on the multiple second recognition results, obtain a second output result; based on the first output result and the second output result, obtain a recognition result of the document to be recognized.

[0024] In a possible implementation method, the recognition unit is used to divide the document to be recognized into blocks according to different types of data to obtain blocks of text data and blocks of image data; input the blocks of text data into an optical character recognition (OCR) engine; and input the blocks of text data and corresponding recognition prompt words, and the blocks of image data and corresponding recognition prompt words into multiple different multimodal large language models, respectively.

[0025] In a possible implementation method, the first recognition result includes a sub-recognition result for each row; the output unit is used to respectively obtain weight information corresponding to the OCR engine and the multiple different multimodal large language models; for any sub-recognition result in the first recognition result, determine at least one target recognition result that meets the matching requirements with the sub-recognition result from the multiple second recognition results; based on the weight information corresponding to the OCR engine and the multiple different multimodal large language models, the sub-recognition results and the at least one target recognition result, count the weighted weights corresponding to the same recognition results; use the recognition result corresponding to the maximum weighted weight sum as the first output item; and obtain the first output result based on each first output item.

[0026] In one possible implementation method, the recognition unit is used to input the sub-graph corresponding to the position indication information in the sub-recognition result and the corresponding recognition prompt word into the multiple different multimodal large language models to obtain multiple third recognition results if at least one target recognition result that meets the matching requirements with the sub-recognition result is not determined; and based on the multiple third recognition results, re-determine at least one target recognition result that meets the matching requirements with the sub-recognition result.

[0027] In a possible implementation method, the output unit is configured to merge the sub-recognition results in a plurality of consecutive rows into one sub-recognition result if the sub-recognition results in a plurality of consecutive rows are all included in at least one second recognition result.

[0028] In one possible implementation method, the output unit is used to use the recognition result with the largest weighted weight sum as the first output item if the maximum weighted weight sum is greater than the first weight threshold; if the maximum weighted weight sum is not greater than the first weight threshold, obtain the sub-graph corresponding to the position indication information in the sub-recognition result; generate a new sub-graph according to the information corresponding to the sub-graph and any second recognition result; and use the recognition result corresponding to the new sub-graph whose similarity with the sub-graph is greater than the pixel threshold as the first output item.

[0029] In a possible implementation method, the output unit is used for the graph data being a statistical graph; any second recognition result is a key-value pair result set; the weighted sum of the same key-value pair result set is calculated; and the key-value pair result set with the maximum weighted sum is used as the second output result.

[0030] In a possible implementation method, the output unit is used to use the key-value pair whose maximum weighted weight sum is greater than a second weight threshold as the second output item; input the key-value pair whose maximum weighted weight sum is not greater than the second weight threshold and the subgraph where the statistical graph is located into at least one multimodal large language model, determine the key-value pair belonging to the subgraph where the statistical graph is located and use it as the second output item; and obtain the second output result based on each second output item.

[0031] In a third aspect, an embodiment of the present application further provides a computing device, including:

[0032] A memory for storing program instructions;

[0033] The processor is used to call the program instructions stored in the memory, and execute any method of implementing the above-mentioned first aspect according to the obtained program instructions.

[0034] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which computer-readable instructions are stored. When a computer reads and executes the computer-readable instructions, any method of the above-mentioned first aspect is implemented.

[0035] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program executable by a computer device, wherein when the program is run on the computer device, the computer device executes any method for implementing the above-mentioned first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of a document recognition method provided in an embodiment of the present application;

[0037] Figure 2 A flowchart of a document recognition method provided in an embodiment of the present application;

[0038] Figure 3 A flowchart of a document recognition method provided in an embodiment of the present application;

[0039] Figure 4 A flowchart of a document recognition method provided in an embodiment of the present application;

[0040] Figure 5 A schematic diagram of the structure of a document recognition device provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of the structure of a document recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] Figure 1 A flowchart of a document recognition method provided in an embodiment of the present application, the method can be executed by a document recognition device, the document recognition device can be a terminal device or a module for a terminal device, or a server or a module for a server. The present application does not limit the execution subject of the method.

[0043] The method comprises the following steps:

[0044] Step 101: input a document to be recognized into an optical character recognition (OCR) engine to obtain a first recognition result output by the OCR engine.

[0045] In one possible implementation method, the OCR engine is an OCR engine based on deep learning and / or an OCR engine based on statistical machine learning, which is hereinafter referred to as a traditional OCR engine.

[0046] In a possible implementation method, the document to be recognized is input into at least one OCR engine, and a first recognition result output by each OCR engine is obtained. The present application does not limit the number of OCR engines, which can be one or more. Multiple OCR engines can solve the singularity problem.

[0047] Step 102: input the document to be recognized and recognition prompt words of different types of data into a plurality of different multimodal large language models respectively, and obtain a plurality of second recognition results output by the plurality of different multimodal large language models.

[0048] In a possible implementation method, the multimodal large language model is a multimodal large language model with image processing capabilities. The multimodal large language model can be used to process both image data and text data.

[0049] Optionally, the present application does not limit the number of multimodal large language models, and the document to be recognized can be input into one or more multimodal large language models.

[0050] Optionally, a multimodal large language model such as open source models such as GLM-4v-9b, Qwen-VL, or an external model with a closed-source open API interface.

[0051] Step 103: for text data, align the first recognition result and the plurality of second recognition results to obtain a first output result.

[0052] Step 104, for the graph data, based on the multiple second recognition results, obtain a second output result.

[0053] Step 105: Obtain a recognition result of the document to be recognized based on the first output result and the second output result.

[0054] The above scheme, on the one hand, combines the OCR engine and the multimodal large language model, which can solve the single problem of the traditional OCR engine and also realize the interpretation of image data; on the other hand, the document to be recognized is divided into text data and image data, and the recognition results of the OCR engine and multiple multimodal large language models are aligned for the text data and the image data respectively, so that the recognition result of the document to be recognized is more reliable, so that the recognition result of the document to be recognized can be obtained accurately and effectively, thereby improving the accuracy of OCR recognition.

[0055] In a possible implementation method, before the above step 101, before inputting the document to be recognized into the optical character recognition (OCR) engine, it also includes: dividing the document to be recognized into blocks according to different types of data to obtain blocks of text data and blocks of image data. Among them, text data includes plain text, tables, etc.; image data includes QR codes, statistical charts, etc. This application does not limit the specific forms of text data and image data.

[0056] Optionally, before the above step 101, each block is also preprocessed, and the preprocessing includes but is not limited to denoising, resolution enhancement, etc., which is not limited in the present application.

[0057] Optionally, an image edge detection algorithm or a deep learning algorithm is used to divide the document to be identified into blocks, so that each block intersects only at the edge, and all blocks can be spliced ​​into the document to be identified.

[0058] Optionally, each block is classified to determine the blocks of text data and the blocks of image data corresponding to the document to be identified. The present application does not limit the algorithm of the classification process.

[0059] In a possible implementation method, the blocks of the text data are input into an optical character recognition (OCR) engine to obtain a first recognition result output by the OCR engine. The blocks of the text data and the corresponding recognition prompt words, the blocks of the image data and the corresponding recognition prompt words are respectively input into the multiple different multimodal large language models to obtain multiple second recognition results output by the multiple different multimodal large language models.

[0060] Optionally, the output results of multiple different multimodal large language models are processed into a unified format for output. If it is text data, the information of the cell corresponding to each row is obtained: <row number, column number, text content>; if it is graph data, it is subdivided, and for statistical graphs, the response results of multiple multimodal large language models are extracted; if it is a QR code, yes / no is output.

[0061] Optionally, different recognition prompt words are set for different types of data.

[0062] For example, the recognition prompt word corresponding to the text data is "Give you a picture, please identify the content of the picture and convert it into markdown text"; then you can get structured markdown text data, and then you can use the markdown parser to parse the block recognition result: <coordinate axis, text content>. It should be noted here that the multimodal large language model may not be able to give the OCR recognition score and coordinate axis.

[0063] The recognition prompt word corresponding to the statistical chart data is "Please identify the image and output structured information. All content should be output in json format. If it cannot be parsed, output {}. Please do not lose the image information. A similar format is as follows: {"Annual sales volume of two-wheeled electric vehicles in 2018": 32.2 million units, "Annual sales volume of two-wheeled electric vehicles in 2019": 36.8 million units}".

[0064] The recognition prompt word corresponding to the QR code data is "Please identify whether the picture has a QR code. You only need to answer whether there is a QR code. If there is, output a word YES, otherwise output a word NO."

[0065] The above scheme, on the one hand, inputs different types of data into different recognition engines, which can obtain accurate and effective recognition results for different types of data; on the other hand, setting different recognition prompt words for different types of data can improve the recognition accuracy of the multimodal large language model, thereby improving the accuracy of the recognition results of the documents to be recognized.

[0066] In a possible implementation method, the first recognition result includes a sub-recognition result for each row; the above step 103 aligns the first recognition result with the plurality of second recognition results to obtain a first output result, such as Figure 2 As shown, the following steps are included:

[0067] Step 201, respectively obtain weight information corresponding to the OCR engine and the multiple different multimodal large language models.

[0068] In a possible implementation method, weight information corresponding to the traditional OCR engine and each multimodal large language model is set respectively. The present application does not limit the weight values ​​corresponding to the traditional OCR engine and each multimodal large language model. Exemplarily, the weight values ​​corresponding to the traditional OCR engine and each multimodal large language model are the same.

[0069] Optionally, the sum of the weights corresponding to the traditional OCR engine and each multimodal large language model is 1.

[0070] Optionally, the weight information corresponding to the traditional OCR engine and each multimodal large language model can be assigned through manual experience, or learned using machine learning methods, or obtained through statistical methods.

[0071] Step 202: for any sub-recognition result in the first recognition result, determine at least one target recognition result that meets a matching requirement with the sub-recognition result from the plurality of second recognition results.

[0072] In one possible implementation method, the traditional OCR engine recognizes the row content, and when a cell consists of multiple rows, it will be recognized in multiple blocks; while the multimodal large language model has a strong ability to recognize complete semantic blocks, and at this time it is necessary to merge the regions and then align them. Therefore, before the above step 201, it also includes: if the sub-recognition results of multiple consecutive rows are all in at least one second recognition result, then the sub-recognition results of the multiple consecutive rows are merged as one sub-recognition result. This solution makes full use of the semantic analysis integrity advantage of the multimodal large language model engine and solves the pain point of cross-row recognition of cells in the traditional OCR engine.

[0073] Alternatively, if the result of the traditional OCR engine is relatively short, and the output of most large models is relatively long, it is determined that cross-row content may occur, and region merging is required.

[0074] Optionally, a layout analysis method is used to determine whether the recognition results of the traditional OCR engine need to be merged. The layout analysis method includes one or more of the following: edge detection, color texture detection, and deep learning technology. Exemplarily, if edge detection finds that there are multiple rows of data in the same cell, the first recognition results corresponding to the multiple rows of data in the cell are merged into one row.

[0075] In a possible implementation method, the region merging algorithm includes one or more of the following: merging based on distance (less than a specific range), based on clustering algorithm, based on feature similarity, based on edge, based on semantic analysis method, etc.

[0076] In a possible implementation method, for the merged area, since there are multiple OCR recognition scores and multiple text results, the text results can be combined, and the recognition scores can be replaced by the mean or the maximum value. At the same time, the merged position needs to be updated to obtain a new result: <block position, OCR recognized text content, OCR recognition score>.

[0077] Step 203: According to the weight information corresponding to the OCR engine and the multiple different multimodal large language models, the sub-recognition results and the at least one target recognition result, a weighted sum corresponding to the same recognition result is calculated.

[0078] The first recognition result includes a sub-recognition result for each row.

[0079] In a possible implementation method, based on the sub-recognition result and the at least one target recognition result, weighted weights corresponding to engines with the same recognition results are summed, and the recognition result corresponding to the maximum weighted weight is selected as the first output item.

[0080] Step 204: taking the maximum weighted value and the corresponding recognition result as the first output item.

[0081] Step 205: Obtain the first output result based on each first output item.

[0082] Optionally, the output result format of the first output item is <first output item, weighted confidence, traditional engine recognition score>, wherein the weighted confidence is the maximum weighted sum.

[0083] In one possible implementation method, for the weighted confidence and the traditional engine recognition score, a rule engine can be used or a fusion scoring model can be learned to output the final confidence score.

[0084] In the above scheme, the first output item is determined by the recognition results of multiple engines, so the confidence of the first output item is relatively reliable and can be directly used by downstream tasks, so the first output result can be obtained accurately and effectively.

[0085] In a possible implementation method, in the above step 201, determining at least one target recognition result that meets the matching requirement with the sub-recognition result from the multiple second recognition results includes: if the matching degree between any second recognition result and the sub-recognition result is greater than or equal to a first threshold, then determining that the second recognition result meets the matching requirement with the sub-recognition result. This application does not limit the value of the first threshold.

[0086] Optionally, the traditional OCR engine result is matched with the recognition result of each multimodal large language model respectively, wherein the matching method includes one or more of the following: string fuzzy matching algorithm, string matching algorithm such as edit distance, method based on word or character statistics, etc.

[0087] In a possible implementation method, if the degree of match between any second recognition result and the sub-recognition result is greater than or equal to a first threshold, the position indication information corresponding to the sub-recognition result is added to the second recognition result. Because the recognition result of the traditional OCR engine includes position information, but the multimodal large language model is better at semantic parsing and understanding rather than extracting image position blocks, it is necessary to map the recognition result of the multimodal large language model to the recognition result of the traditional OCR engine.

[0088] In a possible implementation method, for each recognition result of the traditional OCR engine, find out the one that has not been matched in the corresponding large language model recognition result in order, find the one with the smallest error within a tolerable error threshold for matching, and mark the recognition result of the large language model as matched. For the matched recognition result, add the position indication information corresponding to the traditional OCR engine recognition result to the matched recognition result, and for the unmatched recognition result, re-recognition and other operations can be performed.

[0089] Optionally, if some recognition results of the traditional OCR engine are not matched, this is allowed and can be marked as unsuccessful matching.

[0090] In a possible implementation method, in the above step 201, if at least one target recognition result that meets the matching requirements with the sub-recognition result is not determined, the sub-graph corresponding to the position indication information in the sub-recognition result and the corresponding recognition prompt word are input into the multiple different multimodal large language models to obtain multiple third recognition results; based on the multiple third recognition results, at least one target recognition result that meets the matching requirements with the sub-recognition result is re-determined.

[0091] Optionally, in addition to not determining at least one target recognition result that meets the matching requirements with the sub-recognition result, it also includes: the confidence level given by the traditional OCR engine is relatively low, or if you want to further improve the accuracy of the recognition result, you can input the sub-image and the corresponding recognition prompt word into the multiple different multimodal large language models to obtain multiple third recognition results.

[0092] Optionally, according to the corresponding position indication information in the sub-recognition result, the document to be recognized is cut to obtain the sub-image, and the sub-image is preprocessed; the preprocessing includes one or more of the following: denoising and super-resolution enhancement. The reason for enhancing the resolution is mainly that the recognition effect is not good, and there is a certain possibility that the original image has noise or resolution problems.

[0093] In a possible implementation method, since the multimodal large language model has semantic analysis capabilities, it will automatically combine or split the text in the sub-graph; at this time, it is necessary to design recognition prompt words to inform the multimodal large language model that the sub-graph is a complete content and does not need to be cut. The sub-graph and the corresponding recognition prompt words are input into the multiple different multimodal large language models, including: designing recognition prompt words, and inputting the sub-graph and the designed recognition prompt words into the multiple different multimodal large language models.

[0094] In a possible implementation method, in the above step 203, if the maximum weighted weight sum is greater than the first weight threshold, the recognition result with the largest weighted weight sum is used as the first output item; if the maximum weighted weight sum is not greater than the first weight threshold, the sub-graph corresponding to the position indication information in the sub-recognition result is obtained; a new sub-graph is generated based on the information corresponding to the sub-graph and any second recognition result; and the recognition result corresponding to the new sub-graph whose similarity with the sub-graph is greater than the pixel threshold is used as the first output item.

[0095] Optionally, in addition to the case where the maximum weighted sum is not greater than the first weight threshold, it also includes one or more of the following: the recognition score of the traditional OCR engine is low, the sub-recognition result does not match the output results of the major language models, and the number of matches between the sub-recognition result and the output results of the major language models is insufficient.

[0096] In one possible implementation method, if the maximum weighted sum is not greater than the first weight threshold, then obtaining a subgraph corresponding to the position indication information in the sub-recognition result, including: if the maximum weighted sum is not greater than the first weight threshold, then cutting the document to be recognized according to the corresponding position indication information in the sub-recognition result to obtain the subgraph.

[0097] In one possible implementation method, the font style, size, font content and other information corresponding to the sub-graph are obtained, and a new sub-graph is generated through any multimodal large language model according to the information corresponding to the sub-graph and any second recognition result. Of course, in addition to the multimodal large language model, a new sub-graph can also be generated through a traditional neural network model, and this application does not limit the method for generating a new sub-graph.

[0098] Specifically, according to the corresponding position indication information in the sub-recognition result, the document to be recognized is cut to obtain the sub-image, and then the image algorithm is used to obtain the font style and size corresponding to the sub-image. Then, the technology of generating pictures using text recognized by each engine is used to generate a new image, and the similarity is calculated based on the new image and the sub-image. If the similarity score is high, because the pixel-level matching credibility is high, it can be considered that the recognition result is likely to be correct, and the voting result is set as the second recognition result of the corresponding engine. At the same time, the confidence is weighted to a relatively large confidence, such as setting the confidence to 90%.

[0099] In a possible implementation method, if the maximum weighted sum is not greater than the first weight threshold, then obtain the subgraph corresponding to the position indication information in the sub-recognition result, input the subgraph into other multimodal large language models, and obtain a new recognition result; compare the new recognition result with the old recognition result, and if the new recognition result meets the matching requirements with any old recognition result, use the new recognition result as the first output item. The old recognition result includes sub-recognition results and / or multiple second recognition results. This application does not limit the first quantity threshold.

[0100] In a possible implementation method, the graph data is a statistical graph; then the step of determining the second output result is as follows: Figure 3 As shown, the following steps are included:

[0101] Step 301: Calculate the weighted sum of the same key-value pair result sets based on the multiple second recognition results.

[0102] Among them, any second recognition result is a key-value pair result set.

[0103] For graph data, since traditional OCR engines cannot recognize it, there are only recognition results of multiple multimodal large language models. Therefore, it is only necessary to determine the second output result based on the recognition results of multiple multimodal large language models. Among them, the recognition result of each multimodal large language model is a key-value pair result set. A similar format is as follows: [{"Annual sales volume of two-wheeled electric vehicles in 2018": 32.2 million units, "Annual sales volume of two-wheeled electric vehicles in 2019": 36.8 million units}].

[0104] Step 302: taking the maximum weighted weight and the corresponding key-value pair result set as the second output result.

[0105] In one possible implementation method, the key-value pair corresponding to the maximum weighted weight sum greater than the second weight threshold is used as the second output item; the key-value pair whose maximum weighted weight sum is not greater than the second weight threshold and the subgraph where the statistical graph is located are input into at least one multimodal large language model to determine the key-value pair belonging to the subgraph where the statistical graph is located and use it as the second output item; the second output result is obtained based on each second output item. The present application does not limit the second quantity threshold. In this solution, if the maximum weighted weight sum is not greater than the second weight threshold, the semantic analysis capability of the multimodal large language model is used to determine whether the identified key-value pair is located on the statistical graph, so as to accurately and effectively determine the second output result.

[0106] In a possible implementation method, the key-value pairs whose maximum weighted weight sum is not greater than the second weight threshold and the subgraph where the statistical graph is located are input into at least one multimodal large language model, and the key-value pairs belonging to the subgraph where the statistical graph is located are determined as the second output items, including: determining a first result set and a second result set according to the key-value pair result set corresponding to each second recognition result; the first result set is used to store the same key-value pairs in the key-value pair result set corresponding to each second recognition result; the second result set is used to store key-value pairs that are different from the first result set in the key-value pair result set corresponding to any second recognition result; the second result set and the subgraph are input into at least one multimodal large language model, and the at least one multimodal large language model is used to analyze whether any key-value pair in the second result set is located in the subgraph where the statistical graph is located; if any key-value pair in the second result set is located in the subgraph where the statistical graph is located, the key-value pair is stored in a third result set; and the first result set and the third result set are used as the second output results.

[0107] Exemplarily, the second recognition result of the first multimodal large language model is [{“Annual sales volume of two-wheeled electric vehicles in 2018”: 32.2 million units, “Annual sales volume of two-wheeled electric vehicles in 2019”: 36.8 million units}], and the second recognition result of the second multimodal large language model is [{“Annual sales volume of two-wheeled electric vehicles in 2018”: 32.2 million units, “Annual sales volume of two-wheeled electric vehicles in 2020”: 39.6 million units}]; then the first result set is [{“Annual sales volume of two-wheeled electric vehicles in 2018”: 32.2 million units}]; the second result set is [“Annual sales volume of two-wheeled electric vehicles in 2019”: 36.8 million units}, “Annual sales volume of two-wheeled electric vehicles in 2020”: 39.6 million units}]; the statistical chart of the second result set is input into the multimodal large language model, and the multimodal large language model is used to analyze “{“Annual sales volume of two-wheeled electric vehicles in 2019”: 32.2 million units}] Whether "{"Annual sales volume of electric vehicles in 2019": 36.8 million units}" and "{"Annual sales volume of two-wheeled electric vehicles in 2020": 39.6 million units}" are in the statistical chart, if the multimodal large language model determines that "{"Annual sales volume of two-wheeled electric vehicles in 2019": 36.8 million units}" is in the statistical chart, then "{"Annual sales volume of two-wheeled electric vehicles in 2019": 36.8 million units}" is added to the third result set; if the multimodal large language model determines that {"Annual sales volume of two-wheeled electric vehicles in 2020": 39.6 million units}" is not in the statistical chart, there is no need to add {"Annual sales volume of two-wheeled electric vehicles in 2020": 39.6 million units}" to the third result set; then the second output result is [{"Annual sales volume of two-wheeled electric vehicles in 2018": 32.2 million units, "Annual sales volume of two-wheeled electric vehicles in 2019": 36.8 million units}].

[0108] In one possible implementation, the key-value pairs are converted into<k1.k2.k3…kn,v> format, and then count the number of result sets with the same key-value pairs after conversion.

[0109] In one possible implementation method, the confidence setting method of the second output result includes one or more of the following: the sum of the number of times each key-value pair appears / the total number of key-value pairs that appear in the recognition result of the large language model, or using a weighted calculation method, i.e., sigma (the proportion of key-value pairs that appear in the final result in each model * voting weight), or using statistical features to train a regression model for confidence score output.

[0110] In a possible implementation method, in addition to the above steps 301 to 302, the second output result can also be determined in the following manner: the second recognition result of the large language model with the strongest image capability is selected as the second output result.

[0111] Optionally, the number of similar large language models is calculated, where the similar large language models refer to other large language models whose similarity to the second recognition result of the large language model with the strongest image capability is greater than a similarity threshold. The present application does not limit the similarity threshold, and specifically, a similarity threshold of 1 means that the second recognition result of the similar large language model is the same as the second recognition result of the large language model with the strongest image capability.

[0112] Optionally, the confidence level of the second output result is: the number of similar large language models / the total number of large language models, and the output format is <output result of the most capable large model, confidence level>.

[0113] Optionally, the confidence of the second output result may also be: the weighted sum of the most capable large language model and similar large language models.

[0114] The above scheme determines the second output result of the document to be recognized through the recognition results of multiple large language models. The confidence of the second output result is relatively credible, so the second output result can be obtained accurately and effectively, thereby improving the accuracy of OCR recognition.

[0115] In a possible implementation method, the image data is a QR code; then the step of determining the second output result is as follows: Figure 4 As shown, the following steps are included:

[0116] Step 401: Based on the multiple second recognition results, count the number of multimodal large language models that recognize that the image data is a QR code.

[0117] Step 402: Determine a second output result based on the weighted sum of the multimodal large language model that identifies that the image data is a QR code.

[0118] In a possible implementation method, if the sum of the weighted weights of the multimodal large language model that identifies the image data as a QR code is greater than or equal to the third weight threshold, the second output result is determined to be "yes" or "YES"; if the sum of the weighted weights of the multimodal large language model that identifies the image data as a QR code is less than the third weight threshold, the second output result is determined to be "no" or "NO"; wherein the second output result of "yes" or "YES" indicates that the image data is a QR code; the second output result of "no" or "NO" indicates that the image data is not a QR code. This application does not limit the second threshold.

[0119] In a possible implementation method, the confidence level of the second output result is the number of codes recognized as QR codes / the total number of large language models.

[0120] In a possible implementation method, the confidence level of the second output result may also be the weighted sum of the major language models that identify the QR code.

[0121] The above scheme determines the second output result of the document to be recognized through the recognition results of multiple large language models. The confidence of the second output result is relatively credible, so the second output result can be obtained accurately and effectively, thereby improving the accuracy of OCR recognition.

[0122] Based on the same technical concept, Figure 5 A document recognition device 500 provided in an embodiment of the present application is exemplarily shown. Figure 5 As shown, it includes: a recognition unit 501 and an output unit 502. The recognition unit 501 is used to input the document to be recognized into an optical character recognition (OCR) engine to obtain a first recognition result output by the OCR engine; input the document to be recognized and recognition prompt words of different types of data into multiple different multimodal large language models respectively to obtain multiple second recognition results output by the multiple different multimodal large language models; the output unit 502 is used to align the first recognition result with the multiple second recognition results for text data to obtain a first output result; for image data, based on the multiple second recognition results, obtain a second output result; based on the first output result and the second output result, obtain the recognition result of the document to be recognized.

[0123] In a possible implementation method, the recognition unit 501 is used to divide the document to be recognized into blocks according to different types of data to obtain blocks of text data and blocks of image data; input the blocks of text data into an optical character recognition (OCR) engine; and input the blocks of text data and corresponding recognition prompt words, and the blocks of image data and corresponding recognition prompt words into multiple different multimodal large language models, respectively.

[0124] In a possible implementation method, the first recognition result includes a sub-recognition result for each row; the output unit 502 is used to respectively obtain weight information corresponding to the OCR engine and the multiple different multimodal large language models; for any sub-recognition result in the first recognition result, determine at least one target recognition result that meets the matching requirements with the sub-recognition result from the multiple second recognition results; based on the weight information corresponding to the OCR engine and the multiple different multimodal large language models, the sub-recognition results and the at least one target recognition result, count the weighted weights corresponding to the same recognition results; the first recognition result includes a sub-recognition result for each row; the recognition result corresponding to the maximum weighted weight sum is used as the first output item; and the first output result is obtained based on each first output item.

[0125] In a possible implementation method, the recognition unit 501 is used to input the sub-graph corresponding to the position indication information in the sub-recognition result and the corresponding recognition prompt word into the multiple different multimodal large language models to obtain multiple third recognition results if at least one target recognition result that meets the matching requirements with the sub-recognition result is not determined; and based on the multiple third recognition results, re-determine at least one target recognition result that meets the matching requirements with the sub-recognition result.

[0126] In a possible implementation method, the output unit 502 is configured to merge the sub-recognition results in a plurality of consecutive rows into one sub-recognition result if the sub-recognition results in a plurality of consecutive rows are all included in at least one second recognition result.

[0127] In one possible implementation method, the output unit 502 is used to use the recognition result with the largest weighted weight sum as the first output item if the maximum weighted weight sum is greater than the first weight threshold; if the maximum weighted weight sum is not greater than the first weight threshold, obtain the sub-graph corresponding to the position indication information in the sub-recognition result; generate a new sub-graph according to the information corresponding to the sub-graph and any second recognition result; and use the recognition result corresponding to the new sub-graph whose similarity with the sub-graph is greater than the pixel threshold as the first output item.

[0128] In a possible implementation method, the output unit 502 is used for the graph data to be a statistical graph; any second recognition result is a key-value pair result set; the weighted sum of the same key-value pair result set is calculated; and the key-value pair result set with the maximum weighted sum is used as the second output result.

[0129] In a possible implementation method, the output unit 502 is used to use the key-value pair whose maximum weighted weight sum is greater than the second weight threshold as the second output item; input the key-value pair whose maximum weighted weight sum is not greater than the second weight threshold and the subgraph where the statistical graph is located into at least one multimodal large language model, determine the key-value pair belonging to the subgraph where the statistical graph is located and use it as the second output item; and obtain the second output result based on each second output item.

[0130] Based on the same technical concept, the embodiment of the present application provides a document recognition device 600, which can be a computing device, for example. Figure 6 As shown, a document recognition device 600 includes at least one processor 601 and a memory 602 connected to the at least one processor. The specific connection medium between the processor 601 and the memory 602 is not limited in the embodiment of the present application. Figure 6 For example, the processor 601 and the memory 602 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0131] In the embodiment of the present application, the memory 602 stores instructions that can be executed by at least one processor 601, and the at least one processor 601 can execute the above-mentioned document recognition method by executing the instructions stored in the memory 602.

[0132] Among them, the processor 601 is a control center of the document recognition device 600, which can use various interfaces and lines to connect various parts of the computer equipment, and perform resource settings by running or executing instructions stored in the memory 602 and calling data stored in the memory 602. Optionally, the processor 601 may include one or more determination units, and the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 601. In some embodiments, the processor 601 and the memory 602 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0133] Processor 601 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0134] The memory 602 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 602 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 602 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 602 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0135] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer-executable program. The computer-executable program is used to enable a computer to execute a document recognition method listed in any of the above-mentioned methods.

[0136] An embodiment of the present application provides a computer program product, including a computer program executable by a computer device. When the program is run on the computer device, the computer device executes a document recognition method listed in any of the above-mentioned ways.

[0137] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0138] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0139] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0140] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0141] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A document recognition method, characterized in that: include: Inputting the document to be recognized into an optical character recognition (OCR) engine to obtain a first recognition result output by the OCR engine; Inputting the document to be recognized and recognition prompt words of different types of data into a plurality of different multimodal large language models respectively, and obtaining a plurality of second recognition results output by the plurality of different multimodal large language models; For text data, aligning the first recognition result with the plurality of second recognition results to obtain a first output result; For the image data, based on the plurality of second recognition results, obtain a second output result; Based on the first output result and the second output result, a recognition result of the document to be recognized is obtained.

2. The method according to claim 1, characterized in that Before inputting the document to be recognized into the optical character recognition (OCR) engine, the method further includes: According to different types of data, the document to be identified is divided into blocks to obtain blocks of text data and blocks of image data; Input the document to be recognized into the optical character recognition (OCR) engine, including: Input the blocks of the text data into the OCR engine; The document to be recognized and the recognition prompt words of different types of data are respectively input into a plurality of different multimodal large language models, including: The blocks of the text data and the corresponding recognition prompt words, the blocks of the image data and the corresponding recognition prompt words are respectively input into the multiple different multimodal large language models.

3. The method according to claim 1, characterized in that The first recognition result includes a sub-recognition result for each row; The aligning the first recognition result with the plurality of second recognition results to obtain a first output result includes: Respectively obtaining weight information corresponding to the OCR engine and the plurality of different multimodal large language models; For any sub-recognition result in the first recognition result, determine at least one target recognition result that meets a matching requirement with the sub-recognition result from the plurality of second recognition results; According to the weight information corresponding to the OCR engine and the multiple different multimodal large language models, the sub-recognition results and the at least one target recognition result, a weighted sum corresponding to the same recognition result is counted; the first recognition result includes a sub-recognition result for each row; The maximum weighted value and the corresponding recognition result are taken as the first output item; The first output result is obtained based on each first output item.

4. The method according to claim 3, characterized in that The method further comprises: If at least one target recognition result that meets the matching requirements with the sub-recognition result is not determined, the sub-graph corresponding to the position indication information in the sub-recognition result and the corresponding recognition prompt word are input into the multiple different multimodal large language models to obtain multiple third recognition results; based on the multiple third recognition results, at least one target recognition result that meets the matching requirements with the sub-recognition result is re-determined.

5. The method according to claim 3, characterized in that Before determining at least one target recognition result that meets the matching requirement with the sub-recognition result, the method further includes: If multiple consecutive rows of sub-recognition results are all in at least one second recognition result, the multiple consecutive rows of sub-recognition results are merged into one sub-recognition result.

6. The method according to claim 3, characterized in that The step of taking the recognition result having the largest weight and the largest value as the first output item includes: If the maximum weighted weight sum is greater than the first weight threshold, taking the recognition result with the maximum weighted weight sum as the first output item; If the maximum weighted sum is not greater than the first weight threshold, obtain the sub-graph corresponding to the position indication information in the sub-recognition result; generate a new sub-graph based on the information corresponding to the sub-graph and any second recognition result; and use the recognition result corresponding to the new sub-graph whose similarity with the sub-graph is greater than the pixel threshold as the first output item.

7. The method according to claim 1, characterized in that The graph data is a statistical graph; any second recognition result is a key-value pair result set; The obtaining a second output result based on the plurality of second recognition results comprises: Count the weighted sum of the result sets with the same key-value pairs; The key-value pair result set with the maximum weighted weight sum is used as the second output result.

8. The method according to claim 1, characterized in that The step of taking the key-value pair result set with the highest weight as the second output result includes: The key-value pair with the maximum weighted weight and greater than the second weight threshold is used as the second output item; Inputting the key-value pairs whose maximum weighted sum is not greater than the second weight threshold and the subgraph where the statistical graph is located into at least one multimodal large language model, determining the key-value pairs belonging to the subgraph where the statistical graph is located as the second output item; The second output result is obtained based on each second output item.

9. A document recognition device, characterized in that: Including recognition unit and output unit: The recognition unit is used to input the document to be recognized into an optical character recognition (OCR) engine to obtain a first recognition result output by the OCR engine; input the document to be recognized and recognition prompt words of different types of data into a plurality of different multimodal large language models respectively to obtain a plurality of second recognition results output by the plurality of different multimodal large language models; The output unit is used to align the first recognition result with the plurality of second recognition results for text data to obtain a first output result; For the image data, based on the plurality of second recognition results, obtain a second output result; Based on the first output result and the second output result, a recognition result of the document to be recognized is obtained.

10. A computing device, characterized in that include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 8 according to the obtained program instructions.

11. A computer-readable storage medium, characterized in that: The method comprises computer-readable instructions, and when a computer reads and executes the computer-readable instructions, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The invention comprises a computer program executable by a computer device, and when the program is run on the computer device, the computer device is caused to execute the steps of the method according to any one of claims 1 to 8.

Images