Trusted document information extraction method, system, device and storage medium

Abstract

The application discloses a kind of trusted document information extraction method, system, equipment and storage medium, they are corresponding scheme, in scheme: by automatically constructing negative sample, it solves the problem that manual labeling negative sample is impractical and lacks diversity;Meanwhile, a kind of joint optimization strategy is designed, for example, only the gradient of the indicator indicating error is allowed to be back propagated for graph-text pair, while for positive example graph-text pair, the gradient of text sequence and indicator is back propagated, the strategy ensures that negative sample will not negatively affect the ability of model to generate correct results;And, a self-evaluation mechanism is introduced, so that the model can evaluate and identify the results generated by itself;Finally, during inference, the model can identify the output that seems to be semantically coherent but is actually incorrect by judging the visual consistency of the generated text and image, thereby improving the reliability of the information extraction results.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, computer vision and natural language processing, and particularly relates to a trusted document information extraction method, system, device and storage medium. BACKGROUND

[0002] Existing document information extraction (DIE) methods can be mainly divided into two categories: methods relying on optical character recognition (OCR) and end-to-end methods not relying on OCR. Methods relying on OCR, such as document 1: C., Yang, Z.: Omniparser: A unified framework for text spotting, key information extraction and table recognition. arXiv preprint arXiv: 2403.19128 (2024).

[0003] Methods relying on OCR usually adopt a two-stage paradigm: first, use OCR tools to extract text, and then combine the OCR results with image features for document understanding (for example, document 2: Kim, D., Kim, Y., Kim, D., Lim, Y., Kim, G., Kil, T.: Scob: Universal text understanding via character-wise supervised contrastive learning with online text rendering for bridging domain gap. In: Proceedings of the IEEE / CVF International Conference on Computer Vision. pp. 19562-19573 (2023)). Although this method is effective, it has the following inherent defects: the dependence on OCR tools leads to additional computational overhead and error propagation, because imperfect OCR results will affect downstream performance. In addition, high dependence makes the model vulnerable to OCR engine and reading order errors.

[0004] In recent years, emerging end-to-end methods such as Donut (Document Understanding Transformer), SeRum (Selective Region Understanding Model), and OmniParser (Unified Framework for Text Spotting, Key Information Extraction, and Table Recognition) can directly process documents without the need for pre-processed OCR tools. Transformer is a self-attention neural network.

[0005] The above three methods correspond to the following documents in order: Document 3: Kim, G., Hong, T., Yim, M., Nam, J., Park, J., Yim, J., Hwang, W., Yun, S., Han, D., Park, S.: Ocr-free document understanding transformer. In European Conference on Computer Vision. pp. 498-517. Springer (2022); Document 4: Cao, H., Bao, C., Liu, C., Chen, H., Yin, K., Liu, H., Liu, Y., Jiang, D., Sun, X.: Attention where it matters: Rethinking visual document understanding with selective region concentration. In: Proceedings of the IEEE / CVF International Conference on Computer Vision. pp. 19517-19527 (2023); Document 5: Wan, J., Song, S., Yu, W., Liu, Y., Cheng, W., Huang, F., Bai, X., Yao, C., Yang, Z.: OmniParser: A unified framework for text spotting, key information extraction, and table recognition. arXiv preprint arXiv:2403.19128 (2024).

[0006] However, the above method directly applies the generative model structure without fully considering its inherent shortcomings. In dealing with visual-language tasks that require balanced multi-modal integration, existing models tend to over-rely on language information, leading to insufficient utilization of visual information and resulting in visually obvious errors or semantically coherent but factually incorrect prediction results, such as Figure 1 As shown in (a) and (b), the document images in the two parts of the figure are from the evaluation set of existing document understanding, and the two document images involved here are real photocopies of shopping tickets, reflecting the text pixel loss due to printing problems in real scenarios. The text sequence below each document image is the error result generated by the existing end-to-end model document model. This unbalanced multi-modal integration and utilization undermines the reliability of the extraction result.

[0007] Therefore, the present application is proposed. SUMMARY

[0008] The purpose of the present application is to provide a reliable document information extraction method, system, device and storage medium to solve the problem of over-reliance on language information and unbalanced multi-modal integration of existing end-to-end generative models in document information extraction tasks, thereby improving the reliability of information extraction results.

[0009] The purpose of the present application is achieved by the following technical solutions:

[0010] A reliable document information extraction method, comprising:

[0011] constructing an end-to-end generative model;

[0012] collecting a plurality of document images and corresponding real text sequences, replacing characters in each real text sequence based on character scores to obtain corresponding error text sequences, adding indicators at specified positions of the real text sequences and the error text sequences, and then combining them with the corresponding positive and negative example image-text pairs of the document images; and constructing query information: the query information of the positive example image-text pair is the question corresponding to the real text sequence, and the query information of the negative example image-text pair is the error text sequence and the corresponding question; wherein the indicators are used to indicate the category of the text sequence;

[0013] inputting the positive and negative example image-text pairs and the query information into the end-to-end generative model, and decoding the text sequence and the indicator from the document image and the query information by the end-to-end generative model; for the positive example image-text pair, constructing a positive sample loss function combining the decoded text sequence and indicator, and the real text sequence with added indicators; for the negative example image-text pair, constructing a negative sample loss function combining the decoded indicator; training the end-to-end generative model using the positive sample loss function and the negative sample loss function;

[0014] Input the document image to be processed and the query information into the trained end-to-end generation model, and decode to obtain a target text sequence and an indicator, and determine the category of the target text sequence according to the indicator.

[0015] A trusted document information extraction system for implementing the foregoing method, comprising:

[0016] A model construction unit for constructing an end-to-end generation model;

[0017] A training data construction unit for collecting a plurality of document images and corresponding real text sequences, replacing characters in each real text sequence based on character scores to obtain corresponding error text sequences, adding indicators at specified positions of the real text sequences and the error text sequences respectively, and then combining the corresponding document images to form positive and negative example image-text pairs, and constructing query information: the query information of the positive example image-text pair is a question corresponding to the real text sequence, and the query information of the negative example image-text pair is an error text sequence and a corresponding question; wherein the indicator is used to indicate the category of the text sequence;

[0018] A model training unit for inputting the positive and negative example image-text pairs and the query information into the end-to-end generation model, and decoding the text sequence and the indicator from the document image and the query information by the end-to-end generation model; for the positive example image-text pair, constructing a positive sample loss function by combining the decoded text sequence and the indicator, and the real text sequence with the added indicator; for the negative example image-text pair, constructing a negative sample loss function by combining the decoded indicator; and training the end-to-end generation model by using the positive sample loss function and the negative sample loss function;

[0019] A trusted document information extraction unit for inputting the document image to be processed and the query information into the trained end-to-end generation model, and decoding to obtain a target text sequence and an indicator, and determining the category of the target text sequence according to the indicator.

[0020] A processing device, comprising: one or more processors; a memory for storing one or more programs;

[0021] When the one or more programs are executed by the one or more processors, the one or more processors implement the foregoing method.

[0022] A readable storage medium storing a computer program, which, when executed by a processor, implements the foregoing method.

[0023] It can be seen from the technical solutions provided by the application that by automatically constructing negative samples, the problem of impracticality and lack of diversity in manual labeling of negative samples is solved; meanwhile, a joint optimization strategy is designed, for negative example image-text pairs, only the gradient of the indicator indicating an error is allowed to be back propagated, while for positive example image-text pairs, the gradients of the text sequence and the indicator are both back propagated, which ensures that the negative samples do not have a negative impact on the ability of the model to generate correct results; and a self-evaluation mechanism is introduced, so that the model can evaluate and identify the results generated by itself; finally, during reasoning, the model can identify the output that seems to be semantically coherent but is actually wrong by judging the visual consistency of the generated text and the image, thereby improving the reliability of the information extraction result. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0025] Figure 1 An error result schematic diagram of the existing end-to-end model document model generation provided by the embodiments of the present application.

[0026] Figure 2 A flowchart of a trusted document information extraction method provided by the embodiments of the present application.

[0027] Figure 3 A schematic diagram of an end-to-end generation model architecture and its workflow provided by the embodiments of the present application.

[0028] Figure 4 A schematic diagram of the effect of the trained end-to-end generation model provided by the embodiments of the present application.

[0029] Figure 5 A schematic diagram of a trusted document information extraction system provided by the embodiments of the present application.

[0030] Figure 6 A schematic diagram of a processing device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0032] First, the terms that can be used in the present text are described as follows:

[0033] The term "and / or" means either one or both, for example, X and / or Y means three cases including "X" or "Y" or "X and Y".

[0034] The terms "include", "contain", "have", "possess" or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example: including a technical feature element (such as raw materials, components, ingredients, carriers, dosage forms, materials, sizes, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or articles, etc.), should be interpreted as not only including the explicitly listed technical feature elements, but also including other technical feature elements that are not explicitly listed but are known in the art.

[0035] The term "consisting of" means excluding any technical feature elements not explicitly listed. If this term is used in the claims, the term will make the claim closed, so that it does not contain technical feature elements other than the explicitly listed technical feature elements, except for conventional impurities related thereto. If the term only appears in a certain clause of the claim, it is only limited to the elements explicitly listed in that clause, and the elements described in other clauses are not excluded from the overall claim.

[0036] A trusted document information extraction method, system, device and storage medium provided by the present application are described in detail below. The content not described in detail in the embodiments of the present application belongs to the prior art known to those skilled in the art. If no specific conditions are specified in the embodiments of the present application, the conventional conditions or the conditions recommended by the manufacturer are used. If no manufacturer of the reagent or instrument used in the embodiments of the present application is specified, it is a conventional product that can be purchased on the market.

[0037] Embodiment one

[0038] The embodiments of the present application provide a trusted document information extraction method, as shown in Figure 2 The method mainly includes the following steps:

[0039] Step 1, constructing an end-to-end generation model.

[0040] In the embodiments of the present application, an end-to-end generation model containing a visual encoder and a query text decoder is constructed.

[0041] Step 2, constructing training data, including positive and negative example image-text pairs and corresponding query information.

[0042] In the embodiment of the present application, a data set containing a document image and a corresponding real text sequence is collected, the real text sequence is taken as a positive sample, a character in each positive sample is replaced based on a character score to obtain a corresponding error text sequence, and the error text sequence is taken as a negative sample; a corresponding indicator is added at a specified position of the positive sample and the negative sample respectively, and the indicator is used to indicate the category of the text sequence, specifically, the positive sample is indicated as correct text, and the negative sample is indicated as error text; the positive sample with the added indicator and the corresponding document image are taken as a positive example image-text pair, and the negative sample with the added indicator and the corresponding document image are taken as a negative example image-text pair.

[0043] In the embodiment of the present application, the replacing of the character in each positive sample based on the character score to obtain the corresponding error text sequence comprises: inputting the positive sample corresponding document image into a given model to obtain a character output with a score; wherein the score in the character output with the score represents the score of the character at the corresponding position, and the decoded text sequence in the subsequent text is composed of the characters with the highest scores at each position; a plurality of positions are selected, and the character with the second highest score at the corresponding position is used to replace the character at the corresponding position in the positive sample to obtain the corresponding error text sequence; compared with the real text sequence, the error text sequence is visually similar but incorrect in content (semantically coherent).

[0044] In the embodiment of the present application, the number of the plurality of positions is one or more, which can be set by a user according to actual conditions or experience, and the present application does not make specific limitations.

[0045] The given model comprises a baseline model and an end-to-end generation model currently being trained, the baseline model is a model obtained after training the end-to-end generation model using only the positive example image-text pair; and the baseline model or the end-to-end generation model currently being trained is selected as the given model according to the progress.

[0046] Specifically, the training stage is divided into three stages according to the training progress (training round), the given model in the first stage is the baseline model, the given model in the second stage is randomly selected from the baseline model and the end-to-end generation model currently being trained, and the given model in the third stage is the end-to-end generation model currently being trained.

[0047] In the embodiment of the present application, query information is also constructed for the positive example image-text pair and the negative example image-text pair, wherein the query information of the positive example image-text pair is a question corresponding to the real text sequence, and the query information of the negative example image-text pair is the error text sequence and a corresponding question. For example, the document image is an image of a ship ticket, the real text sequence is Haigun, that is, it is the ship name in the document image, and the query information is the ship name; the error text sequence is Haidia, and the query information contains the ship name and Haidia.

[0048] It should be noted that the step numbers above are only used to distinguish different steps, and do not limit the execution order of the steps, and the specific execution order can be determined according to the step content.

[0049] Step 3, training an end-to-end generation model.

[0050] In the embodiment of the application, the positive and negative example image-text pairs and the corresponding query information are input into the end-to-end generation model, and the end-to-end generation model decodes the text sequence and the indicator from the document image and the query information; specifically: the visual encoder extracts features from the document image in the positive and negative example image-text pairs, the query text decoder decodes the text sequence and the corresponding indicator in combination with the query information and the features extracted by the visual encoder; for the positive example image-text pair, a positive sample loss function is constructed in combination with the decoded text sequence and the corresponding indicator, and the positive sample and the corresponding indicator; for the negative example image-text pair, a negative sample loss function is constructed in combination with the decoded indicator; the end-to-end generation model is trained by using the positive sample loss function and the negative sample loss function.

[0051] In the embodiment of the application, the visual encoder extracts features from the document image in the positive and negative example image-text pairs, and the query text decoder decodes the text sequence and the corresponding indicator in combination with the query information and the features extracted by the visual encoder, which includes: the visual encoder maps the document image into initial features and serializes them into a group of embedded features as the extracted features; the query text decoder includes a query decoder and a text decoder; the query decoder decodes the query information into a query vector, and the text decoder combines the query vector and the features extracted by the visual encoder to generate the text sequence and the corresponding indicator in an autoregressive manner.

[0052] In the embodiment of the application, the positive sample loss function and the negative sample loss function are represented as:

[0053] ;

[0054] ;

[0055] wherein, is the positive sample loss function, is the negative sample loss function, is the real text sequence, is the indicator contained in the positive and negative example image-text pairs; represents the probability of decoding out under the condition that the document image i and the query information q are given, which is calculated in combination with the decoded text sequence and the real text sequence; represents the probability of decoding out the probability of the decoded indicator, which is calculated in combination with the indicator contained in the positive and negative example image-text pair; and log is a logarithmic function.

[0056] In the embodiments of the present application, the visual encoder, the query decoder and the text decoder can all be implemented by using existing structures, and the present application does not limit the specific structures.

[0057] Step 4, trusted document information extraction.

[0058] In the embodiments of the present application, steps 2 and 3 are repeatedly performed until a set stop condition (for example, model convergence) is met, the document image to be processed and the query information are input into the trained end-to-end generation model, the target text sequence and the corresponding indicator are obtained, and it is judged whether the obtained target text sequence is the correct text according to the indicator.

[0059] The above scheme provided by the embodiments of the present application solves the problem of impracticality and lack of diversity of manual labeling of negative samples by automatically constructing negative samples; at the same time, a positive and negative example image-text pair joint optimization strategy is designed, for the negative example image-text pair, only the gradient of the indicator indicating the error is allowed to be back propagated, and for the positive example image-text pair, the gradients of the text sequence and the indicator are both back propagated, the strategy ensures that the negative example image-text pair will not have a negative impact on the ability of the model to generate correct results; and a self-evaluation mechanism is introduced, so that the end-to-end generation model can evaluate and identify the results generated by itself; finally, during inference, the end-to-end generation model can identify the output that seems to be semantically coherent but is actually incorrect by judging the visual consistency of the generated text and the image, thereby improving the reliability of the information extraction result.

[0060] In order to more clearly show the technical solutions provided by the present application and the technical effects produced, the method provided by the embodiments of the present application is described in detail below with specific embodiments.

[0061] I. Overall introduction of the scheme.

[0062] 1. Self-evaluation mechanism.

[0063] The present application introduces a self-evaluation mechanism, so that the end-to-end generation model can evaluate and identify the results generated by itself. The mechanism trains the end-to-end generation model to distinguish between positive and negative samples by adding indicators at designated positions in the text sequence. The mechanism promotes the end-to-end generation model to enhance the utilization of visual information without weakening the utilization of language information.

[0064] For example, the indicator can be added at the end position of the text sequence, and the indicator is divided into two types, one of which is used to indicate that the text sequence is the correct text, and the following mark form can be used: <eos>Another kind of mark is used to indicate the text sequence is error text, which is <error_eos>.

[0065] 2. Negative sample automatic construction algorithm.

[0066] In the embodiment of the present application, an algorithm for automatically generating high-quality negative samples is proposed. In the training process, according to the score output generated by the given model (i.e. the baseline model and the end-to-end generation model currently being trained), the characters with high score ranking are selected for replacement, so as to automatically construct error text sequences similar to real text sequences in vision, and combine the corresponding document images (i.e. the document images corresponding to the corresponding real text sequences) to form negative samples. Through this algorithm, the problem of impracticality and lack of diversity of manual labeling of negative samples is solved.

[0067] 3. Joint training optimization strategy.

[0068] A joint training optimization strategy is designed to optimize positive and negative example image-text pairs simultaneously. For negative example image-text pairs, only the gradient of the error indicator is allowed to be backpropagated; for positive example image-text pairs, the gradients of both the text sequence and the indicator are backpropagated. This strategy ensures that negative example image-text pairs do not negatively affect the model's ability to generate correct results.

[0069] II. Detailed introduction of the scheme.

[0070] The following mainly introduces the model architecture, workflow, self-evaluation mechanism, negative sample automatic construction algorithm and joint training optimization strategy.

[0071] 1. Introduction of model architecture and workflow.

[0072] The present application provides an end-to-end generation model for document information extraction (DIE), which is a technology for recognizing, locating and extracting specific structured information from different types of document images (such as invoices, receipts, forms, certificates, etc.), and belongs to an important part of document image understanding.

[0073] The end-to-end generation model does not need to rely on external OCR tools and directly extracts structured information from document images. It mainly consists of a vision encoder and a query-text decoder, including a query decoder and a text decoder, as shown in Figure 3 .

[0074] The visual encoder is responsible for mapping the input document image into initial features and serializing them into a set of embedding features; the query text decoder then decodes the text sequence and indicator in a self-recursive manner based on the image features from the visual encoder and the input query information.

[0075] Figure 3 In the illustrated example, examples of document images and query information are provided, and the document images here also embody the situation of text pixel loss due to printing problems in the real scene, and further embody the situation of character occlusion existing in the real scene.

[0076] 2. Self-evaluation mechanism.

[0077] The present application introduces a self-evaluation task in model training. Specifically, the end-to-end generation model is required to add a special indicator after generating the text sequence: for correct samples (i.e. the real text sequence), the additional <eos>The <error_eos> label is added to mark the error text sequence as incorrect.

[0078] When the end-to-end generation model encounters an error text sequence that is syntactically correct but does not match the content of the document image, it cannot determine its correctness based on language information alone. At this time, in order to accurately predict the <error_eos> label, the end-to-end generation model needs to make more in-depth use of visual information to identify the inconsistencies between the text sequence and the document image, such as identifying visually similar but actually incorrect characters. This mechanism forces the model to rely more on visual information when faced with semantically coherent but incorrect results, thereby achieving effective use of visual information.

[0079] 3. Negative sample automatic construction algorithm

[0080] To solve the problem of lack of high-quality negative samples, the present application proposes a negative sample automatic construction algorithm. This algorithm uses a baseline model trained only with positive samples or the end-to-end generation model currently being trained to obtain the baseline model's character output with scores during training, and selects a second candidate character (the second highest score character) that is visually similar to replace the corresponding real character, thereby automatically generating error text sequences that are visually similar to the real text sequences in the positive samples but incorrect in content. These automatically generated error text sequences will be added with the corresponding indicator <error_eos> and form high-quality negative sample data with the corresponding document images, greatly improving the effectiveness of the self-evaluation mechanism. The algorithm is described in Table 1.

[0081] Table 1: Negative sample automatic construction algorithm

[0082]

[0083] In Table 1, the meaning of each symbol is explained as follows: Function gen_negative_samples() is the name of the entire algorithm, which is a function for generating negative samples; gen_model is the negative sample generation model, that is, the baseline model described above; model is the main model, that is, the end-to-end generation model currently being trained (referred to as the currently trained model); the algorithm can use the currently trained end-to-end model to generate negative samples; i is a document image, q is query information; e is the current cycle (Epoch), which is the current iteration number in the training process; m is the maximum cycle number (Max_epochs), that is, the number of rounds of training required; flag is a flag (similar to a threshold), which is a dynamic flag that changes according to the training progress, used to control the probability of selecting a certain model (gen_model or model) to generate negative samples; prob is a probability, which is a randomly generated value between 0 and 1, used to compare with flag to achieve random selection of models; torch.no_grad() is a no-gradient calculation (context manager), which is a common usage in PyTorch (an open-source deep learning framework), and all calculations in Table 1 are not tracked for gradients; gen_logits is the generated logits, that is, the original, non-normalized prediction scores (corresponding to the character scores in the text) before outputting the final probability (such as Softmax); get_negative() is a function for obtaining negative samples, which is a core auxiliary function that receives gen_logits and then selects the second highest score character to replace the real character, thereby constructing a visually similar but incorrect error sample sequence; negative_samples are negative samples.

[0084] For ease of understanding, the algorithm shown in Table 1 is explained line by line as follows. The algorithm divides the entire process into three stages, and as the training progresses, the strategy for generating negative samples gradually transitions from "relying on the baseline model" to "relying on the main model itself", thereby realizing the evolution from simple negative samples to hard negative mining.

[0085] Line 1: It is clear that the algorithm requires two models as input: one is the baseline model gen_model for generating negative samples, and the other is the main model model being trained.

[0086] Line 2: The input data required for the current batch (batch) is obtained, including the document image i, the corresponding real text sequence q, and the current cycle e and the total cycle number m for judging the training progress.

[0087] Line 3: Early training phase (first 1 / 3 of epochs).

[0088] Line 4: Set flag to -1.

[0089] Line 5: Mid training phase (middle 1 / 3 of epochs).

[0090] Line 6: Set flag to 0.5.

[0091] Line 7: Late training phase (last 1 / 3 of epochs).

[0092] Line 8: Set flag to 1.

[0093] Line 9: End this condition for setting flag.

[0094] Line 10: Generate a random number between 0 and 1 and assign it to prob.

[0095] Line 11: Enter a code environment that does not compute gradients. Ensure that the following negative sample generation process does not affect the training of the model.

[0096] Line 12: Decide which model to use based on the values of prob and flag. In the early training phase (flag = -1): prob (0~1) is always greater than -1, so the code in the following if block (Line 13~Line 14) will always be executed. In the mid training phase (flag = 0.5): prob has about a 50% chance of being greater than 0.5, so there is a 50% chance that the code in the following if block will be executed. In the late training phase (flag = 1): prob (0~1) will never be greater than 1, so the code in this if block will not be executed, and the else part (Line 16~Line 17) will always be executed.

[0097] Line 13: Use the negative sample generation model (gen_model) to process document image i and real text sequence q, and output the predicted character scores. In the early training phase, completely rely on this more stable baseline model to generate negative samples.

[0098] Line 14: Call the get_negative function to generate the final negative sample based on the logits output by gen_model, by replacing the second candidate character, etc.

[0099] Line 15: If the condition prob > flag is not met (always in the late training phase, and with a 50% chance in the mid training phase).

[0100] Line 16: Use the main model (model) being trained to generate the predicted logits by itself.

[0101] Line 17: Also call the get_negative function, this time based on the prediction of the main model model. The negative samples generated in this way are "hard negatives" for the main model, because they are the samples that the model itself is most likely to confuse.

[0102] Line 18: End the decision-making judgment. At this time, the final generated negative samples have been stored in the negative_samples variable.

[0103] 4. Joint optimization training strategy.

[0104] The present application designs a precise joint training optimization strategy to ensure that negative sample training does not have a negative impact on the normal generation ability of the model. The strategy optimizes the loss function of the positive samples and the negative samples at the same time. The positive sample loss function is defined as follows:

[0105] ;

[0106] wherein, is the real text sequence, is an indicator, allowing the gradient of the text sequence and the indicator to be backpropagated. Corresponding to the example provided in the foregoing, the indicator in this case is <eos>.

[0107] The loss function of the negative sample is as follows:

[0108] ;

[0109] Wherein, only the gradient of the indicator indicating error is allowed to be back-propagated, and the gradient of the text sequence part is prevented from being back-propagated. Corresponding to the example provided in the foregoing, the indicator in this case is < error_eos >.

[0110] The overall loss function is the sum of the positive sample loss function and the negative sample loss function, and is expressed as follows:

[0111] .

[0112] The above joint optimization training strategy provided by the present application ensures that the end-to-end generation model can effectively learn to identify errors and will not weaken its ability to generate correct results due to the training of negative samples, and finally realizes the balanced integration and utilization of visual and language information.

[0113] Based on the above introduction, the overall process of the present application can be divided into two parts: the model training stage and the model reasoning and application stage.

[0114] (1) Model training stage.

[0115] (1.1) Data input: Prepare a data set containing document images and corresponding real text sequences.

[0116] (1.2) Model construction: Construct an end-to-end generation model composed of a visual encoder and a query text decoder, and initialize the model parameters.

[0117] (1.3) Positive case image-text pair construction: Add an <eos>The label, combined with the corresponding document image, is constructed as a positive example image-text pair.

[0118] (1.4) Negative image-text pair construction: In each training batch, an algorithm for automatically constructing negative samples is used to construct a corresponding negative sample for each positive sample. The algorithm first uses the currently trained model to perform a forward propagation on the document image to obtain logits (character scores), i.e., character output with scores; then, according to the logits, the second-best character that is visually most similar to the real character is selected and replaced into the corresponding position of the real text sequence, thereby generating an incorrect error text sequence that is visually similar but incorrect in content, i.e., a negative sample; an <error_eos> label is added at the end of the error text sequence, and then combined with the corresponding document image to form a negative example image-text pair.

[0119] (1.5) Joint optimization.

[0120] Positive sample loss: Calculate the cross-entropy loss of the positive sample, and the loss calculation range includes the entire text sequence and <eos>Token. All gradients are backpropagated.

[0121] Negative sample loss: calculate the cross-entropy loss of the negative sample, but only the prediction of the <error_eos> token is calculated for loss, preventing the gradient backpropagation of the text sequence part.

[0122] Total loss: add the positive sample loss and the negative sample loss to get the total loss, and use this total loss to update the model parameters.

[0123] Repeat (1.3)~(1.5), and iterate the training until the model converges.

[0124] (2) Model inference and application stage.

[0125] (2.1) Input: input the document image to be processed and the query information into the trained end-to-end generation model. The query information here can be set by the user according to the needs to guide the model to decode the required text sequence from the document image.

[0126] (2.2) Text generation: the trained end-to-end generation model extracts image features through the visual encoder, and generates text sequences and corresponding indicators through the query text decoder in an autoregressive manner.

[0127] (2.3) Evaluation and output: judge the category of the target text sequence according to the indicator. If the indicator is <eos>If the indicator is <error_eos>, the text sequence is determined to be untrusted, and the user can be prompted for manual review or reprocessing. This process enables the end-to-end generation model to provide a confidence signal about the reliability of its output, thereby enhancing the trustworthiness of the entire document information extraction scheme.

[0128] III. Effect description

[0129] The present application solves the inherent defects of existing end-to-end generation models, brings significant technical progress and positive effects, mainly reflected in the following aspects.

[0130] 1. Significantly improves the accuracy and robustness of information extraction.

[0131] By balancing the use of visual and language information, the present application effectively reduces errors caused by excessive reliance on language by the model. The experimental results on the CORD, Ticket and SROIE public benchmark datasets are shown in Table 2. Obviously, the performance of the present application is significantly better than the prior art. For example, on the CORD dataset, the F1 score (an evaluation indicator that takes into account the precision and recall of the model) of the present method (Ours in Table 2) reaches 97.4%, which is 2.1% higher than the state-of-the-art OmniParser method (95.3%) and 2.9% higher than the Donut method (94.5%). This shows that the present application can more accurately extract the required information from the document and has stronger generalization ability on different types of documents.

[0132] Table 2: Quantitative comparison results with other models

[0133]

[0134] The existing methods are divided into two groups in Table 2. (1) The first group: OCR-based methods; BERT: Bidirectional Encoder Representations from Transformers, a bidirectional encoder representation model based on Transformers. It is a landmark pre-training language model in the field of natural language processing, but does not itself contain the ability to process document layout or visual information. BROS: BERT Relying On Spatiality, a pre-training language model designed specifically for understanding scanned documents, which encodes text, layout, and visual information together through a new pre-training strategy. LayoutLM: Layout Language Model, a landmark model proposed by Microsoft, which adds two-dimensional position embeddings (2D Position Embeddings) to BERT, allowing the model to understand the spatial layout information of text on the document page. LayoutLMv2: Layout Language Model v2, an improved version of LayoutLM, which not only considers text and layout, but also integrates the visual features (image information) of the document into the pre-training stage, achieving multi-modal fusion of text-layout-image. LayoutLMv3: Layout Language Model v3, a further simplification and optimization of the LayoutLM series, which unifies the multi-modal representation of text and image, adopts a more concise architecture and pre-training objectives such as "Word-Patch Alignment", and improves efficiency and performance. (2) The second group is end-to-end or OCR-free methods; Dessurt: Document End-to-End Understanding and Recognition Framework, an extensible and general end-to-end document-level information extraction model. Donut: Document Understanding Transformer, an innovative OCR-free model that generates structured text results directly from document images like an image-to-text translation task, without the need for pre-OCR. DocParser: Document Parsing Framework, an OCR-free document information extraction model based on Transformer, similar to Donut. OmniParser: Unified Document Understanding Framework, a unified framework designed to handle multiple styles of document information extraction. SeRum-prompt: Selective Region Understanding Model.#Params in Table 2 represents the model parameter amount of each method, M is million, the first group of methods contains two parts of parameters, the part before the plus sign is the OCR model parameter, and the part after the plus sign represents the information extraction module parameter, and the second group of methods only contains the OCR model parameter.

[0135] 2. The reliability of the model result is improved.

[0136] The self-evaluation mechanism of the present application enables the model to have the ability of self-verification. The model can identify those outputs that seem to be semantically coherent but are actually wrong by judging the visual consistency of the generated text and image. As shown in Figure 4 , the predicted text sequence and the corresponding indicator ( <eos>, the label below is the real text sequence, considering the occlusion problem of characters in the document image, therefore, Figure 4 As shown in the four dashed boxes, only part of the area in the document image is displayed; similarly, these document images also reflect the loss of text pixels due to various problems in the real scene.

[0137] This is of great significance in practical applications, for example, when processing key information documents such as invoices and receipts, it can greatly reduce the risk caused by model errors, and improve the reliability and credibility of the scheme.

[0138] 3. Reduced dependence on high-quality labeled data.

[0139] Traditional reliability improvement methods usually require a large amount of manual annotation to construct negative samples, which is not feasible in practice. The negative sample automatic construction algorithm proposed in the present application solves this problem. It can automatically generate high-quality negative samples during training, greatly reducing the dependence on manual annotation, making the model training more efficient and scalable, and reducing research and development costs.

[0140] 4. Achieved more efficient multi-modal integration.

[0141] Through the joint training optimization strategy designed with care, the present application optimizes the generation ability and evaluation ability of the model during training. This not only promotes the model to perform excellently in the generation task, but also simultaneously enables it to better integrate and utilize information from the two modalities of document images and text, achieving a deeper and more balanced multi-modal integration.

[0142] From the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0143] Embodiment Two

[0144] The present application also provides a trusted document information extraction system, which is mainly used to implement the method provided by the foregoing embodiments, as shown in the figure, the system mainly includes: Figure 5

[0145] A model construction unit is configured to construct an end-to-end generation model.

[0146] ​The training data construction unit collects multiple document images and their corresponding real text sequences. For each real text sequence, characters are replaced based on character scores to obtain the corresponding error text sequence. Indicators are added at specified positions in both the real and error text sequences, and then these are combined with the corresponding document images to form positive and negative image-text pairs. Furthermore, query information is constructed: the query information for positive image-text pairs is the question corresponding to the real text sequence, and the query information for negative image-text pairs is the error text sequence and its corresponding question. The indicators are used to identify the category of the text sequence.

[0147] The model training unit is used to input positive and negative example image-text pairs and query information into the end-to-end generation model. The end-to-end generation model uses the document image and query information to decode the text sequence and indicators. For positive example image-text pairs, a positive sample loss function is constructed by combining the decoded text sequence and indicators with the real text sequence with added indicators. For negative example image-text pairs, a negative sample loss function is constructed by combining the decoded indicators. The end-to-end generation model is trained using the positive sample loss function and the negative sample loss function.

[0148] The trusted document information extraction unit inputs the document image to be processed and the query information into the trained end-to-end generation model, decodes it to obtain the target text sequence and indicator, and determines the category of the target text sequence based on the indicator.

[0149] Since the main technical details of this system have been described in detail in previous embodiments, they will not be repeated here.

[0150] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0151] Example 3

[0152] The present invention also provides a processing device, such as Figure 6 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0153] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0154] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0155] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0156] The output device can be a display terminal;

[0157] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0158] Example 4

[0159] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0160] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0161] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.< / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos>

Claims

1. A method of trusted document information extraction, characterized by, The method comprises the following steps: constructing an end-to-end generation model; collecting a plurality of document images and corresponding real text sequences, replacing characters in each real text sequence based on character scores to obtain corresponding error text sequences, adding indicators at specified positions of the real text sequences and the error text sequences respectively, and then combining the corresponding document images to form positive and negative example image-text pairs; and constructing query information: the query information of the positive example image-text pair is the question corresponding to the real text sequence, and the query information of the negative example image-text pair is the error text sequence and the corresponding question; wherein the indicators are used to indicate the category of the corresponding text sequence; inputting the positive and negative example image-text pairs and the query information into the end-to-end generation model, and decoding the text sequence and the indicators from the end-to-end generation model using the document images and the query information; for the positive example image-text pair, constructing a positive sample loss function by combining the decoded text sequence and indicators, and the real text sequence with added indicators; for the negative example image-text pair, constructing a negative sample loss function by combining the decoded indicators; and training the end-to-end generation model using the positive sample loss function and the negative sample loss function; inputting a document image to be processed and query information into the trained end-to-end generation model, decoding to obtain a target text sequence and indicators, and determining the category of the target text sequence according to the indicators.

2. The method of claim 1, wherein, The method of replacing characters in each real text sequence based on character scores to obtain corresponding error text sequences comprises: inputting the corresponding document image of the real text sequence into a given model to obtain a character output with scores; wherein the scores in the character output with scores represent the scores of the characters at the corresponding positions; selecting a number of positions, and replacing the characters at the corresponding positions in the real text sequence with the characters at the corresponding positions that have the second highest scores to obtain corresponding error text sequences.

3. The method of claim 2, wherein, The given model comprises a baseline model and an end-to-end generation model that is currently being trained, and the baseline model or the end-to-end generation model that is currently being trained is selected as the given model according to the training progress; wherein the baseline model is a model obtained by training the end-to-end generation model using only positive example image-text pairs.

4. The method of claim 3, wherein, Selecting the baseline model or the end-to-end generation model that is currently being trained as the given model according to the training progress comprises: dividing the training stage into three stages according to the training progress, then the given model in the first stage is the baseline model, the given model in the second stage is randomly selected from the baseline model and the end-to-end generation model that is currently being trained, and the given model in the third stage is the end-to-end generation model that is currently being trained.

5. The method of claim 1, wherein, The method of decoding the text sequence and the indicators from the end-to-end generation model using the document images and the query information comprises: The end-to-end generation model comprises a visual encoder and a query text decoder. The visual encoder extracts features from the document image, and the query text decoder combines the query information and the extracted features to decode the text sequence and the corresponding indicators.

6. The method of claim 5, wherein, The method further comprises: The visual encoder maps the document image into initial features and serializes the initial features into a set of embedded features as the extracted features. The query text decoder comprises a query decoder and a text decoder; the query decoder decodes query information into a query vector, and the text decoder decodes a text sequence and a corresponding indicator in an autoregressive manner by combining the query vector and features extracted by a visual encoder.

7. The method of claim 1, wherein, The positive sample loss function and the negative sample loss function are represented as: ； ； wherein, is a positive sample loss function, is a negative sample loss function, is a ground-truth text sequence, is an indicator contained in the positive-negative example image-text pair; denotes a probability of decoding out under the condition that the given document image i and the query information q are given, which is calculated in combination with the decoded text sequence and the ground-truth text sequence; denotes a probability of decoding out under the condition that the given document image i and the query information q are given, which is calculated in combination with the decoded indicator and the indicator contained in the positive-negative example image-text pair; log is a logarithmic function.

8. A trusted document information extraction system, characterized by, The method comprises the following steps: The model construction unit is configured to construct an end-to-end generation model. The training data construction unit is configured to collect a plurality of document images and corresponding real text sequences, replace characters in each real text sequence based on character scores to obtain corresponding error text sequences, add indicators at specified positions of the real text sequences and the error text sequences respectively, and then form positive and negative example image-text pairs corresponding to the document images, and construct query information: the query information of the positive example image-text pair is a question corresponding to the real text sequence, and the query information of the negative example image-text pair is an error text sequence and a corresponding question; wherein the indicators are used to indicate the category of the text sequence. The model training unit is configured to input the positive and negative example image-text pairs and the query information into the end-to-end generation model, decode text sequences and indicators from the document images and the query information by using the end-to-end generation model, construct a positive sample loss function by combining the decoded text sequences and indicators, the real text sequence with added indicators, and the error text sequence for the positive example image-text pair, construct a negative sample loss function by combining the decoded indicators for the negative example image-text pair, and train the end-to-end generation model by using the positive sample loss function and the negative sample loss function. The trusted document information extraction unit inputs a document image to be processed and query information into the trained end-to-end generation model, decodes target text sequences and indicators, and determines the category of the target text sequences according to the indicators.

9. A processing device, characterized by The computer program product comprises a computer readable storage medium having computer readable program code embodied therein, the computer readable program code comprising: one or more processors; memory for storing one or more programs; wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.

10. A readable storage medium, storing a computer program, characterized in that, The computer program, when executed by a processor, implements the method of any one of claims 1-7.

Images