Key information extraction method and device, equipment, medium and product

Abstract

The invention discloses a key information extraction method and device, equipment, a medium and a product. The method comprises the steps that a to-be-tested document image and a problem text sequence of the to-be-tested document image are received; obtaining a target key information extraction model; the to-be-tested document image and the question text sequence of the to-be-tested document image are input into a target key information extraction model, an answer text sequence of the to-be-tested document image is generated, and the answer text sequence of the to-be-tested document image is obtained by querying a target mark generated by the target key information extraction model. The hidden state of the target mark implicitly encodes the physical position information of the answer in the to-be-tested document image. According to the method, the target mark is introduced, the target key information extraction model is guided to perceive the physical position information of the answer file sequence in the to-be-tested document image, and the answer is positioned through the physical position information, so that the answer text sequence of the to-be-tested document image is obtained, and the accuracy and effect of key information extraction are improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method, apparatus, device, medium, and product for extracting key information. Background Technology

[0002] With the continuous advancement of information digitization and office automation, image documents have become an indispensable medium in many business processes such as enterprise management, financial reimbursement, and contract review. This trend has driven the urgent need for technologies that automatically extract key information from image documents, thus giving rise to the development of Key Information Extraction (KIE) technology.

[0003] Currently, key information extraction methods can convert images into parsable text and their location information using an OCR module, and then combine this with a text understanding model for entity recognition, or directly input the image into the model and output the text result.

[0004] However, the above methods only output text results and lack visual evidence, which leads to low accuracy in extracting key information. Summary of the Invention

[0005] The main objective of this application is to provide a method, apparatus, device, medium, and product for extracting key information, which can improve the accuracy and effectiveness of key information extraction.

[0006] To achieve the above objectives, firstly, this application provides a method for extracting key information, including: Receive the image of the document to be tested and a sequence of question texts related to the image of the document to be tested; Model for extracting key target information; The test document image and the question text sequence of the test document image are input into the target key information extraction model to generate the answer text sequence of the test document image. The answer text sequence of the test document image is obtained by target tag query generated by the target key information extraction model. The hidden state of the target tag implicitly encodes the physical location information of the answer in the test document image.

[0007] In one embodiment, obtaining a target key information extraction model includes: Construct a pre-trained dataset; The initial key information extraction model is trained using a pre-trained dataset to obtain the target key information extraction model.

[0008] In one implementation, constructing a pre-trained dataset includes: Obtain training document images; The training document images are labeled to obtain the HTML code structure information corresponding to the training document images; The HTML code structure information is input into a pre-defined large language model to generate text features of the training document image. The text features include question-answer pair sequences and actual physical location information of the answers. The question-answer pair sequences include the question text sequence of the training document image and the answer text sequence of the training document image. The actual physical location information of the answers refers to the actual physical location of the answers in the initial document image. Use the training document images and the corresponding text features of the training document images as training samples; A pre-training dataset is constructed by using a predetermined number of training samples.

[0009] In one embodiment, the initial key information extraction model is trained using a pre-trained dataset to obtain the target key information extraction model, including: Obtain initial key information and extract the target loss function of the model; The initial key information extraction model is trained by using the training document images and the question text sequences of the training document images in the pre-training dataset as inputs and the answer text sequences of the training document images in the pre-training dataset as outputs. Once the target loss function of the initial key information extraction model reaches its minimum value, the target key information extraction model is obtained.

[0010] In one embodiment, obtaining the target loss function of the initial key information extraction model includes: Obtain the first loss function and the second loss function; Based on the first loss function, the second loss function, and the balancing weight coefficient, the target loss function of the initial key information extraction model is determined.

[0011] In one embodiment, obtaining the first loss function includes: The visual encoder in the initial key information extraction model is used to transform the training document images and generate visual features. Obtain the initial key information and extract the answer text sequence of the current position output by the model; The first loss function is constructed based on the visual features and the answer text sequence of the current location.

[0012] In one embodiment, obtaining the second loss function includes: The initial key information is obtained by extracting the first label generated by the model, wherein the hidden state of the first label implicitly encodes the physical location information of the answer in the training document image; A linear transformation is performed on the hidden state of the first marker to obtain the query vector; The query vector is used to search a predefined coordinate vocabulary, and the position probability distribution is obtained through a preset function. A second loss function is constructed based on the location probability distribution.

[0013] In one embodiment, obtaining the first tag generated by the initial key information extraction model includes: The visual encoder in the initial key information extraction model is used to transform the training document images and generate visual features. The question text sequence of the training document image is transformed to obtain question text features; By fusing visual features and question text features, a fused feature is obtained; The fused features are decoded by the text decoder in the initial key information extraction model to generate the first label.

[0014] In one implementation, a second loss function is constructed based on the location probability distribution, including: Calculate the weighted expected value of the location probability distribution as the predicted value for each coordinate point; The predicted values ​​for each coordinate point are summed up to obtain the predicted values ​​for all coordinate points. A second loss function is constructed based on the predicted values ​​and the true values ​​of all coordinate points.

[0015] Secondly, embodiments of this application provide a key information extraction device, including: The receiving module is used to receive the image of the document to be tested and the sequence of question texts for the image of the document to be tested. The acquisition module is used to acquire the target key information extraction model; The information extraction module is used to input the image of the document to be tested and the question text sequence of the image of the document to be tested into the target key information extraction model to generate the answer text sequence of the image of the document to be tested. The answer text sequence of the image of the document to be tested is obtained by querying the target tags generated by the target key information extraction model. The hidden state of the target tags implicitly encodes the physical location information of the answer in the image of the document to be tested.

[0016] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the methods described above.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the methods described above.

[0018] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the methods described above.

[0019] This application provides a method, apparatus, device, medium, and product for extracting key information. The method includes: first, receiving a test document image and a sequence of question texts related to the test document image; obtaining a target key information extraction model; then, inputting the test document image and the sequence of question texts related to the test document image into the target key information extraction model to generate an answer text sequence for the test document image. The answer text sequence for the test document image is obtained through a target tag query generated by the target key information extraction model. The hidden state of the target tag implicitly encodes the physical location information of the answer in the test document image. By introducing target tags, this application guides the target key information extraction model to first perceive the physical location information of the answer file sequence in the test document image. The answer is located using the physical location information, thereby obtaining the answer text sequence of the test document image, improving the accuracy and effectiveness of key information extraction. Attached Figure Description

[0020] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application. In the drawings: Figure 1 This is a flowchart illustrating a key information extraction method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a method for constructing a pre-trained dataset according to an embodiment of this application; Figure 3 This is a flowchart illustrating another method for extracting key target information provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a key information extraction device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the computer device provided in the embodiments of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0022] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein.

[0023] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0024] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0025] It should be understood that in this application, "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product or device.

[0026] It should be understood that in this application, "multiple" refers to two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, "and / or B" can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "Contains A, B, and C", "Contains A, B, and C" means that all three A, B, and C are contained; "Contains A, B, or C" means that one of A, B, and C is contained; "Contains A, B, and / or C" means that any one, two, or three of A, B, and C are contained.

[0027] It should be understood that in this application, "B corresponding to A", "B corresponding to A", "A corresponds to B", or "B corresponds to A" means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information. Matching A and B is defined as a similarity between A and B that is greater than or equal to a preset threshold.

[0028] Depending on the context, "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection."

[0029] The data involved in this application may be data authorized by the tester or fully authorized by all parties. The collection, dissemination, and use of the data shall comply with the relevant laws, regulations and standards of the relevant countries and regions. The implementation methods / executives of this application may be combined with each other.

[0030] The technical solutions of this application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0031] The present application will now be described in conjunction with the accompanying drawings and specific embodiments.

[0032] With the continuous advancement of information digitization and office automation, image documents have become an indispensable medium in many business processes such as enterprise management, financial reimbursement, and contract review. This trend has driven the urgent need for technologies that automatically extract key information from image documents, thus giving rise to the development of Key Information Extraction (KIE) technology. KIE technology aims to automatically identify and extract key entities such as company names, dates, amounts, and addresses from complex and visually rich document images, providing structured input for various information management systems.

[0033] Most existing KIE technologies rely on Optical Character Recognition (OCR) as an intermediate processing step. The OCR module converts images into parsable text and its location information, which is then combined with a text understanding model for entity recognition. While this OCR-based processing has achieved some success, its performance is heavily dependent on OCR quality and suffers from significant computational latency and error propagation issues. For example, in cases of low image quality or complex layouts, OCR output often contains typos, text truncation, or lost location information, leading to a significant drop in overall KIE performance.

[0034] To circumvent the drawbacks of OCR modules, a series of OCR-free end-to-end methods, such as the Donut model, have been proposed in recent years. These methods directly input images into the model and output structured text results. These approaches significantly improve inference efficiency and reduce reliance on external tools. However, these models generally suffer from a key drawback: they can only output plain text answers and cannot provide the specific location of the answer within the image, i.e., they lack the ability to provide visual grounding. This makes the models difficult to interpret, verify, or use for downstream tasks that require localization information, such as information highlighting and automatic annotation.

[0035] While OCR-free end-to-end methods eliminate the OCR step and improve processing efficiency, they generally lack the ability to provide visual evidence, failing to pinpoint the location of the answer within the image and impacting the model's interpretability. To incorporate visual information, some methods attempt to explicitly resolve positional coordinates; however, training heavily relies on precisely labeled coordinate data from downstream sources, which is difficult to obtain in real-world scenarios, resulting in poor model generalization and unstable application. Furthermore, explicit coordinate output is prone to error accumulation, further interfering with the quality of generated text.

[0036] In summary, while existing Key Information Extraction (KIE) methods have made some progress, they still have several shortcomings in practical applications, mainly reflected in the following aspects: Over-reliance on OCR modules leads to poor system robustness: Current mainstream KIE methods heavily rely on OCR engines to provide accurate text and location information. However, in scenarios with poor document image quality, complex layouts, or diverse languages, OCR is prone to character recognition errors and structural parsing failures. These errors can cascade and propagate to subsequent models, resulting in a significant drop in overall performance. Furthermore, the OCR module itself suffers from structural bottlenecks such as high inference latency and poor cross-domain adaptability, making it difficult to meet the demands for efficient and universal document understanding.

[0037] The lack of a stable and effective mechanism for providing visual evidence is a significant issue. While OCR-free models eliminate the OCR process and improve end-to-end modeling capabilities, they generally only output plain text answers, lacking a correspondence with image regions. This results in poor interpretability and uncontrollability, making it difficult to serve demanding downstream localization tasks. Some existing methods attempt to incorporate visual information through explicit coordinate regression, but these rely on precisely labeled data, are difficult to generalize, and are prone to error propagation.

[0038] To address the aforementioned issues, this application proposes a method for extracting key information.

[0039] Please see Figure 1 , Figure 1 This is a flowchart illustrating a key information extraction method provided in an embodiment of this application. Figure 1As shown, it includes the following steps: Step S101: Receive the image of the document to be tested and the sequence of question texts for the image of the document to be tested.

[0040] The user provides a document image to be tested, such as an image of an invoice that has never been seen before, and a sequence of text questions about the image, such as "What is the total amount of the invoice?".

[0041] Step S102: Obtain the target key information extraction model.

[0042] To obtain the target key information extraction model, a pre-training dataset needs to be constructed first, and then the initial key information extraction model is trained using the pre-training dataset to obtain the target key information extraction model.

[0043] The process of constructing a pre-training dataset includes: acquiring training document images; labeling the training document images to obtain the corresponding HTML code structure information; inputting the HTML code structure information into a pre-defined large language model to generate text features of the training document images, wherein the text features include question-answer pair sequences and the actual physical location information of the answers, wherein the question-answer pair sequence includes the question text sequence and the answer text sequence of the training document images, and the actual physical location information of the answers refers to the actual physical location of the answers in the initial document images; using the training document images and the corresponding text features as training samples; and constructing a pre-training dataset using a pre-defined number of training samples.

[0044] For example, such as Figure 2 As shown, a large number of table image samples can be collected from multiple public table recognition datasets as training document images. These images are all document images in real-world scenarios, such as financial statements, invoices, and contract forms, ensuring that the images have diverse layout structures and visual styles.

[0045] Then, each training document image, such as a table image, is preprocessed. For example, the answer is marked in the table image and its row and column are marked. The structured information (including structured tags) is extracted and converted into standard HTML code format, thereby obtaining the HTML code structure information corresponding to the training document image.

[0046] It should be noted that HTML code can accurately describe the logical structure of a table (such as row and column relationships, cell merging, etc.), providing a semantic basis for the subsequent generation of question-and-answer pairs.

[0047] Then, the HTML code structure information corresponding to the above training document images is input into a preset large language model (such as GPT-4), and with the designed prompt words, such as standardized prompt word templates, the corresponding text features are automatically generated, including the question-answer pair sequence and the actual physical location information of the answer.

[0048] Each question-answer pair in the question-answer pair sequence includes: Question text sequence: such as "Q1: What is the date of the notice of IFB as per the table?", "Q2: What bid number is associated with the procurement information in the table?", "Q3: What is the name of the project in the table?"; Answer text sequence: such as "A1: Auguet 30, 1999", "A2: CMC991508", "A3: Labor Market Development Project".

[0049] In addition, it also includes logical position information: the row and column position of the answer (e.g., L1: 2nd, 2nd (2nd row, 2nd column).

[0050] Once the logical location information is obtained, post-processing is performed. This involves tracing back to the corresponding polygon bounding box coordinates (usually four-point or eight-point coordinates) of the cell in the annotation of the training document image based on the logical location (row and column index) of the cell in the HTML code. Through this mapping relationship, the logical location is converted into the actual physical location information in the image, forming the spatial coordinate representation of the answer in the image. The row and column position of the answer is displayed in the table image, such as the cell corresponding to the thick line in the table image.

[0051] Finally, the training document images and their corresponding text features are used as training samples. The above steps are repeated to process a large number of training document images, generating a sufficient number of training samples. These samples are then integrated into a pre-training dataset (TVG).

[0052] The embodiments of this application obtain images and text content aligned in a pre-trained dataset, answers with visual location basis, and cover various table structures and question-and-answer types, exhibiting good diversity and generalization ability, providing a basis for subsequent key information extraction.

[0053] The process involves training an initial key information extraction model using a pre-trained dataset to obtain a target key information extraction model. This includes: obtaining the target loss function of the initial key information extraction model; using training document images and question text sequences from the pre-trained dataset as inputs to the initial key information extraction model, and using answer text sequences from the training document images in the pre-trained dataset as outputs to train the initial key information extraction model; and obtaining the target key information extraction model once the target loss function of the initial key information extraction model reaches its minimum value.

[0054] The process of obtaining the target loss function of the initial key information extraction model includes: obtaining a first loss function and a second loss function; and determining the target loss function of the initial key information extraction model based on the first loss function, the second loss function, and the balancing weight coefficients.

[0055] The process of obtaining the first loss function includes: obtaining visual features generated by transforming the training document image through the visual encoder in the initial key information extraction model; obtaining the answer text sequence at the current position output by the initial key information extraction model; and constructing the first loss function based on the visual features and the answer text sequence at the current position.

[0056] like Figure 3 As shown, the training document image is input into the visual encoder (such as the ViT-based encoder) of the initial key information extraction model (STNet). The visual encoder performs hierarchical feature extraction and transformation on the training document image, and finally outputs a visual feature that can comprehensively represent the visual information of the image. This visual feature captures all key visual cues in the image, such as text, layout, and font.

[0057] The initial key information extraction model receives a sequence of question texts (e.g., "What is the date of the notice of IFB as per the table?") as input, and the text decoder generates the sequence of answer texts step by step in an autoregressive manner. This means that when generating the i-th answer token, the initial key information extraction model relies on the following three parts of information: visual features extracted from the image, the input question sequence, and all previously generated answer token sequences (i.e., from the first to the previous token).

[0058] At each step of generating the answer token, the initial key information extraction model calculates a conditional probability distribution: .

[0059] This conditional probability distribution represents the probability that the current token is a word in the vocabulary, given all contextual information.

[0060] For each position i in the answer text sequence, calculate the negative log-likelihood between the conditional probability distribution predicted by the initial key information extraction model and the true answer token.

[0061] Finally, the loss values ​​at all positions are summed and averaged to obtain the average loss of the entire sequence, i.e., the first loss function, which is expressed as follows:

[0062] in, It is the actual length of the answer text sequence. It is the conditional probability distribution predicted by the initial key information extraction model for each position i in the answer text sequence.

[0063] The process of obtaining the second loss function includes: obtaining a first label generated by the initial key information extraction model, wherein the hidden state of the first label implicitly encodes the physical location information of the answer in the training document image; performing a linear transformation on the hidden state of the first label to obtain a query vector; using the query vector to search in a predefined coordinate vocabulary and obtaining the position probability distribution through a preset function; and constructing the second loss function based on the position probability distribution.

[0064] The process of obtaining the first tag generated by the initial key information extraction model includes: obtaining visual features by transforming the training document image through the visual encoder in the initial key information extraction model; transforming the question text sequence of the training document image to obtain question text features; fusing the visual features and question text features to obtain fused features; and decoding the fused features through the text decoder in the initial key information extraction model to generate the first tag.

[0065] The second loss function is constructed based on the location probability distribution, including: calculating the weighted expected value of the location probability distribution as the predicted value for each coordinate point; summing up the predicted values ​​for each coordinate point to obtain the predicted values ​​for all coordinate points; and constructing the second loss function based on the relationship between the predicted values ​​for all coordinate points and the true values ​​for all coordinate points.

[0066] like Figure 3As shown, firstly, the training document images are input into the visual encoder of the initial key information extraction model and transformed into a rich set of visual features that encode the visual content and structural information of the images. Simultaneously, the question text sequence (e.g., "What is the date of the notice of IFB as per the table?") is transformed into question text features through the embedding layer of the initial key information extraction model.

[0067] Subsequently, in the text decoder of the initial key information extraction model, visual features and problem text features are deeply fused to form fused features that include both visual and textual context.

[0068] The text decoder performs autoregressive decoding based on this fused feature. Before generating the answer text, the model first generates a special first tag, namely... <see>The tag implicitly encodes the physical location information of the image region corresponding to the current answer in its hidden state. This mechanism realizes the core idea of ​​"see first, then describe," that is, perceive the location first, and then generate the text.

[0069] To decode explicit coordinate information from implicit hidden states, the initial key information extraction model... <see>Hidden state of the marker Perform a series of operations: Hide state By applying a linear transformation layer, the vector is projected onto a new feature space to generate a query vector. This vector is used to query coordinate information. Its linear transformation formula is as follows:

[0070] Then, the initial key information extraction model maintains a predefined coordinate vocabulary. (This can be viewed as a lookup table containing 1000 learnable vectors), and the location probability distribution is obtained through softmax. The calculation formula is as follows:

[0071] Next, the weighted expected value of the obtained position probability distribution is calculated. This is used as the predicted value for each coordinate point. The specific formula is as follows:

[0072] Since the physical location of an answer region is represented by a polygonal bounding box containing 8 coordinate values ​​(i.e., the x and y coordinates of 4 points), the 8 predicted coordinate values ​​are compared with the 8 true coordinate values ​​provided in the training samples. This means comparing the predicted values ​​with all coordinate points. Compared with the true value The mean squared error (MSE) between the two losses is used to construct the second loss function, and the specific calculation formula is as follows:

[0073] After obtaining the first loss function and the second loss function, they can be added together to obtain the target loss function of the initial key information extraction model. The specific formula is as follows:

[0074] in, These are the balancing weighting coefficients.

[0075] After obtaining the target loss function of the initial key information extraction model, the training document images and question text sequences from the pre-training dataset are used as inputs to the initial key information extraction model, and the answer text sequences from the training document images in the pre-training dataset are used as outputs. The initial key information extraction model is trained until its target loss function reaches its minimum value, at which point the model is considered converged and training is complete. The trained model parameters are then saved, resulting in the final target key information extraction model.

[0076] Step S103: Input the image of the document to be tested and the question text sequence of the image of the document to be tested into the target key information extraction model to generate the answer text sequence of the image of the document to be tested. The answer text sequence of the image of the document to be tested is obtained by querying the target marker generated by the target key information extraction model. The hidden state of the target marker implicitly encodes the physical location information of the answer in the image of the document to be tested.

[0077] The user provides a test document image, such as an image of an invoice that has never been seen before, and a sequence of question texts about the image, such as "What is the date of the notice of IFB as per the table?", which is then input into the target key information extraction model.

[0078] The visual encoder of the target key information extraction model processes the image to extract its rich visual features. The decoder of the target key information extraction model then begins inference and generation in an autoregressive manner. This process strictly follows the "see first, then describe" pattern learned during training: generating target labels (…). <see>(Tag): The decoder doesn't initially generate a specific answer text, but rather that special... <see>The tag. The hidden state of this tag implicitly encodes the physical location information of the image region that the model "sees" and that is related to the answer. This means that the model has already located the region on the image before outputting any text.

[0079] exist <see>After the tokens are marked, the decoder continues to generate subsequent tokens in an autoregressive manner, gradually outputting the complete answer text sequence, such as "Auguet 30,1999". This answer generation process is influenced by... <see>The tags guide and constrain the visual perception information encoded by them, thus resulting in higher accuracy.

[0080] The direct output of the target key information extraction model is a sequence of answer text, providing the user with the necessary key information. Simultaneously, while the model does not directly output coordinates, the physical location information of the answer within the image is fully encoded. <see>The marker is in a hidden state.

[0081] If users need to verify the correctness of answers or perform downstream location tasks, such as highlighting the answer area on an image, a simple physical decoder (a set of linear transformation layers and a coordinate vocabulary) that has been learned during the training phase can be used. <see>By parsing the hidden state of the markers, the coordinates of the polygon bounding box where the answer is located can be recovered, thus enabling the visualization of the visual evidence.

[0082] This application provides a key information extraction method, including: first, receiving a test document image and a question text sequence of the test document image; obtaining a target key information extraction model; then, inputting the test document image and the question text sequence of the test document image into the target key information extraction model to generate an answer text sequence of the test document image. The answer text sequence of the test document image is obtained through target tag query generated by the target key information extraction model, and the hidden state of the target tag implicitly encodes the physical location information of the answer in the test document image. By introducing target tags, this application guides the target key information extraction model to first perceive the physical location information of the answer file sequence in the test document image, thereby locating the answer through the physical location information and obtaining the answer text sequence of the test document image, thus improving the accuracy and effectiveness of key information extraction.

[0083] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0084] The following are device embodiments of this application. For details not described in detail, please refer to the corresponding method embodiments described above.

[0085] Figure 4 This diagram illustrates a key information extraction device according to an embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. The key information extraction device includes: The receiving module 401 is used to receive the image of the document to be tested and the sequence of question texts for the image of the document to be tested; Module 402 is used to acquire the target key information extraction model; The information extraction module 403 is used to input the image of the document to be tested and the question text sequence of the image of the document to be tested into the target key information extraction model to generate the answer text sequence of the image of the document to be tested. The answer text sequence of the image of the document to be tested is obtained by querying the target marker generated by the target key information extraction model. The hidden state of the target marker implicitly encodes the physical location information of the answer in the image of the document to be tested.

[0086] In one embodiment, the acquisition module 402 is further configured to construct a pre-trained dataset; The initial key information extraction model is trained using a pre-trained dataset to obtain the target key information extraction model.

[0087] In one embodiment, the acquisition module 402 is further configured to acquire training document images; The training document images are labeled to obtain the HTML code structure information corresponding to the training document images; The HTML code structure information is input into a pre-defined large language model to generate text features of the training document image. The text features include question-answer pair sequences and actual physical location information of the answers. The question-answer pair sequences include the question text sequence of the training document image and the answer text sequence of the training document image. The actual physical location information of the answers refers to the actual physical location of the answers in the initial document image. Use the training document images and the corresponding text features of the training document images as training samples; A pre-training dataset is constructed by using a predetermined number of training samples.

[0088] In one embodiment, the acquisition module 402 is further configured to acquire the target loss function of the initial key information extraction model; The initial key information extraction model is trained by using the training document images and the question text sequences of the training document images in the pre-training dataset as inputs and the answer text sequences of the training document images in the pre-training dataset as outputs. Once the target loss function of the initial key information extraction model reaches its minimum value, the target key information extraction model is obtained.

[0089] In one embodiment, the acquisition module 402 is further configured to acquire a first loss function and a second loss function; Based on the first loss function, the second loss function, and the balancing weight coefficient, the target loss function of the initial key information extraction model is determined.

[0090] In one embodiment, the acquisition module 402 is further configured to acquire visual features generated by transforming the training document image through the visual encoder in the initial key information extraction model; Obtain the initial key information and extract the answer text sequence of the current position output by the model; The first loss function is constructed based on the visual features and the answer text sequence of the current location.

[0091] In one embodiment, the acquisition module 402 is further configured to acquire a first tag generated by the initial key information extraction model, wherein the hidden state of the first tag implicitly encodes the physical location information of the answer in the training document image; A linear transformation is performed on the hidden state of the first marker to obtain the query vector; The query vector is used to search a predefined coordinate vocabulary, and the position probability distribution is obtained through a preset function. A second loss function is constructed based on the location probability distribution.

[0092] In one embodiment, the acquisition module 402 is further configured to acquire visual features generated by transforming the training document image through the visual encoder in the initial key information extraction model; The question text sequence of the training document image is transformed to obtain question text features; By fusing visual features and question text features, a fused feature is obtained; The fused features are decoded by the text decoder in the initial key information extraction model to generate the first label.

[0093] In one embodiment, the acquisition module 402 is further configured to calculate the weighted expected value of the location probability distribution as the predicted value for each coordinate point; The predicted values ​​for each coordinate point are summed up to obtain the predicted values ​​for all coordinate points. A second loss function is constructed based on the predicted values ​​and the true values ​​of all coordinate points.

[0094] This application provides a key information extraction device, specifically used for: first receiving a test document image and a question text sequence of the test document image, obtaining a target key information extraction model, then inputting the test document image and the question text sequence of the test document image into the target key information extraction model to generate an answer text sequence of the test document image. The answer text sequence of the test document image is obtained through a target tag query generated by the target key information extraction model, and the hidden state of the target tag implicitly encodes the physical location information of the answer in the test document image. This application introduces target tags to guide the target key information extraction model to first perceive the physical location information of the answer file sequence in the test document image, thereby locating the answer through the physical location information and obtaining the answer text sequence of the test document image, thus improving the accuracy and effectiveness of key information extraction.

[0095] This application Figure 5 A schematic diagram of a computer device is provided. (Example) Figure 5 As shown, the computer device 5 in this embodiment includes a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, it implements the steps in the various key information extraction method embodiments described above, for example... Figure 1 Steps 101 to 103 are shown. Alternatively, when processor 501 executes computer program 503, it implements the functions of each module / unit in the above-described embodiments of the key information extraction device, for example... Figure 4 The functions of modules / units 401 to 403 are shown.

[0096] This application also provides a readable storage medium storing a computer program, which, when executed by a processor, is used to implement the key information extraction method provided in the various embodiments described above.

[0097] The readable storage medium can be a computer storage medium or a communication medium. A communication medium includes any medium that facilitates the transfer of computer programs from one location to another. A computer storage medium can be any available medium accessible to a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application-Specific Integrated Circuit (ASIC). Alternatively, the ASIC can be located in a user equipment. Of course, the processor and the readable storage medium can also exist as discrete components in a communication device. The readable storage medium can be a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0098] This application also provides a program product including executable instructions stored in a readable storage medium. At least one processor of the device can read the executable instructions from the readable storage medium, and the execution of the executable instructions by the at least one processor causes the device to implement the key information extraction methods provided in the various embodiments described above.

[0099] In the embodiments of the above-described device, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0100] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.< / see> < / see> < / see> < / see> < / see> < / see> < / see> < / see>

Claims

1. A method for extracting key information, characterized in that, include: Receive an image of a document to be tested and a sequence of question texts related to the document image; Model for extracting key target information; The image of the document to be tested and the question text sequence of the image of the document to be tested are input into the target key information extraction model to generate the answer text sequence of the image of the document to be tested. The answer text sequence of the image of the document to be tested is obtained by target tag query generated by the target key information extraction model. The hidden state of the target tag implicitly encodes the physical location information of the answer in the image of the document to be tested.

2. The key information extraction method as described in claim 1, characterized in that, The target key information extraction model includes: Construct a pre-trained dataset; The initial key information extraction model is trained using the pre-trained dataset to obtain the target key information extraction model.

3. The key information extraction method as described in claim 2, characterized in that, The construction of the pre-trained dataset includes: Obtain training document images; The training document images are labeled to obtain the HTML code structure information corresponding to the training document images; The HTML code structure information is input into a preset large language model to generate text features of the training document image. The text features include a question-answer pair sequence and the actual physical location information of the answer. The question-answer pair sequence includes a question text sequence of the training document image and an answer text sequence of the training document image. The actual physical location information of the answer refers to the actual physical location of the answer in the initial document image. The training document image and the text features corresponding to the training document image are used as training samples; The pre-training dataset is constructed using a preset number of training samples.

4. The key information extraction method as described in claim 2, characterized in that, The step of training the initial key information extraction model using the pre-trained dataset to obtain the target key information extraction model includes: Obtain the target loss function of the initial key information extraction model; The initial key information extraction model is trained by using the training document images and the question text sequences of the training document images in the pre-training dataset as inputs and the answer text sequences of the training document images in the pre-training dataset as outputs. Once the target loss function of the initial key information extraction model reaches its minimum value, the target key information extraction model is obtained.

5. The key information extraction method as described in claim 4, characterized in that, The objective loss function for obtaining the initial key information extraction model includes: Obtain the first loss function and the second loss function; Based on the first loss function, the second loss function, and the balancing weight coefficient, the target loss function of the initial key information extraction model is determined.

6. The key information extraction method as described in claim 5, characterized in that, The process of obtaining the first loss function includes: The visual encoder in the initial key information extraction model is used to transform the training document image to generate visual features; Obtain the answer text sequence for the current position output by the initial key information extraction model; The first loss function is constructed based on the visual features and the answer text sequence at the current location.

7. The key information extraction method as described in claim 5, characterized in that, The process of obtaining the second loss function includes: Obtain the first tag generated by the initial key information extraction model, wherein the hidden state of the first tag implicitly encodes the physical location information of the answer in the training document image; A linear transformation is performed on the hidden state of the first marker to obtain the query vector; The query vector is used to retrieve data from a predefined coordinate vocabulary, and the position probability distribution is obtained through a preset function. The second loss function is constructed based on the location probability distribution.

8. The key information extraction method as described in claim 7, characterized in that, The step of obtaining the first tag generated by the initial key information extraction model includes: The visual encoder in the initial key information extraction model is used to transform the training document image to generate visual features; The question text sequence of the training document image is transformed to obtain question text features; The visual features and the question text features are fused to obtain the fused features; The fused features are decoded using the text decoder in the initial key information extraction model to generate the first tag.

9. The key information extraction method as described in claim 7, characterized in that, The construction of the second loss function based on the location probability distribution includes: Calculate the weighted expected value of the location probability distribution as the predicted value for each coordinate point; The predicted values ​​for each coordinate point are summed to obtain the predicted values ​​for all coordinate points. The second loss function is constructed based on the predicted values ​​and the true values ​​of all coordinate points.

10. A key information extraction device, characterized in that, include: The receiving module is used to receive the image of the document to be tested and the sequence of question texts of the image of the document to be tested; The acquisition module is used to acquire the target key information extraction model; The information extraction module is used to input the image of the document to be tested and the question text sequence of the image of the document to be tested into the target key information extraction model to generate the answer text sequence of the image of the document to be tested. The answer text sequence of the image of the document to be tested is obtained by querying the target marker generated by the target key information extraction model. The hidden state of the target marker implicitly encodes the physical location information of the answer in the image of the document to be tested.

11. A computer device, characterized in that, Includes a memory, and one or more processors communicatively connected to the memory; The memory stores instructions that can be executed by the one or more processors, which, when executed, enable the one or more processors to implement the key information extraction method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, Includes a program or instructions that, when run on a computer, implement the key information extraction method according to any one of claims 1 to 9.

13. A computer program product, characterized in that, The method includes a computer program that, when executed by a processor, implements the key information extraction method according to any one of claims 1 to 9.

Images