Bill information extraction method and device and electronic equipment

Abstract

The application discloses a kind of bill information extraction method, device and electronic equipment.Therein, the method includes: obtaining target bill image, wherein, target bill image includes multiple text regions, and multiple text regions include corresponding text parameters respectively;Determine the spatial parameters corresponding to multiple text regions respectively, and determine the association between multiple text regions;According to the text parameters and spatial parameters corresponding to multiple text regions respectively, and the association between multiple text regions, determine the target bill information corresponding to target bill image.The present application solves the technical problem that the bill information cannot be extracted efficiently and accurately in related technologies when extracting bill information.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more specifically, to a method, apparatus, and electronic device for extracting invoice information. Background Technology

[0002] In related technologies, due to the significant differences in format and field content among different types of invoices, traditional OCR technology often struggles to effectively handle the diverse invoice types. Therefore, when extracting invoice information, the diversity of invoice types and other issues result in the inability to extract invoice information efficiently and accurately.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides a method, apparatus, and electronic device for extracting invoice information, which at least solves the technical problem in the related art that in extracting invoice information, it is impossible to extract invoice information efficiently and accurately.

[0005] According to one aspect of the present invention, a method for extracting invoice information is provided, comprising: acquiring a target invoice image, wherein the target invoice image includes a plurality of text regions, each of the plurality of text regions including corresponding text parameters; determining spatial parameters corresponding to the plurality of text regions respectively, and determining the association relationship between the plurality of text regions; and determining target invoice information corresponding to the target invoice image based on the text parameters and spatial parameters corresponding to the plurality of text regions respectively, and the association relationship between the plurality of text regions.

[0006] Optionally, based on the text parameters and spatial parameters corresponding to the plurality of text regions, and the correlation between the plurality of text regions, the target ticket information corresponding to the target ticket image is determined, including: determining the fusion feature vector corresponding to each of the plurality of text regions based on the text parameters and spatial parameters corresponding to each of the plurality of text regions, and the correlation between the plurality of text regions; determining the weighted graph corresponding to the target ticket image based on the fusion feature vector corresponding to each of the plurality of text regions, wherein the weighted graph includes nodes corresponding to key features and nodes corresponding to non-key features, wherein the key features are features whose corresponding importance index is greater than a predetermined importance threshold, and the non-key features are features whose corresponding importance index is less than or equal to the predetermined importance threshold; determining the nonlinear feature matrix corresponding to the target ticket image based on the weighted graph corresponding to the target ticket image, wherein the nonlinear feature matrix is ​​a matrix obtained by weighting the key features and reducing the weight of the non-key features; determining the target classification label corresponding to each of the plurality of text regions based on the nonlinear feature matrix corresponding to the target ticket image; and determining the target ticket information corresponding to the target ticket image based on the text parameters and target classification labels corresponding to each of the plurality of text regions.

[0007] Optionally, determining the weighted graph corresponding to the target ticket image based on the fusion feature vectors corresponding to the plurality of text regions includes: determining the nodes corresponding to the plurality of fusion feature vectors to obtain a node set; determining the association weight value between each node in the node set and other nodes based on the association relationship between each node in the node set and other nodes to obtain an association weight set; determining the neighbor nodes corresponding to each node in the node set based on the association weight set to obtain an edge set; and constructing the weighted graph based on the node set, the association weight set, and the edge set.

[0008] Optionally, determining the nonlinear feature matrix corresponding to the target ticket image based on the weighted graph corresponding to the target ticket image includes: determining the adjacency matrix and node feature matrix corresponding to the weighted graph; inputting the adjacency matrix and node feature matrix into a linear transformation layer to obtain a linear feature matrix, wherein the linear transformation layer includes a predetermined weight matrix; and inputting the linear feature matrix into an activation function to obtain a nonlinear feature matrix.

[0009] Optionally, obtaining the target invoice image includes: determining an initial invoice image; determining, based on the initial invoice image, whether the initial invoice image needs repair; if the determination result is that the initial invoice image needs repair, determining a defective sub-image and adjacent sub-images corresponding to the initial invoice image, wherein the adjacent sub-images are sub-images in the initial invoice image whose proximity to the defective sub-image is greater than a proximity threshold; and repairing the defective sub-images based on the adjacent sub-images to obtain the target invoice image.

[0010] Optionally, the step of repairing the defective sub-image based on the adjacent sub-images to obtain the target ticket image includes: repairing the defective sub-image based on the adjacent sub-images to obtain a repaired ticket image; determining a physical degradation function, wherein the physical degradation function is a function used for defect simulation of the ticket image; inputting the repaired ticket image into the physical degradation function to obtain a degraded ticket image; determining a similarity index between the degraded ticket image and the initial ticket image; and determining the repaired ticket image as the target ticket image if the similarity index is greater than a similarity threshold.

[0011] Optionally, determining the fusion feature vector corresponding to each of the multiple text regions based on the text parameters and spatial parameters corresponding to each of the multiple text regions, as well as the correlation between the multiple text regions, includes: determining the spatial feature vector corresponding to each of the multiple text regions based on the spatial parameters corresponding to each of the multiple text regions; determining the text feature vector corresponding to each of the multiple text regions based on the text parameters corresponding to each of the multiple text regions; determining the style parameters corresponding to each of the multiple text regions; determining the visual feature vector corresponding to each of the multiple text regions based on the style parameters corresponding to each of the multiple text regions; and determining the fusion feature vector corresponding to each of the multiple text regions based on the text feature vector, the visual feature vector, and the spatial feature vector.

[0012] According to one aspect of the present invention, a bill information extraction device is provided, comprising: an acquisition module for acquiring a target bill image, wherein the target bill image includes a plurality of text regions, each of the plurality of text regions including corresponding text parameters; a first determination module for determining spatial parameters corresponding to the plurality of text regions and determining the association relationship between the plurality of text regions; and a second determination module for determining target bill information corresponding to the target bill image based on the text parameters and spatial parameters corresponding to the plurality of text regions and the association relationship between the plurality of text regions.

[0013] According to one aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the invoice information extraction method described in any of the preceding embodiments.

[0014] According to one aspect of the present invention, a computer-readable storage medium is provided, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the ticket information extraction method described in any of the preceding claims.

[0015] In this embodiment of the invention, a target invoice image is acquired, wherein the target invoice image includes multiple text regions, each of which includes corresponding text parameters; spatial parameters corresponding to each of the multiple text regions are determined, and the correlation between the multiple text regions is determined; based on the text parameters and spatial parameters corresponding to each of the multiple text regions, as well as the correlation between the multiple text regions, target invoice information corresponding to the target invoice image is determined. By acquiring a target invoice image including multiple text regions, each of which includes corresponding text parameters, an accurate data foundation is provided for the subsequent invoice information extraction process, thereby facilitating the accurate location of key information in the target invoice image based on its text regions. By determining the spatial parameters of each of the multiple text regions in the target invoice image, it is helpful to accurately identify and locate key information in the invoice, and by determining the correlation between these text regions, it is helpful to accurately understand the logical connections between the text regions, thereby enabling accurate identification and parsing of invoice information. By utilizing textual and spatial parameters extracted from text regions, it is possible to achieve targeted and efficient information extraction. By combining the correlation between text regions, the target ticket information corresponding to the target ticket image can be determined. This realizes the conversion of unstructured information in the ticket image into structured data, reduces misidentification in the ticket information extraction process, and improves the efficiency and accuracy of ticket information extraction and processing. In this way, it solves the technical problem in related technologies that it is impossible to extract ticket information efficiently and accurately. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0017] Figure 1 This is a flowchart of a method for extracting invoice information according to an embodiment of the present invention;

[0018] Figure 2 This is a flowchart of a method for extracting invoice information in an optional embodiment of the present invention;

[0019] Figure 3 This is a technical framework diagram of image restoration in an optional embodiment of the present invention;

[0020] Figure 4 This is a flowchart of image restoration in an optional embodiment of the present invention;

[0021] Figure 5 This is a structural block diagram of a ticket information extraction device according to an embodiment of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0025] OCR: OCR (Optical Character Recognition) is a technology that converts text in an image into editable and searchable text. OCR technology can be used in document scanning, image processing, automated data entry, and other fields.

[0026] CRNN: CRNN (Convolutional Recurrent Neural Network) is a deep learning model that combines Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). It is commonly used for sequence generation tasks, such as text recognition and image caption generation. CRNNs can capture local features in images (through CNNs) and process sequence data (through RNNs), thereby achieving accurate text recognition in images.

[0027] Text-Perceptron-Det: Text-Perceptron-Det is a text detector in OCR (Optical Character Recognition). Its main function is to detect text regions in images and extract text information from them. It can learn and recognize text features in images using deep learning techniques, such as convolutional neural networks (CNNs).

[0028] Word2Vec is a word embedding technique for natural language processing. It maps each word in the vocabulary to a fixed-size vector space, ensuring that semantically similar words are also close in distance within the vector space. Word2Vec can capture the semantic relationships between words, providing effective feature representations for natural language processing tasks.

[0029] GCN: GCN (Graph Convolutional Network) is a type of neural network model specifically designed for processing graph-structured data. By applying convolutional operations to graph-structured data, GCN can capture local and global features between nodes, thereby enabling tasks such as classification and regression of graph-structured data.

[0030] GNN: GNN (Graph Neural Network) is a type of neural network model specifically designed for processing graph-structured data. GNN captures information from the graph structure by iteratively updating the representations of nodes, thereby enabling tasks such as classification, regression, and link prediction for graph-structured data.

[0031] GAN: GAN (Generative Adversarial Network) is a deep learning model consisting of two networks: a generator and a discriminator. The generator is responsible for producing realistic data samples, while the discriminator is responsible for distinguishing between real and generated data. Through continuous adversarial training between the two networks, GAN can generate increasingly realistic data samples and can be applied to fields such as image generation and speech synthesis.

[0032] PyTorch: PyTorch is an open-source deep learning framework that provides all the basic building blocks needed to build and train various neural networks, such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), etc.

[0033] TensorFlow: TensorFlow is an open-source machine learning framework that provides a flexible architecture for defining complex data flow graphs and executing efficiently on multiple platforms. TensorFlow supports GPU and TPU accelerated computation and provides various APIs, such as the Keras high-level interface, to facilitate developers in quickly building models.

[0034] Softmax classifier: The softmax classifier is a commonly used multi-class classification model. It supports multi-class tasks by mapping input features to probability distributions of multiple classes. Softmax classifiers are typically used in the output layer of neural networks to transform the network's output into a probability distribution, thereby enabling class prediction.

[0035] pix2pix: pix2pix is ​​an image transformation model based on Conditional Generative Adversarial Networks (GANs), designed to transform one image into another. It is suitable for image-to-image transformation tasks. pix2pix provides an input image, allowing the generator to learn how to generate an output image with specific target characteristics. A discriminator then distinguishes the generated image from the real target image.

[0036] Cycle GAN: Cycle GAN (Cycle-Consistent Generative Adversarial Network) is a deep learning model for unsupervised image translation, a variant of Generative Adversarial Networks (GANs) for image-to-image translation tasks. Cycle GAN can transform images of one style into images of another without paired training samples.

[0037] VRD: Visual Relationship Detection is a computer vision task designed to identify visual relationships between objects in an image, such as "a person riding a bicycle" or "a bird perched on a tree." VRD technology can be applied to image understanding, scene analysis, and other fields.

[0038] F1 Score: The F1 score is a commonly used evaluation metric in classification tasks. It is the harmonic mean of precision and recall, used to comprehensively evaluate the model's performance. The F1 score ranges from 0 to 1, with higher values ​​indicating better model performance. The F1 score can be applied to both binary and multi-class classification tasks.

[0039] Example 1

[0040] According to an embodiment of the present invention, an embodiment of a method for extracting invoice information is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0041] Figure 1 This is a flowchart of a bill information extraction method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0042] S102, Obtain the target ticket image, wherein the target ticket image includes multiple text regions, and each of the multiple text regions includes corresponding text parameters;

[0043] In step S102 of this application, the target ticket image is obtained.

[0044] This involves the target document image, which is a scanned or photographed document image from which information needs to be extracted. This target document image can be any type of financial document, such as invoices, receipts, train tickets, flight itineraries, taxi receipts, toll receipts, passenger transport invoices, etc.

[0045] This involves text areas, which are the areas corresponding to the text content on the target invoice image. The target invoice image contains multiple text areas, such as the text areas for "invoice code", "invoice number", "date", and "amount" on an invoice.

[0046] This involves text parameters, which are specific parameters describing the text content within a text area. For example, the text content in a text area might be "invoice number" or "unit price".

[0047] By acquiring a target invoice image containing multiple text regions, each region containing corresponding text parameters, an accurate data foundation is provided for the subsequent invoice information extraction process. This helps to accurately locate key information in the target invoice image based on its text regions, thereby reducing misidentification during the information extraction process, improving the efficiency and accuracy of invoice information extraction, and helping to solve the technical problem of inefficient and inaccurate extraction of invoice information.

[0048] It should be noted that there are no specific restrictions on the method for obtaining the target ticket image; it can be customized according to the actual application and scenario.

[0049] S104, determine the spatial parameters corresponding to multiple text regions respectively, and determine the association relationship between multiple text regions;

[0050] In step S104 of this application, spatial parameters corresponding to multiple text regions are determined, and the relationship between multiple text regions is determined.

[0051] This involves spatial parameters, which are the parameters corresponding to the geometric position of the text region in the target document image. These spatial parameters can include the bounding box coordinates of the text region (i.e., the coordinates of the top-left and bottom-right corners), the center point of the text region, the size of the text region (width and height), and the orientation or tilt angle of the text region. For example, for a text region containing "invoice number," its spatial parameters could be the top-left corner coordinates (120, 50) and the bottom-right corner coordinates (300, 100), representing the specific position and size of the text region in the document image.

[0052] This involves relationships, which are the connections between multiple text regions in a target invoice image. These relationships can be based on visual layout, such as the proximity of text regions, the vertical or horizontal alignment of text lines, or the nesting of text regions (e.g., one text region may be contained within another). They can also be based on semantics, such as certain text regions logically accompanying other specific text regions (e.g., "amount" is usually adjacent to "currency"). For example, in an invoice, the "invoice number" text region may be closely adjacent to the "invoice code" text region in terms of both location and semantics.

[0053] Determining the spatial parameters (such as coordinates and dimensions) of multiple text regions within a target document image facilitates accurate identification and location of key information within the document, even in complex layouts. Identifying the relationships between these text regions, including visual proximity and semantic relevance, helps in accurately understanding their logical connections, enabling accurate identification and parsing of document information, particularly for non-standard or complex document formats. By determining the spatial parameters and relationships of multiple text regions, accurate document information identification and extraction can be achieved, reducing false positives and false negatives, while also handling various document formats, thus improving the method's versatility and applicability.

[0054] It should be noted that there are no specific restrictions on the methods for determining the spatial parameters corresponding to multiple text regions and the relationships between multiple text regions. These can be customized according to the actual application and scenario.

[0055] S106, Based on the text parameters and spatial parameters corresponding to multiple text regions, and the correlation between multiple text regions, determine the target ticket information corresponding to the target ticket image.

[0056] In step S106 of this application, target ticket information corresponding to the target ticket image is determined.

[0057] This involves target invoice information, which consists of key data or information extracted from the image of the target invoice. This target invoice information is the main content of the invoice and is used to prove the legality of transactions, payments, and other related actions. For example, the extracted target invoice information may include "Invoice Code: 1234567890", "Invoice Number: 0987654321", "Amount: 234.56 yuan", "Date: April 1, 2023", etc.

[0058] By utilizing textual and spatial parameters extracted from text regions, it is possible to achieve targeted and efficient information extraction. By combining the correlation between text regions, the target ticket information corresponding to the target ticket image can be determined, improving the accuracy of information extraction, reducing false alarms and false negatives, and realizing the conversion of unstructured information in ticket images into structured data. This solves the technical problem of being unable to extract ticket information efficiently and accurately.

[0059] It should be noted that the method for determining the target ticket information corresponding to the target ticket image based on the text parameters and spatial parameters corresponding to multiple text regions, as well as the correlation between multiple text regions, is not specifically limited here and can be customized according to the actual application and scenario.

[0060] Through steps S102-S106 above, a target invoice image is obtained, wherein the target invoice image includes multiple text regions, each containing corresponding text parameters; the spatial parameters corresponding to each of the multiple text regions are determined, and the relationships between the multiple text regions are determined; based on the text parameters and spatial parameters corresponding to each of the multiple text regions, as well as the relationships between the multiple text regions, the target invoice information corresponding to the target invoice image is determined. By obtaining a target invoice image containing multiple text regions, each containing corresponding text parameters, an accurate data foundation is provided for the subsequent invoice information extraction process, thereby facilitating the accurate location of key information in the target invoice image based on its text regions. Determining the spatial parameters of each of the multiple text regions in the target invoice image helps to accurately identify and locate key information in the invoice, and determining the relationships between these text regions helps to accurately understand the logical connections between them, thereby enabling accurate identification and parsing of invoice information. By utilizing textual and spatial parameters extracted from text regions, it is possible to achieve targeted and efficient information extraction. By combining the correlation between text regions, the target ticket information corresponding to the target ticket image can be determined. This realizes the conversion of unstructured information in the ticket image into structured data, reduces misidentification in the ticket information extraction process, and improves the efficiency and accuracy of ticket information extraction and processing. In this way, it solves the technical problem in related technologies that it is impossible to extract ticket information efficiently and accurately.

[0061] As an optional embodiment, the target document information corresponding to the target document image is determined based on the text parameters and spatial parameters corresponding to multiple text regions, as well as the correlation between the multiple text regions. This includes: determining fusion feature vectors corresponding to each of the multiple text regions based on the text parameters and spatial parameters corresponding to each of the multiple text regions, as well as the correlation between the multiple text regions; determining a weighted graph corresponding to the target document image based on the fusion feature vectors corresponding to each of the multiple text regions, wherein the weighted graph includes nodes corresponding to key features and nodes corresponding to non-key features, where key features are features whose corresponding importance index is greater than a predetermined importance threshold, and non-key features are features whose corresponding importance index is less than or equal to the predetermined importance threshold; determining a nonlinear feature matrix corresponding to the target document image based on the weighted graph corresponding to the target document image, wherein the nonlinear feature matrix is ​​a matrix obtained by weighting key features and reducing the weight of non-key features; determining target classification labels corresponding to each of the multiple text regions based on the nonlinear feature matrix corresponding to the target document image; and determining the target document information corresponding to the target document image based on the text parameters and target classification labels corresponding to each of the multiple text regions.

[0062] This embodiment describes the specific steps for determining the target ticket information corresponding to the target ticket image based on the text parameters and spatial parameters corresponding to multiple text regions, as well as the correlation between the multiple text regions.

[0063] This involves a fused feature vector, which is a vector representing the comprehensive features of a text region. It can be a high-dimensional vector obtained by fusing text parameters, spatial parameters, style parameters, etc. For example, the fused feature vector of a "amount" text region may include the amount (text parameter), the position of the text region in the image (spatial parameter), and the image features of the text region (such as font, color, texture, etc.), forming a vector that can comprehensively describe the region.

[0064] This involves a weighted graph, which is used to represent the relationship between different text regions in a ticket image. In this weighted graph, the nodes represent text regions, and the weights of the edges are determined by the relationships between the text regions.

[0065] This involves key features that significantly contribute to identification and classification during the extraction of invoice information. For example, fields such as "amount" and "date" are key features for invoice information extraction because they directly relate to the validity of the invoice.

[0066] This includes non-critical features, which are features that are relatively minor in terms of identification and classification during the extraction of invoice information. For example, advertising text or decorative patterns on invoices are usually considered non-critical features because they do not directly participate in the extraction of key information.

[0067] This involves nodes, which represent text regions in a ticket image. Each node carries a fused feature vector to describe the comprehensive information of the text region.

[0068] This includes an importance index, which is a quantitative index used to measure the importance of features corresponding to a text region in information extraction and classification tasks. For example, the higher the importance index of a feature in a text region, the more important that feature is in analyzing and extracting invoice information.

[0069] This involves a predetermined importance threshold, which is a pre-set threshold used to distinguish between key and non-key features. For example, if the importance index is higher than the predetermined importance threshold, the feature is considered a key feature; if it is lower than or equal to the predetermined importance threshold, it is considered a non-key feature.

[0070] This involves a nonlinear feature matrix, which is a feature matrix obtained after processing with an activation function and is used to capture the complex relationships between features. This nonlinear feature matrix is ​​obtained by weighting key features and reducing the weight of non-key features, reflecting the nonlinear relationships and importance of features at each node in the weighted graph.

[0071] This involves weighted processing, which involves adding weights to key features to reflect their importance in extracting invoice information.

[0072] This involves a weight reduction process, which reduces the weight of non-critical features. This reduces the influence of non-critical features, ensuring that the analysis focuses on the key features directly related to the invoice information.

[0073] This involves target classification labels, which are category labels that each text region should be classified into, such as "invoice code", "invoice number", "amount", "date", etc., used to convert the text regions in the target invoice image into structured information.

[0074] In the steps involved in this embodiment, firstly, a fusion feature vector corresponding to each text region is determined. This fusion feature vector comprehensively considers the text parameters and spatial parameters corresponding to the text region, as well as the correlation relationship with other text regions. Next, based on these fusion feature vectors, a weighted graph is constructed. Nodes in the graph represent text regions, and the edge weights reflect the correlation strength between text regions. When constructing the weighted graph, key features and non-key features are distinguished. Key features correspond to features with an importance index higher than a predetermined importance threshold, while non-key features correspond to features with an importance index lower than or equal to the threshold. Then, the weighted graph is processed to obtain a non-linear feature matrix. This non-linear feature matrix performs weighting on key features and weighting on non-key features. Finally, based on the non-linear feature matrix, the target classification label corresponding to each text region is determined, thereby determining the target ticket information corresponding to the target ticket image.

[0075] By comprehensively considering the textual and spatial parameters corresponding to multiple text regions, as well as the correlations between these regions, fusion feature vectors corresponding to each text region are determined. This helps to accurately identify the true textual information corresponding to multiple text regions and reduce misidentification during the document information extraction process. Based on these fusion feature vectors, a weighted graph is constructed, where nodes represent text regions and edge weights reflect the correlation strength between text regions. In constructing the weighted graph, key and non-key features are distinguished, addressing the issue of varying importance of different features during information extraction. Through weighting and deweighting processes, the prominence of key features is ensured, improving the accuracy of information extraction. Processing the weighted graph yields a nonlinear feature matrix, which can more accurately identify and classify key information in documents, reducing errors in the information extraction process and contributing to improved comprehensiveness of document information extraction. Based on the nonlinear feature matrix, the target classification label corresponding to each text region is determined, thereby identifying the target document information corresponding to the target document image. This effectively solves the problem of converting unstructured information in document images into structured data, improving the efficiency and accuracy of data processing.

[0076] As an optional embodiment, a weighted graph corresponding to the target ticket image is determined based on the fused feature vectors corresponding to multiple text regions, including: determining the nodes corresponding to the multiple fused feature vectors to obtain a node set; determining the association weight value between each node in the node set and other nodes based on the association relationship between each node in the node set and other nodes to obtain an association weight set; determining the neighbor nodes corresponding to each node in the node set based on the association weight set to obtain an edge set; and constructing a weighted graph based on the node set, the association weight set, and the edge set.

[0077] This embodiment describes the specific steps for determining the weighted graph corresponding to the target ticket image based on the fusion feature vectors corresponding to multiple text regions.

[0078] This involves a node set, which is a collection of nodes corresponding to multiple text regions. Each node represents a text region in the target invoice image, and each node carries a fused feature vector. For example, if the target invoice image contains three text regions: "invoice code," "invoice number," and "amount," the node set will contain these three nodes.

[0079] This involves association weight values, which are derived from the relationships between each node and other nodes. These weight values ​​measure the strength of the association between two nodes and are used to construct the weights of edges in the weighted graph. For example, if "invoice code" and "invoice number" are adjacent in the image and have closely related semantic meanings in the corresponding text content, then their association weight value will be high.

[0080] This involves the association weight set, which is a collection of association weight values. For example, for a weighted graph containing three nodes ("invoice code", "invoice number", and "amount"), the association weight set can be the pairwise combination of association weight values ​​between the three nodes.

[0081] This involves neighboring nodes, which are nodes in the weighted graph that are directly connected to the current node via edges.

[0082] This involves an edge set, which is a collection of edges representing the connections between nodes, obtained from the neighboring nodes of each node in the node set.

[0083] In the steps involved in this embodiment, firstly, a node set is obtained by identifying the nodes corresponding to each fused feature vector, with each node representing a text region. Then, based on the association relationships between each node and other nodes, the association weight values ​​between each node in the node set and other nodes are determined, resulting in an association weight set. Next, based on the weight values ​​in the association weight set and the neighboring nodes corresponding to each node in the node set, an edge set is constructed. Finally, based on the node set, the association weight set, and the edge set, a weighted graph corresponding to the target ticket image is constructed.

[0084] By modeling the complex text region relationships in the invoice image into a weighted graph through the above steps, it is helpful to accurately identify and understand the layout and information structure in the target invoice image. This facilitates the accurate capture of the complex relationships between text regions during the subsequent extraction of target invoice information, thereby helping to accurately identify and classify each text region, improving the accuracy and efficiency of information extraction. In turn, it helps to solve the technical problem of being unable to efficiently and accurately extract invoice information due to the diversity of invoice types.

[0085] As an optional embodiment, the nonlinear feature matrix corresponding to the target ticket image is determined based on the weighted graph corresponding to the target ticket image, including: determining the adjacency matrix and node feature matrix corresponding to the weighted graph; inputting the adjacency matrix and node feature matrix into a linear transformation layer to obtain a linear feature matrix, wherein the linear transformation layer includes a predetermined weight matrix; and inputting the linear feature matrix into an activation function to obtain a nonlinear feature matrix.

[0086] This embodiment describes the specific steps for determining the nonlinear feature matrix corresponding to the target invoice image based on the weighted graph corresponding to the target invoice image.

[0087] This involves the adjacency matrix, which is used to represent the connection relationships between nodes in a weighted graph. In a weighted graph, the elements of the adjacency matrix represent the weights of the edges.

[0088] This involves a node feature matrix, which is a matrix used to represent the features corresponding to multiple text regions (nodes). Each row of the node feature matrix corresponds to a node in the weighted graph, and each column corresponds to the features of the node.

[0089] This involves a linear transformation layer, which is included in the deep learning model and is used to process the input data (such as the adjacency matrix and the node feature matrix) through linear transformations. These linear transformations can be matrix multiplication and vector addition.

[0090] This involves a linear feature matrix, which is a feature matrix obtained by linearly transforming the node feature matrix. This linear feature matrix retains the feature information of the nodes.

[0091] This involves a predetermined weight matrix, which is a parameter matrix in the linear transformation layer. This predetermined weight matrix is ​​learned through the training process and is used to adjust the weights of node features in order to better capture the correlation between nodes.

[0092] This involves activation functions, which are nonlinear functions used to introduce nonlinear transformations, enabling deep learning models to learn more complex feature relationships.

[0093] In the steps involved in this embodiment, firstly, the adjacency matrix and node feature matrix corresponding to the weighted graph are determined. The adjacency matrix represents the connection relationships between nodes and their weights, while the node feature matrix contains the feature information of each node. Then, the adjacency matrix and node feature matrix are input to a linear transformation layer for linear transformation (such as matrix multiplication and vector addition) to obtain a linear feature matrix. The predetermined weight matrix in the linear transformation layer is a parameter matrix that has been trained and optimized. Finally, the linear feature matrix is ​​input to an activation function to introduce a nonlinear transformation, generating a nonlinear feature matrix.

[0094] Through the above steps, linear transformations (such as matrix multiplication and vector addition) are performed on the adjacency matrix and node feature matrix corresponding to the weighted graph to obtain a linear feature matrix, which helps the model better understand the relationships between nodes. Inputting the linear feature matrix into the activation function introduces a nonlinear transformation, generating a nonlinear feature matrix. This helps capture complex relationships between nodes, thereby strengthening the correlation between features, improving the model's adaptability to complex document formats and information structures, and facilitating accurate identification and classification of document information in a wider range of scenarios. This improves the efficiency and accuracy of target document information extraction, and ultimately helps solve the technical problem of inefficient and inaccurate document information extraction.

[0095] As an optional embodiment, obtaining the target invoice image includes: determining an initial invoice image; determining, based on the initial invoice image, whether the initial invoice image needs repair; if the determination result is that the initial invoice image needs repair, determining a defective sub-image corresponding to the initial invoice image and adjacent sub-images, wherein the adjacent sub-images are sub-images in the initial invoice image whose proximity to the defective sub-image is greater than a proximity threshold; and repairing the defective sub-images based on the adjacent sub-images to obtain the target invoice image.

[0096] In this embodiment, the specific steps for obtaining the target ticket image are described.

[0097] This involves initial invoice images, which are scanned or photographed images of invoices that have been initially obtained without any processing. For example, an image of a paper invoice that has just been scanned, or an image of a receipt taken with a mobile phone camera, can be considered an initial invoice image.

[0098] This involves determining whether the initial ticket image needs repair. For example, the determination result could be "image is blurry and needs repair" or "image is clear and does not need repair".

[0099] This includes defect sub-images, which are image areas in the initial invoice image that have defects (such as blurriness, damage, or missing parts). For example, the invoice number area may become blurry due to shadows during scanning or paper wear; this part of the image area can be considered a defect sub-image.

[0100] This involves the concept of adjacent sub-images, which are image regions in the initial invoice image that are close to the defective sub-image and whose proximity is greater than a preset proximity threshold. For example, if the "invoice number" region is the defective sub-image, then if the "invoice code" region is relatively close to the "invoice number" region and its clarity is higher than the preset proximity threshold, then the "invoice code" region can be considered an adjacent sub-image and can be used to assist in repairing the "invoice number" region.

[0101] This involves a proximity threshold, which is a pre-set threshold used to determine whether two image regions are adjacent. For example, if the Euclidean distance between two image regions is less than 50 pixels, then the proximity of these two regions is greater than the proximity threshold, and they can be considered adjacent, which can be used as information reference in the repair task.

[0102] In the steps involved in this embodiment, an initial ticket image is first determined. Then, the quality of this initial ticket image is analyzed to determine whether repair processing is required. If the image is determined to have blurriness, damage, or other defects, i.e., the result is determined to require repair, the defective sub-image region in the initial ticket image is identified, as well as adjacent sub-images with a clarity higher than a preset proximity threshold. These clear adjacent sub-images are used to help repair the blurry or missing parts of the defective sub-image. After the repair is completed, the resulting image is the target ticket image.

[0103] Through the above steps, before extracting information from the invoice image, the imaging quality of the acquired initial invoice image (such as whether it is blurry, damaged, or missing) is first judged, avoiding the inaccurate extraction of invoice information due to imaging quality issues. By identifying the defective sub-image region and adjacent sub-images in the initial invoice image when repair is required, it helps to accurately restore the image content of the defective region. This ensures that subsequent extraction of invoice information can be performed on a high-quality image basis, reducing information extraction errors and omissions caused by invoice image quality issues, improving the efficiency and accuracy of invoice information extraction, and thus solving the problem in related technologies where invoice damage and wear prevent efficient and accurate extraction of invoice information.

[0104] As an optional embodiment, the defective sub-image is repaired based on adjacent sub-images to obtain a target ticket image, including: repairing the defective sub-image based on adjacent sub-images to obtain a repaired ticket image; determining a physical degradation function, wherein the physical degradation function is a function used to simulate defects in the ticket image; inputting the repaired ticket image into the physical degradation function to obtain a degraded ticket image; determining the similarity index between the degraded ticket image and the initial ticket image; and determining the repaired ticket image as the target ticket image if the similarity index is greater than a similarity threshold.

[0105] This embodiment describes the specific steps for repairing defective sub-images based on adjacent sub-images to obtain the target ticket image.

[0106] This involves restoring receipt images, which are receipt images obtained after being processed using image restoration algorithms and other techniques. For example, if the "amount" area in the initial receipt image is blurry, after restoration, this area becomes clear and the numbers can be accurately identified; then the restored image is the restored receipt image.

[0107] This involves a physical degradation function, which is a mathematical function used to simulate the image degradation process in reality. This physical degradation function can reintroduce the degradation effect into the restored image to test whether the restoration effect is natural and whether it can still maintain the clarity and information integrity before restoration after the degradation process.

[0108] This involves degraded invoice images, which are obtained by simulating the degradation effect of a restored invoice image using a physical degradation function. For example, the image obtained by processing the restored "amount" area using a fuzzy simulation function is a degraded invoice image.

[0109] This involves a similarity index, which quantifies the similarity between a degraded ticket image and an original ticket image. This index can be calculated based on pixel values, among other things. For example, the similarity index can assess whether the restoration effect is natural and consistent with the original image by determining the differences in pixel values ​​between the degraded ticket image and the original ticket image.

[0110] This involves a similarity threshold, which is a pre-set threshold used to determine whether the degraded ticket image is sufficiently similar to the original ticket image. For example, if the similarity threshold is set to 0.9, then a restoration effect is considered satisfactory if the similarity index is greater than 0.9.

[0111] In the steps involved in this embodiment, firstly, the defective sub-image is repaired based on adjacent sub-images to obtain a repaired ticket image. Then, a physical degradation function is determined to simulate the image degradation process in reality, such as blurring, noise, and insufficient lighting. The repaired ticket image is input into the physical degradation function to obtain a degraded ticket image. Next, a similarity index between the degraded ticket image and the initial ticket image is determined to evaluate the repair effect. If the similarity index is greater than a preset similarity threshold, it indicates that the repaired ticket image retains the original information while maintaining consistency in appearance and information integrity with the initial ticket image. In this case, the repaired ticket image is the final target ticket image.

[0112] Through the above steps, the defective sub-image is repaired based on adjacent sub-images to obtain a repaired ticket image. This repaired ticket image is then input into a physical degradation function to obtain a degraded ticket image. The physical degradation function simulates degradation, ensuring the performance and reliability of the repaired ticket image in practical applications and helping to verify the naturalness and consistency of the repair effect. By determining the similarity index between the degraded ticket image and the initial ticket image, and if the similarity index is greater than a similarity threshold, the repaired ticket image is determined as the target ticket image. This ensures the naturalness of the obtained target ticket image and its consistency with the initial ticket image, avoiding image distortion or information loss caused by repair. This provides a more reliable image foundation for subsequent information extraction, improving the efficiency and accuracy of the entire ticket information extraction process.

[0113] As an optional embodiment, based on the text parameters and spatial parameters corresponding to multiple text regions respectively, and the correlation between multiple text regions, a fusion feature vector corresponding to each of the multiple text regions is determined, including: determining a spatial feature vector corresponding to each of the multiple text regions based on the spatial parameters corresponding to each of the multiple text regions; determining a text feature vector corresponding to each of the multiple text regions based on the text parameters corresponding to each of the multiple text regions; determining style parameters corresponding to each of the multiple text regions; determining a visual feature vector corresponding to each of the multiple text regions based on the style parameters corresponding to each of the multiple text regions; and determining a fusion feature vector corresponding to each of the multiple text regions based on the text feature vector, visual feature vector, and spatial feature vector.

[0114] This embodiment describes the specific steps for determining the fusion feature vectors corresponding to multiple text regions based on the text parameters and spatial parameters corresponding to each of the multiple text regions, as well as the correlation between the multiple text regions.

[0115] This involves spatial feature vectors, which are derived from spatial parameters and used to represent the position and size of the text region within the target document image. These spatial feature vectors may include the center coordinates, width, height, and tilt angle of the text region.

[0116] This involves text feature vectors, which are obtained based on text parameters and are used to represent the text content corresponding to a text region. These feature vectors contain the semantic information of the text content in that text region.

[0117] This involves style parameters, which are derived from text parameters and are used to reflect the visual characteristics of the text area, including background color, border style, etc.

[0118] This involves visual feature vectors, which are feature vectors obtained based on style parameters and used to describe the visual characteristics of text regions on an image. For example, the visual feature vector of a text region may include a description of the color distribution, edge features, and texture of that text region.

[0119] In the steps involved in this embodiment, firstly, based on the spatial parameters of each text region, a spatial feature vector representing the position, size, etc., of the text region in the target document image is determined. Then, based on the text parameters of each text region, a text feature vector representing the text content corresponding to that text region is determined. Next, style parameters corresponding to each text region, such as background color, are determined. Based on the style parameters, a corresponding visual feature vector is determined. Finally, the text feature vector, visual feature vector, and spatial feature vector are combined to obtain a fused feature vector corresponding to each of the multiple text regions.

[0120] Through the above steps, spatial feature vectors are determined based on the spatial parameters of each text region, which helps to accurately determine the layout and relative position of the text region in the target document image. Text feature vectors are determined based on the text parameters of each text region, which helps to accurately capture the text content in subsequent text extraction. By extracting the style parameters of the text regions and determining the corresponding visual feature vectors based on these parameters, visual contextual information is provided for subsequent information extraction. By combining the text feature vectors, visual feature vectors, and spatial feature vectors, fused feature vectors corresponding to multiple text regions are obtained. This helps to achieve a more comprehensive understanding of the contextual information of the text regions when processing documents with complex layouts and information structures, reducing errors in the information extraction process, and thus helping to solve the technical problem of inefficient and inaccurate extraction of document information.

[0121] Based on the above embodiments and optional embodiments, an optional implementation method is provided, which is described in detail below.

[0122] In related technologies, due to the significant differences in format and field content among different types of invoices, traditional OCR technology often struggles to effectively handle diverse invoice types. Furthermore, during the extraction of invoice information, issues such as invoice damage and wear, and poor image quality may arise, resulting in technical problems that prevent the efficient and accurate extraction of invoice information.

[0123] There is currently no effective solution to the above problems.

[0124] In view of this, an optional embodiment of the present invention provides a method for extracting invoice information, which can also be called a method for extracting electronic invoice information based on deep learning. It can effectively solve the technical problem of being unable to extract invoice information efficiently and accurately.

[0125] Figure 2 This is a flowchart of a method for extracting invoice information in an optional embodiment of the present invention. Figure 3 This is a technical framework diagram of image restoration in an optional embodiment of the present invention. Figure 4 This is a flowchart of an optional embodiment of the image restoration method of the present invention, such as... Figure 2 , Figure 3 ,as well as Figure 4 As shown below, a detailed description will be provided.

[0126] S1, Obtain the target ticket image, wherein the target ticket image includes multiple text regions, and each of the multiple text regions includes corresponding text parameters;

[0127] For example, OCR technology is used to detect text regions (same as the text regions mentioned above) in an image (same as the target document image mentioned above) and recognize their content, outputting the text content (same as the text parameters mentioned above) and its location information (same as the spatial parameters mentioned above). This step includes two sub-steps:

[0128] A1. Text detection: Locate text regions in an image.

[0129] A2. Text Recognition: Recognizes the specific text content within the detected text region (same as the text parameters mentioned above).

[0130] Specifically, S1 includes:

[0131] S11, Determine the initial ticket image;

[0132] S12, Based on the initial ticket image, determine whether the initial ticket image needs to be repaired;

[0133] S13, if the result is determined to be the result that the initial ticket image needs to be repaired, determine the defective sub-image and the adjacent sub-image corresponding to the initial ticket image, wherein the adjacent sub-image is the sub-image in the initial ticket image whose proximity to the defective sub-image is greater than the proximity threshold.

[0134] S14. Based on the adjacent sub-images, the defective sub-image is repaired to obtain the target ticket image.

[0135] Specifically, S14 includes:

[0136] S141, Based on adjacent sub-images, repair the defective sub-image to obtain the repaired ticket image;

[0137] S142, Determine the physical degradation function, where the physical degradation function is a function used for defect simulation of the bill image;

[0138] S143, input the restored ticket image into the physical degradation function to obtain the degraded ticket image;

[0139] S144, determine the similarity index between the degraded ticket image and the initial ticket image;

[0140] S145, if the similarity index is greater than the similarity threshold, determine the repaired ticket image as the target ticket image.

[0141] For example, data collection and annotation involves gathering actual scanned images of invoices and manually annotating them, including structured information (key-value pairs of key fields) and OCR information (the location and content of all text in the image). If the original invoice image (same as the initial invoice image mentioned above) is of high quality, subsequent information extraction is performed directly; otherwise, image restoration and enhancement are performed, specifically including:

[0142] B1. Input the scanned or photographed image of the original invoice.

[0143] B2. Using a GAN-based image inpainting network, damaged areas in the image (same as the defective sub-images mentioned above) are repaired to enhance the visual quality of the image. Specifically, this includes:

[0144] B21. Fill in damaged or blurred areas.

[0145] B22. Remove stains or background noise (such as stamps or advertising text). Smooth out surface distortion caused by camera shooting.

[0146] B23. Output the repaired and enhanced high-quality document image (same as the target document image mentioned above).

[0147] S2, determine the spatial parameters corresponding to multiple text regions respectively, and determine the relationship between multiple text regions;

[0148] S3. Based on the text parameters and spatial parameters corresponding to multiple text regions, and the correlation between multiple text regions, determine the target ticket information corresponding to the target ticket image.

[0149] Specifically, S3 includes:

[0150] S31, Based on the text parameters and spatial parameters corresponding to multiple text regions respectively, and the correlation between multiple text regions, determine the fusion feature vectors corresponding to multiple text regions respectively;

[0151] Specifically, S31 includes:

[0152] S311, Based on the spatial parameters corresponding to the multiple text regions respectively, determine the spatial feature vectors corresponding to the multiple text regions respectively;

[0153] S312, Based on the text parameters corresponding to the multiple text regions respectively, determine the text feature vectors corresponding to the multiple text regions respectively;

[0154] S313, determine the style parameters corresponding to each of the multiple text regions;

[0155] S314, Based on the style parameters corresponding to the multiple text regions respectively, determine the visual feature vectors corresponding to the multiple text regions respectively;

[0156] S315. Based on the text feature vector, visual feature vector, and spatial feature vector, determine the fusion feature vector corresponding to each of the multiple text regions.

[0157] S32, based on the fusion feature vectors corresponding to multiple text regions respectively, determine the weighted map corresponding to the target ticket image. The weighted map includes nodes corresponding to key features and nodes corresponding to non-key features. Key features are features whose corresponding importance index is greater than a predetermined importance threshold, and non-key features are features whose corresponding importance index is less than or equal to the predetermined importance threshold.

[0158] Specifically, S32 includes:

[0159] S321, determine the nodes corresponding to multiple fused feature vectors to obtain a node set;

[0160] S322, Based on the association relationship between each node in the node set and other nodes, determine the corresponding association weight value between each node in the node set and other nodes, and obtain the association weight set;

[0161] S323, Based on the association weight set, determine the neighboring nodes corresponding to each node in the node set to obtain the edge set;

[0162] S324: Construct a weighted graph based on the node set, the associated weight set, and the edge set.

[0163] S33. Based on the weighted image corresponding to the target ticket image, determine the nonlinear feature matrix corresponding to the target ticket image. The nonlinear feature matrix is ​​a matrix obtained by weighting the key features and reducing the weight of the non-key features.

[0164] Specifically, S33 includes:

[0165] S331, Determine the adjacency matrix and node feature matrix corresponding to the weighted graph;

[0166] S332, the adjacency matrix and the node feature matrix are input into the linear transformation layer to obtain the linear feature matrix, wherein the linear transformation layer includes a predetermined weight matrix;

[0167] S333: Input the linear feature matrix into the activation function to obtain the nonlinear feature matrix.

[0168] S34, Based on the nonlinear feature matrix corresponding to the target ticket image, determine the target classification labels corresponding to the multiple text regions respectively;

[0169] S35, based on the text parameters and target classification labels corresponding to multiple text regions respectively, determine the target ticket information corresponding to the target ticket image.

[0170] For example, the output of the OCR module is used to construct a graph representation, and information extraction is achieved through a graph neural network (GNN). Specifically, the process of using a graph neural network includes processing the weighted graph with a graph convolutional network (GCN) to learn the high-dimensional feature representation of each node and capture the relationships between text regions. The information extraction task is then transformed into a node classification task, predicting the category label (same as the target classification label mentioned above) for each node (text region), such as "invoice number", "amount", "date", etc.

[0171] The output of the GNN (same as the target classification label mentioned above) is converted into the final structured information (same as the target invoice information mentioned above), including format conversion and error correction. Specifically, this includes:

[0172] C1. Convert the amount to a number format;

[0173] C2. Standardize dates in different formats to a standard format;

[0174] C3. Perform semantic validation and correction on field values ​​(e.g., check whether the amount field is a valid number).

[0175] The field-level F1 score is used to evaluate the effectiveness of information extraction. The F1 score is the harmonic mean of precision and recall, which measures the accuracy of the information extracted by the model.

[0176] In summary, this method can utilize Graph Neural Networks (GNNs) and Generative Adversarial Networks (GANs) to achieve information extraction. It includes the following key steps:

[0177] Step 1: Data collection and annotation;

[0178] During the data collection process, invoices are typically pasted onto a blank sheet of paper, then scanned to generate images and uploaded to the system. These invoice images, after authorization, proceed to subsequent processing. In scenarios involving invoice information entry or verification, the system needs to extract key-value pairs of field content from the invoices. The key is a pre-defined field name (e.g., "invoice code," "invoice number," etc.), and the value is the text content of the corresponding field obtained through image recognition technology. It's important to note that different types of invoices may have some overlap in their key information fields, but they also have their own unique fields. For example, the key fields for a general fixed-amount invoice include "invoice code," "invoice number," and "amount"; while some itineraries (such as flight itineraries) include "date," "amount," "departure station," and "arrival station." Therefore, the key information requirements differ for different invoice types, which places higher demands on the accuracy and adaptability of information extraction.

[0179] To address the aforementioned task requirements, two independent data annotation processes were designed to obtain high-quality annotation information: the "structured information annotation process" and the "OCR information annotation process." The first annotation process aims to generate the "structured information" of the invoice, namely the key-value pairs and their location information of key fields in each image (same as the target invoice image mentioned above). The second annotation process aims to generate the "OCR information" of the invoice (same as the target invoice image mentioned above), namely the text content (same as the text parameters mentioned above) of all text (excluding curved text) in the image and its position coordinates in the image (same as the spatial parameters mentioned above). These two annotation processes provide two different levels of annotation information for subsequent model training, used for key field extraction and full-image text detection and recognition, respectively.

[0180] Both annotation processes employed a "pseudo-label generation + manual review and cleaning" approach to improve annotation efficiency and accuracy. In the first annotation process, a pre-selected document recognition system was used to process the image, automatically generating pseudo-labels containing "structured information." These pseudo-labels included the location information of key fields (same as the spatial parameters mentioned above) and the corresponding text content (same as the text parameters mentioned above). Subsequently, annotators manually reviewed and cleaned the generated pseudo-labels to ensure accuracy. For blurry text that could not be manually recognized, it was uniformly labeled with the "NOT CARE" attribute to avoid interfering with model training. In the second annotation process, an OCR inference engine was first used to perform text detection and recognition on the document image (same as the target document image mentioned above), generating full-image OCR pseudo-labels. Then, the key field text boxes generated in the first annotation process were matched with the text boxes in the OCR inference results (same as the text regions mentioned above) to ensure consistency between the key field annotations and the OCR results.

[0181] During the matching process, if the Intersection over Union (IoU) of a key field text box (same as the text area mentioned above) and the OCR inference text box is greater than 0.5, the coordinates and content of the corresponding text box in the OCR inference result are directly overwritten with the truth value from the first annotation process. If no matching text box is found for a key field text box in the OCR inference result, the field is considered a missed item in the OCR and needs to be directly added to the OCR inference result. Through this matching and correction process, it is ensured that the labels of all key fields in the OCR annotation are manually reviewed and cleaned to ensure their accuracy. However, non-key field text in the background may still contain a certain degree of OCR noise, but this noise will not affect the annotation quality of key fields.

[0182] Through the two annotation processes described above, high-quality annotation of key fields and text throughout the document image was achieved, providing reliable training data for subsequent deep learning-based OCR and information extraction models. This annotation method not only adapts to the needs of various document types but also effectively reduces the workload of manual annotation while ensuring the accuracy and consistency of the annotation results, thus providing an operational foundation for document information extraction tasks.

[0183] Specifically, data acquisition and annotation are the foundational steps of the entire system. Their role is to provide high-quality training and validation data for subsequent deep learning models (such as OCR modules and information extraction models). By annotating the ticket images, the location and content of key fields are clarified, enabling the model to learn how to correctly extract key information from the image. The main steps include:

[0184] (1) Input: Original scanned image or captured image of the invoice (same as the target invoice image mentioned above).

[0185] (2) Output: Structured information annotation (key-value pairs and location information of key fields), OCR information annotation (detection and recognition results of text in the whole image).

[0186] (3) Execution: Completed by a combination of automated tools (specific document recognition system, OCR engine) and manual annotation.

[0187] Step 2: Image restoration and enhancement (optional);

[0188] This step primarily involves constructing a Generative Adversarial Network (GAN) to solve various image inpainting problems (such as image deblurring, dehazing, and deraining). The image inpainting problem can be described as recovering a clear image x (the same as the inpainted ticket image described above) from a degraded input image y (the same as the initial ticket image mentioned above). This mainly includes:

[0189] (1) Generative Network Generate a sharp image from a degraded image y.

[0190] (2) Physical model constraints: Ensure that the generated results are consistent with the input image under the physical degradation model.

[0191] (3) Discrimination network

[0192] Discrimination Network Used to determine whether the generated image is similar in distribution to the real, clear image;

[0193] Discrimination Network Used to determine whether the generated image is identical to the input image after physical degradation.

[0194] Specifically, this step includes the following sub-steps:

[0195] D1. Problem Modeling:

[0196] The degradation process of the input image can be represented by a physical degradation model, as shown in the following formula:

[0197]

[0198] Where: y represents the input degraded image, that is, the input image obtained from the captured or acquired image after being blurred, fogged, raindroped, or otherwise degraded. x represents the clear image. These are physical operators that describe the degradation process and represent the physical model of image degradation. Specifically, they can be:

[0199] 1) Fuzzy operations: such as convolution operations representing fuzzy kernels;

[0200] 2) Fog models: such as atmospheric degradation models (transmittance, background light);

[0201] 3) Raindrop interference: This refers to the interference mode that covers the pixels.

[0202] The output determines the specific physical model. The degradation mechanisms and their parameters (such as fuzzy kernel and transmittance) are clarified for different tasks.

[0203] make and Let i represent the degraded (blurred) image and the clear image, respectively. Let i represent the i-th ticket image and N represent the total number of ticket images.

[0204] D2, GAN network architecture design:

[0205] The network architecture consists of three parts: the generative network. Physical degradation module and discriminant network

[0206] 1) Generative Network Generative Networks It is the core module of image inpainting, responsible for restoring a clear image from a degraded input image y. It adopts a Cycle GAN-style structure, including convolutional layers for feature extraction, residual blocks for processing global context information, and deconvolutional layers for reconstructing a clear image. The generator takes the degraded image y as input and generates the result. This is the output.

[0207] 2) Physical Degradation Module: This module uses a physical degradation model (same as the physical degradation function described above) to generate a clearer image. Simulation degenerates into (Similar to the degraded ticket image mentioned above), to conform to the degradation process experienced by the input image y, thus introducing physical consistency constraints into the network. Specifically, the physical degradation model is set according to the specific task (deblurring, dehazing, deraining, etc.). For example, in deblurring, the degradation model is a convolution operation, with the formula:

[0208]

[0209] Where, k i It is a fuzzy kernel. It is a degraded simulation image (same as the degraded ticket image mentioned above).

[0210] In defogging, the degradation model is the atmospheric scattering model, and the formula is:

[0211]

[0212] Among them, t i It is transmittance, A i It is the atmospheric light value.

[0213] Overall, the module takes a clear image as input. With specific degradation parameters, output the generated degradation simulation map.

[0214] 3) Discriminating network The network consists of two sub-modules. and

[0215] It is used to distinguish generated images Whether it conforms to the distribution of clear images (or "determine whether the generated image is similar to the real clear image in distribution"), that is, to determine whether it is a realistic "high-definition image";

[0216] It is used to identify degradation simulation diagrams Does it match the input image y (or, in other words, "determine whether the physical degradation model generated by the above model is consistent with the input image y)? "Whether it matches the input image y" means ensuring that the generated image conforms to the constraints of the physical model after the degradation model.

[0217] Overall, discriminant networks The input includes a real, clear image x and the generated image. Degradation simulation diagram The network takes an input image y as input. It outputs two discrimination results, which are used to guide the optimization of the generator network.

[0218] These modules are built and implemented by developers, using deep learning frameworks such as PyTorch or TensorFlow for programming.

[0219] D3. Loss Function Design:

[0220] The GAN model of this invention achieves perceptual and physical consistency by jointly optimizing the generator and discriminator networks through multiple loss functions. Specifically, these three loss functions are as follows:

[0221] 1) Combating losses

[0222] This loss function ensures that the distribution of the generated image approximates the distribution of the real image, and the formula is:

[0223]

[0224] 2) Pixel consistency loss:

[0225] This loss function ensures that the generated image is close to the real image in pixel values, including:

[0226] L1 loss of the generated image (same as the repaired ticket image above) and the real image (same as the target ticket image above). The formula is:

[0227]

[0228] L1 loss of the degraded image (same as the degraded ticket image above) and the input image (same as the initial ticket image above). The formula is:

[0229]

[0230] 3) Generator stability loss

[0231] The loss function further improves stability by constraining the generator's output, and the formula is as follows:

[0232]

[0233] Therefore, the total loss The formula is:

[0234]

[0235] Specifically, firstly, adversarial loss is used to constrain the discriminant network. and generative networks The collaboration ensures that the distribution of the generated images closely approximates the distribution of real, sharp images. The adversarial loss is handled by a discriminative network. and The output components, whose goal is to generate an image. More realistic, while ensuring the image remains unchanged even after degradation. With input image y i To maintain consistency. Furthermore, the method uses pixel consistency loss to constrain the pixel differences between the generated image and the real image, specifically including the generated image... Compared to real image x i L1 loss, and degraded image and input image y i The L1 loss is applied. To further stabilize the generator, a generator stability loss is added, which ensures that the generator can recover from degraded images. A clear image x was reconstructed from the image. i The final loss function is a weighted sum of the above components. This process requires adjusting the relevant weight coefficients λ, γ, and μ according to the task to optimize the overall loss objective.

[0236] The inputs to the above module are the generated image, the real image, the degraded image, and their corresponding parameters; the output is the joint loss value (as described above). ), used to guide the parameter updates of the model.

[0237] The final objective function to be solved is:

[0238]

[0239] in, This represents the optimal solution.

[0240] D4. Training Process:

[0241] D41. Training Phase:

[0242] During the training phase, the adversarial generative network of this invention is used. and discriminant networks Joint optimization.

[0243] First, use adversarial generative networks Generate a preliminary clear image from the degraded image y. Image y that is about to degrade i Input generator A preliminary, clear image G(y) is obtained.

[0244] Subsequently, through the physical degradation module Convert to degraded image To simulate the input image y i The process of degradation.

[0245] Next, the two sub-modules of the discriminant network are used respectively. and To generate clear images and degraded simulated images Perform the discrimination and output the discrimination score.

[0246] Finally, through backpropagation of the gradient of the loss function, the generator and discriminator are jointly optimized. The generator continuously improves the quality of the output clear image, while the physical constraint module and discriminator ensure the physical consistency of the simulated degradation image, so that the final generation result is both realistic and in line with the actual degradation mechanism.

[0247] The above process takes a training dataset (containing both degraded and sharp images) as input and outputs optimized model parameters. The entire training process can be completed using a GPU cluster or a deep learning platform.

[0248] D42. Testing Phase:

[0249] During the testing phase, the GAN model of this invention was directly applied to an unseen (untrained) degraded image y. The degraded image was input into the generative network. Then, output a clear image. Without the need for a physical degradation module and a discriminant network, the generator can independently complete image restoration tasks, and the generated clear images can be directly used for subsequent tasks, such as visual analysis or OCR extraction.

[0250] In this phase, the input is a new image from the degradation task, and the output is the corresponding clear image. The testing phase is performed by the end user and is typically run in a real-world deployment environment.

[0251] GAN models learn distributions adversarially through generative and discriminative networks. However, existing GAN methods (such as pix2pix and Cycle GAN) do not consider the physical consistency between the input and generated images, leading to potentially inconsistent generated results with physical degradation models. This new approach, however, proposes a unified framework combining physical model constraints and GANs, enabling generated results to not only perceptually approximate realistic images but also satisfy physical degradation models. This approach is better suited for image inpainting tasks with various degradation scenarios (such as deblurring, dehazing, and deraining). By combining the advantages of physical degradation models and generative adversarial networks (GANs), it recovers sharp images from input degraded images and ensures optimization of the generated results in terms of perceptual and physical consistency.

[0252] Step 3: OCR text detection and recognition;

[0253] This module primarily extracts all text content contained in an image, typically using a mature OCR engine for inference. General-purpose OCR inference engines are usually trained on a large number of natural scene images and possess a certain level of text recognition capability in generalized new scenes.

[0254] The main content includes three OCR settings: ① Text input using a general OCR inference engine; ② Fine-tuning the text detection / recognition model using scenario data and using the model's prediction results as input; ③ Text input directly using OCR ground truth values.

[0255] An OCR-based inference engine is used, employing Text-Perceptron-Det as the text detector and CRNN as the text recognizer, to obtain text coordinates and recognition results based on text lines.

[0256] Step 4: Graph representation construction and information extraction (including graph construction, GNN processing, and node classification);

[0257] This step uses Graph Neural Networks (GNNs) to construct graph representations and extract information. First, in the graph representation construction and information extraction stage, text boxes (same as the text regions mentioned above) and their relationships (same as the association relationships mentioned above) are modeled as a weighted graph to capture the multidimensional relationships between text boxes. Then, a Graph Convolutional Network (GNN) is used to propagate and aggregate features on the graph, updating the feature representation of each text box (same as the non-linear feature matrix mentioned above). Finally, a classifier is used to classify the nodes of each text box and predict its semantic label, thereby extracting key information from the document (such as date, amount, etc.). This entire step involves three modules:

[0258] 1) Encoder module:

[0259] The encoder module is responsible for extracting multimodal features (same as the fused feature vector) from the visually rich document (same as the target ticket image mentioned above), namely text, image and spatial features (same as the text feature vector, visual feature vector and spatial feature vector mentioned above, respectively). It contains multiple text boxes, each with its position coordinates, text content and corresponding local image fragment (same as the spatial parameters, text parameters and style parameters mentioned above, respectively).

[0260] Initial input: Original visually rich documents (e.g., invoices, receipts), for example,

[0261] Text box 1: Position = (Xa1, Yb1, Xa2, Yb2), Text content = "Total", Image fragment = partial screenshot.

[0262] Text box 2: Position = (Xa3, Yb3, Xa4, Yb4), Text content = "23.45", Image clip = partial screenshot.

[0263] Execution: Text embeddings are generated by the Word2Vec model, image embeddings are generated by the CNN model, and spatial embeddings are generated using a position calculation formula.

[0264] ① Text Feature Extractor: The Word2Vec model is used to embed text content.

[0265] Input: Text content (e.g., "Total", "23.45") (same as the text parameters above).

[0266] Output: Text embedding vector T i (Same as the text feature vectors mentioned above.)

[0267] ② Image Feature Extractor: Uses a convolutional neural network to embed image fragments into the text box.

[0268] Input: Image fragment (such as a partial screenshot of "Total") (same as the style parameters above).

[0269] Output: Image embedding vector I i (Same as the visual feature vectors mentioned above.)

[0270] ③ Spatial Feature Calculator: Calculates spatial embedding by combining the bounding box coordinates (Xa1, Yb1, Xa2, Yb2) of the text box.

[0271] Where Xa1 and Xa3 are the x-coordinates of the top left corner of the text box, representing the distance from the left edge of the image to the left edge of the text box; Yb1 and Yb3 are the y-coordinates of the top left corner of the text box, representing the distance from the top edge of the image to the top edge of the text box; Xa2 and Xa4 are the x-coordinates of the bottom right corner of the text box, representing the distance from the left edge of the image to the right edge of the text box; and Yb2 and Yb4 are the y-coordinates of the bottom right corner of the text box, representing the distance from the top edge of the image to the bottom edge of the text box.

[0272] Input: Text box position coordinates (same as the spatial parameters above).

[0273] Output: Spatial embedding vector S i(Same as the spatial feature vectors mentioned above), including: the Euclidean distance between the center point of the text box and the center point of the document, and the polar coordinates (angle and radius) of the center point of the text box.

[0274] Final output: Multimodal feature vector F for each text box i (Similar to the fused feature vectors mentioned above), it includes text embedding vectors, image embedding vectors, and spatial embedding vectors, namely:

[0275] F i =[T i I i S i ]

[0276] 2) Graph representation module:

[0277] This graph representation module is responsible for converting the document image (similar to the target ticket image mentioned above) into a weighted graph to capture the diverse relationships between text boxes.

[0278] Input: The multimodal feature vector F = {F1, F2, ..., F...} generated by the encoder module n}, where each F i This represents the feature of the i-th text box.

[0279] Execution: A custom weighted graph construction method that uses feature distance metrics to calculate edge weights.

[0280] ① Node Definition: Define each text box as a node of the graph, with node characteristics F. i .

[0281] ② Edge definition: Define the edges between nodes to represent the relationship between text boxes. The weight of the edges is calculated using the following metrics: Euclidean distance, Manhattan distance, cosine similarity, correlation of image features, mutual information of image features, etc.

[0282] ③ Neighbor selection: Select the top K neighbors with the highest weight for each node (same as the neighbor nodes mentioned above) and construct the edge set E.

[0283] Output: Construct a weighted graph G = (V, E, Ω), which contains a node set V (features of each text box), an edge set E (edges between each pair of text boxes), and a weight set Ω (relationships / weights between text boxes) (same as the associated weight set mentioned above).

[0284] 3) Node classification module:

[0285] This node classification module is responsible for modeling the task of extracting information from visually rich documents as a node classification problem on a graph, and predicting the label of each text box (same as the target classification label mentioned above).

[0286] Input: The weighted graph G = (V, E, Ω) generated in the second step.

[0287] Execution: Graph Convolutional Network (GCN) classifies nodes through multiple layers of propagation and aggregation (similar to the linear transformation layers mentioned above).

[0288] ① Graph Convolutional Network (GCN): Through multi-layer propagation and aggregation, the feature representation of each node is updated. Each node receives information from its neighboring nodes through adjacency relationships and calculates the weighted sum of feature representations.

[0289]

[0290] Wherein: H (l) These are the node features of layer l. The features of each node are obtained through propagation from the previous layer. W (l) The trainable weight matrix of layer l is used for linear transformation of features (same as the predetermined weight matrix mentioned above), and σ is the activation function (e.g., ReLU or Sigmoid) used to introduce non-linearity and enhance the expressive performance of the model. It is a normalized adjacency matrix (including edge weights) (similar to the adjacency matrix mentioned above, used to control the range and proportion of feature propagation; the structural information of the weighted graph will affect feature updates).

[0291] After propagation layer by layer, the final node feature representation (nonlinear transformation matrix) is generated.

[0292] ② Classifier: The softmax classifier classifies the final features of each node and predicts its semantic label.

[0293] Output: Category labels for each text box (e.g., "Date", "Total Amount", etc.).

[0294] The following is a detailed description, including the three main steps of graph construction, GNN processing, and node classification.

[0295] (1) Graph representation construction:

[0296] The goal of graph construction is to model the text boxes and their relationships in a document as a weighted graph, capturing the semantic, spatial, and visual connections between text boxes. The input is a multimodal feature vector extracted from the encoder, which includes text embeddings, image embeddings, and spatial embeddings.

[0297] First, the method defines each text box in the document as a node in the graph, and the features of the node are represented by the multimodal feature vector of the text box.

[0298] Then, edges are defined between nodes to represent the relationships between text boxes. These edges are established according to different rules, such as spatial proximity (two text boxes are located close to each other in the document), semantic similarity (the text content of two text boxes is semantically similar), or visual feature similarity (the image fragments corresponding to the text boxes have similar features). The weights of the edges are calculated using various distance metrics, including Euclidean distance, cosine similarity, or image feature correlation.

[0299] In addition, to avoid redundant edges interfering with graph modeling, the method selects the top K most relevant neighbors based on weights to construct a sparse structure of the graph.

[0300] Ultimately, the output is a weighted graph, where nodes represent text boxes, and edges and their weights characterize the multidimensional relationships between text boxes.

[0301] The main goal of this step is to model the text boxes and their relationships in a visually rich document (VRD) as a weighted graph, capturing the semantic, spatial, and visual relationships between the text boxes.

[0302] Input: A set of multimodal feature vectors F = {F1, F2, ..., F...} from the encoder module n}, where each F i A text box represents a multimodal feature, including text embedding, image embedding, and spatial embedding.

[0303] Execution process:

[0304] The graph construction process includes three key steps: node definition, edge definition, and weight calculation.

[0305] ① Node definition: Each text box is considered a node in a graph, and the features of a node are defined by its multimodal feature vector F. i Each node carries semantic information (text embedding), image information (image embedding), and spatial information (spatial embedding) of the text box, thus comprehensively representing the attributes of the text box.

[0306] ② Edge Definition: Edges are added between nodes to represent the relationships between text boxes. Edges are established based on the following rules:

[0307] Spatial proximity: If two text boxes are located close to each other in a document, an edge is established between them.

[0308] Semantic similarity: If the text content of two text boxes is semantically similar, then an edge is established between them.

[0309] Visual feature similarity (same as style parameters above): If the images of two text boxes are similar in the feature space, then an edge is established between them.

[0310] ③ Edge weight calculation: Edge weights are calculated using various distance metrics, such as:

[0311] Euclidean distance: measures the spatial proximity of text boxes.

[0312] Cosine similarity: measures the semantic similarity of text content.

[0313] Image feature correlation (same as style parameters above): measures the visual similarity of local image fragments within the text box.

[0314] ③ Neighbor selection: Select the top K neighbors with the highest weight for each node to construct a sparse graph structure (same as the weighted graph above) to avoid the influence of too many redundant edges.

[0315] Output: A weighted graph G = (V, E, Ω), where:

[0316] Node set Represents a text box (same as the text region mentioned above), including the features of each text box (same as the fused feature vector mentioned above), where n indicates that there are n text boxes (graph nodes);

[0317] Edge set This indicates the relationship between text boxes (similar to the association relationship described above), including the edges between each pair of text boxes, i.e., describing the relationship between different nodes, such as v (i) and v (j) The connectivity status;

[0318] Weight set This represents the weight of each edge, assigns a weight to each edge, and captures the multidimensional relationships between nodes.

[0319] Furthermore, X∈R n×d Node feature matrix (same as the node feature matrix above), representing the feature vector of each node, where d is the feature dimension.

[0320] (2) Graph Neural Network (GNN) processing:

[0321] After the weighted graph is constructed, the system inputs it into a Graph Neural Network (GNN). Utilizing the graph's structural information and node features, it performs feature propagation and aggregation to capture the global contextual relationships between text boxes. In each GNN layer, each node receives information from its neighbors and updates its own feature representation. The propagation mechanism controls the propagation of neighbor information through a normalized adjacency matrix, while simultaneously performing a weighted summation using a trainable parameter matrix, ultimately generating a new feature representation for each node at that layer. Activation functions are used to introduce non-linearity, thereby enhancing the model's expressive power. After multiple propagation layers, each node's features not only include its own information but also incorporate contextual information from neighboring nodes and even more distant nodes. Furthermore, multi-layer propagation expands the receptive field of nodes, enabling the model to capture long-distance relationships within the document, thus improving the accuracy of semantic representation of text boxes. After this stage of processing, the output is the final feature representation of each node. After feature aggregation, the semantic and spatial dependencies between nodes are effectively fused.

[0322] The main goal is to capture the global relationships between text boxes by using graph neural networks (GNNs) to propagate and aggregate features on a weighted graph.

[0323] Input: Weighted graph G = (V, E, Ω), containing node features, edges and their weights.

[0324] Execution process: GNN updates the feature representation of each node through multi-layer propagation (similar to the linear transformation layer mentioned above) and feature aggregation mechanism, so that it not only contains its own information, but also incorporates the contextual information of its neighboring nodes. The specific process is as follows:

[0325] ① Feature propagation: In each layer, each node receives information from its neighboring nodes through adjacency relationships. The input H of the first layer GCN (0) This is the original feature matrix X of the node (same as the node feature matrix mentioned above). That is, for layer 0:

[0326] H (0) =X

[0327] This setup allows the model to learn higher-level feature representations by propagating layer by layer based on the original node features.

[0328] ② Feature aggregation: During propagation, node features are updated layer by layer. The final feature representation of each node not only includes its own information but also incorporates information from its neighboring nodes. This feature aggregation mechanism can capture the global relationships between text boxes.

[0329] ③ Multi-layer propagation: Through multi-layer propagation, GNNs can gradually expand their receptive field and capture relational information from farther distances. For example, the first layer aggregates information from direct neighbors, and the second layer aggregates information from second-order neighbors.

[0330] Output: The updated node feature matrix (same as the nonlinear feature matrix above), where the feature representation of each node incorporates contextual information.

[0331] (3) Node classification:

[0332] Node classification is the final step in the graph representation method, aiming to predict the semantic label of each text box based on the node's feature representation. The input is a node feature matrix processed by a GNN; these features have incorporated contextual relationships and possess strong semantic expressive power. The classifier maps node features to a fixed number of class spaces through a fully connected layer and uses a softmax function to calculate the probability distribution of a node belonging to each class. For example, a text box might be classified as "date," "amount," or "invoice number." Finally, the classifier uses the class with the highest probability distribution as the predicted label for that node. Through this process, each text box in the document is assigned a specific semantic category, thus achieving the goal of information extraction. The output is the classification result for each text box, i.e., its semantic label.

[0333] The main objective is to model the node classification task on the graph as an information extraction problem and predict the semantic label of each text box.

[0334] Input: The node feature matrix after GNN processing (same as the nonlinear feature matrix mentioned above).

[0335] Execution process:

[0336] Node classification is accomplished using a simple classifier (such as a softmax classifier), and the specific process is as follows:

[0337] ① Classifier Design: A graph convolutional network (GCN) with m layers is used, aiming to classify each node on a custom document graph. In the last layer, the output of the GCN is consistent with the number of classification labels and is mapped to the probability distribution of each class through a softmax classifier. The formula is as follows:

[0338]

[0339] Where Z is the output matrix of the last layer, representing the probability distribution of node classification, with each node outputting the probability of each category. It is the normalized adjacency matrix, H (m) W is the node feature matrix of the m-th layer GCN, containing the aggregated features of each node from the previous layer. (m) is the trainable weight matrix of the m-th layer, used to linearly transform the features to the probability space of the classification dimension. softmax is the normalized activation function that maps the features of each node to the class probability distribution.

[0340] Defined as:

[0341]

[0342] in, It is the adjacency matrix A plus the self-loop (identity matrix I) n The purpose of adding the identity matrix is ​​to create a self-loop for each node, enabling it to incorporate its own features. It is the degree matrix in the normalization process, used to standardize each row of the adjacency matrix (i.e., the sum of the weights of each row is 1).

[0343] softmax is defined as:

[0344]

[0345] Among them, softmax(sof i ) represents the vector sof i (Same as above The result of applying the softmax function to the i-th element in ().

[0346] Ultimately, softmax maps the features of each node to the class space, making the sum of the elements in each row of Z equal to 1, which can be interpreted as a probability distribution.

[0347] ② Category Prediction: Based on the probability distribution output by softmax, the category with the highest probability is selected as the final predicted label for the node. For example, a text box may be classified as "Date", "Amount", "Invoice Number", etc.

[0348] In the node classification task, the cross-entropy loss function is used to optimize the classification of labeled nodes. The formula for the loss function is as follows:

[0349]

[0350] in, The value of the loss function, Type i,num This represents the value of the i-th text box belonging to the num-th label class. i,num This represents the probability that the i-th text box belongs to the num-th label class, where NUM represents the total number of labels.

[0351] Output: The category label for each node (same as the target category label above), that is, the semantic category of each text box.

[0352] Generally speaking, the figure shows that the construction and information extraction phase realizes the modeling and extraction of complex document information through three steps: graph modeling, GNN feature propagation, and node classification. Starting from multi-modal features, a weighted graph is constructed to capture the multi-dimensional connections of text boxes; then, feature fusion and context modeling are achieved through GNN layers; finally, each node is classified into the corresponding category through a classifier to output the key information of the document. The graph construction algorithm, GNN model, and classifier are the core executors in this phase, and they work together to solve the problems of modeling and information extraction of diverse text box relationships in the document.

[0353] Step 5: Post-processing (formatting and error correction);

[0354] This step is mainly responsible for converting the model output into the output form of the final solution, mainly including mapping the predicted values back to the dictionary values, filtering special symbols, empty texts, etc. It should be noted that in actual business applications, the post-processing form of data will be customized according to the requirements of downstream tasks. For example, convert the amount into a uniformly formatted number for storage, and for different date formats, such as 2024 / 08 / 06, 202407-07, August 6, 2024, store them using a unified regular expression to facilitate subsequent data storage or processing.

[0355] Through this step, the extracted bill information can be automatically converted into standardized structured data, which is convenient for direct storage in a relational database or for further processing in subsequent processes. This step improves the availability and usability of data and meets the high-efficiency requirements of modern data processing.

[0356] Step 6: Evaluation metric (F1 value evaluation).

[0357] This step is mainly for the information extraction results, so the evaluation metric of this step directly uses the field-level F1 value for evaluation. This metric is calculated from the precision P and recall R:

[0358] [[ID=ID=18]]

[0359] Among them, TP represents the number of fields in all predictions that match the information structured truth value, FP represents the number of fields predicted that do not appear in the information structured truth value, and FN represents the number of fields in the information structured truth value that are not matched. ε is a small numerical constant (0.001) to prevent the appearance of blank images without truth values. The F1 value of the entire dataset is the average of the F1 values of each image.

[0360] Based on the above task settings, deep learning is used to implement the extraction of electronic credential information.

[0361] Through the above optional implementation methods, at least the following beneficial effects can be achieved:

[0362] (1) Compared with related technologies, this invention, by determining the spatial parameters of each of the multiple text regions in the target ticket image, helps to accurately identify and locate key information in the ticket. Furthermore, by determining the relationships between these text regions, it helps to accurately understand the logical connections between them, thereby enabling accurate identification and parsing of ticket information. Utilizing the text and spatial parameters extracted from the text regions helps to achieve targeted and efficient information extraction. Combining the relationships between text regions, the target ticket information corresponding to the target ticket image is determined, realizing the conversion of unstructured information in the ticket image into structured data. This reduces misidentification during the ticket information extraction process, improves the efficiency and accuracy of ticket information extraction, and thus solves the technical problem in related technologies where ticket information cannot be extracted efficiently and accurately.

[0363] (2) Compared with related technologies, this invention, by comprehensively considering the text parameters and spatial parameters corresponding to multiple text regions, as well as the correlation between multiple text regions, determines the fusion feature vectors corresponding to each of the multiple text regions. This helps to accurately identify the real text information corresponding to multiple text regions and reduce misidentification in the process of extracting invoice information. Based on these fusion feature vectors, a weighted graph is constructed, where nodes represent text regions and the weights of edges reflect the correlation strength between text regions. When constructing the weighted graph, key features and non-key features are distinguished, solving the problem of different features having different importance in the information extraction process. Through weighting and deweighting, the prominence of key features is ensured, improving the accuracy of information extraction. By processing the weighted graph, a nonlinear feature matrix is ​​obtained, which can more accurately identify and classify key information in invoices, reducing errors in the information extraction process and helping to improve the comprehensiveness of invoice information extraction in the future. By determining the target classification label corresponding to each text region based on the nonlinear feature matrix, the target invoice information corresponding to the target invoice image is determined, effectively solving the problem of converting unstructured information in invoice images into structured data and improving the efficiency and accuracy of data processing.

[0364] (3) Compared to related technologies, this invention, by determining the spatial parameters (such as coordinates and dimensions) of multiple text regions in the target document image, helps to accurately identify and locate key information in the document, even in complex layouts. Determining the relationships between these text regions, including visual proximity and semantic relevance, helps to accurately understand the logical connections between them, thereby enabling accurate identification and parsing of document information, especially for non-standard or complex document formats. By determining the spatial parameters and relationships of multiple text regions, document information can be accurately identified and extracted, reducing misidentification and omissions, while also handling various document formats, thus improving the versatility and applicability of the method.

[0365] (4) Compared with related technologies, this invention obtains a linear feature matrix by performing linear transformations (such as matrix multiplication and vector addition) on the adjacency matrix and node feature matrix corresponding to the weighted graph, which helps the model better understand the relationship between nodes. Inputting the linear feature matrix into the activation function introduces a nonlinear transformation, generating a nonlinear feature matrix, which helps capture the complex relationships between nodes, thereby strengthening the correlation between features, improving the model's adaptability to complex document formats and information structures, and helping to accurately identify and classify document information in a wider range of scenarios. This improves the efficiency and accuracy of target document information extraction, and thus helps solve the technical problem of inefficient and inaccurate document information extraction.

[0366] (5) Compared with related technologies, before extracting information from the invoice image, the present invention first judges the imaging quality (such as whether it is blurry, damaged, or missing) of the initial invoice image, avoiding the inability to accurately extract invoice information due to imaging quality problems. By determining that the result requires repair, the defective sub-image region and adjacent sub-images in the initial invoice image are identified, which helps to accurately restore the image content of the defective region. This ensures that the subsequent extraction of invoice information can be carried out on the basis of high-quality images, reducing information extraction errors and omissions caused by invoice image quality problems, improving the efficiency and accuracy of invoice information extraction, and thus solving the problem in related technologies where invoice information extraction is not efficient and accurate due to problems such as invoice damage and wear.

[0367] (6) Compared with related technologies, this invention determines spatial feature vectors based on the spatial parameters of each text region, which helps to accurately determine the layout and relative position of the text region in the target document image. Determining text feature vectors based on the text parameters of each text region helps to accurately capture the text content in the subsequent text. By extracting the style parameters of the text region and determining the corresponding visual feature vectors based on the style parameters, visual context information is provided for subsequent information extraction. By integrating the text feature vectors, visual feature vectors, and spatial feature vectors, fused feature vectors corresponding to multiple text regions are obtained, which helps to achieve a more comprehensive understanding of the context information of the text region when processing documents with complex layouts and information structures, reducing errors in the information extraction process, and thus helping to solve the technical problem of inefficient and inaccurate extraction of document information.

[0368] (7) Compared with related technologies, this invention can better address the problem of complex invoice formats. For fixed formats, it can quickly and accurately extract preset fields. For open formats, it can dynamically identify key-value pairs, enabling flexible field extraction and a wider range of applications. Furthermore, it not only utilizes graph neural networks (GNNs) in deep learning technology to achieve automated adaptation to diverse invoice types and parsing of format information, improving the accuracy of information extraction for different types of invoices, but also introduces image restoration functionality through generative adversarial networks (GANs) to solve the problem of missing or blurred text caused by invoice damage or wear, enhancing the robustness of information extraction. It also provides an end-to-end automated solution, achieving highly efficient automated processing from invoice scanning / photographing input to key information extraction and standardized structured output, making it suitable for batch business scenarios.

[0369] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0370] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0371] Example 2

[0372] According to embodiments of the present invention, an apparatus for implementing the above-described method for extracting invoice information is also provided. Figure 5 This is a structural block diagram of a ticket information extraction device according to an embodiment of the present invention, such as... Figure 5 As shown, the device includes: an acquisition module 502, a first determination module 504, and a second determination module 506. The device will be described in detail below.

[0373] The acquisition module 502 is used to acquire a target ticket image, wherein the target ticket image includes multiple text regions, and each of the multiple text regions includes corresponding text parameters; the first determination module 504 is connected to the acquisition module 502, and is used to determine the spatial parameters corresponding to the multiple text regions respectively, and to determine the association relationship between the multiple text regions; the second determination module 506 is connected to the first determination module 504, and is used to determine the target ticket information corresponding to the target ticket image based on the text parameters and spatial parameters corresponding to the multiple text regions respectively, and the association relationship between the multiple text regions.

[0374] It should be noted that the above-mentioned acquisition module 502, the first determination module 504 and the second determination module 506 correspond to steps S102 to S106 in the method for extracting invoice information. The multiple modules and the corresponding steps are the same in terms of implementation examples and application scenarios, but are not limited to the content disclosed in the above embodiment 1.

[0375] Example 3

[0376] According to another aspect of the present invention, an electronic device is also provided, comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor is configured to execute instructions to implement the document information extraction method of any of the above embodiments.

[0377] Example 4

[0378] According to another aspect of the present invention, a computer-readable storage medium is also provided, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the ticket information extraction method described above.

[0379] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0380] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0381] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0382] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0383] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0384] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0385] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for extracting invoice information, characterized in that, include: Obtain a target ticket image, wherein the target ticket image includes multiple text regions, and each of the multiple text regions includes corresponding text parameters; Determine the spatial parameters corresponding to the multiple text regions respectively, and determine the relationship between the multiple text regions. The spatial parameters include the bounding box coordinates of the text region, the center point position of the text region, the size of the text region, and the direction or tilt angle of the text region. The relationship between the multiple text regions is a relationship based on visual layout. The relationship based on visual layout represents the positional proximity of the text regions, the vertical or horizontal alignment of the text lines, and the nesting relationship of the text regions. Based on the text parameters and spatial parameters corresponding to the multiple text regions, and the correlation between the multiple text regions, the target ticket information corresponding to the target ticket image is determined; The determination of target ticket information corresponding to the target ticket image, based on the text parameters and spatial parameters corresponding to the multiple text regions and the correlation between the multiple text regions, includes: determining fusion feature vectors corresponding to the multiple text regions based on the text parameters and spatial parameters corresponding to the multiple text regions and the correlation between the multiple text regions; determining a weighted graph corresponding to the target ticket image based on the fusion feature vectors corresponding to the multiple text regions, wherein the weighted graph includes nodes corresponding to key features and nodes corresponding to non-key features, wherein key features are features whose corresponding importance index is greater than a predetermined importance threshold, and non-key features are features whose corresponding importance index is less than or equal to the predetermined importance threshold; determining a nonlinear feature matrix corresponding to the target ticket image based on the weighted graph corresponding to the target ticket image, wherein the nonlinear feature matrix is ​​a matrix obtained by weighting the key features and reducing the weight of the non-key features; determining target classification labels corresponding to the multiple text regions based on the nonlinear feature matrix corresponding to the target ticket image; and determining target ticket information corresponding to the target ticket image based on the text parameters and target classification labels corresponding to the multiple text regions.

2. The method according to claim 1, characterized in that, The step of determining the weighted image corresponding to the target ticket image based on the fusion feature vectors corresponding to the multiple text regions includes: Identify the nodes corresponding to multiple fused feature vectors to obtain a node set; Based on the association relationships between each node in the node set and other nodes, determine the corresponding association weight values ​​between each node in the node set and other nodes to obtain an association weight set; Based on the association weight set, determine the neighboring nodes corresponding to each node in the node set to obtain the edge set; The weighted graph is constructed based on the node set, the associated weight set, and the edge set.

3. The method according to claim 1, characterized in that, The step of determining the nonlinear feature matrix corresponding to the target invoice image based on the weighted graph corresponding to the target invoice image includes: Determine the adjacency matrix and node feature matrix corresponding to the weighted graph; The adjacency matrix and the node feature matrix are input into a linear transformation layer to obtain a linear feature matrix, wherein the linear transformation layer includes a predetermined weight matrix; The linear feature matrix is ​​input into the activation function to obtain the nonlinear feature matrix.

4. The method according to claim 1, characterized in that, The acquisition of the target ticket image includes: Determine the initial ticket image; Based on the initial ticket image, determine whether the initial ticket image needs to be repaired; If the result is determined to be that the initial ticket image needs to be repaired, a defective sub-image and a neighboring sub-image are determined corresponding to the initial ticket image, wherein the neighboring sub-image is a sub-image in the initial ticket image whose proximity to the defective sub-image is greater than a proximity threshold. Based on the adjacent sub-images, the defective sub-image is repaired to obtain the target ticket image.

5. The method according to claim 4, characterized in that, The step of repairing the defective sub-image based on the adjacent sub-images to obtain the target ticket image includes: Based on the adjacent sub-images, the defective sub-images are repaired to obtain a repaired ticket image; Determine the physical degradation function, wherein the physical degradation function is a function used for defect simulation of the invoice image; The restored ticket image is input into the physical degradation function to obtain a degraded ticket image; Determine the similarity index between the degraded ticket image and the initial ticket image; If the similarity index is greater than the similarity threshold, the repaired ticket image is determined to be the target ticket image.

6. The method according to any one of claims 1 to 5, characterized in that, The step of determining the fusion feature vector corresponding to each of the multiple text regions based on their respective text parameters and spatial parameters, as well as the correlation between the multiple text regions, includes: Based on the spatial parameters corresponding to the multiple text regions, determine the spatial feature vectors corresponding to the multiple text regions respectively; Based on the text parameters corresponding to the multiple text regions, determine the text feature vectors corresponding to the multiple text regions respectively; Determine the style parameters corresponding to the multiple text regions respectively; Based on the style parameters corresponding to the multiple text regions, determine the visual feature vectors corresponding to the multiple text regions respectively; Based on the text feature vector, the visual feature vector, and the spatial feature vector, a fusion feature vector corresponding to each of the multiple text regions is determined.

7. A device for extracting ticket information, characterized in that, include: An acquisition module is used to acquire a target ticket image, wherein the target ticket image includes multiple text regions, and each of the multiple text regions includes corresponding text parameters; The first determining module is used to determine the spatial parameters corresponding to the plurality of text regions respectively, and to determine the relationship between the plurality of text regions. The spatial parameters include the bounding box coordinates of the text region, the center point position of the text region, the size of the text region, and the direction or tilt angle of the text region. The relationship between the plurality of text regions is a relationship based on visual layout. The relationship based on visual layout represents the positional proximity of the text regions, the vertical or horizontal alignment of the text lines, and the nesting relationship of the text regions. The second determining module is used to determine the target ticket information corresponding to the target ticket image based on the text parameters and spatial parameters corresponding to the plurality of text regions respectively, as well as the correlation between the plurality of text regions, so as to realize the ticket information extraction method as described in any one of claims 1 to 6.

8. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the invoice information extraction method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the ticket information extraction method as described in any one of claims 1 to 6.

Images