Bill information extraction method and device and electronic equipment

Abstract

The invention discloses a bill information extraction method and device and electronic equipment. The method comprises the steps that a target bill image is acquired, the target bill image comprises a plurality of text areas, and the plurality of text areas respectively comprise corresponding text parameters; determining spatial parameters respectively corresponding to the plurality of text regions, and determining an association relationship among the plurality of text regions; and determining target bill information corresponding to the target bill image according to the text parameters and the spatial parameters respectively corresponding to the plurality of text areas and the association relationship among the plurality of text areas. According to the invention, the technical problem that the bill information cannot be efficiently and accurately extracted when the bill information is extracted in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method, device and electronic equipment for extracting bill information. Background Art

[0002] In the related technologies, since bill types vary greatly in format and field content, and traditional OCR technology often has difficulty in effectively dealing with a variety of bill types, when extracting bill information, due to the diversity of bill types and other issues, there is a problem of being unable to efficiently and accurately extract bill information.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a method, device and electronic device for extracting bill information, so as to at least solve the technical problem in the related art that the bill information cannot be extracted efficiently and accurately.

[0005] According to one aspect of an embodiment of the present invention, a method for extracting bill information is provided, comprising: acquiring a target bill image, wherein the target bill image includes a plurality of text areas, and the plurality of text areas respectively include corresponding text parameters; determining spatial parameters corresponding to the plurality of text areas respectively, and determining an association relationship between the plurality of text areas; determining target bill information corresponding to the target bill image based on the text parameters and spatial parameters corresponding to the plurality of text areas respectively, and the association relationship between the plurality of text areas.

[0006] Optionally, according to the text parameters and spatial parameters respectively corresponding to the multiple text areas, and the association relationship between the multiple text areas, the target bill information corresponding to the target bill image is determined, including: according to the text parameters and spatial parameters respectively corresponding to the multiple text areas, and the association relationship between the multiple text areas, the fused feature vectors respectively corresponding to the multiple text areas are determined; according to the fused feature vectors respectively corresponding to the multiple text areas, a weighted graph corresponding to the target bill image is determined, wherein the weighted graph includes nodes corresponding to key features and nodes corresponding to non-key features, the key features are features whose corresponding importance index is greater than a predetermined importance threshold, and the non-key features are features whose corresponding importance index is less than or equal to a predetermined importance threshold; according to the weighted graph corresponding to the target bill image, a nonlinear feature matrix corresponding to the target bill image is determined, wherein the nonlinear feature matrix is ​​a matrix obtained by weighting the key features and reducing the weights of the non-key features; according to the nonlinear feature matrix corresponding to the target bill image, the target classification labels respectively corresponding to the multiple text areas are determined; according to the text parameters and target classification labels respectively corresponding to the multiple text areas, the target bill information corresponding to the target bill image is determined.

[0007] Optionally, determining the weighted graph corresponding to the target bill image based on the fused feature vectors corresponding to the multiple text areas respectively includes: determining the nodes corresponding to the multiple fused feature vectors respectively to obtain a node set; determining the association weight values ​​corresponding to each node in the node set and other nodes based on the association relationship between each node in the node set and other nodes to obtain an association weight set; determining the neighbor nodes corresponding to each node in the node set respectively based on the association weight set to obtain an edge set; constructing the weighted graph based on the node set, the association weight set, and the edge set.

[0008] Optionally, determining the nonlinear feature matrix corresponding to the target bill image based on the weighted graph corresponding to the target bill image includes: determining an adjacency matrix and a node feature matrix corresponding to the weighted graph; inputting the adjacency matrix and the node feature matrix into a linear transformation layer to obtain a linear feature matrix, wherein the linear transformation layer includes a predetermined weight matrix; inputting the linear feature matrix into an activation function to obtain a nonlinear feature matrix.

[0009] Optionally, acquiring the target bill image includes: determining an initial bill image; determining whether the initial bill image needs to be repaired based on the initial bill image; when the determination result is that the initial bill image needs to be repaired, determining a defect sub-image corresponding to the initial bill image, and an adjacent sub-image, wherein the adjacent sub-image is a sub-image in the initial bill image whose degree of proximity to the defect sub-image is greater than a proximity threshold; based on the adjacent sub-image, repairing the defect sub-image to obtain the target bill image.

[0010] Optionally, repairing the defective sub-image based on the adjacent sub-image to obtain the target bill image includes: repairing the defective sub-image based on the adjacent sub-image to obtain a repaired bill image; determining a physical degradation function, wherein the physical degradation function is a function for defect simulation of a bill image; inputting the repaired bill image into the physical degradation function to obtain a degraded bill image; determining a similarity index between the degraded bill image and the initial bill image; and when the similarity index is greater than a similarity threshold, determining that the repaired bill image is the target bill image.

[0011] Optionally, the method of determining fused feature vectors corresponding to the multiple text areas respectively based on the text parameters and spatial parameters respectively corresponding to the multiple text areas, and the association relationship between the multiple text areas, includes: determining the spatial feature vectors corresponding to the multiple text areas respectively based on the spatial parameters respectively corresponding to the multiple text areas; determining the text feature vectors corresponding to the multiple text areas respectively based on the text parameters respectively corresponding to the multiple text areas; determining the style parameters respectively corresponding to the multiple text areas; determining the visual feature vectors corresponding to the multiple text areas respectively based on the style parameters respectively corresponding to the multiple text areas; and determining the fused feature vectors corresponding to the multiple text areas respectively based on the text feature vectors, the visual feature vectors and the spatial feature vectors.

[0012] According to one aspect of an embodiment of the present invention, a bill information extraction device is provided, comprising: an acquisition module for acquiring a target bill image, wherein the target bill image comprises a plurality of text areas, and the plurality of text areas respectively comprise corresponding text parameters; a first determination module for determining spatial parameters respectively corresponding to the plurality of text areas, and determining an association relationship between the plurality of text areas; and a second determination module for determining target bill information corresponding to the target bill image based on the text parameters and spatial parameters respectively corresponding to the plurality of text areas, and the association relationship between the plurality of text areas.

[0013] According to one aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement any of the above-mentioned ticket information extraction methods.

[0014] According to one aspect of an embodiment of the present invention, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute any of the above-mentioned ticket information extraction methods.

[0015] In an embodiment of the present invention, a target bill image is obtained, wherein the target bill image includes multiple text areas, and the multiple text areas respectively include corresponding text parameters; the spatial parameters corresponding to the multiple text areas are determined, and the association relationship between the multiple text areas is determined; based on the text parameters and spatial parameters corresponding to the multiple text areas, and the association relationship between the multiple text areas, the target bill information corresponding to the target bill image is determined. By obtaining a target bill image including multiple text areas, and the multiple text areas respectively include corresponding text parameters, an accurate data basis is provided for the subsequent bill information extraction process, thereby facilitating the subsequent accurate positioning of key information in the target bill image based on the text areas of the target bill image. By determining the spatial parameters of each of the multiple text areas in the target bill image, it is helpful to accurately identify and locate the key information in the bill in the subsequent process, and by determining the association relationship between these text areas, it is helpful to accurately understand the logical connection between the text areas, so that the bill information can be accurately identified and parsed. By utilizing the text parameters and spatial parameters extracted from the text area, it is helpful to achieve targeted and efficient information extraction. Combined with the correlation between text areas, the target bill information corresponding to the target bill image is determined, and the unstructured information in the bill image is converted into structured data, which reduces misidentification in the bill information extraction process and improves the efficiency and accuracy of bill information extraction processing, thereby solving the technical problem in related technologies that bill information cannot be extracted efficiently and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0017] Figure 1 is a flow chart of a method for extracting bill information according to an embodiment of the present invention;

[0018] Figure 2 is a flow chart of a method for extracting bill information in an optional embodiment of the present invention;

[0019] Figure 3 is a technical framework diagram of image restoration in an optional embodiment of the present invention;

[0020] Figure 4 is a flowchart of image restoration in an optional embodiment of the present invention;

[0021] Figure 5 It is a structural block diagram of a bill information extraction device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0025] OCR: OCR (Optical Character Recognition) is the abbreviation of optical character recognition. It is a technology that converts text in an image into editable and searchable text. OCR technology can be used in document scanning, image processing, automated data entry and other fields.

[0026] CRNN: CRNN (Convolutional Recurrent Neural Network) is a deep learning model that combines convolutional neural network (CNN) and recurrent neural network (RNN). It is usually used for sequence generation tasks, such as text recognition, image description generation, etc. CRNN can capture local features in the image (through CNN) and process sequence data (through RNN), thereby achieving accurate recognition of text in the image.

[0027] Text-Perceptron-Det: Text-Perceptron-Det is a text detector in OCR (Optical Character Recognition). Its main function is to detect text areas in images and extract text information from images. It can learn and recognize text features in images through deep learning techniques such as convolutional neural networks (CNN).

[0028] Word2Vec: Word2Vec is a word embedding technology used for natural language processing. It maps each word in the vocabulary into a vector space of fixed size, so that semantically similar words are close in distance in the vector space. Word2Vec can capture the semantic relationship between words and provide an effective feature representation for natural language processing tasks.

[0029] GCN: GCN (Graph Convolutional Network) is the abbreviation of graph convolutional network. It is a neural network model specially designed for processing graph structure data. By applying convolution operations on graph structure data, GCN can capture local and global features between nodes, thereby achieving tasks such as classification and regression of graph structure data.

[0030] GNN: GNN (Graph Neural Network) is the abbreviation of Graph Neural Network. It is a neural network model specifically designed to process graph structure data. GNN captures information in the graph structure by iteratively updating the representation of nodes, thereby achieving tasks such as classification, regression, and link prediction of graph structure data.

[0031] GAN: GAN (Generative Adversarial Network) is the abbreviation of Generative Adversarial Network. It is a deep learning model consisting of two networks: the generator and the discriminator. The generator is responsible for generating realistic data samples, while the discriminator is responsible for distinguishing between real data and generated data. Through continuous adversarial training of the two networks, GAN can generate more and more realistic data samples, which can be applied to image generation, speech synthesis and other fields.

[0032] PyTorch: PyTorch is an open-source deep learning framework that provides all the basic building blocks required to build and train various neural networks such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), etc.

[0033] TensorFlow: TensorFlow is an open source machine learning framework that provides a flexible architecture to define complex data flow graphs and efficiently execute them on multiple platforms. TensorFlow supports GPU and TPU accelerated computing and provides multiple APIs, such as the Keras high-level interface, to facilitate developers to quickly build models.

[0034] Softmax classifier: Softmax classifier is a commonly used multi-classification model. It supports multi-classification tasks by mapping input features to probability distributions of multiple categories. Softmax classifier is usually used in the output layer of neural networks to convert the output of neural networks into probability distributions, thereby achieving category prediction.

[0035] pix2pix: pix2pix is ​​an image conversion model based on conditional generative adversarial networks (Conditional GANs), which aims to convert one image into another image. It is suitable for image-to-image conversion tasks. pix2pix provides an input image and allows the generator to learn how to generate an output image with specific target characteristics from the image. The discriminator is used to distinguish between the generated image and the real target image.

[0036] Cycle GAN: Cycle GAN (Cycle-Consistent Generative Adversarial Network) is a deep learning model for unsupervised image translation, that is, a variant of the Generative Adversarial Network (GAN) for image-to-image translation tasks. Cycle GAN can convert images of one style into another style without paired training samples.

[0037] VRD: VRD (Visual Relationship Detection) is the abbreviation of visual relationship detection. It is a computer vision task that aims to identify the visual relationship between objects in an image, such as "a person riding a bicycle", "a bird standing on a tree", etc. VRD technology can be applied to image understanding, scene analysis and other fields.

[0038] F1 value: F1 value is one of the commonly used evaluation indicators in classification tasks. It is the harmonic mean of precision and recall, and is used to comprehensively evaluate the performance of the model. The F1 value ranges from 0 to 1, and the higher the value, the better the performance of the model. The F1 value can be applied to both binary and multi-classification tasks.

[0039] Example 1

[0040] According to an embodiment of the present invention, an embodiment of a method for extracting bill information is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0041] Figure 1 is a flow chart of a method for extracting bill information according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0042] S102, acquiring a target bill image, wherein the target bill image includes a plurality of text regions, each of which includes a corresponding text parameter;

[0043] In step S102 provided in the present application, a target bill image is acquired.

[0044] The target bill image is a bill image obtained by scanning or photographing and for which information extraction is required. The target bill image can be any type of financial bill, such as an invoice, receipt, train ticket, flight itinerary, taxi invoice, toll invoice, passenger invoice, etc.

[0045] Among them, a text area is involved, which is the area corresponding to the text content on the target bill image. The target bill image contains multiple text areas, for example, text areas such as "invoice code", "invoice number", "date", and "amount" on an invoice.

[0046] Among them, a text parameter is involved, and the text parameter is a specific parameter corresponding to the text content in the text area, for example, the text content in the text area is "invoice number", the text content in the text area is "unit price", etc.

[0047] By acquiring a target bill image including multiple text areas, each of which includes corresponding text parameters, an accurate data basis is provided for the subsequent bill information extraction process, which helps to accurately locate the key information in the target bill image based on the text area of ​​the target bill image, thereby helping to reduce misidentification in the information extraction process, improve the efficiency and accuracy of the bill information extraction process, and help to solve the technical problem of not being able to extract bill information efficiently and accurately.

[0048] It should be noted that there is no specific limitation on the method of acquiring the target bill image, and the method can be customized according to the actual application and scenario.

[0049] S104, determining the spatial parameters corresponding to the multiple text regions respectively, and determining the association relationship between the multiple text regions;

[0050] In step S104 provided in the present application, the spatial parameters corresponding to the multiple text regions are determined, and the association relationship between the multiple text regions is determined.

[0051] Among them, the spatial parameters are involved, which are the relevant parameters corresponding to the geometric position of the text area in the target bill image. The spatial parameters may include the bounding box coordinates of the text area (i.e., the coordinates of the upper left corner and the lower right corner), the center point position of the text area, the size of the text area (width and height), the direction or tilt angle of the text area, etc. For example, for a text area containing "invoice number", its spatial parameters may be the upper left corner coordinates (120, 50) and the lower right corner coordinates (300, 100), indicating the specific position and size of the text area in the bill image.

[0052] Among them, an association relationship is involved, which is the relationship between multiple text regions in the target bill image. The association relationship can be based on visual layout, such as the positional proximity of text regions, the vertical or horizontal alignment of text lines, and the nested relationship of text regions (for example, a text region may be contained in another text region), or it can be based on semantics. For example, some text regions are usually logically accompanied by other specific text regions (such as "amount" is usually adjacent to "currency"). For example, in an invoice, the "invoice number" text region may be closely adjacent to the "invoice code" text region in terms of position, and also closely related in terms of semantics.

[0053] By determining the spatial parameters (such as coordinates and size) of multiple text regions in the target bill image, it is helpful to accurately identify and locate key information in the bill, and the position of each text region can be accurately identified even in a complex layout. By determining the association between these text regions, which can include proximity in visual layout and semantic relevance, it is helpful to accurately understand the logical connection between text regions, so that accurate identification and parsing of bill information can be achieved, especially for bills with non-standard or complex formats. By determining the spatial parameters and associations of multiple text regions, it is possible to accurately identify and extract bill information, reduce misidentification and missed identification, and cope with bills of various formats, thereby improving the versatility and scope of application of the method.

[0054] It should be noted that there is no specific limitation on the method of determining the spatial parameters corresponding to the multiple text regions and the method of determining the association relationship between the multiple text regions, and the settings can be customized according to the actual application and scenario.

[0055] S106, determining target bill information corresponding to the target bill image according to the text parameters and space parameters respectively corresponding to the plurality of text regions, and the association relationship between the plurality of text regions.

[0056] In step S106 provided in the present application, target bill information corresponding to the target bill image is determined.

[0057] Among them, the target bill information is involved, which is the key data or information extracted from the target bill image. The target bill information is the main content of the bill and is used to prove the legality of transactions, payments, etc. For example, the extracted target bill information may include "invoice code: 1234567890", "invoice number: 0987654321", "amount: 234.56 yuan", "date: April 1, 2023", etc.

[0058] By utilizing the text parameters and spatial parameters extracted from the text area, it is helpful to achieve targeted and efficient information extraction. Combined with the correlation between text areas, the target bill information corresponding to the target bill image is determined, thereby improving the accuracy of extracted information, reducing false positives and negatives, and realizing the conversion of unstructured information in the bill image into structured data, thereby solving the technical problem of being unable to extract bill information efficiently and accurately.

[0059] It should be noted that the method for determining the target bill information corresponding to the target bill image based on the text parameters and space parameters corresponding to multiple text areas, and the association relationship between the multiple text areas, is not specifically limited here and can be customized according to actual applications and scenarios.

[0060] Through the above steps S102-S106, a target bill image is obtained, wherein the target bill image includes multiple text areas, each of which includes corresponding text parameters; the spatial parameters corresponding to the multiple text areas are determined, and the association relationship between the multiple text areas is determined; based on the text parameters and spatial parameters corresponding to the multiple text areas, and the association relationship between the multiple text areas, the target bill information corresponding to the target bill image is determined. By obtaining a target bill image including multiple text areas, and each of which includes corresponding text parameters, an accurate data basis is provided for the subsequent bill information extraction process, thereby facilitating the subsequent accurate positioning of key information in the target bill image based on the text areas of the target bill image. By determining the spatial parameters of each of the multiple text areas in the target bill image, it is helpful to accurately identify and locate the key information in the bill in the subsequent process, and by determining the association relationship between these text areas, it is helpful to accurately understand the logical connection between the text areas, so that the bill information can be accurately identified and parsed. By utilizing the text parameters and spatial parameters extracted from the text area, it is helpful to achieve targeted and efficient information extraction. Combined with the correlation between text areas, the target bill information corresponding to the target bill image is determined, and the unstructured information in the bill image is converted into structured data, which reduces misidentification in the bill information extraction process and improves the efficiency and accuracy of bill information extraction processing, thereby solving the technical problem in related technologies that bill information cannot be extracted efficiently and accurately.

[0061] As an optional embodiment, the target bill information corresponding to the target bill image is determined based on the text parameters and spatial parameters corresponding to the multiple text areas, as well as the association relationship between the multiple text areas, including: determining the fused feature vectors corresponding to the multiple text areas according to the text parameters and spatial parameters corresponding to the multiple text areas, as well as the association relationship between the multiple text areas; determining the weighted graph corresponding to the target bill image according to the fused feature vectors corresponding to the multiple text areas, wherein the weighted graph includes nodes corresponding to key features and nodes corresponding to non-key features, the key features are features whose corresponding importance index is greater than a predetermined importance threshold, and the non-key features are features whose corresponding importance index is less than or equal to the predetermined importance threshold; determining the nonlinear feature matrix corresponding to the target bill image according to the weighted graph corresponding to the target bill image, wherein the nonlinear feature matrix is ​​a matrix obtained by weighting the key features and reducing the weights of the non-key features; determining the target classification labels corresponding to the multiple text areas according to the nonlinear feature matrix corresponding to the target bill image; determining the target bill information corresponding to the target bill image according to the text parameters and target classification labels corresponding to the multiple text areas.

[0062] In this embodiment, specific steps of determining the target bill information corresponding to the target bill image according to the text parameters and space parameters respectively corresponding to the multiple text regions and the association relationship between the multiple text regions are described.

[0063] Among them, a fused feature vector is involved, which is a vector used to characterize the comprehensive features of a text area, and can be a high-dimensional vector obtained by fusing text parameters, space parameters, style parameters, etc. For example, a fused feature vector of an "amount" text area may include the amount value (text parameter), the position of the text area in the image (space parameter), and the image features of the text area (such as font, color, texture, etc.), forming a vector that can fully describe the area.

[0064] Among them, a weighted graph is involved, which is a graph used to represent the relationship between different text areas in a bill image, and the nodes in the weighted graph represent text areas, and the weights of the edges are determined by the association relationship between the text areas.

[0065] Among them, key features are involved, which are features that have important contributions to identification and classification in the process of bill information extraction. For example, fields such as "amount" and "date" are key features for bill information extraction because they are directly related to the validity of the bill.

[0066] Among them, non-key features are involved, which are relatively minor features for identification and classification in the process of bill information extraction. For example, advertising text or decorative patterns in bills are usually regarded as non-key features because they are not directly involved in the extraction of key information.

[0067] Among them, nodes are involved, which represent the text area in the bill image. Each node carries a fused feature vector for describing the comprehensive information of the text area.

[0068] Among them, the importance index is involved, which is a quantitative index used to measure the importance of the features corresponding to the text area in the information extraction and classification tasks. For example, the higher the importance index corresponding to a feature in the text area, the more important the feature is in analyzing and extracting bill information.

[0069] Among them, a predetermined importance threshold is involved, which is a preset threshold used to distinguish key features from non-key features. For example, if the importance index is higher than the predetermined importance threshold, the feature is considered a key feature; if it is lower than or equal to the predetermined importance threshold, it is considered a non-key feature.

[0070] Among them, a nonlinear feature matrix is ​​involved. This nonlinear feature matrix is ​​a feature matrix obtained after being processed by an activation function, and is used to capture the complex relationship between features. This nonlinear feature matrix is ​​obtained by weighting key features and down-weighting non-key features, reflecting the nonlinear relationship and importance of each node feature in the weighted graph.

[0071] Among them, weighted processing is involved, and the weighted processing is a processing of increasing the weight corresponding to the key feature to reflect the importance of the key feature in extracting the bill information.

[0072] Among them, the downgrading process is involved, which is to reduce the weights corresponding to non-critical features. The impact of non-critical features can be weakened by reducing the weights to ensure that the key features directly related to the bill information are focused on during the analysis process.

[0073] Among them, the target classification label is involved, which is the category label that each text area should be classified into, such as "invoice code", "invoice number", "amount", "date", etc., which is used to convert the text area in the target bill image into structured information.

[0074] In the steps involved in this embodiment, first, a fused feature vector corresponding to each text area is determined, and the fused feature vector comprehensively considers the text parameters and spatial parameters corresponding to the text area, as well as the correlation relationship with other text areas. Then, based on these fused feature vectors, a weighted graph is constructed, in which the nodes in the graph represent text areas, and the weights of the edges reflect the correlation strength between text areas. When constructing the weighted graph, key features and non-key features are distinguished. Key features correspond to features with an importance index higher than a predetermined importance threshold, while non-key features correspond to features lower than or equal to the threshold. Then, the weighted graph is processed to obtain a nonlinear feature matrix, which weights the key features and reduces the weights of non-key features. Finally, based on the nonlinear feature matrix, the target classification label corresponding to each text area is determined, and then the target bill information corresponding to the target bill image is determined.

[0075] By comprehensively considering the text parameters and spatial parameters corresponding to multiple text regions, as well as the correlation between multiple text regions, the fused feature vectors corresponding to multiple text regions are determined, which helps to accurately identify the real text information corresponding to multiple text regions and reduce misidentification in the process of bill information extraction. Based on these fused feature vectors, a weighted graph is constructed. The nodes in the graph represent text regions, and the weights of the edges reflect the correlation strength between text regions. When constructing the weighted graph, key features and non-key features are distinguished, which solves the problem of different importance of different features in the information extraction process. Through weighted processing and weight reduction processing, the prominence of key features is ensured, and the accuracy of information extraction is improved. By processing the weighted graph, a nonlinear feature matrix is ​​obtained, which can more accurately identify and classify the key information in the bill, reduce the error in the information extraction process, and help to improve the comprehensiveness of bill information extraction in the future. By determining the target classification label corresponding to each text region based on the nonlinear feature matrix, and then determining the target bill information corresponding to the target bill image, the problem of converting unstructured information in the bill image into structured data is effectively solved, and the efficiency and accuracy of data processing are improved.

[0076] As an optional embodiment, a weighted graph corresponding to a target bill image is determined based on fused feature vectors corresponding to multiple text areas, including: determining nodes corresponding to multiple fused feature vectors to obtain a node set; determining the association weight values ​​corresponding to each node in the node set and other nodes based on the association relationship between each node in the node set and other nodes to obtain an association weight set; determining neighbor nodes corresponding to each node in the node set to obtain an edge set based on the association weight set; and constructing a weighted graph based on the node set, the association weight set, and the edge set.

[0077] In this embodiment, the specific steps of determining a weighted graph corresponding to a target bill image based on fused feature vectors corresponding to a plurality of text regions are described.

[0078] Among them, a node set is involved, which is a set of nodes corresponding to multiple text regions. Each node represents a text region in the target bill image, and each node carries a fused feature vector. For example, if there are three text regions of "invoice code", "invoice number" and "amount" in the target bill image, the node set will contain these three nodes.

[0079] Among them, the association weight value is involved. The association weight value is obtained based on the association relationship between each node and other nodes. It measures the strength of the association between two nodes and is used to form the weight of the edge in the weighted graph. For example, if the "invoice code" and "invoice number" are located adjacent to each other in the image and the semantic meanings corresponding to the text content are closely related, the association weight value between them will be higher.

[0080] Among them, an association weight set is involved, which is a set of association weight values. For example, for a weighted graph containing three nodes ("invoice code", "invoice number", "amount"), the association weight set can be the association weight values ​​of the three nodes in pairs.

[0081] Among them, a neighbor node is involved, and the neighbor node is a node that is directly connected to the current node through an edge in the weighted graph.

[0082] Among them, an edge set is involved, which is a set constructed by edges representing the associations between nodes, obtained based on the neighbor nodes corresponding to each node in the node set.

[0083] In the steps involved in this embodiment, first, the nodes corresponding to each fused feature vector are obtained to obtain a node set, each node representing a text area. Then, based on the association relationship between each node and other nodes, the association weight value corresponding to each node in the node set and other nodes is determined to obtain an association weight set. Next, according to the weight value in the association weight set, the neighbor nodes corresponding to each node in the node set are respectively constructed to construct an edge set. Finally, according to the node set, the association weight set, and the edge set, a weighted graph corresponding to the target bill image is constructed.

[0084] Through the above steps, the complex text area relationship in the bill image is modeled as a weighted graph, which helps to accurately identify and understand the layout and information structure in the target bill image, and facilitates the subsequent process of extracting the target bill information corresponding to the target bill image. It is convenient to accurately capture the complex associations between text areas, thereby helping to accurately identify and classify each text area, improve the accuracy and efficiency of information extraction, and further help solve the technical problem of not being able to efficiently and accurately extract bill information due to the diversity of bill types when extracting bill information.

[0085] As an optional embodiment, based on the weighted graph corresponding to the target bill image, a nonlinear feature matrix corresponding to the target bill image is determined, including: determining an adjacency matrix and a node feature matrix corresponding to the weighted graph; inputting the adjacency matrix and the node feature matrix into a linear transformation layer to obtain a linear feature matrix, wherein the linear transformation layer includes a predetermined weight matrix; inputting the linear feature matrix into an activation function to obtain a nonlinear feature matrix.

[0086] In this embodiment, the specific steps of determining the nonlinear feature matrix corresponding to the target bill image based on the weighted graph corresponding to the target bill image are described.

[0087] Among them, the adjacency matrix is ​​involved, which is a matrix used to represent the connection relationship between nodes in a weighted graph. In a weighted graph, the elements of the adjacency matrix represent the weights of the edges.

[0088] Among them, a node feature matrix is ​​involved, which is a matrix used to represent the features corresponding to multiple text regions (nodes). Each row of the node feature matrix corresponds to a node in the weighted graph, and each column corresponds to the feature of the node.

[0089] Among them, a linear transformation layer is involved. The linear transformation layer is a layer included in the deep learning model, which is used to process input data (such as adjacency matrix and node feature matrix) through linear transformation. The linear transformation can be matrix multiplication and vector addition.

[0090] Among them, a linear feature matrix is ​​involved. The linear feature matrix is ​​a feature matrix obtained after the node feature matrix is ​​linearly transformed. The linear feature matrix retains the feature information of the node.

[0091] Among them, a predetermined weight matrix is ​​involved, which is a parameter matrix in the linear transformation layer. The predetermined weight matrix is ​​learned through a training process and is used to adjust the weights of node features to better capture the correlation relationship between nodes.

[0092] Among them, the activation function is involved, which is a nonlinear function used to introduce nonlinear transformation, so that the deep learning model can learn more complex feature relationships.

[0093] In the steps involved in this embodiment, first, the adjacency matrix and the node feature matrix corresponding to the weighted graph are determined, the adjacency matrix represents the connection relationship and its weight between the nodes, and the node feature matrix contains the feature information of each node. Then, the adjacency matrix and the node feature matrix are input to the linear transformation layer, and linear transformation (such as matrix multiplication and vector addition) is performed to obtain a linear feature matrix. The predetermined weight matrix in the linear transformation layer is a parameter matrix optimized through training. Finally, the linear feature matrix is ​​input to the activation function, nonlinear transformation is introduced, and a nonlinear feature matrix is ​​generated.

[0094] Through the above steps, the adjacency matrix and the node feature matrix corresponding to the weighted graph are linearly transformed (such as matrix multiplication and vector addition) to obtain a linear feature matrix, which helps the model better understand the relationship between nodes. The linear feature matrix is ​​input into the activation function, and a nonlinear transformation is introduced to generate a nonlinear feature matrix, which helps to capture the complex relationship between nodes, thereby strengthening the correlation between features, improving the model's adaptability to complex bill formats and information structures, and helping to accurately identify and classify bill information in a wider range of scenarios. The efficiency and accuracy of target bill information extraction are improved, which in turn helps to solve the technical problem of not being able to efficiently and accurately extract bill information when extracting bill information.

[0095] As an optional embodiment, obtaining a target bill image includes: determining an initial bill image; determining whether the initial bill image needs to be repaired based on the initial bill image; when the determination result is that the initial bill image needs to be repaired, determining a defect sub-image corresponding to the initial bill image, and an adjacent sub-image, wherein the adjacent sub-image is a sub-image in the initial bill image whose degree of proximity to the defect sub-image is greater than a proximity threshold; based on the adjacent sub-image, repairing the defect sub-image to obtain the target bill image.

[0096] In this embodiment, the specific steps of acquiring the target bill image are described.

[0097] The initial bill image is a preliminarily acquired bill image obtained by scanning or photographing without any processing. For example, a paper invoice image just scanned by a scanner or a receipt image taken by a mobile phone camera can be regarded as an initial bill image.

[0098] Among them, it involves a determination result, which is a judgment result of whether the initial bill image needs to be repaired. For example, the determination result may be "the image is blurred and needs to be repaired" or "the image is clear and does not need to be repaired".

[0099] Among them, defect sub-images are involved, which are image areas with defects (such as blur, damage, or missing parts) in the initial bill image. For example, the invoice number area may become blurred due to shadows or paper wear during scanning, and this part of the image area can be regarded as a defect sub-image.

[0100] Among them, the adjacent sub-image is involved, and the adjacent sub-image is an image area that is adjacent to the defective sub-image in the initial bill image and whose proximity is greater than a preset proximity threshold. For example, if the "invoice number" area is a defective sub-image, then if the "invoice code" area is relatively close to the "invoice number" area and its clarity is higher than the preset proximity threshold, then the "invoice code" area can be regarded as an adjacent sub-image and can be used to assist in repairing the "invoice number" area.

[0101] Among them, the proximity threshold is involved, which is a pre-set threshold used to determine whether two image regions are adjacent. For example, if the Euclidean distance between two image regions is less than 50 pixels, then the proximity of the two regions is greater than the proximity threshold, and they can be considered to be adjacent and can be used as information reference in the restoration task.

[0102] In the steps involved in this embodiment, the initial bill image is first determined, and then the quality of this initial bill image is analyzed to determine whether it needs to be repaired. If it is determined that the image is blurred, damaged or has other defects, that is, the result is determined to be in need of repair, then the defective sub-image area in the initial bill image and the adjacent sub-images adjacent to the defective sub-image with a clarity higher than a preset adjacent threshold are determined, and these clear adjacent sub-images are used to help repair the blurred or missing parts of the defective sub-image. After the repair is completed, the obtained image is the target bill image.

[0103] Through the above steps, before extracting the information of the bill image, the imaging quality of the acquired initial bill image (such as whether it is blurred, damaged or missing) is first judged, thereby avoiding the inability to accurately extract the bill information due to the imaging quality problem of the bill image. By determining the defective sub-image area and the adjacent sub-image in the initial bill image when the result is determined to be in need of repair, it is helpful to accurately restore the image content of the defective area, thereby ensuring that the subsequent extraction of the marked bill information can be carried out on the basis of high-quality images, reducing information extraction errors and omissions caused by bill image quality problems, and improving the efficiency and accuracy of bill information extraction, thereby solving the problem in the related technology that when extracting bill information, the bill information cannot be extracted efficiently and accurately due to bill damage and wear.

[0104] As an optional embodiment, a defective sub-image is repaired based on adjacent sub-images to obtain a target bill image, including: repairing the defective sub-image based on adjacent sub-images to obtain a repaired bill image; determining a physical degradation function, wherein the physical degradation function is a function for simulating defects in a bill image; inputting the repaired bill image into the physical degradation function to obtain a degraded bill image; determining a similarity index between the degraded bill image and the initial bill image; and determining that the repaired bill image is a target bill image when the similarity index is greater than a similarity threshold.

[0105] In this embodiment, the specific steps of repairing a defective sub-image based on adjacent sub-images to obtain a target bill image are described.

[0106] Among them, it involves repairing bill images, which are bill images obtained after repair processing through image repair algorithms and other technologies. For example, if the "amount" area in the initial bill image is blurred, after repair, this area becomes clear and the number can be accurately recognized, then the repaired image is the repaired bill image.

[0107] Among them, a physical degradation function is involved, which is a mathematical function used to simulate the image degradation process in reality. The physical degradation function can reintroduce the degradation effect of the repaired image to verify whether the restoration effect is natural and whether it can still maintain the clarity and information integrity before restoration after the degradation process.

[0108] Among them, a degraded bill image is involved, which is a bill image obtained after the restoration bill image is simulated by a physical degradation function to simulate the degradation effect. For example, the image obtained by processing the "amount" area after restoration through a fuzzy simulation function is a degraded bill image.

[0109] Among them, a similarity index is involved, which is an index used to quantify the similarity between the degraded bill image and the initial bill image, and can be calculated based on pixel values, etc. For example, the similarity index can evaluate whether the restoration effect is natural and consistent with the original image by determining the difference in pixel values ​​between the degraded bill image and the initial bill image.

[0110] Among them, a similarity threshold is involved, which is a pre-set threshold used to determine whether the degraded bill image is sufficiently similar to the initial bill image. For example, if the similarity threshold is set to 0.9, then when the similarity index is greater than 0.9, the restoration effect meets the requirements.

[0111] In the steps involved in this embodiment, first, the defective sub-image is repaired based on the adjacent sub-images to obtain a repaired bill image. Then, a physical degradation function is determined to simulate the degradation process of the image in reality, such as blur, noise, insufficient light, etc. The repaired repaired bill image is input into the physical degradation function to obtain a degraded bill image after degradation. Next, the similarity index between the degraded bill image and the initial bill image is determined to evaluate the repair effect. If the similarity index is greater than the preset similarity threshold, it indicates that the repaired repaired bill image retains the original information while maintaining the appearance consistency and information integrity with the initial bill image. In this case, the repaired repaired bill image is the final target bill image.

[0112] Through the above steps, the defective sub-image is repaired according to the adjacent sub-images to obtain the repaired bill image, and the repaired bill image is input into the physical degradation function to obtain the degraded bill image. The degradation is simulated by the physical degradation function to ensure the performance and reliability of the repaired bill image in the actual application scenario, which is helpful to verify the naturalness and consistency of the repair effect. By determining the similarity index between the degraded bill image and the initial bill image; when the similarity index is greater than the similarity threshold, the repaired bill image is determined to be the target bill image, which ensures the naturalness of the target bill image and its consistency with the initial bill image, avoiding image distortion or information loss caused by repair, thereby providing a more reliable image basis for subsequent information extraction and improving the efficiency and accuracy of the entire bill information extraction process.

[0113] As an optional embodiment, based on the text parameters and spatial parameters corresponding to the multiple text areas, as well as the correlation between the multiple text areas, fused feature vectors corresponding to the multiple text areas are determined, including: based on the spatial parameters corresponding to the multiple text areas, determining the spatial feature vectors corresponding to the multiple text areas; based on the text parameters corresponding to the multiple text areas, determining the text feature vectors corresponding to the multiple text areas; determining the style parameters corresponding to the multiple text areas; determining the visual feature vectors corresponding to the multiple text areas based on the style parameters corresponding to the multiple text areas; determining the fused feature vectors corresponding to the multiple text areas based on the text feature vectors, the visual feature vectors and the spatial feature vectors.

[0114] In this embodiment, specific steps of determining fused feature vectors corresponding to a plurality of text regions respectively according to text parameters and space parameters corresponding to the plurality of text regions respectively and the association relationship between the plurality of text regions are described.

[0115] Among them, a spatial feature vector is involved, which is obtained according to the spatial parameters and is used to represent the position and size of the text area in the target bill image. The spatial feature vector may include the center coordinates, width and height, tilt angle, etc. of the text area.

[0116] Among them, a text feature vector is involved. The text feature vector is obtained according to text parameters and is used to represent the feature vector of the text content corresponding to the text area, and contains the semantic information of the text content in the text area.

[0117] Among them, style parameters are involved. The style parameters are obtained according to text parameters and are used to reflect the visual presentation characteristics of the text area, including background color, border style, etc.

[0118] Among them, a visual feature vector is involved, which is a feature vector obtained according to style parameters and used for the visual characteristics of a text region on an image. For example, a visual feature vector of a text region may include a description of color distribution, edge features, and texture in the text region.

[0119] In the steps involved in this embodiment, first, according to the spatial parameters of each text area, a spatial feature vector for representing the position, size, etc. of the text area in the target bill image is determined, and according to the text parameters of each text area, a text feature vector for representing the text content corresponding to the text area is determined. Next, the style parameters corresponding to each text area, such as background color, etc., are determined, and according to the style parameters, the corresponding visual feature vector is determined, and the text feature vector, visual feature vector, and spatial feature vector are integrated to obtain fused feature vectors corresponding to multiple text areas.

[0120] Through the above steps, the spatial feature vector is determined according to the spatial parameters of each text area, which helps to accurately determine the layout and relative position of the text area in the target bill image. According to the text parameters of each text area, the text feature vector is determined, which helps to accurately capture the text content in the text. By extracting the style parameters of the text area and determining the corresponding visual feature vector based on the style parameters, visual context information is provided for subsequent information extraction. By integrating the text feature vector, the visual feature vector, and the spatial feature vector, a fused feature vector corresponding to multiple text areas is obtained, which helps to achieve a more comprehensive understanding of the context information of the text area when processing bills with complex layouts and information structures, and reduce errors in the information extraction process, thereby helping to solve the technical problem of not being able to efficiently and accurately extract bill information when extracting bill information.

[0121] Based on the above embodiments and optional embodiments, an optional implementation is provided, which is described in detail below.

[0122] In related technologies, due to the large differences in format and field content among bill types, traditional OCR technology often finds it difficult to effectively cope with a variety of bill types. In addition, when extracting bill information, there may be problems such as bill damage and wear, and poor bill image quality, resulting in a technical problem of being unable to efficiently and accurately extract bill information.

[0123] To address the above-mentioned problems, no effective solution has been proposed yet.

[0124] In view of this, a bill information extraction method is provided in an optional embodiment of the present invention, which can also be called an electronic bill information extraction method based on deep learning, which can effectively solve the technical problem of not being able to extract bill information efficiently and accurately when extracting bill information.

[0125] Figure 2 is a flow chart of a method for extracting bill information in an optional embodiment of the present invention, Figure 3 is a technical framework diagram of image restoration in an optional embodiment of the present invention, Figure 4 is a flowchart of image restoration in an optional embodiment of the present invention, such as Figure 2 , Figure 3 ,as well as Figure 4 As shown, a detailed description is given below.

[0126] S1, obtaining a target bill image, wherein the target bill image includes a plurality of text regions, each of which includes a corresponding text parameter;

[0127] For example, use OCR technology to detect the text area (same as the above text area) in the image (same as the above target bill image) and recognize its content, and output the text content (same as the above text parameters) and its position information (same as the above space parameters). This step includes two sub-steps:

[0128] A1. Text detection: locate the text area in the image.

[0129] A2. Text recognition: Identify the specific text content in the detected text area (same as the above text parameters).

[0130] Specifically, S1 includes:

[0131] S11, determining an initial bill image;

[0132] S12, determining whether the initial bill image needs to be repaired based on the initial bill image;

[0133] S13, when the determination result is that the initial bill image needs to be repaired, determining a defect sub-image corresponding to the initial bill image and an adjacent sub-image, wherein the adjacent sub-image is a sub-image in the initial bill image whose proximity to the defect sub-image is greater than a proximity threshold;

[0134] S14, repairing the defective sub-image based on adjacent sub-images to obtain a target bill image.

[0135] Specifically, S14 includes:

[0136] S141, repairing the defective sub-image according to adjacent sub-images to obtain a repaired bill image;

[0137] S142, determining a physical degradation function, wherein the physical degradation function is a function used for defect simulation of a bill image;

[0138] S143, inputting the repaired bill image into a physical degradation function to obtain a degraded bill image;

[0139] S144, determining a similarity index between the degraded bill image and the initial bill image;

[0140] S145: When the similarity index is greater than the similarity threshold, determine that the restored bill image is the target bill image.

[0141] For example, data collection and annotation include collecting actual scanned bill images and manually annotating the images, including structural information (Key-Value pairs of key fields) and OCR information (the location and content of all text in the image). If the original bill image (same as the initial bill image mentioned above) is of high quality, subsequent information extraction is performed directly, otherwise image restoration and enhancement are performed, including:

[0142] B1. Input the scanned image or photographed image of the original bill.

[0143] B2. Use the GAN-based image restoration network to complete the damaged areas in the image (same as the defect sub-image above) and enhance the visual quality of the image. Specifically include:

[0144] B21. Fill in damaged and blurred areas.

[0145] B22, remove stains or interfering background information (such as stamps or advertising text, etc.). Smooth surface distortion caused by camera shooting, etc.

[0146] B23. Output the repaired and enhanced high-quality document image (same as the target bill image mentioned above).

[0147] S2, determining the spatial parameters corresponding to the multiple text regions respectively, and determining the association relationship between the multiple text regions;

[0148] S3, determining target bill information corresponding to the target bill image according to the text parameters and space parameters respectively corresponding to the multiple text regions, and the association relationship between the multiple text regions.

[0149] Specifically, S3 includes:

[0150] S31, determining fused feature vectors corresponding to the multiple text regions respectively according to the text parameters and space parameters respectively corresponding to the multiple text regions, and the association relationship between the multiple text regions;

[0151] Specifically, S31 includes:

[0152] S311, determining spatial feature vectors corresponding to the plurality of text regions respectively according to the spatial parameters corresponding to the plurality of text regions respectively;

[0153] S312, determining text feature vectors corresponding to the plurality of text regions respectively according to the text parameters corresponding to the plurality of text regions respectively;

[0154] S313, determining style parameters corresponding to the multiple text areas respectively;

[0155] S314, determining visual feature vectors corresponding to the plurality of text regions respectively according to the style parameters corresponding to the plurality of text regions respectively;

[0156] S315, determining fused feature vectors corresponding to the plurality of text regions respectively according to the text feature vector, the visual feature vector, and the spatial feature vector.

[0157] S32, determining a weighted graph corresponding to the target bill image based on the fused feature vectors corresponding to the multiple text regions, wherein the weighted graph includes nodes corresponding to key features and nodes corresponding to non-key features, the key features being features whose corresponding importance index is greater than a predetermined importance threshold, and the non-key features being features whose corresponding importance index is less than or equal to the predetermined importance threshold;

[0158] Specifically, S32 includes:

[0159] S321, determining nodes corresponding to the multiple fused feature vectors respectively, and obtaining a node set;

[0160] S322, determining the corresponding association weight value between each node in the node set and other nodes according to the association relationship between each node in the node set and other nodes, and obtaining an association weight set;

[0161] S323, determining the neighbor nodes corresponding to each node in the node set according to the association weight set, and obtaining an edge set;

[0162] S324, construct a weighted graph based on the node set, the associated weight set, and the edge set.

[0163] S33, determining a nonlinear feature matrix corresponding to the target bill image according to the weighted graph corresponding to the target bill image, wherein the nonlinear feature matrix is ​​a matrix obtained by weighting key features and reducing weights on non-key features;

[0164] Specifically, S33 includes:

[0165] S331, determining an adjacency matrix and a node feature matrix corresponding to the weighted graph;

[0166] S332, inputting the adjacency matrix and the node feature matrix into a linear transformation layer to obtain a linear feature matrix, wherein the linear transformation layer includes a predetermined weight matrix;

[0167] S333, input the linear feature matrix into the activation function to obtain a nonlinear feature matrix.

[0168] S34, determining target classification labels corresponding to the plurality of text regions respectively according to the nonlinear feature matrix corresponding to the target bill image;

[0169] S35, determining the target bill information corresponding to the target bill image according to the text parameters and target classification labels respectively corresponding to the plurality of text regions.

[0170] For example, the output of the OCR module is used to construct a graph representation, and information extraction is achieved through a graph neural network (GNN). The specific use of the graph neural network to process the weighted graph includes using a graph convolutional network (GCN) to process the weighted graph, learn the high-dimensional feature representation of each node, and capture the relationship between text regions. The information extraction task is converted into a node classification task, and the category label of each node (text region) is predicted (the same as the target classification label mentioned above), such as "invoice number", "amount", "date", etc.

[0171] Convert the output of GNN (same as the target classification label) into the final structured information (same as the target ticket information), including format conversion, error correction, etc. Specifically:

[0172] C1. Convert the amount into digital format;

[0173] C2. Unify dates in different formats into a standard format;

[0174] C3. Perform semantic verification and correction on field values ​​(such as checking whether the amount field is a valid number).

[0175] The F1 value at the field level is used to evaluate the effect of information extraction. The F1 value is the harmonic average of precision and recall, which is used to measure the accuracy of information extracted by the model.

[0176] In summary, this method can use graph neural network (GNN) and generative adversarial network (GAN) to achieve information extraction. It includes the following key steps:

[0177] Step 1: Data collection and annotation;

[0178] During the collection process, the bill is usually pasted on a piece of white paper, and then an image is generated by a scanning device and uploaded to the system. After these bill images are authorized, they enter the subsequent processing flow. In the scenario of invoice information entry or review, the system needs to extract field content based on Key-Value pairs from the bill. Among them, Key is a pre-set field name (such as "invoice code", "invoice number", etc.), and Value is the text content of the corresponding field obtained through image recognition technology. It should be noted that different types of bills may have certain overlaps in their key information fields, but they also have their own unique fields. For example, the key fields of a general fixed-amount invoice include "invoice code", "invoice number" and "amount"; while the key fields of some itineraries (such as airplane itineraries) include "date", "amount", "departure station" and "arrival station". It can be seen that there are differences in the key information requirements of different types of bills, which puts higher requirements on the accuracy and adaptability of information extraction.

[0179] In response to the above task requirements, two independent data annotation processes were designed to obtain high-quality annotation information, namely the "structured information annotation process" and the "OCR information annotation process". The goal of the first annotation process is to generate the "structured information" of the bill, that is, the Key-Value pairs of key fields in each image (the same as the above target bill image) and their position information; the goal of the second annotation process is to generate the "OCR information" of the bill (the same as the above target bill image), that is, the text content of all text (excluding curved text) in the image (the same as the above text parameters) and its position coordinates in the image (the same as the above spatial parameters). These two annotation processes provide two different levels of annotation information for the subsequent model training, which are used for key field extraction and full-image text detection and recognition respectively.

[0180] In both annotation processes, the method of "pseudo-label generation + manual review and cleaning" is adopted to improve the annotation efficiency and accuracy. In the first annotation process, the image is processed using a pre-selected bill recognition system to automatically generate pseudo-labels of "structured information". These pseudo-labels include the location information of key fields (same as the above spatial parameters) and the corresponding text content (same as the above text parameters). Subsequently, the annotators will manually review and clean the generated pseudo-labels to ensure the accuracy of the annotation results. For fuzzy text that cannot be recognized manually, it is uniformly annotated as the "NOT CARE" attribute to avoid interference with model training. In the second annotation process, the bill image (same as the target bill image) is firstly detected and recognized based on the OCR inference engine to generate the OCR pseudo-label of the whole image. Then, the key field text box generated in the first annotation process is matched with the text box in the OCR inference result (same as the above text area) to ensure that the annotation of the key field is consistent with the OCR result.

[0181] During the matching process, if the intersection over union (IoU) between a key field text box (the same as the above text area) and the OCR inference text box is greater than 0.5, the coordinates and content of the corresponding text box in the OCR inference result are directly overwritten with the true value of the first annotation process; if a key field text box does not find a matching text box in the OCR inference result, the field is considered to be an OCR missed item and needs to be directly added to the OCR inference result. Through this matching and correction process, it is ensured that the labels of all key fields in the OCR annotation are manually reviewed and cleaned to ensure their accuracy. However, the non-key field text in the background may still contain a certain degree of OCR noise, but this noise will not affect the annotation quality of the key fields.

[0182] Through the above two annotation processes, high-quality annotation of key fields and full-image text in bill images is achieved, providing reliable training data for subsequent deep learning-based OCR models and information extraction models. This annotation method can not only meet the needs of various bill types, but also effectively reduce the workload of manual annotation, while ensuring the accuracy and consistency of the annotation results, providing an operational basis for bill information extraction tasks.

[0183] Specifically, data collection and annotation are the basic links of the entire system, and their role is to provide high-quality training data and verification data for subsequent deep learning models (such as OCR modules, information extraction models, etc.). By annotating the bill images and clarifying the location and content of key fields, the model can learn how to correctly extract key information from the images. The main steps include:

[0184] (1) Input: original bill scanned image or photographed image (same as the target bill image mentioned above).

[0185] (2) Output: structured information annotation (Key-Value pairs and location information of key fields), OCR information annotation (detection and recognition results of text in the entire image).

[0186] (3) Execution: Completed by automated tools (specific bill recognition system, OCR engine) and manual annotation).

[0187] Step 2: Image restoration and enhancement (optional);

[0188] This step mainly builds a generative adversarial network (GAN) to solve various image restoration problems (such as image deblurring, defogging, and deraining). The image restoration problem can be described as restoring a clear image x (same as the restored bill image) from a degraded input image y (same as the initial bill image above). It mainly includes:

[0189] (1) Generate a network Generate a sharp image from a degraded image y

[0190] (2) Physical model constraints: ensure that the generated results are consistent with the input image under the physical degradation model.

[0191] (3) Discriminant network

[0192] Discriminative network Used to determine whether the generated image is similar to the real clear image in distribution;

[0193] Discriminative network Used to determine whether the generated image is consistent with the input image after physical degradation.

[0194] Specifically, this step includes the following sub-steps:

[0195] D1. Problem Modeling:

[0196] The degradation process of the input image is expressed as a physical degradation model, and the formula is as follows:

[0197]

[0198] Where: y represents the input degraded image, that is, the input image obtained from the captured or collected image after blurring, fogging, raindrops or other degradation processes. x represents the clear image, It is a physical operator that describes the degradation process and characterizes the physical model of image degradation. Specifically, it can be:

[0199] 1) Blur operation: such as convolution operation to represent blur kernel;

[0200] 2) Fog model: such as atmospheric degradation model (transmittance, background light);

[0201] 3) Raindrop interference: represented as the interference pattern of covered pixels.

[0202] Output determines the specific physical model Clarify the degradation mechanism and its parameters (such as blur kernel, transmittance) for different tasks.

[0203] make and represent the degraded (blurred) image and the clear image respectively, i represents the i-th bill image, and N represents the total number of bill images.

[0204] D2. GAN network architecture design:

[0205] The network architecture consists of three parts: Generative network Physical degradation module and discriminant network

[0206] 1) Generate a network Generative Network It is the core module of image restoration, responsible for restoring a clear image from the degraded input image y It adopts a Cycle GAN-style structure, including convolutional layers for feature extraction, residual blocks for processing global context information, and deconvolutional layers to complete the reconstruction of clear images. The generator takes the degraded image y as input and generates the result is the output.

[0207] 2) Physical degradation module: The generated clear image is transformed into The simulation degenerates to (same as the above degraded bill image) to conform to the degradation process experienced by the input image y, thereby introducing physical consistency constraints in the network. Specifically, the physical degradation model is set according to the specific task (deblurring, defogging, deraining, etc.). For example, in deblurring, the degradation model is a convolution operation, and the formula is:

[0208]

[0209] Among them, k i is the blur kernel, It is a degradation simulation image (same as the above-mentioned degraded bill image).

[0210] In dehazing, the degradation model is the atmospheric scattering model, and the formula is:

[0211]

[0212] Among them, t i is the transmittance, A i is the atmospheric light value.

[0213] In general, the input of this module is the generated clear image and specific degradation parameters, output the generated degradation simulation diagram

[0214] 3) Discriminant network The network consists of two submodules and

[0215] It is used to discriminate the generated image Whether it conforms to the distribution of clear images (or "determine whether the generated image is similar to the real clear image in distribution"), that is, determine whether it is a realistic "high-definition image";

[0216] It is used to judge the degradation simulation diagram Is it consistent with the input image y (or "judge whether the above physical degradation model generates Whether it matches the input image y"), that is, to ensure that the result of the generated image after the degradation model conforms to the constraints of the physical model.

[0217] In general, the discriminant network The input includes the real clear image x, the generated image Degradation simulation diagram and input image y. The network outputs two discriminant results, which are used to guide the optimization of the generation network.

[0218] These modules are built and implemented by developers, and are programmed using deep learning frameworks such as PyTorch or TensorFlow.

[0219] D3. Loss function design:

[0220] The GAN model of the present invention achieves perceptual consistency and physical consistency by jointly optimizing the generative network and the discriminative network through multiple loss functions. Specifically, the following three loss functions are used:

[0221] 1) Fighting Losses

[0222] This loss function ensures that the distribution of generated images is close to the distribution of real images. The formula is:

[0223]

[0224] 2) Pixel consistency loss:

[0225] This loss function ensures that the generated image is close to the real image in terms of pixel values, including:

[0226] L1 loss of the generated image (same as the repaired bill image above) and the real image (same as the target bill image above) The formula is:

[0227]

[0228] L1 loss of the degraded image (same as the degraded bill image above) and the input image (same as the initial bill image above) The formula is:

[0229]

[0230] 3) Generator stability loss

[0231] This loss function further improves stability by constraining the output of the generator, and the formula is:

[0232]

[0233] Therefore, the total loss The formula is:

[0234]

[0235] Specifically, first, adversarial loss is used to constrain the discriminative network and the generative network The collaboration ensures that the distribution of generated images is close to the distribution of real clear images. The adversarial loss is determined by the discriminative network. and The output of More realistic while ensuring degraded images With the input image y i In addition, the method constrains the pixel difference between the generated image and the real image through pixel consistency loss, including the generated image With the real image x i L1 loss, and degraded image and the input image y i To further stabilize the generator, we add a generator stability loss, that is, to ensure that the generator can be trained from degraded images. Reconstruct the clear image x again i The final loss function is a weighted sum of the above parts. This process requires adjusting the relevant weight coefficients λ, γ, and μ according to the task to optimize the total loss target.

[0236] The input of the above module is the generated image, the real image, the degraded image and their corresponding parameters, and the output is the joint loss value (same as above ), which is used to guide the parameter update of the model.

[0237] The final objective function to be solved is:

[0238]

[0239] in, Indicates the optimal solution.

[0240] D4. Training process:

[0241] D41, Training phase:

[0242] In the training phase, the adversarial generative network of the present invention is used and the discriminant network Joint optimization.

[0243] First, we use the adversarial generative network Generate a preliminary clear image from the degraded image y The image y is about to be degraded i Input Generator Get the preliminary clear image G(y).

[0244] Then, the physical degradation module Convert to degraded image To simulate the input image y i degradation process.

[0245] Next, use the two submodules of the discriminant network respectively and To generate clear images and degraded simulated images Perform discrimination and output discrimination score.

[0246] Finally, through the gradient backpropagation of the loss function, the generator and the discriminator are optimized together, and the generator continuously improves the quality of the output clear image. At the same time, the physical constraint module and the discriminator ensure the physical consistency of the image after simulated degradation, so that the final generation result is both realistic and consistent with the actual degradation mechanism.

[0247] In the above process, the input is the training data set (including degraded images and real clear images), and the output is the optimized model parameters. The entire training process can be completed by using a GPU cluster or a deep learning platform.

[0248] D42, Testing Phase:

[0249] In the testing phase, the GAN model of the present invention is directly applied to the unseen (untrained) degraded image y. The degraded image is input to the generative network After that, output clear image Without the need for physical degradation modules and discriminant networks, the generator can independently complete image restoration tasks, and the generated clear images can be directly used for subsequent tasks such as visual analysis or OCR extraction.

[0250] In this phase, the input is the new image in the degradation task, and the output is the corresponding clear image. The testing phase is completed by the end user and is usually run in the actual deployment environment.

[0251] The GAN model uses a generative network and a discriminative network to learn the distribution adversarially, but existing GAN methods (such as pix2pix and Cycle GAN) do not consider the physical consistency between the input image and the generated image, resulting in the generated results not being able to conform to the physical degradation model. However, this method proposes a unified framework that combines physical model constraints and GAN, so that the generated results are not only perceptually close to the real image, but also meet the physical degradation model. This method is more suitable for image restoration tasks in various degraded scenarios (such as deblurring, defogging, and deraining). It combines the advantages of physical degradation models and generative adversarial networks (GANs) to restore clear images from input degraded images and ensure that the generated results are optimized in terms of perception and physical consistency.

[0252] Step 3: OCR text detection and recognition;

[0253] This module mainly extracts all text content contained in the image, which is usually directly inferred by a mature OCR engine. The general OCR inference engine is generally trained based on a large number of natural scene images and has a certain text recognition capability in generalized new scenes.

[0254] The main contents include 3 OCR settings: 1. Text input using a general OCR inference engine for inference; 2. Fine-tuning the text detection / recognition model using the scenario data and using the model prediction results as input; 3. Directly using OCR true value text input.

[0255] It adopts an OCR inference engine, using Text-Perceptron-Det as a text detector and CRNN as a text recognizer, and can obtain text coordinates and recognition results based on text lines.

[0256] Step 4: Graph representation construction and information extraction (including graph construction, GNN processing, and node classification);

[0257] This step can use a graph neural network (GNN) to construct a graph representation and complete information extraction. First, in the graph representation construction and information extraction stage, the text boxes (same as the text areas mentioned above) and their relationships (same as the association relationships mentioned above) are modeled as weighted graphs to capture the multi-dimensional relationships between text boxes. Then, the graph convolutional network (GNN) is used to propagate and aggregate features on the graph and update the feature representation of each text box (same as the nonlinear feature matrix mentioned above). Finally, the classifier is used to classify the nodes of each text box and predict its semantic label, thereby extracting the key information in the document (such as date, amount, etc.). The entire step involves three modules:

[0258] 1) Encoder module:

[0259] The encoder module is responsible for extracting multimodal features (same as the above-mentioned fused feature vector) from a visually rich document (same as the above-mentioned target bill image), namely text, image and spatial features (same as the above-mentioned text feature vector, visual feature vector, and spatial feature vector, respectively), including multiple text boxes, each of which has its position coordinates, text content and corresponding local image fragments (same as the above-mentioned spatial parameters, text parameters and style parameters, respectively).

[0260] Initial input: original visually rich documents (e.g., invoices, receipts), e.g.

[0261] Text box 1: Position = (Xa1, Yb1, Xa2, Yb2), text content = "Total", image fragment = partial screenshot.

[0262] Text box 2: Position = (Xa3, Yb3, Xa4, Yb4), text content = "23.45", image fragment = partial screenshot.

[0263] Implementation: Text embedding is generated by the Word2Vec model, image embedding is generated by the CNN model, and spatial embedding is generated by the position calculation formula.

[0264] ① Text feature extractor: Use the Word2Vec model to embed text content.

[0265] Input: text content (such as "Total", "23.45") (same as the above text parameters).

[0266] Output: text embedding vector T i (Same as the text feature vector above).

[0267] ②Image feature extractor: Use convolutional neural network to embed the image fragment of the text box.

[0268] Input: Image fragment (e.g. a partial screenshot of "Total") (same style parameters as above).

[0269] Output: Image embedding vector I i (Same as the visual feature vector above).

[0270] ③ Spatial feature calculator: Combine the bounding box coordinates of the text box (Xa1, Yb1, Xa2, Yb2) to calculate the spatial embedding,

[0271] Among them, Xa1, Xa3: the horizontal coordinate of the upper left corner of the text box, indicating the distance from the left border of the image to the left border of the text box, Yb1, Yb3: the vertical coordinate of the upper left corner of the text box, indicating the distance from the top border of the image to the top border of the text box, Xa2, Xa4: the horizontal coordinate of the lower right corner of the text box, indicating the distance from the left border of the image to the right border of the text box, Yb2, Yb4: the vertical coordinate of the lower right corner of the text box, indicating the distance from the top border of the image to the bottom border of the text box.

[0272] Input: Text box location coordinates (same as the above space parameters).

[0273] Output: spatial embedding vector S i(Same as the above spatial feature vector), including: the Euclidean distance between the center point of the text box and the center point of the document, and the polar coordinates (angle and radius) of the center point of the text box.

[0274] Final output: multimodal feature vector F for each text box i (same as the above fused feature vector), including text embedding vector, image embedding vector and space embedding vector, namely:

[0275] F i =[T i , I i , S i ]

[0276] 2) Graph representation module:

[0277] The graph representation module is responsible for converting the document image (same as the target bill image mentioned above) into a weighted graph to capture the diverse relationships between text boxes.

[0278] Input: Multimodal feature vector F = {F1, F2, ..., F n}, where each F i Represents the features of the i-th text box.

[0279] Implementation: A custom weighted graph construction method that uses a feature distance metric to calculate edge weights.

[0280] ①Node definition: define each text box as a node of the graph, and the node feature is F i .

[0281] ② Edge definition: Define the edges between nodes to represent the relationship between text boxes. The weight of the edge is calculated using the following metrics: Euclidean distance, Manhattan distance, cosine similarity, correlation of image features, mutual information of image features, etc.

[0282] ③Neighbor selection: Select the top K neighbors with the highest weight for each node (the same as the neighbor nodes mentioned above) and construct the edge set E.

[0283] Output: Construct a weighted graph G = (V, E, Ω), which contains the node set V (features of each text box), the edge set E (edges between each pair of text boxes), and the weight set Ω (relationship / weight between text boxes) (same as the association weight set mentioned above).

[0284] 3) Node classification module:

[0285] The node classification module is responsible for modeling the information extraction task of visually rich documents as a node classification problem on a graph and predicting the label of each text box (the same as the target classification label mentioned above).

[0286] Input: The weighted graph G = (V, E, Ω) generated in the second step.

[0287] Execution: Graph Convolutional Network (GCN), through multi-layer propagation and aggregation (same as the linear transformation layer mentioned above), realizes node classification.

[0288] ① Graph Convolutional Network (GCN): Through multi-layer propagation and aggregation, the feature representation of each node is updated. Each node receives information from its neighbor nodes through adjacency relationships and calculates the feature representation after weighted summation:

[0289]

[0290] Where: H (l) is the node feature of the lth layer. The feature of each node is obtained by propagating through the previous layer. (l) is the trainable weight matrix of the lth layer for linear transformation features (same as the predetermined weight matrix mentioned above), σ is the activation function (such as ReLU or Sigmoid), which is used to introduce nonlinearity and enhance the expression performance of the model. It is a normalized adjacency matrix (including edge weights) (same as the above adjacency matrix, used to control the range and proportion of feature propagation. The structural information of the weighted graph will affect the feature update.

[0291] After layer-by-layer propagation, the final node feature representation (non-linear transformation matrix) is generated.

[0292] ②Classifier: The softmax classifier classifies the final features of each node and predicts its semantic label.

[0293] Output: Category label for each text box (such as "Date", "Total Amount", etc.).

[0294] The following is a detailed description, including three main steps: graph construction, GNN processing, and node classification.

[0295] (1) Graph representation construction:

[0296] The goal of graph construction is to model the text boxes of a document and their relations as a weighted graph, capturing the semantic, spatial, and visual connections between text boxes. The input is the multimodal feature vectors extracted from the encoder, which include text embeddings, image embeddings, and spatial embeddings.

[0297] First, the method defines each text box in the document as a node in the graph, and the features of the node are represented by the multimodal feature vector of the text box.

[0298] Then, edges are defined between nodes to represent the association between text boxes. These edges are established based on different rules, such as spatial proximity (the positions of two text boxes in the document are close), semantic similarity (the text content of two text boxes is semantically similar), or visual feature similarity (the image fragment features corresponding to the text boxes are similar). The weights of the edges are calculated using a variety of distance metrics, including Euclidean distance, cosine similarity, or image feature correlation.

[0299] In addition, in order to avoid the interference of redundant edges on graph modeling, the method selects the top K most relevant neighbors based on weights to construct the sparse structure of the graph.

[0300] Ultimately, the output is a weighted graph where nodes represent text boxes and edges and their weights characterize the multidimensional relationships between text boxes.

[0301] The main goal of this step is to model the text boxes and their relations in a visually rich document (VRD) as a weighted graph, capturing the semantic, spatial, and visual relations between text boxes.

[0302] Input: A set of multimodal feature vectors F = {F1, F2, ..., F n}, where each F i Represents the multimodal features of a text box, including text embedding, image embedding, and spatial embedding.

[0303] Execution process:

[0304] The graph construction process includes three key steps: node definition, edge definition, and weight calculation:

[0305] ①Node definition: Each text box is regarded as a node in the graph, and the node features are represented by its multimodal feature vector F i Each node carries the semantic information (text embedding), image information (image embedding), and spatial information (spatial embedding) of the text box, thus fully representing the properties of the text box.

[0306] ② Edge definition: Add edges between nodes to represent the relationship between text boxes. The establishment of edges is based on the following rules:

[0307] Spatial proximity: An edge is established between two text boxes if they are located close to each other in the document.

[0308] Semantic Similarity: An edge is established between two text boxes if their text contents are semantically similar.

[0309] Visual feature similarity (same as style parameter above): If the image embeddings of two text boxes are similar in feature space, an edge is established between them.

[0310] ③ Edge weight calculation: The edge weight is calculated using a variety of distance measurement methods, such as:

[0311] Euclidean distance: measures the spatial proximity of text boxes.

[0312] Cosine similarity: measures the semantic similarity of text content.

[0313] Image feature correlation (same as style parameter as above): measures the visual similarity of local image fragments of the text box.

[0314] ③ Neighborhood selection: Select the top K neighbors with the highest weight for each node, build a sparse graph structure (same as the weighted graph above), and avoid the influence of too many redundant edges.

[0315] Output: A weighted graph G = (V, E, Ω), where:

[0316] Node Set Represents a text box (same as the text area above), including the features of each text box (same as the fused feature vector above), and n represents n text boxes (graph nodes);

[0317] Edge Set Represents the relationship between text boxes (same as the above association relationship), including the edges between each pair of text boxes, that is, describing the relationship between different nodes, such as v (i) and v (j) connectivity of

[0318] Weight Set Represents the weight of each edge, assigns weight to each edge, and captures the multi-dimensional relationship between nodes.

[0319] In addition, X∈R n×d : Node feature matrix (same as the node feature matrix above), representing the feature vector of each node, where d is the feature dimension.

[0320] (2) Graph Neural Network (GNN) Processing:

[0321] After the weighted graph is constructed, the system inputs it into the graph neural network (GNN), and uses the structural information of the graph and the node features to propagate and aggregate features to capture the global contextual relationship between text boxes. In each layer of GNN, each node receives information from its neighbor nodes and updates its own feature representation. The propagation mechanism controls the propagation of neighbor information through the normalized adjacency matrix, and combines the trainable parameter matrix for weighted summation to finally generate a new feature representation for each node in this layer. The activation function is used to introduce nonlinearity to improve the expressiveness of the model. After multiple layers of propagation, the features of each node include not only its own information, but also the contextual information of neighbor nodes and more distant nodes. In addition, multi-layer propagation expands the receptive field of the node, enabling the model to capture long-distance relationships in the document, thereby improving the accuracy of the semantic expression of the text box. After this stage of processing, the output is the final feature representation of each node. After feature aggregation, the semantic and spatial dependencies between nodes are effectively integrated.

[0322] The main goal is to capture the global relationships between text boxes by performing feature propagation and aggregation on weighted graphs through graph neural networks (GNNs).

[0323] Input: Weighted graph G = (V, E, Ω), including node features, edges and their weights.

[0324] Execution process: GNN updates the feature representation of each node through multi-layer propagation (same as the linear transformation layer mentioned above) and feature aggregation mechanism, so that it not only contains its own information, but also incorporates the contextual information of neighboring nodes. The specific process is as follows:

[0325] ① Feature propagation: In each layer, each node receives information from its neighbor nodes through adjacency relationships. The input H of the first layer GCN (0) is the original feature matrix X of the node (same as the node feature matrix above). That is, for the 0th layer:

[0326] H (0) =X

[0327] This setting enables the model to learn higher-level feature representations through layer-by-layer propagation based on the original node features.

[0328] ② Feature aggregation: During the propagation process, node features are updated layer by layer, and the final feature representation of each node not only contains its own information, but also incorporates the information of its neighboring nodes. This feature aggregation mechanism can capture the global relationship between text boxes.

[0329] ③ Multi-layer propagation: Through multi-layer propagation, GNN can gradually expand the receptive field and capture more distant relationship information. For example, the first layer aggregates the information of direct neighbors, and the second layer aggregates the information of second-order neighbors.

[0330] Output: The updated node feature matrix (same as the nonlinear feature matrix above), where the feature representation of each node incorporates contextual information.

[0331] (3) Node classification:

[0332] This node classification is the last step of the graph representation method, and the goal is to predict the semantic label of each text box based on the feature representation of the node. The input is the node feature matrix processed by GNN. These features have integrated the contextual relationship and have strong semantic expression ability. The classifier maps the node features to a fixed number of category spaces through a fully connected layer, and uses the softmax function to calculate the probability distribution of the node belonging to each category. For example, a text box may be classified as a label such as "date", "amount" or "invoice number". Finally, the classifier uses the category with the largest value in the probability distribution as the predicted label of the node. Through this process, each text box in the document will be assigned a specific semantic category, thereby achieving the goal of the information extraction task. The output is the classification result corresponding to each text box, that is, its semantic label.

[0333] That is, the main goal is to model the node classification task on the graph as an information extraction problem and predict the semantic label of each text box.

[0334] Input: Node feature matrix after GNN processing (same as the nonlinear feature matrix above).

[0335] Execution process:

[0336] Node classification is done through a simple classifier (such as softmax classifier), the specific process is as follows:

[0337] ① Classifier design: A graph convolutional network (GCN) with m layers is used, and the goal is to perform classification tasks on each node on the custom document graph. In the last layer, the output of GCN is consistent with the number of classification labels and is mapped to the probability distribution of each class through a softmax classifier. The formula is as follows:

[0338]

[0339] Among them, Z is the output matrix of the last layer, which represents the probability distribution of node classification. Each node outputs the probability corresponding to each category. is the normalized adjacency matrix, H (m) is the node feature matrix of the mth layer GCN, which contains the aggregated features of the previous layer of each node, W (m) It is the trainable weight matrix of the mth layer, which is used to linearly transform the features into the probability space of the classification dimension. Softmax is a normalized activation function that maps the features of each node to the category probability distribution.

[0340] Defined as:

[0341]

[0342] in, is the adjacency matrix A plus the self-loop (the identity matrix I n The purpose of adding the unit matrix is ​​to add a self-loop to each node so that it can integrate its own features. It is the degree matrix in the normalization process, which is used to standardize the rows of the adjacency matrix (that is, the sum of the weights of each row is 1).

[0343] The softmax is defined as:

[0344]

[0345] Among them, softmax(sof i ) represents the vector sof i (Same as above ) is the result of applying the softmax function to the i-th element in .

[0346] Finally, softmax maps the features of each node to the category space so that the sum of the elements in each row of Z is 1, which can be interpreted as a probability distribution.

[0347] ② Category prediction: According to the probability distribution of softmax output, the category with the highest probability is selected as the final predicted label of the node. For example, a text box may be classified as "date", "amount", "invoice number", etc.

[0348] In the node classification task, the cross entropy loss function is used to optimize the labeled nodes. The formula of the loss function is as follows:

[0349]

[0350] in, The value of the loss function, Type i,num Indicates that the i-th text box belongs to the value of the num-th class label, Prob i,num It represents the probability that the i-th text box belongs to the num-th class label, and NUM represents the total number of labels.

[0351] Output: The classification label of each node (same as the target classification label above), that is, the semantic category of each text box.

[0352] In general, the graph representation construction and information extraction stage achieves modeling and extraction of complex document information through three steps: graph modeling, GNN feature propagation, and node classification. Starting from multimodal features, a weighted graph is constructed to capture the multidimensional connections of text boxes; then, feature fusion and context modeling are achieved through the GNN layer; finally, each node is classified into the corresponding category through the classifier, and the key information of the document is output. The graph construction algorithm, GNN model, and classifier are the core executors of this stage. They work together to solve the modeling and information extraction problems of the diverse text box relationships in the document.

[0353] Step 5: Post-processing (formatting and error correction);

[0354] This step is mainly responsible for converting the model output into the output form of the final solution, mainly including mapping the predicted value back to the dictionary value, filtering special symbols, empty text, etc. It should be noted that in actual business applications, the post-processing form of the data will be customized according to the requirements of downstream tasks. For example, the amount is converted into a number in a unified format for storage. For dates in different formats, such as 2024 / 08 / 06, 202407-07, and August 6, 2024, they are stored using a unified regular expression to facilitate subsequent data storage or processing.

[0355] This step can automatically convert the extracted bill information into standardized structured data, which can be directly stored in a relational database or used for further processing in subsequent processes. This step improves the availability and ease of use of the data and meets the efficient requirements of modern data processing.

[0356] Step 6: Evaluation indicators (F1 value evaluation).

[0357] This step is mainly aimed at the information extraction results, so the evaluation index of this step directly uses the field-level F1 value for evaluation. This index is calculated by the precision rate P and the recall rate R:

[0358]

[0359] Among them, TP represents the number of predicted fields that match the information structured truth value, FP represents the number of predicted fields that do not appear in the information structured truth value, and FN represents the number of fields that are not matched in the information structured truth value. ε is a small constant (0.001) to prevent the appearance of blank images with no truth value. The F1 value of the entire dataset is the average F1 value of each image.

[0360] Based on the above task settings, deep learning is used to extract electronic credential information.

[0361] Through the above optional implementation, at least the following beneficial effects can be achieved:

[0362] (1) Compared with the related art, the present invention helps to accurately identify and locate the key information in the bill by determining the spatial parameters of each of the multiple text regions in the target bill image, and helps to accurately understand the logical connection between the text regions by determining the association relationship between these text regions, so as to accurately identify and analyze the bill information. By using the text parameters and spatial parameters extracted from the text region, it helps to achieve the pertinence and efficiency of information extraction, and combined with the association relationship between the text regions, the target bill information corresponding to the target bill image is determined, and the unstructured information in the bill image is converted into structured data, which reduces the misidentification in the bill information extraction process, improves the efficiency and accuracy of the bill information extraction process, and thus solves the technical problem in the related art that the bill information cannot be extracted efficiently and accurately when extracting the bill information.

[0363] (2) Compared with the related art, the present invention determines the fusion feature vectors corresponding to the multiple text regions by comprehensively considering the text parameters and spatial parameters corresponding to the multiple text regions, as well as the correlation between the multiple text regions, which helps to accurately identify the real text information corresponding to the multiple text regions and reduce the misidentification in the process of bill information extraction. Based on these fusion feature vectors, a weighted graph is constructed, in which the nodes represent the text regions, and the weights of the edges reflect the correlation strength between the text regions. When constructing the weighted graph, key features and non-key features are distinguished, which solves the problem of different importance of different features in the information extraction process. Through weighted processing and weight reduction processing, the prominence of key features is ensured, and the accuracy of information extraction is improved. By processing the weighted graph, a nonlinear feature matrix is ​​obtained, which can more accurately identify and classify the key information in the bill, reduce the error in the information extraction process, and help to improve the comprehensiveness of bill information extraction in the future. By determining the target classification label corresponding to each text region based on the nonlinear feature matrix, and then determining the target bill information corresponding to the target bill image, the problem of converting unstructured information in the bill image into structured data is effectively solved, and the efficiency and accuracy of data processing are improved.

[0364] (3) Compared with the related art, the present invention helps to accurately identify and locate the key information in the bill by determining the spatial parameters (such as coordinates and size) of each of the multiple text areas in the target bill image, and can accurately identify the position of each text area even in a complex layout. By determining the association between these text areas, which can include the proximity in visual layout and the semantic relevance, it helps to accurately understand the logical connection between text areas, so that accurate identification and parsing of bill information can be achieved, especially for bills with non-standard or complex formats. By determining the spatial parameters and associations of each of the multiple text areas, it is possible to accurately identify and extract bill information, reduce misidentification and missed identification, and cope with bills of various formats, thereby improving the versatility and scope of application of the method.

[0365] (4) Compared with the related art, the present invention obtains a linear feature matrix by performing linear transformation (such as matrix multiplication and vector addition) on the adjacency matrix and node feature matrix corresponding to the weighted graph, which helps the model better understand the relationship between nodes. The linear feature matrix is ​​input into the activation function, and a nonlinear transformation is introduced to generate a nonlinear feature matrix, which helps to capture the complex relationship between nodes, thereby strengthening the correlation between features, improving the adaptability of the model to complex bill formats and information structures, and helping to accurately identify and classify bill information in a wider range of scenarios, improving the efficiency and accuracy of target bill information extraction, and thus helping to solve the technical problem of not being able to efficiently and accurately extract bill information when extracting bill information.

[0366] (5) Compared with the related art, the present invention first judges the imaging quality of the acquired initial bill image (such as whether it is blurred, damaged or missing) before extracting the information of the bill image, thereby avoiding the inability to accurately extract the bill information due to the bill image imaging quality problem. By determining the defective sub-image area and the adjacent sub-image in the initial bill image when the result is determined to be in need of repair, it is helpful to accurately restore the image content of the defective area, thereby ensuring that the subsequent extraction of the marked bill information can be carried out on the basis of high-quality images, reducing information extraction errors and omissions caused by bill image quality problems, and improving the efficiency and accuracy of bill information extraction, thereby solving the problem in the related art that when extracting bill information, the bill information cannot be extracted efficiently and accurately due to bill damage and wear.

[0367] (6) Compared with the related art, the present invention determines the spatial feature vector according to the spatial parameters of each text area, which helps to accurately determine the layout and relative position of the text area in the target bill image. According to the text parameters of each text area, the text feature vector is determined, which helps to accurately capture the text content in the text. By extracting the style parameters of the text area and determining the corresponding visual feature vector according to the style parameters, visual context information is provided for subsequent information extraction. By integrating the text feature vector, the visual feature vector, and the spatial feature vector, a fused feature vector corresponding to multiple text areas is obtained, which helps to achieve a more comprehensive understanding of the context information of the text area when processing bills with complex layouts and information structures, reduce errors in the information extraction process, and thus help solve the technical problem of not being able to efficiently and accurately extract bill information when extracting bill information.

[0368] (7) Compared with the related art, the present invention can better deal with the problem of complexity of bill format. For fixed format, it can quickly and accurately extract preset fields. For open format, it can dynamically identify Key-Value pair information, realize flexible field extraction, and has a wider scope of application. In addition, it not only uses the graph neural network (GNN) in deep learning technology to realize the automatic adaptation of diversified bill types and the analysis of format information, and improve the information extraction accuracy of different types of bills, but also introduces the image repair function through the adversarial generative network (GAN) to solve the problem of missing or blurred text caused by bill damage, wear, etc., and enhance the robustness of information extraction. It also provides an end-to-end automated solution, from bill scanning / photographing input to key information extraction and standardized structured output, to achieve efficient and automated processing throughout the process, which is suitable for batch business scenarios.

[0369] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0370] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention.

[0371] Example 2

[0372] According to an embodiment of the present invention, a device for implementing the above-mentioned bill information extraction method is also provided. Figure 5 is a structural block diagram of a bill information extraction device according to an embodiment of the present invention. Figure 5 As shown, the device includes: an acquisition module 502, a first determination module 504 and a second determination module 506. The device is described in detail below.

[0373] An acquisition module 502 is used to acquire a target bill image, wherein the target bill image includes multiple text areas, and the multiple text areas respectively include corresponding text parameters; a first determination module 504 is connected to the above-mentioned acquisition module 502, and is used to determine the spatial parameters corresponding to the multiple text areas respectively, and determine the association relationship between the multiple text areas; a second determination module 506 is connected to the above-mentioned first determination module 504, and is used to determine the target bill information corresponding to the target bill image based on the text parameters and spatial parameters corresponding to the multiple text areas respectively, and the association relationship between the multiple text areas.

[0374] It should be noted here that the above-mentioned acquisition module 502, the first determination module 504 and the second determination module 506 correspond to steps S102 to S106 in the implementation of the bill information extraction method, and the instances and application scenarios implemented by the multiple modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiment 1.

[0375] Example 3

[0376] According to another aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor, wherein the processor is configured to execute the instructions to implement any of the above-mentioned ticket information extraction methods.

[0377] Example 4

[0378] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute any of the above-mentioned ticket information extraction methods.

[0379] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0380] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0381] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0382] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0383] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0384] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0385] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for extracting bill information, characterized in that: include: Acquire a target bill image, wherein the target bill image includes a plurality of text areas, and the plurality of text areas respectively include corresponding text parameters; Determine the spatial parameters corresponding to the multiple text areas respectively, and determine the association relationship between the multiple text areas; The target bill information corresponding to the target bill image is determined according to the text parameters and the space parameters respectively corresponding to the multiple text areas, and the association relationship between the multiple text areas.

2. The method according to claim 1, characterized in that Determining target bill information corresponding to the target bill image according to the text parameters and space parameters respectively corresponding to the multiple text areas, and the association relationship between the multiple text areas, includes: Determining fused feature vectors corresponding to the plurality of text regions respectively according to the text parameters and space parameters respectively corresponding to the plurality of text regions and the association relationship between the plurality of text regions; Determine a weighted graph corresponding to the target bill image according to the fused feature vectors corresponding to the multiple text regions, wherein the weighted graph includes nodes corresponding to key features and nodes corresponding to non-key features, the key features are features whose corresponding importance index is greater than a predetermined importance threshold, and the non-key features are features whose corresponding importance index is less than or equal to the predetermined importance threshold; Determining a nonlinear feature matrix corresponding to the target bill image according to the weighted graph corresponding to the target bill image, wherein the nonlinear feature matrix is ​​a matrix obtained by weighting the key features and reducing the weights of the non-key features; Determining target classification labels corresponding to the plurality of text regions respectively according to a nonlinear feature matrix corresponding to the target bill image; According to the text parameters and target classification labels respectively corresponding to the plurality of text areas, target bill information corresponding to the target bill image is determined.

3. The method according to claim 2, characterized in that The step of determining a weighted graph corresponding to the target bill image based on the fused feature vectors corresponding to the plurality of text regions respectively includes: Determine the nodes corresponding to the multiple fused feature vectors respectively, and obtain a node set; Determine the corresponding association weight value between each node in the node set and other nodes according to the association relationship between each node in the node set and other nodes, and obtain an association weight set; Determine the neighbor nodes corresponding to each node in the node set according to the association weight set to obtain an edge set; The weighted graph is constructed according to the node set, the associated weight set, and the edge set.

4. The method according to claim 2, characterized in that: The step of determining a nonlinear feature matrix corresponding to the target bill image based on the weighted graph corresponding to the target bill image comprises: Determining an adjacency matrix and a node feature matrix corresponding to the weighted graph; Inputting the adjacency matrix and the node feature matrix into a linear transformation layer to obtain a linear feature matrix, wherein the linear transformation layer includes a predetermined weight matrix; The linear feature matrix is ​​input into the activation function to obtain a nonlinear feature matrix.

5. The method according to claim 1, characterized in that The step of acquiring the target bill image comprises: Determining an initial bill image; Determining, based on the initial bill image, whether the initial bill image needs to be repaired; In the case where the determination result is that the initial bill image needs to be repaired, determining a defect sub-image corresponding to the initial bill image and an adjacent sub-image, wherein the adjacent sub-image is a sub-image in the initial bill image whose proximity to the defect sub-image is greater than a proximity threshold; The defective sub-image is repaired according to the adjacent sub-images to obtain the target bill image.

6. The method according to claim 5, characterized in that The step of repairing the defective sub-image according to the adjacent sub-image to obtain the target bill image includes: Repairing the defective sub-image according to the adjacent sub-images to obtain a repaired bill image; Determining a physical degradation function, wherein the physical degradation function is a function used for defect simulation of a bill image; Inputting the repaired bill image into the physical degradation function to obtain a degraded bill image; Determining a similarity index between the degraded bill image and the initial bill image; When the similarity index is greater than a similarity threshold, the restored bill image is determined to be the target bill image.

7. The method according to any one of claims 2 to 6, characterized in that The step of determining the fused feature vectors corresponding to the plurality of text regions respectively according to the text parameters and the space parameters respectively corresponding to the plurality of text regions and the association relationship between the plurality of text regions comprises: Determining spatial feature vectors corresponding to the plurality of text regions respectively according to the spatial parameters corresponding to the plurality of text regions respectively; Determining text feature vectors corresponding to the plurality of text regions respectively according to text parameters corresponding to the plurality of text regions respectively; Determine style parameters corresponding to the multiple text areas respectively; Determining visual feature vectors corresponding to the plurality of text regions respectively according to style parameters corresponding to the plurality of text regions respectively; According to the text feature vector, the visual feature vector, and the spatial feature vector, fused feature vectors corresponding to the multiple text regions are determined.

8. A bill information extraction device, characterized in that: include: An acquisition module, used for acquiring a target bill image, wherein the target bill image includes a plurality of text areas, and the plurality of text areas respectively include corresponding text parameters; A first determination module, used to determine the space parameters corresponding to the plurality of text regions respectively, and to determine the association relationship between the plurality of text regions; The second determination module is used to determine the target bill information corresponding to the target bill image according to the text parameters and space parameters respectively corresponding to the multiple text areas and the association relationship between the multiple text areas.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the ticket information extraction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the ticket information extraction method as described in any one of claims 1 to 7.

Images