Bill information extraction method, device and bill information extraction system
Through the bill information extraction model of the multimodal fusion module and the graph inference module, combined with visual semantic fusion features and node features, the problems of poor universality and low efficiency of bill information extraction in the existing technology are solved, and the bill information extraction is realized for different versions of bills. Accurate and efficient extraction and context understanding.
Patent Information
- Application Number
- CN202311870102.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-29
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2043-12-29
AI Technical Summary
When processing different versions of bill information, the existing bill information extraction methods have poor generality, low execution efficiency and easy to generate ambiguity, making it difficult to extract bill information accurately and efficiently.
The bill information extraction model of multimodal fusion module, graph inference module and classification module is adopted. Through visual semantic fusion of feature vectors and node feature vectors, combined with text position information, visual features and semantic features are fused to infer the correlation between text areas. , improve extraction efficiency and accuracy.
It realizes accurate and efficient extraction of bill information of different versions, enhances context understanding, reduces ambiguity, and improves the efficiency and accuracy of bill information extraction.
Smart Images

Figure CN117831056B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of computer vision technology, and in particular, relates to a bill information extraction method, device and bill information extraction system. Background Art
[0002] Invoice information extraction is the process of automatically identifying and extracting key information on paper or electronic invoices, such as extracting and identifying invoice numbers, dates, amounts, product names, etc. Automatic invoice information extraction can enable information that originally required manual identification and entry to be processed automatically and quickly and accurately, which can improve the efficiency and accuracy of invoice information identification and entry, save costs, and support big data analysis of invoice information.
[0003] At present, there are usually two ways to extract bill information: one is based on template matching recognition. This type of method has a fixed bill format, poor versatility, high development difficulty, requires a large number of templates and rules, and is difficult to maintain; the other is based on natural language processing. This type of method has low execution efficiency and produces ambiguity. Summary of the invention
[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a bill information extraction method, device and bill information extraction system, which are applicable to bills of different versions and can accurately and efficiently extract bill information.
[0005] In a first aspect, the present application provides a method for extracting bill information, the method comprising:
[0006] Acquire a bill image to be processed, wherein the bill image includes a plurality of text areas;
[0007] Performing text detection and recognition on the bill image to determine a plurality of text position information and a plurality of text content information of the bill image, wherein the text position information and the text content information correspond to the text area in a one-to-one manner;
[0008] Inputting the bill image, the text position information and the text content information into a bill information extraction model to obtain the bill information output by the bill information extraction model;
[0009] The bill information extraction model includes a multimodal fusion module, a graph reasoning module and a classification module. The bill image, the text position information and the text content information are input into the bill information extraction model to obtain the bill information output by the bill information extraction model, further including:
[0010] Inputting the bill image, the text position information and the text content information into the multimodal fusion module to obtain a visual semantic fusion feature vector output by the multimodal fusion module;
[0011] Inputting the visual semantic fusion feature vector and the text position information into the graph reasoning module, performing reasoning according to the regional position relationship of the multiple text regions, and obtaining a node feature vector output by the graph reasoning module;
[0012] The node feature vector is input into the classification module to obtain the bill information output by the classification module.
[0013] According to the bill information extraction method of the present application, the visual features and semantic features corresponding to each text area in the bill image are fused through the bill information extraction model, and reasoning is performed based on the regional position relationship of multiple text areas. By fusion of visual features and semantic features and fully considering the position information of the text in the bill image, the correlation between them can be inferred based on the position relationship of each text area, thereby enhancing the understanding of the context and improving the efficiency and accuracy of bill information extraction.
[0014] According to an embodiment of the present application, the step of inputting the visual semantic fusion feature vector and the text position information into the graph reasoning module, performing reasoning according to the regional position relationship of the multiple text regions, and obtaining the node feature vector output by the graph reasoning module includes:
[0015] Based on the text position information, determining the region position relationship of the plurality of text regions;
[0016] Based on the region position relationship, determining weight information between nodes in the visual semantic fusion feature vector, each node in the visual semantic fusion feature vector corresponds to one text region;
[0017] A node feature vector is obtained based on the weight information and the visual semantic fusion feature vector.
[0018] According to an embodiment of the present application, the multimodal fusion module includes a visual feature extraction module, a semantic feature extraction module and a feature fusion module, and the bill image, the text position information and the text content information are input into the multimodal fusion module to obtain the visual semantic fusion feature vector output by the multimodal fusion module, including:
[0019] Inputting the bill image and the text position information into the visual feature extraction module to obtain a visual feature vector output by the visual feature extraction module;
[0020] Inputting the text content information and the text position information into the semantic feature extraction module to obtain a semantic feature vector output by the semantic feature extraction module;
[0021] The visual feature vector and the semantic feature vector are input into the feature fusion module to obtain the visual-semantic fusion feature vector.
[0022] According to an embodiment of the present application, the step of inputting the bill image and the text position information into the visual feature extraction module to obtain the visual feature vector output by the visual feature extraction module includes:
[0023] Performing convolution processing on the bill image to obtain a first feature map corresponding to the bill image;
[0024] Based on the text position information, the text position information corresponding to each of the text regions is mapped to the first feature map, and the mapped first feature map is pooled with a region of interest to obtain the visual feature vector.
[0025] According to an embodiment of the present application, the step of inputting the text content information and the text position information into the semantic feature extraction module to obtain the semantic feature vector output by the semantic feature extraction module includes:
[0026] Mapping the text content information to the one-hot encoding space of the semantic feature extraction module to obtain a one-hot encoding vector;
[0027] Based on the text position information and the one-hot encoding vector, the semantic feature vector is obtained.
[0028] According to an embodiment of the present application, the step of inputting the node feature vector into the classification module to obtain the ticket information output by the classification module includes:
[0029] Based on the node feature vector, classify each of the text regions to obtain label information corresponding to each of the text regions;
[0030] Based on the tag information, the ticket information is obtained.
[0031] In a second aspect, the present application provides a bill information extraction device, the device comprising:
[0032] An acquisition module, used for acquiring a bill image to be processed, wherein the bill image includes a plurality of text areas;
[0033] A first processing module, configured to perform text detection and recognition on the bill image, and determine a plurality of text position information and a plurality of text content information of the bill image, wherein the text position information and the text content information correspond to the text area in a one-to-one manner;
[0034] A second processing module, configured to input the bill image, the text position information and the text content information into a bill information extraction model to obtain the bill information output by the bill information extraction model;
[0035] The bill information extraction model includes a multimodal fusion module, a graph reasoning module and a classification module, and the second processing module is further used for:
[0036] Inputting the bill image, the text position information and the text content information into the multimodal fusion module to obtain a visual semantic fusion feature vector output by the multimodal fusion module;
[0037] Inputting the visual semantic fusion feature vector and the text position information into the graph reasoning module, performing reasoning according to the regional position relationship of the multiple text regions, and obtaining a node feature vector output by the graph reasoning module;
[0038] The node feature vector is input into the classification module to obtain the bill information output by the classification module.
[0039] According to the bill information extraction method of the present application, the visual features and semantic features corresponding to each text area in the bill image are fused through the bill information extraction model, and reasoning is performed based on the regional position relationship of multiple text areas. By fusion of visual features and semantic features and fully considering the position information of the text in the bill image, the correlation between them can be inferred based on the position relationship of each text area, thereby enhancing the understanding of the context and improving the efficiency and accuracy of bill information extraction.
[0040] In a third aspect, the present application provides a bill information extraction system, the system comprising:
[0041] server;
[0042] A client, the client is connected to the server for communication, the client sends a receipt image to be processed to the server, the server executes the receipt information extraction method described in the first aspect above, and sends receipt information corresponding to the receipt image to the client;
[0043] Among them, the server is deployed with a bill information extraction model.
[0044] According to the bill information extraction method of the present application, the visual features and semantic features corresponding to each text area in the bill image are fused through the bill information extraction model, and reasoning is performed based on the regional position relationship of multiple text areas. By fusion of visual features and semantic features and fully considering the position information of the text in the bill image, the correlation between them can be inferred based on the position relationship of each text area, thereby enhancing the understanding of the context and improving the efficiency and accuracy of bill information extraction.
[0045] According to one embodiment of the present application, the system further includes:
[0046] A model training end, the model training end is connected to the server for communication, the model training end is used to train the bill information extraction model and deploy the bill information extraction model to the server;
[0047] The server receives the bill information extraction request sent by the client, saves the bill image sent by the client, and outputs it to the model training end;
[0048] When the number of the bill images received by the model training end is equal to the image number threshold, the bill images with the image number threshold are added to the training set of the bill information extraction model, and the bill information extraction model is fine-tuned to obtain an updated bill information extraction model;
[0049] The model training end outputs the updated bill information extraction model to the server, and deploys the updated bill information extraction model on the server.
[0050] In a fourth aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for extracting ticket information as described in the first aspect above is implemented.
[0051] In a fifth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for extracting bill information as described in the first aspect above is implemented.
[0052] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the bill information extraction method as described in the first aspect above.
[0053] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0055] Figure 1 This is one of the flow charts of the method for extracting bill information provided in the embodiment of the present application;
[0056] Figure 2It is a structural diagram of a bill information extraction model provided in an embodiment of the present application;
[0057] Figure 3 This is the second flow chart of the method for extracting bill information provided in the embodiment of the present application;
[0058] Figure 4 This is the third flow chart of the method for extracting bill information provided in the embodiment of the present application;
[0059] Figure 5 It is a structural schematic diagram of a bill information extraction device provided in an embodiment of the present application;
[0060] Figure 6 It is a structural diagram of a bill information extraction system provided in an embodiment of the present application;
[0061] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0062] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.
[0063] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0064] The following, in conjunction with the accompanying drawings, describes in detail the ticket information extraction method, ticket information extraction device, ticket information extraction system, electronic device and readable storage medium provided in the embodiments of the present application through specific embodiments and their application scenarios.
[0065] The ticket information extraction method may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.
[0066] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).
[0067] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse and a joystick.
[0068] The ticket information extraction method provided in the embodiment of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the ticket information extraction method. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The ticket information extraction method provided in the embodiment of the present application is described below using an electronic device as an example of the execution subject.
[0069] like Figure 1 As shown, the bill information extraction method includes: step 110, step 120 and step 130.
[0070] Step 110: Acquire a bill image to be processed, where the bill image includes multiple text areas.
[0071] The bill image is a visual representation of a paper or electronic bill captured in digital form. The bill image can be in the form of a scan, a photo, etc. The bill image can be a locally saved image or a networked image.
[0072] It is understandable that the area where the bill is located can be cut out from the image including the bill to obtain the bill image, or the bill image can be directly obtained.
[0073] In this embodiment, the text area is an image area containing text information, for example, the text information includes the title, issuer, amount, date, currency type, payment period, number, etc.
[0074] In this embodiment, the bill image includes multiple text areas. In different bill images, the positions of the text areas may not be fixed, and different bill images may contain different types of text information.
[0075] Step 120: Perform text detection and recognition on the bill image to determine multiple text position information and multiple text content information of the bill image. The text position information and the text content information correspond to the text area one by one.
[0076] The text position information represents the position of the text area in the bill image. For example, the text area can be determined by a rectangular text box, and the text position information can be the coordinates of the four vertices of the text box.
[0077] In this embodiment, a text detection method may be used to detect a text box corresponding to each text area, and the four vertex coordinates corresponding to the text box are used as text position information.
[0078] For example, taking the center of the bill image as the origin, the four vertex coordinates of the text box corresponding to a certain text area are detected to be (1, 2), (3, 2), (1, 3), and (3, 3). Then the text position information corresponding to the text area is (1, 2), (3, 2), (1, 3), and (3, 3).
[0079] In this embodiment, the text content information represents the text content corresponding to the text area. For example, the text content may be Arabic numerals, Chinese characters, English letters, etc.
[0080] In this embodiment, the text box corresponding to the text area can be determined first to obtain text position information, and then the image corresponding to the text area can be captured based on the text position information, and text recognition can be performed on the captured image to obtain the text content information corresponding to the text area.
[0081] In this embodiment, optical character recognition (OCR) or a deep learning model may be used for text detection and recognition.
[0082] Step 130: Input the bill image, text position information and text content information into the bill information extraction model to obtain the bill information output by the bill information extraction model.
[0083] The bill information extraction model can be trained based on the bill image sample set and the corresponding text position information sample set and text content information sample set as well as the sample label sets corresponding to these sample sets.
[0084] In this embodiment, if Figure 2 As shown in the figure, the bill information extraction model includes a multimodal fusion module (corresponding to the Dual Modality Fusion Module in the figure), a graph reasoning module (corresponding to the Graph Reasoning Module in the figure) and a classification module (corresponding to the Classification Module in the figure). The multimodal fusion module can be used for feature fusion, the graph reasoning module can be used for graph reasoning, and the classification module can be used to classify data or predict labels.
[0085] like Figure 3As shown, step 130, inputting the bill image, text position information and text content information into the bill information extraction model to obtain the bill information output by the bill information extraction model further includes:
[0086] Step 131: input the bill image, text position information and text content information into a multimodal fusion module to obtain a visual semantic fusion feature vector output by the multimodal fusion module.
[0087] In this embodiment, each element in the visual semantic fusion feature vector corresponds to a text area, and the value of the element is a fusion of the visual features and semantic features of the text area. The visual semantic fusion feature vector may contain fusion features corresponding to all text areas in the bill image.
[0088] The visual features may be determined by the bill image, and may be the color, shape, texture, edge, outline, etc. of the bill image.
[0089] Semantic features can be determined by text content information, which can be the meaning of the text, such as words, phrases, topics, emotions, entities, etc.
[0090] In this embodiment, the visual semantic fusion feature may be a feature obtained by fusion of the visual feature and the semantic feature of the corresponding text region through calculation, for example, the visual feature and the semantic feature may be added together.
[0091] In this embodiment, the text area of the bill image can be determined by the text position information, thereby obtaining the visual features and semantic features of each text area.
[0092] In this embodiment, the input of the multimodal fusion module is the bill image, text position information and text content information. The multimodal fusion module fuses the visual features and semantic features of each text area of the bill image through the bill image, text position information and text content information. The output of the multimodal fusion module is a visual semantic fusion feature vector.
[0093] Step 132: input the visual semantic fusion feature vector and the text position information into the graph reasoning module, perform reasoning based on the regional position relationship of multiple text regions, and obtain the node feature vector output by the graph reasoning module.
[0094] The region position relationship of the plurality of text regions may represent the position relationship between the respective text regions, for example, may be the distance between the respective text regions.
[0095] In this embodiment, the position of each text region can be determined according to the text position information, so as to calculate the distance between the text regions.
[0096] In this step, each text region may be in a network structure, each text region may be a node in the network, and each node has visual semantic fusion features of the corresponding text region.
[0097] In this embodiment, the network where each text area is located can be inferred based on the regional position relationship of multiple text areas, and the features of adjacent nodes can be added to the visual semantic fusion features of the text area corresponding to each node to obtain the node features of the text area. The vector composed of the node features of each text area is the node feature vector.
[0098] In this embodiment, the graph reasoning module performs reasoning based on the regional positional relationship of multiple text regions, and adds the features of adjacent text regions to the visual semantic fusion features of each text region. The correlation between the text regions can be inferred based on the positional relationship of the text regions, thereby better understanding the overall context of the bill image, enhancing the understanding of the context, and reducing ambiguity. The obtained node features are more accurate, thereby improving the accuracy of the subsequent classification process.
[0099] Step 133: input the node feature vector into the classification module to obtain the bill information output by the classification module.
[0100] In this embodiment, the input of the classification module is the node feature vector, the classification module classifies each node, ie each text area, according to the node feature vector, and the output of the classification module is the bill information.
[0101] The bill information is the information contained in the bill image, for example, the name of the issuer is A, the name of the payee is B, the bill amount is C, the issue date is D, etc.
[0102] In this step, the classification module can use support vector machine (SVM), neural network (NN) and decision tree (DT) to achieve classification.
[0103] In the related art, methods for extracting bill information are generally divided into two categories. One category is methods based on template matching or rules, such as regularized matching for bill information extraction. Such methods are usually based on industry knowledge, have poor versatility, and are mostly used for text image types with fixed layouts. The other category is methods based on deep learning. Such methods have low execution efficiency, cannot match text information and spatial information, and are prone to ambiguity.
[0104] In an embodiment of the present application, the visual features and semantic features corresponding to each text area in the bill image are fused in the multimodal fusion module of the bill information extraction model. In the graph reasoning module of the bill information extraction model, reasoning is performed based on the regional position relationship of multiple text areas, so that the features of other text areas are added to the visual semantic fusion features corresponding to each text area. The visual features and semantic features are fused and the position information of the text in the bill image is fully considered. The correlation between them can be inferred based on the positional relationship of each text area, thereby enhancing the understanding of the context and improving the efficiency and accuracy of bill information extraction.
[0105] According to the bill information extraction method of the present application, the visual features and semantic features corresponding to each text area in the bill image are fused through the bill information extraction model, and reasoning is performed based on the regional position relationship of multiple text areas. By fusion of visual features and semantic features and fully considering the position information of the text in the bill image, the correlation between them can be inferred based on the position relationship of each text area, thereby enhancing the understanding of the context and improving the efficiency and accuracy of bill information extraction.
[0106] In some embodiments, the visual semantic fusion feature vector and the text position information are input into the graph reasoning module, and reasoning is performed according to the regional position relationship of multiple text regions to obtain the node feature vector output by the graph reasoning module, including:
[0107] Determining a region position relationship of a plurality of text regions based on the text position information;
[0108] Based on the regional position relationship, the weight information between the nodes in the visual semantic fusion feature vector is determined, and each node in the visual semantic fusion feature vector corresponds to a text region;
[0109] The feature vector is fused based on the weight information and visual semantics to obtain the node feature vector.
[0110] In this embodiment, the text position information may be coordinate information corresponding to the text area, for example, the vertex coordinates and the center point coordinates corresponding to the text area.
[0111] In this embodiment, the distance between text regions and the relative shape relationship between text regions may be determined based on the text position information, thereby determining the region position relationship.
[0112] In this embodiment, the distance between the text areas may be the distance between the coordinates of the center points corresponding to the text areas.
[0113] In this embodiment, the length and width of the text area can be obtained based on the coordinates of the rectangle vertices corresponding to the text area, and the relative shape relationship between the text areas can be determined based on the length and width of the text area.
[0114] In this embodiment, the region position relationship between the two text regions may be determined based on the distance between the two text regions and the relative shape relationship between the two text regions.
[0115] For example, there are text regions i and j in the bill image, and the coordinates of the center point of text region i are (x i ,y i ), the coordinates of the center point of text area j are (x j ,y j ).
[0116] Calculate the horizontal distance between text area i and text area j as Δx ij =x j -x i , calculate the vertical distance between text area i and text area j as Δy ij =y j -y i .
[0117] The relative spatial position distance between text region i and text region j is Where d is a normalization constant, and || represents the concat operation, i.e., the concatenation operation.
[0118] The relative shape relationship between text region i and text region j is where w i is the horizontal length of text area i, h i is the vertical length of text area i, w j is the horizontal length of text area j, h j is the vertical length of text area j, and Represents relative shape information.
[0119] The area position relationship between text area i and text area j is
[0120] In this embodiment, the text regions can form a network, each node of the network corresponds to a text region, each element in the visual semantic fusion feature vector corresponds to a visual semantic fusion feature of a text region, and the elements in the visual semantic fusion feature vector corresponding to the text region can be called nodes.
[0121] In this embodiment, determining the weight information between nodes in the visual semantic fusion feature vector is to determine the weights of the edges between nodes in the network composed of each text region.
[0122] For example, the region position relationship between text region i and text region j is r ij, the regional position relationship r between text region i and text region j ij The edge weight e between the embedded text region i and the text region j ij ∈ε,
[0123]
[0124] f ij =n i ||e′ ij ||n j
[0125] e ij =M(f ij )
[0126] Among them, E is a region position relationship r ij Embedding into a linear transformation of a 1D representation.
[0127] It refers to the L2 normalization operation, that is, in deep learning, normalization is performed through the L2 norm, dividing a vector by its L2 norm so that the length of the vector (L2 norm) is equal to 1.
[0128] f ij Yes i and n j The concat representation and normalized spatial relationship embedding, n i is the visual semantic fusion feature of text region i, n j is the visual semantic fusion feature of text region j.
[0129] M is a ij becomes a scalar e ij Multilayer Perception (MLP) operation.
[0130] In this embodiment, the node feature vector may be obtained based on the weight information and the visual semantic fusion feature vector. For example, the weight information and the visual semantic fusion feature vector may be multiplied.
[0131] In some embodiments, the multimodal fusion module includes a visual feature extraction module, a semantic feature extraction module and a feature fusion module, and the bill image, text location information and text content information are input into the multimodal fusion module to obtain a visual semantic fusion feature vector output by the multimodal fusion module, including:
[0132] Input the bill image and text position information into the visual feature extraction module to obtain the visual feature vector output by the visual feature extraction module;
[0133] Inputting the text content information and the text position information into the semantic feature extraction module to obtain the semantic feature vector output by the semantic feature extraction module;
[0134] The visual feature vector and the semantic feature vector are input into the feature fusion module to obtain the visual-semantic fusion feature vector.
[0135] Among them, the visual feature extraction module can be used to extract the visual features of each text area in the bill image, the semantic feature extraction module can be used to extract the semantic features of each text area in the bill image, and the feature fusion module can be used to fuse the visual features and semantic features of each text area in the bill image.
[0136] In this embodiment, the visual feature extraction module can extract visual features based on the U-net network, and the semantic feature extraction module can extract semantic features based on the bidirectional long short-term memory network (Bidirectional Long Short-Term Memory, Bi-LSTM).
[0137] In this embodiment, each element in the visual feature vector corresponds to the visual feature of a text area, and each element in the semantic feature vector corresponds to the semantic feature of a text area. The visual feature vector may include the visual features of all text areas, and the semantic feature vector may include the semantic features of all text areas.
[0138] In this embodiment, the feature fusion module may operate the visual features and the semantic features to fuse the visual features and the semantic features, for example, adding the visual features and the semantic features.
[0139] For another example, feature fusion is performed through Kronecker product, v i is the visual feature of text region i, t i is the semantic feature of text region i, and the visual semantic fusion feature of text region i is P is a learnable linear transformation, represents the Kronecker product operation.
[0140] In this embodiment, the visual features and semantic features of each text region in the bill image are fused to obtain the visual semantic fusion features of each text region. The visual semantic fusion features of all text regions can form a visual semantic fusion feature vector.
[0141] In some embodiments, the bill image and text position information are input into a visual feature extraction module to obtain a visual feature vector output by the visual feature extraction module, including:
[0142] Performing convolution processing on the bill image to obtain a first feature map corresponding to the bill image;
[0143] Based on the text position information, the text position information corresponding to each text area is mapped to the first feature map, and the mapped first feature map is pooled with a region of interest to obtain a visual feature vector.
[0144] For example, the bill image and text position information are input into the visual feature extraction module, and convolution processing is performed based on U-Net to extract visual features, and the feature map seg1 of the last layer of the convolutional neural network (CNN), that is, the first feature map, is obtained.
[0145] The text position information is mapped into the first feature map, and the mapped first feature map seg1 is subjected to region of interest pooling (ROIPooling) to realize feature extraction and obtain a visual feature vector.
[0146] In some embodiments, inputting text content information and text position information into a semantic feature extraction module to obtain a semantic feature vector output by the semantic feature extraction module includes:
[0147] Map the text content information to the one-hot encoding space of the semantic feature extraction module to obtain a one-hot encoding vector;
[0148] Based on the text position information and the one-hot encoding vector, a semantic feature vector is obtained.
[0149] For example, first, a character set table with a length of 11377 is collected, and the character set table covers numbers (0-9), letters (az, AZ), Chinese characters, special characters, etc., and characters not in the character table are uniformly marked as "unkown".
[0150] Then, the text content information is mapped character by character to the one-hot encoding space, which can be a 32-dimensional one-hot encoding vector Where i represents the i-th text region, j represents the j-th character in the text region, and the text position information can determine the text region.
[0151] Next, Project it into a low-dimensional space and extract semantic features based on a bidirectional long short-term memory network (Bi-LSTM) Where W ij Represents the projection matrix of a one-hot encoded vector.
[0152] In some embodiments, the node feature vector is input into a classification module to obtain the ticket information output by the classification module, including:
[0153] Based on the node feature vector, each text area is classified to obtain the label information corresponding to each text area;
[0154] Based on the tag information, obtain the ticket information.
[0155] In this embodiment, the classification module can perform classification based on the fully connected layer and activation function of the neural network.
[0156] In this embodiment, during the training of the classification module, the following loss function can be used for supervised learning:
[0157]
[0158] Among them, y i is the category of the i-th text region, CE (Cross Entropy Loss) is the cross entropy loss function, n i L is the node feature of the i-th text region.
[0159] In this embodiment, the node feature vector may include node features of all text regions, and each text region may be classified based on the node feature vector to obtain label information corresponding to each text region.
[0160] In this embodiment, the label information may be a number, such as a first label, a second label, a third label, and the like.
[0161] In this embodiment, the label information may correspond to a specific bill information category, for example, the first label corresponds to the issuer, the second label corresponds to the payee, and the third label corresponds to the bill amount.
[0162] In this embodiment, the bill information can be obtained based on the label information of each text area and the text content information of each text area. For example, the name of the issuer is A, the name of the payee is B, the bill amount is C, the issue date is D, etc., where the issuer, payee, bill amount, and issue date are the label information corresponding to the text area, and the specific content of the issuer, payee, bill amount, and issue date is the text content information in the corresponding text area.
[0163] In some embodiments, Figure 4 As shown, the method may also include inputting an image, i.e., image loading, image pre-processing, image parsing, and outputting structured key information.
[0164] Before performing text detection and recognition on the bill image, the method may include:
[0165] Image loading, that is, loading the bill image from the storage medium or the network, and reading the bill image into the memory for subsequent processing.
[0166] The image pre-processing stage mainly includes image grayscale, image denoising, image scaling, data normalization and standardization, data format conversion, batch processing, etc.
[0167] The image analysis stage mainly includes layout analysis of bill images, document orientation classification, bill segmentation, bill classification, seal removal, etc.
[0168] Perform text detection and recognition on the bill image after image analysis to determine multiple text position information and multiple text content information of the bill image.
[0169] After the bill image, text position information and text content information are input into the key information extraction module, i.e., the bill information extraction model, the method may include:
[0170] The stage of outputting structured key information mainly includes operations such as text error correction and result formatting.
[0171] The bill information extraction method provided in the embodiment of the present application can be executed by a bill information extraction device. In the embodiment of the present application, the bill information extraction device provided in the embodiment of the present application is described by taking the bill information extraction device executing the bill information extraction method as an example.
[0172] The embodiment of the present application also provides a device for extracting bill information.
[0173] like Figure 5 As shown, the bill information extraction device includes:
[0174] An acquisition module 510 is used to acquire a bill image to be processed, where the bill image includes a plurality of text areas;
[0175] The first processing module 520 is used to perform text detection and recognition on the bill image, determine multiple text position information and multiple text content information of the bill image, and the text position information and the text content information correspond to the text area one by one;
[0176] The second processing module 530 is used to input the bill image, text position information and text content information into the bill information extraction model to obtain the bill information output by the bill information extraction model;
[0177] The bill information extraction model includes a multimodal fusion module, a graph reasoning module and a classification module, and the second processing module is further used for:
[0178] Input the bill image, text location information and text content information into the multimodal fusion module to obtain the visual semantic fusion feature vector output by the multimodal fusion module;
[0179] The visual semantic fusion feature vector and text position information are input into the graph reasoning module, and reasoning is performed based on the regional position relationship of multiple text regions to obtain the node feature vector output by the graph reasoning module;
[0180] The node feature vector is input into the classification module to obtain the ticket information output by the classification module.
[0181] According to the bill information extraction method of the present application, the visual features and semantic features corresponding to each text area in the bill image are fused through the bill information extraction model, and reasoning is performed based on the regional position relationship of multiple text areas. By fusion of visual features and semantic features and fully considering the position information of the text in the bill image, the correlation between them can be inferred based on the position relationship of each text area, thereby enhancing the understanding of the context and improving the efficiency and accuracy of bill information extraction.
[0182] In some embodiments, the second processing module 530 is used to determine the region position relationship of multiple text regions based on the text position information;
[0183] Based on the regional position relationship, the weight information between the nodes in the visual semantic fusion feature vector is determined, and each node in the visual semantic fusion feature vector corresponds to a text region;
[0184] The feature vector is fused based on the weight information and visual semantics to obtain the node feature vector.
[0185] In some embodiments, the second processing module 530 is used to input the bill image and text position information into the visual feature extraction module to obtain the visual feature vector output by the visual feature extraction module;
[0186] Inputting the text content information and the text position information into the semantic feature extraction module to obtain the semantic feature vector output by the semantic feature extraction module;
[0187] The visual feature vector and the semantic feature vector are input into the feature fusion module to obtain the visual-semantic fusion feature vector.
[0188] In some embodiments, the second processing module 530 is used to perform convolution processing on the bill image to obtain a first feature map corresponding to the bill image;
[0189] Based on the text position information, the text position information corresponding to each text area is mapped to the first feature map, and the mapped first feature map is pooled with a region of interest to obtain a visual feature vector.
[0190] In some embodiments, the second processing module 530 is used to map the text content information to the one-hot encoding space of the semantic feature extraction module to obtain a one-hot encoding vector;
[0191] Based on the text position information and the one-hot encoding vector, a semantic feature vector is obtained.
[0192] In some embodiments, the second processing module 530 is used to classify each text region based on the node feature vector to obtain label information corresponding to each text region;
[0193] Based on the tag information, obtain the ticket information.
[0194] The bill information extraction device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not make specific limitations.
[0195] The bill information extraction device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0196] The bill information extraction device provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented by the method embodiment are not described here.
[0197] The embodiment of the present application also provides a bill information extraction system.
[0198] like Figure 6 As shown, the bill information extraction system includes a server and a client.
[0199] The client communicates with the server, the client sends the bill image to be processed to the server, and the server executes the above-mentioned bill information extraction method and sends the bill information corresponding to the bill image to the client;
[0200] Among them, the server is deployed with a bill information extraction model.
[0201] According to the bill information extraction method of the present application, the visual features and semantic features corresponding to each text area in the bill image are fused through the bill information extraction model, and reasoning is performed based on the regional position relationship of multiple text areas. By fusion of visual features and semantic features and fully considering the position information of the text in the bill image, the correlation between them can be inferred based on the position relationship of each text area, thereby enhancing the understanding of the context and improving the efficiency and accuracy of bill information extraction.
[0202] In some embodiments, Figure 6 As shown, the bill information extraction system also includes:
[0203] The model training end is connected to the server for communication. The model training end is used to train the bill information extraction model and deploy the bill information extraction model to the server.
[0204] The server receives the bill information extraction request sent by the client, saves the bill image sent by the client, and outputs it to the model training end;
[0205] When the number of bill images received by the model training end is equal to the image number threshold, bill images of the image number threshold number are collected and added to the training set of the bill information extraction model, and the bill information extraction model is fine-tuned to obtain an updated bill information extraction model;
[0206] The model training end outputs the updated bill information extraction model to the server and deploys the updated bill information extraction model on the server.
[0207] In this embodiment, the image number threshold is a preset value. For example, the image number threshold may be 500. That is, after the model training end receives 500 bill images, the model training end recycles the 500 bill images, adds them to the training set of the bill information extraction model, and performs fine-tuning training on the bill information extraction model.
[0208] In this embodiment, the model training end can pre-label and manually verify the bill image dataset and start model fine-tuning training.
[0209] In this embodiment, during the training and reasoning process of the bill information extraction method of the embodiment of the present application, the length of the data can be set to a constant value by means of the padding value of the neural network.
[0210] For example, setting the total number of text lines in a single image to 800 and the total number of characters in a single text line to 400 can facilitate subsequent model conversion.
[0211] In this embodiment, the bill information extraction model trained by the model training end can be compressed and quantized through the tensor inference engine TensorRT to reduce inference time and memory usage.
[0212] In this embodiment, the bill information extraction model and the corresponding pre- and post-processing can be deployed to the server, and regularization and rule-based processes can be added to implement a full set of service links for bill information extraction.
[0213] In some embodiments, Figure 7 As shown, an embodiment of the present application also provides an electronic device 700, including a processor 701, a memory 702, and a computer program stored in the memory 702 and executable on the processor 701. When the program is executed by the processor 701, each process of the above-mentioned bill information extraction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0214] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.
[0215] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned bill information extraction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0216] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0217] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned bill information extraction method when executed by a processor.
[0218] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0219] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned bill information extraction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0220] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0221] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0222] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0223] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
[0224] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0225] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.
Claims
1. A method for extracting bill information, characterized in that: include: Acquire a bill image to be processed, wherein the bill image includes a plurality of text areas; Performing text detection and recognition on the bill image to determine a plurality of text position information and a plurality of text content information of the bill image, wherein the text position information and the text content information correspond to the text area in a one-to-one manner; Inputting the bill image, the text position information and the text content information into a bill information extraction model to obtain the bill information output by the bill information extraction model; The bill information extraction model includes a multimodal fusion module, a graph reasoning module and a classification module. The bill image, the text position information and the text content information are input into the bill information extraction model to obtain the bill information output by the bill information extraction model, further including: Inputting the bill image, the text position information and the text content information into the multimodal fusion module to obtain a visual semantic fusion feature vector output by the multimodal fusion module; Inputting the visual semantic fusion feature vector and the text position information into the graph reasoning module, performing reasoning according to the regional position relationship of the multiple text regions, and obtaining a node feature vector output by the graph reasoning module; Inputting the node feature vector into the classification module to obtain the bill information output by the classification module; The step of inputting the visual semantic fusion feature vector and the text position information into the graph reasoning module, and performing reasoning according to the regional position relationship of the multiple text regions to obtain the node feature vector output by the graph reasoning module includes: Based on the text position information, determining the region position relationship of the plurality of text regions; Based on the region position relationship, determining weight information between nodes in the visual semantic fusion feature vector, each node in the visual semantic fusion feature vector corresponds to one text region; Obtaining a node feature vector based on the weight information and the visual semantic fusion feature vector; The region position relationship between two text regions is determined based on the distance between the two text regions and the relative shape relationship between the two text regions.
2. The method for extracting bill information according to claim 1, characterized in that: The multimodal fusion module includes a visual feature extraction module, a semantic feature extraction module and a feature fusion module. The bill image, the text position information and the text content information are input into the multimodal fusion module to obtain a visual semantic fusion feature vector output by the multimodal fusion module, including: Inputting the bill image and the text position information into the visual feature extraction module to obtain a visual feature vector output by the visual feature extraction module; Inputting the text content information and the text position information into the semantic feature extraction module to obtain a semantic feature vector output by the semantic feature extraction module; The visual feature vector and the semantic feature vector are input into the feature fusion module to obtain the visual-semantic fusion feature vector.
3. The method for extracting bill information according to claim 2, characterized in that: The step of inputting the bill image and the text position information into the visual feature extraction module to obtain the visual feature vector output by the visual feature extraction module includes: Performing convolution processing on the bill image to obtain a first feature map corresponding to the bill image; Based on the text position information, the text position information corresponding to each of the text regions is mapped to the first feature map, and the mapped first feature map is pooled with a region of interest to obtain the visual feature vector.
4. The method for extracting bill information according to claim 2, characterized in that: The step of inputting the text content information and the text position information into the semantic feature extraction module to obtain the semantic feature vector output by the semantic feature extraction module includes: Mapping the text content information to the one-hot encoding space of the semantic feature extraction module to obtain a one-hot encoding vector; Based on the text position information and the one-hot encoding vector, the semantic feature vector is obtained.
5. The method for extracting bill information according to any one of claims 1 to 4, characterized in that: The step of inputting the node feature vector into the classification module to obtain the ticket information output by the classification module includes: Based on the node feature vector, classify each of the text regions to obtain label information corresponding to each of the text regions; Based on the tag information, the ticket information is obtained.
6. A bill information extraction device, characterized in that: include: An acquisition module, used for acquiring a bill image to be processed, wherein the bill image includes a plurality of text areas; A first processing module, configured to perform text detection and recognition on the bill image, and determine a plurality of text position information and a plurality of text content information of the bill image, wherein the text position information and the text content information correspond to the text area in a one-to-one manner; A second processing module, configured to input the bill image, the text position information and the text content information into a bill information extraction model to obtain the bill information output by the bill information extraction model; The bill information extraction model includes a multimodal fusion module, a graph reasoning module and a classification module, and the second processing module is further used for: Inputting the bill image, the text position information and the text content information into the multimodal fusion module to obtain a visual semantic fusion feature vector output by the multimodal fusion module; Inputting the visual semantic fusion feature vector and the text position information into the graph reasoning module, performing reasoning according to the regional position relationship of the multiple text regions, and obtaining a node feature vector output by the graph reasoning module; Inputting the node feature vector into the classification module to obtain the bill information output by the classification module; The second processing module is further used for: Based on the text position information, determining the region position relationship of the plurality of text regions; Based on the region position relationship, determining weight information between nodes in the visual semantic fusion feature vector, each node in the visual semantic fusion feature vector corresponds to one text region; A node feature vector is obtained based on the weight information and the visual semantic fusion feature vector.
7. A bill information extraction system, characterized in that: include: server; A client, wherein the client is in communication connection with the server, the client sends a receipt image to be processed to the server, and the server executes the receipt information extraction method according to any one of claims 1 to 5, and sends receipt information corresponding to the receipt image to the client; Among them, the server is deployed with a bill information extraction model.
8. The bill information extraction system according to claim 7, characterized in that: Also includes: A model training end, the model training end is connected to the server for communication, the model training end is used to train the bill information extraction model and deploy the bill information extraction model to the server; The server receives the bill information extraction request sent by the client, saves the bill image sent by the client, and outputs it to the model training end; When the number of the bill images received by the model training end is equal to the image number threshold, the bill images with the image number threshold are added to the training set of the bill information extraction model, and the bill information extraction model is fine-tuned to obtain an updated bill information extraction model; The model training end outputs the updated bill information extraction model to the server, and deploys the updated bill information extraction model on the server.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the bill information extraction method as described in any one of claims 1-5 is implemented.
