Method, device, equipment and storage medium for extracting key information from bank receipt

Abstract

The present invention discloses a method for extracting key information of a bank receipt, comprising: performing text block recognition on a received bank receipt image to obtain each text block; extracting and fusing the image visual features and text features of each text block to obtain a multimodal feature; respectively calculating the edge weights between each text block, and calculating the edge weight features; fusing the multimodal features and the edge weight features to obtain a fused feature; performing classification operations on each fused feature using an information classification model to obtain a key information category of each text block; screening each target key information category belonging to a preset key information category group, extracting key information from the target text block corresponding to each target key information category, and obtaining the key information of the bank receipt. The present invention avoids the cumbersome setting of templates, effectively improves the accuracy of extracting key information of a bank receipt, and enhances robustness. The present invention also discloses a device, equipment, and storage medium, which have corresponding technical effects.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and computer-readable storage medium for extracting key information from a bank receipt. Background Art

[0002] With the continuous development of artificial intelligence technology, the automatic extraction of key information can effectively reduce manpower and improve work efficiency. Extracting key information from document images is the key to achieving automated office. Bank receipts are valid vouchers for individuals or units to handle business at banks. It is necessary to extract key information from bank receipt images.

[0003] The existing method for extracting key information from bank receipts mainly uses a template matching-based method to extract key information, which cannot effectively work on bank receipt images that do not match the template. The template setting is cumbersome and is not robust to text recognition errors.

[0004] In summary, how to effectively solve the problems of cumbersome template setting and low robustness of the existing key information extraction method of bank receipts is an urgent problem that technicians in this field need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a method for extracting key information of a bank receipt, which avoids the cumbersome setting of templates, effectively improves the accuracy of extracting key information of a bank receipt, and enhances robustness; another purpose of the present invention is to provide a device, equipment and computer-readable storage medium for extracting key information of a bank receipt.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] A method for extracting key information of a bank receipt, comprising:

[0008] Perform text block recognition on the received bank receipt image to obtain each text block;

[0009] Extracting image visual features and text features of each text block, and fusing the image visual features and the text features to obtain multimodal features;

[0010] Calculating edge weights between each pair of the text blocks respectively, and calculating edge weight features according to the edge weights corresponding to each text block;

[0011] Adaptively and dynamically fuse the multimodal features and edge weight features of each text block to obtain fused features;

[0012] Using the pre-trained information classification model to perform classification operations on each of the fused features, to obtain key information categories corresponding to each of the text blocks;

[0013] Target key information categories belonging to a preset key information category group are screened from the key information categories, and key information extraction operations are performed on target text blocks corresponding to the target key information categories to obtain bank receipt key information.

[0014] In a specific embodiment of the present invention, the classification operation of each fusion feature is performed using the pre-trained information classification model, including:

[0015] Using the pre-trained graph-aware reasoning model to perform encoding and decoding operations on each of the fused features to obtain each target feature;

[0016] The pre-trained information classification model is used to perform classification operations on each of the target features.

[0017] In a specific implementation of the present invention, after screening each target key information category belonging to a preset key information category group from each of the key information categories, before performing a key information extraction operation on the text blocks corresponding to each of the target key information categories, the method further includes:

[0018] Using the information classification model to perform classification operations on the edge weight features of each target text block to obtain classification results of each key-value pair;

[0019] Determine whether the classification results of each of the key-value pairs are consistent with the preset key-value pair information;

[0020] If so, the step of performing key information extraction operations on the text blocks corresponding to each target key information category is executed.

[0021] In a specific implementation of the present invention, the edge weight features of each target text block are classified using the information classification model, including:

[0022] The edge weight features of each target text block are classified using an information classification model that minimizes the edge weight loss function.

[0023] In a specific embodiment of the present invention, the classification operation of each fusion feature is performed using the pre-trained information classification model, including:

[0024] The information classification model obtained by pre-training and minimizing the information loss function is used to perform classification operations on each of the fused features.

[0025] In a specific implementation of the present invention, the text block recognition is performed on the received bank receipt image, including:

[0026] Optical character recognition technology is used to identify text blocks in the received bank receipt images.

[0027] A device for extracting key information of a bank receipt, comprising:

[0028] The text block recognition module is used to perform text block recognition on the received bank receipt image to obtain each text block;

[0029] A multimodal feature acquisition module, used for extracting image visual features and text features of each text block, and fusing the image visual features and the text features to obtain multimodal features;

[0030] An edge weight feature calculation module, used to calculate the edge weights between each of the text blocks, and calculate the edge weight features according to the edge weights corresponding to each text block;

[0031] The fusion feature acquisition module is used to adaptively and dynamically fuse the multimodal features and edge weight features of each text block to obtain fusion features;

[0032] A key information category acquisition module is used to classify each of the fusion features using a pre-trained information classification model to obtain the key information category corresponding to each of the text blocks;

[0033] The bank receipt key information acquisition module is used to filter each target key information category belonging to a preset key information category group from each of the key information categories, perform key information extraction operations on the target text blocks corresponding to each of the target key information categories, and obtain the bank receipt key information.

[0034] In a specific implementation of the present invention, the key information category obtaining module includes:

[0035] A feature encoding and decoding submodule is used to use the pre-trained graph perception reasoning model to perform encoding and decoding operations on each of the fusion features to obtain each target feature;

[0036] The feature classification submodule is used to perform classification operations on each of the target features using the pre-trained information classification model.

[0037] A device for extracting key information of a bank receipt, comprising:

[0038] Memory for storing computer programs;

[0039] A processor is used to implement the steps of the method for extracting key information of the bank receipt as described above when executing the computer program.

[0040] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for extracting key information from a bank receipt as described above.

[0041] The key information extraction method of a bank receipt provided by the present invention performs text block recognition on a received bank receipt image to obtain each text block; extracts image visual features and text features of each text block, and fuses the image visual features and text features to obtain multimodal features; calculates edge weights between each text block, and calculates edge weight features according to edge weights corresponding to each text block; adaptively and dynamically fuses the multimodal features and edge weight features of each text block to obtain fused features; uses a pre-trained information classification model to classify each fused feature to obtain key information categories corresponding to each text block; selects each target key information category belonging to a preset key information category group from each key information category, and performs key information extraction operations on target text blocks corresponding to each target key information category to obtain key information of the bank receipt.

[0042] It can be seen from the above technical solution that by fusing the image visual features and text features of each text block in the bank receipt image, a multimodal feature is obtained, and the edge weight features of each text block are calculated, and the multimodal feature and the edge weight feature are adaptively and dynamically fused, so that the obtained fused feature can better represent the global information. The information classification model is pre-trained, and the fused features are classified using the information classification model, thereby realizing the extraction of key information of the bank receipt. The present invention avoids the cumbersome setting of the template, and by classifying the fused features, the accuracy of extracting key information of the bank receipt is effectively improved, and the robustness is enhanced.

[0043] Correspondingly, the present invention also provides a bank receipt key information extraction device, equipment and computer-readable storage medium corresponding to the above-mentioned bank receipt key information extraction method, which has the above-mentioned technical effects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 It is an implementation flow chart of a method for extracting key information of a bank receipt in an embodiment of the present invention;

[0046] Figure 2Another implementation flow chart of the method for extracting key information of a bank receipt in an embodiment of the present invention;

[0047] Figure 3 This is a structural block diagram of a device for extracting key information from a bank receipt in an embodiment of the present invention;

[0048] Figure 4 This is a structural block diagram of a key information extraction device for a bank receipt in an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of the specific structure of a device for extracting key information from a bank receipt provided in this embodiment. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0051] See also Figure 1 , Figure 1 This is a flow chart of an implementation of a method for extracting key information from a bank receipt in an embodiment of the present invention. The method may include the following steps:

[0052] S101: Perform text block recognition on the received bank receipt image to obtain various text blocks.

[0053] When receiving the bank receipt image, the bank receipt key information extraction center performs text block (box) recognition on the received bank receipt image to obtain each text block. When performing text block recognition on the bank receipt image, generally, each text block is recognized for each line of text.

[0054] S102: extracting image visual features and text features of each text block, and fusing the image visual features and text features to obtain multimodal features.

[0055] Each text block contains image visual features and text features. After identifying each text block in the bank receipt image, the image visual features and text features of each text block are extracted. For example, the image visual features and text features can be extracted using the lightweight mobilenetv3-small model structure and the Bi_LSTM model respectively. The extracted image visual features and text features are fused to obtain multimodal features.

[0056] S103: Calculate the edge weights between each pair of text blocks respectively, and calculate the edge weight feature according to each edge weight corresponding to each text block.

[0057] It is defined that all text blocks on the image are connected in pairs to form a topological structure. After identifying each text block in the bank receipt image, the edge weights between each text block are calculated respectively, and the edge weight features are calculated according to the edge weights corresponding to each text block. The edge weight feature encodes the spatial information of each text block in the overall layout of the bank receipt image.

[0058] S104: Adaptively and dynamically fuse the multimodal features and edge weight features of each text block to obtain fused features.

[0059] After fusing the image visual features and text features to obtain multimodal features and calculating the edge weight features, the multimodal features and edge weight features of each text block are adaptively and dynamically fused to obtain fused features. The multimodal features of each text block are embedded in the edge weight features (i.e., the edge relationship information of the topological graph), and the fused features are jointly formed as the node information of the topological graph, i.e., each text block can be regarded as a node.

[0060] S105: Using the pre-trained information classification model to perform classification operations on each fusion feature, and obtain the key information category corresponding to each text block.

[0061] According to the key information that needs to be determined in the bank receipt, a category label is formulated. For example, 16 key information categories required in business scenarios are pre-formulated, including: Payer name keyword (Payer_name_key), Payer name value (Payer_name_value), Payee name keyword (Payee_name_key), Payee name value (Payee_name_value), Payer account keyword (Payer_account_key), Payer account value (Payer_account_value), Payee account keyword (Payee_account_key), Payee account value (Payee_account_value), Amount keyword (Money_key), Amount value (Money_value), Date keyword (Time_key), Date value (Time_value), Summary keyword (Summary_key), Summary value (Summary_value), Others, Ignore. Among them, the first 14 are mainly key values ​​of key information, Others represents unimportant information, and Ignore represents the label value of the text block with empty text.

[0062] The model is trained using a training set of text blocks annotated with corresponding category labels to obtain an information classification model. After adaptively and dynamically fusing the multimodal features and edge weight features of each text block to obtain the fused features, the information classification model is used to classify each fused feature to obtain the key information category corresponding to each text block.

[0063] S106: Filter target key information categories belonging to a preset key information category group from the key information categories, perform key information extraction operations on target text blocks corresponding to the target key information categories, and obtain bank receipt key information.

[0064] A key information category group consisting of key information categories to be selected is pre-set. After the pre-trained information classification model is used to classify each fused feature and obtain the key information category corresponding to each text block, each target key information category belonging to the preset key information category group is screened from each key information category. The key information extraction operation is performed on the target text block corresponding to each target key information category to obtain the key information of the bank receipt. Multimodal features are obtained by fusing the image visual features and text features of each text block in the bank receipt image, and the edge weight features of each text block are calculated. The multimodal features are adaptively and dynamically fused with the edge weight features, so that the obtained fused features can better represent the global information and improve the accuracy of bank receipt key information extraction in complex scenarios.

[0065] It can be seen from the above technical solution that by fusing the image visual features and text features of each text block in the bank receipt image, a multimodal feature is obtained, and the edge weight features of each text block are calculated, and the multimodal feature and the edge weight feature are adaptively and dynamically fused, so that the obtained fused feature can better represent the global information. The information classification model is pre-trained, and the fused features are classified using the information classification model, thereby realizing the extraction of key information of the bank receipt. The present invention avoids the cumbersome setting of the template, and by classifying the fused features, the accuracy of extracting key information of the bank receipt is effectively improved, and the robustness is enhanced.

[0066] It should be noted that, based on the above embodiment, the embodiment of the present invention also provides corresponding improved solutions. In the subsequent embodiments, the same steps or corresponding steps as those in the above embodiment can be referenced to each other, and the corresponding beneficial effects can also be referenced to each other, which will not be repeated one by one in the following improved embodiments.

[0067] See also Figure 2 , Figure 2 This is another implementation flow chart of the method for extracting key information of a bank receipt in an embodiment of the present invention. The method may include the following steps:

[0068] S201: Using optical character recognition technology to perform text block recognition on the received bank receipt image to obtain various text blocks.

[0069] After receiving the bank receipt image to be extracted, the optical character recognition technology (OCR) is used to perform text block recognition on the received bank receipt image to obtain each text block. For each bank receipt image, the optical character recognition technology is used to perform text detection and text recognition to obtain the text block position information and corresponding text content of each bank receipt image. In which, the detected text block is represented as r i ={x i ,y i ,h i , w i ,s i}, x i ,y i ,h i , w i Respectively represent the horizontal and vertical coordinates of the upper left corner of the text block and its height and width, s i Represents the recognized text content.

[0070] S202: extracting image visual features and text features of each text block, and fusing the image visual features and text features to obtain multimodal features.

[0071] The lightweight mobilenetv3-small model structure is used to extract the image visual features of the text block, and the feature dimension is assumed to be 16.

[0072] Predefine the font library, and the font size can be set according to the actual application scenario, including common Chinese characters, rare characters, and alphanumeric symbols. Map the text into a vector form according to the index information of each character in the font library, pad the text vector to a vector of the same length, and then project it into a low-dimensional space, and then send it to the Bi_LSTM model to obtain text features. Set the dimension of the final text feature to 256.

[0073] A fully connected layer is used to transform the dimensions of the above two features to 780, and then a dot product operation is performed to obtain the intermediate features. Then a fully connected layer is performed to obtain the final multimodal feature VAT, with a feature dimension of 256.

[0074] S203: Calculate the edge weights between each pair of text blocks respectively, and calculate the edge weight feature according to each edge weight corresponding to each text block.

[0075] When calculating the edge weight feature E, the spatial relationship information is first initialized:

[0076] Δx ij =x j -x i ;

[0077] Δy ij =y j -y i ;

[0078]

[0079]

[0080]

[0081] Where Δx ij Represents the horizontal distance between two text blocks, Δy ij represents the vertical distance between two text blocks, d is a constant of 10, and || represents the connection operation. Indicates the relative spatial position of two text blocks. Encodes the aspect ratio and relative shape information of the text block, r ij It means connecting the relative spatial position with the aspect ratio and relative shape information of the text block, that is, the spatial information finally obtained, the size of which is (n×n×5), where n represents the number of text blocks on the entire bank receipt image.

[0082] The spatial information is transformed and embedded into a vector feature of dimension 256. ij , can be calculated by the following formula:

[0083]

[0084] e ij =M(n i ||e i ' j ||n j );

[0085] Where L represents a linear transformation, represents l2 normalization, e i ' j Represents r ij The intermediate features obtained after linear transformation and normalization, M(·) represents a simple multi-layer perceptron structure, n i Represents the multimodal features of text block i, n j Represents the multimodal features of text block j. Finally, the edge weight feature e of the topological graph is obtained ij This constitutes the edge weight feature set E with a dimension of 768.

[0086] S204: Adaptively and dynamically fuse the multimodal features and edge weight features of each text block to obtain fused features.

[0087] The multimodal features and edge weight features obtained above are adaptively fused. The core of this method is the attention coefficient of the adaptive dynamic learning feature. The main steps are as follows:

[0088] The edge weight features of the topology graph are passed through a fully connected layer to reduce the dimension of the feature to 256, which is the same as the multimodal feature dimension. In order to learn the corresponding attention coefficient, the multimodal features and edge weight features are connected, and a fully connected layer parameterized by the weight matrix W is introduced to capture the relationship between the connection features. Then the LeakyReLU activation function is applied, and then the output unit of the LeakyRelu activation function is processed by the softmax function and L2-norm normalization. The corresponding expression is as follows:

[0089] M=l2(softmax(LeakyReLU([VAT||E]W)));

[0090] Among them, the softmax function represents the feature transformation of the input multimodal features and edge weight features, and W is the weight matrix.

[0091] Obtain the weight vector values ​​m1 and m2 that measure the importance of the multimodal feature VAT and the edge weight feature E respectively. The two weight features are normalized by L2-norm to obtain M i =[m1||m2], the matrix size of m1 is consistent with the multimodal feature matrix, and the size of m2 is consistent with the edge weight feature matrix. Therefore, the multimodal and edge weight features are adaptively fused to obtain the fused feature, denoted as Z, with a dimension of 256, and the formula is as follows:

[0092] Z=m1⊙VAT+m2⊙E;

[0093] Where ⊙ represents the Hadamard product of the matrices.

[0094] S205: Use the pre-trained graph-aware reasoning model to perform encoding and decoding operations on each fusion feature to obtain each target feature.

[0095] A graph-aware reasoning model is obtained by pre-training, such as a transformer model. After adaptively and dynamically fusing the multimodal features and edge weight features of each text block to obtain fused features, the graph-aware reasoning model is used to encode and decode each fused feature to obtain each target feature.

[0096] The above-obtained fusion feature Z is used as the input of the transformer-based graph-aware reasoning model. The transformer adopts an encoder-decoder architecture. It first uses an encoding component composed of several layers of encoding layers containing self-attention to learn and transform the input features, transform the features into an intermediate state feature, and then uses the decoding component corresponding to the encoding component to decode the intermediate state feature. After decoding by the decoding component, a mask value is obtained, and the mask value is multiplied by the original input feature to obtain the final feature. By inputting the fusion feature into the transformer model for encoding-decoding, the correlation of the document information distribution is learned. The transformer model can effectively consider the global information of the data and improve the accuracy of extracting key information from bank receipts.

[0097] S206: Using the information classification model obtained by pre-training to minimize the information loss function, classification operations are performed on each target feature to obtain the key information category corresponding to each text block.

[0098] You can set the information loss function loss_node, and the calculation formula is as follows:

[0099]

[0100] in, is the final category output of the information classification model, represents the true category, and CE(·) represents the cross entropy.

[0101] The information classification model with the information loss function minimized by pre-training is used to classify each target feature to obtain the key information category corresponding to each text block. By minimizing the information loss function, the accuracy of feature classification by the information classification model is improved.

[0102] S207: Filtering target key information categories belonging to a preset key information category group from the key information categories.

[0103] S208: Using the information classification model that minimizes the edge weight loss function to perform classification operations on the edge weight features of each target text block, and obtain classification results for each key-value pair.

[0104] When training the information classification model, you can set the edge weight loss function loss_edge, and the calculation formula is as follows:

[0105]

[0106] in, The predicted value of the final key-value pair classification result of the information classification model. It is the real key-value pair information.

[0107] The edge weight feature of each target text block is classified using the information classification model that minimizes the edge weight loss function to obtain the classification results of each key-value pair. The key-value pair classification results between the text blocks corresponding to the keyword and value can be set to 1, the self-corresponding value is -1, and the others are all 0. By minimizing the edge weight loss function, the accuracy of the information classification model in edge weight feature classification is improved.

[0108] S209: Determine whether the classification results of each key-value pair are consistent with the preset key-value pair information, if not, execute step S210, if yes, execute step S211.

[0109] After obtaining the classification results of each key-value pair, determine whether the classification results of each key-value pair are consistent with the preset key-value pair information. If not, it means that there are text blocks in the target text blocks corresponding to each target key information category that do not require key information extraction operations, and execute step S210. If so, it means that the target text blocks corresponding to each target key information category are all text blocks that require key information extraction operations, and execute step S211.

[0110] S210: Eliminate the target text blocks corresponding to the key-value pair classification results that are inconsistent with the preset key-value pair information, and perform key information extraction operations on the target text blocks corresponding to each target key information category after the text blocks are eliminated to obtain the bank receipt key information.

[0111] When it is determined that there are key-value pair classification results that are inconsistent with the corresponding preset key-value pair information in each key-value pair classification result, it means that there are text blocks in the target text blocks corresponding to each target key information category that do not need to perform key information extraction operations. The target text blocks corresponding to the key-value pair classification results that are inconsistent with the preset key-value pair information are eliminated, and the key information extraction operations are performed on the target text blocks corresponding to each target key information category after the text blocks are eliminated to obtain the bank receipt key information.

[0112] S211: Perform key information extraction operations on the target text blocks corresponding to each target key information category to obtain the bank receipt key information.

[0113] This embodiment is different from the embodiment 1 corresponding to the technical solution claimed for protection by the independent claim 1, and further adds the technical solutions claimed for protection by the dependent claims 2 to 7. Of course, according to different actual conditions and requirements, the technical solutions claimed for protection by the dependent claims can be flexibly combined without affecting the integrity of the solutions, so as to better meet the requirements of different usage scenarios. This embodiment only provides one of the solutions with the most solutions and the best effects. Due to the complexity of the situation, it is impossible to list all possible solutions one by one. Those skilled in the art should be able to realize that there can be many examples based on the basic method principles provided by this application in combination with actual conditions, and they should all be within the scope of protection of this application without sufficient creative work.

[0114] Corresponding to the above method embodiment, the present invention further provides a key information extraction device for a bank receipt. The key information extraction device for a bank receipt described below and the key information extraction method for a bank receipt described above can be referenced to each other.

[0115] See also Figure 3 , Figure 3 This is a structural block diagram of a device for extracting key information of a bank receipt in an embodiment of the present invention. The device may include:

[0116] The text block recognition module 31 is used to perform text block recognition on the received bank receipt image to obtain each text block;

[0117] A multimodal feature acquisition module 32 is used to extract the image visual features and text features of each text block, and perform a fusion operation on the image visual features and the text features to obtain a multimodal feature;

[0118] The edge weight feature calculation module 33 is used to calculate the edge weights between each text block, and calculate the edge weight feature according to each edge weight corresponding to each text block;

[0119] A fusion feature acquisition module 34 is used to adaptively and dynamically fuse the multimodal features and edge weight features of each text block to obtain a fusion feature;

[0120] The key information category obtaining module 35 is used to perform classification operations on each fusion feature using the pre-trained information classification model to obtain the key information category corresponding to each text block;

[0121] The bank receipt key information obtaining module 36 is used to filter each target key information category belonging to a preset key information category group from each key information category, and perform key information extraction operations on the target text blocks corresponding to each target key information category to obtain the bank receipt key information.

[0122] It can be seen from the above technical solution that by fusing the image visual features and text features of each text block in the bank receipt image, a multimodal feature is obtained, and the edge weight features of each text block are calculated, and the multimodal feature and the edge weight feature are adaptively and dynamically fused, so that the obtained fused feature can better represent the global information. The information classification model is pre-trained, and the fused features are classified using the information classification model, thereby realizing the extraction of key information of the bank receipt. The present invention avoids the cumbersome setting of the template, and by classifying the fused features, the accuracy of extracting key information of the bank receipt is effectively improved, and the robustness is enhanced.

[0123] In a specific implementation of the present invention, the key information category obtaining module 35 includes:

[0124] The feature encoding and decoding submodule is used to use the pre-trained graph-aware reasoning model to perform encoding and decoding operations on each fusion feature to obtain each target feature;

[0125] The feature classification submodule is used to classify each target feature using the pre-trained information classification model.

[0126] In a specific embodiment of the present invention, the device may further include:

[0127] The key-value pair classification result obtaining module is used to, after selecting each target key information category belonging to a preset key information category group from each key information category, and before performing a key information extraction operation on the text block corresponding to each target key information category, perform a classification operation on the edge weight features of each target text block using an information classification model to obtain each key-value pair classification result;

[0128] A judgment module is used to judge whether the classification result of each key-value pair is consistent with the preset key-value pair information;

[0129] The bank receipt key information obtaining module 36 is specifically a module which performs key information extraction operations on the text blocks corresponding to each target key information category when it is determined that the classification results of each key-value pair are consistent with the preset key-value pair information.

[0130] In a specific embodiment of the present invention, the key-value pair classification result obtaining module is specifically a module that uses an information classification model that minimizes an edge weight loss function to perform classification operations on the edge weight features of each target text block.

[0131] In a specific embodiment of the present invention, the key information category acquisition module 35 is specifically a module that uses the information classification model obtained by pre-training to minimize the information loss function to perform classification operations on each fusion feature.

[0132] In a specific implementation of the present invention, the text block recognition module 31 is specifically a module that uses optical character recognition technology to perform text block recognition on the received bank receipt image.

[0133] Corresponding to the above method embodiment, see Figure 4 , Figure 4 This is a schematic diagram of a key information extraction device for a bank receipt provided by the present invention, and the device may include:

[0134] A memory 332, for storing computer programs;

[0135] The processor 322 is used to implement the steps of the method for extracting key information of the bank receipt of the above method embodiment when executing the computer program.

[0136] For details, please refer to Figure 5 , Figure 5 A schematic diagram of the specific structure of a key information extraction device for a bank receipt provided in this embodiment, which may have relatively large differences due to different configurations or performances, may include a processor (central processing units, CPU) 322 (for example, one or more processors) and a memory 332, wherein the memory 332 stores one or more computer applications 342 or data 344. The memory 332 may be a temporary storage or a permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), each of which may include a series of instruction operations in the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332, and execute a series of instruction operations in the memory 332 on the key information extraction device 301 for the bank receipt.

[0137] The bank receipt key information extraction device 301 may also include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input and output interfaces 358 , and / or one or more operating systems 341 .

[0138] The steps in the method for extracting key information from a bank receipt described above can be implemented by the structure of a device for extracting key information from a bank receipt.

[0139] Corresponding to the above method embodiment, the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps can be implemented:

[0140] The received bank receipt image is subjected to text block recognition to obtain each text block; the image visual features and text features of each text block are extracted, and the image visual features and text features are fused to obtain multimodal features; the edge weights between each text block are calculated respectively, and the edge weight features are calculated according to the edge weights corresponding to each text block; the multimodal features and edge weight features of each text block are adaptively and dynamically fused to obtain fused features; the pre-trained information classification model is used to classify each fused feature to obtain the key information category corresponding to each text block; from each key information category, each target key information category belonging to the preset key information category group is selected, and the key information extraction operation is performed on the target text block corresponding to each target key information category to obtain the key information of the bank receipt.

[0141] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0142] For an introduction to the computer-readable storage medium provided by the present invention, please refer to the above method embodiment, and the present invention will not be elaborated here.

[0143] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the devices, equipment and computer-readable storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0144] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the technical solution and core ideas of the present invention. It should be pointed out that for ordinary technicians in this technical field, the present invention can also be improved and modified without departing from the principles of the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A method for extracting key information from a bank receipt, characterized in that: include: Perform text block recognition on the received bank receipt image to obtain each text block; Extracting image visual features and text features of each text block, and fusing the image visual features and the text features to obtain multimodal features; Calculating edge weights between each pair of the text blocks respectively, and calculating edge weight features according to the edge weights corresponding to each text block; Adaptively and dynamically fuse the multimodal features and edge weight features of each text block to obtain fused features; Using the pre-trained information classification model to perform classification operations on each of the fused features, to obtain key information categories corresponding to each of the text blocks; Target key information categories belonging to a preset key information category group are screened from the key information categories, and key information extraction operations are performed on target text blocks corresponding to the target key information categories to obtain bank receipt key information.

2. The method for extracting key information from a bank receipt according to claim 1, characterized in that: The pre-trained information classification model is used to classify each of the fusion features, including: Using the pre-trained graph-aware reasoning model to perform encoding and decoding operations on each of the fused features to obtain each target feature; The pre-trained information classification model is used to perform classification operations on each of the target features.

3. The method for extracting key information of a bank receipt according to claim 1 or 2, characterized in that: After selecting target key information categories belonging to a preset key information category group from the key information categories, and before performing key information extraction operations on text blocks corresponding to the target key information categories, the method further includes: Using the information classification model to perform classification operations on the edge weight features of each target text block to obtain classification results of each key-value pair; Determine whether the classification results of each of the key-value pairs are consistent with the preset key-value pair information; If so, the step of performing key information extraction operations on the text blocks corresponding to each target key information category is executed.

4. The method for extracting key information from a bank receipt according to claim 3, characterized in that: Using the information classification model to classify the edge weight features of each target text block includes: The edge weight features of each target text block are classified using an information classification model that minimizes the edge weight loss function.

5. The method for extracting key information from a bank receipt according to claim 1, characterized in that: The pre-trained information classification model is used to classify each of the fusion features, including: The information classification model obtained by pre-training and minimizing the information loss function is used to perform classification operations on each of the fused features.

6. The method for extracting key information from a bank receipt according to claim 1, characterized in that: Perform text block recognition on the received bank receipt image, including: Optical character recognition technology is used to identify text blocks in the received bank receipt images.

7. A device for extracting key information from a bank receipt, characterized in that: include: The text block recognition module is used to perform text block recognition on the received bank receipt image to obtain each text block; A multimodal feature acquisition module, used for extracting image visual features and text features of each text block, and fusing the image visual features and the text features to obtain multimodal features; An edge weight feature calculation module, used to calculate the edge weights between each of the text blocks, and calculate the edge weight features according to the edge weights corresponding to each text block; The fusion feature acquisition module is used to adaptively and dynamically fuse the multimodal features and edge weight features of each text block to obtain fusion features; A key information category acquisition module is used to classify each of the fusion features using a pre-trained information classification model to obtain the key information category corresponding to each of the text blocks; The bank receipt key information acquisition module is used to filter each target key information category belonging to a preset key information category group from each of the key information categories, perform key information extraction operations on the target text blocks corresponding to each of the target key information categories, and obtain the bank receipt key information.

8. The key information extraction device of the bank receipt according to claim 7 is characterized in that: The key information category acquisition module includes: A feature encoding and decoding submodule is used to use the pre-trained graph perception reasoning model to perform encoding and decoding operations on each of the fusion features to obtain each target feature; The feature classification submodule is used to perform classification operations on each of the target features using the pre-trained information classification model.

9. A device for extracting key information from a bank receipt, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the key information extraction method of the bank receipt as claimed in any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for extracting key information of a bank receipt as claimed in any one of claims 1 to 6.

Images