Automatic bill information processing method based on Transform architecture

Abstract

The invention discloses an automatic bill information processing method based on a Transform architecture, and the method comprises the following steps: inputting a bill image, and carrying out the segmentation preprocessing of the bill image, and forming a plurality of image blocks; mapping each image block to a smaller vector space, and then forming a vector sequence of each image block in combination with position information; a Transform encoder performs two-stage feature extraction on the vector sequence of each image block; then the output vector passes through an additional linear layer; and finally, completing classification and extraction of bill character information through a softmax function, and selecting a character category with the highest confidence coefficient as prediction according to probability distribution of output vectors of the bill character information. A recursive prediction method is adopted for the sequence characters until terminators appear, and finally complete bill character content is generated. The method has high adaptability to bill format diversity, can significantly improve processing efficiency and real-time performance, and has strong model generalization ability.

Description

Technical Field

[0001] The present invention relates to the field of automated bill information processing, and in particular to an automated bill information processing method based on a Transformer architecture. Background Art

[0002] In the field of automated bill information processing, although a variety of technologies such as optical character recognition (OCR) have been proposed and applied in large quantities, existing solutions still have some significant pain points. The diversity of bill formats, the unevenness of image quality, the low processing efficiency, the lack of real-time performance and the limited accuracy have always been the bottleneck of the industry's development.

[0003] First, the diversity of bill formats makes it difficult to generalize the automated system. Since bills in different industries and regions have significant differences in format and layout, a general automated processing system often requires expensive and time-consuming customized development.

[0004] Secondly, the inhomogeneity of image quality, such as blur, tilt, uneven lighting or noise, seriously affects the accuracy of text recognition. Although existing technologies can provide acceptable recognition rates in some cases, the accuracy often drops significantly when faced with complex layouts or handwritten texts.

[0005] In addition, the efficiency of the existing processing flow cannot be ignored. Many systems need to go through complex image pre-processing, feature extraction, text recognition and post-processing when processing bills. These steps not only increase processing time, but also increase the possibility of errors.

[0006] Lack of real-time performance is another key issue, especially in industries such as financial services, healthcare, and retail, which have strict requirements on the speed of bill processing. However, existing systems often cannot achieve fast processing without compromising accuracy. Finally, the issue of accuracy remains a core challenge in bill information processing. Although existing technologies can provide acceptable recognition rates in some cases, performance often drops significantly when faced with complex layouts or handwritten text.

[0007] The patent document with the authorization announcement number CN115019327B discloses a fragmented bill recognition method and system based on fragmented bill segmentation and Transformer network. It constructs a fragmented bill recognition model with Transformer network, simplifies the complexity of the network structure, and utilizes a general pre-training model, which greatly reduces the pressure of manual data annotation and improves the training efficiency of the model. In addition, in the loss function used in training the fragmented bill recognition model, the weight calculation of the vocabulary in the constructed fragmented bill vocabulary is added, so that the model training focuses on the vocabulary with high word frequency in the fragmented bill content recognition scenario, further improving the model's ability to recognize the content of fragmented bills. However, this solution still has the following problems:

[0008] 1. Insufficient coverage of specific domain terms: Although the solution uses the Jieba word segmentation tool to build a fragmented bill vocabulary, this method may have insufficient coverage when processing bills containing a large number of professional domain terms or emerging words. This indicates that the model may not be able to fully capture the special words in the bill, thus affecting the accuracy and completeness of recognition.

[0009] 2. Limited real-time processing capabilities: In the process of building a fragmented bill dictionary, all words and placeholders need to be <s>、 <pad> 、< / pad> < / s> and <unk>As the key, the index is used as the value and written in JSON file format. This step involves data processing and file writing operations, which consumes significant computing resources and time, thus affecting the real-time processing capability of the entire system. This means that in scenarios with high efficiency requirements, the current processing flow may not be able to meet the needs of real-time response.

[0010] 3. The generalization ability of the model needs to be improved: The current model is mainly trained for specific types of bills, which limits its generalization ability when processing diversified bills. In actual applications, the format and content of bills may vary greatly, and the model needs to be able to adapt to these changes to ensure high recognition rates in various situations. Summary of the invention

[0011] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and provide an automated bill information processing method based on the Transformer architecture.

[0012] The purpose of the present invention is achieved through the following technical solutions:

[0013] The automated bill information processing method based on the Transformer architecture includes the following steps:

[0014] S1, input the bill image and perform block preprocessing to form multiple image blocks;

[0015] S2, map each image block to a smaller vector space, and then combine the position information to form a vector sequence for each image block;

[0016] S3, Transformer encoder performs two-stage feature extraction on the vector sequence of each image block: In the first stage, the multi-head self-attention mechanism is used to capture the global context information between the image block sequences; In the second stage, the output of the multi-head attention is nonlinearly mapped and feature transformed through two fully connected layers in the feedforward neural network to further enhance the feature expression capability;

[0017] The output vector of the Transformer encoder then passes through an additional linear layer, which maps the high-dimensional feature vector to an output dimension relevant to the task;

[0018] Finally, the classification and extraction of bill text information is completed through the softmax function;

[0019] S4. According to the probability distribution of the output vector of the softmax function, the text category with the highest confidence is selected as the prediction; a recursive prediction method is used for the sequence text until a terminator appears, and finally the complete text content of the bill is generated.

[0020] In step S1, the bill image is represented as a two-dimensional matrix , , is the height of the bill image, is the width of the bill image, is the real number set of the bill image, indicating that each pixel value in the image is a real number; Indicates that the input bill image is a two-dimensional real number matrix with height multiplied by width;

[0021] Image of the receipt Perform block preprocessing and divide it into non-overlapping blocks of fixed size , is the side length of each block; is the total number of blocks; is the number of channels of the input image.

[0022] In step S2, each of the image blocks is linearly projected into an embedding vector. The embedding vector corresponding to the image patch The calculation is as follows:

[0023] ;

[0024] is the projection matrix, which is used to map the image block to a smaller vector space; For the The pixel value of the image block is flattened into a one-dimensional vector; is a bias term used to adjust the results;

[0025] Record the position information of each image block for subsequent processing; The position information of the image blocks in the image is , No. The vector sequence of image blocks combined with position information The calculation method is as follows:

[0026] .

[0027] In step S3, the Transformer encoder performs nonlinear mapping and feature conversion on the output of the multi-head attention through two fully connected layers in the feedforward neural network, which is completed by the following formula:

[0028] ;

[0029] is the weight matrix of the first fully connected layer in the feedforward neural network; is the bias term of the first fully connected layer in the feedforward neural network; is the ReLu activation function, All negative values ​​in the result are set to 0, and positive values ​​remain unchanged; is the weight matrix of the second fully connected layer in the feedforward neural network; is the bias term of the second fully connected layer in the feedforward neural network; is the output of the feedforward neural network.

[0030] In step S3, the linear layer in the Transformer encoder is used to convert the encoder output into the final bill text information, and classified by the softmax function:

[0031] ;

[0032] is the weight matrix; is the bias term; is the input vector; is the output vector, which contains the probability distribution of text information after being processed by the softmax function.

[0033] In step S4, the probability value of each element in the output vector of the softmax function represents the confidence of the corresponding text category, so the Transformer encoder will select the category corresponding to the maximum probability value as the prediction result of the current text information based on the probability distribution; if the output is a sequence of text, each character category is recursively predicted step by step until the terminator is predicted; finally, all predicted character categories are mapped to actual text information to generate complete bill text content.

[0034] At the same time, the present invention provides:

[0035] A server comprises a processor and a memory, wherein the memory stores at least one program, and the program is loaded and executed by the processor to implement the above-mentioned automated bill information processing method based on the Transformer architecture.

[0036] A computer-readable storage medium stores at least one program, which is loaded and executed by a processor to implement the above-mentioned automated bill information processing method based on the Transformer architecture.

[0037] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0038] 1. The present invention has high adaptability to the diversity of bill formats.

[0039] The present invention uses a multi-head self-attention mechanism based on the Transformer architecture to globally model the contextual relationship between image blocks in the bill and adapt to the diverse bill layouts. Compared with the traditional OCR method that relies on templated bill formats, the present invention can effectively adapt to bills of different formats, languages ​​and regions, reducing the customized development costs caused by layout differences.

[0040] 2. The present invention can significantly improve processing efficiency and real-time performance.

[0041] The present invention reduces the complex image pre-processing and post-processing steps by optimizing the process, and at the same time, with the help of the parallel processing capability of the Transformer architecture, it achieves efficient data flow and calculation when processing bills. Compared with the traditional OCR method that requires multi-stage processing, the present invention can complete the extraction of bill information in a shorter time, especially in high-concurrency scenarios, showing excellent real-time performance, meeting the strict requirements of financial, medical and retail industries for bill processing speed.

[0042] 3. The present invention has strong model generalization ability.

[0043] The present invention optimizes the pre-trained model of Transformer so that the system can quickly adapt to different types of bill formats and contents. Compared with the models in the prior art that are highly dependent on specific bill types, the model of the present invention can handle more diverse bill contents and scenarios, reducing the need for model retraining due to scenario switching, thereby improving the generalization performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flowchart of the automated bill information processing method based on the Transformer architecture. DETAILED DESCRIPTION

[0045] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0046] like Figure 1 , the automated bill information processing method based on the Transformer architecture includes the following steps:

[0047] 1. Image input and preprocessing

[0048] The input bill image is represented as a two-dimensional matrix , , is the height of the bill image, is the width of the bill image, is the real number set of the bill image, indicating that each pixel value in the image is a real number; Indicates that the input bill image is a A two-dimensional real matrix of height by width.

[0049] Image of the receipt Perform block preprocessing and divide it into non-overlapping blocks of fixed size , is the side length of each block; is the total number of blocks; is the number of channels of the input image.

[0050] 2. Feature extraction and encoding

[0051] Each of the image blocks is linearly projected into an embedding vector, The embedding vector corresponding to the image patch The calculation is as follows:

[0052] ;

[0053] is the projection matrix, which is used to map the image block to a smaller vector space; For the The pixel value of the image block is flattened into a one-dimensional vector; is a bias term used to adjust the results;

[0054] Record the position information of each image block for subsequent processing; The position information of the image blocks in the image is , No. The vector sequence of image blocks combined with position information The calculation method is as follows:

[0055] .

[0056] 3. Transformer Encoder

[0057] The Transformer encoder performs two-stage feature extraction on the vector sequence of each image block: In the first stage, the multi-head self-attention mechanism is used to capture the global context information between the image block sequences. The multi-head self-attention mechanism captures the dependencies between different positions through parallel calculations of multiple attention heads, thereby enhancing the understanding of the global context; In the second stage, the output of the multi-head attention is nonlinearly mapped and feature transformed through two fully connected layers in the feedforward neural network to further enhance the feature expression capability;

[0058] Among them, in the second stage, the Transformer encoder performs nonlinear mapping and feature conversion on the output of multi-head attention through two fully connected layers in the feedforward neural network, which is completed by the following formula:

[0059] ;

[0060] is the weight matrix of the first fully connected layer in the feedforward neural network; is the bias term of the first fully connected layer in the feedforward neural network; is the ReLu activation function, All negative values ​​in the result are set to 0, and positive values ​​remain unchanged; is the weight matrix of the second fully connected layer in the feedforward neural network; is the bias term of the second fully connected layer in the feedforward neural network; is the output of the feedforward neural network.

[0061] The output vector of the Transformer encoder then passes through an additional linear layer, which maps the high-dimensional feature vector to an output dimension relevant to the task;

[0062] Finally, the classification and extraction of bill text information is completed through the softmax function;

[0063] Transformer encoder structure and function: Transformer encoder is the core component of this method. Its main function is to encode the input vector sequence through a multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention mechanism is used to capture the global dependencies between the parts of the input vector sequence, while the feedforward neural network (FFN) further performs nonlinear transformation and feature extraction on the features processed by the attention mechanism.

[0064] Relationship between feedforward neural network and linear layer: Each layer of Transformer encoder contains a multi-head self-attention mechanism and a feedforward neural network module. The feedforward neural network is a module composed of two layers of fully connected linear layers and nonlinear activation functions. The first linear layer maps the input features to a higher dimensional space, and the second linear layer remaps the high-dimensional features back to the original dimension, and combines residual connections to improve the stability of the network.

[0065] The specific role of the linear layer: After the final output of the Transformer encoder, an additional linear layer is added to convert the high-dimensional feature vector into a fixed dimension of the target space (such as the number of categories or output a specific character probability distribution), and normalize it through the softmax function to complete the classification task. This additional linear layer is not the inherent feedforward neural network inside the Transformer encoder, but an output layer specially designed for the task target in the automated bill processing method.

[0066] 4. Output and decoding

[0067] Use the linear layer in Transformer to convert the output of the encoder into the final bill text information and classify it through the softmax function:

[0068] ;

[0069] is the weight matrix; is the bias term; is the input vector; is the output vector, which contains the probability distribution of text information after being processed by the softmax function.

[0070] Since the probability value of each element in the output vector y represents the confidence of the corresponding text category, the Transformer encoder will select the category corresponding to the maximum probability value as the prediction result of the current text information based on the probability distribution; if the output is a sequence of text, it will recursively predict each character category step by step until the terminator is predicted. Finally, all predicted character categories are mapped to actual text information to generate the complete text content of the bill.

[0071] At the same time, the present invention provides:

[0072] A server comprises a processor and a memory, wherein the memory stores at least one program, and the program is loaded and executed by the processor to implement the above-mentioned automated bill information processing method based on the Transformer architecture.

[0073] A computer-readable storage medium stores at least one program, which is loaded and executed by a processor to implement the above-mentioned automated bill information processing method based on the Transformer architecture.

[0074] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.< / unk>

Claims

1. The automated bill information processing method based on the Transformer architecture is characterized by: The following steps are involved: S1, input the bill image and perform block preprocessing to form multiple image blocks; S2, map each image block to a smaller vector space, and then combine the position information to form a vector sequence for each image block; S3, Transformer encoder performs two-stage feature extraction on the vector sequence of each image block: In the first stage, the multi-head self-attention mechanism is used to capture the global context information between the image block sequences; In the second stage, the output of the multi-head attention is nonlinearly mapped and feature transformed through two fully connected layers in the feedforward neural network to further enhance the feature expression capability; The output vector of the Transformer encoder then passes through an additional linear layer, which maps the high-dimensional feature vector to an output dimension relevant to the task; Finally, the classification and extraction of bill text information is completed through the softmax function; S4, select the text category with the highest confidence as the prediction according to the probability distribution of the output vector of the softmax function; A recursive prediction method is used for the sequence text until a terminator appears, and finally the complete bill text content is generated.

2. The automated bill information processing method based on the Transformer architecture according to claim 1 is characterized in that: In step S1, the bill image is represented as a two-dimensional matrix , , is the height of the bill image, is the width of the bill image, is the real number set of the bill image, indicating that each pixel value in the image is a real number; Indicates that the input bill image is a two-dimensional real number matrix with height multiplied by width; Image of the receipt Perform block preprocessing and divide it into non-overlapping blocks of fixed size , is the side length of each block; is the total number of blocks; is the number of channels of the input image.

3. The automated bill information processing method based on the Transformer architecture according to claim 1 is characterized in that: In step S2, each of the image blocks is linearly projected into an embedding vector. The embedding vector corresponding to the image patch The calculation is as follows: ； is the projection matrix, which is used to map the image block to a smaller vector space; For the The pixel value of the image block is flattened into a one-dimensional vector; is a bias term used to adjust the results; Record the position information of each image block for subsequent processing; The position information of the image blocks in the image is , No. The vector sequence of image blocks combined with position information The calculation method is as follows: 。 4. The automated bill information processing method based on the Transformer architecture according to claim 1 is characterized in that: In step S3, the Transformer encoder performs nonlinear mapping and feature conversion on the output of the multi-head attention through two fully connected layers in the feedforward neural network, which is completed by the following formula: ； is the weight matrix of the first fully connected layer in the feedforward neural network; is the bias term of the first fully connected layer in the feedforward neural network; is the ReLu activation function, All negative values ​​in the result are set to 0, and positive values ​​remain unchanged; is the weight matrix of the second fully connected layer in the feedforward neural network; is the bias term of the second fully connected layer in the feedforward neural network; is the output of the feedforward neural network.

5. The automated bill information processing method based on the Transformer architecture according to claim 1 is characterized in that: In step S3, the linear layer in the Transformer encoder is used to convert the encoder output into the final bill text information, and classified by the softmax function: ； is the weight matrix; is the bias term; is the input vector; is the output vector, which contains the probability distribution of text information after being processed by the softmax function.

6. The automated bill information processing method based on the Transformer architecture according to claim 1 is characterized in that: In step S4, the probability value of each element in the output vector of the softmax function represents the confidence of the corresponding text category, so the Transformer encoder will select the category corresponding to the maximum probability value as the prediction result of the current text information based on the probability distribution; if the output is a sequence of text, each character category is recursively predicted step by step until the terminator is predicted; Finally, all predicted character categories are mapped to actual text information to generate the complete bill text content.

7. A server, comprising a processor and a memory, characterized in that: At least one program is stored in the memory, and the program is loaded and executed by the processor to implement the automated bill information processing method based on the Transformer architecture as described in any one of claims 1 to 6.

8. A computer-readable storage medium, wherein at least one program is stored in the storage medium, characterized in that: The program is loaded and executed by a processor to implement the automated bill information processing method based on the Transformer architecture as described in any one of claims 1 to 6.

Images