Invoice text recognition method and system based on multi-scale residual and attention mechanism

Abstract

This invention discloses an invoice text recognition method and system based on multi-scale residuals and attention mechanisms, comprising the following steps: preprocessing the invoice image to segment it into individual character images; constructing a multi-scale residual CNN network, inputting the individual character images into the network to obtain the representation vectors of the individual character images; constructing a BiGRU network based on an attention mechanism to obtain the text recognition result for each sample; training the text recognition network composed of the multi-scale residual CNN network and the BiGRU network based on an attention mechanism, finding the optimal network parameters to obtain the optimal text recognition network, and using the optimal text recognition network to recognize text information from the invoice image; the advantages of this invention are: it can extract more comprehensive features, solve the gradient vanishing problem, and has high text recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of computer imaging, and more specifically to a method and system for recognizing invoice text based on multi-scale residuals and attention mechanisms. Background Technology

[0002] In recent years, with the rapid development of invoice systems, value-added tax (VAT) invoices, serving as accounting and tax payment certificates in commercial activities, have been widely used across various sectors. Currently, many institutions rely primarily on finance department personnel for data processing, statistics, entry, and archiving of expense reports. These processes waste significant human, material, and time resources and are prone to data entry errors. Optical character recognition (OCR) is a key technology for solving these problems. It can quickly and accurately identify textual information in invoices, effectively saving human and material costs, improving invoice entry efficiency, and reducing human error. It plays a crucial role in the identification and entry of invoices.

[0003] Currently, text recognition methods can be broadly categorized into traditional methods and deep learning methods. Traditional methods primarily include template matching algorithms, anchor point localization, and geometric feature extraction. These methods can identify information from scanned VAT images. However, due to variations in brightness, sharpness, blurriness, and noise levels in captured images, traditional text recognition methods suffer from relatively poor accuracy, limiting their practical application. Deep learning methods, with their more complex network structures, have been successfully applied to complex tasks such as computer vision, natural language processing, and speech recognition. Compared to traditional methods, deep learning-based text recognition methods offer better stability and generalization capabilities, are less affected by image quality, and can accurately identify text information from complex invoice images. Therefore, deep learning-based text recognition has become a leading research trend, but it still has the following shortcomings:

[0004] (1) Existing deep learning methods use single-scale convolutional kernels to extract features from invoice image information. For example, a scene text recognition method based on multi-scale features disclosed in Chinese Patent Publication No. CN111967471A is difficult to extract more comprehensive feature representations from invoice images. In addition, when the network is too deep, gradient vanishing is likely to occur during the network training process.

[0005] (2) Existing text recognition methods only consider the recognition result of a single character, without considering the relationship between characters before and after, and ignore the contextual semantic information in the invoice text, resulting in low accuracy of text recognition. Summary of the Invention

[0006] The technical problem to be solved by this invention is that existing invoice text recognition methods are difficult to extract more comprehensive features, and are prone to gradient vanishing and low text recognition accuracy.

[0007] This invention solves the above-mentioned technical problems through the following technical means: an invoice text recognition method based on multi-scale residuals and attention mechanisms, comprising the following steps:

[0008] Step 1: Preprocess the invoice image and segment it into individual character images;

[0009] Step 2: Construct a multi-scale residual CNN network. Input a single character image into the network, and use convolutional layers with different kernel sizes to perform convolution operations on the image data respectively. Then perform max pooling operation. The pooling result is concatenated and fused with the input vector to obtain the representation vector of a single character image.

[0010] Step 3: Construct a BiGRU network based on the attention mechanism. Input the representation vector of a single character image in each sample into the network to obtain the contextual semantic information in the sample and input it into the text recognition model to obtain the text recognition result for each sample.

[0011] Step 4: Train the text recognition network composed of the multi-scale residual CNN network and the attention-based BiGRU network, find the optimal network parameters, obtain the optimal text recognition network, and use the optimal text recognition network to recognize text information from the invoice image.

[0012] Beneficial effects: This invention uses convolutional kernels of different sizes to perform convolution operations on image data to obtain more comprehensive feature representations. It introduces a residual network structure to alleviate the gradient vanishing problem and proposes a BiGRU network based on an attention mechanism to obtain the semantic relationship of the context. It considers the contextual semantic information between individual characters and uses an attention mechanism to distinguish the relative importance of different characters, effectively improving the accuracy of text recognition and realizing text recognition of invoice images.

[0013] Further, step 1 includes:

[0014] Step 1.1: Collect the original image data of the invoice using an image acquisition device;

[0015] Step 1.2: Perform image correction, grayscale conversion, and threshold segmentation denoising on the original image data;

[0016] Step 1.3: Segment the processed original image data into individual character images, label each individual character image to obtain the true label corresponding to each character image. The processed data is represented as D = {(x1,y1),(x2,y2),...,(x...}i ,y i ),...,(x n ,y n )}, where x i ={x i1 ,x i2 ,...,x ij ,...,x im Let} be the set of m single-character images in the i-th sample, and x ij For the j-th single character image in the i-th sample, y i ={y i1 ,y i2 ,...,y ij ,...,y im Let} be the set of labels in the i-th sample, and y ij For x ij The actual label data, 1≤i≤n, 1≤j≤m.

[0017] Furthermore, step 2 includes:

[0018] Step 2.1: Construct a multi-scale residual CNN network. This network includes multiple parallel convolutional layers with different kernel sizes, and pooling layers connected to each convolutional layer. Input the single character image data from each sample into the multi-scale residual CNN network, and perform convolution operations on the image data using convolutional layers with different kernel sizes to obtain the output vector f. l , l represents the label of the convolution kernel;

[0019] Step 2.2, the output vector f after convolution l The input pooling layer performs max pooling, and then the output vector after pooling is padded and concatenated with the input vector of the convolutional layer to obtain the representation vector of a single character image.

[0020] Step 2.3: Repeat steps 2.1 and 2.2 until the individual character image data for each sample has been calculated, thus obtaining the feature representation of the individual character image in each sample as {(f1,y1),(f2,y2),...,(f i ,y i ),...,(f n ,y n )},f i ={f i1 ,f i2 ,...,f ij ,...,f im}, f ij Let be the representation vector of the j-th single character image in the i-th sample.

[0021] Furthermore, in step 2.1, convolutional layers with different kernel sizes are used to perform convolution operations on the image data. The calculation method for the convolution operation is as shown in formula (1):

[0022]

[0023] Among them, f l w is the output vector after convolution. l and b l These are the learnable parameters of the convolutional layer. This represents the convolution operation. is the ReLU activation function, and l represents the label of the convolution kernel.

[0024] Furthermore, step 3 includes:

[0025] Step 3.1: Initialize time step q = 1. When q = 1, initialize the hidden vectors of the BiGRU network. vector As the query vector; when q > 1, the hidden state vector of the BiGRU network at time step q-1 is used. As a query vector;

[0026] Step 3.2: Using the representation vector of a single character image in each sample as the key and value, the BiGRU network performs weighted fusion of the representation vectors of the single character images using an attention mechanism;

[0027] Step 3.3: After weighted fusion via the attention mechanism, calculate the hidden state vector at the q-th time step in the BiGRU network.

[0028] Step 3.4: Construct a text recognition model based on a fully connected layer network. As input, predict the recognition result of the q-th single character image in the i-th input sample;

[0029] Step 3.5: Execute q = q + 1. When 1 ≤ q ≤ m, repeat steps 3.1-3.4 to obtain the recognition results of m individual character images of the i-th input sample. Otherwise, proceed to step 4.

[0030] Furthermore, the calculation method for the attention-based weighted fusion of the representation vectors of a single character image by the BiGRU network in step 3.2 is as follows:

[0031]

[0032]

[0033]

[0034] Where V, W, and b are learnable parameters in the attention mechanism, and e i =[e i1 ,e i2 ,...,e ij ,...,e im ] is {f i1 ,f i2 ,...,f ij ,...,f im The weight vector of}, after normalization, yields α. ij , representing f in the i-th input sample ij Weights, exp(·) is an exponential function with the natural constant e as its base.

[0035] Furthermore, step 3.3 includes:

[0036] Will The output of the previous time step is concatenated with the output of the previous time step and then input into the BiGRU network. After calculation, the hidden state vector of the q-th time step in the BiGRU network can be obtained. The calculation method is as follows:

[0037]

[0038] in, and These are the reset gate and update gate of the BiGRU at the q-th time step in the i-th sample, respectively. Let W be the candidate hidden vector of BiGRU at time step q in the i-th sample, σ(·) be the sigmoid activation function, tanh(·) be the hyperbolic tangent activation function, and W be the hidden vector of BiGRU at time step q in the i-th sample. r1 W r2 W z1 W z2 W s1 and W s2 This is the weight matrix in BiGRU, b r b z and b s This is the corresponding bias.

[0039] Furthermore, the text recognition model based on a fully connected layer network constructed in step 3.4 is shown in formula (6):

[0040]

[0041] Among them, W predict and b predict These are the learnable parameters in the text recognition model. φ is the output of the q-th single character image in the i-th input sample, and φ is the softmax function.

[0042] Furthermore, step 4 includes:

[0043] Step 4.1: Using cross-entropy as the loss function, update the learnable parameters in the text recognition network based on multi-scale residual CNN network and attention mechanism BiGRU using the stochastic gradient descent method to obtain the network parameters when the loss function reaches its minimum value. Substitute these network parameters into the text recognition network to obtain the optimized text recognition network. The loss function is shown in Equation (7):

[0044]

[0045] Where Loss is the loss function, W represents the learnable parameters in the text recognition network based on multi-scale residual CNN and attention mechanism BiGRU, and λ is the penalty factor.

[0046] Step 4.2: Use the text recognition network trained in Step 4.1, based on a multi-scale residual CNN network and an attention mechanism BiGRU, to recognize text information from the invoice image.

[0047] This invention also provides an invoice text recognition system based on multi-scale residuals and attention mechanisms, comprising:

[0048] The preprocessing section is used to preprocess the invoice image and segment it into individual character images;

[0049] The convolutional pooling part is used to construct a multi-scale residual CNN network. A single character image is input into the network, and convolutional layers with different kernel sizes are used to perform convolution operations on the image data respectively. Then, max pooling is performed, and the pooling result is concatenated and fused with the input vector to obtain the representation vector of a single character image.

[0050] The text recognition part is used to construct a BiGRU network based on the attention mechanism. The representation vector of a single character image in each sample is input into the network to obtain the contextual semantic information in the sample and input it into the text recognition model to obtain the text recognition result for each sample.

[0051] The training part is used to train the text recognition network composed of the multi-scale residual CNN network and the attention-based BiGRU network, to find the optimal network parameters, obtain the optimal text recognition network, and use the optimal text recognition network to recognize text information from the invoice image.

[0052] Furthermore, the preprocessing section is also used for:

[0053] Step 1.1: Collect the original image data of the invoice using an image acquisition device;

[0054] Step 1.2: Perform image correction, grayscale conversion, and threshold segmentation denoising on the original image data;

[0055] Step 1.3: Segment the processed original image data into individual character images, label each individual character image to obtain the true label corresponding to each character image. The processed data is represented as D = {(x1,y1),(x2,y2),...,(x...} i ,y i ),...,(x n ,y n )}, where x i ={x i1 ,x i2 ,...,x ij ,...,x im Let} be the set of m single-character images in the i-th sample, and x ij For the j-th single character image in the i-th sample, y i ={y i1 ,y i2 ,...,y ij ,...,y im Let} be the set of labels in the i-th sample, and y ij For x ij The actual label data, 1≤i≤n, 1≤j≤m.

[0056] Furthermore, the convolutional pooling portion is also used for:

[0057] Step 2.1: Construct a multi-scale residual CNN network. This network includes multiple parallel convolutional layers with different kernel sizes, and pooling layers connected to each convolutional layer. Input the single character image data from each sample into the multi-scale residual CNN network, and perform convolution operations on the image data using convolutional layers with different kernel sizes to obtain the output vector f. l , l represents the label of the convolution kernel;

[0058] Step 2.2, the output vector f after convolution l The input pooling layer performs max pooling, and then the output vector after pooling is padded and concatenated with the input vector of the convolutional layer to obtain the representation vector of a single character image.

[0059] Step 2.3: Repeat steps 2.1 and 2.2 until the individual character image data for each sample has been calculated, thus obtaining the feature representation of the individual character image in each sample as {(f1,y1),(f2,y2),...,(f i,y i ),...,(f n ,y n )},f i ={f i1 ,f i2 ,...,f ij ,...,f im}, f ij Let be the representation vector of the j-th single character image in the i-th sample.

[0060] Furthermore, in step 2.1, convolutional layers with different kernel sizes are used to perform convolution operations on the image data. The calculation method for the convolution operation is as shown in formula (1):

[0061]

[0062] Among them, f l w is the output vector after convolution. l and b l These are the learnable parameters of the convolutional layer. This represents the convolution operation. is the ReLU activation function, and l represents the label of the convolution kernel.

[0063] Furthermore, the text recognition section is also used for:

[0064] Step 3.1: Initialize time step q = 1. When q = 1, initialize the hidden vectors of the BiGRU network. vector As the query vector; when q > 1, the hidden state vector of the BiGRU network at time step q-1 is used. As a query vector;

[0065] Step 3.2: Using the representation vector of a single character image in each sample as the key and value, the BiGRU network performs weighted fusion of the representation vectors of the single character images using an attention mechanism;

[0066] Step 3.3: After weighted fusion via the attention mechanism, calculate the hidden state vector at the q-th time step in the BiGRU network.

[0067] Step 3.4: Construct a text recognition model based on a fully connected layer network, and then... i q As input, predict the recognition result of the q-th single character image in the i-th input sample;

[0068] Step 3.5: Execute q = q + 1. When 1 ≤ q ≤ m, repeat steps 3.1-3.4 to obtain the recognition results of m individual character images of the i-th input sample. Otherwise, proceed with the training section.

[0069] Furthermore, the calculation method for the attention-based weighted fusion of the representation vectors of a single character image by the BiGRU network in step 3.2 is as follows:

[0070]

[0071]

[0072]

[0073] Where V, W, and b are learnable parameters in the attention mechanism, and e i =[e i1 ,e i2 ,...,e ij ,...,e im ] is {f i1 ,f i2 ,...,f ij ,...,f im The weight vector of}, after normalization, yields α. ij , representing f in the i-th input sample ij Weights, exp(·) is an exponential function with the natural constant e as its base.

[0074] Furthermore, step 3.3 includes:

[0075] Will The output of the previous time step is concatenated with the output of the previous time step and then input into the BiGRU network. After calculation, the hidden state vector of the q-th time step in the BiGRU network can be obtained. The calculation method is as follows:

[0076]

[0077] in, and These are the reset gate and update gate of the BiGRU at the q-th time step in the i-th sample, respectively. Let W be the candidate hidden vector of BiGRU at time step q in the i-th sample, σ(·) be the sigmoid activation function, tanh(·) be the hyperbolic tangent activation function, and W be the hidden vector of BiGRU at time step q in the i-th sample. r1 W r2 W z1 W z2 W s1 and Ws2 This is the weight matrix in BiGRU, b r b z and b s This is the corresponding bias.

[0078] Furthermore, the text recognition model based on a fully connected layer network constructed in step 3.4 is shown in formula (6):

[0079]

[0080] Among them, W predict and b predict These are the learnable parameters in the text recognition model. φ is the output of the q-th single character image in the i-th input sample, and φ is the softmax function.

[0081] Furthermore, the training portion is also used for:

[0082] Step 4.1: Using cross-entropy as the loss function, update the learnable parameters in the text recognition network based on multi-scale residual CNN network and attention mechanism BiGRU using the stochastic gradient descent method to obtain the network parameters when the loss function reaches its minimum value. Substitute these network parameters into the text recognition network to obtain the optimized text recognition network. The loss function is shown in Equation (7):

[0083]

[0084] Where Loss is the loss function, W represents the learnable parameters in the text recognition network based on multi-scale residual CNN and attention mechanism BiGRU, and λ is the penalty factor.

[0085] Step 4.2: Use the text recognition network trained in Step 4.1, based on a multi-scale residual CNN network and an attention mechanism BiGRU, to recognize text information from the invoice image.

[0086] The advantages of this invention are:

[0087] (1) This invention uses convolution kernels of different sizes to perform convolution operations on image data to obtain more comprehensive feature representations. It introduces a residual network structure to alleviate the gradient vanishing problem and proposes a BiGRU network based on the attention mechanism to obtain the semantic relationship of the context. It considers the contextual semantic information between individual characters and uses the attention mechanism to distinguish the relative importance of different characters, effectively improving the accuracy of text recognition and realizing text recognition of invoice images.

[0088] (2) This invention is applicable to invoice text recognition in various scenarios. It can quickly and accurately identify text information from invoice images and has high application value. Attached Figure Description

[0089] Figure 1 This is a flowchart of the invoice text recognition method based on multi-scale residuals and attention mechanisms disclosed in Embodiment 1 of the present invention;

[0090] Figure 2 This is a schematic diagram of the structure of the multi-scale residual CNN network in the invoice text recognition method based on multi-scale residual and attention mechanism disclosed in Embodiment 1 of the present invention;

[0091] Figure 3 This is a schematic diagram of the BiGRU network structure in the invoice text recognition method based on multi-scale residuals and attention mechanisms disclosed in Embodiment 1 of the present invention.

[0092] Figure 4 This is a schematic diagram of the text recognition network structure in the invoice text recognition method based on multi-scale residuals and attention mechanisms disclosed in Embodiment 1 of the present invention. Detailed Implementation

[0093] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0094] Example 1

[0095] like Figure 1 As shown, the invoice text recognition method based on multi-scale residuals and attention mechanisms includes the following steps:

[0096] Step 1: Acquire invoice images, preprocess the invoice images, and segment the invoice images into individual character images;

[0097] Step 1.1: Collect the original image data of the invoice using an image acquisition device;

[0098] Step 1.2: In order to improve the resolution of the invoice image, the original image data is preprocessed, specifically including image correction, grayscale processing and threshold segmentation for noise reduction;

[0099] Step 1.3: Segment the processed invoice image into individual character images, perform data augmentation on each individual character image, specifically including rotation and random occlusion, and then label each individual character image to obtain the corresponding ground truth label. The processed data can be represented as D = {(x1,y1),(x2,y2),...,(x...}i ,y i ),...,(x n ,y n )}, where x i ={x i1 ,x i2 ,...,x ij ,...,x im Let} be the set of m single-character images in the i-th sample, and x ij For the j-th single character image in the i-th sample, y i ={y i1 ,y i2 ,...,y ij ,...,y im Let} be the set of labels in the i-th sample, and y ij For x ij The actual label data, 1≤i≤n, 1≤j≤m;

[0100] Step 2, as follows Figure 2 As shown, a multi-scale residual CNN network is constructed. A single character image is input into the network to obtain the representation vector of the single character image.

[0101] Step 2.1: Construct a multi-scale residual CNN network and initialize the network parameters. Input the single character image data from each sample into the multi-scale residual CNN network, and perform convolution operations on the image data using convolution kernels of different sizes. In this embodiment, there are 3 convolution kernels with sizes of 3×3, 5×5, and 7×7, respectively. The calculation method for the convolution operation is shown in formula (1):

[0102]

[0103] Among them, f l w is the output vector after convolution. l and b l These are the learnable parameters of the convolutional layer. This represents the convolution operation. Here, is the ReLU activation function, and l represents the kernel label.

[0104] Step 2.2: Process the output vector f after convolution. l Perform max pooling, then pad the pooled output vector, concatenate it with the input vector of the convolutional layer, and use the concatenated vector as the input of the next convolutional layer.

[0105] Step 2.3: Repeat steps 2.1 and 2.2 until the individual character image data for each sample has been calculated. This yields the feature representation of each individual character image in each sample, which can be represented as {(f1,y1),(f2,y2),...,(f...}. i ,y i ),...,(f n ,y n )},f i ={f i1 ,f i2 ,...,f ij ,...,f im}, f ij Let be the representation vector of the j-th single character image in the i-th sample;

[0106] Step 3, as follows Figure 3 As shown, a BiGRU network based on an attention mechanism is constructed. The representation vector of a single character image in each sample is input into the network to obtain the contextual semantic information in the sample, and then the text recognition result of each sample is obtained.

[0107] Step 3.1: Initialize time step q = 1. When q = 1, initialize the hidden vector of BiGRU. vector As the query vector; when q > 1, the hidden state vector of the BiGRU unit at time step q-1 is used. As a query vector;

[0108] Step 3.2: Using the representation vector of each individual character image in each sample as the key and value, calculate the similarity between the query vector and the key. Based on the similarity, use softmax to calculate the importance of the representation vector of each character image. The calculation method is as follows:

[0109]

[0110]

[0111]

[0112] Where V, W, and b are learnable parameters in the attention mechanism, and e i =[e i1 ,e i2 ,...,e ij ,...,e im ] is {f i1 ,f i2 ,...,f ij ,...,f imThe weight vector of}, after normalization, yields α. ij , representing f in the i-th input sample ij Weights, exp(·) is an exponential function with the natural constant e as its base;

[0113] Step 3.3: After weighted fusion using the attention mechanism, The output of the previous time step is concatenated with the output of the previous time step and then input into the BiGRU. After calculation, the hidden state vector of the q-th time step in the BiGRU can be obtained. The calculation method is as follows:

[0114]

[0115] in, and These are the reset gate and update gate of the BiGRU at the q-th time step in the i-th sample, respectively. Let W be the candidate hidden vector of BiGRU at time step q in the i-th sample, σ(·) be the sigmoid activation function, tanh(·) be the hyperbolic tangent activation function, and W be the hidden vector of BiGRU at time step q in the i-th sample. r1 W r2 W z1 W z2 W s1 and W s2 This is the weight matrix in BiGRU, b r b z and b s This is the corresponding bias;

[0116] Step 3.4: Construct a text recognition model based on a fully connected layer network. As input, predict the recognition result of the q-th single character image in the i-th input sample, as shown in formula (6):

[0117]

[0118] Among them, W predict and b predict These are the learnable parameters in a text recognition network. φ is the output of the q-th single character image in the i-th input sample, where φ is the softmax function;

[0119] Step 3.5: Execute q = q + 1. When 1 ≤ q ≤ m, repeat steps 3.1-3.4 to obtain the recognition results of m individual character images of the i-th input sample. Otherwise, proceed to step 4;

[0120] Step 4, as follows Figure 4As shown, a text recognition network based on multi-scale residual CNN and attention mechanism BiGRU is trained to find the optimal network parameters, and the performance of the model is evaluated using the recognition accuracy evaluation metric.

[0121] Step 4.1: Use cross-entropy as the loss function of the text recognition network based on multi-scale residual CNN and attention mechanism BiGRU. Update the learnable parameters in the text recognition network based on multi-scale residual CNN and attention mechanism BiGRU using the stochastic gradient descent method to obtain the network parameters when the loss function reaches its minimum value. The cross-entropy loss function is shown in Equation (7):

[0122]

[0123] Where Loss is the loss function, W represents the learnable parameters in the text recognition network based on multi-scale residual CNN and attention mechanism BiGRU, and λ is the penalty factor.

[0124] Step 4.2: Use the text recognition network trained in Step 4.1, based on multi-scale residual CNN and attention mechanism BiGRU, to recognize text information from invoice images, and use the accuracy evaluation metric to evaluate the performance of the model.

[0125] Through the above technical solutions, this invention uses convolutional kernels of different sizes to perform convolution operations on image data to obtain more comprehensive feature representations. It introduces a residual network structure to alleviate the gradient vanishing problem, improving model stability and generalization performance. Furthermore, it proposes a BiGRU network based on an attention mechanism to obtain semantic relationships within the context, considering the contextual semantic information between individual characters and using an attention mechanism to distinguish the relative importance of different characters, effectively improving the accuracy of text recognition and achieving text recognition of invoice images.

[0126] Example 2

[0127] Based on Embodiment 1, Embodiment 2 of the present invention also provides an invoice text recognition system based on multi-scale residuals and attention mechanisms, including:

[0128] The preprocessing section is used to preprocess the invoice image and segment it into individual character images;

[0129] The convolutional pooling part is used to construct a multi-scale residual CNN network. A single character image is input into the network, and convolutional layers with different kernel sizes are used to perform convolution operations on the image data respectively. Then, max pooling is performed, and the pooling result is concatenated and fused with the input vector to obtain the representation vector of a single character image.

[0130] The text recognition part is used to construct a BiGRU network based on the attention mechanism. The representation vector of a single character image in each sample is input into the network to obtain the contextual semantic information in the sample and input it into the text recognition model to obtain the text recognition result for each sample.

[0131] The training part is used to train the text recognition network composed of the multi-scale residual CNN network and the attention-based BiGRU network, to find the optimal network parameters, obtain the optimal text recognition network, and use the optimal text recognition network to recognize text information from the invoice image.

[0132] Specifically, the preprocessing section is also used for:

[0133] Step 1.1: Collect the original image data of the invoice using an image acquisition device;

[0134] Step 1.2: Perform image correction, grayscale conversion, and threshold segmentation denoising on the original image data;

[0135] Step 1.3: Segment the processed original image data into individual character images, label each individual character image to obtain the true label corresponding to each character image. The processed data is represented as D = {(x1,y1),(x2,y2),...,(x...} i ,y i ),...,(x n ,y n )}, where x i ={x i1 ,x i2 ,...,x ij ,...,x im Let} be the set of m single-character images in the i-th sample, and x ij For the j-th single character image in the i-th sample, y i ={y i1 ,y i2 ,...,y ij ,...,y im Let} be the set of labels in the i-th sample, and y ij For x ij The actual label data, 1≤i≤n, 1≤j≤m.

[0136] More specifically, the convolutional pooling part is also used for:

[0137] Step 2.1: Construct a multi-scale residual CNN network. This network includes multiple parallel convolutional layers with different kernel sizes, and pooling layers connected to each convolutional layer. Input the single character image data from each sample into the multi-scale residual CNN network, and perform convolution operations on the image data using convolutional layers with different kernel sizes to obtain the output vector f. l , l represents the label of the convolution kernel;

[0138] Step 2.2, the output vector f after convolution l The input pooling layer performs max pooling, and then the output vector after pooling is padded and concatenated with the input vector of the convolutional layer to obtain the representation vector of a single character image.

[0139] Step 2.3: Repeat steps 2.1 and 2.2 until the individual character image data for each sample has been calculated, thus obtaining the feature representation of the individual character image in each sample as {(f1,y1),(f2,y2),...,(f i ,y i ),...,(f n ,y n )},f i ={f i1 ,f i2 ,...,f ij ,...,f im}, f ij Let be the representation vector of the j-th single character image in the i-th sample.

[0140] More specifically, in step 2.1, convolutional layers with different kernel sizes are used to perform convolution operations on the image data, and the calculation method for the convolution operation is as shown in formula (1):

[0141]

[0142] Among them, f l w is the output vector after convolution. l and b l These are the learnable parameters of the convolutional layer. This represents the convolution operation. is the ReLU activation function, and l represents the label of the convolution kernel.

[0143] More specifically, the text recognition section is also used for:

[0144] Step 3.1: Initialize time step q = 1. When q = 1, initialize the hidden vectors of the BiGRU network. vector As the query vector; when q > 1, the hidden state vector of the BiGRU network at time step q-1 is used. As a query vector;

[0145] Step 3.2: Using the representation vector of a single character image in each sample as the key and value, the BiGRU network performs weighted fusion of the representation vectors of the single character images using an attention mechanism;

[0146] Step 3.3: After weighted fusion via the attention mechanism, calculate the hidden state vector at the q-th time step in the BiGRU network.

[0147] Step 3.4: Construct a text recognition model based on a fully connected layer network. As input, predict the recognition result of the q-th single character image in the i-th input sample;

[0148] Step 3.5: Execute q = q + 1. When 1 ≤ q ≤ m, repeat steps 3.1-3.4 to obtain the recognition results of m individual character images of the i-th input sample. Otherwise, proceed with the training section.

[0149] More specifically, the calculation method for the attention mechanism weighted fusion of the representation vector of a single character image by the BiGRU network in step 3.2 is as follows:

[0150]

[0151]

[0152]

[0153] Where V, W, and b are learnable parameters in the attention mechanism, and e i =[e i1 ,e i2 ,...,e ij ,...,e im ] is {f i1 ,f i2 ,...,f ij ,...,f im The weight vector of}, after normalization, yields α. ij , representing f in the i-th input sample ij Weights, exp(·) is an exponential function with the natural constant e as its base.

[0154] More specifically, step 3.3 includes:

[0155] Will The output of the previous time step is concatenated with the output of the previous time step and then input into the BiGRU network. After calculation, the hidden state vector of the q-th time step in the BiGRU network can be obtained. The calculation method is as follows:

[0156]

[0157] in, and These are the reset gate and update gate of the BiGRU at the q-th time step in the i-th sample, respectively. Let W be the candidate hidden vector of BiGRU at time step q in the i-th sample, σ(·) be the sigmoid activation function, tanh(·) be the hyperbolic tangent activation function, and W be the hidden vector of BiGRU at time step q in the i-th sample. r1 W r2 W z1 W z2 W s1 and W s2 This is the weight matrix in BiGRU, b r b z and b s This is the corresponding bias.

[0158] More specifically, the text recognition model based on a fully connected layer network constructed in step 3.4 is shown in formula (6):

[0159]

[0160] Among them, W predict and b predict These are the learnable parameters in the text recognition model. φ is the output of the q-th single character image in the i-th input sample, and φ is the softmax function.

[0161] More specifically, the training section is also used for:

[0162] Step 4.1: Using cross-entropy as the loss function, update the learnable parameters in the text recognition network based on multi-scale residual CNN network and attention mechanism BiGRU using the stochastic gradient descent method to obtain the network parameters when the loss function reaches its minimum value. Substitute these network parameters into the text recognition network to obtain the optimized text recognition network. The loss function is shown in Equation (7):

[0163]

[0164] Where Loss is the loss function, W represents the learnable parameters in the text recognition network based on multi-scale residual CNN and attention mechanism BiGRU, and λ is the penalty factor.

[0165] Step 4.2: Use the text recognition network trained in Step 4.1, based on a multi-scale residual CNN network and an attention mechanism BiGRU, to recognize text information from the invoice image.

[0166] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An invoice text recognition method based on multi-scale residuals and attention mechanisms, characterized in that, Includes the following steps: Step 1: Preprocess the invoice image and segment it into individual character images; Step 2: Construct a multi-scale residual CNN network. Input a single character image into the network, and use convolutional layers with different kernel sizes to perform convolution operations on the image data respectively. Then perform max pooling operation. The pooling result is concatenated and fused with the input vector to obtain the representation vector of a single character image. Step 2.1: Construct a multi-scale residual CNN network. This network includes multiple parallel convolutional layers with different kernel sizes, and pooling layers connected to each convolutional layer. Input the single character image data from each sample into the multi-scale residual CNN network, and perform convolution operations on the image data using convolutional layers with different kernel sizes to obtain the output vector f. l , l represents the label of the convolution kernel; Step 2.2, the output vector f after convolution l The input pooling layer performs max pooling, and then the output vector after pooling is padded and concatenated with the input vector of the convolutional layer to obtain the representation vector of a single character image. Step 2.3: Repeat steps 2.1 and 2.2 until the individual character image data for each sample has been calculated, thus obtaining the feature representation of the individual character image in each sample as {(f1,y1),(f2,y2),...,(f i ,y i ),...,(f n ,y n )},f i ={f i1 ,f i2 ,...,f ij ,...,f im }, f ij Let be the representation vector of the j-th single character image in the i-th sample; Step 3: Construct a BiGRU network based on the attention mechanism. Input the representation vector of a single character image in each sample into the network to obtain the contextual semantic information in the sample and input it into the text recognition model to obtain the text recognition result for each sample. Step 3.1: Initialize time step q = 1. When q = 1, initialize the hidden vectors of the BiGRU network. vector As the query vector; when q > 1, the hidden state vector of the BiGRU network at time step q-1 is used. As a query vector; Step 3.2: Using the representation vector of a single character image in each sample as the key and value, the BiGRU network performs weighted fusion of the representation vectors of the single character images using an attention mechanism; Step 3.3: After weighted fusion via the attention mechanism, calculate the hidden state vector at the q-th time step in the BiGRU network. Step 3.4: Construct a text recognition model based on a fully connected layer network. As input, predict the recognition result of the q-th single character image in the i-th input sample; Step 3.5: Execute q = q + 1. When 1 ≤ q ≤ m, repeat steps 3.1-3.4 to obtain the recognition results of m individual character images of the i-th input sample. Otherwise, proceed to step 4; Step 4: Train the text recognition network composed of the multi-scale residual CNN network and the attention-based BiGRU network, find the optimal network parameters, obtain the optimal text recognition network, and use the optimal text recognition network to recognize text information from the invoice image.

2. The invoice text recognition method based on multi-scale residuals and attention mechanisms according to claim 1, characterized in that, Step 1 includes: Step 1.1: Collect the original image data of the invoice using an image acquisition device; Step 1.2: Perform image correction, grayscale conversion, and threshold segmentation denoising on the original image data; Step 1.3: Segment the processed original image data into individual character images, label each individual character image to obtain the true label corresponding to each character image. The processed data is represented as D = {(x1,y1),(x2,y2),...,(x...} i ,y i ),...,(x n ,y n )}, where x i ={x i1 ,x i2 ,...,x ij ,...,x im Let} be the set of m single-character images in the i-th sample, and x ij For the j-th single character image in the i-th sample, y i ={y i1 ,y i2 ,...,y ij ,...,y im Let} be the set of labels in the i-th sample, and y ij For x ij The actual label data, 1≤i≤n, 1≤j≤m.

3. The invoice text recognition method based on multi-scale residuals and attention mechanisms according to claim 1, characterized in that, In step 2.1, convolutional layers with different kernel sizes are used to perform convolution operations on the image data. The calculation method for the convolution operation is as shown in formula (1): Among them, f l w is the output vector after convolution. l and b l These are the learnable parameters of the convolutional layer. This represents the convolution operation. is the ReLU activation function, and l represents the label of the convolution kernel.

4. The invoice text recognition method based on multi-scale residuals and attention mechanisms according to claim 1, characterized in that, In step 3.2, the BiGRU network performs attention-based weighted fusion of the representation vectors of individual character images as follows: Where V, W, and b are learnable parameters in the attention mechanism, and e i =[e i1 ,e i2 ,...,e ij ,...,e im ] is {f i1 ,f i2 ,...,f ij ,...,f im The weight vector of}, after normalization, yields α. ij , representing f in the i-th input sample ij Weights, exp(·) is an exponential function with the natural constant e as its base.

5. The invoice text recognition method based on multi-scale residuals and attention mechanisms according to claim 1, characterized in that, Step 3.3 includes: Will The output of the previous time step is concatenated with the output of the previous time step and then input into the BiGRU network. After calculation, the hidden state vector of the q-th time step in the BiGRU network can be obtained. The calculation method is as follows: Where, r i q and These are the reset gate and update gate of the BiGRU at the q-th time step in the i-th sample, respectively. Let W be the candidate hidden vector of BiGRU at time step q in the i-th sample, σ(·) be the sigmoid activation function, tanh(·) be the hyperbolic tangent activation function, and W be the hidden vector of BiGRU at time step q in the i-th sample. r1 W r2 W z1 W z2 W s1 and W s2 This is the weight matrix in BiGRU, b r b z and b s This is the corresponding bias.

6. The invoice text recognition method based on multi-scale residuals and attention mechanisms according to claim 1, characterized in that, The text recognition model based on a fully connected layer network constructed in step 3.4 is shown in formula (6): Among them, W predict and b predict These are the learnable parameters in the text recognition model. φ is the output of the q-th single character image in the i-th input sample, and φ is the softmax function.

7. The invoice text recognition method based on multi-scale residuals and attention mechanisms according to claim 1, characterized in that, Step 4 includes: Step 4.1: Using cross-entropy as the loss function, update the learnable parameters in the text recognition network based on multi-scale residual CNN network and attention mechanism BiGRU using the stochastic gradient descent method to obtain the network parameters when the loss function reaches its minimum value. Substitute these network parameters into the text recognition network to obtain the optimized text recognition network. The loss function is shown in Equation (7): Where Loss is the loss function, W represents the learnable parameters in the text recognition network based on multi-scale residual CNN and attention mechanism BiGRU, and λ is the penalty factor. Step 4.2: Use the text recognition network trained in Step 4.1, based on a multi-scale residual CNN network and an attention mechanism BiGRU, to recognize text information from the invoice image.

8. An invoice text recognition system based on multi-scale residuals and attention mechanisms, characterized in that, include: The preprocessing section is used to preprocess the invoice image and segment it into individual character images; The convolutional pooling part is used to construct a multi-scale residual CNN network. A single character image is input into this network, and convolutional layers with different kernel sizes are used to perform convolution operations on the image data. Then, max pooling is performed, and the pooling result is concatenated with the input vector to obtain the representation vector of the single character image. The convolutional pooling part is also used for: Step 2.1: Construct a multi-scale residual CNN network. This network includes multiple parallel convolutional layers with different kernel sizes, and pooling layers connected to each convolutional layer. Input the single character image data from each sample into the multi-scale residual CNN network, and perform convolution operations on the image data using convolutional layers with different kernel sizes to obtain the output vector f. l , l represents the label of the convolution kernel; Step 2.2, the output vector f after convolution l The input pooling layer performs max pooling, and then the output vector after pooling is padded and concatenated with the input vector of the convolutional layer to obtain the representation vector of a single character image. Step 2.3: Repeat steps 2.1 and 2.2 until the individual character image data for each sample has been calculated, thus obtaining the feature representation of the individual character image in each sample as {(f1,y1),(f2,y2),...,(f i ,y i ),...,(f n ,y n )},f i ={f i1 ,f i2 ,...,f ij ,...,f im }, f ij Let be the representation vector of the j-th single character image in the i-th sample; The text recognition part is used to construct an attention-based BiGRU network. The representation vector of a single character image in each sample is input into this network to obtain the contextual semantic information of the sample, which is then fed into the text recognition model to obtain the text recognition result for each sample. The text recognition part is also used for: Step 3.1: Initialize time step q = 1. When q = 1, initialize the hidden vectors of the BiGRU network. vector As the query vector; when q > 1, the hidden state vector of the BiGRU network at time step q-1 is used. As a query vector; Step 3.2: Using the representation vector of a single character image in each sample as the key and value, the BiGRU network performs weighted fusion of the representation vectors of the single character images using an attention mechanism; Step 3.3: After weighted fusion via the attention mechanism, calculate the hidden state vector at the q-th time step in the BiGRU network. Step 3.4: Construct a text recognition model based on a fully connected layer network. As input, predict the recognition result of the q-th single character image in the i-th input sample; Step 3.5: Execute q = q + 1. When 1 ≤ q ≤ m, repeat steps 3.1-3.4 to obtain the recognition results of m individual character images of the i-th input sample. Otherwise, proceed with the training phase; The training part is used to train the text recognition network composed of the multi-scale residual CNN network and the attention-based BiGRU network, to find the optimal network parameters, obtain the optimal text recognition network, and use the optimal text recognition network to recognize text information from the invoice image.

Images