Bill text recognition system and method based on deep learning

Abstract

The invention relates to the technical field of bill text recognition, and discloses a bill text recognition system and method based on deep learning. The method comprises the following steps: acquiring target bill original image data containing a multi-channel pixel matrix and spatial resolution information; based on a matching result of the bill edge features and a preset template, geometric distortion correction is carried out on the original image, and a corrected bill image is generated; inputting the corrected image into a pre-training text region detection network to obtain positioning information containing text line boundary coordinates and region confidence; text line image blocks are extracted according to the positioning information, character segmentation preprocessing is executed, and a character-level image sequence is generated; calling a deep character recognition model to classify the sequence character by character, and generating an initial text recognition result; semantic verification and error correction are performed on the initial result based on the bill type knowledge base, final structured text data are generated, and the processing requirements of bills of different types and qualities can be met.

Description

Technical Field

[0001] This invention relates to the field of invoice text recognition technology, specifically to an invoice text recognition system and method based on deep learning. Background Technology

[0002] With the development of computer vision technology, document text recognition technology based on traditional image processing algorithms has emerged. This type of technology typically preprocesses document images through steps such as image grayscale conversion, binarization, and edge detection. Then, it uses methods like projection segmentation and connected component analysis to achieve text region localization and character segmentation. Finally, it completes character recognition through template matching or simple feature extraction algorithms. However, document types are diverse, and different types of documents vary significantly in layout, font style, and printing quality. Traditional image processing algorithms have poor adaptability and struggle to achieve stable text region localization and character segmentation for different document types. Furthermore, document acquisition is easily affected by factors such as shooting angle, lighting conditions, and paper wrinkles, leading to problems like geometric distortion, blurring, and noise interference in document images. Traditional algorithms have limited ability to process such complex images, often resulting in text region localization errors, incomplete character segmentation, and character recognition errors, failing to meet the accuracy and stability requirements of actual business applications for document text recognition.

[0003] In recent years, deep learning technology has made significant progress in the field of computer vision, providing a new technical direction for invoice text recognition. While some deep learning-based invoice text recognition methods have emerged, existing methods still have many shortcomings. Some methods focus only on model optimization in the text recognition stage, neglecting the importance of invoice image preprocessing. For invoice images with geometric distortions, directly inputting them into the text recognition model leads to a significant drop in recognition accuracy. Other methods, although incorporating image preprocessing steps, suffer from insufficient robustness in preprocessing algorithms, making it difficult to handle different degrees and types of geometric distortions. Furthermore, in the text region detection and character segmentation stages, existing methods are poorly adapted to invoices with complex layouts, easily resulting in missed or false detections of text regions, as well as segmentation errors caused by character adhesion or breakage. Simultaneously, most methods lack effective semantic verification mechanisms, failing to correct errors in a timely manner even during character recognition, ultimately making it difficult to guarantee the accuracy of the output text information. Summary of the Invention

[0004] The purpose of this invention is to provide a deep learning-based invoice text recognition system and method to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides a deep learning-based method for recognizing invoice text, the method comprising: Obtain the original image data of the target ticket, wherein the original image data includes a multi-channel pixel matrix and spatial resolution information; Geometric distortion correction is performed on the original image data to generate a corrected ticket image. The geometric distortion correction is based on the matching result of the ticket edge features and the preset template. The corrected ticket image is input into a pre-trained text region detection network to generate text region localization information, which includes text line boundary coordinates and region confidence. Based on the text region positioning information, extract text line image blocks, and perform character segmentation preprocessing on the text line image blocks to generate a character-level image sequence; The deep character recognition model is invoked to perform character-by-character classification processing on the character-level image sequence to generate initial text recognition results; Based on the knowledge base of invoice types, semantic verification and error correction are performed on the initial text recognition results to generate the final structured text data.

[0006] Preferably, the geometric distortion correction process performed on the original image data includes: Extract the set of ticket edge contour points from the original image data. The set of edge contour points is generated jointly by multi-scale gradient detection and Hough transform. Calculate the perspective transformation matrix between the edge contour point set and the standard ticket template, wherein the perspective transformation matrix includes rotation parameters and scaling factors; Perform bilinear interpolation on the original image data according to the perspective transformation matrix to generate a distorted corrected image. The corrected image is subjected to brightness equalization processing to make the difference in illumination intensity in different areas lower than a preset threshold.

[0007] Preferably, inputting the corrected ticket image into a pre-trained text region detection network includes: A feature pyramid network is used to extract multi-scale convolutional feature maps of the corrected image, which contain low-level detail features and high-level semantic features. Candidate text region boxes are generated through a region proposal network, and the generation of the candidate text region boxes is based on the sliding window detection results of the multi-scale convolutional feature map; The non-maximum suppression algorithm is used to filter redundant region boxes with an overlap rate exceeding a threshold, while retaining the text region location information with the highest confidence.

[0008] Preferably, the character segmentation preprocessing performed on the text line image block includes: Perform vertical projection analysis on the text line image block to generate a character gap histogram; The character segmentation boundary is determined based on the trough position of the character gap histogram, and the character segmentation boundary is used to divide independent character image regions; Normalization scaling is performed on the segmented character image regions to ensure that all character images maintain a uniform size.

[0009] Preferably, the step of calling the deep character recognition model to perform character-by-character classification processing on the character-level image sequence includes: A convolutional neural network is used to extract local feature vectors from the character-level image sequence. These local feature vectors contain information about stroke direction and structural distribution. By modeling the contextual dependencies of character sequences using a bidirectional long short-term memory network, a character-level probability distribution matrix is ​​generated. The probability distribution matrix is ​​decoded by combining the connection-time classification algorithm to generate an initial text recognition result sequence.

[0010] Preferably, the semantic verification and error correction processing of the initial text recognition result based on the invoice type knowledge base includes: The corresponding set of field rules is loaded from the knowledge base according to the invoice type identifier. The set of field rules includes length constraints and character type restrictions. The initial text recognition results are subjected to rule matching detection, and abnormal fields that do not conform to the rules are marked; The similarity between the abnormal field and the candidate correction item is calculated using the edit distance algorithm, and the optimal correction result is selected to replace the abnormal field.

[0011] Preferably, the method further includes: Collect sample invoice recognition errors from actual business scenarios within a preset period; Calculate the identification bias features of the erroneous samples, wherein the identification bias features include a character confusion matrix and a field missing rate; The classification weight parameters of the deep character recognition model are dynamically adjusted based on the recognition deviation characteristics.

[0012] Preferably, the method further includes: The final structured text data is subjected to key information extraction processing to generate a bill element index table; The document element index table is mapped to the business database to verify the logical consistency of the data.

[0013] Preferably, the method further includes: When a stamp or handwritten annotation is detected in the document image, the additional content recognition process is initiated. An attention mechanism network is used to separate the stamp area from the text area, avoiding interference from the stamp with the text recognition results.

[0014] Preferably, the present invention also includes a deep learning-based invoice text recognition system, the system including a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the deep learning-based invoice text recognition method described above.

[0015] Compared with the prior art, the beneficial effects of the present invention are: In the image preprocessing stage of the invoice, this method addresses the geometric distortion problem that easily occurs during invoice acquisition by proposing a geometric distortion correction approach based on matching invoice edge features with a preset template. By accurately matching invoice edge features with the preset template, geometric distortions caused by different shooting angles and varying degrees of wrinkling can be effectively corrected, generating a corrected invoice image. This correction method does not rely on a single image feature but combines the inherent structural features of the invoice with the preset template, demonstrating good adaptability to invoice images of different types and degrees of distortion. It lays a stable image foundation for subsequent text region detection, character segmentation, and recognition, avoiding processing errors in subsequent stages caused by image geometric distortion, and improving the accuracy of invoice text recognition from the source.

[0016] In the text region detection stage, a pre-trained text region detection network is used to generate text region localization information, which includes text line boundary coordinates and region confidence scores. This pre-trained network, trained on a large amount of different types of invoice image data, learns the characteristic patterns of text regions under different invoice layouts, demonstrating strong detection capabilities for text regions on complex invoice layouts. Simultaneously, the introduction of region confidence scores helps to filter out reliable text regions, reducing missed and false detections and ensuring the accuracy of subsequent text line image block extraction, thus providing high-quality processing objects for character segmentation and recognition.

[0017] In the character segmentation and recognition stage, after extracting text line image blocks based on text region location information, character segmentation preprocessing is performed to generate character-level image sequences, which are then classified character by character using a deep character recognition model. This staged processing approach first accurately segments the text line into independent character images, avoiding the impact of character adhesion or breakage on the recognition results. The deep character recognition model, through deep learning algorithms, learns the deep features of characters, exhibiting stronger recognition capabilities for characters with different font styles and printing quality compared to traditional character recognition algorithms. This effectively reduces the character recognition error rate and improves the accuracy of the initial text recognition results.

[0018] This method introduces a semantic verification and error correction process based on a knowledge base of ticket types. The knowledge base stores text structure rules and semantic logical relationships for different types of tickets. After obtaining the initial text recognition result, semantic verification is performed on the initial result according to the rules in the knowledge base to determine whether the text information conforms to the layout logic and data format requirements of that type of ticket. For logical errors and formatting errors found during the verification process, automatic error correction is performed, such as correcting text that does not conform to the date format or adjusting data that does not conform to the logical relationships of ticket fields. This step effectively compensates for the lack of semantic verification in existing methods. Even if a small number of errors occur during character recognition, they can be corrected through semantic verification and error correction mechanisms, improving the reliability of the final structured text data. Attached Figure Description

[0019] Figure 1 This is a schematic diagram illustrating the working principle of the deep learning-based invoice text recognition method described in this invention. Figure 2 Flowchart for geometric distortion correction processing; Figure 3 A flowchart for processing deep character recognition models. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1This invention provides a deep learning-based method for recognizing invoice text. The method includes: acquiring raw image data of the target invoice, which includes a multi-channel pixel matrix and spatial resolution information. The raw image data may originate from a scanning device or digital camera, and its format includes, but is not limited to, RGB or grayscale images. The spatial resolution information involves the image size and pixel density. Geometric distortion correction is performed on the raw image data to generate a corrected invoice image. The geometric distortion correction is based on the matching results between the invoice edge features and a preset template. The preset template stores the geometric attributes of a standard invoice, such as the four corner coordinates and aspect ratio. The corrected invoice image is input into a pre-trained text region detection network to generate text region localization information, which includes the text line boundary coordinates and region confidence. The text region detection network adopts a deep learning architecture and can adapt to various invoice layouts. Text line image blocks are extracted based on the text region localization information, and character segmentation preprocessing is performed on the text line image blocks to generate a character-level image sequence. The character segmentation preprocessing aims to decompose continuous text lines into independent character units. A deep character recognition model is invoked to perform character-by-character classification on the character-level image sequence, generating initial text recognition results. The deep character recognition model uses convolutional and recurrent neural network components to capture character features. Semantic verification and error correction are then performed on the initial text recognition results based on a document type knowledge base, generating final structured text data. The document type knowledge base includes field rules and common error patterns to ensure the accuracy and consistency of the output data.

[0022] Example 1: See Figure 2 Multi-scale gradient detection first constructs a Gaussian pyramid representation of the input original image, analyzes image gradient features at different scales, and generates a series of image layers with decreasing resolution through progressive downsampling. At each scale level, the gradient magnitude and orientation field of each pixel are calculated. This multi-scale analysis effectively captures edge information from coarse to fine granularity, enhancing the detection capability for blurred or weak edges. Gradient calculation typically uses Sobel or Scharr operators for convolution operations to obtain approximate derivatives of the image in the horizontal and vertical directions. The Hough transform is applied to the gradient detection results to identify straight or curved features in the image. This transform maps points in the image space to a parameter space for cumulative voting, detecting possible edge segments by setting angular resolution and distance parameters. The generation of the edge contour point set is an iterative optimization process that fuses multi-scale gradient information with the Hough transform detection results. Redundant edge responses are removed through non-maximum suppression, ultimately outputting a set of continuous point coordinates that accurately describe the boundary of the document.

[0023] Calculating the perspective transformation matrix between the edge contour point set and the standard ticket template requires establishing a geometric correspondence between the two planes. The standard ticket template pre-stores the coordinates of the four corner points of an ideal ticket and its geometric attributes such as aspect ratio. This template data originates from statistical analysis and normalization processing of a large number of real tickets. The point set matching process employs a feature point correspondence algorithm, comparing the geometric characteristics of the edge contour point set, such as curvature and angular distribution, to find the optimal transformation relationship with the template. The perspective transformation matrix is ​​calculated based on the least squares principle or a random sampling consensus algorithm, estimating the projection transformation parameters by solving an overdetermined system of equations. This matrix includes elements such as rotation angle, translation amount, and anisotropic scaling factor. The matrix solution process considers the influence of outliers in the point set, employing robust estimation methods to eliminate noise interference and ensure the accuracy and stability of the transformation relationship. The obtained perspective transformation matrix can completely describe the spatial mapping relationship from the original image to the standard coordinate system, providing a mathematical foundation for subsequent image correction.

[0024] When performing bilinear interpolation on the original image data based on the perspective transformation matrix, it is necessary to establish a pixel mapping relationship between the original image and the target image. Bilinear interpolation is an image resampling technique that determines the gray value of a target pixel by calculating the weighted average of its four neighboring pixels in the original image. This interpolation method effectively avoids the jagged edges that may occur with nearest-neighbor interpolation and has lower computational complexity than cubic interpolation. In practice, the position of each pixel in the target image in the original image is first calculated based on the perspective transformation matrix. Then, a weighted average is calculated based on the gray values ​​of the four pixels surrounding that position, with the weights determined by the relative distance between the pixels. The interpolation process needs to handle image boundary conditions, using methods such as mirror filling or constant expansion to handle out-of-bounds coordinates and ensure the integrity of the entire transformation process. Through this interpolation operation, geometric distortions in the original image are corrected, generating a visually natural and standard-compliant ticket image.

[0025] Brightness equalization is the final step in geometric distortion correction, aiming to eliminate uneven illumination in the image. Brightness equalization employs adaptive histogram matching, dividing the image into multiple overlapping or non-overlapping sub-regions. Within each region, a grayscale histogram is independently calculated and its distribution adjusted. The process analyzes the overall brightness distribution characteristics of the image, adjusting local grayscale values ​​using a contrast-limited adaptive histogram equalization method while limiting noise amplification that might be introduced by over-enhancement. The equalization parameters are dynamically adjusted based on the image content, maintaining appropriate contrast enhancement for text areas and avoiding over-processing for background areas. The goal of brightness equalization is to control the difference in illumination intensity between different regions of the image within a preset threshold range. This threshold is set according to the specific application scenario and is typically derived from statistical analysis of extensive experimental data. The processed image exhibits a uniform brightness distribution, creating favorable visual conditions for subsequent text detection and recognition.

[0026] In the complete process of geometric distortion correction, the specific implementation of multi-scale gradient detection requires consideration of the selection and optimization of scale parameters. The Gaussian pyramid is constructed using a progressive downsampling method, with the downsampling ratio typically set to √2. The number of layers is determined based on the original image resolution to ensure sufficient edge information is retained even at the highest layer. At each scale layer, the kernel size of the gradient operator needs to be adapted to the scale factor; larger scales use larger kernels to capture macroscopic edge features, while smaller scales use smaller kernels to retain detailed information. After gradient calculation, non-maximum suppression is performed, retaining local gradient maxima as candidate edge points, while a hysteresis thresholding method is applied to distinguish between strong and weak edges. The implementation of the Hough transform requires a reasonable setting of the discretization precision of the parameter space; excessively high precision increases computational burden, while excessively low precision affects detection accuracy. In practical applications, asymptotic probabilistic Hough transform is usually used to improve computational efficiency while ensuring detection accuracy. Solving the perspective transformation matrix involves complex numerical calculations and requires handling possible singular matrix cases. When there is a significant difference between the edge contour point set and the template point set, the algorithm automatically activates a backup affine transformation estimation method to ensure robustness in extreme cases. After matrix solving, a quality assessment is required, analyzing the transformation accuracy through reprojection error analysis. A re-detection mechanism is triggered when the error exceeds a set threshold. The implementation of bilinear interpolation requires special attention to numerical precision, employing floating-point arithmetic to ensure accurate coordinate mapping, while optimizing processing speed through parallel computation. For large images, interpolation operations are performed in blocks to reduce memory usage and improve processing efficiency.

[0027] In brightness equalization, the selection of sub-region size requires a trade-off between local adaptability and computational complexity, and is typically dynamically determined based on the size of the text region in the image. During equalization, the signal-to-noise ratio (SNR) of the image is monitored to avoid introducing excessive noise in low-contrast areas. The processed image undergoes quality assessment to check for over-enhancement or loss of detail, and parameter adjustments and reprocessing are performed as necessary. The entire geometric distortion correction process forms a complete quality control closed loop, ensuring that the output image meets the requirements of subsequent processing through multi-stage verification. Parameter settings during the correction process are adaptive, dynamically adjusting according to the characteristics of the input image. For different types of ticket images, the system loads corresponding parameter configuration files to ensure the targeting of the processing strategy. All processing steps have detailed logging and error handling mechanisms for easy problem tracking and system optimization. The geometric distortion correction module adopts a pipelined processing architecture, with efficient data flow between processing stages through a caching mechanism, supporting real-time processing requirements. The module design considers scalability requirements, allowing new correction algorithms and processing strategies to be easily integrated into the existing framework.

[0028] Example 2: The Feature Pyramid Network achieves feature fusion through a top-down path and lateral connections. In the top-down path, high-level feature maps are upsampled to match the size of the low-level feature maps. Then, features from different levels are element-wise added through lateral connections. This design allows the network to possess strong semantic features at multiple scales. Low-level features contain rich edge and texture information, which helps to accurately locate text region boundaries, while high-level features provide contextual information, helping to distinguish between text and non-text regions. The network backbone typically uses classic structures such as ResNet or VGG, pre-trained on the ImageNet dataset, and then fine-tuned using ticket image data to adapt to specific domain features. The generation process of multi-scale convolutional feature maps involves multiple convolutional layers and pooling operations. Each level corresponds to a different receptive field, enabling the detection of text regions of different sizes. Spatial information is preserved between feature maps through skip connections, avoiding the loss of detailed features in deep networks. When generating candidate text region boxes through the Region Proposal Network, this network performs sliding window detection at each level of the feature pyramid. The Region Proposal Network (RPN) sets multiple anchor boxes of different scales and aspect ratios at each feature map location. The size of these anchor boxes is designed based on the general characteristics of text lines, typically using flat rectangles to match the banded distribution of text. For each anchor box, the RPN predicts two outputs: a text / non-text classification score and an adjusted bounding box coordinate value. A sliding window moves across the feature map in a convolutional manner, calculating the overlap between the anchor box and the ground truth text region at each location and regressing the precise bounding box coordinates. The candidate box generation process uses a fully convolutional network, which efficiently processes the entire image and avoids redundant computation. The generated candidate boxes include location coordinates and a confidence score, representing the probability that the region contains text.

[0029] The process of filtering redundant bounding boxes using the non-maximum suppression algorithm employs an iterative selection mechanism. The algorithm first sorts candidate boxes based on their confidence scores, then calculates the intersection-union ratio (IU) of the highest-scoring box with all remaining boxes. IU measures the degree of overlap between two rectangular regions, calculated as the ratio of their intersection area to their union area. When the IU of two boxes exceeds a preset threshold, they are considered to have detected the same text region; the box with the higher confidence score is retained, while the box with the lower confidence score is removed. The non-maximum suppression process proceeds from highest to lowest confidence score, comparing each retained box with subsequent boxes to ensure that highly overlapping detections do not exist in the final result. Threshold settings must balance recall and precision; excessively high thresholds may lead to missed detections, while excessively low thresholds may result in duplicate detections. After non-maximum suppression, the retained text region location information includes precise bounding box coordinates and region confidence scores.

[0030] After text region detection, text line image patches are extracted based on the localization information. These patches are rectangular regions cropped from the corrected document image. The cropping process considers the coordinate information of the bounding box, maintains image quality through bilinear interpolation, and records the positional relationship of each image patch in the original image. The extracted text line image patches may contain tilted or curved text, requiring further processing before character-level recognition. Vertical projection analysis is a core step in the character segmentation preprocessing of the text line image patches. Vertical projection analysis generates a histogram reflecting the character distribution by summing the gray values ​​of each pixel column in the horizontal direction of the text line image. In the binarized text line image, the pixel values ​​of character regions are lower, while the pixel values ​​of background regions are higher, resulting in a clear peak and trough feature in the projection histogram. Peaks correspond to the pixel column regions where characters are located, and troughs correspond to the gaps between characters. Histogram generation needs to consider image quality factors, using filtering to smooth out noise-induced jitter and ensure the accuracy of trough locations. Determining the character segmentation boundary based on the trough positions of the character gap histogram is a crucial step. The detection of trough positions employs a local minimum search algorithm. This algorithm scans the entire histogram, searching for points that continuously decrease and then increase in width as candidate trough positions. Then, it filters these troughs based on their depth and width, eliminating false troughs caused by noise. Determining the character segmentation boundary also needs to consider character connectivity; regions with abnormal widths may require secondary segmentation. The selection of the segmentation boundary must ensure the integrity of each character region while avoiding dividing a single character into multiple parts.

[0031] When performing normalization scaling on segmented character image regions, it is necessary to maintain the character's aspect ratio and structural features. Normalization adjusts character images of different sizes to a uniform dimensions, typically using a standard size of 28×28 or 32×32 pixels. The scaling algorithm employs bilinear interpolation or region interpolation methods to maintain the recognizability of character strokes while adjusting the size. During normalization, the character image is also centered by calculating the centroid position of the character pixels and shifting the character to the center of the image. Furthermore, the image grayscale values ​​are normalized to the 0-1 range to reduce the impact of lighting conditions on the recognition results. The normalized character-level image sequence is arranged in the order of the original text lines, forming a standardized input for deep character recognition models. The specific implementation of the feature pyramid network needs to consider the issue of unifying the number of channels in feature maps at different levels. Typically, 1×1 convolutions are used to adjust the channel dimension, enabling feature maps from different levels to be fused. Region proposal networks require a balance of positive and negative samples during training. Positive samples are anchor boxes with an intersection-union (IU) ratio greater than 0.7 with the ground truth bounding boxes, while negative samples are anchor boxes with an IU ratio less than 0.3. The implementation of non-maximum suppression algorithms requires optimization of computational efficiency; for a large number of candidate boxes, a divide-and-conquer strategy or approximation algorithms are used to accelerate processing.

[0032] Vertical projection analysis requires tilt correction when processing slanted text. This is achieved by detecting the text line angle using Hough transform and performing rotation correction. Determining character segmentation boundaries also considers the writing characteristics of specific languages, such as the square structure of Chinese characters or the hyphenation in English. During normalization scaling, characters with disproportionate height-to-width ratios are padded with white borders to maintain their proportions. The entire text region detection and character segmentation process forms a complete preprocessing pipeline, with corresponding quality control mechanisms at each stage to ensure that the output data meets the input requirements of the deep character recognition model. The text region detection network is trained end-to-end, with a loss function comprising both classification and regression losses, and network parameters are optimized using gradient descent. The parameter settings for character segmentation preprocessing are adaptive, automatically adjusting the projection analysis window size based on the height of the text lines. The system records key parameters and intermediate results during processing for troubleshooting and algorithm optimization. The entire implementation emphasizes a balance between processing efficiency and accuracy, employing multi-threaded parallel processing to accelerate computationally intensive operations.

[0033] Example 3: See Figure 3The deep character recognition model begins with character-by-character classification of character-level image sequences. This model uses a convolutional neural network (CNN) to extract local feature vectors from character images. These local feature vectors contain key information such as stroke direction, curvature changes, and structural distribution. A CNN typically consists of multiple convolutional layers, pooling layers, and activation function layers stacked alternately. Convolutional layers use small filters sliding across the image to capture local patterns such as edges and corners through convolution operations. Pooling layers downsample the feature maps, preserving salient features while reducing the number of parameters. Activation functions introduce non-linear transformations to enhance the network's expressive power. Network training uses backpropagation to optimize weight parameters and employs supervised learning with a large number of labeled character images. The loss function is typically chosen as cross-entropy or mean squared error to minimize the difference between predicted values ​​and true labels. During feature extraction, the network generates high-dimensional feature vectors that encode the visual characteristics of characters, providing input for subsequent sequence modeling. When modeling the contextual dependencies of character sequences using a bidirectional long short-term memory (LSTM) network, this network can simultaneously consider contextual information and capture long-term dependency patterns between characters. Bidirectional Long Short-Term Memory (LSTM) networks consist of two recurrent networks: a forward network and a backward network. The forward network processes the sequence in chronological order, while the backward network processes it in reverse order. The hidden state at each time step integrates information from both directions. Internally, the network includes input gates, forget gates, and output gates. The input gate controls the inflow of new information, the forget gate determines which historical information needs to be retained, and the output gate adjusts the output ratio of the current state. For a character sequence, the feature vector of each character serves as the input for each time step. The network learns the contextual patterns in the sequence, such as common character combinations or grammatical constraints, through a gating mechanism. Updating the hidden state involves complex weight calculations, which are implemented using sigmoid and tanh functions for gating. The final output is a probability distribution matrix for each time step, representing the confidence level of each character category.

[0034] When decoding the probability distribution matrix using a connection-series classification algorithm, this algorithm can handle situations where the input and output sequences have different lengths, directly generating an initial text recognition result sequence. Connection-series classification introduces whitespace as placeholders, allowing the model to output sequences of variable length, and calculates the sum of probabilities for all possible paths through dynamic programming. The decoding process searches for the most probable character sequence, selecting the optimal path through either beam search or greedy decoding strategies. Greedy decoding selects the character with the highest probability each time, while beam search retains multiple candidate paths to improve accuracy. The loss function of connection-series classification is based on maximizing the sequence probability, and gradients are calculated during training using a forward-backward algorithm to optimize model parameters. For the probability distribution matrix, the algorithm merges duplicate characters and removes whitespace to generate the final text string, thus addressing cases of inaccurate character segmentation or character concatenation.

[0035] In the specific implementation of convolutional neural networks, the network architecture may contain multiple convolutional blocks. Each block consists of convolutional layers, batch normalization layers, and ReLU activation functions. Batch normalization accelerates the training process and improves stability, while the ReLU function provides sparse activation to avoid gradient vanishing. The number of filters increases with network depth, gradually increasing from an initial 32 or 64 to 256 or 512 to capture more complex features. Pooling layers typically use max pooling with a window size of 2x2 or 3x3 and a stride of 2 to reduce the feature map size. Fully connected layers map the flattened feature vectors to the number of character classes, outputting a score for each class. Data augmentation techniques such as rotation, scaling, or elastic deformation are applied to the training data to improve the model's generalization ability. Network initialization uses He initialization or Xavier initialization to ensure smooth gradient flow in the early stages of training. The implementation of bidirectional long short-term memory networks needs to consider the unfolding of time steps. For character sequences of variable length, the network unifies the input size by padding or truncating, but uses a masking mechanism to ignore invalid time steps. The number of hidden layer units is set according to task complexity, typically 128 or 256, and the number of layers may be 2 or 3 to balance model capacity and computational cost. Gradient clipping is applied during training to prevent gradient explosion. The optimizer is Adam or RMSprop, and the learning rate is dynamically adjusted. The probability distribution matrix is ​​generated using the softmax function, which converts the network output into class probabilities, ensuring that the sum of all probabilities is 1. In sequence modeling, attention mechanisms are sometimes integrated into the network, allowing the model to focus on relevant time steps and improving the performance of long sequence processing.

[0036] Decoding optimization of connection-based temporal classification algorithms involves various strategies. Prefix search algorithms are used for accurate decoding but are computationally expensive. In practice, beam search is often used to balance efficiency and accuracy. The beam width parameter controls the number of candidate paths, usually set to 5 or 10. A larger beam width results in more accurate results but slower speed. Language model ensemble is a common extension, introducing n-gram or recurrent neural network language models to adjust character probability distributions and correct context-based errors. During training, the connection-based temporal classification loss function is defined as the negative log probability sum, and the loss is minimized through gradient descent. During the inference phase, the model processes the entire sequence once and outputs the recognition result without pre-segmenting characters. Training data for deep character recognition models needs to cover various fonts, sizes, and styles, including printed and handwritten characters, to enhance robustness. The training set contains tens of thousands to millions of character images, with annotation information including character sequence and position information. Data preprocessing includes normalization, denoising, and augmentation to ensure input consistency. Model evaluation uses character error rate or word error rate metrics to measure performance on independent test sets. During deployment, the model can be quantized into integer or floating-point format to optimize inference speed, making it suitable for embedded or cloud environments. The feature extraction capabilities of convolutional neural networks are enhanced through deep structures; residual connections or dense connections alleviate gradient vanishing, allowing networks to reach tens or even hundreds of layers. Feature vectors can be further compressed or encoded, reducing dimensionality before being input into recurrent networks. Context modeling using bidirectional long short-term memory networks is particularly suitable for continuous text, such as date or amount fields on invoices, where strong dependencies exist between characters. Each element of the probability distribution matrix corresponds to a time step and a character class; low-probability predictions are filtered by thresholding to reduce noise impact. The implementation of the connection-based temporal classification algorithm needs to handle the merging rules of whitespace characters, which represent character intervals or silent segments and are removed during decoding. The algorithm supports the compression of repeated characters unless the context requires repetition. Sequence alignment of training data is a challenge; connection-based temporal classification does not require mandatory alignment, simplifying annotation. The model integrates multiple networks or uses ensemble learning to improve recognition accuracy.

[0037] In document text recognition scenarios, deep character recognition models need to adapt to specific domain character sets, such as numbers, letters, and symbols, and may be extended to support Chinese characters or special characters. Real-time requirements influence model design; lightweight networks such as MobileNet or SqueezeNet may be adopted, sacrificing a small amount of accuracy for speed. Error analysis helps identify common confusing characters, such as "O" and "0", which can be improved through post-processing or data balancing. The entire process involves the collaboration of multiple components: convolutional neural networks are responsible for visual feature extraction, bidirectional long short-term memory networks capture temporal dependencies, and connection-based temporal classification decoding generates sequences. Pipeline optimization ensures end-to-end efficiency, and model updates adapt to new data through online learning. Implementation details such as batch size, learning rate scheduling, and regularization techniques fine-tune model performance to meet practical application requirements.

[0038] The loss function for connection-based temporal classification can be expressed as: , in: Indicates the loss value. It is the training dataset. It is an input character-level image sequence. It is the corresponding real label sequence. The model is given input Time output The conditional probability is calculated. This loss function aims to maximize the probability of a correct sequence by updating the network parameters through gradient backpropagation.

[0039] Example 4: Semantic verification, error correction, and dynamic adjustment mechanisms based on a knowledge base for invoice types are used to verify and correct the initial text recognition results, while simultaneously optimizing model performance by collecting error samples. Semantic verification and error correction first load the corresponding set of field rules from the knowledge base based on the invoice type identifier. This set includes length constraints and character type restrictions; for example, for VAT invoices, the rules might specify a 12-digit invoice code and a date format of YYYY-MM-DD. The knowledge base stores the specifications for different invoice types in a structured format, and rules are loaded through database queries or configuration files. When performing rule matching detection on the initial text recognition results, the system applies regular expressions or finite state machines to verify each field. For example, it checks whether numeric fields contain letters or symbols, marking abnormal fields that do not conform to the rules. These abnormal fields are recorded and trigger the error correction process. The similarity between the abnormal field and the candidate correction item is calculated using the edit distance algorithm. The edit distance algorithm, such as Levenshtein distance, measures the minimum number of edit operations (insertion, deletion, replacement) between strings. The candidate correction items are drawn from the historical correct values ​​or common value library of the knowledge base. The algorithm compares all candidates and selects the item with the smallest distance as the optimal correction result to replace the abnormal field.

[0040] The dynamic adjustment mechanism collects error samples of invoice recognition from actual business scenarios within a preset period, which can be set weekly or monthly. Error samples originate from manual review feedback, system logs, or user reports. Sample data includes original images, recognition results, and correct annotations. The mechanism calculates the recognition deviation features of the error samples, including a character confusion matrix and field missing rates. The character confusion matrix statistically analyzes the frequency of misidentified character pairs, such as the confusion between "0" and "O". The field missing rate measures the proportion of missed detections for a specific field. Based on the recognition deviation features, the classification weight parameters of the deep character recognition model are dynamically adjusted. This adjustment process is achieved through online learning or incremental learning, for example, using stochastic gradient descent to update the weights of fully connected layers to reduce the recurrence of common error patterns.

[0041] Taking the recognition process of a VAT invoice as an example, the initial text recognition result generates field values ​​such as invoice code, invoice number, invoice date "2023-02-30", and amount "five hundred yuan". The semantic verification module loads a set of rules based on the invoice type identifier "VAT invoice". The rules include that the invoice code must be 12 digits long, the date must be valid, and the amount must be in numeric format. The rule matching detection found that the invoice date "2023-02-30" was invalid (February has no 30th) and marked it as an abnormal field. At the same time, the amount "five hundred yuan" did not conform to the numeric format rules. The edit distance algorithm calculates similarity for abnormal fields. For the date field, candidate correction terms include "2023-02-28" and "2023-02-29". The algorithm calculates the edit distance between "2023-02-30" and each candidate. The minimum distance corresponds to "2023-02-28" (distance 1, replacing "0" with "8"), so it is replaced with "2023-02-28". For the amount field, candidates include "500 yuan" and "500.00". After distance calculation, "500 yuan" has the smallest distance (inserting "5" and "0"), so it is corrected to "500 yuan". The dynamic adjustment mechanism collects error samples in a monthly cycle. If multiple date recognition errors are found, the character confusion matrix shows a high confusion frequency between "3" and "8", and the field missing rate shows a 5% false negative rate for the amount field, the model is adjusted by increasing the penalty weight for confused character pairs to enhance the detection sensitivity of the amount field. Refer to Table 1, which shows the rules for the invoice field and the rule definitions for semantic verification. Table 1: Bill Field Rules Table

[0042] In the implementation of semantic verification and error correction processing, the construction of the knowledge base depends on the accumulation of historical bill data, and the rule definition is completed by analyzing the common features of a large number of real bills. For example, the rule for the date field is based on the validity check of the Gregorian date, and the rule for the amount field excludes Chinese capital numbers. The rule matching detection uses an automated script to batch process field values. For complex rules such as date validity, a date parsing library is called for verification; the character type limit uses Unicode attribute checks. The implementation of the edit distance algorithm optimizes the calculation efficiency, calculates the string difference through dynamic programming, and the sources of candidate corrections include high-frequency value statistics and domain dictionaries. The algorithm sets a threshold to filter out candidates with low similarity to avoid incorrect corrections. The results after error correction are recorded in the database for knowledge base update and model retraining. The error sample collection of the dynamic adjustment mechanism is implemented through the system monitoring module, and the collection process includes data cleaning, de-identification, and annotation to ensure the quality of the samples. The calculation of the recognition deviation features uses statistical analysis tools, and the character confusion matrix shows the misrecognition relationship between characters in the form of a two-dimensional table. The field missing rate is aggregated according to the field type. The model weight adjustment is executed when the inference service is idle, and a rolling update strategy is adopted to avoid service interruption. The adjusted model is verified through A / B testing. The entire implementation process emphasizes closed-loop optimization, and the collection of error samples and model adjustment form an iterative cycle to continuously improve the recognition accuracy.

[0043] In specific operations, the semantic verification module is integrated into the end of the text recognition pipeline and executed as a post-processing step. The verification results output structured data and record exception logs at the same time. The dynamic adjustment mechanism is decoupled from the model service, and error samples are processed asynchronously through a message queue to ensure the system response performance. The implementation details include parameter tuning, such as the threshold setting of the edit distance and the strictness control of rule matching. These parameters are dynamically configured according to business requirements. The processing of VAT invoices in the example shows a typical application scenario, and similar logic applies to other bill types such as receipts and bills, which are generalized by adapting the knowledge base rules. The reliability of semantic verification and error correction processing depends on the completeness of the knowledge base, and the rule update mechanism regularly synchronizes the latest business specifications. When the edit distance algorithm processes similar strings, it considers phonetic or morphological errors, such as the confusion between the Chinese numbers "one" and "壹". The algorithm is extended to support phonetic similarity calculation. The sample collection of the dynamic adjustment mechanism covers various error types, including false positives and false negatives. The deviation feature analysis identifies the weaknesses of the system and guides targeted optimization. The rule examples in the table reflect common constraint types. In actual implementation, the rule base may contain hundreds of rules covering all bill fields. The rule matching engine uses parallel processing to improve throughput for batch processing of large-scale data. The edit distance calculation is optimized by using a Trie tree to accelerate candidate retrieval and reduce the calculation overhead. In the model adjustment stage, the weight update is based on small batches of samples to prevent overfitting, and the adjusted model version management supports a rollback mechanism.

[0044] Example 5: The final structured text data undergoes key information extraction processing to generate a document element index table. This index table is then mapped to the business database to verify logical consistency. Simultaneously, when a seal or handwritten annotation is detected in the document image, an additional content recognition process is initiated, employing an attention mechanism network to separate interference regions. Key information extraction processing is based on predefined document type templates. The system loads corresponding extraction rules according to the document category identifier. These rules clearly define the core data fields to be extracted and their positional relationships within the text structure. Taking a VAT invoice as an example, its final structured text data includes multiple fields such as buyer's name, taxpayer identification number, bank account number, goods name, quantity, unit price, amount, tax rate, and tax amount. The key information extraction module filters out the elements that need to be indexed from these fields, such as invoice code, invoice number, invoice date, buyer's name, and total amount. The extraction process combines a rule engine with pattern matching. Fields with fixed locations are extracted directly using coordinates, while fields with variable locations are determined through keyword triggering and contextual analysis; for example, identifying the "total amount" field. The generated invoice element index table is stored in a key-value pair structure. Each index entry contains a field name, field value, confidence score, and original location information. This index table is organized in JSON or XML format for easy subsequent querying and verification. When mapping the invoice element index table to the business database, the system selects the corresponding mapping rules and verification process based on the invoice type. The mapping is implemented through a database connection interface or API call. The system first queries the business database for records matching the key fields in the index table; for example, searching for the corresponding invoice in the invoice management system using the invoice code and invoice number. When verifying logical consistency, the system compares the field values ​​in the index table with the corresponding values ​​in the business database records, checking for contradictions or differences, such as whether the amount is consistent, whether the date is within a reasonable range, and whether the buyer information matches. The verification process takes into account business rules and constraints, such as an invoice being reimbursed only once and the amount not exceeding the budget limit. When inconsistencies are found, an anomaly report is generated and the item requiring manual review is marked. The mapping verification results are fed back into the system log, and the invoice processing status is updated to ensure the integrity and traceability of the data chain.

[0045] The additional content recognition process is triggered based on image analysis results. The system automatically initiates this process when a stamp or handwritten annotation is present in the document image. Stamp detection is achieved through color segmentation and shape analysis. The system first converts the image from the RGB color space to the HSV color space, segments potential stamp areas using thresholds for the red or blue channels, and then confirms the stamp shape through contour detection and geometric feature analysis. Handwritten annotation detection is based on handwriting texture analysis and connected component analysis to identify areas that significantly differ from printed text features. After detecting these additional contents, the system activates an attention mechanism network for region separation. This network generates a spatial attention map, quantifying the importance of each image region to the text recognition task. The attention mechanism network typically employs an encoder-decoder structure. The encoder extracts multi-scale features from the image, and the decoder generates an attention weight map through upsampling and dot product operations. Text regions have higher weights, while non-text regions, such as stamps or annotations, have lower weights. By applying the attention weight map, the system can suppress activation responses in non-text regions and highlight text regions, thereby reducing the interference of additional content on the text recognition process.

[0046] In the specific implementation of key information extraction, the system needs to handle the diversity of invoice formats, as different template variations may exist for the same type of invoice. The system automatically selects the closest extraction template through a template matching algorithm, based on the global layout features of the invoice and the relative positional relationships of key fields. For invoices with complex table structures, the extraction process employs cell merging and cross-row parsing techniques to ensure the integrity of continuous content. The generation of the index table also includes data standardization steps, such as unifying date formats and converting numerical units, to ensure that invoice data from different sources have a consistent representation. A full-text search index is built when the index table is stored, supporting fast keyword-based queries and statistical analysis. The implementation of business database association mapping needs to consider the diversity of database types. The system supports multiple data sources such as relational databases and NoSQL databases through the adapter pattern. The mapping process uses a transaction mechanism to ensure data consistency; when verification detects serious inconsistencies, the operation is automatically rolled back and an alarm is triggered. Verification rules support custom configuration; different levels of strictness can be set for different business scenarios. For example, financial audits require strict matching of all fields, while archiving may only verify key fields. The mapping results will generate a difference report, which records in detail the comparison results and difference types of each field, providing a basis for subsequent processing.

[0047] In the additional content recognition process, the training of the attention mechanism network requires a large amount of labeled data, including pixel-level segmentation masks of text and non-text regions in the image. The network training employs supervised learning, with a loss function combining segmentation accuracy and text recognition accuracy as multi-task learning objectives. During the inference phase, the network processes the entire invoice image, outputting the probability that each pixel belongs to a text region, and then generates a binarized attention mask through thresholding. Multiplying this mask by the original image suppresses features in non-text regions and enhances features in text regions, thereby improving the input quality of subsequent text recognition modules. For particularly complex additional content interference, the system can use an iterative processing approach, performing secondary analysis and correction on low-confidence regions after the initial recognition. A specific example of key information extraction involves a transportation invoice containing a complex table. The structured text data of this invoice includes multiple modules such as shipper, consignee, goods details, and freight calculation. The system first identifies the invoice type as a "road transport invoice," loads the corresponding extraction template, and defines the elements to be indexed, including key fields such as invoice number, invoice date, shipper name, consignee name, total weight of goods, and total freight. The extraction module analyzes the text's layout to locate table areas, then parses the product details row by row, automatically merging content from cells spanning multiple rows. The generated index table contains standardized field values, such as dates converted to "YYYY-MM-DD" format and amounts converted to numeric format with two decimal places. The index table also records the extraction confidence level for each field, providing a reference for subsequent validation.

[0048] For this transportation invoice, the system uses a business database mapping mechanism. It queries relevant transportation records in the logistics management database, narrowing the search range by invoice number and date range. The verification process compares the shipper and consignee information in the index table with the contract data in the database, checks the weight of the goods and freight amount against pricing rules, and verifies that the invoice status has not been duplicated. If a discrepancy is found between the freight amount and the contract pricing in the database, the system determines whether manual review is required based on a preset tolerance threshold, recording the discrepancy details for subsequent analysis. Additional content recognition is applied to this transportation invoice. The invoice image has a red "Reimbursed" stamp in the lower right corner and a handwritten "Urgent" annotation on the left. The stamp detection algorithm segments a circular area using a red channel threshold, and contour analysis confirms the stamp's shape features. The handwritten annotation is identified through handwriting curvature analysis and morphological differences from printed characters. An attention mechanism network processes the entire image, generating an attention weight map, assigning low weights to the stamp and handwritten areas and high weights to the table text areas. The weighted image features input to the text recognition module effectively avoid interference from stamped text and handwritten annotations on the content of formal forms, thus improving the recognition accuracy of key fields.

[0049] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0050] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A deep learning-based method for recognizing text on a document, characterized by, The method comprises the following steps: obtain the original image data of the target bill, which contains a multi-channel pixel matrix and spatial resolution information; perform geometric distortion correction processing on the original image data to generate a corrected bill image, wherein the geometric distortion correction processing is based on the matching results of bill edge features and a preset template; input the corrected bill image into a pre-trained text region detection network to generate text region positioning information, which contains text line boundary coordinates and region confidence; extract text line image blocks according to the text region positioning information and perform character segmentation preprocessing on the text line image blocks to generate a character-level image sequence; invoke a deep character recognition model to perform character-by-character classification processing on the character-level image sequence to generate an initial text recognition result; perform semantic verification and error correction processing on the initial text recognition result based on a bill type knowledge base to generate final structured text data. 2.The deep learning-based bill text recognition method of claim 1, wherein, The geometric distortion correction processing on the original image data comprises: extracting a set of bill edge contour points in the original image data, which is generated by jointing multi-scale gradient detection and Hough transform; calculating the perspective transformation matrix of the set of edge contour points and a standard bill template, which contains rotation parameters and scaling factors; performing bilinear interpolation operation on the original image data according to the perspective transformation matrix to generate a corrected image that eliminates distortion; performing brightness equalization processing on the corrected image to make the illumination intensity difference of different regions below a preset threshold. 3.The deep learning-based bill text recognition method of claim 2, wherein, The input of the corrected bill image into a pre-trained text region detection network comprises: extracting multi-scale convolution feature maps of the corrected image using a feature pyramid network, which contains low-level detail features and high-level semantic features; generating candidate text region boxes through a region proposal network, which is based on the sliding window detection results of the multi-scale convolution feature maps; filtering redundant region boxes with an overlap rate exceeding a threshold using a non-maximum suppression algorithm and retaining the text region positioning information with the highest confidence. 4.The deep learning-based bill text recognition method of claim 3, wherein, The character segmentation preprocessing on the text line image blocks comprises: performing vertical projection analysis on the text line image blocks to generate a character gap histogram; determining character segmentation boundaries according to the trough positions of the character gap histogram, which are used to divide independent character image regions; performing normalization scaling processing on the segmented character image regions to make all character images maintain a uniform size. 5.The deep learning-based bill text recognition method of claim 4, wherein, The character-by-character classification processing on the character-level image sequence using a deep character recognition model comprises: extracting local feature vectors of the character-level image sequence using a convolutional neural network, which contains stroke direction and structure distribution information; modeling the context dependency of character sequences through a bidirectional long short-term memory network to generate a character-level probability distribution matrix; decoding the probability distribution matrix combined with a connection time sequence classification algorithm to generate an initial text recognition result sequence. 6.The deep learning-based bill text recognition method of claim 5, wherein, The bill type knowledge base performs semantic verification and error correction processing on the initial text recognition result, including: Loading a corresponding field rule set from the knowledge base according to the bill type identifier, the field rule set including length constraints and character type restrictions; Performing rule matching detection on the initial text recognition result to mark abnormal fields that do not conform to the rules; Calculating the similarity between the abnormal fields and candidate correction items using an edit distance algorithm, and selecting the optimal correction result to replace the abnormal fields.

7. The deep learning-based bill text recognition method according to claim 6, characterized in that, The method further includes: Collecting bill recognition error samples in actual business scenarios within a preset period; Calculating the recognition bias features of the error samples, including a character confusion matrix and a field missing rate; Dynamically adjusting the classification weight parameters of the deep character recognition model according to the recognition bias features. 8.The deep learning-based bill text recognition method of claim 7, wherein, The method further includes: Performing key information extraction processing on the final structured text data to generate a bill element index table; Associating and mapping the bill element index table with a business database to verify the logical consistency of the data. 9.The deep learning-based bill text recognition method of claim 8, wherein, The method further includes: When detecting that there is a seal or handwritten annotation in the bill image, starting an additional content recognition process; Using an attention mechanism network to separate the seal area and the text area to avoid interference of the seal with the text recognition result. 10.A deep learning based bill text recognition system, characterized in that, A processor and a memory are included, the memory stores a computer program, and the processor executes the computer program to implement the deep learning-based bill text recognition method of any one of claims 1 to 9.

Images