A handwritten numeral string recognition method based on an improved residual network (ResNet)

Abstract

This invention relates to the fields of pattern recognition and artificial intelligence, specifically to a handwritten digit string recognition technology based on an improved residual network. The method includes: acquiring a digit image of the target and normalizing its size; then inputting the normalized image into a recognition model. The recognition model comprises four main modules: a feature extraction sub-network, a feature fusion and enhancement module, an adaptive attention module, and a classification decision module. In the feature fusion module, multi-scale features from different locations are fused; in the adaptive attention module, a two-level attention mechanism is used to improve recognition performance; and in the classification decision module, only a fully connected network is used to improve inference speed. This solution, through feature fusion, multi-level attention, and a fully connected classification network, effectively improves the performance and inference speed of digit string recognition.

Description

Technical Field

[0001] This invention relates to the fields of pattern recognition and artificial intelligence, specifically to a handwritten digit string recognition technology based on an improved residual network (ResNet), which is particularly suitable for scenarios involving efficient and accurate recognition of strings containing multiple handwritten digits, such as financial document digit string input, postal tracking number recognition, and interpretation of digit strings on educational examination answer sheets. Background Technology

[0002] In today's digital age, handwritten digit recognition has a wide range of applications across various industries. Traditional handwritten digit recognition methods, such as those combining template matching, feature engineering, and shallow machine learning models, often struggle to achieve ideal recognition accuracy and efficiency when faced with complex and diverse handwriting styles, character deformations, noise interference, and differences in character spacing and size.

[0003] With the rise of deep learning technology, Convolutional Neural Networks (CNNs) have achieved great success in image recognition due to their powerful feature extraction capabilities. ResNet, as a classic deep CNN architecture, effectively alleviates the vanishing and exploding gradient problems during deep network training by introducing residual blocks, enabling the network to learn image features more deeply and effectively. However, existing ResNet-based handwritten digit recognition schemes still have some limitations. For example, they do not delve deeply enough into the contextual semantic relationships between characters in the digit string, lack effective mechanisms for adaptively focusing on key character regions, and their recognition performance significantly degrades when processing long digit strings or complex handwriting styles.

[0004] The core objective of this invention is to provide an efficient and robust method and system for recognizing handwritten digit strings. By innovatively improving the traditional ResNet architecture, incorporating a multi-scale feature fusion strategy and an adaptive attention mechanism, it can more fully capture the local details and global semantic information of characters in the digit string, effectively handling various complex writing situations. Furthermore, the network structure abandons the serial mode of sequence generation, enabling more effective utilization of the hardware's parallel computing capabilities, thereby significantly improving the accuracy and efficiency of handwritten digit string recognition. Summary of the Invention

[0005] To address the issues of low efficiency and insufficient robustness in existing character recognition methods, this invention innovatively improves the traditional ResNet architecture by incorporating a multi-scale feature fusion strategy and an adaptive attention mechanism. This allows for a more comprehensive capture of local details and global semantic information of characters in digit strings, effectively handling various complex writing scenarios. Furthermore, the network structure abandons the serial mode of sequence generation, enabling more efficient utilization of the hardware's parallel computing capabilities, thereby significantly improving the recognition accuracy and efficiency of handwritten digit strings. To achieve the above objectives, the technical solution adopted by this invention is as follows:

[0006] Overall Network Architecture Overview

[0007] The handwritten digit recognition network ResNet_rec proposed in this invention is an end-to-end deep learning model, mainly composed of four key parts: a feature extraction sub-network, a feature fusion and enhancement module, an adaptive attention module, and a classification decision module. (Reference) Figure 1 The overall network consists of 5 steps, and the overall network structure is as follows: Figure 2 As shown.

[0008] (1) Feature extraction subnetwork

[0009] The feature extraction subnetwork is built on a multi-level residual block structure, aiming to extract feature information at different levels and scales from the input handwritten digit string image.

[0010] Initial Residual Layer (Layer 0): This layer takes a single-channel (1 input channel) image of a handwritten digit string as input and uses BasicBlock for feature extraction. The stride is set to (1,1) to ensure that as much original detail information as possible is preserved in the initial stage. The number of output feature channels of this layer is the pre-defined number of planes, laying the foundation for subsequent feature abstraction and downsampling operations.

[0011] Downsampling Residual Layers (Layer 1-Layer 4): From Layer 1 to Layer 4, each layer consists of multiple stacked basic residual blocks, with the number of feature channels increasing by a factor of 2 (planes*2, planes*4, planes*8, and planes*16 respectively). Furthermore, except for Layer 4, which has a stride of (2,1), the stride of each of the other layers is set to (2,2), achieving downsampling of the feature map in the spatial dimension. This design enables the network to extract features from digit strings at different scales, gradually abstracting more representative and discriminative feature representations from local stroke textures to global character structures.

[0012] (2) Feature fusion and enhancement module

[0013] This module mainly consists of a series of convolutional layers and batch normalization layers, which are used to further process and fuse the features output by the feature extraction subnetwork.

[0014] Conv 5 and Conv 6: These two convolutional layers both have a kernel size of 3, a stride of 1, and padding of 1. They aim to further enhance and refine the feature maps while maintaining their spatial dimensions. Following the convolutional operations are batch normalization layers (BN5 and BN6) to accelerate network convergence and improve model stability.

[0015] Conv 7 and Conv 9: These two layers use a special padding method (such as (0,1)) to reduce the size of the feature map in one dimension during convolution, thereby extracting more condensed and representative feature information. Similarly, batch normalization layers (BN7 and BN9) follow the convolution.

[0016] Conv 8: Continues to perform convolution processing on features, with the same kernel size, stride, and padding settings as Conv 5 and Conv 6, further enhancing the expressive power of features.

[0017] The feature maps output from Conv 5, Conv 7, and Conv 9 are concatenated along the height dimension to effectively fuse features at different scales and levels. This multi-scale feature fusion method can fully utilize the feature information extracted from different convolutional layers, improving the network's comprehensive ability to represent character features. A schematic diagram is shown below. Figure 2 As shown.

[0018] (3) Adaptive attention module

[0019] To better capture the contextual information and semantic relationships between characters in a numeric string, this invention introduces an adaptive attention mechanism. This module mainly consists of a feature transformation layer and an attention calculation layer.

[0020] Feature Transformation Layer: This layer reshapes the concatenated feature map output from the feature fusion and enhancement modules to fit the input requirements of the fully connected layers. Then, feature transformation is performed through two fully connected layers, featrans1 and featrans2, mapping the features to a new feature space. The ReLU activation function is used between the fully connected layers to increase the network's non-linear expressive power, enabling it to better learn the complex relationships between features.

[0021] Attention Calculation Layer: The transformed features are reshaped and input into the fully connected layers attfc1 and attfc2. attfc1 performs a linear transformation on the input features, followed by non-linear processing using the tanh activation function to introduce non-linear characteristics. attfc2 further transforms the processed features, outputting an attention matrix. The attention matrix is ​​then normalized in its last dimension using the Softmax function to obtain the attention weights for each position. These weights are used to perform a weighted summation of the features, thereby highlighting important feature information and suppressing irrelevant information, achieving adaptive focusing on key character regions.

[0022] (4) Classification Decision Module

[0023] The classification decision module is used to classify and predict the category of each character in the handwritten digit string. This module also uses a fully connected layer structure. This invention employs a two-level classification structure.

[0024] Intermediate classification stage: Features processed by the adaptive attention module are input into fully connected layers clafc1 and clafc2 for intermediate classification prediction. A ReLU activation function is used between the fully connected layers to increase the network's non-linearity. The intermediate classification results initially reflect the possible category of each character, but further processing is needed to enhance the modeling of dependencies between characters.

[0025] Final classification stage: The intermediate classification results are processed again by the attention module (composed of attfc3 and attfc4), which works on a similar principle to the previous attention calculation layer, further enhancing the modeling of dependencies between characters. Finally, the final classification result is output through the fully connected layers clafc3 and clafc4. The number of categories in the classification result is classnum, where classnum represents the number of digit categories (e.g., for digit recognition of 0-9, classnum is 11, with an additional category used to represent spaces or invalid characters).

[0026] The beneficial effects of this invention are:

[0027] This invention utilizes a multi-level residual block structure and multi-scale feature fusion technology to enable the network to comprehensively and deeply capture the local and global features of handwritten digit strings, effectively addressing issues such as different writing styles, character deformation, and noise interference, thereby significantly improving recognition accuracy. The introduction of an adaptive attention mechanism further enhances the network's ability to capture contextual information and semantic relationships between characters, allowing the model to recognize each character more accurately. Furthermore, the network is implemented using only convolutional and fully connected layers, which leverages the parallel computing capabilities of the hardware to improve the model's inference and recognition speed. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart of a digital string recognition method provided in an embodiment of the present invention;

[0030] Figure 2 This is an overall network architecture diagram provided in one embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of multi-scale feature fusion provided in an embodiment of the present invention; Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0033] The specific implementation of the above concept is described below.

[0034] (I) Data Preparation and Collection: A wide range of handwritten digit string image data were collected from various sources to ensure the data covered handwritten samples from different age groups, writing habits, and font styles, thereby increasing the diversity and representativeness of the data. Simultaneously, the data was labeled to clearly define the digit category corresponding to each character in each digit string.

[0035] Data preprocessing: The collected image data is preprocessed by converting the image to grayscale and normalizing it to an image with a height of 80. The width is scaled proportionally. The normalized image is then placed on an 80*1600 background image with all zeros. The pixel values ​​are divided by 255 and normalized to the [0,1] range to obtain the preprocessed image I.

[0036] Dataset partitioning: The preprocessed dataset is divided into training, validation, and test sets according to a certain ratio (e.g., 7:2:1). The training set is used for model parameter learning, the validation set is used to evaluate the model's performance during training, adjust hyperparameters, and prevent overfitting, and the test set is used to finally evaluate the model's generalization ability and recognition accuracy.

[0037] (II) Model Training

[0038] Model initialization: The ResNet_rec network is initialized using randomly initialized parameters. A suitable initialization method can be chosen, such as Xavier initialization or He initialization, to ensure that the network parameters have a reasonable distribution at the start of training, which is beneficial for network convergence.

[0039] Define the loss function and optimizer: Use either the multi-label classification loss function (MultiLabelSoftMarginLoss) or the cross-entropy loss function (CrossEntropyLoss) as the model's loss function to measure the difference between the model's predictions and the true labels. For the optimizer, choose commonly used optimization algorithms such as stochastic gradient descent (SGD) or Adam, and set appropriate hyperparameters such as learning rate, momentum, and weight decay. The learning rate can be dynamically adjusted, such as through learning rate decay, gradually reducing the learning rate during training to improve the model's convergence accuracy.

[0040] Training process: Training data is input into the model in batches, forward propagation is performed to calculate the output results, and the loss value is calculated according to the loss function. Then, the gradient is calculated using the backpropagation algorithm, and the optimizer is used to update the model parameters. During training, the model's performance is evaluated periodically using a validation set, and metrics such as accuracy and loss value on the validation set are recorded. Based on the performance on the validation set, the learning rate and other hyperparameters are adjusted to prevent overfitting.

[0041] (III) Model Evaluation

[0042] The trained model was comprehensively evaluated using a test set. Metrics such as recognition accuracy were calculated to assess the model's performance in recognizing different categories of digits. Simultaneously, the model's performance under varying handwriting styles, character spacing, and noise interference was analyzed to identify weaknesses and provide a basis for further optimization.

[0043] (iv) Practical Application

[0044] The trained model is then deployed to real-world application scenarios. For input handwritten digit string images, the same preprocessing operations as the training data are performed first, and then the preprocessed image is input into the model for recognition. The model outputs the classification result for each character, and the recognition result for the entire digit string is determined based on these results. During practical applications, new data is continuously collected to update and optimize the model online, adapting to constantly changing handwriting styles and application requirements.

[0045] It is understood that the embodiments of the present invention do not constitute a specific limitation on a handwritten digit string recognition method. In other embodiments of the present invention, a handwritten digit string recognition method may include more or fewer modules than illustrated, or combine some modules, split some modules, or arrange different modules. The illustrated modules may be implemented in hardware, software, or a combination of software and hardware.

[0046] The information interaction and execution process between the modules in the above-mentioned device are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description in the method embodiment of the present invention, and will not be repeated here.

[0047] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0048] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0049] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for handwritten digit string recognition based on improved Residual Network (ResNet), characterized in that, Includes the following steps: (1) Preprocessing steps: After grayscale conversion of the image to be recognized, normalize it to an image with a height of 80 and scale the width proportionally. Then, place the normalized image on an 80*1600 background image with all zeros and divide the pixel values ​​by 255 to normalize them to the [0,1] range to obtain the preprocessed image I. (2) Feature extraction steps: Image I is used as network input and passed through residual layer group (Layer 0), each layer is composed of multiple basic residual blocks stacked together, with a stride of (1,1) to extract initial features. The number of output feature channels is the number of planes set in advance. Then, it is passed through downsampling residual layer group (Layer 1-Layer 4), each layer is composed of multiple basic residual blocks stacked together, with the number of feature channels increasing by a factor of 2. The stride of Layer 1-Layer 3 is set to (2,2), and the stride of Layer 4 is set to (2,1) to realize downsampling of feature map and extraction of features at different scales. (3) Feature fusion and enhancement steps: The features output from the feature extraction step are processed by a sequence of convolutional layers (Conv 5-Conv 9). The kernel size of Conv 5, Conv 6, and Conv 8 is 3, the stride is 1, and the padding is 1. Conv 7 and Conv 9 use a special padding method (such as (0,1)). Each convolutional layer is followed by a batch normalization layer. The feature maps output by Conv 5, Conv 7, and Conv 9 are spliced ​​in a specific dimension to achieve multi-scale feature fusion. (4) Adaptive attention calculation steps: The shape of the stitched feature map output from the feature fusion and enhancement steps is adjusted to fit the input requirements of the fully connected layer. The adjusted features are then input to the fully connected layer featrans1 for linear transformation. The input dimension of this fully connected layer is the adjusted feature dimension of the stitched feature map, and the output dimension is planes*64. A ReLU activation function is applied to the output of featrans1 to introduce non-linearity, enhancing the network's ability to learn complex feature relationships. The result of the ReLU activation is then input to the fully connected layer featrans2, which also has a planes*64 input and output dimension, and undergoes another linear transformation to obtain the transformed feature representation. The transformed features obtained from the feature transformation operation are reshaped into a one-dimensional vector. This one-dimensional vector is then input into a fully connected layer attfc1. The input dimension of attfc1 is the product of planes*64 and the maximum convolution length max_conv_len, and the output dimension is max_conv_len. The tanh activation function is applied to the output of attfc1, mapping its value to the interval [-1,1], introducing non-linearity to better model the complex relationships between features. The result after tanh activation is then input into a fully connected layer attfc2. The input dimension of attfc2 is max_conv_len, and the output dimension is the product of max_conv_len and max_conv_len. The output of attfc2 is reshaped into a matrix of shape (batch_size, max_conv_len, max_conv_len) to obtain the attention matrix. The attention matrix is ​​then normalized by applying the Softmax function to the last dimension, ensuring the sum of elements in each row is 1, thus obtaining the attention weight for each position. The normalized attention matrix is ​​then used to perform a weighted summation of the transformed features obtained from the feature transformation operation; that is, matrix multiplication is used to multiply the attention matrix with the transformed features. This process highlights important feature information and suppresses irrelevant information, achieving adaptive focusing on key character regions. (5) Classification decision step: Input the features processed by the adaptive attention calculation step into the fully connected layers clafc1 and clafc2 for intermediate classification prediction, and use the ReLU activation function between layers; The intermediate classification results are processed again by the attention module consisting of attfc3 and attfc4, and the processing method is similar to the adaptive attention calculation steps. The final classification result is output through the fully connected layers clafc3 and clafc4. The number of categories in the classification result is classnum, where classnum represents the number of categories of digits.

2. The method of claim 1, wherein The Basic Block consists of two convolutional layers and a shortcut connection. When the stride is not 1 or the number of input channels does not match the number of output channels, the shortcut connection is adjusted using convolutional layers and batch normalization layers.

3. The method of claim 1, wherein In the feature extraction step, Layer 1 has 'planes' as its input feature channels and 'planes*2' as its output feature channels; Layer 2 has 'planes*2' as its input feature channels and 'planes*4' as its output feature channels; Layer 3 has 'planes*4' as its input feature channels and 'planes*8' as its output feature channels; and Layer 4 has 'planes*8' as its input feature channels and 'planes*16' as its output feature channels.

4. The method of claim 1, wherein In the feature fusion and enhancement step, the role of Conv 5 and Conv 6 is to enhance and refine the feature map, keeping the spatial size of the feature map unchanged; the role of Conv 7 and Conv 9 is to reduce the size of the feature map in one dimension and extract more representative features.

5. The method of claim 1, wherein In the attention processing of the adaptive attention calculation step and the classification decision step, the tanh activation function is used to introduce nonlinear characteristics, and the Softmax function is used to normalize the attention matrix to obtain the attention weight of each position.

Images