Manchu printed form document identification method and system based on Swin-Transform
Through the encoder-decoder structure and feature extraction and analysis module based on Swin-Transformer, the accuracy problem of character segmentation and reorganization in Manchu recognition is solved, and efficient recognition and accurate conversion of Manchu printed documents are achieved.
Patent Information
- Application Number
- CN202510954574.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-11
- Publication Date
- 2025-09-23
AI Technical Summary
Existing Manchu recognition technology has deficiencies in the accuracy of character segmentation and reorganization, and existing models have limitations in image detail extraction and semantic sequence capture. In particular, there is a lack of effective whole-word recognition strategies in printed Manchu recognition.
An encoder-decoder structure based on Swin-Transformer is adopted. Through local window division and self-attention calculation, the decoder process is optimized in combination with the Feature Extraction and Analysis Module (FEAM) to achieve efficient recognition of Manchu word group features.
The accuracy and robustness of Manchu printed document recognition have been improved, and it can effectively convert printed document images into editable text, making it suitable for complex and long text recognition tasks.
Smart Images
Figure CN120689887A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of optical character recognition, and in particular to a method and system for recognizing Manchu printed documents based on Swin-Transformer. Background Art
[0002] Existing Manchu recognition technologies generally rely on character segmentation strategies, using projection or stroke growth methods to divide Manchu images into basic units—characters or strokes—and then applying simple classification models for recognition. This approach aims to significantly reduce the amount of training data required while improving recognition efficiency. However, the complex vocabulary of Manchu still limits the accuracy of character segmentation, and currently no mature technology exists to effectively reconstruct characters.
[0003] The current leading technologies include C-RNN-based whole-word recognition strategies and Transformer-based whole-word recognition strategies. C-RNN-based systems and Transformer-based architectures have been successfully deployed in printed text recognition tasks in multiple languages and have demonstrated significant recognition performance. However, there is a lack of similar research in the field of printed Manchu recognition. The whole-word model based on C-RNN and Transformer is essentially an encoder-decoder architecture model. C-RNN performs well in extracting image features, but its performance is inferior to Transformer in capturing semantic sequences. Although the Transformer decoder excels at processing sequential data, it has limitations in effectively extracting image details. Vision-Transformer introduced the Transformer architecture to the image field. However, this method adopts a global computation strategy when executing the attention mechanism, fails to effectively identify the correlation between local image features, and imposes a huge computational burden. Summary of the Invention
[0004] The present invention aims to overcome the shortcomings of the prior art and provide a method for recognizing printed Manchu documents based on Swin-Transformer, comprising the following steps:
[0005] Step 1: Input the Manchu word image and, after preprocessing, segment the image into non-overlapping image blocks; perform linear mapping on each image block to obtain a tensor of fixed feature dimension C; add a position code PE to the tensor of each image block;
[0006] Step 2: Input the character feature encoding and position encoding PE after preprocessing Manchu segmentation into the Swin-Transformer encoder, perform local window division and self-attention calculation, divide the input features into multiple windows of size M×M, and calculate the window self-attention in each window. Move the window between two consecutive layers to obtain cross-window information; through hierarchical design, gradually reduce the resolution of the feature map and increase the channel dimension. The output feature map size changes at each stage, and the image is changed from H×W×C to H / 32×W / 32×8C to obtain the encoded feature tensor;
[0007] Step 3: Input the feature tensor corresponding to the character sequence output by the encoder into the decoder. The model decodes based on the feature vector F output by the Swin-Transformer encoder and the historical input of the decoder. The feature extraction and analysis module detects prediction errors and generates loss feedback for prediction correction to obtain the predicted character sequence.
[0008] Furthermore, the input Manchu word image is preprocessed to segment the image into non-overlapping image blocks; each image block is linearly mapped to obtain a tensor of fixed feature dimension C, including:
[0009] Assume that the input image size is H×W and the size of each image block is P×P, then the number of image blocks is:
[0010]
[0011] Each image patch is linearly mapped to a tensor E of fixed feature dimension C i :
[0012] E i =W P ·P i +b P , E i ∈R C .
[0013] Furthermore, the adding of position encoding PE to the tensor of each image block includes:
[0014] In each patch tensor E input Add position code PE:
[0015] E input =E+PE
[0016] The position code PE is:
[0017]
[0018] Where pos is the position index of the patch, i is the dimension index, and d is the embedding dimension.
[0019] Furthermore, the aforementioned calculation of window self-attention within each window, moving the window between two consecutive layers, and obtaining cross-window information includes:
[0020] The input features are divided into windows of size M×M, and the total number of windows is:
[0021]
[0022] Calculate self-attention for the features within each window:
[0023]
[0024] Among them, Q, K, and V are obtained by linear transformation of features within the window, d k is the attention head dimension; a moving window mechanism is adopted, and adjacent layer windows are moved by M / 2 pixels, and cross-window information interaction is achieved through circular shift operations.
[0025] Furthermore, the feature extraction and analysis module is used to detect prediction errors and generate loss feedback for prediction correction, including:
[0026] At each time step t, the first t characters of the current prediction and the real characters are converted into images, and the predicted image and the real image are generated through the Render function; the features F of the predicted image and the real image are extracted through CNN pred =CNN(I pred ), F true =CNN(I true ), calculate the difference between the predicted features and the true features, and generate the loss L diff :
[0027] L diff =MSE(F pred , F true )
[0028] L diff Feedback is sent to the decoder to update the parameters and re-predict the character at the current time step.
[0029] A Manchu printed document recognition system based on Swin-Transformer, which applies the Manchu printed document recognition method based on Swin-Transformer, includes a data preprocessing module, a Swin Transformer encoding module, and a decoder module;
[0030] The data preprocessing module is used to receive a Manchu word image and segment the image into non-overlapping image blocks; perform linear mapping on each image block to obtain a tensor with a fixed feature dimension C; introduce position encoding and add a position encoding PE to the tensor of each image block;
[0031] The Swin Transformer encoding module is used to receive the character feature encoding and position encoding PE after Manchu text segmentation output by the data preprocessing module; perform local window division and self-attention calculation, divide the input features into multiple windows (windows) of size M×M, and calculate window self-attention within each window; move the window between two consecutive layers to capture cross-window information; through hierarchical design, gradually reduce the resolution of the feature map while increasing the channel dimension to obtain the encoded tensor;
[0032] The decoder module is used to receive the feature tensor corresponding to the character sequence output by the Swin Transformer encoding module; during the decoding process, decoding is performed based on the feature vector F output by the Swin-Transformer encoder and the decoder history input;
[0033] A feature extraction and analysis module (FEAM) is introduced to detect prediction errors and generate loss feedback to correct the prediction and obtain the final predicted character sequence.
[0034] The present invention addresses the unique characteristics of Manchu word formation and proposes a model called Swin-Manchu OCR. This model is a Manchu optical character recognition model tailored to these characteristics and can effectively convert printed document images into editable text. This unique characteristic of Manchu word formation results in low accuracy rates for current mainstream recognition models. The Swin-Manchu OCR model, designed based on current whole-word recognition techniques, employs a typical encoder-decoder architecture, using the Swin-Transformer as the decoder to better extract image details.
[0035] The decoder of the Swin-Manchu OCR model incorporates a module specifically designed to optimize feature extraction and analysis, FEAM. FEAM measures the deviation between predicted output and actual value, generating an additional loss feedback signal to optimize the decoder's decoding process and improve the accuracy of Swin-Manchu OCR in character sequence prediction. FEAM improves decoder decoding performance and increases prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 The figure is a flowchart of the Manchu printed document recognition method based on Swin-Transformer;
[0037] Figure 2 Schematic diagram of the internal structure of adjacent Swin-Transformer blocks;
[0038] Figure 3 Schematic diagram of the Swin-Transformer hierarchical structure;
[0039] Figure 4 Schematic diagram of the size change of the image feature map;
[0040] Figure 5 This is a schematic diagram of the principle of the feature extraction and analysis module;
[0041] Figure 6 Schematic diagram of encoding real values as tensors. DETAILED DESCRIPTION
[0042] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the following.
[0043] The features and performance of the present invention are further described in detail below with reference to the embodiments.
[0044] like Figure 1 As shown in FIG, the Manchu printed document recognition method based on Swin-Transformer includes the following steps:
[0045] Step 1: Input the Manchu word image and, after preprocessing, segment the image into non-overlapping image blocks; perform linear mapping on each image block to obtain a tensor of fixed feature dimension C; add a position code PE to the tensor of each image block;
[0046] Step 2: Input the character feature encoding and position encoding PE after preprocessing Manchu segmentation into the Swin-Transformer encoder, perform local window division and self-attention calculation, divide the input features into multiple windows of size M×M, and calculate the window self-attention in each window. Move the window between two consecutive layers to obtain cross-window information; through hierarchical design, gradually reduce the resolution of the feature map and increase the channel dimension. The output feature map size changes at each stage, and the image is changed from H×W×C to H / 32×W / 32×8C to obtain the encoded feature tensor;
[0047] In step three, the feature tensor corresponding to the character sequence output by the encoder is input into the decoder. The model decodes the character sequence based on the feature vector F output by the Swin-Transformer encoder and the decoder's historical input. The feature extraction and analysis module detects prediction errors and generates loss feedback to correct the prediction, resulting in a predicted character sequence.
[0048] Specifically, the application proposes a new model architecture for Manchu OCR (Optical Character Recognition) tasks. This architecture uses a Swin-Transformer encoder and incorporates a feature extraction and analysis module into the decoder to implement Manchu character encoding. This results in a Swin-Transformer-based recognition model. The architecture includes the following steps:
[0049] 1. Data Preprocessing
[0050] Input: Manchu word image
[0051] Output: Image after full character segmentation and its feature vector E i ,i=1,…,N
[0052] Method: The input image is preprocessed and divided into a series of non-overlapping image patches. Assuming the input image size is H×W and the size of each image patch is P×P, the total number of patches is 3-1:
[0053]
[0054] Each image patch is linearly mapped to a tensor E of fixed feature dimension C i , as shown in formula 3-2:
[0055] E i =W P ·P i +b P , E i ∈R C (3)
[0056] Similar to the standard Transformer, position encoding (Position Embedding) is still introduced here. In order to make the model understand the relative position of the patch, the tensor E of each patch is input Adding the position code PE, the calculation result is formula 3-3:
[0057]
[0058] The calculation formula for the position encoding PE is 3-4, where pos is the position index of the patch, i is the dimension index, and d is the embedding dimension.
[0059]
[0060] 2.Swin Transformer encoding
[0061] Input: Character feature code E after Manchu segmentation i ,i=1,…,N; position code PE
[0062] Output: Encoded tensor
[0063] Method: Unlike the global self-attention mechanism in the standard Transformer, the Swin-Transformer adopts a window self-attention mechanism. It reduces computational complexity by dividing the local window and introduces a hierarchical structure to gradually aggregate contextual information. The encoding process of the entire Swin-Transformer encoder is as follows:
[0064] First, the local window division and self-attention (Window Self-Attention) input feature E input It is divided into multiple M×M windows. The patch in each window will calculate the window self-attention instead of the global self-attention. The window size is M, and the total number of windows is Equation 3-5:
[0065]
[0066] For each window W i , we calculate Self-Attention within the window, the calculation formula is 3-6, similar to the self-attention in the standard Transformer: where Q, K, V are obtained by linearly transforming the feature vector of the patch within the window. k is the dimension of the attention head.
[0067]
[0068] Window self-attention is only calculated in a local range, which significantly reduces the computational complexity from O(N 2 ) is reduced to Where W is the window size. The local window ensures that the attention calculation is concentrated on the local area of the character, reducing the amount of calculation and facilitating the capture of details such as character edges and strokes.
[0069] Shifted Window Attention: In order to allow the model to capture cross-window information, Swin-Transformer introduces a shifted window mechanism between two consecutive layers. Specifically, the first layer is divided into fixed windows with a size of M×M. The l+1 layer shifts the window by The distance between the windows is re-divided. Through this alternating moving window operation, the information between windows can be effectively transferred, allowing the model to capture a wider range of context information. The specific structure diagrams of the two are as follows Figure 2 As shown:
[0070] Assume The feature map output by the previous Swin-Transformer block, after passing through two consecutive Swin-Transformer blocks, the result is z l+1 , and z l+1 The calculation follows formula (8)
[0071]
[0072] For two adjacent blocks, conventional window partitioning uses W-MSA, which is essentially a multi-head attention layer that calculates attention within each window. The other uses SW-MSA. By partitioning the windows, attention can be calculated between different windows, thus better capturing the connections between windows, which is also crucial for the prediction process. Window movement breaks the limitations of fixed partitioning and enables information exchange between windows. This helps model global relationships between characters.
[0073] Finally, the hierarchical feature representation is as follows Figure 3 As shown in Figure 2, Swin-Transformer gradually reduces the resolution of feature maps through a hierarchical design while increasing the channel dimension. The output feature map size changes at each stage are as follows:
[0074] Stage 1: Input size H×W is downsampled to Extract fine-grained edge features of characters.
[0075] Stage 2: Further downsampling to Capture the local combination features of characters.
[0076] Stage 3: Then downsample to Learn the overall semantic information of words.
[0077] Stage 4: The final downsampling is Extract global context information.
[0078] After four stages of attention calculation, the image changes from H×W×C to H / 32×W / 32×8C. The dimension of the feature map changes during the entire decoding process as follows: Figure 4 There are four types of Swin-Transformer as shown in Table 3-1.
[0079] Multi-scale feature extraction enables the model to balance character-level details with word-level global representations. Downsampling effectively reduces the data size, thereby reducing computational resource requirements. This process also helps extract more generalizable high-level features, improving the model's generalization capabilities. This gradual downsampling allows the model to capture multi-scale contextual features, applicable to characters of varying sizes and complexities. This improved decoder is therefore suitable for Manchu font recognition.
[0080] Table 3-1 Swin-Transformer Types
[0081]
[0082] 3. Decoder
[0083] Input: Feature tensor corresponding to the encoder output character sequence
[0084] Output: Predicted character sequence Output: Predicted character sequence
[0085] Assume that the Manchu character sequence is: True sequence Y = (y1, y2, y3) The decoder output of the model predicts the character sequence in: The prediction is correct and equal to y1 and y2. The prediction error is not equal to y3. The feature extraction and analysis module is used to detect the error, generate loss feedback, and correct it. In the step-by-step decoding process, the model is based on the feature vector output by the Swin-Transformer encoder. The quantity F and the decoder history input, the decoder decoding process is as follows:
[0086] (1) Time step t = 1: Prediction input: start token <start>Output: The probability distribution is as shown in Equation 9, and the predicted word is output through Softmax. Result: Assuming that the prediction is correct, that is
[0087] P(y1)=Softmax(FFN(Attention(F, <start>))) (9)
[0088] (2) Time step t = 2: prediction Input: the output at the previous moment, Output: Probability distribution, predicted characters output by Softmax Result: Assuming the prediction is correct, that is The calculation formula is 10.
[0089]
[0090] (3) Time step t = 3: prediction Input: Output at the previous moment Output: Probability distribution as in Equation 11, predicted characters are output through Softmax Result: Assume that the prediction is wrong, i.e.
[0091]
[0092] From the above steps, we can see that the error detection and correction is performed at time step t=3, when the feature extraction and analysis module intervenes. The process is as follows:
[0093] (1) Constructing predicted images and real images
[0094] Convert the first three predicted characters and real characters output by the current decoder into images respectively: predicted character sequence Generate predicted image I pred . The real character sequence Y 1:3 =(y1,y2,y3), generate real image I true These images can be concatenated or rendered into a fixed format, such as 3-11:
[0095]
[0096] (2) Input Feature Extraction and Analysis Module (FEAM)
[0097] The predicted image I pred and real image I true Input to the feature extraction and analysis module. This module uses convolutional neural network (CNN) to extract the features of the two images, and the calculation results are:
[0098] F pred =CNN(I pred ), F true =CNN(I true ) (13)
[0099] Calculate the difference between the predicted features and the true features to generate the loss L diff . This can be calculated using the mean squared error (MSE):
[0100] L diff =MSE(F pred ,F true ) (14)
[0101] After receiving the loss feedback, the decoder updates its parameters and re-predicts the character at time step t=3. Make it closer to the true value y3.
[0102] This feature extraction and analysis module effectively captures the fine-grained differences between predicted and real characters, and optimizes the decoder's predictions through loss feedback, ultimately improving the recognition accuracy and robustness of the OCR model. In practical applications, this module is particularly well-suited for recognizing complex or long text, significantly reducing cumulative error and improving final output quality.
[0103] 4. Model loss function
[0104] In actual training, the input true value label will be converted into a tensor through encoding, such as Figure 6 As shown, comparing these tensors with the tensors predicted by the decoder results in a loss. Based on the detailed introduction to the encoding and decoding parts of the encoder-decoder, combined with the theoretical analysis of the Transformer model in Chapter 2, the loss function for the Transformer model is typically the cross-entropy loss function. This is a common loss function for classification tasks and is suitable for word prediction during language model generation. Below, I will detail the input and output of this loss function in the Transformer model.
[0105]
[0106] For the Manchu recognition task, N is the total number of Manchu characters in a word, y i is the true label of each character in the Manchu word sequence, p i It is the model's predicted probability for each character. For each time step, the cross entropy loss will calculate the loss value based on the degree of match between the model's predicted probability and the actual character, and sum the losses of all time steps to obtain the overall loss of the full character sequence. This application adds a feature extraction module based on the original decoder, so the character with the highest probability will be output at each time step, but this character may not be consistent with the actual character. The feature extraction module compares at the pixel level. If the predicted character is consistent with the actual value, the gap between pixels will be small. If it is inconsistent, there will be a gap. This gap between pixels is extracted and compared by the feature extraction module and represented by L1 loss. Since the character predicted at each time step will incur a loss, these losses are summed up, y i is the true value of the character, Predict values for each time step of the model.
[0107]
[0108] Through the analysis of the above process loss function, the final loss function formula of the model is obtained:
[0109] L total =(1-α)L cross +αL img (17).< / start> < / start>
Claims
1. A Manchu printed document recognition method based on Swin-Transformer, characterized by: The steps include: Step 1: Input the Manchu word image and, after preprocessing, segment the image into non-overlapping image blocks; perform linear mapping on each image block to obtain a tensor of fixed feature dimension C; add a position code PE to the tensor of each image block; Step 2: Input the character feature encoding and position encoding PE after preprocessing Manchu segmentation into the Swin-Transformer encoder, perform local window division and self-attention calculation, divide the input features into multiple windows of size M×M, and calculate the window self-attention in each window. Move the window between two consecutive layers to obtain cross-window information; through hierarchical design, gradually reduce the resolution of the feature map and increase the channel dimension. The output feature map size changes at each stage, and the image is changed from H×W×C to H / 32×W / 32×8C to obtain the encoded feature tensor; Step 3: The feature tensor corresponding to the character sequence output by the encoder is input into the decoder. The model decodes based on the feature vector F output by the Swin-Transformer encoder and the decoder history input; The feature extraction and analysis module detects prediction errors and generates loss feedback to correct the prediction and obtain the predicted character sequence.
2. The method for recognizing printed Manchu documents based on Swin-Transformer according to claim 1, characterized in that: The input Manchu word image is pre-processed to segment the image into non-overlapping image blocks; Perform linear mapping on each image block to obtain a tensor of fixed feature dimension C, including: Assume that the input image size is H×W and the size of each image block is P×P, then the number of image blocks is: Each image patch is linearly mapped to a tensor E of fixed feature dimension C i : HAVE BEEN i =W P ·P i +b P ,HAVE BEEN i ∈R C .
3. The method for recognizing printed Manchu documents based on Swin-Transformer according to claim 2, characterized in that: The position encoding PE is added to the tensor of each image block, including: In the tensor E of each image block input Add position code PE: It is input =E+PE The position code PE is: Where pos is the position index of the image block, i is the dimension index, and d is the Embedding dimension.
4. The method for recognizing printed Manchu documents based on Swin-Transformer according to claim 1, characterized in that: The window self-attention is calculated within each window, and the window is moved between two consecutive layers to obtain cross-window information, including: The input features are divided into windows of size M×M, and the total number of windows is: Calculate self-attention for the features within each window: Among them, Q, K, and V are obtained by linear transformation of features within the window, d k is the attention head dimension; a moving window mechanism is adopted, and adjacent layer windows are moved by M / 2 pixels, and cross-window information interaction is achieved through circular shift operations.
5. The method for recognizing printed Manchu documents based on Swin-Transformer according to claim 1, characterized in that: The feature extraction and analysis module is used to detect prediction errors and generate loss feedback for prediction correction, including: At each time step t, the first t characters of the current prediction and the real characters are converted into images, and the predicted image and the real image are generated through the Render function; the features F of the predicted image and the real image are extracted through CNN pred =CNN(I pred ), F true =CNN(I true ), calculate the difference between the predicted features and the true features, and generate the loss L diff : L diff =MSE(F pred ,F true ) L diff Feedback is sent to the decoder to update the parameters and re-predict the character at the current time step.
6. The Manchu printed document recognition system based on Swin-Transformer is characterized by: The method for recognizing Manchu printed documents based on Swin-Transformer according to any one of claims 1 to 5 is applied, comprising a data preprocessing module, a Swin Transformer encoding module, and a decoder module; The data preprocessing module is used to receive a Manchu word image and segment the image into non-overlapping image blocks; perform linear mapping on each image block to obtain a tensor with a fixed feature dimension C; introduce position encoding and add a position encoding PE to the tensor of each image block; The Swin Transformer encoding module is used to receive the character feature encoding and position encoding PE after Manchu text segmentation output by the data preprocessing module; perform local window division and self-attention calculation, divide the input features into multiple windows (windows) of size M×M, and calculate window self-attention within each window; move the window between two consecutive layers to capture cross-window information; through hierarchical design, gradually reduce the resolution of the feature map while increasing the channel dimension to obtain the encoded tensor; The decoder module is used to receive the feature tensor corresponding to the character sequence output by the Swin Transformer encoding module; during the decoding process, decoding is performed based on the feature vector F output by the Swin-Transformer encoder and the decoder history input; A feature extraction and analysis module (FEAM) is introduced to detect prediction errors and generate loss feedback to correct the prediction and obtain the final predicted character sequence.