A small seal script recognition method based on Swin Transformer

Abstract

The application discloses a small seal character recognition method based on SwinTransformer. Small seal character pictures and corresponding traditional Chinese character pictures are selected as inputs of a model, and the small seal character pictures after preprocessing are first sent to an encoder E based on SwinTransformer to obtain global features of the pictures; then, a parallel three-branch structure is adopted by the network, i.e., a traditional Chinese character picture generation branch, a small seal character picture generation branch and a small seal character prediction branch. The Swin-Transformer module is used to replace the encoder of a traditional recognition model, and the W-MSA mechanism and the SW-MSA mechanism of the Swin-Transformer improve the global feature extraction capability of the pictures, so that the module can complete the high-difficulty task of small seal character recognition, and can realize the function of converting the small seal character pictures into traditional Chinese character pictures.

Description

Technical Field

[0001] This invention relates to the fields of image and text recognition and deep learning technology, specifically to a method for recognizing Small Seal script based on Swing Transformer. Background Technology

[0002] Chinese, as one of the oldest writing systems in the world, has a history of over six thousand years. As the carrier of Chinese culture, Chinese characters are relied upon to study ancient Chinese literature and the migration and transmission of culture. The study of ancient Chinese characters is of great significance to the exploration of ancient Chinese history and has therefore always been a very important research direction. In the continuous evolution of ancient Chinese characters, Small Seal Script, as a product of the standardization of weights and measures by Qin Shi Huang, weakened its pictographic nature and became more symbolic, reducing confusion and difficulty in writing and reading, and playing an important role in the development of modern Chinese. Even today, Small Seal Script is frequently found in paintings, calligraphy, seals, company trademarks, and posters. However, because Small Seal Script differs significantly in form from the simplified characters we commonly use today, most people without professional knowledge of ancient Chinese characters often lack the ability to recognize Small Seal Script. Therefore, we need a method to identify Small Seal Script.

[0003] Currently, there are few image-text recognition technologies for Small Seal script, and the accuracy rate is relatively low. The main reasons are: 1. Small Seal script is rich in pictographic and symbolic information and has diverse styles. Different calligraphers from different dynasties have different writing styles for Small Seal script, and even the same character may have significant differences in shape, resulting in insufficient recognition accuracy. Most existing Small Seal script recognition technologies directly classify the feature vectors after extracting image feature information to obtain the recognition result, which wastes the rich structural information contained in Small Seal script, making Small Seal script recognition difficult; 2. Most existing Small Seal script and other ancient Chinese character recognition technologies are based on Convolutional Neural Networks (CNN) or Recurrent Neural Networks (RNN). Pooling operations in CNNs can lose some valuable information in the image and ignore the correlation between the local and the whole. As the number of network layers increases, RNNs will suffer from gradient vanishing or gradient exploding when processing long sequence scenes. Past network structures have limited ability to extract information from images, resulting in low accuracy when performing difficult tasks such as Small Seal script recognition.

[0004] The Transformer structure can acquire global information without stacking, unlike CNNs which are limited by their receptive field. It also overcomes the limitation of RNN models not being able to perform parallel computation, possessing excellent feature extraction capabilities. Recently, the Transformer has appeared with increasing frequency in the field of computer vision. The Swin Transformer increases the interaction between windows by using window self-attention (W-MSA) and moving window self-attention (SW-MSA) mechanisms, greatly improving the Transformer's ability to extract global features from images. If a Small Seal Script recognition network can be built using the Swin Transformer structure, its excellent feature extraction capabilities can significantly improve the accuracy of Small Seal Script recognition tasks. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a Small Seal Script recognition method based on Swing Transformer. This method fully utilizes the highly similar and rich structural information contained between Small Seal Script and Traditional Chinese characters. It designs a network structure that simultaneously recognizes Small Seal Script and converts Small Seal Script images into Traditional Chinese character images. Small Seal Script images and their corresponding Traditional Chinese character images are selected as inputs to the model. After preprocessing, the Small Seal Script images are first fed into an encoder E built on Swing Transformer to encode global image features. Subsequently, the network adopts a parallel three-branch structure, with the Traditional Chinese character image generation branch using a Traditional Chinese character image generation decoder G built on Swing Transformer. t The global features of the image are decoded into corresponding traditional Chinese character images, and then a traditional Chinese character image discriminator D is used. t The image generation effect is evaluated. The Small Seal Script image generation branch adopts the same network structure as the Traditional Chinese Script image generation branch, decoding the global features of the image into the corresponding Small Seal Script image and performing discrimination, thereby verifying and improving the feature extraction effect of the decoder. Using the same method, the Traditional Chinese Script image corresponding to the Small Seal Script image is fed into E to generate the global features of the image, and then fed into the Traditional Chinese Script image generation branch and the Small Seal Script image generation branch respectively to generate the corresponding Traditional Chinese Script image and Small Seal Script image, thereby constraining the encoding effect of E and the image generation effect of the Traditional Chinese Script image generation branch and the Small Seal Script image generation branch. The Small Seal Script prediction branch uses a Small Seal Script prediction classifier E built based on SwingTransformer. r The system classifies the global features of the Small Seal script image to obtain the Small Seal script recognition result, thereby completing the Small Seal script recognition task with high accuracy. It can also realize the Small Seal script-to-Traditional Chinese character conversion function, converting the Small Seal script image into the corresponding Traditional Chinese character image.

[0006] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0007] Step S1: Construct a Small Seal Script image training set X s The test set, and the traditional Chinese character image X corresponding to the small seal script image. t And perform preprocessing;

[0008] Step S2: Divide the Small Seal image from S1 into image blocks and perform embedding encoding, then feed them into E built based on SwinTransformer to obtain the global image features F of the Small Seal image. s ;

[0009] Step S3: Extract the global image features F from step S2. s The traditional Chinese image generation branch is used, employing the traditional Chinese image generation decoder G built based on SwinTransformer. t Decode the Small Seal script features into the corresponding Traditional Chinese character image Y st And use the Traditional Chinese image discriminator D t Judge the image generation effect;

[0010] Step S4: Extract the global image features F from step S2. s The small seal image generation branch is fed in, and the small seal image generation decoder G, built based on SwinTransformer, is used. s Decode the Small Seal script features into the corresponding Small Seal script image Y. ss And use the Small Seal Script image discriminator D s Judge the image generation effect;

[0011] Step S5: Select the corresponding traditional Chinese character image mentioned in S1 as the network input, and repeat steps S2 to S4 to decode the corresponding traditional Chinese character image into global features F. t The branch generated by the traditional Chinese image will be the F t Decoded into Traditional Chinese character Y tt The image is analyzed and branch F is generated from the Small Seal script image. t Decoded into Small Seal Script image Y ts The decoding effect of the decoder and the image generation effect of the traditional Chinese character image generation branch and the traditional Chinese character image generation branch are verified by using the structural information that is very similar between the small seal script and the traditional Chinese character.

[0012] Step S6: Extract the global features F of the Small Seal Script image from S2. s The data is fed into the Small Seal Script prediction branch, and a Small Seal Script prediction classifier E built based on SwinTransformer is used. r For F s The classification process yields the final prediction result Y for Small Seal script. l ;

[0013] Step S7: Calculate the model loss function L, which includes the encoder loss L of E. F Image generation loss L for Traditional Chinese character image generation branch and Small Seal script image generation branch I The prediction loss L of the Small Seal script prediction branch C The network is trained via backpropagation;

[0014] Step S8: Use the Small Seal Script Recognition Task Test Set to verify the recognition accuracy of the Small Seal Script Recognition Task.

[0015] The beneficial effects of this invention are as follows:

[0016] This invention utilizes the structural similarity between Small Seal script and Traditional Chinese characters, and uses the corresponding Traditional Chinese character images as input to the model. This achieves the effect of global feature decoding of auxiliary constraint E and expands the dataset, thereby improving the image generation quality of both the Traditional Chinese image generation branch and the Small Seal script image generation branch.

[0017] Based on the structural similarity between Small Seal script and Traditional Chinese characters, this invention adds two additional branches for generating images: one for Traditional Chinese characters and one for generating images of Small Seal script. The Traditional Chinese character image generation branch can constrain the Small Seal script feature F through the conversion between Traditional and Small Seal scripts. s Features of Traditional Chinese Characters F t To achieve consistency, thereby improving E's ability to encode image style information, the small seal image generation branch generates Y through constraints. ts and Y ss , and the input image X s Consistency further constrains the encoding capability of E. All three branches are indispensable and mutually constraining, ultimately enabling the model to achieve high accuracy in Small Seal script recognition.

[0018] This invention proposes a novel Small Seal Script recognition model based on Swin-Transformer. The Swin-Transformer module replaces the CNN or RNN module in the encoder or decoder of the traditional recognition model. The W-MSA and SW-MSA mechanisms of Swin-Transformer improve the global feature extraction capability of the image, enabling the module to complete the difficult task of Small Seal Script recognition and to convert Small Seal Script images into Traditional Chinese character images. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the Swin-Transformer-based Small Seal Script recognition network model structure;

[0020] Figure 2 This is a schematic diagram of the main steps of the present invention;

[0021] Figure 3 This is a schematic diagram of the network model structure of encoder E;

[0022] Figure 4 A schematic diagram of the network model structure for generating branches for traditional Chinese characters and small seal script images;

[0023] Figure 5 A schematic diagram of the network model structure for the Small Seal script prediction branch. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] This invention, based on deep learning technologies, implements a Small Seal Script recognition method using the Swing Transformer. The network includes an encoder E that parses Small Seal Script images into global image features, and three parallel sub-network branches: a Traditional Chinese character image generation branch that decodes global image features into Traditional Chinese character images, and a Small Seal Script image generation branch that decodes global image features into Small Seal Script images. These two branches work together to improve the feature extraction performance of E. The Small Seal Script prediction branch can obtain the Small Seal Script recognition result based on the global features of the Small Seal Script image. See the network structure diagram of this invention. Figure 1 .

[0026] This invention applies the Swing Transformer structure to the fields of character recognition and image style transfer, realizing the recognition of Small Seal script images and the conversion of Small Seal script images into Traditional Chinese character images, providing a new approach to Small Seal script recognition tasks. The invention will be further described below with reference to the accompanying drawings. See also... Figure 2 The steps of the present invention will be described in detail below:

[0027] Step S1 involves constructing a small seal script image training set X. s The test set, and the traditional Chinese character image X corresponding to the small seal script image. t And perform preprocessing, the specific method is as follows:

[0028] S1.1 Collect images of 6 different Small Seal script fonts from the Small Seal script font library, with 3666 characters for each font. Select 2 fonts with significantly different shapes as the test set fonts, and the remaining 4 fonts as the training set fonts. Randomly divide the 3666 Small Seal script characters into two groups: one group with 2914 characters is used as the training set font, and the other group with 752 characters is used as the test set font. Pair the Small Seal script images of the 2914 training set fonts (the 4 fonts in the training set) with their corresponding Traditional Chinese character images to form the Small Seal script recognition task training set. Use the Small Seal script images of the 752 test set fonts (the 2 fonts in the test set) as the Small Seal script recognition task test set.

[0029] S1.2 Divide the pixels in the image into 256 gray levels from 0 to 255 according to their gray values. Select a gray level threshold of 200 and perform binarization processing on the image. Set the gray value of pixels with a gray value greater than or equal to 200 to 255 and the gray value of pixels with a gray value less than 200 to 200. Then, reshape the image to a size of 256 pixels wide and 256 pixels high using bilinear interpolation, resulting in a small seal script image of size 1×256×256 and its corresponding traditional Chinese character image of size 1×256×256. The bilinear interpolation method is a mature existing technology.

[0030] Step S2 involves dividing the Small Seal Script image from S1 into image blocks and embedding them into the encoder E to obtain the global image features F of the Small Seal Script image. s The network structure diagram of E is shown below. Figure 3 The specific method is as follows:

[0031] S2.1 The small seal image with a size of 1×256×256 mentioned in S1 is fed into the block coding layer. The block coding layer specifically includes a convolutional layer with 1 input channel, 96 output channels, a stride of 4, and a kernel size of 4×4, and outputs block features with a dimension of 96×64×64.

[0032] S2.2 Merge the last two dimensions of the block feature with a size of 96×64×64 to obtain a block feature with a size of 96×4096. Then, swap the order of the two dimensions to obtain a block feature with a size of 4096×96.

[0033] S2.3 Create a downsampling array downsample to store the global features of the downsampled image. Store the 4096×96 block features mentioned above into downsample.

[0034] S2.4 Next, the features are fed into the Swing Transformer block [reference]. The Swing Transformer block specifically includes a W-MSA Transformer layer and an SW-MSA Transformer layer. The W-MSA Transformer layer and the SW-MSA Transformer layer are mature technologies in Swing Transformer, which help reduce the computational complexity of the model and allow the features to have a global receptive field. The Swing Transformer block does not change the feature dimension and outputs features of size 4096×96.

[0035] S2.5 The features are fed into the Patch Partition layer. Patch Partition is a mature technology in SwinTransformer. Patch Partition reduces the size of the global features of the input image to 1 / 4 of the input size and increases the number of channels to twice the input size, resulting in a global feature of size 1024×192.

[0036] S2.6 Store the 1024×192 feature in downsample;

[0037] S2.7 is similar to steps S2.4 to S2.6. The features are sent into the Swing Transformer block and then into the Patch Partition. The feature size becomes 256×384. The features are then stored in downsample.

[0038] S2.8 is similar to steps S2.4 and S2.5, where the feature is sent into the Swing Transformer block and then into the Patch Partition, and the feature size becomes 64×768.

[0039] S2.9 The features are fed into the W-MSA Transformer layer to calculate the global features of the image, and the output feature size is still 64×768;

[0040] S2.10 Repeat step S2.9, feeding the feature into the W-MSA Transformer layer to calculate the global features of the feature and output feature F. s The size remains 64×768, serving as the global feature of the output image of E.

[0041] Step S3 involves using the global features F of the Small Seal Script image from S2. s The traditional Chinese character image generation branch is fed in, and the global features of the image are decoded into the corresponding traditional Chinese character image Y.st And determine, the network structure diagram of the traditional Chinese character image generation branch is shown below. Figure 4 The specific method is as follows:

[0042] S3.1 The feature F s The image is fed into a block expansion layer. This layer increases the size of the global features of the input image to four times the input size and reduces the number of channels to half the input size, expanding the features from 64×768 to 256×384. The specific operations of the block expansion layer are as follows:

[0043] S3.1.1 Use a linear layer with an output dimension twice that of the input dimension to double the channel dimension of the feature, that is, to increase the size of the feature from 64×768 to 64×1536. The linear layer is a mature existing technology.

[0044] S3.1.2 Reshape the feature by increasing the dimensionality of the global image feature of the feature by 4 times and reducing the dimensionality of the channel of the feature by 1 / 4 times. The feature is reshaped from 64×1536 to 256×384.

[0045] S3.1.3 The features are fed into a layer normalization (LN) with a dimension of 384, and the output feature size is still 256×384, where LN is an existing mature technology.

[0046] S3.2 Extract the E feature with a size of 256×384 from the downsample, and concatenate it with the feature in the channel dimension. After concatenation, the size becomes 256×768. Then, use a linear layer with an output dimension of 1 / 2 of the input dimension to reduce the dimensionality of the feature in the channel dimension, and the feature size becomes 256×384 again.

[0047] S3.3 The features are fed into the Swing Transformer block to calculate the global features of the image, and the output global feature size is still 256×384;

[0048] S3.4 is similar to steps S3.1 to S3.3. The feature is expanded to 1024×192 using a block expansion layer. The feature is then concatenated and dimensionality reduced with the 1024×192 feature in the downsample. Finally, it is fed into the SwinTransformer block to output a global feature of size 1024×192.

[0049] S3.5 is similar to steps S3.1 to S3.3. The feature is expanded to 4096×96 using a block expansion layer. The feature is then concatenated and dimensionality reduced with the 4096×96 feature in the downsample. Finally, it is fed into the SwinTransformer block to output a global feature of size 4096×96.

[0050] S3.6 The features are fed into a 4x block expansion layer. The block expansion layer increases the size of the global features of the input image to 16 times the input size, expanding the features from 4096×96 to 65536×96. The specific operation of the 4x block expansion layer is as follows:

[0051] S3.6.1 Use a linear layer with an output dimension 16 times that of the input dimension to increase the channel dimension of the feature to 16 times the original dimension, that is, increase the size of the feature from 4096×96 to 4096×1546.

[0052] S3.6.2 Reshape the feature by increasing the dimensionality of the global image feature of the feature by 16 times and reducing the dimensionality of the feature channel by 1 / 16 times. The feature is reshaped from 4096×1546 to 65536×96.

[0053] S3.6.3 The features are fed into an LN with a dimension of 96, and the output feature size is still 65536×96;

[0054] S3.7 The features are fed into the channel dimensionality reduction layer to obtain the generated traditional Chinese character image Y. st The specific operation of the channel dimensionality reduction layer is as follows:

[0055] S3.7.1 Split the global feature dimension of the image of the feature, and reshape the feature from 65536×96 to 256×256×96;

[0056] S3.7.2 The global feature dimension and channel dimension of the image of the feature are swapped, and the feature size is changed from 256×256×96 to 96×256×256;

[0057] S3.7.3 The features are fed into a convolutional layer with 96 input channels, 1 output channel, a stride of 1, and a kernel size of 1×1, outputting a traditional Chinese character image Y with dimensions of 1×256×256. st ;

[0058] S3.8 Transfer the traditional Chinese character image Y st Send to D t In the process, the image discrimination result is obtained, and the D... t The specific steps are as follows:

[0059] S3.8.1 The image is fed into a convolutional layer with 1 input channel, 64 output channels, a stride of 2, and a kernel size of 5×5, and the output is a discriminative feature with a dimension of 64×128×128.

[0060] S3.8.2 The discriminant features are nonlinearly transformed using the Rectified Linear Unit (ReLU) activation function, and the output discriminant feature size remains 64×128×128, where ReLU is a mature existing technology;

[0061] S3.8.3 The discriminative features are fed into a convolutional layer with 64 input channels, 128 output channels, a stride of 2, and a kernel size of 5×5, and the output discriminative features with dimensions of 128×64×64 are obtained.

[0062] S3.8.4 The discriminant features are fed into a batch normalization layer (BN) with a dimension of 128, and the output discriminant feature dimension is still 128×64×64, where BN is an existing mature technology;

[0063] S3.8.5 The ReLU activation function is used to perform a nonlinear transformation on the discriminant features, and the size of the output discriminant features remains 128×64×64;

[0064] S3.8.6 is similar to steps S3.8.3 to S3.8.5. The discriminative features are fed into a convolutional layer with an input dimension of 128 and an output dimension of 256, and then fed into a BN and ReLU activation function with a dimension of 256 to obtain discriminative features of size 256×32×32.

[0065] S3.8.7 Stretch the discriminative features into a 262144-dimensional vector and feed it into a linear layer with an output dimension of 1 to obtain D. t Judgment results.

[0066] Step S4 will use the global features F of the image s The image is fed into the Small Seal script image generation branch, where the global features of the image are decoded into the corresponding Small Seal script character image Y. ss It is determined that the network model structure of the Small Seal Script image generation branch is completely consistent with that of the Traditional Chinese Character image generation branch. Similar to operation S3, the global image features F from S2 are used. s Input the small seal image to generate a branch, using G s Generate corresponding small seal script image Y ss and using D s Identify the image.

[0067] Step S5 uses the corresponding traditional Chinese character image X mentioned in S1. t As model input, repeat steps S2 to S4, using E to convert the traditional Chinese character X. tEncoding as global image features F t Then, the traditional Chinese character image is fed in parallel to generate the branch output traditional Chinese character image Y. tt The result is then evaluated and fed into the Small Seal Script image generation branch to generate Small Seal Script image Y. ts And determine the result.

[0068] Step S6 will process the global features F of the image. s The image is fed into the Small Seal Script prediction branch, and the Small Seal Script image is classified globally to obtain the Small Seal Script recognition result Y. l The network structure diagram of the predicted branch in Small Seal script can be found here. Figure 5 The specific method is as follows:

[0069] S6.1 The feature is fed into the block compression layer, and the block compression layer will input the F s The global feature size of the image is reduced to 1 / 4 of the input size, resulting in a global feature size of 16×768. The specific operation of the block compression layer is as follows:

[0070] S6.1.1 Reshape the features to 8×8×768, sampling the first and second dimensions of the global image features with a stride of 2, and then F... s It is divided into four blocks, f1 to f4, with a size of 4×4×768. Then, f1 to f4 are spliced ​​together in the channel dimension to obtain a global feature with a size of 4×4×3072. Then, the first and second dimensions of the feature are merged to obtain a global feature with a size of 16×3072.

[0071] S6.1.2 The global features are fed into an LN layer with a dimension of 3072, and then a linear layer with an output dimension of 1 / 4 of the input dimension is used to reduce the number of channels of the features to 1 / 4 of the original, resulting in a global feature of size 16×768.

[0072] S6.2 The features are fed into the Swing Transformer block to calculate the global features of the image, and the output global feature size is still 16×768;

[0073] S6.3 is similar to steps S6.1 and S6.2. The features are sent to the block compression layer to reduce the size of the global features of the image to 1 / 4 of the input size, resulting in a global feature of size 4×768. The feature is then sent to the Swing Transformer block to calculate the global features of the image.

[0074] S6.4 After merging and flattening the first and second dimensions of the feature, a feature vector with a length of 3072 is obtained;

[0075] S6.5 The vector is fed into a linear layer with an input dimension of 3072 and an output dimension of 3666 to reshape the vector dimension to 3666. Then, a normalized exponential function (softmax) is used to classify the features to obtain the recognition result Y of the Small Seal script. l The softmax function is a mature existing technology.

[0076] Step S7 calculates the loss function and backpropagates to train the network, where the loss function can be expressed as L = λ. F L F +L I +L C Where L represents the overall loss function of the model, L F λ represents the encoder loss. F L represents the weights of the encoder loss. I L represents the image generation loss. c Representing the prediction loss in Small Seal script, preferably, in the embodiments of the present invention, λ F =15, the specific components and calculation method of the loss function are as follows:

[0077] S7.1 Calculate encoder loss L F To fully utilize the semantic information of the structural similarity between Small Seal script and Traditional Chinese characters, the global features F of the Small Seal script image obtained by E encoding should be preserved as much as possible. s L maintains consistent global features with its corresponding traditional Chinese character image. F The calculation formula is:

[0078] L F =MSE(F s F t )

[0079] Where MSE(F) s F t ) indicates the calculation of F s and F t The mean squared error between them;

[0080] S7.2 Calculate the image generation loss L I L I The constrained image generation results are more realistic, thus enabling the conversion of Small Seal script images into Traditional Chinese character images, and enhancing E's global feature extraction capabilities. I The calculation formula is:

[0081] L I =L GAN +λ L1 L L1

[0082] Where L GAN L represents the image generation loss.L1 λ represents the image consistency loss. L1 The weight representing the image consistency loss, preferably, in this embodiment of the invention, λ L1 =70, the L GAN The traditional Chinese character image generation branch and the small seal script image generation branch are required to generate high-quality images and correctly identify the images. GAN The calculation formula is:

[0083] L GAN =log D s (X s )+log(1-D s (Y ss ))+log(1-D s (Y st ))

[0084] +log D t (X t )+log(1-D t (Y st ))+log(1-D t (Y tt ))

[0085] Where D s (X) represents D s The discrimination result for image X, D t (X) represents D t The discrimination result for image X; the L L1 The generated image must be identical to the corresponding input image. Ll The calculation formula is:

[0086] L L1 =MAE(X s Y ss )+MAE(X s Y ts )+MAE(X t Y st )+MAE(X t Y tt )

[0087] MAE(X,Y) represents the calculation of the mean absolute error between images X and Y.

[0088] S7.2 Calculate the prediction loss L in Small Seal script C L C The constrained small seal script prediction branch predicts the correct small seal script recognition result, L C The calculation formula is:

[0089] L label =CE(X)l Y l )

[0090] Where X l Representing the true result of Small Seal script, CE(X) l Y l ) indicates the calculation of X l and Y l Cross-entropy between them.

[0091] Step S8 uses a test set to verify the recognition accuracy of the Small Seal script recognition task. The test set from S1 is fed into the trained network model to test the Small Seal script recognition accuracy. The present invention achieves an accuracy of 75.91% in recognizing Small Seal script characters or character sets that have never been seen before, as shown in Table 1:

[0092] Table 1 compares the recognition accuracy of this invention with other deep learning Chinese character recognition models:

[0093]

[0094] This invention utilizes the structural similarities between Small Seal script and Traditional Chinese characters, and builds a model based on the efficient Swing Transformer module that can recognize Small Seal script and generate corresponding Traditional Chinese character images. It can recognize Small Seal script images of fonts or character sets that have never been seen before with high accuracy.

Claims

1. A method for recognizing Small Seal script based on Swing Transformer, characterized in that... Includes the following steps: Step S1: Construct a Small Seal Script image training set The test set, and the traditional Chinese character images corresponding to the small seal script images. The image was preprocessed to obtain a small seal script image with a size of 1×256×256. Step S2: Divide the Small Seal image from Step S1 into image blocks and perform embedding encoding, then feed the blocks into an encoder built on SwinTransformer. In the process, the global features of the image corresponding to the Small Seal script image are obtained. ; The encoder The network structure is as follows: S2.1 The small seal image with a size of 1×256×256 mentioned in S1 is fed into the block coding layer. The block coding layer specifically includes a convolutional layer with 1 input channel, 96 output channels, a stride of 4, and a kernel size of 4×4, and outputs a block feature with a dimension of 96×64×64. S2.2 Merge the last two dimensions of the block feature with a size of 96×64×64 to obtain a block feature with a size of 96×4096. Then, swap the order of the two dimensions to obtain a block feature with a size of 4096×96. S2.3 Create a downsampling array downsample to store the global features of the downsampled image. Store the 4096×96 block features mentioned above into downsample. S2.4 Next, the features are fed into the Swing Transformer block, which specifically includes a W-MSA Transformer layer and an SW-MSA Transformer layer. The Swing Transformer block does not change the feature dimension and outputs a feature of size 4096×96. S2.5 The features are fed into the block merging layer. The merging layer reduces the size of the global features of the input features to 1 / 4 of the input size and increases the number of channels to twice the input size, resulting in a global feature of size 1024×192. S2.6 Store the 1024×192 feature in downsample; S2.7 The features are fed into the Swing Transformer block, then into the merging layer, and the feature size becomes 256×384. The features are then stored in downsample. S2.8 The feature is fed into the Swing Transformer block and then into the merge layer, and the feature size becomes 64×768; S2.9 The features are fed into the W-MSA Transformer layer to calculate the global features of the image, and the output feature size is still 64×768; S2.10 Repeat step S2.9, feeding the features into the W-MSA Transformer layer to calculate the global features of the features and output the features. The size remains 64×768, as The output image global features; Step S3: Extract the global features of the image from step S2. The traditional Chinese image generation branch is used, employing a traditional Chinese image generation decoder built based on SwinTransformer. Global features of the image Decoded into corresponding Traditional Chinese character images And use a Traditional Chinese image discriminator Judge the image generation effect; Step S4: Extract the global features of the image from step S2. The small seal image generation branch is fed in, using a small seal image generation decoder built on SwinTransformer. Global features of the image Decoded into the corresponding Small Seal script image And use the Small Seal Script image discriminator Judge the image generation effect; Step S5: Select the corresponding traditional Chinese character image mentioned in S1 as the network input, and repeat steps S2 to S4 to decode the corresponding traditional Chinese character image into global features. The branch is generated by using traditional Chinese characters. Decode into Traditional Chinese The image is analyzed and a branch is generated from the Small Seal script image. Decoded into Small Seal Script Image The decoding effect of the decoder and the image generation effect of the traditional Chinese character image generation branch and the traditional Chinese character image generation branch are verified by using the structural information that is very similar between the small seal script and the traditional Chinese character. Step S6: Global features of the Small Seal Script image described in Step S2 The input is fed into the Small Seal Script prediction branch, using a Small Seal Script prediction classifier built on SwinTransformer. right The classification process yielded the final prediction results for the Small Seal script. ; Step S7: Calculate the model loss function including encoder loss Image generation loss for Traditional Chinese character image generation branch and Small Seal script image generation branch The loss of the small seal script prediction branch The network is trained via backpropagation; Step S8: Use the Small Seal Script Recognition Task Test Set to verify the recognition accuracy of the Small Seal Script Recognition Task.

2. The method for recognizing Small Seal script based on Swing Transformer according to claim 1, characterized in that... Step S1 is implemented as follows: S1.1 Collect images of 6 small seal script fonts from the small seal script font library, with 3666 small seal script characters for each font. Select 2 fonts with significant shape differences as the test set fonts, and the remaining 4 fonts as the training set fonts. Randomly divide the 3666 small seal script characters into two groups: one group with 2914 small seal script characters is used as the training set font, and the other group with 752 small seal script characters is used as the test set font. Pair the small seal script images of the 2914 training set fonts and their corresponding traditional Chinese character images to form the small seal script recognition task training set. Use the small seal script images of the 752 test set fonts as the small seal script recognition task test set. S1.2 Divide the pixels in the image into 256 gray levels from 0 to 255 according to their gray values. Select a gray level threshold of 200 and perform binarization processing on the image. Set the gray value of pixels with a gray value greater than or equal to 200 to 255 and the gray value of pixels with a gray value less than 200 to 200. Then, reshape the image to a size of 256 pixels wide and 256 pixels high using bilinear interpolation to obtain a small seal script image of size 1×256×256 and its corresponding traditional Chinese character image of size 1×256×256.

3. The method for recognizing Small Seal script based on Swing Transformer according to claim 2, characterized in that... Step S3 is implemented as follows: S3.1 The features The image is fed into a block expansion layer, which increases the size of the global features of the input features to 4 times the input size and reduces the number of channels to 1 / 2 times the input size, expanding the features from 64×768 to 256×384. S3.2 Extract the 256×384 element from the downsample. The features are concatenated with the features in the channel dimension, and the size of the concatenated features becomes 256×768. Then, a linear layer with an output dimension of 1 / 2 of the input dimension is used to reduce the dimensionality of the features in the channel dimension, and the size of the features becomes 256×384 again. S3.3 The features are fed into the Swing Transformer block to calculate the global features of the image, and the output global feature size is still 256×384; S3.4 Similar to steps S3.1 to S3.3, the feature is expanded to 1024×192 using a block expansion layer. The feature is then concatenated and dimensionality reduced with the 1024×192 feature in the downsample, and then fed into the SwinTransformer block to output a global feature of size 1024×192. S3.5 Similar to steps S3.1 to S3.3, the feature is expanded to 4096×96 using a block expansion layer. The feature is then concatenated and dimensionality reduced with the 4096×96 feature in the downsample, and then fed into the Swin Transformer block to output a global feature of size 4096×96. S3.6 The features are fed into a 4x block expansion layer. The block expansion layer increases the size of the global features of the image with the input features to 16 times the input size, expanding the features from 4096×96 to 65536×96. S3.7 The features are fed into the channel dimensionality reduction layer to obtain the generated traditional Chinese character image. The specific operation of the channel dimensionality reduction layer is as follows: S3.8 Transfer the traditional Chinese character image Send in In the process, the image discrimination results are obtained.

4. The method for recognizing Small Seal script based on Swing Transformer according to claim 3, characterized in that... The specific operations of the block expansion layer in step S3.1 are as follows: S3.1.1 Use a linear layer with an output dimension twice that of the input dimension to double the channel dimension of the feature, that is, to increase the size of the feature from 64×768 to 64×1536. S3.1.2 Reshape the feature by increasing the dimensionality of the global image feature of the feature by 4 times and reducing the dimensionality of the channel of the feature by 1 / 4 times. The feature is reshaped from 64×1536 to 256×384. S3.1.3 The features are fed into a 384-dimensional layer for normalization, and the output feature size remains 256×384.

5. A method for recognizing Small Seal script based on Swing Transformer according to claim 3 or 4, characterized in that... The specific operation of the 4x block expansion layer in step S3.6 is as follows: S3.6.1 Using a linear layer with an output dimension 16 times that of the input dimension, the channel dimension of the feature is increased to 16 times the original dimension, that is, the feature size is increased from 4096×96 to 4096×1546. S3.6.2 Reshape the feature by increasing the dimensionality of the global image feature of the feature by 16 times and reducing the dimensionality of the channel of the feature by 1 / 16 times. The feature is reshaped from 4096×1546 to 65536×96. S3.6.3 The features are fed into an LN with a dimension of 96, and the output feature size remains 65536×96.

6. A method for recognizing Small Seal Script based on Swing Transformer according to claim 3 or 4, characterized in that... The specific operation of the channel dimensionality reduction layer in step S3.7 is as follows: S3.7.1 Split the global feature dimension of the image of the feature, and reshape the feature from 65536×96 to 256×256×96; S3.7.2 The global feature dimension and channel dimension of the image of the feature are swapped, and the feature size is changed from 256×256×96 to 96×256×256; S3.7.3 The features are fed into a convolutional layer with 96 input channels, 1 output channel, a stride of 1, and a kernel size of 1×1, outputting a traditional Chinese character image with dimensions of 1×256×256. .

7. A method for recognizing Small Seal Script based on Swing Transformer according to claim 3 or 4, characterized in that... The steps described in S3.8 The specific steps are as follows: S3.8.1 The image is fed into a convolutional layer with 1 input channel, 64 output channels, a stride of 2, and a kernel size of 5×5, and the output is a discriminative feature with a dimension of 64×128×128. S3.8.2 The discriminant features are nonlinearly transformed using the Rectified Linear Unit (ReLU) activation function, and the output discriminant feature size remains 64×128×128, where ReLU is a mature existing technology; S3.8.3 The discriminant features are fed into a convolutional layer with 64 input channels, 128 output channels, a stride of 2, and a kernel size of 5×5, and the output discriminant features with dimensions of 128×64×64 are obtained. S3.8.4 The discriminative features are fed into a batch normalization layer (BN) with a dimension of 128, and the output discriminative feature dimension is still 128×64×64, where BN is an existing mature technology; S3.8.5 The discriminant features are nonlinearly transformed using the ReLU activation function, and the size of the output discriminant features remains 128×64×64; S3.8.6 Similar to steps S3.8.3 to S3.8.5, the discriminative features are fed into a convolutional layer with an input dimension of 128 and an output dimension of 256, and then fed into a BN and ReLU activation function with a dimension of 256 to obtain discriminative features of size 256×32×32. S3.8.7 Stretch the discriminative features into a 262144-dimensional vector and feed it into a linear layer with an output dimension of 1 to obtain... Judgment results.

8. The method for recognizing Small Seal script based on Swing Transformer according to claim 7, characterized in that... The specific operation of step S6 is as follows: S6.1 The feature is fed into the block compression layer, and the block compression layer will input the feature. The global feature size of the image is reduced to 1 / 4 of the input size, resulting in a global feature size of 16×768. The specific operation of the block compression layer is as follows: S6.1.1 The features are reshaped into 8×8×768, and samples are taken at a stride of 2 in the first and second dimensions of the global image features. The feature is divided into four blocks, f1 to f4, with a size of 4×4×768. Then, f1 to f4 are concatenated along the channel dimension to obtain a global feature with a size of 4×4×3072. Subsequently, the first and second dimensions of the feature are merged to obtain a global feature with a size of 16×3072. S6.1.2 The global features are fed into an LN layer with a dimension of 3072, and then a linear layer with an output dimension of 1 / 4 of the input dimension is used to reduce the number of channels of the features to 1 / 4 of the original, resulting in a global feature of size 16×768. S6.2 The features are fed into the Swing Transformer block to calculate the global features of the image, and the output global feature size is still 16×768; S6.3 Similar to steps S6.1 and S6.2, the features are fed into the block compression layer to reduce the size of the global features of the image to 1 / 4 of the input size, resulting in a global feature of size 4×768. Then, the feature is fed into the Swing Transformer block to calculate the global features of the image. S6.4 After merging and flattening the first and second dimensions of the feature, a feature vector with a length of 3072 is obtained; S6.5 The vector is fed into a linear layer with an input dimension of 3072 and an output dimension of 3666 to reshape the vector dimension to 3666. Then, a normalized exponential function is used to classify the features to obtain the recognition result of the Small Seal script. .

9. A method for recognizing Small Seal Script based on Swing Transformer according to claim 8, characterized in that... The specific operation of step S7 is as follows: Step S7 calculates the loss function and backpropagates the training network, wherein the loss function is expressed as: ,in The loss function representing the overall model. Indicates encoder loss, The weights represent the encoder loss. Indicates the image generation loss. The loss function, representing the prediction loss in Small Seal script, is composed of the following components and is calculated as follows: S7.1 Calculate encoder loss To fully utilize the semantic information of the structural similarity between the Small Seal script and the Traditional Chinese characters, we should try our best to ensure that the characters are written in the same style. Global features of the Small Seal Script image obtained by encoding It maintains the same global features as its corresponding traditional Chinese character image. The calculation formula is: in Indicates calculation and The mean squared error between them; S7.2 Calculate the image generation loss , The constraints result in more realistic image generation, thus enabling the conversion of Small Seal script images into Traditional Chinese character images, and enhancing... Global feature extraction capability The calculation formula is: in This represents the image generation loss. This indicates image consistency loss. The weights representing the image consistency loss are... The The traditional Chinese character image generation branch and the small seal script image generation branch are required to generate high-quality images and correctly identify the images. The calculation formula is: in express The discrimination result for image X, express The discrimination result for image X; the stated The generated image must be identical to the corresponding input image. The calculation formula is: in This indicates the calculation of the mean absolute error between images X and Y; S7.2 Calculate the prediction loss for Small Seal script , The constrained Small Seal script prediction branch predicts the correct Small Seal script recognition result. The calculation formula is: in This represents the true result of Small Seal script. Indicates calculation and Cross-entropy between them.

Images