A method and system for ancient character recognition based on a hybrid framework of CNN and Vision Transformer

Abstract

This invention discloses an ancient character recognition method and system based on a hybrid CNN and Vision Transformer framework. The method includes: constructing an original dataset Dy; preprocessing the ancient character images in the original dataset Dy; cropping the ancient character images to a predetermined standard size; expanding the original dataset Dy; constructing a complete dataset; performing data augmentation on the expanded dataset Dk to obtain the complete dataset Dw; constructing a hybrid CNN and Vision Transformer framework model (pyramid structure network model); integrating contextual information, extracting spatial location information, and extracting fine-grained features using the hybrid CNN and Vision Transformer framework model; and classifying and recognizing the ancient characters using the pyramid structure network model. The ancient character recognition method and system based on the hybrid CNN and Vision Transformer framework of this invention have advantages such as using pseudo-data to supplement the few categories of ancient character images, and achieving accurate recognition of characters in ancient character images with severe noise and incompleteness even with a small data scale.

Description

Technical Field

[0001] This invention relates to a character recognition technology, and more particularly to an ancient character recognition method and system based on a hybrid framework of CNN and Vision Transformer. Background Technology

[0002] Ancient writing systems such as oracle bone script and bamboo slips are precious intangible cultural heritage of the Chinese nation. The primary function of these ancient scripts was to record various activities in ancient society, and their stroke and structural rules laid the foundation for the early development of the Chinese language and writing system. Therefore, the study of ancient writing systems is of great value to the study of Chinese etymology and calligraphy.

[0003] Deep learning specifically refers to machine learning based on deep neural network models and methods. It has evolved from statistical machine learning and artificial neural network algorithms, combined with the development of modern big data and high-performance computing. The most important technical characteristic of deep learning is its ability to automatically extract features; these extracted features are also called deep features or deep feature representations. Compared to manually designed features, deep features are more powerful and robust. Therefore, the essence of deep learning is feature representation learning. Deep neural networks are the model foundation for deep learning's ability to automatically extract features; deep neural networks are essentially a series of nested nonlinear transformations. Currently, deep learning appears to be the most promising technical approach to solving the major scientific and technological problem of strong artificial intelligence, and it is also a research hotspot in the fields of computer science, big data science, and artificial intelligence. In recent years, deep learning methods have been increasingly applied to the field of ancient character recognition.

[0004] However, deep learning still faces many technical challenges in the recognition of ancient Chinese characters. First, ancient Chinese characters have been passed down for thousands of years. The image quality of many ancient characters has severely deteriorated, and some characters only have partial images, lacking complete text images, which hinders effective recognition. These text images are often fragmented and subject to severe noise interference during recognition, while also suffering from intra-class variation and inter-class similarity issues. Second, the training of deep learning models typically requires a large number of labeled samples. However, given the limited number of unearthed ancient characters, the data size of ancient character datasets is relatively small, often affected by long-tailed class distributions. Based on these two reasons, existing methods for ancient character recognition face certain difficulties. Summary of the Invention

[0005] To avoid the shortcomings of the existing technologies, this invention provides an ancient character recognition method based on a hybrid framework of CNN and VisionTransformer, which can accurately recognize characters in ancient character images with severe noise and defects even with a small data scale.

[0006] The present invention adopts the following technical solution to solve the technical problem.

[0007] The present invention provides a method for ancient character recognition based on a hybrid framework of CNN and Vision Transformer, comprising the following steps:

[0008] Step 1: Construct the original dataset Dy; collect images of ancient characters and filter, classify and label them to obtain the original dataset Dy;

[0009] Step 2: Preprocess the ancient text images in the original dataset Dy; crop the ancient text images to a predetermined standard size, and count the number n of ancient text images in each category;

[0010] Step 3: Expand the original dataset Dy; the number of ancient text images n is less than the preset value n. y The original dataset Dy is expanded by classifying the dataset and generating pseudo-data to obtain the expanded dataset Dk.

[0011] Step 4: Construct the complete dataset; Perform data augmentation on the expanded dataset Dk to obtain the complete dataset Dw;

[0012] Step 5: Construct a hybrid CNN and Vision Transformer framework model (pyramid structure network model); the hybrid CNN and Vision Transformer framework model integrates contextual information, extracts spatial location information, and extracts fine-grained features;

[0013] Step 6: Classify and identify ancient scripts using the pyramid structure network model described above.

[0014] The ancient character recognition method based on a hybrid framework of CNN and Vision Transformer of the present invention is also characterized by:

[0015] Furthermore, in step 2, the ancient character image is cropped to a size of 256×256.

[0016] Furthermore, in step 2, after counting the number n of ancient text images in each category, the imbalance intensity BPH of the data distribution in the original dataset Dy is calculated.

[0017] Furthermore, in step 3, a self-attention generative adversarial network (SAGAN) is used as an image generation module to generate pseudo data.

[0018] Furthermore, in step 3, the self-attention generative adversarial network (SAGAN) includes a generator and a discriminator.

[0019] Furthermore, in step 3, during the process of constructing the complete dataset, the header data is first subjected to random undersampling before data augmentation.

[0020] Furthermore, in step 4, during the data augmentation process, data augmentation is performed based on the augmentation intensity coefficient λ.

[0021] Furthermore, the data augmentation process includes flipping, fading, scratch removal, noise addition, and blurring.

[0022] Furthermore, in step 5, the hybrid CNN and Vision Transformer framework model includes a shallow CNN module, a DCSP module, a deep LFGP module, and a Vision Transformer module.

[0023] This invention also discloses an ancient character recognition system based on a hybrid framework of CNN and Vision Transformer, including an ancient character data acquisition module, an ancient character preprocessing module, an image generation module, a data augmentation module, a context modeling module, a spatial information extraction module, a fine-grained feature extraction module, and an ancient character recognition module;

[0024] The ancient script data acquisition module is used to collect images of ancient scripts, classify and label all ancient script images, and obtain the original dataset Dy.

[0025] The ancient script preprocessing module is used to crop the ancient script images and count the number n of ancient script images in each category of the ancient script image classification.

[0026] The image generation module is used to generate pseudo data from the preprocessed ancient script dataset using a self-attention generative adversarial network (SAGAN), and then adds the pseudo data to the original dataset Dy to obtain an expanded dataset Dk.

[0027] The data augmentation module is used to perform data augmentation processing on the expanded dataset Dk to obtain the complete dataset Dw;

[0028] The context modeling module uses the MobileViT module to model the global context;

[0029] The spatial information extraction module uses the DCSP module to obtain the spatial coordinate information of the X and Y coordinates of the image in parallel using average pooling and max pooling operations;

[0030] The fine-grained feature extraction module uses the LFGP module to employ an overlapping patch partitioning strategy to enhance feature interaction, thereby enabling the learning of finer local details.

[0031] The ancient script recognition module uses a hybrid pyramid framework SAF-ViT network combining CNN and Vision Transformer to train and classify ancient scripts, thereby enabling the recognition of ancient scripts and outputting the recognition results.

[0032] Compared with existing technologies, the beneficial effects of this invention are reflected in:

[0033] This invention discloses an ancient character recognition method and system based on a hybrid CNN and Vision Transformer framework, including: constructing an original dataset Dy; preprocessing the ancient character images in the original dataset Dy; cropping the ancient character images to a predetermined standard size; expanding the original dataset Dy; constructing a complete dataset; performing data augmentation on the expanded dataset Dk to obtain the complete dataset Dw; constructing a hybrid CNN and Vision Transformer framework model (pyramid structure network model); the hybrid CNN and Vision Transformer framework model integrates contextual information, extracts spatial location information, and extracts fine-grained features; and classifying and recognizing the ancient characters using the pyramid structure network model.

[0034] The ancient character recognition method and system based on the hybrid framework of CNN and Vision Transformer of the present invention has the advantages of accurately recognizing characters in ancient character images with severe noise and incompleteness by supplementing the very few categories of ancient character images with pseudo data and with small data scale. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating the ancient character recognition algorithm based on a hybrid framework of CNN and Vision Transformer of the present invention. Figure 2 This is a block diagram of the ancient character recognition system based on a hybrid framework of CNN and Vision Transformer according to the present invention; Figure 3 This is an example diagram illustrating the noise and form of ancient character images presented in this invention. Ancient character images present challenges in recognition, including those with severe noise and intra-class variations. Figure 4This is an average quantity distribution chart of classification categories in one embodiment of the present invention, used to represent the severity of long-tail distribution. The average quantity distribution of each category reflects the long-tail problem in the data. Figure 5 This is a flowchart illustrating the data processing procedure in this invention; Figure 6 This is a diagram illustrating how the MobileViT module models the global context: Expand → Transformer → Collapse. Figure 7 This is a schematic diagram of the network flow of SAF-ViT based on a hybrid framework of CNN and Vision Transformer according to the present invention; Figure 8 It is a system website for ancient Chinese character recognition from the applicant's institution, Anhui University. Detailed Implementation

[0036] See Figures 1 to 8 The present invention provides a method for ancient character recognition based on a hybrid framework of CNN and Vision Transformer, comprising the following steps:

[0037] Step 1: Construct the original dataset Dy; collect images of ancient characters and filter, classify and label them to obtain the original dataset Dy;

[0038] First, the images of ancient characters (bamboo slips) were screened. Then, all collected images were categorized and labeled to construct the original dataset. Screening primarily involved removing images of ancient characters that showed no trace of writing or were completely contaminated by noise. Labeling was performed by experts from the School of Humanities, who used their expertise to determine the corresponding modern Chinese translation of each image. The ancient character images were then categorized according to their translated names.

[0039] When collecting images of ancient characters, all images should be included regardless of whether they are heavily noisy or partially damaged. When classifying and labeling all collected images, it is essential to ensure that the identification of the ancient characters is performed by experts to guarantee the accuracy of the classification.

[0040] Step 2: Preprocess the ancient text images in the original dataset Dy; crop the ancient text images to a predetermined standard size, and count the number n of ancient text images in each category;

[0041] The images of ancient characters are preprocessed, cropped to a size of 256×256, and the number of ancient character images n1 to n in all b categories is counted. b Then calculate the average number M of ancient text images across all categories: Where n1 is the number of ancient Chinese characters in the first category, and n k Let be the number of ancient text images in the k-th category, and b be the total number of categories, i.e., all ancient text images are divided into b categories, 1≤k≤b.

[0042] Step 3: Expand the original dataset Dy; the number of ancient text images n is less than the preset value n. y The original dataset Dy is expanded by classifying the dataset and generating pseudo-data to obtain the expanded dataset Dk.

[0043] In one embodiment, n y Set to 10. When the number of ancient text images n in a category (i.e., the number of samples n in that category) is less than 10, pseudo-data is added to that category to increase the number of ancient text images in that category to 10.

[0044] Step 4: Construct the complete dataset; Perform data augmentation on the expanded dataset Dk to obtain the complete dataset Dw;

[0045] Data augmentation (including flipping, fading, and blurring) is performed on the augmented dataset Dk. The augmentation intensity is determined based on the augmentation intensity coefficient λ to construct a diverse and balanced complete dataset Dw.

[0046] Step 5: Construct a hybrid CNN and Vision Transformer framework model (pyramid structure network model); the hybrid CNN and Vision Transformer framework model integrates contextual information, extracts spatial location information, and extracts fine-grained features;

[0047] This invention designs a pyramid structure network model based on MobileViT. The pyramid structure network model includes a shallow CNN module, a DCSP module, a deep LFGP module, and a Vision Transformer module. The shallow CNN module is responsible for extracting local information; the DCSP module is responsible for extracting spatial location information; the deep LFGP module is responsible for extracting fine-grained features; and the Vision Transformer module is responsible for integrating contextual information.

[0048] Step 6: Classify and identify ancient scripts using the pyramid structure network model described above.

[0049] Images of ancient texts are classified using the SAF-ViT pyramid structure network model. The SAF-ViT model is primarily composed of a hybrid framework of CNN and Vision Transformer, which better integrates local and global information to obtain the classification results.

[0050] The SAF-ViT pyramid structure network model was used to train and recognize ancient text images, yielding recognition results. The complete dataset Dw was divided into training, validation, and test sets in a 6:2:2 ratio. The network was trained using the training set, and the validation set was used to observe the network's training progress. Finally, the trained model was used to classify and recognize the test set to evaluate its performance.

[0051] Figure 7 The system website of the applicant's institution, Anhui University, is used for ancient character recognition. Users can upload images of ancient characters to the website for recognition, and then the system will identify the category of the ancient characters and output the corresponding modern characters.

[0052] In practice, in step 2, the ancient text image is cropped to a size of 256×256.

[0053] In specific implementation, in step 2, after counting the number n of ancient characters in each category, the imbalance intensity BPH of the data distribution in the original dataset Dy is calculated.

[0054] During the preprocessing of ancient text images, it is essential to ensure that they are cropped to a size of 256×256, while preserving the text area of ​​the ancient text image, ideally placing the text area in the center of the image. When counting the number of ancient text images for each category, all categories must be covered. Then, the imbalance strength (BPH) of the data distribution is analyzed based on the number of ancient text images for all categories. The imbalance strength is the imbalance ratio. The imbalance ratio is the ratio of the largest number of samples to the smallest number of samples in each category. For example, in the original dataset Dy, there are 5 categories; the number of ancient text images in the 5 categories are 2000, 1200, 600, 200, and 10, respectively. Then the average value M = (2000 + 1200 + 600 + 200 + 10) / 5 = 802. The category with the most samples included 2000 ancient character images, while the category with the fewest samples included 10 ancient character images. Therefore, the imbalance ratio is 200:1, meaning the imbalance strength (BPH) is 200:1. Generally, the BPH relates to the long-tailed distribution in the experiment; a smaller BPH generally results in better experimental outcomes.

[0055] In specific implementation, in step 3, the self-attention generative adversarial network SAGAN is used as the image generation module to generate pseudo data.

[0056] In specific implementation, in step 3, the self-attention generative adversarial network (SAGAN) includes a generator and a discriminator.

[0057] In specific implementation, the image generation module of the present invention adopts a self-attention generative adversarial network SAGAN (Self-Attention Generative Adversarial Network) to generate realistic fake data that is difficult to distinguish.

[0058] Most image generation models based on generative adversarial networks are constructed using convolutional layers. However, relying solely on convolutional layers to model long-range dependencies in images is computationally inefficient. To address this limitation, self-attention is incorporated into the generative adversarial network framework to construct a self-attention generative adversarial network SAGAN, enabling the generator and discriminator of the self-attention generative adversarial network SAGAN to effectively simulate the interactions between spatially distant regions. Through such an interaction method, the pyramid structure network model can learn the spatial position relationships of the glyphs of ancient characters in ancient character pictures, and when processing information, it is necessary to obtain two far-apart parts in the image. For example, for the character "small", the pyramid structure network model can identify the position relationship between the left and right points in "small", and then be more accurate and realistic when generating character pictures.

[0059] The process of generating fake data includes the following steps:

[0060] Step 31: Input noise and the label of the ancient character picture into the generator; in the generator, a series of processes such as convolution, normalization, and upsampling are performed. The generator contains a self-attention module (for modeling long-range dependencies in pictures) that is used to generate more realistic pictures.

[0061] Step 32: Input the generated picture and the label of the ancient character picture into the discriminator, and a series of processes such as convolution, normalization, and downsampling are performed in the discriminator. The generator also contains a self-attention module that outputs whether the discriminator image is real or fake.

[0062] Step 33: Through multiple rounds of adversarial training between the generator and the discriminator, the generator can generate more effective pictures, which are output as the final fake data (i.e., the above-mentioned more realistic pictures). When training SAGAN to generate fake data, the wgan_gp loss function with gradient penalty is used to prevent mode collapse.

[0063] In the self-attention generative adversarial network SAGAN, the specific method for implementing self-attention is: project the image features into two different feature domains f and g to calculate the attention score β j,i , and the calculation formula is shown in the following formula (1);

[0064] (1);

[0065] In formula (1), the attention score βj,i This represents the degree of attention the self-attention module in the self-attention generative adversarial network SAGAN pays to the i-th spatial location when synthesizing the j-th region (i.e., position j); for example, if the image features are divided into a 100*100 matrix, the degree of attention paid to the original input feature position i(80,80) when synthesizing a new feature position j(30,30). N represents the number of spatial locations from the feature maps of the previous hidden layers. i and x j It is a feature map generated by the image generation module as pseudo-data. i,j For x i and x j The similarity score. f(x) i ) represents the key vector at position i. g(x) j ) represents the query vector at position j. T represents the matrix transpose.

[0066] In formula (1), x i and x j As input, the output of the previous layer in the self-attention generation adversarial network SAGAN is used; where x∈R C×N Where C is the number of channels in the feature map, and N is the total number of spatial locations in the feature map (N = height H × width W). The H × W two-dimensional grid is "flattened" into a one-dimensional vector for easier computation. Self-attention computation first generates three metrics: query, key, and value, and then calculates the correlation between position i (key) and position j (query). Specifically, f(x) is calculated. i (the key vector at position i) and g(x) j The dot product of the query vector at position j is used. The larger the dot product, the more semantically related the features at positions i and j are. In formula (1), the softmax function (summing and normalizing exp(sij) over all i positions) is used to calculate the similarity score S. i,j Transform it into a probability distribution; β j,i This indicates the extent to which the network should focus on (assign weights to) the features at position i when synthesizing or judging the features of the j-th region.

[0067] The output Y of the self-attention module in the self-attention generative adversarial network SAGAN j See formula (2) below;

[0068] (2);

[0069] In formula (2), N represents the number of spatial locations from the feature maps of the preceding hidden layers; O j This represents weighted aggregation, where for output position j, the value vector h(x) of all positions i needs to be calculated. iWeighted average of h(x) i V is the value vector at position i. V() represents the linear transformation W. v ∈ R C×C The aggregated vector of dimension C+ is projected back to the original number of channels C using V(), so that it can be added to other convolutional paths in the network to fuse information. Output Y j x j The original input features are the output of the previous layer in the self-attention generative adversarial network SAGAN. γ is a learnable scalar parameter (initially initialized to 0), a gain coefficient that controls the global information O. j The degree of contribution to the final output. Y j This is the final output feature of the self-attention module, and it is also the feature passed to the next layer of the network.

[0070] like Figure 2 As shown, the pseudo data generated by the self-attention generative adversarial network SAGAN is input into the image enhancement module. The enhanced image is then processed sequentially by the context modeling module, the spatial information extraction module, and the fine-grained feature extraction module. Finally, the ancient script recognition module converts the ancient script into modern script.

[0071] In specific implementation, during step 3, when constructing the complete dataset, the header data is first subjected to random undersampling before data augmentation.

[0072] In specific implementation, in step 4, during the data augmentation process, data augmentation is performed based on the augmentation intensity coefficient λ.

[0073] In practice, the data augmentation process includes flipping, fading, scratch removal, noise addition, and blurring.

[0074] like Figure 5 This describes the entire data processing procedure of the present invention. First, the average number of samples in each category of the original dataset Dy is calculated.

[0075] (1) such as Figure 5 Then, using the average sample size as a reference, random undersampling is performed on the head class data (data volume greater than the average sample size). Data is randomly sampled, and the sampled images are then augmented. For example, if a category has 200 images, and 100 are randomly undersampled, then the 100 unsampled images are essentially deleted.

[0076] (2) For tail-class data (data volume less than the average number of samples) where data is extremely scarce, a self-attention generative adversarial network (SAGAN) is used to generate a large number of realistic synthetic samples, and the "generated data" (i.e., the "pseudo-data" mentioned above) is added to the samples. These synthetic samples retain the basic features of the original characters, while introducing subtle and reasonable changes in glyph structure and background texture. Then data augmentation is performed.

[0077] (3) Subsequently, we introduce an augmentation strength coefficient λ derived from the distribution of the original dataset Dy. The augmentation strength coefficient λ is crucial for adjusting the strength of various data augmentation operations, such as flipping, noise addition, and blurring.

[0078] The enhancement strength coefficient λ is determined by the size of the number of classes in the entire dataset. A higher λ value corresponds to a more significant, robust, and diverse enhancement effect. The calculation process of the enhancement strength coefficient λ is shown in the following formula (3);

[0079] (3);

[0080] In formula (3), λ q Let be the enhancement strength coefficient for the q-th category, b be the number of all categories in the original dataset Dy, and n be the number of categories. k Let n be the number of ancient text images in the k-th category. q q represents the number of ancient character images in the q-th category.

[0081] In specific implementation, step 5 includes a hybrid CNN and Vision Transformer framework model comprising a shallow CNN module, a DCSP module, a deep LFGP module, and a Vision Transformer module.

[0082] like Figure 5 As shown, the pyramid structure network model of this invention is a hybrid framework of CNN and Vision Transformer. By extracting features at different scales, the pyramid structure network model can effectively integrate information from images for recognition.

[0083] like Figure 4The existing traditional MobileViT module uses an unfolding → Transformer → folding modeling approach, which effectively preserves spatial location information. In this invention, a hybrid framework of CNN (Convolutional Neural Network) and Vision Transformer (ViT) is applied to the field of ancient character recognition, achieving good performance. After analyzing the shape, noise, and other features of ancient character images, spatial prior information and fine-grained feature extraction are introduced into the model to guide its learning. Experiments demonstrate the effectiveness of this method. Table 1 below compares this invention with existing methods.

[0084]

[0085] In Table 1: Top-1 accuracy is the number of correctly predicted samples divided by the total number of samples. Precision means that when the model correctly identifies a sample, its confidence is very high. Recall focuses on the ability to identify positive samples; high recall means that the model can find the vast majority of truly positive samples. It measures how many of all actually positive samples the model successfully predicted. F1-Score is the harmonic mean of precision and recall, used to measure the model's overall performance in terms of precision and recall. As shown in Table 1, the method of this invention demonstrates significant improvements and advancements across all metrics compared to existing similar methods.

[0086] The DCSP module uses average pooling (hereinafter referred to as a) and max pooling (hereinafter referred to as m) operations in parallel to model the spatial location information of the image through the following formula, as shown in the following formula (4);

[0087] (4);

[0088] In formula (4), XAvgPool and YavgPool are the average pooling operation functions applied to the c-th channel along the height h (X-axis) and width w (Y-axis), respectively. XMaxPool and YMaxPool are the maximum pooling operation functions applied to the c-th channel along the height h (X-axis) and width w (Y-axis), respectively. c represents the index of the c-th channel, c = 1, 2, 3, ..., C. This is the intermediate feature vector after the average pooling path has undergone preliminary encoding; The features of all pixels in the column with width w in the c-th channel; Let be the feature of all pixels in the row with height h in the c-th channel; δ() is the ReLU6 activation function. In practice, it can be defined as δ(x) = min(6, max(x, 0)). This is the intermediate feature vector after the initial encoding of the max pooling path.

[0089] Finally, the spatial coordinate information of the X and Y coordinates is concatenated, as shown in the following formula (5);

[0090]

[0091] (5);

[0092] In formula (5), Y(i,j) is the output feature map after splicing spatial coordinate information, which is a 2C-dimensional vector; Concat() is the splicing function; σ is the Sigmoid activation function, which maps the convolution output to the (0, 1) interval and serves as the attention weight. Let H be the horizontal attention weight matrix, with shape H × 1 × C. It represents the importance of each height position (row) to each channel. Let i be the C-dimensional weight vector of the horizontal attention weight matrix at the i-th row (height). Y is the vertical attention weight matrix, with a shape of 1×W×C. It represents the importance of each width position (column) to each channel. 'j' is the C-dimensional weight vector of the vertical attention weight matrix at the j-th column (width). F(i,j) is the C-dimensional vector of the original input feature map at position (i,j). m (i, j) represents the feature of the final output of the max pooling path at position (i, j); it is modulated by attention in both the horizontal and vertical directions, thereby enhancing the long-range spatial dependency features associated with position (i, j). a (i,j) represents the feature of the final output of the average pooling path at position (i,j).

[0093] Finally, by combining all Y(i,j) in formula (5), we obtain the output feature map with spatial location information.

[0094] The LFGP module uses an overlapping patch partitioning strategy to enhance feature interaction and learns finer local details through the following formula (6);

[0095] (6);

[0096] In formula (6), MSA() is the multi-head self-attention function, which is the core operation function of the Transformer architecture; Xout is the output of MSA(), Xo The input to MSA() is the output from the previous layer; W Q W K and W V It is a weight matrix shared by all patches, used to weight the input feature X. o The projection is onto three different representation spaces, used to compute queries, keys, and values, respectively.

[0097] like Figure 7 The LFGP modules reside in layers three and four, receiving output features from layers two and three respectively, and then performing MV2↓ (downsampling) on ​​the results. The DCSP module above receives output features from the first two MV2 convolutional layers. The MobileViT modules reside in layers three, four, and five. Layers three and four receive features refined by LFGP, while layer five integrates high-level semantic information, receiving features processed by layers one through four.

[0098] like Figure 7 The MobileViT model of this invention adopts a hierarchical design, fusing convolution and attention mechanisms at different levels to form a feature extraction process from local to global:

[0099] (1) Assume that the original input feature x has dimensions [B, 3, 256, 256] (batch size × number of channels × height × width) and is input into the first layer; B is the batch size. x is processed by 3×3 convolution and multiple MV2 convolution blocks. Each block is processed through expansion → depthwise convolution → compression to gradually enhance the feature expression capability, keeping the dimensions [B, 16, 128, 128] unchanged and input into the second layer.

[0100] (2) Then, downsampling is performed through the MV2↓ module, the number of channels C is increased to 32, the spatial size is halved, and the feature dimension becomes [B, 32, 64, 64]; the DCSP module enhances the spatial perception capability of the mid-layer features through the dual paths of mean pooling and max pooling, combined with coordinate attention in the horizontal and vertical directions, providing an enhanced feature foundation for subsequent processing, and inputting the processed features into the third layer.

[0101] (3) Both the third and fourth layers process features in different dimensions (third layer [B, 64, 32, 32], fourth layer [B, 80, 16, 16]). The LFGP module extracts fine-grained local features through a dynamic patching strategy to enhance local detail representation. The MobileViT module serializes the features, performs global dependency modeling through Transformer, and then inputs them into the next layer.

[0102] (4) The fifth layer performs the final integration, fusing the multi-scale and multi-granular features extracted from the first four layers into a unified semantic representation, providing discriminative features for classification tasks, with dimensions of [B, 96, 8, 8].

[0103] (5) The sixth layer outputs the classification results through channel expansion (dimension [B, 960, 8, 8]), global pooling, and fully connected layers. The entire process forms a spatial compression from 256×256 to 8×8 and a semantic deepening from 3 channels to 960 channels. Through the alternating use of convolution, attention, and downsampling, a hierarchical visual representation is gradually constructed.

[0104] The Vision Transformer module uses the MobileViT module from the MobileViT network. MobileViT is a lightweight Transformer architecture designed for mobile and edge devices, aiming to combine the local modeling advantages of Convolutional Neural Networks (CNNs) with the global dependency capture capabilities of Visual Transformers (ViTs).

[0105] like Figure 4 In the process of global context modeling, the Vision Transformer module, in order to learn a global representation with spatial inductive bias, takes the input X... L Expand into N non-overlapping flat patches X U See formula (7) below;

[0106] (7);

[0107] In formula (7), P is the number of patches, and X is the number of patches. U (p) is to make X L The p-th flat patch after expansion; h ≤ n and w ≤ n are the height and width of the patch, respectively. Then, for each patch, the inter-block relationship is encoded by applying Transformer, as shown in the following formula (8);

[0108] (8);

[0109] In formula (8), this ensures that neither the patch order nor the spatial order of pixels within each patch is lost. Therefore, we can further fold X. G (p) Obtain the output X F .

[0110] like Figure 4 Assume the input X∈R H×W×CH, W, and C represent the height, width, and number of channels of the input feature map X, respectively.

[0111] (1) By using 1×1 convolution (pointwise convolution), the number of channels is projected from C to a higher dimension d (d > C) to obtain XL: XL∈ R H×W×d High-dimensional features that contain local information.

[0112] (2) Divide the two-dimensional feature map XL into a series of non-overlapping, flattened image blocks X. U P = w * h: Number of pixels per image patch. N = (H * W) / P: Total number of image patches. X U For the expanded tensor, X U (p) ∈R N×d For a fixed internal position p, extract the features of all N image patches at that position; X U It is a sequence of shape (N, d) containing pixel features distributed in different regions of the image but in the same local relative position.

[0113] (3)X G (p) ∈ R N×d This is the output sequence after processing by the Transformer. It is folded back into its original spatial location, reassembling the P sequences into a single two-dimensional feature map, following the original image block division method. X F ∈ R H ×W×d It is a feature map that preserves the original spatial structure (pixel order, block order), where each pixel feature incorporates global information. Finally, X... F The output of the Vision Transformer module is the projection from the high-dimensional space d back to the low-dimensional space C.

[0114] This invention also discloses an ancient character recognition system based on a hybrid framework of CNN and Vision Transformer, including an ancient character data acquisition module, an ancient character preprocessing module, an image generation module, a data augmentation module, a context modeling module, a spatial information extraction module, a fine-grained feature extraction module, and an ancient character recognition module;

[0115] The ancient script data acquisition module is used to collect images of ancient scripts, classify and label all ancient script images, and obtain the original dataset Dy.

[0116] The ancient script preprocessing module is used to crop the ancient script images and count the number n of ancient script images in each category of the ancient script image classification.

[0117] The image generation module is used to generate pseudo data from the preprocessed ancient script dataset using a self-attention generative adversarial network (SAGAN), and then adds the pseudo data to the original dataset Dy to obtain an expanded dataset Dk.

[0118] The data augmentation module is used to perform data augmentation processing on the expanded dataset Dk to obtain the complete dataset Dw;

[0119] The context modeling module uses the MobileViT module to model the global context;

[0120] The spatial information extraction module uses the DCSP module to obtain the spatial coordinate information of the X and Y coordinates of the image in parallel using average pooling and max pooling operations;

[0121] The fine-grained feature extraction module uses the LFGP module to employ an overlapping patch partitioning strategy to enhance feature interaction, thereby enabling the learning of finer local details.

[0122] The ancient script recognition module uses a hybrid pyramid framework SAF-ViT network combining CNN and Vision Transformer to train and classify ancient scripts, thereby enabling the recognition of ancient scripts and outputting the recognition results.

[0123] like Figure 2 and Figure 7 , Figure 2 The context modeling module in the middle is Figure 7 The third and fourth layer MobileViT modules; the spatial information extraction module is Figure 7 The second layer of the DCSP module. The fine-grained feature extraction module is... Figure 7 The third and fourth layers are LFGP modules. The Self-Attention Generative Adversarial Network (SAGAN) consists of a total of 6 layers: Layer 2 (DCSP) → establishes a spatial framework, enabling understanding of the macroscopic spatial layout of ancient characters and outputting spatially perceptive features → passed to subsequent layers for processing. Layer 3 / 4 (LFGP) → supplements local details, capturing microscopic features such as strokes, transitions, and noise-affected areas, outputting fine-grained features → fused with global features. Layer 3 / 4 (MobileViT) → integrates global context, establishing long-term dependencies between character components and outputting context-rich features → used for final classification.

[0124] Logical Relationship: The DCSP module provides the feature foundation for the MobileViT module to enhance spatial awareness: DCSP enhances the sensitivity of features to spatial coordinates, and MobileViT establishes global dependencies based on spatial enhancement.

[0125] The LFGP module and the MobileViT module are complementary. The LFGP module is responsible for fine-grained feature extraction and local feature extraction, while the MobileViT module works collaboratively at layers 3 and 4 for global relationship modeling.

[0126] This invention presents an ancient script recognition method and system based on a hybrid CNN and Vision Transformer framework. When collecting images of ancient scripts (bamboo slips), all images are included in the dataset regardless of whether they are heavily noisy or partially incomplete. After classifying and labeling the images, a generative adversarial network is applied to generate pseudo-data for categories with only a few images to expand the dataset. Then, special data augmentation is applied to perfect the dataset, effectively addressing the problem of limited unearthed ancient scripts leading to small dataset sizes and the influence of long-tailed category distributions. The MobileViT module is used to model the global context, enabling the learning of global representations with spatial inductive bias. An innovative hybrid CNN and Vision Transformer pyramid framework is proposed and used, combining spatial prior information extraction from the DCSP module and fine-grained feature extraction from the LFGP module to obtain more effective information. Finally, the proposed SAF-ViT model is used to achieve the classification and recognition of ancient scripts.

[0127] To achieve effective recognition of ancient Chinese characters (bamboo slips), this invention proposes a data augmentation strategy that combines Self-Attention Generative Adversarial Network (SAGAN) with adaptive data augmentation. This data augmentation strategy is specifically designed to address the issue of the small size of ancient Chinese character datasets, which are easily affected by long-tailed class distributions. It aims to enhance the representation of tail classes, ultimately producing a comprehensive and class-balanced dataset. This invention also proposes a hybrid CNN and Vision Transformer framework (SAF-ViT) model specifically for ancient Chinese character recognition. This hybrid CNN and Vision Transformer framework model utilizes CNNs capable of capturing long-range dependencies to extract and downsample local features from ancient Chinese character images, while simultaneously employing Transformers to model global contextual information, overcoming the limitations of traditional Chinese character recognition. This invention also introduces two key components: (1) Local Region Self-Attention (LFGP), which enhances the extraction of fine-grained local features; and (2) Dual-Coordinate Spatial Location Feature Extraction (DCSP), which provides crucial spatial prior information to guide feature learning.

[0128] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0129] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An ancient character recognition method based on a hybrid framework of CNN and Vision Transformer, characterized in that, Includes the following steps: Step 1: Construct the original dataset Dy; collect images of ancient characters and filter, classify and label them to obtain the original dataset Dy; Step 2: Preprocess the ancient text images in the original dataset Dy; crop the ancient text images to a predetermined standard size, and count the number n of ancient text images in each category; Step 3: Expand the original dataset Dy; the number of ancient text images n is less than the preset value n. y The original dataset Dy is expanded by classifying the dataset and generating pseudo-data to obtain the expanded dataset Dk. Step 4: Construct the complete dataset; Perform data augmentation on the expanded dataset Dk to obtain the complete dataset Dw; Step 5: Construct a hybrid CNN and Vision Transformer framework model; the hybrid CNN and Vision Transformer framework model integrates contextual information, extracts spatial location information, and extracts fine-grained features; Step 6: Classify and identify ancient scripts using the pyramid structure network model described above.

2. The ancient character recognition method based on a hybrid framework of CNN and Vision Transformer as described in claim 1, characterized in that, In step 2, the ancient text image is cropped to a size of 256×256.

3. The ancient character recognition method based on a hybrid framework of CNN and Vision Transformer as described in claim 1, characterized in that, In step 2, after counting the number n of ancient text images in each category, the imbalance intensity BPH of the data distribution in the original dataset Dy is calculated.

4. The ancient character recognition method based on a hybrid framework of CNN and Vision Transformer as described in claim 1, characterized in that, In step 3, the self-attention generative adversarial network SAGAN is used as the image generation module to generate pseudo data.

5. The ancient character recognition method based on a hybrid framework of CNN and Vision Transformer as described in claim 4, characterized in that, In step 3, the self-attention generative adversarial network (SAGAN) includes a generator and a discriminator.

6. The ancient character recognition method based on a hybrid framework of CNN and Vision Transformer as described in claim 1, characterized in that, In step 3, during the process of constructing the complete dataset, the header data is first subjected to random undersampling before data augmentation.

7. The ancient character recognition method based on a hybrid framework of CNN and Vision Transformer as described in claim 1, characterized in that, In step 4, during the data augmentation process, data augmentation is performed based on the augmentation intensity coefficient λ.

8. The ancient character recognition method based on a hybrid framework of CNN and Vision Transformer as described in claim 1, characterized in that, The data augmentation processes include flipping, fading, scratch removal, noise addition, and blurring.

9. The ancient character recognition method based on a hybrid framework of CNN and Vision Transformer as described in claim 1, characterized in that, In step 5, the hybrid CNN and Vision Transformer framework model includes a shallow CNN module, a DCSP module, a deep LFGP module, and a Vision Transformer module.

10. An ancient script recognition system based on a hybrid framework of CNN and Vision Transformer, characterized in that, It includes modules for ancient script data acquisition, ancient script preprocessing, image generation, data augmentation, context modeling, spatial information extraction, fine-grained feature extraction, and ancient script recognition. The ancient script data acquisition module is used to collect images of ancient scripts, classify and label all ancient script images, and obtain the original dataset Dy. The ancient script preprocessing module is used to crop the ancient script images and count the number n of ancient script images in each category of the ancient script image classification. The image generation module is used to generate pseudo data from the preprocessed ancient script dataset using a self-attention generative adversarial network (SAGAN), and then adds the pseudo data to the original dataset Dy to obtain an expanded dataset Dk. The data augmentation module is used to perform data augmentation processing on the expanded dataset Dk to obtain the complete dataset Dw; The context modeling module uses the MobileViT module to model the global context; The spatial information extraction module uses the DCSP module to obtain the spatial coordinate information of the X and Y coordinates of the image in parallel using average pooling and max pooling operations; The fine-grained feature extraction module uses the LFGP module to employ an overlapping patch partitioning strategy to enhance feature interaction, thereby enabling the learning of finer local details. The ancient script recognition module uses a hybrid pyramid framework SAF-ViT network combining CNN and Vision Transformer to train and classify ancient scripts, thereby enabling the recognition of ancient scripts and outputting the recognition results.

Images