Similar handwritten Chinese character recognition method based on feature fusion TL-ResNet18 network

Abstract

This invention relates to a similar handwritten Chinese character recognition method based on a feature fusion-based TL-ResNet18 network. It obtains the similarity probability between handwritten Chinese characters by training a neural network and replaces the distance index in the density peak clustering algorithm with this probability. This overcomes the limitation of traditional density peak clustering in classifying the high-dimensional feature vectors represented by images, and can improve the recognition rate of printed Chinese characters within tables by up to 5.71%. A feature fusion-based TL-ResNet18 network is designed, which, based on the traditional ResNet-18 model, incorporates a deep transfer learning model and a feature fusion module, increasing the recognition rate of similar handwritten Chinese characters to 97.06% and the overall recognition rate of handwritten Chinese characters to 97.45%.

Description

Technical Field

[0001] This invention relates to a similar handwritten Chinese character recognition method based on feature fusion TL-ResNet18 network, belonging to the field of handwritten Chinese character recognition technology. Background Technology

[0002] With the continuous development of technology, computers, mobile phones, and other devices have become important tools for image processing. Images contain a large amount of Chinese character information, and Chinese characters are the core of Chinese information processing, holding significant importance. For computers and other devices, the Chinese character information in images needs preliminary processing and feature extraction before it can be recognized and used. To improve the processing capabilities of computers and other devices for Chinese character information, Chinese character recognition technology has emerged. This technology converts Chinese character images into digital information and has been widely applied in various aspects of life, such as digital document management, enterprise office automation, invoice management, and license plate recognition. The development and application of Chinese character recognition technology have made the processing and transmission of Chinese information more convenient and efficient, while simultaneously driving the advancement of Chinese information towards digitization, intelligence, and automation.

[0003] Handwritten Chinese character recognition is far more difficult than printed Chinese character recognition due to the irregular handwriting and other factors. Depending on the method of operation, handwritten Chinese character recognition can be divided into online and offline methods. Online handwritten Chinese character recognition refers to the real-time recognition of Chinese characters written on electronic devices. These characters have strict stroke order information and real-time processing, resulting in a higher recognition rate. Offline handwritten Chinese character recognition refers to the recognition of handwritten Chinese characters existing in images. Due to the wide variety of types, complex structures, and diverse handwriting styles, offline handwritten Chinese characters have a high degree of similarity, which is the key challenge and focus of handwritten Chinese character recognition. Furthermore, the lack of stroke order information makes recognition even more difficult, leading many researchers to conduct extensive research and propose numerous algorithms for offline handwritten Chinese character recognition.

[0004] Existing methods typically use confusion matrices to extract similar handwritten Chinese character datasets and design targeted classifiers to improve recognition rates. However, confusion matrices have the limitation of not being able to perform multi-class classification on similar handwritten Chinese character datasets, thus increasing the dataset's storage space. Furthermore, existing methods still suffer from low recognition accuracy when identifying similar handwritten Chinese characters, which reduces the overall recognition rate. Summary of the Invention

[0005] To overcome the shortcomings of existing research, this invention provides a similar handwritten Chinese character recognition method based on a feature fusion-based TL-ResNet18 network. By training a neural network to obtain the similarity probability between handwritten Chinese characters, this method replaces the distance index in the density peak clustering algorithm, overcoming the limitation of traditional density peak clustering in classifying high-dimensional feature vectors represented by images. This method can improve the recognition rate of printed Chinese characters within tables by up to 5.71%. Furthermore, a feature fusion-based TL-ResNet18 network is designed. Based on the traditional ResNet-18 model, a deep transfer learning model and a feature fusion module are added, increasing the recognition rate of similar handwritten Chinese characters to 97.06% and the overall recognition rate of handwritten Chinese characters to 97.45%.

[0006] The specific steps of the similar handwritten Chinese character recognition method based on feature fusion TL-ResNet18 network are as follows:

[0007] Step 1: Input the image of the handwritten Chinese character;

[0008] Step 2: Training the ResNet-18 neural network;

[0009] Step 3: Classify handwritten Chinese characters;

[0010] Step 4: Divide the similar handwritten Chinese character dataset using the density peak clustering method;

[0011] To address the issue of increased storage space caused by the inability of confusion matrices to perform multi-classification of similar handwritten Chinese character datasets, a multi-classification algorithm based on density peak clustering for similar handwritten Chinese character datasets is proposed. This algorithm obtains the similarity probability between handwritten Chinese characters by training a neural network and replaces the distance metric in the density peak clustering algorithm with this probability, overcoming the limitation of traditional density peak clustering in classifying the high-dimensional feature vectors represented by multiple images. To solve the problem of poor performance of existing classifiers in similar handwritten Chinese character recognition, this invention presents a TL-ResNet18 network design method based on feature fusion.

[0012] Step 5: Identify similar Chinese characters using the TL-ResNet18 network based on feature fusion.

[0013] The density peak clustering method in step four specifically includes:

[0014] S4.1: Calculate the x values ​​of the two data points. i and x j Distance P between ij ':

[0015] P ij =1-P ij (1)

[0016] Among them, P ij P represents the confusion probability obtained from training the ResNet-18 network. In traditional density peak clustering, the Euclidean distance metric measures the similarity between two features; the smaller the Euclidean distance value, the more similar the two features are. ij This means that the larger the value, the more similar the two categories are; therefore, P needs to be adjusted. ij Transform into P in formula (1) ij ' indicates that P ij The smaller the value, the more similar the two data points are;

[0017] S4.2: Calculate the local density of data points using a Gaussian kernel. The specific formula is as follows:

[0018]

[0019] Where, d c It is the cutoff distance, let d. ij The values ​​at the 1% to 2% level after sorting in ascending order;

[0020] S4.3: Calculate the distance δ between a data point and its nearest higher-density point. i :

[0021]

[0022] If a data point is a point of maximum local density, its distance is calculated as follows:

[0023]

[0024] The values ​​of local density ρ and distance δ were obtained. A ρ-δ decision graph was constructed to represent the distribution of all data points in the ρ-δ space. Points with higher ρ and δ in the decision graph were selected as cluster centers.

[0025] Using γ in formula (5) as the single criterion for cluster center selection, the formula is as follows:

[0026] γ=ρ*δ (5)

[0027] According to the above formula, the data points are sorted in descending order of γ to construct a γ decision map; the data points with larger γ values ​​are determined as cluster centers, and the remaining non-center data points are assigned to the clusters of the nearest higher-density points, thus completing the clustering process.

[0028] Step five specifically includes:

[0029] S5.1: Using the ImageNet dataset as the source domain for transfer learning experiments, the ResNet-18 network was built and trained using the PyTorch framework, and the trained weight parameters were saved. This model can be expressed by the following mathematical equation:

[0030] F S =f ResNet-18 (ω,ω fc ;x i (6) S5.2: The similar handwritten Chinese character dataset based on improved density peak clustering is used as the target domain for the transfer learning experiment. Since ResNet-18 and the feature fusion-based TL-ResNet18 network have the same convolutional layer structure, some weights of the convolutions trained in the source domain can be extracted and loaded into the convolutional part of TL-ResNet18. The weights loaded into convolutional layer 1 are frozen, meaning that these weights do not participate in the model training. Only the network layers after convolutional layer 1 need to be trained. The TL-ResNet18 network can be represented by the following formula:

[0031] F T =f TL-ResNet18 (ω,ω new-fc ;x i (7)

[0032] Where F T The output of the model is represented by ω, which represents the weight parameter of the convolutional part in the deep transfer convolutional neural network. This parameter is the same as the parameter in equation (6). new-fc The weight parameters x represent the network layers trained after convolutional layer 1 of the model. i Training data representing the target domain.

[0033] S5.3: Based on the TL-ResNet18 network built in S5.2, a feature fusion module is added. This module performs adjacent convolutional layer fusion, cross convolutional layer fusion, and three-layer convolutional layer fusion on the four convolutional layers of the TL-ResNet18 network respectively, and designs ten TL-ResNet18 networks based on feature fusion.

[0034] S5.4: The ten networks built in S5.3 were subjected to ablation experiments on a dataset of similar handwritten Chinese characters. Based on the TL-ResNet18 network, the adjacent layers, cross-convolutional layers, and three-layer convolutional layers of the network were fused to identify similar Chinese characters.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] This invention proposes an improved density peak clustering algorithm to address the problems in existing density peak clustering algorithms, such as the inability of distance metrics to measure the distance between high-dimensional feature vectors represented by handwritten Chinese characters and the difficulty in multi-classifying similar handwritten Chinese character datasets, thereby improving the recognition rate of similar handwritten Chinese characters.

[0037] This invention proposes a TL-ResNet18 network based on feature fusion. This network makes full use of the deep transfer learning model and feature fusion module. The TL-ResNetF3 network, which fuses convolutional layer 4 and convolutional layer 5, better preserves the difference information between similar handwritten Chinese characters, thus achieving the best recognition effect. Attached Figure Description

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 This is a flowchart of the similar handwritten Chinese character recognition method based on feature fusion TL-ResNet18 network of the present invention;

[0040] Figure 2 This is a schematic diagram of the TL-ResNet18 network based on feature fusion according to the present invention;

[0041] Figure 3 This is a schematic diagram of the feature fusion module of adjacent convolutional layers in this invention;

[0042] Figure 4 This is a schematic diagram of the ResNet-18 network based on cross-layer feature fusion according to the present invention;

[0043] Figure 5 This is a diagram of the three-layer fusion network structure of the present invention;

[0044] Figure 6 Here are the loss function curves of the ten fusion networks of this invention (1);

[0045] Figure 7 Here are the loss function curves of the ten fusion networks of this invention (2);

[0046] Figure 8 This is a comparison curve of the top-1 accuracy of the ResNet-18 network and the TL-ResNet18 network of this invention.

[0047] Figure 9 This is a comparison chart of the top-1 accuracy of the fusion network of adjacent convolutional layers in this invention;

[0048] Figure 10 This is a comparison chart of the top-1 accuracy of the cross-layer feature fusion network of this invention;

[0049] Figure 11 This is a comparison curve of the top-1 accuracy of the three-layer convolutional layer fusion network of the present invention;

[0050] Figure 12 This is a comparison curve of the top-1 accuracy of the TL-ResNetF3 and TL-ResNetF6 networks of this invention.

[0051] Figure 13 These are the eight network loss function curves of this invention;

[0052] Figure 14 This is a comparison chart of the top-1 accuracy of the ResNet-18 and TL-ResNet18 networks of this invention;

[0053] Figure 15 A comparison of visual features extracted using ResNet-18 and TL-ResNetF3 networks for some handwritten Chinese characters in this invention (1);

[0054] Figure 16 A comparison of visual features extracted using ResNet-18 and TL-ResNetF3 networks for some handwritten Chinese characters in this invention (2);

[0055] Figure 17 Comparison of visual features extracted using ResNet-18 and TL-ResNetF3 networks for some handwritten Chinese characters in this invention (3). Detailed Implementation

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0057] The specific steps of the similar handwritten Chinese character recognition method based on feature fusion TL-ResNet18 network are as follows:

[0058] Step 1: Input the image of the handwritten Chinese character;

[0059] Step 2: Training the ResNet-18 neural network;

[0060] Step 3: Classify handwritten Chinese characters;

[0061] Step 4: Divide the similar handwritten Chinese character dataset using the density peak clustering method;

[0062] To address the issue of increased storage space caused by the inability of confusion matrices to perform multi-classification of similar handwritten Chinese character datasets, a multi-classification algorithm based on density peak clustering for similar handwritten Chinese character datasets is proposed. This algorithm obtains the similarity probability between handwritten Chinese characters by training a neural network and replaces the distance metric in the density peak clustering algorithm with this probability, overcoming the limitation of traditional density peak clustering in classifying the high-dimensional feature vectors represented by multiple images. To solve the problem of poor performance of existing classifiers in similar handwritten Chinese character recognition, this invention presents a TL-ResNet18 network design method based on feature fusion.

[0063] Step 5: Identify similar Chinese characters using the TL-ResNet18 network based on feature fusion.

[0064] The density peak clustering method in step four specifically includes:

[0065] S4.1: Calculate the x values ​​of the two data points. i and x j Distance P between ij ':

[0066] P ij =1-P ij (1)

[0067] Among them, P ij P represents the confusion probability obtained from training the ResNet-18 network. In traditional density peak clustering, the Euclidean distance metric measures the similarity between two features; the smaller the Euclidean distance value, the more similar the two features are. ij This means that the larger the value, the more similar the two categories are; therefore, P needs to be adjusted. ij Transform into P in formula (1) ij ' indicates that P ij The smaller the value, the more similar the two data points are;

[0068] S4.2: Calculate the local density of data points using a Gaussian kernel. The specific formula is as follows:

[0069]

[0070] Where, d c It is the cutoff distance, let d. ij The values ​​at the 1% to 2% level after sorting in ascending order;

[0071] S4.3: Calculate the distance δ between a data point and its nearest higher-density point. i :

[0072]

[0073] If a data point is a point of maximum local density, its distance is calculated as follows:

[0074]

[0075] The values ​​of local density ρ and distance δ were obtained. A ρ-δ decision graph was constructed to represent the distribution of all data points in the ρ-δ space. Points with higher ρ and δ in the decision graph were selected as cluster centers.

[0076] Using γ in formula (5) as the single criterion for cluster center selection, the formula is as follows:

[0077] γ=ρ*δ (5)

[0078] According to the above formula, the data points are sorted in descending order of γ to construct a γ decision map; the data points with larger γ values ​​are determined as cluster centers, and the remaining non-center data points are assigned to the clusters of the nearest higher-density points, thus completing the clustering process.

[0079] Detailed Algorithm for Improving Density Peak Clustering to Divide Similar Handwritten Chinese Character Datasets

[0080] Input: Images of handwritten Chinese characters dataset

[0081] 1. Train the ResNet-18 neural network on a handwritten Chinese character dataset;

[0082] 2. Train the ResNet-18 neural network to obtain the confusion matrix, and calculate the confusion probability P between handwritten Chinese characters. ij ;

[0083] 3. Based on the similarity probability formula and the Euclidean distance formula, the confusion probability in step two is transformed into the distance index in equation (2);

[0084] 4. Calculate the local density of data points according to formula (3);

[0085] 5. Calculate the distance between the data point and the nearest point with higher density according to formula (4);

[0086] 6. Calculate the γ value of each point according to formula (5), and select the cluster center points according to the descending order of the γ values;

[0087] 7. Assign non-central data points to the cluster containing the nearest, higher-density point;

[0088] 8. Output the cluster labels of the data points to complete the clustering process.

[0089] Output: Similar handwritten Chinese character dataset.

[0090] TL-ResNet18 network based on feature fusion:

[0091] By incorporating a deep transfer learning model and a feature fusion module into the traditional ResNet-18 model, the recognition rate of handwritten Chinese characters is improved. The similar handwritten Chinese character recognition process based on feature fusion using TL-ResNet18 is as follows: Figure 1 As shown.

[0092] A feature fusion module was added to the TL-ResNet18 network, completing the construction of a ResNet-18 network based on deep transfer learning and feature fusion. Its structure is as follows: Figure 2 As shown. The network architecture designed in this invention is as follows:

[0093] (1) Using the ImageNet dataset as the source domain for the transfer learning experiment, the ResNet-18 network was used to train the ImageNet dataset to obtain the weights for training the model.

[0094] (2) Import the weights of the first convolutional layer of the ResNet-18 network after training into the handwritten Chinese character recognition network based on ResNet-18 network;

[0095] (3) Based on this network, convolutional layers 2, 3, 4 and 5 were fused by adjacent layers, cross layers and three convolutional layers respectively, thus building ten TL-ResNet18 networks based on feature fusion.

[0096] These ten networks are:

[0097] A TL-ResNet18 network based on feature fusion of convolutional layer 2 and convolutional layer 3;

[0098] A TL-ResNet18 network based on feature fusion of convolutional layer 3 and convolutional layer 4;

[0099] A TL-ResNet18 network based on feature fusion of convolutional layer 4 and convolutional layer 5;

[0100] A TL-ResNet18 network based on feature fusion of convolutional layer 2 and convolutional layer 5;

[0101] A TL-ResNet18 network based on feature fusion of convolutional layer 2 and convolutional layer 4;

[0102] A TL-ResNet18 network based on feature fusion of convolutional layer 3 and convolutional layer 5;

[0103] A TL-ResNet18 network based on feature fusion of convolutional layer 2, convolutional layer 3 and convolutional layer 4;

[0104] A TL-ResNet18 network based on feature fusion of convolutional layer 2, convolutional layer 3 and convolutional layer 5;

[0105] A TL-ResNet18 network based on feature fusion of convolutional layer 2, convolutional layer 4 and convolutional layer 5;

[0106] The TL-ResNet18 network is based on feature fusion of convolutional layers 3, 4, and 5.

[0107] These ten networks are abbreviated as: TL-ResNetF1, TL-ResNetF2, TL-ResNetF3, TL-ResNetF4, TL-ResNetF5, TL-ResNetF6, TL-ResNetF7, TL-ResNetF8, TL-ResNetF9, and TL-ResNetF10.

[0108] The ResNet-18 network based on feature fusion of adjacent convolutional layers is divided into three types:

[0109] A ResNet-18 network based on feature fusion of convolutional layer 2 and convolutional layer 3;

[0110] A ResNet-18 network based on feature fusion of convolutional layer 3 and convolutional layer 4;

[0111] A ResNet-18 network based on feature fusion of convolutional layer 4 and convolutional layer 5.

[0112] For simplicity, they will be referred to as ResNetF1, ResNetF2, and ResNetF3 networks, respectively. Their feature fusion module is as follows: Figure 3 As shown.

[0113] The specific process of the adjacent convolutional layer feature fusion module is as follows:

[0114] (1) Use an adaptive max pooling layer to compress the height and width dimensions of the feature map output by convolutional layer A to 1, resulting in a 1×1 feature map.

[0115] (2) The compressed feature vector is mapped to the range [0,1] using the softmax function to obtain the weight coefficients on each channel. These weight coefficients are then multiplied by the compressed 1×1 feature map to obtain a new feature vector. The new feature vector is then activated using the ReLU function. The activated feature vector is then passed through an average pooling layer to reduce the height and width of the feature vector.

[0116] (3) The repeat function is used to transform the size of the feature vector to the same size as convolutional layer B, and then the feature vector of convolutional layer A is added to the feature vector of convolutional layer B to achieve feature fusion between convolutional layer A and convolutional layer B. Finally, the fused features are input into subsequent network layers for classification, thereby improving the performance and generalization ability of the model.

[0117] In ResNetF1, convolutional layer 2 replaces convolutional layer A and convolutional layer 3 replaces convolutional layer B; in ResNetF2, convolutional layer 3 replaces convolutional layer A and convolutional layer 4 replaces convolutional layer B; and in ResNetF3, convolutional layer 4 replaces convolutional layer A and convolutional layer 5 replaces convolutional layer B.

[0118] ResNet-18 networks based on cross-layer feature fusion are divided into three types:

[0119] A ResNet-18 network based on feature fusion of convolutional layer 2 and convolutional layer 5;

[0120] A ResNet-18 network based on feature fusion of convolutional layer 2 and convolutional layer 4;

[0121] A ResNet-18 network based on feature fusion of convolutional layer 3 and convolutional layer 5.

[0122] These are respectively referred to as ResNetF4, ResNetF5, and ResNetF6 networks. Their feature fusion modules are as follows: Figure 4 As shown, this module fuses shallow and deep features from cross-layer convolutional layers to enhance the model's expressive power.

[0123] The specific process of the cross-layer feature fusion module is as follows:

[0124] 1) Perform a convolution operation with a 1×1 kernel on convolutional layer A. The purpose is to compress the size of convolutional layer A to the same image size as convolutional layer B.

[0125] 2) The compressed convolutional layer A is downsampled to the same height and width as convolutional layer B by an adaptive average pooling layer, and then added to the feature vector of convolutional layer B to obtain a vector that integrates the features of convolutional layer A and convolutional layer B.

[0126] 3) The fused features are input into subsequent network layers for classification, which improves the model's feature extraction capabilities.

[0127] In ResNetF4, convolutional layer 2 replaces convolutional layer A and convolutional layer 5 replaces convolutional layer B; in ResNetF5, convolutional layer 2 replaces convolutional layer A and convolutional layer 4 replaces convolutional layer B; and in ResNetF6, convolutional layer 3 replaces convolutional layer A and convolutional layer 5 replaces convolutional layer B.

[0128] The ResNet-18 network based on feature fusion of three convolutional layers is divided into four types:

[0129] A ResNet-18 network based on feature fusion of convolutional layer 2, convolutional layer 3 and convolutional layer 4;

[0130] A ResNet-18 network based on feature fusion of convolutional layer 2, convolutional layer 3 and convolutional layer 5;

[0131] A ResNet-18 network based on feature fusion of convolutional layer 2, convolutional layer 4 and convolutional layer 5;

[0132] A ResNet-18 network based on feature fusion of convolutional layers 3, 4, and 5.

[0133] These are respectively referred to as ResNetF7, ResNetF8, ResNetF9, and ResNetF10 networks, and their network architectures are as follows: Figure 5 As shown.

[0134] The specific process of the three-layer feature fusion module is as follows:

[0135] 1) Merge convolutional layers A and C across layers or adjacent layers, and also merge convolutional layers B and C across layers or adjacent layers. Convolutional layer C represents the deepest feature layer, while convolutional layers A and B represent shallow features.

[0136] 2) The features fused from convolutional layers A and C are added together with the features fused from convolutional layers B and C to obtain the features fused from the three convolutional layers A, B, and C.

[0137] 3) Input the fused features into subsequent network layers for classification to improve the model's feature extraction capabilities.

[0138] Step five specifically includes:

[0139] S5.1: Using the ImageNet dataset as the source domain for transfer learning experiments, the ResNet-18 network was built and trained using the PyTorch framework, and the trained weight parameters were saved. This model can be expressed by the following mathematical equation:

[0140] F S =f ResNet-18 (ω,ω fc ;x i(6) S5.2: The similar handwritten Chinese character dataset based on improved density peak clustering is used as the target domain for the transfer learning experiment. Since ResNet-18 and the feature fusion-based TL-ResNet18 network have the same convolutional layer structure, some weights of the convolutions trained in the source domain can be extracted and loaded into the convolutional part of TL-ResNet18. The weights loaded into convolutional layer 1 are frozen, that is, these weights do not participate in the training of the model. Only the network layers after convolutional layer 1 need to be trained. The TL-ResNet18 network can be represented by the following formula:

[0141] F T =f TL-ResNet18 (ω,ω new-fc ;x i (7)

[0142] Where F T The output of the model is represented by ω, which represents the weight parameter of the convolutional part in the deep transfer convolutional neural network. This parameter is the same as the parameter in equation (6). new-fc The weight parameters x represent the network layers trained after convolutional layer 1 of the model. i Training data representing the target domain.

[0143] S5.3: Based on the TL-ResNet18 network built in S5.2, a feature fusion module is added. This module performs adjacent convolutional layer fusion, cross convolutional layer fusion, and three-layer convolutional layer fusion on the four convolutional layers of the TL-ResNet18 network respectively, and designs ten TL-ResNet18 networks based on feature fusion.

[0144] S5.4: The ten networks built in S5.3 were subjected to ablation experiments on a segmented dataset of similar handwritten Chinese characters. Using the traditional ResNet-18 network as a baseline, a deep transfer model was added to this network to verify whether TL-ResNet18 can improve the recognition rate of similar handwritten Chinese characters. Then, using the TL-ResNet18 network as a baseline, the adjacent layers, cross-convolutional layers, and three-layer convolutional layers of this network were fused respectively, and the experimental results were compared with those of the TL-ResNet18 network to verify whether the feature fusion module can improve the recognition rate of similar handwritten Chinese characters. The ten networks were compared and analyzed on the HWDB1.0 and HWDB1.1 datasets.

[0145] An experiment to recognize handwritten Chinese characters:

[0146] In the ablation experiment for similar handwritten Chinese character recognition, the performance of these four networks on the similar handwritten Chinese character classification task was comprehensively evaluated using evaluation metrics such as loss function, accuracy, precision, recall, F1 score, top-5 accuracy, and top-10 accuracy. This also allowed for a relatively objective assessment of the advantages and disadvantages of different methods. All models used the cross-entropy loss function. The experimental results are as follows: Figure 6 and Figure 7 As shown. Figure 6 and Figure 7 (a)-(l) are the loss function curves for ResNet-18, TL-ResNet18, TL-ResNetF1, TL-ResNetF2, TL-ResNetF3, TL-ResNetF4, TL-ResNetF5, TL-ResNetF6, TL-ResNetF7, TL-ResNetF8, TL-ResNetF9, and TL-ResNetF10 networks, respectively. The horizontal axis represents the number of iterations (epochs), and the vertical axis represents the loss function value of each fused network during the testing process. The closer the loss function is to 0, the better the network performance. Figure 6 and Figure 7 Analyzing the ten figures (c)-(l) in the figure, it can be found that the loss function curves of these ten fusion networks gradually converge to 0 as the epoch increases, indicating that these networks are in a good stable state during the test.

[0147] Table 1 shows the six evaluation metrics of the ten feature fusion networks. Taking ResNet-18 as the benchmark, TL-ResNet18, TL-ResNet18 based on adjacent convolutional layer fusion, and TL-ResNet18 based on cross-layer feature fusion all showed significant improvements in the six evaluation metrics. However, the evaluation metrics of the network based on three-layer feature fusion were lower than those of ResNet-18. This indicates that adjacent convolutional layer feature fusion and cross-layer feature fusion modules can effectively improve the recognition rate of similar handwritten Chinese characters, while the network based on three-layer convolutional layer fusion has no effect on improving the recognition rate and may even reduce the recognition rate of traditional networks.

[0148] Table 1 Ablation Experiment

[0149]

[0150] Furthermore, the data analysis in Table 1 shows that the TL-ResNetF3 network performs better in recognizing similar handwritten Chinese characters, with its six evaluation index values ​​being higher than those of the other three networks. This indicates that the TL-ResNetF3 network has the best performance in improving the recognition rate of similar handwritten Chinese characters.

[0151] Using the traditional ResNet-18 network as a benchmark, the top-1 accuracy comparison results of the ResNet-18 network and the TL-ResNet18 network are as follows: Figure 8 As shown in the figure, the horizontal axis represents the number of iterations (epochs), and the vertical axis represents the top-1 accuracy. The graph shows that the ResNet-18 network's top-1 accuracy gradually increases from a value close to 0, reaching a maximum of 95.86%, while the TL-ResNet18 network's top-1 accuracy gradually rises from around 18%, reaching a maximum of 96.07%. This indicates that the TL-ResNet18 network achieved good classification performance in the initial stages of testing.

[0152] Using the TL-ResNet18 network as a benchmark, the top-1 accuracy changes of the TL-ResNet18 network and the TL-ResNet18 network based on adjacent layer fusion during the test are shown in the figure below. Figure 9 As shown. From Figure 9 It can be seen that as the number of epochs increases, the top-1 accuracy of the four networks also increases.

[0153] Furthermore, among the three fusion networks TL-ResNetF1, TL-ResNetF2, and TL-ResNetF3, TL-ResNetF3 demonstrates superior performance in recognizing similar handwritten Chinese characters, achieving a top-1 accuracy 0.84% ​​higher than TL-ResNetF2 and 0.71% higher than TL-ResNetF1. This indicates that, compared to the other two adjacent-layer fusion methods, the TL-ResNetF3 network, based on the fusion of convolutional layers 4 and 5, exhibits a significant performance advantage in recognizing similar handwritten Chinese characters.

[0154] Using the TL-ResNet18 network as a benchmark, the top-1 accuracy changes of the TL-ResNet18 network and the TL-ResNet18 network during the testing process are shown in the following figure. Figure 10 As shown, the top-1 accuracy of the four networks gradually increases with the increase of epochs.

[0155] From Table 1 and Figure 10Analysis shows that the top-1 accuracy of the TL-ResNetF4 network is 0.65% higher than that of the TL-ResNet18 network; the top-1 accuracy of the TL-ResNetF5 network is 0.63% higher than that of the TL-ResNet18 network; and the top-1 accuracy of the TL-ResNetF6 network is 0.93% higher than that of the TL-ResNet18 network. This indicates that cross-layer feature fusion can also improve the recognition rate of similar handwritten Chinese characters to a certain extent. Furthermore, among the three fusion networks (TL-ResNetF4, TL-ResNetF5, and TL-ResNetF6), the TL-ResNetF6 network achieves the best performance in recognizing similar handwritten Chinese characters, with a top-1 accuracy 0.28% higher than that of the TL-ResNetF4 network and 0.3% higher than that of the TL-ResNetF5 network. This indicates that, compared to the other two cross-layer fusion methods, the TL-ResNetF6 network based on the fusion of convolutional layers 3 and 5 has a greater performance advantage in improving the recognition rate of similar handwritten Chinese characters.

[0156] Using the TL-ResNet18 network as a benchmark, the top-1 accuracy changes of the TL-ResNet18 network with three convolutional layers and the TL-ResNet18 network during the testing process are shown in the figure below. Figure 11 As shown, the top-1 accuracy of the four networks gradually increases with the number of epochs. Among them, the highest top-1 accuracies of TL-ResNet18, TL-ResNetF7, TL-ResNetF8, TL-ResNetF9, and TL-ResNetF10 networks reach 96.07%, 92.5%, 92.29%, 92.59%, and 91.96%, respectively.

[0157] Comparative analysis revealed that the six evaluation metrics of TL-ResNetF7, TL-ResNetF8, TL-ResNetF9, and TL-ResNetF10 networks, which are based on feature fusion of three convolutional layers, were all lower than those of the TL-ResNet18 network. Therefore, it can be concluded that the TL-ResNet18 network, based on feature fusion of three convolutional layers, is not suitable for improving the recognition rate of similar handwritten Chinese characters.

[0158] like Figure 12The figure shows a comparison of the top-1 accuracy of the TL-ResNetF3 network (which achieves the best adjacent-layer fusion performance) and the TL-ResNetF6 network (which achieves the best cross-layer fusion performance). Furthermore, analysis of Table 1 reveals that the top-1 accuracy of the TL-ResNetF3 network is 0.21% higher than that of the TL-ResNetF6 network. This indicates that among all fusion networks, the TL-ResNetF3 network exhibits the best performance in recognizing similar handwritten Chinese characters.

[0159] Handwritten Chinese character recognition experiment results

[0160] (1) Ablation test

[0161] This experiment evaluates the network performance by comparing six evaluation metrics of ResNet-18, TL-ResNet18, TL-ResNet18 network with adjacent layer feature fusion, and TL-ResNet18 network with cross-layer feature fusion. Figure 13 As shown, this paper presents the loss function curves of the above eight networks.

[0162] Depend on Figure 13 As the number of training epochs increases, the loss functions of all networks gradually decrease and stabilize, indicating that the performance of these eight networks is relatively stable during testing. Furthermore, by comparing the rate of loss function descent of different networks, it can be found that the loss function descent rate of the TL-ResNet18 network, the adjacent convolutional layer fusion network, and the cross-layer fusion network is slightly faster than that of the traditional ResNet-18 network. This suggests that the adjacent layer fusion and cross-layer fusion networks can reach a stable state more quickly.

[0163] The ablation experiment results are shown in Table 2. The table reveals that compared to the traditional ResNet-18 and TL-ResNet18 networks, the six performance metrics of TL-ResNetF1, TL-ResNetF2, TL-ResNetF3, TL-ResNetF4, TL-ResNetF5, and TL-ResNetF6 networks all show improvement to some extent, indicating that feature fusion between adjacent layers and cross-layer features can effectively improve the recognition rate of handwritten Chinese characters. Furthermore, among all the fused networks, the TL-ResNetF6 network achieves the highest values ​​across all six performance metrics, demonstrating its optimal performance on the entire handwritten Chinese character dataset.

[0164] Table 2 Ablation Experiment

[0165]

[0166]

[0167] like Figure 14 The graph shows a comparison of the top-1 accuracy of the ResNet-18 network and the TL-ResNet18 network. As the number of epochs increases, the top-1 accuracy of both networks also increases, reaching peak values ​​of 95.90% and 96.12%, respectively.

[0168] To fully demonstrate the significant advantages of the TL-ResNetF3 network, which boasts the best recognition performance, in the handwritten Chinese character recognition task, this invention presents a comparison of the visual features of the TL-ResNetF3 network and the basic ResNet-18 network on 15 types of handwritten Chinese characters, as shown in the image. Figure 15-17 As shown in (a)-(o), in each set of images, the left side displays the original image of the handwritten Chinese character, the middle side displays the visual feature map extracted using the traditional ResNet-18 network, and the right side displays the visual feature map extracted by the TL-ResNetF3 model proposed in this patent. The red areas represent the extracted Chinese character features. Based on these images, the differences and advantages / disadvantages of the two models in handwritten Chinese character feature extraction can be clearly compared.

[0169] By observation Figure 15-17 The experimental images clearly show that the proposed model can extract more refined features from handwritten Chinese characters. This network can better focus on key regions of handwritten Chinese characters, such as the differences between similar handwritten characters. In contrast, the feature maps extracted using the ResNet-18 network contain a large number of irrelevant features and regions. The feature maps extracted using the TL-ResNetF3 network exhibit higher positional accuracy, almost accurately locating the key and detailed features of the image.

[0170] Comparative Analysis of Handwritten Chinese Character Recognition Experiment Results:

[0171] Table 3 presents comparative experimental results of different methods proposed in recent years on the HWDB1.0 and HWDB1.1 handwritten Chinese character datasets. Analysis is conducted based on three metrics: accuracy, storage space, and inference time, where inference time represents the computation time for a single sample. The comparison reveals that the model proposed in this patent exhibits excellent performance in handwritten Chinese character recognition.

[0172] Table 3 Comparison of Handwritten Chinese Character Recognition Rate

[0173]

[0174] The Melnyk-Net network is an efficient network with excellent convolutional modules and dimensionality reduction methods; however, compared to the model presented in this paper, it still has higher storage space and inference time. Furthermore, the MSCS+ASA+SCL network combines multi-scale convolutional shuffling modules and attention-based feature space aggregation to improve the model's performance in handwritten Chinese character recognition. Compared to this network, although the accuracy of our patented model is slightly lower, it saves some storage space and inference time, demonstrating significant advantages.

[0175] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A similar handwritten Chinese character recognition method based on feature fusion TL-ResNet18 network, characterized in that: Includes the following steps: Step 1: Input the image of the handwritten Chinese character; Step 2: Training the ResNet-18 neural network; Step 3: Classify handwritten Chinese characters; Step 4: Divide the similar handwritten Chinese character dataset using the density peak clustering method; Step 5: Identify similar Chinese characters using the TL-ResNet18 network based on feature fusion; Step five specifically includes: S5.1: Using the ImageNet dataset as the source domain for transfer learning experiments, the ResNet-18 network was built and trained using the PyTorch framework. The trained weight parameters were saved. The model is expressed by the following mathematical equation: F S =f ResNet-18 (oh,oh fc ;x i ) (6) S5.2: A similar handwritten Chinese character dataset based on density peak clustering is used as the target domain for the transfer learning experiment. ResNet-18 and the feature fusion-based TL-ResNet18 network have the same convolutional layer structure. Partial weights from the convolutions trained in the source domain are extracted and loaded into the convolutional part of TL-ResNet18. The weights loaded into convolutional layer 1 are frozen, meaning these weights do not participate in model training; only the network layers after convolutional layer 1 need to be trained. The TL-ResNet18 network is represented by the following formula: F T =f TL-ResNet18 (oh,oh new-fc ;x i ) (7) Among them F T The output of the model is represented by ω, which represents the weight parameter of the convolutional part in the deep transfer convolutional neural network. This parameter is the same as the parameter in equation (6). new-fc The weight parameters x represent the network layers trained after convolutional layer 1 of the model. i Training data representing the target domain; S5.3: Based on the TL-ResNet18 network built in S5.2, a feature fusion module is added. This module performs adjacent convolutional layer fusion, cross-convolutional layer fusion, and three-layer convolutional layer fusion on convolutional layers 2, 3, 4 and 5 of the TL-ResNet18 network respectively, and designs ten TL-ResNet18 networks based on feature fusion. S5.4: The ten networks built in S5.3 were subjected to ablation experiments on a segmented dataset of similar handwritten Chinese characters. The experimental results were compared with the TL-ResNet18 network to obtain a TL-ResNet18 network model that combines convolutional layers 4 and 5, which was then used for similar Chinese character recognition.

2. The similar handwritten Chinese character recognition method based on feature fusion TL-ResNet18 network according to claim 1, characterized in that: The density peak clustering method in step four specifically includes: S4.1: Calculate the x values ​​of the two data points. i and x j Distance P between ij ': P ij '=1-P ij (1) Among them, P ij P represents the confusion probability obtained from training the ResNet-18 network. ij This means that the larger the value, the more similar the two categories are; therefore, P needs to be adjusted. ij Transform into P in formula (1) ij ' indicates that P ij The smaller the value, the more similar the two data points are; S4.2: Calculate the local density of data points using a Gaussian kernel. The specific formula is as follows: Where, d c It is the cutoff distance, let d. ij The values ​​at the 1% to 2% level after sorting in ascending order; S4.3: Calculate the distance δ between a data point and its nearest higher-density point. i : If a data point is a point of maximum local density, its distance is calculated as follows: The values ​​of local density ρ and distance δ were obtained. A ρ-δ decision graph was constructed to represent the distribution of all data points in the ρ-δ space. Points with higher ρ and δ in the decision graph were selected as cluster centers. Using γ in formula (5) as the single criterion for cluster center selection, the formula is as follows: γ=ρ*δ (5) According to the above formula, the data points are sorted in descending order of γ to construct a γ decision map; the data points with larger γ values ​​are determined as cluster centers, and the remaining non-center data points are assigned to the clusters of the nearest higher-density points, thus completing the clustering process.

Images