A Chinese character image restoration method considering the structural characteristics of Chinese characters

Abstract

As a branch of image restoration, text restoration has received widespread attention in recent years. However, most of the current text restoration is aimed at English, a text with a simple structure, and there is less research on Chinese, a text with a complex and diverse structure. At the same time, most of the current image restoration work, including text restoration, is aimed at damaged images, and there is not much research on the restoration of smeared images. The present invention proposes a text restoration algorithm using a font structural feature attention network. It can effectively perform structural restoration on Chinese character images that are damaged, smeared, or both. The font structural feature attention network is based on the fact that, unlike English, Chinese characters are of many types and have very different structures. If traditional image restoration technology is used, it is difficult to perform specific restoration on the different structural features of Chinese characters. Therefore, the present invention starts from the structure of Chinese characters and proposes a font structural feature attention network for specific restoration of different Chinese characters.

Description

Technical Field

[0001] The present invention relates to technical fields including computer vision, artificial intelligence, image restoration, etc., and is a Chinese character image restoration method that takes into account the structural characteristics of Chinese character shapes. Background Art

[0002] Image inpainting, a popular research area in computer vision, aims to restore pixel features from damaged areas of a damaged image and plays a key role in many computer vision applications. In recent years, text inpainting, a sub-branch of image inpainting, has garnered significant attention. However, due to the unique nature of text, even the slightest structural difference between the inpainted result and the true image can significantly compromise the visual quality, contributing to the difficulty of text inpainting.

[0003] In recent years, image restoration technology based on deep learning networks has developed rapidly. At the 2016 CVPR conference, Pathak D et al. [1] first proposed an image restoration algorithm by combining CNN and GAN networks. Many improved algorithms have emerged based on this algorithm. However, the restoration of such methods is only concentrated on the rectangular area in the center of the image. Iizuka et al. [2] and Yu et al. [3] broke through the interference hypothesis of the central rectangular box and proposed to repair irregular patterns. However, such algorithms often need to rely on complex post-processing. The PConv (Partial Convolution, PConv) layer proposed by Liu et al. [4] of NVIDIA uses a traditional convolutional network and does not rely on additional post-processing. It repairs irregular interference parts by continuously learning the missing area and the original photo.

[0004] With the advancement of image restoration technology, the restoration technology for text images has also gradually developed. Text restoration tends to restore local lines from the perspective of visual features. References [5] and [6] are applicable to phonetic characters. However, they are not applicable to complex ideographic characters such as Chinese characters. Subsequently, some methods have tried to introduce glyph features. Zhang et al. [7] proposed a hierarchical learning network called HAN to extract hierarchical features to reconstruct damaged strokes of printed Chinese characters. Li et al. [8] proposed a generative adversarial network combined with contextual loss to improve the restoration effect of handwritten Chinese characters. However, the above methods often generate some pseudo-Chinese characters composed of strokes that seem reasonable but do not actually exist. Due to the lack of interaction between adjacent characters in text images, these methods are likely to produce blurred characters.

[0005] Obviously, relying solely on glyph information cannot produce accurate text images. In recent years, great progress has been made in the interaction of visual and semantic information. STEFANN[9] aims to modify the text content in an image at the character level, which uses label embedding to represent and edit each character. Some cross-modal tasks

[10]

[11] attempt to learn feature embeddings from word images and text labels for word localization and recognition tasks. Recently, the idea of ​​SEED

[12] is to use word embeddings from a pre-trained language model to make up for the lack of contextual semantic information. In SNR

[13] , a global semantic reasoning module is introduced to capture global semantic context for accurate scene text recognition. These studies show that semantic modules that process contextual semantic information can be applied to image processing tasks including text repair.

[0006] Since the attention mechanism was proposed, it has been increasingly used to improve the performance of deep learning algorithms. Like other neural network-based methods, the attention mechanism attempts to mimic the way the human brain processes data. Human vision does not process the entire image at once, it only focuses on specific parts. In this way, the focused part of the human visual field can be perceived at "high resolution" while the surrounding environment is at "low resolution". In other words, it gives higher weights to the relevant parts while minimizing the irrelevant parts and giving them lower weights. This allows the brain to accurately and efficiently process and focus on the most important parts instead of processing the entire visual space. It was originally developed for natural language processing applications in 2014

[14] , and since then it has been widely used in different applications

[15] , especially computer vision tasks

[16]

[17] . Its potential to enhance most CNN-based methods has received widespread attention

[18] . In addition, it has been combined with recurrent neural network models and graph neural networks. The main idea behind the attention mechanism is to give different weights to different information. At present, the attention mechanism has begun to be applied to the field of image restoration and has achieved good results

[19]

[20] . However, there are few studies on the application of the attention mechanism to the field of text restoration.

[0007] In summary, although there have been many studies on using deep neural networks to repair text, there are few studies on repairing Chinese characters, and the existing technologies have problems such as generating blurred characters. To address these problems, this paper starts from the structure of Chinese characters and proposes a text repair algorithm for damaged text using a font structural feature attention network. According to the structure of different Chinese characters, it can repair Chinese characters with damaged, smeared, or both images. Summary of the Invention

[0008] The present invention aims to repair damaged or smeared Chinese character images that appear in different real-life scenarios, such as steles, ancient books, calligraphy works and other cultural relics, and solve the problem that the Chinese characters are complex in structure, huge in number, and have many similar characters, which affect the repair effect. A glyph structural feature attention network TAN is proposed. The present invention uses a preprocessing network to perform preliminary repair processing on the input Chinese character image to reduce noise and enhance the structural features of Chinese characters. VGG and the global semantic supervision network are used to perform image context feature supervision and context semantic supervision on the input Chinese character image, so that the repair result conforms to the global font features and context semantics, and reduces the influence of similar characters on the repair result. The main repair object of the present invention is Chinese characters with complex structures that are damaged or smeared. Inspired by the attention mechanism, the present invention takes the Chinese character glyph structure as inspiration and proposes a glyph structural feature attention network TAN to solve the problem of complex and diverse Chinese character glyph structures, segment and select image features to remove features that are irrelevant to the glyph structure and features that hinder glyph repair, and obtain better repair effects.

[0009] The Chinese character image restoration algorithm that considers the structural characteristics of Chinese characters has the following steps:

[0010] Based on the U-Net network, we improved and upgraded it by building a preprocessing network. We used skip connections between the encoder and decoder layers, symmetrically, to pass feature information lost by encoder convolution to the decoder. We also used 2×2 convolution kernels and stride 2 convolution layers instead of traditional pooling layers. We used the generated dataset to train the preprocessing network.

[0011] A convolutional neural network is used to build a glyph structural feature attention network (TAN). The preprocessed output is concatenated with the original image as input, and the input features are divided into four different categories: left-right, left-center-right, top-bottom, and top-center-bottom. The input image features are decomposed into corresponding feature blocks. After convolution and deconvolution of the feature blocks, the resulting features are concatenated to restore the feature size. The image features obtained from the four parts are weighted averaged, with the weight of each part obtained through network training.

[0012] Use VGG and global semantic supervision networks to supervise the training process of the glyph structure attention network;

[0013] We tested a dataset of damaged or smudged Chinese characters. After processing the damaged images through a preprocessing network, they were fed into the glyph structural feature attention network (TAN), which yielded relatively good restoration performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1The invention is able to repair damaged, smeared or both damaged Chinese characters largely due to the network, which extracts text information from complex images through image preprocessing.

[0015] Figure 2 It is the main network structure diagram of the present invention.

[0016] Figure 3 This is the structure of the glyph structural feature attention network (TAN). Based on the structural characteristics of Chinese characters, we split the image into four different types, training different characters with different weights to achieve the goal of character-specific repair.

[0017] Figure 4 It is a display of the results. The three pictures in each group are respectively the input, the result picture of the present invention repairing damaged and smudged text, i.e. the output, and the real picture. DETAILED DESCRIPTION

[0018] In order to more clearly explain the purpose, technical process and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0019] The present invention proposes a text repair algorithm for damaged text using a Chinese character structural feature attention network. The main steps include: constructing a damaged and smudged Chinese character dataset, training a pre-trained network, building a main repair network, and testing the repair results of the repair network on Chinese character images.

[0020] Step 1: Construct a dataset of damaged and smudged Chinese characters

[0021] The purpose of this invention is to repair damaged or smudged Chinese characters. This requires multiple images of damaged or smudged characters in different fonts as input. However, very few real datasets that meet this requirement and also have complete images for calculating loss are sufficient for training. Therefore, we generated a dataset using masks in 32 fonts. This dataset contains approximately one million text images, providing good coverage of most commonly used characters.

[0022] Step 2: Train the pre-trained network

[0023] like Figure 1As shown, the preprocessing network used in the present invention is improved and upgraded based on the U-Net network, with jump connections between the symmetrical network layers of the encoder and decoder, and the feature information lost by the encoder convolution is passed to the decoder. In order to reserve more space to generate better area masks and achieve the purpose of downsampling, a convolution layer with a size of 2×2 and a stride of 2 is used to replace the traditional pooling layer. In addition, more convolution and deconvolution layers are used to obtain better preprocessing effects. The preprocessing network performs some simple denoising and repair on the input damaged Chinese character image and extracts the text information in the image, which can provide better input for the subsequent repair network.

[0024] Step 3: Train the repair network

[0025] like Figure 2 As shown, Figure 1 The trained preprocessing network is used as the pretraining network. The result image obtained by the preprocessing network will be used to calculate the L1 and L2 distances with the target image. The weighted combination of L1 and L2 distances constitutes the loss of the network. Considering that the preprocessing network will lose some information of the input image during the restoration process, the result of the preprocessing network and the input image are connected together and passed to the subsequent network. Figure 3 As shown, the Glyph Structural Feature Attention Network (TAN) is essentially a combination of convolutional layers that functions similarly to an attention mechanism. It selects input image features and removes features irrelevant to the glyph structure or those that hinder glyph restoration. The design of this innovative network is primarily inspired by the glyph structure of Chinese characters. Since Chinese characters can be divided into left-right, top-bottom, and other structural divisions, the input features are partitioned into four different types: left-right, left-center-right, top-bottom, and top-center-bottom. These four divisions split the input image features into corresponding feature blocks. This partitioning approach divides the Chinese character image features into smaller blocks, allowing for more similar structural features across different characters. These feature blocks are then subjected to convolution and deconvolution operations for feature selection and restoration. The resulting features are then concatenated according to the previous partitioning process to restore the feature size. Finally, for further image feature selection, the image features obtained from the four partitions are weighted averaged, with the weights for each component trained by the network. The weight setting takes into account the different glyph structures of different Chinese characters, so it is necessary to train a set of different weights for each Chinese character to give each Chinese character specificity, which also provides a constraint for repairing similar characters. Its expression is as follows:

[0026]

[0027] where X j represents the input feature image, fi It is a feature splitting adaptive mapping. A total of 4 splitting methods are used in the network, so the value of i ranges from 1 to 4, and j is the number of the target character of the input feature image X, k ij For the corresponding f i Under the weight of number j, each word will have a different weight vector {k1, k2, k3, k4};

[0028] In order to ensure the correctness of the glyph structure attention network training, we introduce the pre-trained VGG network to supervise it. The target image passes through some encoding layers of the VGG network to obtain features of the same size and number as the glyph structure network output. The L2 distance between this feature and the feature obtained by the attention network is used as the network loss function. The loss function of the glyph structure attention network TAN is L attention , which is expressed as follows:

[0029]

[0030] where X in , X in2 They are the input damaged feature image and the second convolution output of the repair network, for glyph structure attention network, For the pre-trained VGG feature extraction network, To repair the deconvolution output of the corresponding size in the network. The MSE loss used in the formula is as follows:

[0031]

[0032] Among them, X and Y can be of any shape, each containing n elements. For an image, n is the number of pixels of the image, that is, the product of the width and height of the image. The sum of the squares of the absolute values ​​of the differences corresponding to the n elements and the result divided by n is the MSE loss.

[0033] The global semantic supervision network of this invention uses the network architecture GSSM proposed by Wang

[21] , and learns from the ideas and thoughts of the network in this paper, adds constraints on the semantics of text images and the correlation of context image structures to the repair network, designs an NLP network structure, namely the global semantic supervision network, and calculates it with the pre-trained BERT network to obtain the global semantic loss. To a certain extent, it strengthens the training effect of the glyph structure attention network in the form of implicit index, so that the glyph structure attention network can better train weights for different structural styles. At the same time, it and VGG serve as semantic loss and image loss respectively to strengthen the repair effect in the two dimensions of language and image. The loss function of the semantic supervision network is denoted as L sem , which is expressed as follows:

[0034] Lsem =1-cos(y,e)

[0035] Where E is the pre-trained BERT model for the sentence E = {c1, c2, ..., c n} is obtained by word embedding, y is the semantic supervision network according to each word c i Corresponding image img i Processed to obtain.

[0036] In the pre-processing network and the repair network, we introduce the consistency loss L consist , by strengthening the consistency of the feature map of the damaged image before and after processing by the preprocessing network and the repair network, the network repair ability is enhanced. consist The expression is as follows:

[0037]

[0038] where X truth is the damaged input image X in The uncorrupted target image, To preprocess the network, To repair the network, λ is a hyperparameter that can be adjusted according to experimental results. The corresponding expression of SoothL1 loss used in the formula is as follows:

[0039]

[0040] When the difference between the predicted value and the groundtruth is small (the absolute value difference is less than 1), the L2 loss is actually used; when the difference is large, the L1 loss is translated. SoothL1 loss is actually a combination of L2 loss and L1 loss, and it has some advantages of both L2 loss and L1 loss.

[0041] The total loss of training is expressed as follows:

[0042] L all =L consist +L attention +L sem

[0043] Step 4: Test the restoration results of the Chinese character image by the restoration network.

[0044] like Figure 4 The figure shows the test results of the Chinese character image restoration algorithm proposed by the present invention, which takes into account the structural characteristics of Chinese characters. It can be seen that it has a good restoration effect on images that are damaged, smeared, or both.

Claims

1. A Chinese character image restoration method that considers the structural characteristics of Chinese characters. The general steps are as follows: (1) Step 1: Construct a dataset of damaged and smudged Chinese characters; The above restoration method requires the use of damaged and smeared Chinese character image datasets, using synthetic data or self-collected data; (2) Step 2: training the preprocessing network; The preprocessing network built using deep learning is based on an improved U-Net network. It uses skip connections between the symmetrical network layers of the encoder and decoder to pass feature information lost by encoder convolution to the decoder. To preserve more space to generate better region masks and achieve downsampling, a 2×2 convolution kernel and a stride of 2 convolution layers are used instead of traditional pooling layers. More convolution and deconvolution layers are also used to achieve better preprocessing effects. The preprocessing network performs simple denoising and restoration on the damaged Chinese character input images and extracts the text information in the images, providing better input for the subsequent restoration network. (3) Step 3: Training the repair network; The glyph structural feature attention network TAN is essentially a combination of convolutional layers, which plays a role similar to the attention mechanism. It can select the input image features and remove the features that are irrelevant to the glyph structure and the features that hinder glyph repair. Its expression is as follows: where X j represents the input feature image, f i It is a feature splitting adaptive mapping. A total of 4 splitting methods are used in the network, so the value of i ranges from 1 to 4, and j is the number of the target character of the input feature image X, k ij For the corresponding f i Under the weight of number j, each word will have a different weight vector {k1, k2, k3, k4}; In order to ensure the correctness of the glyph structure attention network training, the pre-trained VGG network is introduced to supervise it; the loss function of the glyph structure attention network TAN is L attention , which is expressed as follows: where X in , X in2 They are the input damaged feature image and the second convolution output of the repair network, for glyph structure attention network, For the pre-trained VGG feature extraction network, To repair the deconvolution output of the corresponding size in the network; The global semantic supervision network strengthens the training effect of the glyph structure attention network in the form of implicit index to some extent, so that the glyph structure attention network can better train the weights for different structural styles. At the same time, it and VGG serve as semantic loss and image loss respectively to strengthen the restoration effect in the two dimensions of language and image. The loss function of the semantic supervision network is denoted as L sem , which is expressed as follows: L sem =1-cos(y,e) Where E is the pre-trained BERT model for the sentence E = {c1, c2, ..., c n } is obtained by word embedding, y is the semantic supervision network according to each word c i Corresponding image img i Processing is performed; In the pre-processing network and the repair network, we introduce the consistency loss L consist , by strengthening the consistency of the feature map of the damaged image before and after processing by the preprocessing network and the repair network, the network repair ability is enhanced, L consist The expression is as follows: where X truth is the damaged input image X in The uncorrupted target image, To preprocess the network, To repair the network, λ is a hyperparameter that can be adjusted according to experimental results, L smoothL1 The corresponding expression of loss is as follows: When the difference between the predicted value and the groundtruth is small (the absolute value difference is less than 1), the L2 loss is actually used; when the difference is large, the L1 loss is translated. SoothL1 loss is actually a combination of L2 loss and L1 loss. It has some advantages of both L2 loss and L1 loss. The total loss of training is expressed as follows: L all =L consist +L attention +L sem (4) Step 4: Test the restoration results of the restoration network on the Chinese character image; The network built in step 3 is tested on a dataset of damaged and smeared Chinese character images, and can achieve good restoration results for text images that are damaged, smeared, or both.

Images