Font Repair Method and System Based on Deep Meta-Learning and Generative Adversarial Network

Abstract

The present invention discloses a font restoration method and system based on deep meta-learning and generative adversarial networks. The steps of the method are as follows: Use a data acquisition module to obtain an existing calligraphy font dataset Dataset-1 and the stroke and structure data of the font; Use a data processing module to process the data in different ways for different tasks; Use the Font-Meta module to complete the incomplete calligraphy font; Use a font review module to find the best restored font; Use a font output module to output the restored font. The font restoration method and system proposed by the present invention can learn features such as the font strokes, structure, and style of Chinese characters based on existing data samples, and are more comprehensive than the prior art when completing the missing parts of Chinese character fonts. When applied in the field of calligraphy font restoration, the present invention can reduce labor costs and improve the accuracy and integrity of font restoration.

Description

Technical Field

[0001] The present invention mainly relates to the fields of computer vision image processing and picture text restoration technology, and particularly relates to a calligraphy font restoration method and system based on deep meta-learning and recurrent generative adversarial network of font strokes, structures, and contours. Background Art

[0002] With the rapid development of deep learning technology and the popularization of artificial intelligence applications, related new technologies have brought great convenience to people's production and life. Nowadays, traditional Chinese culture is being valued by everyone, and font calligraphy is an important part of traditional culture. When obtaining Chinese calligraphy fonts from ancient and modern times, the calligraphy fonts are often missing or damaged due to their long history or other factors. How to repair these calligraphy fonts is a hot research topic now.

[0003] In recent years, various methods for repairing damaged and incomplete calligraphy fonts have emerged in an endless stream, mainly divided into traditional methods and deep learning methods. Traditional methods such as the method for restoring inscriptions based on the description of Chinese character image contours in Patent CN105069766A obtain a set of component stroke templates by segmenting the font structure and strokes of an existing calligraphy dataset. Then, during the repair process, the stroke with the highest similarity matching degree in the stroke template is found for filling and repair. Although this type of method can restore the font structure well, the stroke styles filled for fonts with a more scribbled style (such as cursive script, running script, etc.) have a large difference.

[0004] Now, in the context of the rapid development of deep learning technology, some latest works introduce the ideas of deep neural networks and generative adversarial networks to repair damaged calligraphy fonts. In the method for restoring and completing incomplete Chinese calligraphy based on generative adversarial network in Patent CN110765339A and the method for restoring damaged ancient Chinese characters based on conditional adversarial network in Patent CN110335212A, it is proposed to use generative adversarial network or conditional generative adversarial network for font restoration, that is, directly putting the font to be repaired into the neural network model for training to obtain the repaired font image, and making adjustments and modifications on a global scale, without considering font structure and other features. In the method and system for automatically restoring calligraphy font library based on style transfer in Patent CN110570481A, it is proposed to use the method of style transfer for calligraphy font restoration. Although this method greatly reduces the workload of traditional font segmentation and the generated effect is also good, the style transfer model used requires a paired dataset, and such a dataset is difficult to obtain in actual application scenarios. It is difficult for us to obtain the font image corresponding to this Chinese character in this calligraphy category.

[0005] Patent CN112435196 proposes a text repair method and system based on deep learning to perform repairs using deep learning methods. This method first outputs missing strokes through a text integrity detection module, and then uses a missing stroke matching module to match similar style strokes for the missing strokes. This method combines stroke information and generative adversarial network methods for repair, but does not consider features such as Chinese character font structure and local relationships. And in actual scenarios, the dataset of calligraphy fonts to be repaired that we can obtain is small, and this method cannot solve the problem of calligraphy font repair in small sample scenarios.

[0006] Therefore, there is an urgent need for a method that takes into account multi-dimensional font features and uses small sample calligraphy font data for missing repair. Summary of the Invention

[0007] The purpose of the present invention is to provide a font repair method and system based on deep meta-learning and generative adversarial networks, which uses deep meta-learning methods to learn multi-dimensional font features such as Chinese character font strokes, outlines, structures, and local relationships, and a method for repairing defective fonts through small sample calligraphy font data.

[0008] To achieve the above object, in the first aspect of the present invention, a font repair method based on deep meta-learning and generative adversarial networks is provided, which is characterized in that it includes the following steps:

[0009] S1. Use a data acquisition module to obtain an existing calligraphy font dataset Dataset-1 and stroke and structure data of the font;

[0010] S2. Use a data processing module to process the data in different ways for different tasks;

[0011] S3. Use the Font-Meta module to complete the incomplete calligraphy font;

[0012] S4. Use a font review module to find the best completed font;

[0013] S5. Use a font output module to output the repaired font.

[0014] Further, the steps for constructing the dataset Dataset-1 in S1 are:

[0015] S11. Obtain the calligraphy work to be repaired;

[0016] S12. Use a coverage matrix to crop the calligraphy font in the original entire calligraphy work image, and expand or compress the obtained cropped image to a picture with a size of 256×256.

[0017] S13, converting the image after the unified size into a single channel, performing binarization processing, and obtaining a binary image of the word;

[0018] S14. Binarize the picture set to construct the dataset Dataset-1.

[0019] Furthermore, the data set Dataset-1 is further processed into Dataset-11, and the specific steps are as follows:

[0020] S21. Obtain complete calligraphy fonts and artistic fonts;

[0021] S22, selecting the picture with the largest image entropy as data;

[0022] S23, randomly generating irregular shapes of different sizes as font masks to simulate the defect of calligraphy fonts;

[0023] S24 and font masks are added to the dataset Dataset-11 to obtain a set of class missing images;

[0024] S25. Pair the class-missing image sets and construct the dataset Dataset-11.

[0025] Further, the S3 includes:

[0026] S31. Construct font completion network FDR-Net, cyclic generative adversarial network and font structure review model;

[0027] S32. Use the MAML method to pre-train the font completion network FDR-Net;

[0028] S33, putting the font repaired by the font completion network FDR-Net into the cyclic generative adversarial network for local style adjustment, and outputting the font image after style conversion.

[0029] Furthermore, the dataset Dataset-11 is put into the Font-Meta module for learning, and the font completion network FDR-Net is initialized and pre-trained, the steps are as follows:

[0030] S321, obtaining font stroke data;

[0031] S322, obtaining font structure data;

[0032] S323, constructing a dataset of incomplete simulated calligraphy fonts, and pairing the incomplete data with the original data;

[0033] S324, construct a font completion network FDR-Net model;

[0034] S325. Train the font completion network FDR-Net.

[0035] Further, the S33 includes:

[0036] S331. Initialize and pre-train the CycleGAN;

[0037] S332. Obtain the calligraphy font data to be repaired, and finely tune the font completion network FDR-Net to perform style and font content structure learning;

[0038] S333. After pre-training and fine-tuning, obtain the finely tuned font completion network FDR-Net;

[0039] S334. Complete the missing part to obtain a preliminary repaired picture;

[0040] S335. Input the preliminary repaired picture into the CycleGAN for local style transfer to obtain the transferred image.

[0041] Further, the pre-trained font completion network FDR-Net can complete the stroke structure of the missing part to obtain the completed calligraphy font M1.

[0042] Further, the S4 includes:

[0043] S41. Pre-train the stroke integrity network, structure integrity network, and style similarity network;

[0044] S42. Input the style-converted image into the stroke integrity network for scoring to obtain Score1;

[0045] S43. Input the style-converted image into the structure integrity network for scoring to obtain Score2;

[0046] S44. Input the style-converted image into the style similarity network for scoring to obtain Score3;

[0047] S45. By calculating the weighted average of the scoring results of the stroke integrity network, structure integrity network, and style similarity network respectively, the final score sequence can be obtained, and the repaired calligraphy font picture with the highest score is selected for output.

[0048] Further, the Font-Meta module includes the FDR-Net module and the CycleGAN module;

[0049] The FDR-Net module is used to learn how to complete the font through the network and deep meta-learning methods;

[0050] The CycleGAN module is used to complete the local style conversion of fonts.

[0051] In the second aspect of the present invention, a font restoration system based on deep meta-learning and generative adversarial networks is provided, which is characterized in that it includes the following modules:

[0052] The data acquisition module is used to acquire the existing calligraphy font dataset Dataset-1 and the stroke and structure data of the font; the data processing module is used to process the data in different ways for different tasks;

[0053] The Font-Meta module is used to complete the restoration of incomplete calligraphy fonts;

[0054] The font review module is used to find the best restored font;

[0055] The font output module is used to output the restored font.

[0056] The beneficial technical effects of the present invention are at least as follows:

[0057] (1) Compared with the prior art, the method based on deep meta-learning proposed by the present invention can learn the characteristics of Chinese character fonts such as strokes, structures, and styles based on existing data samples, and considers more comprehensive factors when completing the missing parts of Chinese character fonts than the prior art;

[0058] (2) Since there is less data for the calligraphy fonts to be restored, and the prior art all learns based on the premise of a large amount of known data, the present invention uses the mechanism of meta-learning and only needs a small number of new data samples to infer the data characteristics of this type from the existing knowledge pool, greatly reducing the quantity requirement for new data.

[0059] (3) The prior art all requires paired datasets for style conversion, and the use of the CycleGAN can complete font style conversion without paired datasets. At the same time, the addition of the font review module further improves the quality of calligraphy font restoration. In the field of calligraphy font restoration, the present invention can greatly reduce the labor cost and improve the accuracy and integrity of font restoration. Description of the Drawings

[0060] The present invention is further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation to the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the following drawings without creative efforts.

[0061] Figure 1 It is a schematic structural diagram of the font restoration system according to the embodiment of the present invention.

[0062] Figure 2It is a schematic diagram of the Chinese character font structure according to an embodiment of the present invention.

[0063] Figure 3 It is a flowchart of the MAML algorithm according to an embodiment of the present invention.

[0064] Figure 4 It is a schematic diagram of the algorithm flow of the cyclic generative adversarial network according to an embodiment of the present invention. Detailed implementation manners

[0065] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation of the present invention.

[0066] The present invention provides a calligraphy font restoration method based on deep meta-learning to recognize the structure, strokes, outlines and other features of fonts and combine cyclic generative adversarial networks for style conversion. Using the existing calligraphy font dataset for meta-training dataset S1, by learning the features such as font strokes, font structure and font content in the existing font library and the process of filling in missing strokes of fonts, prior knowledge of different dimensional features of each font and the meta-model Font-Meta and the font structure review model FSR-Net are obtained. Then, the data to be restored is processed to obtain dataset S2, which is put into Font-Meta to learn the strokes, structure, content and style of this type of calligraphy font, and the font restored by the font filling network FDR-Net of Font-Meta is put into the cyclic generative adversarial network for style re-transfer to obtain a complete calligraphy font picture with the same style as the original font.

[0067] In one embodiment, as Figure 1 shown, a calligraphy font restoration system based on deep meta-learning and cyclic generative adversarial networks is provided, which includes the following modules:

[0068] A data acquisition module for acquiring the existing calligraphy font dataset Dataset-1 and the stroke and structure data of the font;

[0069] A data processing module for processing the data in different ways for different tasks;

[0070] The Font-Meta module for filling in the missing calligraphy font;

[0071] A font review module for finding the best filled font;

[0072] A font output module for outputting the restored font.

[0073] The calligraphy font restoration method based on deep meta-learning and recurrent generative adversarial network provided in this embodiment includes the following steps:

[0074] The steps for constructing the dataset Dataset-1 described in S1 are:

[0075] S11. Obtain the calligraphy work to be restored;

[0076] S12, the original whole calligraphy work image is used to perform calligraphy font cropping using a covering matrix, and the cropped image is expanded or compressed to a picture of 256×256 in size;

[0077] S13, converting the image after the uniform size into a single channel, performing binarization processing, and obtaining a binary image of the word;

[0078] S14. The processed binary image set is the data set Dataset-1.

[0079] The specific steps to construct the existing Chinese font dataset Dataset-1 are as follows:

[0080] First, we need to obtain all existing standard font and art font datasets, and use the K-Means method to perform data clustering, and randomly extract 10 key data samples from each category. Then, Dataset-1 is divided into Dataset-11, Dataset-12, Dataset-13, and Dataset-14 according to different processing methods and different purposes. The processing methods and different usage purposes are shown in S1101-S1104:

[0081] S1101, Dataset-11 is the dataset used by the font completion network. The feature of this dataset is that it randomly generates irregular shapes of different sizes with pixel values ​​of 0 as font masks to simulate the missing conditions of calligraphy fonts. The masks are added to the new datasets to obtain a set of class missing images. The class missing image set is paired with the original image to construct a dataset. Finally, the dataset Dataset-11 is split into a support set and a query set (the support set and the query set will be introduced in S13);

[0082] S1102, Dataset-12 is the data set used by the stroke integrity network in the font review module. Obtain the stroke data set, divide it into 101 stroke types according to the Xinhua Dictionary, design the stroke separation network A, and separate the strokes of the Chinese character images in the Dataset-1 data set;

[0083] S1103 and Dataset-13 are datasets used by the structural integrity network in the font review module. According to Wikipedia, Chinese fonts mainly have 12 different structures, such as up and down, left and right, and surrounding. Figure 2 As shown;

[0084] S1104, Dataset-14 is the dataset used by the style similarity network in the font review module. It is the dataset obtained by binarizing the Dataset-1 data.

[0085] The dataset Dataset-1 is further processed into Dataset-11. The specific steps are as follows:

[0086] S21. Obtain normal calligraphy and artistic fonts;

[0087] S22, using font information entropy to select the best data;

[0088] S23, randomly generating irregular shapes of different sizes as font masks to simulate the defect of calligraphy fonts;

[0089] S24 and font masks are added to the dataset Dataset-11 to obtain a set of class missing images;

[0090] S25. Pair the class-missing image sets and construct the dataset Dataset-11.

[0091] The S3 includes:

[0092] S31. Construct the font completion network FDR-Net, loop the generation of adversarial networks and font structure audit models, initialize and pre-train the font completion network FDR-Net, which is a simple variational autoencoder structure, mainly including encoder modules and decoder modules. The encoder and decoder are composed of convolutional layers, normalization layers, pooling layers, etc. The network size and number of layers of the encoder and decoder can be set arbitrarily. In detail, in this example, a 5×5 convolution kernel and a 2×2 pooling are used, the stride is 1, and there are 6 convolution layers. The number of convolution kernels is 32, 32, 64, 128, 256, and 256 respectively.

[0093] Initialize and pre-train the cyclic generative adversarial network, which mainly consists of two generators G and F and two discriminators D1 and D2.

[0094] Generator G: Learn the mapping G: X→Y, where X is the original font style; Y is the font style generated by generator G. The main purpose of generator G is to learn a mapping that makes G(x) and Y similar.

[0095] Generator F: Learn the mapping F: Y → X, receive the target font style, and convert it into a style similar to the original font style. The main purpose of Generator F is to learn a mapping that can make F(G(x)) similar to X.

[0096] The network structures of Generators G and F are composed of 3 convolutional blocks, 2 residual blocks, and 2 upsampling blocks. Each convolutional block contains a 2D convolutional layer and 1 BatchNorm layer, and uses ReLU as the activation function. Each residual block contains two 2D convolutional layers, and there is a batch normalization layer behind each convolutional layer, with the set momentum value being 0.8. Each upsampling block contains a 2D transposed convolutional layer and uses ReLU as the activation function.

[0097] Discriminator D1: Mainly responsible for distinguishing the images generated by Generator F (denoted as F(Y)) and the real images in the target domain (denoted as X).

[0098] Discriminator D2: Mainly responsible for distinguishing the images generated by Generator G (denoted as G(x)) and the real images in the target domain (denoted as Y).

[0099] The architectures of Discriminators D1 and D2 are similar to the discriminator network architecture in PatchGAN, including 5 convolutional layers and 5 BatchNorm layers.

[0100] S32. Use the MAML method to pre-train the font completion network FDR-Net;

[0101] Use the MAML method to pre-train the font completion network FDR-Net. Dataset description of Dataset1: Dataset1 is called the D-meta-train dataset. Suppose there are φ types of fonts in the Dataset1 dataset, Font 1 ~Font φ , where M is the number of samples contained in each font. This dataset is divided into N Tasks. Each Task is a set of 20 pairs of incomplete fonts and complete fonts after adding masks to fonts of different styles, such as At the same time, each task is divided into a support set and a query set. In the tasks of this patent, the paired 5 groups of datasets are called the support set, and the other 15 groups of datasets are used as the query set. Each Task is equivalent to a piece of data in the training process of an ordinary deep learning model. Therefore, we need to repeatedly extract several Tasks from the training data distribution to form a batch, and then use the Adam optimizer for optimization.

[0102] First, define the task. Let the calligraphy data to be repaired be set as F, and the data after repair by the completion network be set as O. Then each task is where R represents the completion network. In this embodiment, R θ is used to represent the font generator with parameters θ. When the model learns the i-th task T i , the parameters θ become θ' i , adapting to the current task T i The parameters θ' i are obtained by updating the model parameters through m-step gradient descent using the support set. For one step of gradient descent, the calculation formula is

[0103]

[0104]

[0105] The query set loss function is:

[0106]

[0107] For the sum of losses for all N tasks in total, the meta-learning objective function is:

[0108]

[0109] The entire pre-training process is shown in Algorithm 1, and the purpose is to obtain the font completion network FDR-Net:

[0110] The FDR-Net module uses the generation network and the method of deep meta-learning to learn how to complete the font. The specific meta-training process is:

[0111] First are the first two Requirements. The first Requirement refers to the distribution of Tasks in Dmeta-train. We can repeatedly randomly sample Tasks to form a Task pool composed of several Ts as the training set of MAML, as Figure 3 shown. The second Requirement is the learning rate. MAML is based on double gradients, and each iteration contains a process of two parameter updates, so there are two learning rates that can be adjusted.

[0112] Step 1: Randomly initialize the model parameters;

[0113] Step 2: It is a loop, which can be understood as one round of iteration process or one Epoch. Of course, there can also be multiple Epochs in the pre-training process, which is equivalent to setting Epoch;

[0114] Step 3: Randomly sample several (for example, 5) Tasks to form a batch;

[0115] Steps 4 - 7: The first gradient update process.

[0116] Duplicate an original model, calculate new parameters, and use them in the calculation process of the second-round gradient. For each task in the batch, update the model's parameters separately (updating 5 times for 5 tasks). Note that this process can be repeatedly executed multiple times in the algorithm, but this layer of loop is not reflected in the pseudocode.

[0117] Step 5: Use the support set in a certain Task in the batch to calculate the gradient of each parameter.

[0118] Step 6: Update the first gradient.

[0119] Steps 4 - 7: After completion, MAML finishes the first gradient update. Next, based on the parameters obtained from the first gradient update, calculate the second gradient update through gradient by gradient. The gradient calculated during the second gradient update is directly applied to the original model through Adam, which is the gradient that the model actually uses to update its parameters.

[0120] Step 8: This corresponds to the process of the second gradient update. The loss calculation method here is roughly the same as in Step 5, but there are two differences: First, instead of updating the gradient using the loss of each task separately, we calculate the sum of the losses of a batch, just like in the common model training process, and perform stochastic gradient descent Adam on the gradient; Second, the samples involved in the calculation here are the query set in the Task. In our example, that is 5-way * 15 = 75 samples, aiming to enhance the generalization ability of the model on the Task and avoid overfitting to the support set.

[0121] After Step 8, the model finishes training in this batch and starts to return to Step 3 to continue sampling the next batch.

[0122] The above is the entire process of pre-training MAML to obtain FDR-Net.

[0123] Next, when facing font completion data and new font completion Tasks, we will fine-tune on the basis of FDR-Net to obtain M-fine-tune.

[0124] The fine-tuning process is roughly the same as the pre-training process, with the following differences:

[0125] In Step 1, for fine-tuning, there is no need to randomly initialize the parameters anymore. Instead, use the trained FDR-Net to initialize the parameters;

[0126] In Step 3, for fine-tuning, only one Task needs to be extracted for learning, and naturally there is no need to form a batch. The fine-tuning uses the support set of this Task to train the model and the query set to test the model;

[0127] There is no Step 8 in fine-tuning because the query set of the Task is used to test the model, and the target images are unknown to the model. Therefore, there is no second gradient update in the fine-tuning process, but the parameters are directly updated using the results of the first gradient calculation.

[0128] S33. Put the font repaired by the completion network FDR-Net into the cyclic generative adversarial network for local style adjustment, and output the font image after style conversion. Specifically, only a small number of target-style calligraphy fonts (which can be complete or partial fonts) need to be obtained, and with the characteristics of the cyclic generative adversarial network - without the need for a paired style data set, the style conversion of the original target font can be completed. The training of this model involves adversarial loss and cycle consistency loss:

[0129] The adversarial loss matches the distribution of the generated font images and the distribution of the target domain:

[0130]

[0131] In Formula 5, x is the original font style and y is the target font style. The discriminator D Y attempts to distinguish the style generated by the mapping G (i.e., G(X)) from the target font style y. The discriminator D X attempts to distinguish the style generated by the mapping F (i.e., F(Y)) from the original font style.

[0132] The cycle consistency loss is used to prevent the transformers G and F in the learning from contradicting each other. If only the adversarial loss is used, the network will map the same set of input font images to any random combination of images of the target font. Therefore, any mapping obtained can learn an output similar to the target probability distribution. The probabilities x i and y i will have many mapping ways between them. The cycle consistency loss solves this problem by reducing the number of possible mappings. Then the loss function formula of the cycle consistency is as

[0133] shown in Formula 6.

[0134]

[0135] If the cycle consistency loss is used, then the images reconstructed by F(G(x)) and G(F(y)) will be similar to x and y respectively.

[0136] The complete objective function is the weighted sum of the adversarial loss and the cycle consistency loss, as shown in Formula 3.

[0137] L(F, G, D X , D Y ) = L GAN (G, X, Y, D Y ) + L GAN (F, Y, X, D X ) + φL cyc (F, G) (7)

[0138] In formula 7, L GAN (G, D Y , X, Y) is the first adversarial loss, and L GAN (F, D X , Y, X) is the second adversarial loss. The first adversarial loss is calculated based on generator A and discriminator network B, and the second adversarial loss is calculated based on generator network B and discriminator network A. The objective function needs to optimize the function in formula 8 to train CycleGAN.

[0139]

[0140] The training steps of CycleGAN are as Figure 4 shown.

[0141] Put the dataset Dataset-11 into the Font-Meta module for learning, and initialize and pre-train the font completion network FDR-Net. The steps are as follows:

[0142] S321. Obtain font stroke data;

[0143] S322. Obtain font structure data;

[0144] S323. Construct a dataset simulating the mutilation of calligraphy fonts, and pair the mutilated data with the original data;

[0145] S324. Construct the font completion network FDR-Net model;

[0146] S325. Train the font completion network FDR-Net.

[0147] The said S33 includes:

[0148] S331. Initialize and pre-train the cycle generative adversarial network;

[0149] S332. Obtain the calligraphy font data to be repaired, and finely tune the font completion network FDR-Net to perform style and font content structure learning;

[0150] S333. Obtain the finely tuned font completion network FDR-Net after pre-training and fine-tuning;

[0151] S334. Complete the missing parts to obtain the preliminarily repaired image;

[0152] S335. Input the preliminarily repaired image into the cyclic generative adversarial network for local style transfer to obtain the transferred image.

[0153] Pre-train the font completion network FDR-Net, which can complete the stroke structure of the missing parts to obtain the completed calligraphy font M1.

[0154] S4 includes:

[0155] S41. Pre-train the stroke integrity network, structure integrity network, and style similarity network;

[0156] S42. Input the style-converted image into the stroke integrity network for scoring, i.e., stroke integrity scoring. The stroke integrity network is a shallow fully connected neural network, and the number of network layers, neurons, and optimizers can be set arbitrarily. In this example, a 5-layer fully connected layer is used, and the ReLu activation function and SGD optimizer are used for model training to obtain Score1;

[0157] S43. Input the style-converted image into the structure integrity network for scoring. The structure integrity network is a convolutional neural network, and the number of convolutional layers of the convolutional kernel can also be set arbitrarily. In this example, 4 convolutional layers, 4 pooling layers, and 2 fully connected layers are used, and the ReLU activation function and Adam optimizer are used for model training to obtain Score2;

[0158] S44. Input the style-converted image into the style similarity network for scoring. The style similarity network is similar to the structure integrity network, with 4 convolutional layers, 4 pooling layers, and 2 fully connected layers, and the ReLU activation function and Adam optimizer are used for model training to obtain Score3;

[0159] S45. By calculating the weighted average of the scoring results of the stroke integrity network, structure integrity network, and style similarity network respectively, the final score sequence can be obtained, and the repaired calligraphy font image with the highest score is selected for output:

[0160] SCORE = α·Score1 + β·Score2 + γ·Score3 (9)

[0161] where α, β, and γ are the weights of the scores of the three networks in the overall score respectively.

[0162] A font repair system based on deep meta-learning and generative adversarial network, characterized in that the system includes:

[0163] A data acquisition module for acquiring the existing calligraphy font dataset Dataset-1 and the stroke and structure data of the font;

[0164] A data processing module for processing the data in different ways for different tasks;

[0165] A Font-Meta module for completing incomplete calligraphy fonts;

[0166] A font review module for finding the best completed font;

[0167] A font output module for outputting the repaired font.

[0168] In summary, this patent proposes a font repair method and system based on deep meta-learning and generative adversarial networks. Using the existing calligraphy font dataset for meta-training dataset S1, by learning the features such as the strokes, structure, and content of the fonts in the existing font library and the process of completing the fonts with missing strokes, the prior knowledge of different dimensional features of each font and the meta-model Font-Meta and the font structure review model FSR-Net are obtained. Then, the data to be repaired is processed to obtain dataset S2, which is put into Font-Meta to learn the strokes, structure, content, and style of this type of calligraphy font. And by putting the font repaired by the font completion network FDR-Net of Font-Meta into the cyclic generative adversarial network for style re-transfer, a complete calligraphy font picture with the same style as the original font is obtained.

[0169] The font repair method and system proposed in the above embodiments of the present invention can learn the features such as the strokes, structure, and style of Chinese characters based on the existing data samples, and are more comprehensive than the prior art in completing the missing parts of Chinese character fonts. Applied in the field of calligraphy font repair, the present invention can reduce labor costs and improve the accuracy and integrity of font repair.

[0170] Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications, substitutions, and deformations can be made to these embodiments without departing from the principles and purposes of the present invention. The scope of the present invention is defined by the claims and their equivalents.

Claims

1. A font restoration method based on deep meta-learning and generative adversarial networks, It is characterized in that It includes the following steps: S1, using the data acquisition module to acquire the existing calligraphy font dataset Dataset-1 and the font stroke and structure data; S2, using the data processing module to process the data in different ways for different tasks; S3, use the Font-Meta module to complete the incomplete calligraphy fonts; S4. Use the font review module to find the best complementary font; S5, using the font output module to output the repaired font; Wherein, in S2, the data set Dataset-1 is further processed into Dataset-11, and the specific steps are: S21. Obtain complete calligraphy fonts and artistic fonts; S22, selecting the picture with the largest image entropy as data; S23, randomly generating irregular shapes of different sizes as font masks to simulate the defect of calligraphy fonts; S24 and font masks are added to the dataset Dataset-11 to obtain a set of class missing images; S25, pair the class missing image sets and construct the dataset Dataset-11; Wherein, the S3 includes: S31. Construct font completion network FDR-Net, cyclic generative adversarial network and font structure review model; S32. Use the MAML method to pre-train the font completion network FDR-Net; S33, putting the font repaired by the font completion network FDR-Net into a cyclic generative adversarial network for local style adjustment, and outputting a font image after style conversion; Wherein, in S32, the dataset Dataset-11 is put into the Font-Meta module for learning, and the font completion network FDR-Net is initialized and pre-trained, and the steps are: S321, obtaining font stroke data; S322, obtaining font structure data; S323, constructing a dataset of incomplete simulated calligraphy fonts, and pairing the incomplete data with the original data; S324, construct a font completion network FDR-Net model; S325. Train the font completion network FDR-Net.

2. According to the font restoration method based on deep meta-learning and generative adversarial network according to claim 1, It is characterized in that The steps for constructing the dataset Dataset-1 in S1 are: S11. Obtain the calligraphy work to be restored; S12, using a covering matrix to crop the calligraphy fonts of the original entire calligraphy work image, and expanding or compressing the cropped image to a picture with a size of 256×256; S13, converting the image after the unified size into a single channel, performing binarization processing, and obtaining a binary image of the word; S14. Binarize the picture set to construct the dataset Dataset-1.

3. According to the font restoration method based on deep meta-learning and generative adversarial network according to claim 2, It is characterized in that The S33 includes: S331, initialization and pre-training of cyclic generative adversarial networks; S332. Obtain the calligraphy font data to be repaired, finely tune the font completion network FDR-Net, and perform style and font content structure learning; S333. After pre-training and fine-tuning, obtain the finely tuned font completion network FDR-Net; S334. Complete the missing part to obtain a preliminarily repaired picture; S335. Input the preliminarily repaired picture into the cyclic generative adversarial network for local style transfer to obtain the transferred image.

4. The font repair method based on deep meta-learning and generative adversarial network according to claim 3, wherein, the pre-trained font completion network FDR-Net can complete the stroke structure of the missing part to obtain the completed calligraphy font M1.

5. The font repair method based on deep meta-learning and generative adversarial network according to claim 1, wherein, S4 includes: S41. Pre-train the stroke integrity network, structure integrity network, and style similarity network; S42. Input the style-converted image into the stroke integrity network for scoring to obtain Score1; S43. Input the style-converted image into the structure integrity network for scoring to obtain Score2; S44. Input the style-converted image into the style similarity network for scoring to obtain Score3; S45. By calculating the weighted average of the scoring results of the stroke integrity network, structure integrity network, and style similarity network respectively, the final score sequence can be obtained, and the repaired calligraphy font picture with the highest score is selected for output.

6. The font repair method based on deep meta-learning and generative adversarial network according to claim 1, wherein, The Font-Meta module includes the FDR-Net module and the CycleGAN module; The FDR-Net module is used to learn how to complete the font through the generative network and deep meta-learning method; The CycleGAN module is used for local style conversion of the completed font.

7. A font repair system based on deep meta-learning and generative adversarial network, wherein, used to implement the font repair method based on deep meta-learning and generative adversarial network according to claim 1, and it includes the following modules: The data acquisition module is used to acquire the existing calligraphy font dataset Dataset-1 and the stroke and structure data of the font; The data processing module is used to process the data in different ways for different tasks; The Font-Meta module is used to complete the incomplete calligraphy font; The font review module is used to find the best completed font; The font output module is used to output the repaired font.

Images