Ink trace repairing method and device, electronic equipment and storage medium
By combining transfer learning and generative adversarial training, a handwriting restoration model has been developed, which solves the problems of high cost and low efficiency in handwriting restoration of calligraphy works and achieves efficient and realistic automated restoration results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 张甲林
- Filing Date
- 2022-09-08
- Publication Date
- 2026-07-21
AI Technical Summary
In existing technologies, the restoration of calligraphy works relies on manual restoration, which is costly and inefficient, making it difficult to achieve low-cost and high-efficiency restoration, especially when faced with diverse calligraphy styles and rare calligraphy works.
By combining transfer learning and generative adversarial training, a handwriting restoration model is generated through pre-trained generative and discriminative models. The model then performs handwriting restoration using a small number of sample handwriting images under a target style, achieving automated restoration.
It reduces the difficulty of obtaining samples required for model training, improves restoration efficiency, and outputs highly realistic restored images. It can assist or replace expert restoration and automate the restoration of calligraphy of different styles.
Smart Images
Figure CN115564665B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and storage medium for handwriting restoration. Background Technology
[0002] Over time, many surviving calligraphy works have become illegible or partially damaged due to improper storage or natural aging. Restoring these works requires skilled professionals with extensive background knowledge and basic techniques to perform manual copying and imitation.
[0003] However, relying solely on manual labor for restoration is extremely costly and time-consuming. Therefore, achieving low-cost, high-efficiency handwriting restoration remains a pressing issue. Summary of the Invention
[0004] This invention provides a handwriting restoration method, apparatus, electronic device, and storage medium to address the shortcomings of existing technologies that rely on manual handwriting restoration, which is time-consuming and labor-intensive.
[0005] This invention provides a handwriting restoration method, comprising:
[0006] Obtain an image of damaged text in the target style;
[0007] The damaged handwriting image is input into the handwriting restoration model to obtain the restored handwriting image output by the handwriting restoration model;
[0008] The handwriting restoration model is obtained by applying the first sample handwriting image under the target style and combining it with the discriminative model to perform generative adversarial training based on the pre-trained generative model; the pre-trained generative model is trained based on the second sample handwriting images under various styles, and the discriminative model is used to distinguish between real images and synthetic images.
[0009] According to a handwriting restoration method provided by the present invention, the handwriting restoration model is trained based on the following steps:
[0010] The first sample handwriting image is randomly covered, and the randomly covered first sample handwriting image is used as the first damaged image. The randomly covered part of the first sample handwriting image is used as the first cropped image.
[0011] The first damaged image is input into the pre-trained generation model to obtain the first repaired image output by the pre-trained generation model;
[0012] The first repaired image and the first cropped image are respectively input into the discrimination model to obtain the discrimination results of the first repaired image and the first cropped image output by the discrimination model.
[0013] Based on the first restored image and the first cropped image, as well as the discrimination results of the first restored image and the first cropped image, the parameters of the pre-trained generative model are iterated to obtain the handwriting restoration model.
[0014] According to a handwriting restoration method provided by the present invention, the method involves iterating the parameters of a pre-trained generative model based on a first restored image and a first cropped image, as well as the discrimination results of the first restored image and the first cropped image, to obtain the handwriting restoration model. The method includes:
[0015] Based on the first repaired image and the first cropped image, determine the reconstruction loss;
[0016] Based on the discrimination results of the first repaired image and the first cropped image, the adversarial loss is determined;
[0017] Based on the reconstruction loss and the adversarial loss, the parameters of the pre-trained generative model are iterated to obtain the handwriting restoration model.
[0018] According to a handwriting restoration method provided by the present invention, the method involves iterating the parameters of a pre-trained generative model based on the reconstruction loss and the adversarial loss to obtain the handwriting restoration model, comprising:
[0019] Based on the reconstruction loss, the parameters of the pre-trained generative model are iterated, and based on the adversarial loss, the parameters of the pre-trained generative model and the discriminative model are iterated to obtain the handwriting restoration model.
[0020] According to a handwriting restoration method provided by the present invention, the step of inputting the first damaged image into the pre-trained generation model to obtain the first restored image output by the pre-trained generation model further includes:
[0021] Determine the initial generative model;
[0022] The second sample handwriting image is randomly covered, and the randomly covered second sample handwriting image is used as the second damaged image. The randomly covered part of the second sample handwriting image is used as the second cropped image.
[0023] The second damaged image is input into the initial generation model to obtain the second repaired image output by the initial generation model;
[0024] Based on the second cropped image and the second repaired image, the parameters of the initial generation model are iterated to obtain the pre-trained generation model.
[0025] According to a handwriting restoration method provided by the present invention, the method further includes iterating the parameters of the pre-trained generative model based on the first restored image and the first cropped image, as well as the discrimination results of the first restored image and the first cropped image, to obtain the handwriting restoration model.
[0026] Obtain test damage images under the target style;
[0027] The test damaged image is input into the handwriting restoration model to obtain the test restored image output by the handwriting restoration model;
[0028] The restoration quality is evaluated based on the test restored image to obtain the evaluation result;
[0029] If the evaluation result does not meet the character shape evaluation criteria, the second sample handwriting image is randomly covered again.
[0030] If the evaluation result meets the character shape evaluation criteria but does not meet the style evaluation criteria, the first sample handwriting image is randomly covered again.
[0031] According to a handwriting restoration method provided by the present invention, the first sample handwriting image and the second sample handwriting image are obtained by data processing, cleaning and data enhancement based on the original handwriting image.
[0032] The present invention also provides a handwriting restoration device, comprising:
[0033] The image acquisition unit is used to acquire images of damaged handwriting under the target style;
[0034] The handwriting restoration unit is used to input the damaged handwriting image into the handwriting restoration model to obtain the restored handwriting image output by the handwriting restoration model;
[0035] The handwriting restoration model is obtained by applying the first sample handwriting image under the target style and combining it with the discriminative model to perform generative adversarial training based on the pre-trained generative model; the pre-trained generative model is trained based on the second sample handwriting images under various styles, and the discriminative model is used to distinguish between real images and synthetic images.
[0036] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the handwriting restoration methods described above.
[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the handwriting restoration method as described above.
[0038] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the handwriting restoration method as described above.
[0039] The handwriting restoration method, apparatus, electronic device, and storage medium provided by this invention utilize a combination of transfer learning and generative adversarial training to obtain a handwriting restoration model for handwriting restoration. It requires only a small number of initial sample handwriting images of the target style to achieve handwriting restoration for that style, reducing the difficulty of obtaining samples needed for model training, improving model training efficiency, and ensuring that the restored handwriting images output by the model are closer to real images, guaranteeing naturalness and realism. The resulting handwriting restoration can assist and replace expert restoration processes, significantly improving the efficiency of handwriting restoration for calligraphy works and automating the restoration of calligraphy works of different styles. Attached Figure Description
[0040] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0041] Figure 1 This is one of the flowcharts illustrating the handwriting restoration method provided by the present invention;
[0042] Figure 2 This is a flowchart illustrating the training method of the handwriting restoration model provided by the present invention;
[0043] Figure 3 This is a schematic diagram of the reconstruction loss used in training the handwriting restoration model provided by the present invention;
[0044] Figure 4 This is a schematic diagram of the adversarial loss used in training the handwriting restoration model provided by this invention;
[0045] Figure 5 This is the second flowchart illustrating the handwriting restoration method provided by this invention;
[0046] Figure 6 This is a schematic diagram of the handwriting restoration device provided by the present invention;
[0047] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0049] Over time, many extant calligraphy works have suffered varying degrees of damage, staining, loss, weathering, and mold due to neglect and improper storage. Furthermore, the natural aging of the calligraphy medium itself can cause the characters to become blurred or partially damaged. Current methods of restoration primarily rely on expert copying, which presents the following problems:
[0050] First, calligraphy, as an ancient art form, encompasses various styles. For example, based on script type, it can be categorized into regular script, seal script, clerical script, running script, and cursive script; based on medium, it can be divided into rubbings, bronze inscriptions, bamboo and silk scrolls, oracle bone script, and stone inscriptions; furthermore, different calligraphers' works possess their own unique characteristics. Therefore, restoring the rich variety and diverse styles of calligraphy requires experts proficient in calligraphy art to utilize considerable background knowledge and basic skills for manual simulation, resulting in a long restoration period and a large workload. Second, many calligraphers' surviving works are rare, making perfect replication difficult even for calligraphy restoration experts. Moreover, the mediums on which many surviving works are displayed are often severely decayed and easily oxidized and corroded, making manual restoration highly susceptible to causing secondary damage.
[0051] With the continuous development of artificial intelligence, image recognition and semantic understanding technologies for text have been applied, making it possible to restore calligraphy based on artificial intelligence. However, due to the complexity of the Chinese language system, the difficulty of its restoration is far greater than that of other languages. How to restore Chinese characters remains an urgent problem to be solved.
[0052] Based on this, the present invention provides a method for handwriting restoration. Figure 1 This is one of the flowcharts illustrating the handwriting restoration method provided by this invention, such as... Figure 1 As shown, the method includes:
[0053] Step 110: Obtain the image of the damaged text in the target style.
[0054] Specifically, the damaged handwriting image is the image that needs to be restored, and the target style is the calligraphy style of the damaged handwriting image. The target style may include the script of the damaged handwriting image, the carrier of the damaged handwriting image, or the calligrapher corresponding to the damaged handwriting image, etc. The embodiments of the present invention do not specifically limit this.
[0055] Step 120: Input the damaged handwriting image into the handwriting restoration model to obtain the restored handwriting image output by the handwriting restoration model;
[0056] The handwriting restoration model is obtained by applying the first sample handwriting image under the target style and combining it with the discriminative model to perform generative adversarial training based on the pre-trained generative model; the pre-trained generative model is trained based on the second sample handwriting images under various styles, and the discriminative model is used to distinguish between real images and synthetic images.
[0057] Specifically, after obtaining the image of the damaged handwriting, the handwriting restoration model can be used to restore the handwriting in the image, thus obtaining a restored handwriting image. This restored handwriting image is used to fill or replace the image areas in the damaged handwriting image that need restoration.
[0058] The handwriting restoration model here is a pre-trained neural network model capable of restoring handwriting in images of a target style. Considering the complexity of the Chinese language and the difficulty of obtaining handwriting images of a single style, this embodiment of the invention combines a first sample handwriting image of the target style with second sample handwriting images of various styles during the training of the handwriting restoration model, applying transfer learning and generative adversarial techniques.
[0059] First, a pre-trained generative model needs to be obtained. This pre-trained model is trained using second-sample handwriting images of various styles. It's understandable that obtaining handwriting images of a single style is difficult, but if the style of the handwriting images is ignored, a large number of sample handwriting images can be easily obtained. Here, the sample handwriting images of various styles collected without considering the style are denoted as second-sample handwriting images. Through a large number of second-sample handwriting images, the pre-trained generative model can learn, during the pre-training phase, the characteristics of font structure, strokes, etc., that need to be considered when universally repairing damaged handwriting for various styles.
[0060] After obtaining the pre-trained generative model, the first sample handwriting image under the target style can be applied in conjunction with the discriminative model for generative adversarial training. Here, the sample handwriting image under the target style is the sample handwriting image belonging to the same style as the damaged handwriting image to be repaired, and is denoted as the first sample handwriting image. It can be understood that obtaining a first sample handwriting image of a single style is more difficult than obtaining a second sample handwriting image of any style, and the scale of image data that can be obtained is smaller. However, since the pre-trained generative model itself already has a preliminary handwriting repair capability, only a small number of first sample handwriting images are needed to achieve efficient transfer learning from the second sample handwriting image of any style to the first sample handwriting image of the target style.
[0061] Furthermore, the pre-trained generative model can serve as the generator in generative adversarial training, and be trained together with the discriminative model, which serves as the discriminator. In generative adversarial training, a first damaged image can be constructed by masking the first sample handwriting image, and the masked area in the first sample handwriting image can be used as the first cropped image.
[0062] The pre-trained generative model, acting as the generator, restores the text on the first damaged input image, outputting a restored image, i.e., the first restored image. The discriminative model, acting as the discriminator, distinguishes between the input image and the first restored image generated by the generator, or the actual first cropped image. In this process, the generator and discriminator work together. The generator aims to output a first restored image that is as similar as possible to the first cropped image, making it difficult for the discriminator to distinguish between the two. The discriminator, on the other hand, aims to output a discrimination result that matches the actual situation of the input image, achieving a more accurate and reliable discrimination effect.
[0063] The method provided in this invention combines transfer learning and generative adversarial training to obtain a handwriting restoration model for handwriting restoration. It requires only a small number of initial sample handwriting images of the target style to achieve handwriting restoration for that style, reducing the difficulty of obtaining samples needed for model training, improving model training efficiency, and ensuring that the restored handwriting images output by the model are closer to real images, guaranteeing naturalness and realism. The resulting handwriting restoration can assist and replace expert restoration processes, significantly improving the efficiency of handwriting restoration for calligraphy works and automating the restoration of calligraphy works of different styles.
[0064] Based on the above embodiments, Figure 2 This is a flowchart illustrating the training method of the handwriting restoration model provided by the present invention, as shown below. Figure 2 As shown, the handwriting restoration model is trained based on the following steps:
[0065] Step 210: Randomly cover the first sample handwriting image, take the randomly covered first sample handwriting image as the first damaged image, and take the randomly covered part of the first sample handwriting image as the first cropped image.
[0066] Step 220: Input the first damaged image into the pre-trained generation model to obtain the first repaired image output by the pre-trained generation model;
[0067] Step 230: Input the first repaired image and the first cropped image into the discrimination model respectively to obtain the discrimination result of the first repaired image and the discrimination result of the first cropped image output by the discrimination model;
[0068] Step 240: Based on the first restored image and the first cropped image, as well as the discrimination results of the first restored image and the first cropped image, perform parameter iteration on the pre-trained generative model to obtain the handwriting restoration model.
[0069] Specifically, in order to achieve transfer learning and generative adversarial training for the pre-trained generative model, it is necessary to first construct the samples required for transfer learning and generative adversarial training, namely the first damaged image and the first cut image obtained from the first sample handwriting image.
[0070] Here, the first sample handwriting image is a real image with complete handwriting. Random masking can be applied to the first sample handwriting image to simulate the damage to real calligraphy handwriting. The randomly masked portion of the first sample handwriting image is denoted as the first damaged image, and the randomly masked portion is denoted as the first cropped image. It can be understood that the first cropped image is the part of the first damaged image that is expected to be restored through handwriting restoration.
[0071] Furthermore, the damage to different first damaged images can be simulated by randomly covering them with polygons of different shapes and sizes at different locations. In the first damaged image, the randomly covered areas will be specially marked.
[0072] After obtaining the first damaged image, it can be input into the pre-trained generative model, which will then repair the characters in the damaged image and output the first repaired image.
[0073] After obtaining the first restored image, it can be input into a discrimination model, which will then judge the input restored image and output a discrimination result. Alternatively, after obtaining the first cropped image, it can also be input into the discrimination model, which will then judge the input cropped image and output a discrimination result. It is understood that the discrimination result here is represented as either "valid" or "fake," where "valid" means the image is a real image, and "fake" means the image is a synthesized virtual image.
[0074] After completing the above operations, the pre-trained generative model can be iterated by combining the first restored image and the first cropped image, as well as the discrimination results of the first restored image and the first cropped image. It can be understood that the difference between the first restored image and the first cropped image reflects the strength of the pre-trained generative model's handwriting restoration ability under the target style. The smaller the difference between the first restored image and the first cropped image, the stronger the pre-trained generative model's handwriting restoration ability under the target style; the larger the difference between the first restored image and the first cropped image, the weaker the pre-trained generative model's handwriting restoration ability under the target style. The discrimination results of the first restored image and the first cropped image reflect the game-theoretic ability of the pre-trained generative model (as a generator) and the discriminator model (as a discriminator) in generative adversarial processes. The closer the discrimination result of the first restored image is to "true," the more realistic the handwriting restoration effect of the pre-trained generative model is, and the weaker the discrimination ability of the discriminator model is. The closer the discrimination result of the first restored image is to "false," the worse the handwriting restoration effect of the pre-trained generative model is, and the stronger the discrimination ability of the discriminator model is. Therefore, by combining the first restored image and the first cropped image, as well as the discrimination results of the first restored image and the first cropped image, a loss function value for the pre-trained generation model can be generated, and then the parameters of the pre-trained generation model can be iterated to obtain the handwriting restoration model.
[0075] Based on any of the above embodiments, step 240 includes:
[0076] Based on the first repaired image and the first cropped image, determine the reconstruction loss;
[0077] Based on the discrimination results of the first repaired image and the first cropped image, the adversarial loss is determined;
[0078] Based on the reconstruction loss and the adversarial loss, the parameters of the pre-trained generative model are iterated to obtain the handwriting restoration model.
[0079] Specifically, Figure 3 This is a schematic diagram of the reconstruction loss used in training the handwriting restoration model provided by this invention, as shown below. Figure 3 As shown, in the process of transfer learning based on the pre-trained generative model, the reconstruction loss can be obtained by comparing the first cropped image and the first repaired image output by the pre-trained generative model. The reconstruction loss here is used to characterize the difference between the first cropped image and the first repaired image, and can be expressed as the following formula:
[0080]
[0081] In the formula, G represents the reconstruction loss; G represents the generator output, i.e. the output of the pre-trained generative model; x represents the missing cropped part of the real image, i.e. the first cropped image; z represents the missing image to be repaired, i.e. the first damaged image; G(z) represents the generator output after repairing the damaged image, i.e. the first repaired image.
[0082] Figure 4 This is a schematic diagram of the adversarial loss used in training the handwriting restoration model provided by this invention, as shown below. Figure 4 As shown, in generative adversarial training where the pre-trained generative model is treated as the generator, the adversarial loss can be determined by measuring the discrimination results output by the discriminator (acting as the discriminator) for the first repaired image and the first cropped image, respectively. Specifically, it can be expressed as the following formula:
[0083]
[0084] Where V represents the adversarial loss; D represents the discriminator output, i.e., the output of the discriminator model; G represents the generator output, i.e., the output of the pre-trained generator model; E represents the expected value; x represents the missing cropped portion of the real image, i.e., the first cropped image; z represents the missing image to be repaired, i.e., the first damaged image; G(z) represents the image repaired portion generated by the generator, i.e., the first repaired image; D(x) and D(G(z)) represent the discrimination results of the discriminator model for the first cropped image and the first repaired image, respectively.
[0085] After obtaining the reconstruction loss and adversarial loss respectively, the parameters of the pre-trained generative model can be iterated by combining the two. Specifically, the two can be added together as the total loss to iterate the parameters of the pre-trained generative model, or the two can be weighted and summed to obtain the total loss, and the parameters of the pre-trained generative model can be iterated based on the total loss. The embodiments of the present invention do not make specific limitations on this.
[0086] In addition, Figure 3 , Figure 4 The pre-trained generative model shown in the figure is represented by an encoder-decoder structure.
[0087] Based on any of the above embodiments, in step 240, the step of iterating the parameters of the pre-trained generative model based on the reconstruction loss and the adversarial loss to obtain the handwriting restoration model includes:
[0088] Based on the reconstruction loss, the parameters of the pre-trained generative model are iterated, and based on the adversarial loss, the parameters of the pre-trained generative model and the discriminative model are iterated to obtain the handwriting restoration model.
[0089] Specifically, in the parallel process of transfer learning and generative adversarial training, the reconstruction loss reflects the ability of the pre-trained generative model to restore handwriting in the target style, and is therefore only used for parameter iteration of the pre-trained generative model. The adversarial loss, on the other hand, reflects the competitive ability of the pre-trained generative model (as the generator) and the discriminator (as the discriminator) in generative adversarial training, and is therefore used for parameter iteration of the overall generative adversarial network composed of the pre-trained generative model and the discriminator. After parameter iteration based on the reconstruction loss and the adversarial loss, the pre-trained generative model becomes a handwriting restoration model capable of restoring handwriting in images of the target style.
[0090] Furthermore, in generative adversarial training, after calculating the adversarial loss, the discriminative model can perform error propagation and optimization on the adversarial loss of this batch according to a certain optimization strategy to obtain a certain discriminative ability. Subsequently, the discriminative model outputs the discrimination result by inputting the first damaged image again through the process of handwriting restoration and discrimination. After that, the weights of the discriminative model are locked, and the adversarial loss is backpropagated to the pre-trained generative model part according to the cross-entropy loss between the discriminative model output and the "true" value. The pre-trained generative model is then optimized according to a strategy different from that of the discriminative model. The above steps are repeated, and the pre-trained generative model and the discriminative model are iterated multiple times according to the game maxima mina optimization theory. The accuracy of the discriminative model result is maintained at around 50%, while the adversarial loss is continuously reduced, so that the output of the pre-trained generative model is as close as possible to the real situation.
[0091] In the method provided by this invention, the pre-trained generative model is optimized based on transfer learning and generative adversarial training. This process effectively captures and extracts the complex features of calligraphic handwriting, overcoming the problem of unsolvable optimization objectives in pre-trained generative models and significantly improving the efficiency of training data utilization.
[0092] Based on any of the above embodiments, before step 220, the method further includes:
[0093] Determine the initial generative model;
[0094] The second sample handwriting image is randomly covered, and the randomly covered second sample handwriting image is used as the second damaged image. The randomly covered part of the second sample handwriting image is used as the second cropped image.
[0095] The second damaged image is input into the initial generation model to obtain the second repaired image output by the initial generation model;
[0096] Based on the second cropped image and the second repaired image, the parameters of the initial generation model are iterated to obtain the pre-trained generation model.
[0097] Specifically, before performing transfer learning and generative adversarial training on the pre-trained generative model, it is necessary to obtain the pre-trained generative model first. The pre-trained generative model can be obtained by training a second sample handwriting image with no style restriction on the initial generative model.
[0098] Here, the initial generative model can be a neural network model with an encoder-decoder structure, and the model parameters are obtained through initialization. In order to pre-train the initial generative model, it is necessary to first construct the samples required for pre-training, namely the second damaged image and the second cropped image obtained from the second sample handwriting image.
[0099] Here, the second sample handwriting image is a real image with complete handwriting. Random masking can be applied to the second sample handwriting image to simulate the damage to real calligraphy. The randomly masked portion of the second sample handwriting image is denoted as the second damaged image, and the randomly masked portion is denoted as the second cropped image. It can be understood that the second cropped image is the part of the second damaged image that is expected to be restored through handwriting restoration.
[0100] After obtaining the second damaged image, it can be input into the initial generation model, which will then perform handwriting restoration on the damaged image and output the second restored image. During this process, the initial generation model encodes and decodes the second damaged image, while extracting feature information such as strokes, character shapes, and meanings to learn handwriting restoration capabilities.
[0101] After completing the above operations, the initial generation model can be iterated by combining the second repaired image and the second cropped image. It can be understood that the difference between the second repaired image and the second cropped image is used to reflect the strength of the initial generation model's ability to repair handwriting regardless of style. The smaller the difference between the second repaired image and the second cropped image, the stronger the initial generation model's ability to repair handwriting regardless of style. The larger the difference between the second repaired image and the second cropped image, the weaker the initial generation model's ability to repair handwriting in the target style.
[0102] Furthermore, based on the second repaired image and the second cropped image, the parameters of the initial generative model are iterated. Specifically, the reconstruction loss can be determined based on the second repaired image and the second cropped image, and the parameters of the initial generative model can be iterated based on the reconstruction loss to obtain the pre-trained generative model.
[0103] It is understandable that determining the reconstruction loss based on the second repaired image and the second cropped image is consistent with the idea of determining the reconstruction loss based on the first repaired image and the first cropped image in the above embodiments. It can also be implemented by applying the formula of reconstruction loss in the above embodiments, which will not be elaborated here.
[0104] Based on any of the above embodiments, step 240 is followed by:
[0105] Obtain test damage images under the target style;
[0106] The test damaged image is input into the handwriting restoration model to obtain the test restored image output by the handwriting restoration model;
[0107] The restoration quality is evaluated based on the test restored image to obtain the evaluation result;
[0108] If the evaluation result does not meet the character shape evaluation criteria, the second sample handwriting image is randomly covered again.
[0109] If the evaluation result meets the character shape evaluation criteria but does not meet the style evaluation criteria, the first sample handwriting image is randomly covered again.
[0110] Specifically, after the handwriting restoration model is trained, it can be tested.
[0111] Specifically, during the testing process, it is first necessary to obtain a test damaged image under the target style. Here, the test damaged image can be obtained by randomly masking a real image under the target style with complete handwriting, or it can be a real image under the target style with incomplete handwriting that is directly obtained. This embodiment of the invention does not make specific limitations on this.
[0112] After obtaining the test damaged image under the target style, the test damaged image can be input into the handwriting restoration model. The handwriting restoration model will then perform handwriting restoration under the target style on the test damaged image, thereby obtaining and outputting the restoration result for the test damaged image, i.e., the test restored image.
[0113] After obtaining the test restoration image, a restoration quality evaluation can be performed on it to obtain the evaluation result. Here, the restoration quality evaluation reflects the quality of the handwriting restoration model in restoring the handwriting to the damaged test image. The restoration quality evaluation can be multi-dimensional, such as including a restoration quality evaluation based on the character shape or style. Specifically, the restoration quality evaluation based on the character shape evaluates the quality of the test restoration image in restoring the character shape and meaning. It can specifically evaluate the distinguishability of the character shape and meaning in the test restoration image, and whether the handwriting in the test restoration image conforms to the basic structure of calligraphy. The restoration quality evaluation based on the style evaluates the quality of the test restoration image in restoring the calligraphy style. Specifically, it can evaluate whether the strokes, turns, etc., of the handwriting in the test restoration image conform to the target style, and whether the structure between the strokes in the handwriting in the test restoration image conforms to the target style.
[0114] After obtaining the evaluation results, it can be determined whether the evaluation results meet the pre-set font evaluation criteria and style evaluation criteria.
[0115] Understandably, if the evaluation results do not meet the character shape evaluation criteria, it means that the handwriting restoration model has a weak ability to restore the character shape and meaning. In this case, it is necessary to randomly cover the second sample handwriting image again to generate a new second damaged image and a second cut image. This handwriting restoration model can then be used as the initial generation model to perform pre-training on the initial generation model, as well as transfer learning and generative adversarial training on the pre-trained model, in order to further optimize the handwriting restoration model.
[0116] If the evaluation result meets the character shape evaluation criteria but not the style evaluation criteria, it means that the handwriting restoration model has a weak restoration ability under the target style. In this case, it is necessary to randomly cover the first sample handwriting image again to generate a new first damaged image and a first cropped image. This way, the handwriting restoration model at this time can be used as a pre-trained generative model. Then, transfer learning and generative adversarial training can be performed on the pre-trained model to achieve further optimization of the handwriting restoration model.
[0117] Based on any of the above embodiments, the first sample handwriting image and the second sample handwriting image are obtained by data processing, cleaning and data enhancement based on the original handwriting image.
[0118] Specifically, before training the model based on the first sample handwriting image and the second sample handwriting image, it is necessary to collect and obtain the first sample handwriting image and the second sample handwriting image.
[0119] Here, regarding the acquisition of the first and second sample handwriting images, we can first collect original handwriting images based on the Chinese calligraphy dataset. The Chinese calligraphy dataset here can be a publicly available dataset from various platforms. The collection process can be assisted by the Python web scraping package (BeautifulSoup). Each original handwriting image collected in this way is a single image containing a single calligraphy character, with a relatively clear difference between the foreground and background colors.
[0120] Each collected original handwriting image may possess different features and attributes, such as being categorized by calligrapher, source of the work, creation time and location, script type, and pen style. After sorting and categorizing, each original handwriting image under each category can be cleaned and processed. First, it is stretched and scaled to the same size (e.g., 64x64 pixels). Second, the images are binarized to make them usable data, and then incorporated into a unified dataset. This unified dataset can be a dataset set up for second sample handwriting images of any style, or a dataset set up for first sample handwriting images of a target style; this embodiment of the invention does not specifically limit this.
[0121] Based on this, data augmentation can be performed on the images in the dataset to expand the dataset size. Specifically, data augmentation can include noise overlay, flipping and rotation, scaling, and other methods.
[0122] It should be noted that the test repair images used for model testing can also be obtained by randomly covering the sample handwriting images obtained based on the above steps. Specifically, the original handwriting images can be first processed, cleaned, and augmented, and then the training and test sets can be divided according to the ratio.
[0123] Based on any of the above embodiments Figure 5 This is the second flowchart illustrating the handwriting restoration method provided by this invention, as shown below. Figure 5 As shown, for the training phase of the handwriting restoration model, firstly, second sample handwriting images of various styles can be collected. Then, through random occlusion, a second damaged image and a second cropped image of the occluded part are obtained. The second damaged image is used as the model input, and the second cropped image is used as the training label to train the initial generation model. During this process, the reconstruction loss is determined based on the second restored image corresponding to the second damaged image and the second cropped image. The reconstruction loss is then input into the optimizer to iterate the parameters of the initial generation model, thereby obtaining the pre-trained generation model.
[0124] Subsequently, for the first sample handwriting images of the pre-collected target style, a first damaged image and a first cropped image of the covered portion can be obtained through random occlusion. The first damaged image is used as input to a pre-trained generative model, which is then combined with a discriminative model for transfer learning and generative adversarial training. During this process, a reconstruction loss is determined based on the first restored image corresponding to the first damaged image and the first cropped image; an adversarial loss is determined based on the discriminative results of the first restored image and the first cropped image; the parameters of the pre-trained generative model are iterated based on the reconstruction loss; and the parameters of both the pre-trained generative model and the discriminative model are iterated based on the adversarial loss, thus obtaining the handwriting restoration model.
[0125] Following this, in the testing phase of the handwriting restoration model, the model can randomly cover the pre-collected test handwriting images of the target style to obtain the covered test damaged images. The test damaged images are then used as input to the handwriting restoration model to obtain the test restored images output by the handwriting restoration model. The restoration quality of the test restored images is then evaluated, and the evaluation results determine whether it is necessary to return to the above training phase for further optimization of the handwriting restoration model.
[0126] The handwriting restoration method provided in this invention differs from ordinary image and text restoration generative networks. The handwriting restoration model combines the characteristics of generative adversarial networks and transfer learning, and designs different optimization objectives. It can be used efficiently and focusedly for the extraction and feature transfer of calligraphy image features, and can restore calligraphy damaged handwriting images of a specified style.
[0127] Furthermore, in the embodiments of the present invention, the damaged images used for model training are all obtained by random masking, thereby enhancing the generalization ability of the handwriting restoration model.
[0128] Furthermore, the embodiments of the present invention apply a feature transfer training objective, which can perform feature transfer learning on the pre-trained generative model after generalization training. When the model has good adaptability to the overall logic of handwriting restoration, it can further learn the specific style of the restoration task to achieve the best task performance.
[0129] The handwriting restoration device provided by the present invention is described below. The handwriting restoration device described below can be referred to in correspondence with the handwriting restoration method described above.
[0130] Figure 6 This is a schematic diagram of the handwriting restoration device provided by the present invention, as shown below. Figure 6 As shown, the device includes:
[0131] Image acquisition unit 610 is used to acquire images of damaged handwriting under the target style;
[0132] The handwriting restoration unit 620 is used to input the damaged handwriting image into the handwriting restoration model to obtain the restored handwriting image output by the handwriting restoration model;
[0133] The handwriting restoration model is obtained by applying the first sample handwriting image under the target style and combining it with the discriminative model to perform generative adversarial training based on the pre-trained generative model; the pre-trained generative model is trained based on the second sample handwriting images under various styles, and the discriminative model is used to distinguish between real images and synthetic images.
[0134] The apparatus provided in this invention uses a combination of transfer learning and generative adversarial training to obtain a handwriting restoration model for handwriting restoration. It requires only a small number of initial sample handwriting images of the target style to achieve handwriting restoration for that style, reducing the difficulty of obtaining samples needed for model training, improving model training efficiency, and ensuring that the restored handwriting images output by the model are closer to real images, guaranteeing naturalness and realism. The resulting handwriting restoration can assist and replace expert restoration processes, significantly improving the efficiency of handwriting restoration for calligraphy works and automating the restoration of calligraphy works of different styles.
[0135] Based on any of the above embodiments, the device further includes a model training unit, used for:
[0136] The first sample handwriting image is randomly covered, and the randomly covered first sample handwriting image is used as the first damaged image. The randomly covered part of the first sample handwriting image is used as the first cropped image.
[0137] The first damaged image is input into the pre-trained generation model to obtain the first repaired image output by the pre-trained generation model;
[0138] The first repaired image and the first cropped image are respectively input into the discrimination model to obtain the discrimination results of the first repaired image and the first cropped image output by the discrimination model.
[0139] Based on the first restored image and the first cropped image, as well as the discrimination results of the first restored image and the first cropped image, the parameters of the pre-trained generative model are iterated to obtain the handwriting restoration model.
[0140] Based on any of the above embodiments, the model training unit is specifically used for:
[0141] Based on the first repaired image and the first cropped image, determine the reconstruction loss;
[0142] Based on the discrimination results of the first repaired image and the first cropped image, the adversarial loss is determined;
[0143] Based on the reconstruction loss and the adversarial loss, the parameters of the pre-trained generative model are iterated to obtain the handwriting restoration model.
[0144] Based on any of the above embodiments, the model training unit is specifically used for:
[0145] Based on the reconstruction loss, the parameters of the pre-trained generative model are iterated, and based on the adversarial loss, the parameters of the pre-trained generative model and the discriminative model are iterated to obtain the handwriting restoration model.
[0146] Based on any of the above embodiments, the model training unit is further configured to:
[0147] Determine the initial generative model;
[0148] The second sample handwriting image is randomly covered, and the randomly covered second sample handwriting image is used as the second damaged image. The randomly covered part of the second sample handwriting image is used as the second cropped image.
[0149] The second damaged image is input into the initial generation model to obtain the second repaired image output by the initial generation model;
[0150] Based on the second cropped image and the second repaired image, the parameters of the initial generation model are iterated to obtain the pre-trained generation model.
[0151] Based on any of the above embodiments, the device further includes a model testing unit, used for:
[0152] Obtain test damage images under the target style;
[0153] The test damaged image is input into the handwriting restoration model to obtain the test restored image output by the handwriting restoration model;
[0154] The restoration quality is evaluated based on the test restored image to obtain the evaluation result;
[0155] If the evaluation result does not meet the character shape evaluation criteria, the second sample handwriting image is randomly covered again.
[0156] If the evaluation result meets the character shape evaluation criteria but does not meet the style evaluation criteria, the first sample handwriting image is randomly covered again.
[0157] Based on any of the above embodiments, the first sample handwriting image and the second sample handwriting image are obtained by data processing, cleaning and data enhancement based on the original handwriting image.
[0158] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a handwriting restoration method, which includes: acquiring a damaged handwriting image under a target style; inputting the damaged handwriting image into a handwriting restoration model to obtain a restored handwriting image output by the handwriting restoration model; the handwriting restoration model is obtained by applying a first sample handwriting image under the target style and combining it with a discriminative model for generative adversarial training based on a pre-trained generative model; the pre-trained generative model is trained based on second sample handwriting images under various styles, and the discriminative model is used to distinguish between real images and synthetic images.
[0159] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0160] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the handwriting restoration method provided by the above methods. The method includes: acquiring a damaged handwriting image under a target style; inputting the damaged handwriting image into a handwriting restoration model to obtain a restored handwriting image output by the handwriting restoration model; the handwriting restoration model is obtained by applying a first sample handwriting image under the target style and combining it with a discriminative model for generative adversarial training based on a pre-trained generative model; the pre-trained generative model is trained based on second sample handwriting images under various styles, and the discriminative model is used to distinguish between real images and synthetic images.
[0161] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the line character restoration method provided by the above methods. The method includes: acquiring a damaged character image under a target style; inputting the damaged character image into a character restoration model to obtain a restored character image output by the character restoration model; the character restoration model is obtained by applying a first sample character image under the target style and combining it with a discriminative model for generative adversarial training based on a pre-trained generative model; the pre-trained generative model is trained based on second sample character images under various styles, and the discriminative model is used to distinguish between real images and synthetic images.
[0162] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0163] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for handwriting restoration, characterized in that, include: Obtain an image of damaged text in the target style; The damaged handwriting image is input into the handwriting restoration model to obtain the restored handwriting image output by the handwriting restoration model; The handwriting restoration model is obtained by applying the first sample handwriting image under the target style and combining it with the discriminative model to perform generative adversarial training based on the pre-trained generative model. The pre-trained generative model is trained based on second sample handwriting images under various styles, and the discriminative model is used to distinguish between real images and synthetic images. The handwriting restoration model was trained based on the following steps: The first sample handwriting image is randomly covered, and the randomly covered first sample handwriting image is used as the first damaged image. The randomly covered part of the first sample handwriting image is used as the first cropped image. The first damaged image is input into the pre-trained generation model to obtain the first repaired image output by the pre-trained generation model; The first repaired image and the first cropped image are respectively input into the discrimination model to obtain the discrimination results of the first repaired image and the first cropped image output by the discrimination model. Based on the first restored image and the first cropped image, as well as the discrimination results of the first restored image and the first cropped image, the parameters of the pre-trained generative model are iterated to obtain the handwriting restoration model.
2. The handwriting restoration method according to claim 1, characterized in that, The step of iterating the parameters of the pre-trained generative model based on the first restored image and the first cropped image, as well as the discrimination results of the first restored image and the first cropped image, to obtain the handwriting restoration model includes: Based on the first repaired image and the first cropped image, determine the reconstruction loss; Based on the discrimination results of the first repaired image and the first cropped image, the adversarial loss is determined; Based on the reconstruction loss and the adversarial loss, the parameters of the pre-trained generative model are iterated to obtain the handwriting restoration model.
3. The handwriting restoration method according to claim 2, characterized in that, The process of iterating the parameters of the pre-trained generative model based on the reconstruction loss and the adversarial loss to obtain the handwriting restoration model includes: Based on the reconstruction loss, the parameters of the pre-trained generative model are iterated, and based on the adversarial loss, the parameters of the pre-trained generative model and the discriminative model are iterated to obtain the handwriting restoration model.
4. The handwriting restoration method according to claim 1, characterized in that, The step of inputting the first damaged image into the pre-trained generation model to obtain the first repaired image output by the pre-trained generation model also includes: Determine the initial generative model; The second sample handwriting image is randomly covered, and the randomly covered second sample handwriting image is used as the second damaged image. The randomly covered part of the second sample handwriting image is used as the second cropped image. The second damaged image is input into the initial generation model to obtain the second repaired image output by the initial generation model; Based on the second cropped image and the second repaired image, the parameters of the initial generation model are iterated to obtain the pre-trained generation model.
5. The handwriting restoration method according to claim 4, characterized in that, The process involves iterating the parameters of the pre-trained generative model based on the first restored image and the first cropped image, as well as the discrimination results of the first restored image and the first cropped image, to obtain the handwriting restoration model. This process further includes: Obtain test damage images under the target style; The test damaged image is input into the handwriting restoration model to obtain the test restored image output by the handwriting restoration model; The restoration quality is evaluated based on the test restored image to obtain the evaluation result; If the evaluation result does not meet the character shape evaluation criteria, the second sample handwriting image is randomly covered again. If the evaluation result meets the character shape evaluation criteria but does not meet the style evaluation criteria, the first sample handwriting image is randomly covered again.
6. The handwriting restoration method according to any one of claims 1 to 5, characterized in that, The first sample handwriting image and the second sample handwriting image are obtained by data processing, cleaning and data enhancement based on the original handwriting image.
7. A handwriting restoration device, characterized in that, include: The image acquisition unit is used to acquire images of damaged handwriting under the target style; The handwriting restoration unit is used to input the damaged handwriting image into the handwriting restoration model to obtain the restored handwriting image output by the handwriting restoration model; The handwriting restoration model is obtained by applying the first sample handwriting image under the target style and combining it with the discriminative model to perform generative adversarial training based on the pre-trained generative model. The pre-trained generative model is trained based on second sample handwriting images under various styles, and the discriminative model is used to distinguish between real images and synthetic images. The handwriting restoration model was trained based on the following steps: The first sample handwriting image is randomly covered, and the randomly covered first sample handwriting image is used as the first damaged image. The randomly covered part of the first sample handwriting image is used as the first cropped image. The first damaged image is input into the pre-trained generation model to obtain the first repaired image output by the pre-trained generation model; The first repaired image and the first cropped image are respectively input into the discrimination model to obtain the discrimination results of the first repaired image and the first cropped image output by the discrimination model. Based on the first restored image and the first cropped image, as well as the discrimination results of the first restored image and the first cropped image, the parameters of the pre-trained generative model are iterated to obtain the handwriting restoration model.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the handwriting restoration method as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the handwriting restoration method as described in any one of claims 1 to 6.