An ancient text image restoration method, system, medium, device and terminal

Abstract

This invention belongs to the field of image processing technology and discloses a method, system, medium, device, and terminal for ancient text image restoration. It constructs a dataset and an ancient text image restoration network model. An edge restoration network is used to repair the edge structure of the ancient text image, reconstructing the edge information of the missing areas. A pre-trained local text learning network is used to perform content restoration on the locally missing areas, obtaining a locally content-restored image. A dual-cross encoder and a multi-scale fusion block are introduced into the content restoration network, and a Laplacian pyramid restoration network is used to progressively restore the image using the low and high layers of the content restoration network. The restored ancient text image and the corresponding undamaged ancient text image are used as inputs to the ancient text image restoration network model for training and optimization. The optimized image restoration network model is then used to restore the missing ancient text image. The ancient text image restored using this invention is more realistic and clearer.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a method, system, medium, device and terminal for ancient text image restoration. Background Technology

[0002] Currently, ancient texts serve as a bridge connecting historical culture and modern life. However, with the passage of time and development, many ancient texts suffer varying degrees of damage during their transmission due to various unstable preservation factors, such as natural aging, mold, dampness, wind and rain erosion, and human-caused damage. These factors can cause the writing to become blurred or partially damaged, thus affecting the integrity and readability of the books. Current ancient text restoration techniques mainly rely on manual restoration, which is time-consuming and inefficient. It not only requires a high level of skill but also often fails to achieve ideal results. Directly restoring ancient texts can easily damage them, which contradicts our goal of preserving them. Over time, the damage to ancient texts continues to worsen, and most damaged texts are difficult to restore effectively. Therefore, using computer-aided image restoration of damaged ancient texts is an efficient and safe solution for ancient text restoration.

[0003] Ancient text image restoration refers to using feature information from known regions to fill in missing content. While deep learning-based image restoration has made significant progress, restoring ancient text images remains challenging. First, there is currently no complete and publicly available dataset of ancient text images, nor a Mask dataset that accurately reflects the actual defects in text images. Second, existing image restoration methods primarily target natural images; failing to consider the characteristics of ancient text images, these methods result in illogical content structure and blurred textures when applied to ancient text images. Furthermore, current image restoration networks perform restoration purely at the image level, raising concerns about the semantic plausibility of missing text, leading to seemingly plausible but ambiguous restoration results. Therefore, a novel method for ancient text image restoration is urgently needed.

[0004] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0005] (1) Current ancient text restoration techniques mainly rely on manual restoration, which has a long cycle and low efficiency. It not only requires high level of manual skills, but also often fails to achieve the desired results. On the other hand, the method of directly restoring the ancient text is counterproductive because the ancient text is easily damaged.

[0006] (2) There is currently no complete and publicly available ancient text image dataset or Mask dataset that matches the actual defects of text images; existing image restoration methods are mainly aimed at natural images. Since they do not take into account the characteristics of ancient text images, they will cause problems such as unreasonable content structure and blurred texture; current image restoration networks restore purely from the image level, which has the problem of the semantic rationality of missing text, resulting in restoration results that seem reasonable but are actually ambiguous. Summary of the Invention

[0007] To address the problems existing in the prior art, this invention provides a method, system, medium, device, and terminal for ancient text image restoration, and particularly relates to a method, system, medium, device, and terminal for ancient text image restoration based on edge guidance and Laplacian pyramid decomposition.

[0008] This invention is implemented as follows: an ancient text image restoration method, comprising: constructing a dataset and an ancient text image restoration network model; using an edge restoration network to restore the edge structure of the ancient text image and reconstruct the edge information of the missing area; using a pre-trained local text learning network to perform content restoration on the local missing area, obtaining a locally content-restored image; introducing a dual cross encoder and a multi-scale fusion block into the content restoration network, and using the low and high layers of the Laplacian pyramid restoration network to progressively restore the image; using the restored ancient text image and the corresponding lossless ancient text image as input to the ancient text image restoration network model, training and optimizing the ancient text image restoration network model, and using the optimized image restoration network model to restore the missing ancient text image.

[0009] Furthermore, the ancient text image restoration method includes the following steps:

[0010] Step 1: Construct the ancient text image dataset, the text image dataset, and the Mask image dataset;

[0011] Step 2: Construct an edge restoration network to extract edges from the damaged ancient text image; use the edge restoration network to restore the missing text structure information, generate a text edge map, and construct an edge guidance map;

[0012] Step 3: Construct a local text learning network. Randomly crop four different missing image patches from the missing area of ​​the input image. Use the pre-trained text learning network to repair the missing text information in the image patches, and stitch the repair results into the missing image to obtain a local content repair image.

[0013] Step four involves constructing a Laplacian pyramid restoration network. This network decomposes the local content restoration map into a high-level detail map and a low-level content map using the Laplacian pyramid. For low-level content restoration, the downsampled edge guide map and the low-level content map are input into the content restoration network to repair the missing areas, generating a low-resolution restoration result map. For high-level detail optimization, the low-resolution restoration result is upsampled and aggregated with the high-level detail map to obtain an intermediate restoration result. The content restoration network then uses the intermediate restoration result and the edge guide map as input to optimize the details of the missing areas, generating a high-quality restoration result.

[0014] Step 5: Construct a discriminator network model. Train and optimize the generator network model and discriminator network model using a loss function to obtain optimized generator network model and discriminator network model. Use the optimized generator network model and discriminator network model to repair the missing ancient text image.

[0015] Furthermore, the generator of the edge restoration network in step two adopts an encoder-decoder structure. The global content network contains an encoder and a decoder. The encoder uses convolution operations, instance normalization and linear rectified activation function to downsample the input image and extract image features. The decoder takes the downsampled image feature results as the network input, performs upsampling operation through nearest neighbor interpolation and reconstructs the image to obtain the restored edge image X.

[0016] Furthermore, in step three, the local text learning network constructs a text image dataset by collecting text from text images, where each image contains one character. During training, a random damaged region is added to each text image in the dataset, forming a damaged text image which is then input into the text learning network, with the corresponding complete text image serving as a constraint. After training, the pre-trained text learning model is applied to text image restoration. Specifically, this includes cropping four 32-bit regions from the input image. 32 different blocks, each containing different missing regions, are input into a pre-trained text learning network to obtain the restoration results of the missing text in the image blocks; the restoration results of the image blocks are then stitched together with the missing image to obtain a local content restoration image P.

[0017] Furthermore, in step four, the Laplacian pyramid restoration network is used at two different scales to progressively restore the damaged image. Specifically, this includes: performing Laplacian pyramid decomposition on the local content restoration map, decomposing it into a high-level detail map and a low-level content map; constructing a content restoration network, using the low-scale edge guide map and content map as inputs to restore the low-level content map; upsampling the restoration result and combining it with the high-level detail map to obtain an intermediate restoration map, and using the edge guide map as inputs to the content restoration network to obtain the final image restoration result I.

[0018] Furthermore, the discriminator network model in step five is the PatchGAN network. The PatchGAN network consists of five convolutional layers with a kernel size of 4×4 and a stride of 1. The output dimensions of the first four convolutional layers are 64, 128, 256, and 512, respectively. The last convolutional layer is input mapping, takes a 30×30 matrix as output, and uses the mean of the output matrix as the output result of the discriminator network model.

[0019] The loss functions include the edge loss function, the text content loss function, and the content loss function; the formula for the edge loss function is as follows:

[0020] ;

[0021] In the formula, and The loss weight is set to =4, =1; It is the discriminator of the edge repair network; It is a generator for edge repair networks.

[0022] ;

[0023] In the formula, It is a complete edge map obtained by preprocessing a real text image. Edge map generated by the edge repair network.

[0024] ;

[0025] In the formula, Represents a text-defective image. Represents the edge of the defect; The mask image is represented by a binary matrix, where 1 represents a missing region and 0 represents an unmissing region.

[0026] The text content loss includes text graph reconstruction loss and adversarial loss. The text content loss function is:

[0027] ;

[0028] In the formula, and The loss weight is set to =4, =1; It is the discriminator for text learning networks; It is a generator for text learning networks.

[0029] ;

[0030] ;

[0031] In the formula, It has a resolution of 32 32 text images, Text images generated for a text learning network. This indicates a missing or damaged text image.

[0032] The Laplacian Pyramid Inpainting Network utilizes a content inpainting network to progressively inpaint images at both high and low levels, comparing the generated image pixel-by-pixel with the original image. Inpainting Loss Including reconstruction losses and generating adversarial losses The content loss function is as follows:

[0033] ;

[0034] In the formula, and The loss weight is set to =1, =0.1.

[0035] ;

[0036] ;

[0037] In the formula, This is the result of low-resolution layer repair. This is the result of downsampling the real image; The high-resolution layer repair result is the final repair result generated by the network; It is a real image; Represents a text-defective image. For edge guidance maps, Represented as a real text image; It is the discriminator of the edge repair network. It is a generator for edge repair networks.

[0038] During network training, the backpropagation algorithm is used to update the parameters of the generator network model and the discriminator network model. After each iteration, the parameters of the discriminator network model are updated once, and then the parameters of the generator network model are updated once. At the same time, the model test score is calculated, and the generator network model with the highest final score is used as the optimized generator network model.

[0039] Another object of the present invention is to provide an ancient text image restoration system applying the aforementioned ancient text image restoration method, the ancient text image restoration system comprising:

[0040] The dataset building module is used to build ancient text image datasets, text image datasets, and Mask image datasets adapted to ancient text image datasets;

[0041] The edge restoration module is used to construct an edge restoration network, which is then used to restore the edge structure of ancient text images and reconstruct the edge information of the missing areas.

[0042] The local restoration module is used to construct a local text learning network, which uses a pre-trained text learning network to restore content in local missing areas, resulting in a locally restored image.

[0043] The progressive repair module is used to construct the Laplacian pyramid repair network, and then use the low and high layers of the content repair network to progressively repair the image.

[0044] The training and optimization module is used to build a discriminator network model. The ancient text image restoration network model is trained and optimized by taking the restored ancient text image and the corresponding undamaged ancient text image as input to the discriminator network. The optimized image restoration network model is then used to restore the damaged ancient text image.

[0045] Another object of the present invention is to provide a computer device including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the ancient text image restoration method.

[0046] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the ancient text image restoration method.

[0047] Another objective of this invention is to provide an information data processing terminal for implementing the ancient text image restoration system.

[0048] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0049] First, the ancient text image restoration method based on edge guidance and Laplacian pyramid decomposition of the present invention first constructs an ancient text image dataset, a text image dataset, and a mask image dataset; the damaged ancient text image undergoes edge extraction, and the obtained damaged edge map is input into an edge restoration network for restoration to reconstruct the edge information of the missing area; then, a pre-trained text learning network is used to perform content restoration on the local missing area to obtain a locally content-restored image; Laplacian decomposition is performed, and the Laplacian pyramid restoration network uses the low-level and high-level layers of the content restoration network to progressively restore the image. A dual cross encoder and a multi-scale fusion block are introduced into the content restoration network to help obtain more effective feature information and generate an image restoration result with complete texture structure; the restored ancient text image and the corresponding undamaged ancient text image are used as input to the discriminator network; the generator network model and the discriminator network model are trained and optimized through a loss function; the optimized generator network model is used to restore the damaged ancient text image.

[0050] This invention provides an ancient text image restoration method based on edge guidance and Laplacian pyramid decomposition. It constructs a novel ancient text image restoration network model. First, an edge restoration network is used to repair the edge structure of the ancient text image, reconstructing the edge information of the missing areas. Then, a pre-trained text learning network is used to perform content restoration on the local missing areas, obtaining a locally content-restored image, which is then subjected to Laplacian decomposition. Next, the Laplacian pyramid restoration network is used to progressively restore the image using the low and high layers of the content restoration network. Furthermore, a dual cross-encoder and a multi-scale fusion block are introduced into the content restoration network, which helps to obtain more effective feature information and generate ancient text image restoration results with complete texture structure.

[0051] Second, the restored ancient text images processed using the ancient text image restoration method based on edge guidance and Laplacian pyramid decomposition provided by this invention are more realistic and clearer.

[0052] This invention utilizes an edge-to-global network model to complete the task of ancient text image restoration, and in terms of visual effect, it can obtain more complete and reasonable ancient text image restoration results.

[0053] Third, as supplementary evidence of the inventive step of the claims of this invention, it is also reflected in the following important aspects:

[0054] (1) The expected benefits and commercial value of the technical solution of this invention after transformation are as follows:

[0055] When repairing damaged ancient books, the complete text image can be input into this model for training, enabling rapid and large-scale repair of damaged ancient books, greatly improving the efficiency of ancient book repair and saving a lot of manpower, material resources and financial resources.

[0056] (2) The technical solution of this invention fills a technical gap in the industry both domestically and internationally:

[0057] To address the current lack of methods for restoring damaged text images, this paper achieves good restoration results through an edge-to-global network model. This method not only effectively restores realistic damaged text images but also performs well on damaged rubbings images, yielding satisfactory results and demonstrating strong generalization ability. With sufficient training data, this method will be adaptable to restoring damaged images in more scenarios.

[0058] (3) Whether the technical solution of the present invention solves the technical problem that people have long wanted to solve but have never been able to solve successfully:

[0059] Current ancient text restoration techniques primarily rely on manual restoration, which is time-consuming and inefficient. It not only demands high levels of technical skill but also often fails to achieve satisfactory results. Direct restoration on ancient texts easily damages them, contradicting our goal of preserving them. Over time, the damage to ancient texts worsens, making effective restoration of most damaged texts impossible. Therefore, this invention utilizes computer-aided restoration of damaged ancient text images, providing an efficient and safe solution for ancient text restoration. Attached Figure Description

[0060] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart of the ancient text image restoration method provided in the embodiments of the present invention;

[0062] Figure 2 This is a schematic diagram illustrating the principle of the ancient text image restoration method provided in this embodiment of the invention;

[0063] Figure 3 This is an interactive diagram of the ancient text image restoration system provided in an embodiment of the present invention;

[0064] Figure 4A This is a schematic diagram of the rubbing image before restoration provided in an embodiment of the present invention;

[0065] Figure 4B This is a schematic diagram of the restored rubbing image provided in an embodiment of the present invention;

[0066] Figure 5A This is a schematic diagram of the ancient text image before restoration, provided in an embodiment of the present invention;

[0067] Figure 5B This is a schematic diagram of the restored ancient text image provided in an embodiment of the present invention; Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0069] The ancient text image restoration method provided in this embodiment of the invention includes the following steps:

[0070] Step 1, Dataset Construction. An ancient text image dataset and a text image dataset were constructed for the restoration analysis of ancient text images. Considering the feasibility of ancient text image restoration, a Mask dataset adapted to the ancient text image dataset was also constructed to allow training to more closely resemble real-world image restoration, achieving the goal of restoring truly damaged ancient text images.

[0071] Step 2: Construct the edge repair network. First, use the edge repair network to repair the missing text structure information, generate a complete text edge map, and construct an edge guidance map.

[0072] Step 3: Construct a local text learning network. Randomly crop four different missing image patches from the missing region of the input image. Use the pre-trained text learning network to repair the missing text information in the image patches, and stitch the repaired results into the missing image to obtain a local content repair image.

[0073] Step 4: Construct a Laplacian pyramid restoration network. Using the Laplacian pyramid, the local content restoration map is decomposed into a high-level detail map and a low-level content map. For low-level content restoration, the downsampled edge guide map and low-level content map are input into the content restoration network to repair the missing areas, generating a low-resolution restoration result map. For high-level detail optimization, the low-resolution restoration result is upsampled and aggregated with the high-level detail map to obtain an intermediate restoration result. The content restoration network uses the intermediate restoration result and the edge guide map as input to optimize the details of the missing areas, generating a high-quality restoration result.

[0074] Step 5: Construct a discriminator network model. Train and optimize the generator network model and discriminator network model using a loss function to obtain optimized generator network model and discriminator network model. Use the optimized generator network model and discriminator network model to repair the missing ancient text image.

[0075] To address the problems existing in the prior art, the present invention provides a method, system, medium, device, and terminal for ancient text image restoration. The present invention will be described in detail below with reference to the accompanying drawings.

[0076] like Figure 1 As shown, the ancient text image restoration method provided in this embodiment of the invention includes the following steps:

[0077] S101, Construct a dataset and an ancient text image restoration network model, and use the edge restoration network to restore the edge structure of the ancient text image and reconstruct the edge information of the missing area;

[0078] S102, Construct a local text learning network, and use the pre-trained local text learning network to repair the content of the local missing areas to obtain a locally repaired image;

[0079] S103 introduces a dual cross encoder and a multi-scale fusion block into the content inpainting network, and uses the low and high layers of the content inpainting network to progressively inpaint the image using the Laplacian pyramid inpainting network.

[0080] S104. The restored ancient text image and the corresponding undamaged ancient text image are used as inputs to the ancient text image restoration network model. The ancient text image restoration network model is trained and optimized, and the optimized image restoration network model is used to restore the damaged ancient text image.

[0081] As a preferred embodiment, such as Figure 2 As shown, the ancient text image restoration method provided in this embodiment of the invention specifically includes the following steps:

[0082] Step 1, Dataset Construction. An ancient text image dataset and a text image dataset were constructed for the restoration analysis of ancient text images. Considering the feasibility of ancient text image restoration, a Mask dataset adapted to the ancient text image dataset was also constructed to allow training to more closely resemble real-world image restoration, achieving the goal of restoring truly damaged ancient text images.

[0083] Step 2: Construct the edge repair network. First, use the edge repair network to repair the missing text structure information, generate a complete text edge map, and construct an edge guidance map.

[0084] Step 3: Construct a local text learning network. Randomly crop four different missing image patches from the missing region of the input image. Use the pre-trained text learning network to repair the missing text information in the image patches, and stitch the repaired results into the missing image to obtain a local content repair image.

[0085] Step 4: Construct the Laplacian pyramid restoration network. Using the Laplacian pyramid, the local content restoration map is decomposed into a high-level detail map and a low-level content map; for low-level content restoration, the downsampled edge guide map and low-level content map are... Figure 1 The input is fed into the content inpainting network to repair the missing areas and generate a low-resolution repair result image. High-level detail optimization upsamples the low-resolution repair result and aggregates it with the high-level detail image to obtain an intermediate repair result. The content inpainting network takes the intermediate repair result and the edge guide map as input to optimize the details of the missing areas and generate a high-quality repair result.

[0086] Step 5: Construct a discriminator network model.

[0087] Step 6: Train and optimize the generator network model and discriminator network model using the loss function to obtain the optimized generator network model and discriminator network model.

[0088] In the ancient text image restoration method based on edge guidance and Laplacian pyramid decomposition provided in this embodiment of the invention, the generator of the edge restoration network adopts an encoder-decoder structure. The global content network contains an encoder and a decoder. The encoder uses convolution operation, instance normalization and linear rectified activation function to downsample the input image and extract image features. The decoder takes the downsampled image feature results as the network input, performs upsampling operation through nearest neighbor interpolation and reconstructs the image to obtain the restored edge image X.

[0089] In the ancient text image restoration method based on edge guidance and Laplacian pyramid decomposition provided in this embodiment of the invention, the text learning network first constructs a text image dataset by collecting text from text images, where each image contains one character. During training, a random damaged region is added to each text image in the dataset to form a damaged text image, which is then input into the text learning network, with the corresponding complete text image serving as a constraint. After training, the pre-trained text learning model is applied to the restoration of the text images. Specifically, four 32-pixel-sized regions are cropped from the input image. 32 different blocks, each containing different missing regions, are input into a pre-trained text learning network to obtain the restoration results of the missing text in the image blocks; finally, the restoration results of the image blocks are stitched together to obtain a local content restoration image P.

[0090] In the ancient text image restoration method based on edge guidance and Laplacian pyramid decomposition provided by this invention, the Laplacian pyramid restoration network uses a content restoration network at two different scales to progressively restore the damaged image. Specifically, firstly, the local content restoration map is decomposed using Laplacian pyramid decomposition, breaking it down into a high-level detail map and a low-level content map; a content restoration network is then constructed, and the low-level content map is restored using the low-scale edge guidance map and content map as inputs; the restoration result is then upsampled and combined with the high-level detail map to obtain an intermediate restoration map, which, along with the edge guidance map, is used as input to the content restoration network to obtain the final image restoration result I.

[0091] In the ancient text image restoration method based on edge guidance and Laplacian pyramid decomposition provided in this embodiment of the invention, the discriminator network model is a PatchGAN network, which consists of 5 convolutional layers with a kernel size of 4×4 and a stride of 1. The output dimensions of the first 4 convolutional layers are 64, 128, 256 and 512, respectively, and the input mapping is the last convolutional layer. A 30×30 matrix is ​​used as the output, and the mean of the output matrix is ​​used as the output result of the discriminator network model.

[0092] In the ancient text image restoration method based on edge guidance and Laplacian pyramid decomposition provided in the embodiments of the present invention, the loss function includes an edge loss function, a text content loss function, and a content loss function.

[0093] The formula for the edge loss function is as follows:

[0094] (1)

[0095] In equation (1), and As the loss weight, and set to [value] during the experiment. =4, =1; It is the discriminator of the edge repair network; It is a generator for edge repair networks.

[0096] (2)

[0097] In equation (2), It is a complete edge map obtained by preprocessing a real text image. Edge map generated by the edge repair network.

[0098] (3)

[0099] In equation (3), Represents a text-defective image. Represents the edge of the defect; The mask image is represented by a binary matrix, where 1 represents a missing region and 0 represents an unmissing region.

[0100] In order for the text learning network to generate reasonable text content as much as possible, the text content loss includes text graph reconstruction loss and adversarial loss. The text content loss function is shown in formula (4):

[0101] (4)

[0102] in, and As the loss weight, and set to [value] during the experiment. =4, =1; It is the discriminator for text learning networks; It is a generator for text learning networks.

[0103] (5)

[0104] (6)

[0105] In equations (5) and (6), It has a resolution of 32 32 text images, Text images generated for a text learning network. This indicates a missing or damaged text image.

[0106] The Laplacian Pyramid Inpainting Network utilizes a content inpainting network to progressively inpaint images at both high and low levels. It constrains the network to generate images that more closely resemble reality by comparing the generated images pixel-by-pixel with the real images. (Inpainting loss) Including reconstruction losses and generating adversarial losses The content loss function is shown in formula (7) below:

[0107] (7)

[0108] In equation (7), and As the loss weight, and set to [value] during the experiment. =1, =0.1.

[0109] (8)

[0110] (9)

[0111] In equation (8), This is the result of low-resolution layer repair. This is the result of downsampling the real image; This is the high-resolution layer repair result, i.e., the final repair result generated by the network; This is a real image.

[0112] In equation (9), Represents a text-defective image. For edge guidance maps, Represented as a real text image; It is the discriminator of the edge repair network. It is a generator for edge repair networks.

[0113] In the ancient text image restoration method based on edge guidance and Laplacian pyramid decomposition provided in this embodiment of the invention, during the network training process, the backpropagation algorithm is used to update the parameters of the generator network model and the discriminator network model. After each iteration, the parameters of the discriminator network model are updated once, and the parameters of the generator network model are updated once. At the same time, the model test score is calculated, and the generator network model with the highest final score is taken as the optimized generator network model.

[0114] like Figure 3 As shown, the ancient text image restoration system provided in this embodiment of the invention includes:

[0115] The dataset building module is used to build ancient text image datasets, text image datasets, and Mask image datasets adapted to ancient text image datasets;

[0116] The edge restoration module is used to construct an edge restoration network, which is then used to restore the edge structure of ancient text images and reconstruct the edge information of the missing areas.

[0117] The local restoration module is used to construct a local text learning network, which uses a pre-trained text learning network to restore content in local missing areas, resulting in a locally restored image.

[0118] The progressive repair module is used to construct the Laplacian pyramid repair network, and then use the low and high layers of the content repair network to progressively repair the image.

[0119] The training and optimization module is used to build a discriminator network model. The ancient text image restoration network model is trained and optimized by taking the restored ancient text image and the corresponding undamaged ancient text image as input to the discriminator network. The optimized image restoration network model is then used to restore the damaged ancient text image.

[0120] As a preferred embodiment, the ancient text image restoration method based on edge guidance and Laplacian pyramid decomposition provided by this invention specifically includes the following steps:

[0121] Step 1: Construct the ancient text image dataset, the Mask dataset, and the text image dataset.

[0122] Step 1.1: Construct an ancient text image dataset;

[0123] Thirty well-preserved modern classical Chinese books were scanned, including: *Wen Ning Meng Qiu*, *The Analects*, *The Three Su's Writings*, *The Analects Explained*, *The New Laws of the Guangxu Emperor of the Qing Dynasty*, *Collected Annotations on Books*, *Mencius*, and *Republic of China Machinery*, resulting in 1305 high-resolution text images. These images were then cropped to 1000 pixels using Photoshop 2017. A 1000-resolution text image patch, and the cropped text image patch is processed to 256. The text images are 256 pixels in size. In the experiment, 10,639 images from the ancient text image dataset were used as the training set to train the network model, and 1,588 images were used as the test set to evaluate the trained model.

[0124] Step 1.2: Construct a text image dataset;

[0125] The ancient text image dataset contains too many characters in a single image, causing the model to ignore the semantic and structural information of the characters themselves. Therefore, to obtain the font style and semantic information of the ancient text image dataset, the dataset is cropped so that each image contains only one complete character. The dataset contains 432 images with a resolution of 32... A complete text image of 32 is input into the auxiliary model for training, which helps the overall network learn text-related information.

[0126] Step 1.3: Construct the Mask dataset;

[0127] Because different text font sizes vary, excessively large mask regions can cause a significant amount of text to disappear completely, preventing the network from acquiring sufficient and effective information, thus rendering the restored content meaningless. Therefore, existing mask datasets used for image restoration are not entirely suitable for ancient text image restoration research. This study employs the most widely used mask generation algorithm to obtain 12,000 mask images with different defect sizes. During training, these images are randomly selected and subjected to flipping, distortion, and stretching to ensure mask diversity and achieve effective restoration.

[0128] Step 2: First, extract the edge map of the damaged image, use the edge inpainting network to repair the missing text structure information, generate a complete text edge map, and construct an edge guide map.

[0129] Step 2.1: Image edge extraction;

[0130] First, the image of the missing ancient text is input into the convolutional network for binarizing historical document images, where it undergoes erosion processing and is subtracted from the binarized image. Finally, pixel inversion is performed to obtain the edge map.

[0131] Step 2.2: The edge repair network consists of an encoder and a decoder;

[0132] The edge inpainting network's generator employs an encoder-decoder structure. The encoder consists of six convolutional layers, while the decoder comprises three composite blocks and one convolutional layer. To allow higher-level convolutions to capture more detail, skip connections are used between the encoder and decoder to improve the accuracy of edge image reconstruction. Instance normalization and rectified linear activation functions are applied after each convolutional layer to ensure the training data is normalized and distributed within a certain range, reducing large fluctuations in data, thus accelerating training and improving the network's generalization ability. The encoder's ordinary convolutional kernel sizes are 7×7, 4×4, 5×5, 4×4, 5×5, and 4×4, with strides varying from 1 to 2, resulting in output dimensions of 64, 128, 256, 256, and 512, respectively. The decoder uses composite blocks for upsampling. These blocks perform upsampling through instance normalization, rectified linear activation functions, a convolution with a kernel of 3 and a stride of 1, and nearest-neighbor interpolation using the Upsampling function.

[0133] Step 3: Construct a text learning network.

[0134] Using a text image dataset, a random damaged region is added to each text image in the dataset, forming a damaged text image which is then input into the text learning network, with the corresponding complete text image serving as a constraint. After training, the pre-trained text learning model is applied to text image restoration. Specifically, four 32-bit regions are cropped from the input image. The network uses 32 different blocks, each containing a different missing region. These missing image blocks are input into a pre-trained text learning network to obtain the repair results of the missing text in the image blocks. Finally, the repair results of the image blocks are stitched together with the missing image to obtain a locally restored image, which is used for subsequent image restoration. This provides more text content information for the Laplacian inpainting network, guiding the network to generate semantically reasonable repair results. The text learning network also adopts an encoder-decoder structure. Its encoder structure consists of 6 convolutional layers, and its decoder structure consists of 3 composite blocks and 1 convolutional layer. The encoder's ordinary convolutional kernel sizes are 7×7, 4×4, 5×5, 4×4, 5×5, and 4×4, respectively, with strides varying from 1 to 2, 1 to 2, 1 to 2, and the output dimensions of each convolution are 64, 128, 256, 256, and 512, respectively. The decoder uses a composite block for upsampling. The composite block performs upsampling by instance normalization, linear rectified activation function, convolution with 3 kernels and stride of 1, and nearest neighbor interpolation using the Upsampling function.

[0135] Step 4: Construct the Laplace pyramid repair network.

[0136] First, the local content restoration image is decomposed using Laplacian pyramid decomposition, breaking it down into a high-level detail image and a low-level content image. A content restoration network is then constructed. Using the low-scale edge guidance image and content image as input, the low-level content image is restored. The restored result is then upsampled and combined with the high-level detail image to obtain an intermediate restored image. This intermediate restored image, along with the edge guidance image, is then used as input to the content restoration network to complete the restoration and optimization of the high-level image. Through this progressive restoration, the Laplacian pyramid restoration network allows the high-level image to obtain rich detail and global information, generating high-quality restoration results. The content restoration networks used for both the low-level and high-level images have the same network structure.

[0137] Content restoration networks are also an encoder-decoder structure. The encoder part is a dual-cross encoder, which uses two encoders to encode features of the damaged image and the edge map to be restored. Simultaneously, to better extract feature information from the image, the bottleneck layer uses eight multi-scale fusion blocks, utilizing different receptive fields to learn global and local features of the image and fuse these features. Then, the decoder decodes the obtained feature information to generate a complete image restoration result. The dual-cross encoder structure includes one edge encoder and one image encoder. The edge encoder takes the edge guide map as input and uses RN during downsampling to normalize the damaged and undamaged areas separately, avoiding the influence of invalid pixels. The content encoder takes the low-level local content restoration map (or intermediate restoration result) as input and uses gated convolution to dynamically learn the optimal mask information during downsampling, updates the mask, and performs a dot product between the mask and the obtained feature map to more accurately segment the damaged area, avoid the influence of invalid pixels, and extract more accurate features.

[0138] The edge encoder consists of four ordinary convolutions, each followed by an RN and ReLU activation function. The kernel sizes are 7×7, 4×4, 4×4, and 3×3, with strides varying from 1 to 1. The output dimensions of each convolution are 64, 256, 512, and 256, respectively. The content encoder consists of four gated convolutions, each followed by an IN and ReLU activation function. The kernel sizes are 5×5, 4×4, 3×3, and 4×4, with strides varying from 1 to 1. The output dimensions of each convolution are 64, 128, 128, and 256, respectively. The bottleneck layer uses eight multi-scale fusion blocks based on dilated convolutions, as proposed earlier. These blocks learn global and local features using dilated convolutions at different scales and then fuse these features to mitigate texture blurring during the restoration process and improve the clarity of the damaged areas.

[0139] The decoder consists of three regular convolutional layers and two upsampling modules. The first two regular convolutional layers are followed by an upsampling function, an RN activation function, and a ReLU activation function. All convolutional layers have a kernel size of 3×3 and a stride of 1, with output dimensions of 512, 256, and 3 respectively. Upsampling uses the PixelShuffle(2) function to double the width and height of the feature map, reducing the number of channels to one-quarter. Each regular convolutional layer is followed by an RN function and a ReLU activation function.

[0140] Step 5: Construct a discriminator network model.

[0141] In this embodiment of the invention, the discriminator network model is a PatchGAN network, which consists of 5 convolutional layers with a kernel size of 4×4 and a stride of 1. The output dimensions of the first 4 convolutional layers are 64, 128, 256 and 512, respectively. The last convolutional layer is input mapping, takes a 30×30 matrix as output, and uses the mean of the output matrix as the output result of the discriminator network model.

[0142] Step 6: Train and optimize the generator network model and discriminator network model using the loss function to obtain the optimized generator network model and discriminator network model.

[0143] The loss functions include the edge loss function, the text content loss function, and the content loss function. The formula for the edge loss function is as follows:

[0144] (1)

[0145] In equation (1), and As the loss weight, and set to [value] during the experiment. =4, =1; It is the discriminator of the edge repair network; It is a generator for edge repair networks.

[0146] (2)

[0147] In equation (2), It is a complete edge map obtained by preprocessing a real text image. Edge map generated by the edge repair network.

[0148] (3)

[0149] In equation (3), Represents a text-defective image. Represents the edge of the defect; The mask image is represented by a binary matrix, where 1 represents a missing region and 0 represents an unmissing region.

[0150] In order for the text learning network to generate reasonable text content as much as possible, the text content loss includes text graph reconstruction loss and adversarial loss. The text content loss function is shown in the following formula (4):

[0151] (4)

[0152] in, and As the loss weight, and set to [value] during the experiment. =4, =1; It is the discriminator of the text learning network. It is a generator for text learning networks.

[0153] (5)

[0154] (6)

[0155] In equations (5) and (6), It has a resolution of 32 32 text images, Text images generated for a text learning network. This indicates a missing or damaged text image.

[0156] The Laplacian Pyramid Inpainting Network utilizes a content inpainting network to progressively inpaint images at both high and low levels. It constrains the network to generate images that more closely resemble reality by comparing the generated images pixel-by-pixel with the real images. (Inpainting loss) Including reconstruction losses and generating adversarial losses The content loss function is shown in formula (7) below:

[0157] (7)

[0158] In equation (7), and As the loss weight, and set to [value] during the experiment. =1, =0.1.

[0159] (8)

[0160] In equation (8), This is the result of low-resolution layer repair; This is the result of downsampling the real image; This is the high-resolution layer repair result, i.e., the final repair result generated by the network; This is a real image.

[0161] (9)

[0162] In equation (9), Represents a text-defective image. For edge guidance maps, Represented as a real text image; It is the discriminator of the edge repair network; It is a generator for edge repair networks.

[0163] The optimized generator network model in this embodiment of the invention is used to process the incomplete ancient text image. Figures 4A-4B These are examples of applications in other scenarios; among them, Figure 4A This is the rubbing image before restoration. Figure 4B This is the restored rubbing image. The ancient text image restoration method based on edge guidance and Laplacian pyramid decomposition provided by this invention was used to process the image, resulting in a complete and reasonable rubbing image.

[0164] The optimized generator network model in this embodiment of the invention is used to process the incomplete ancient text image. Figures 5A-5B This is a specific case; among them, Figure 5A This is the image before restoration. Figure 5B This is the restored image. The ancient text image restoration method based on edge guidance and Laplacian pyramid decomposition provided by this invention is used for processing, resulting in a more complete and reasonable restoration result in terms of visual effect.

[0165] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0166] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for ancient text image inpainting, characterized in that, The application relates to an ancient text image restoration method and device. The ancient text image restoration method comprises the following steps: Step one, constructing an ancient text image dataset, a character image dataset and a Mask image dataset; Step two, constructing an edge restoration network to extract edges of a damaged ancient text image; the edge restoration network is used to restore damaged character structure information, generate a character edge image and construct an edge guide image; Step three, constructing a local character learning network, randomly cropping four different damaged image blocks on a damaged area of an input image; a pre-trained character learning network is used to restore damaged character information in the image blocks, and the restoration results are spliced into the damaged image to obtain a local content restoration image; Step four, constructing a Laplacian pyramid restoration network, the local content restoration image is decomposed into a high-layer detail image and a low-layer content image by the Laplacian pyramid; low-layer content restoration is performed by inputting a down-sampled edge guide image and the low-layer content image into a content restoration network to restore the damaged area and generate a low-resolution restoration result image; high-layer detail optimization is performed by up-sampling the low-resolution restoration result and aggregating the high-layer detail image to obtain an intermediate restoration result; the content restoration network takes the intermediate restoration result and the edge guide image as inputs to optimize the details of the damaged area and generate a high-quality restoration result; Step five, constructing a discriminator network model, training and optimizing the generator network model and the discriminator network model through a loss function to obtain an optimized generator network model and discriminator network model, and using the optimized generator network model and discriminator network model to restore a damaged ancient text image. The generator of the edge restoration network in step two adopts an encoding-decoding structure, a global content network comprises an encoder and a decoder, the encoder adopts convolution operation, instance normalization and a linear rectifier activation function to down-sample an input image and extract image features; 2. The ancient text image restoration method of claim 1, wherein, The decoder takes the down-sampled image feature result as network input, performs nearest neighbor interpolation to complete up-sampling operation and reconstructs an image to obtain a restored edge image X. The local character learning network in step three collects characters in a text image to construct a character image dataset, wherein each image contains one character; during training, a random damaged area is added to each character image in the dataset to form a damaged character image input into the character learning network, and a corresponding complete character image is used as a constraint.

3. The method of claim 1, wherein the step of restoring the ancient text image is performed by using a neural network. ​ After the training is completed, the pre-trained character learning model is applied to the repair of the text image; specifically including: cutting 4 different blocks of 32 32 on the input image, each block containing a different missing area, and inputting the missing image block into the pre-trained character learning network to obtain the repair result of the missing character in the image block; the repair result of the image block is spliced into the missing image to obtain a local content repair image P.

4. The ancient text image restoration method of claim 1, wherein, In the Laplacian pyramid restoration network in step four, the content restoration network is used at two different scales to progressively restore the damaged image, specifically including: performing Laplacian pyramid decomposition on the local content restoration image to decompose the local content restoration image into a high-level detail image and a low-level content image; constructing a content restoration network, taking the low-scale edge guide image and the content image as inputs of the content restoration network, and restoring the low-level content image; upsampling the restoration result and combining the high-level detail image to obtain an intermediate restoration image, and taking the edge guide image and the intermediate restoration image as inputs of the content restoration network to obtain a final image restoration result I.

5. The method of claim 1, wherein the ancient text image restoration method is characterized by, The discriminator network model in step five is a PatchGAN network, which is composed of five convolutional layers with a kernel size of 4x4 and a step size of 1. The output dimensions of the first four convolutional layers are 64, 128, 256, and 512, respectively. The last convolutional layer inputs a 30x30 matrix as output, and the mean value of the output matrix is taken as the output result of the discriminator network model. The loss function includes an edge loss function, a text content loss function, and a content loss function. The formula of the edge loss function is as follows: ； wherein and is a loss weight, set to = 4, = 1; is a discriminator of the edge inpainting network; is a generator of the edge inpainting network; ； In the formula, is a complete edge map obtained by preprocessing a real text image, is an edge map generated by the edge repairing network; ； wherein denotes a text missing map, denotes a missing edge map; is a binary matrix representing a mask map, 1 denotes a missing area, 0 denotes a non-missing area; The text content loss includes a text image reconstruction loss and an adversarial loss, and the text content loss function is: ； wherein and is a loss weight, set to = 4, = 1; is a discriminator of the text learning network; is a generator of the text learning network; ； ； In the formula, is a text image with a resolution of 32 32, is a text image generated by the text learning network, denotes a missing text image; The Laplacian pyramid repair network utilizes the content repair network to repair the image in a progressive manner through high layers and low layers, and compares the generated image with the real image pixel by pixel; the repair loss includes a reconstruction loss and a generative adversarial loss The content loss function is as follows: ； wherein and are loss weights, set to = 1, = 0.1; ； ； In the formula, is the result of low-resolution layer repair, is the result of real image down-sampling; is the high-resolution layer repair result, which is the final repair result generated by the network; is the real image; represents a text missing image, is an edge guide image, represents a text image that is real; is a discriminator of the edge repair network, is a generator of the edge repair network; During network training, the parameters of the generator network model and the parameters of the discriminator network model are updated using the backpropagation algorithm. After each iteration of parameter updating of the discriminator network model, the parameter updating of the generator network model is performed again. The final score of the model is calculated, and the generator network model with the highest final score is taken as the optimized generator network model.

6. An ancient document image restoration system applying the ancient document image restoration method according to any one of claims 1 to 5, characterized by, The ancient text image restoration system includes: A data set construction module for constructing an ancient text image data set, a text image data set, and a Mask image data set compatible with the ancient text image data set; An edge restoration module for constructing an edge restoration network to restore the edge structure of the ancient text image and reconstruct the edge information of the damaged area; A local restoration module for constructing a local text learning network to restore the content of the local damaged area using a pre-trained text learning network to obtain a local content restoration image; A progressive restoration module for constructing a Laplacian pyramid restoration network to progressively restore the image using the low-level and high-level content restoration networks; A training optimization module for constructing a discriminator network model, taking the restored ancient text image and the corresponding undamaged ancient text image as inputs of the discriminator network to train and optimize the ancient text image restoration network model, and using the optimized image restoration network model to restore the damaged ancient text image.

7. A computer device, characterized by The computer device includes a memory and a processor, and the memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of the ancient text image restoration method according to any one of claims 1-5.

8. A computer readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the ancient text image restoration method according to any one of claims 1-5.

9. An information data processing terminal, characterized by The information data processing terminal is used to implement the ancient text image restoration system according to claim 6.

Images