A method for correcting ancient book document image based on text recovery and reconstruction

Abstract

The application discloses a kind of based on text recovery reconstruction ancient document image correction method, it is related to computer technology field, comprising: based on probability diffusion model, residual block and time embedding technology constructs text recovery model;Loss function is constructed, and the training strategy of frequency domain decomposition is introduced, and text recovery model is trained with loss function to obtain the text recovery model after training;The ancient document image is input to above-mentioned model to obtain the ancient document image of preliminary recovery;Based on Enet network and Tnet network constructs document image correction model, Enet network is trained using supervised training and weak supervision training, Tnet network is trained using self-supervised training, to obtain the document image correction model after training;The ancient document image of preliminary recovery is input to the document image correction model after training, to obtain the ancient document image after correction;The application can clearly reconstruct the text of ancient document image, learn the text morphological characteristics of reconstruction, effectively improve the performance of correction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a kind of ancient book document image correction method based on text recovery reconstruction. BACKGROUND

[0002] Ancient books, as an important carrier of Chinese traditional culture and ancient civilization, carry thousands of years of historical memory and cultural heritage. However, in the long process of inheritance, ancient books often suffer from serious paper aging due to limited preservation conditions, humid environment, air pollution and frequent human reading, etc., leading to a series of text degradation problems. Typical manifestations include blurred text, faded strokes, ink smearing, and text pollution caused by dust and mold spots, which greatly affect the readability and integrity of the content of ancient books, limiting their digital preservation and research value. Traditional methods usually rely on manual or simple image enhancement techniques for content recognition and repair, but in the face of complex structure and severely degraded ancient book text images, such methods often fall short. With the rapid development of artificial intelligence and image visual technology, the automatic recovery and reconstruction of ancient book text has become possible. In particular, deep learning-driven text recovery and reconstruction methods can effectively identify and reconstruct blurred or incomplete text content due to degradation, improving the readability and information integrity of ancient book images, and providing strong technical support for the digital protection, research and utilization, and cultural dissemination of ancient books. From the perspective of artificial intelligence visual processing, the text recovery task of ancient book document images is challenging. On the one hand, there are various forms of degradation and often coexist; on the other hand, ancient book text has rich font styles and complex strokes, making it difficult for traditional image processing methods to accurately model the degradation process. In recent years, researchers have attempted to use end-to-end neural network models, combined with character-level structure modeling, semantic context recovery, and image-text joint enhancement strategies, to achieve fine reconstruction of ancient book text regions. Such methods not only significantly surpass traditional methods in terms of recovery accuracy, but also provide a new technical path for large-scale ancient book digitization projects, with important academic research value and cultural protection significance.

[0003] Real ancient book documents inevitably suffer from paper aging problems due to limited preservation conditions and humid environment, leading to text content blurring, stroke fading, and text pollution, etc. At the same time, ancient book document images have the characteristics of severe paper distortion and irregular text layout, making correction more difficult. Existing document image algorithms mostly use pure geometric correction strategies, such as simple CNN structure modeling of complex geometric deformation in ancient book images, focusing on the extraction of global deformation features, which cannot effectively solve the problems of document image aging and degradation, and cannot solve the recovery of local deformation, leading to increased difficulty in extracting ancient book documents, and the quality of extracted images cannot meet actual needs. SUMMARY

[0004] The application provides a method for correcting ancient document image based on text recovery and reconstruction.

[0005] In order to achieve the above-mentioned purpose, the technical scheme of the application is:

[0006] A method for correcting ancient document image based on text recovery and reconstruction, comprising:

[0007] S1: obtaining an ancient document image dataset to form a training set;

[0008] S2: constructing a first residual block and a second residual block, introducing two probability diffusion models, constructing a text recovery model based on the probability diffusion model, the first residual block, the second residual block and a time embedding technology, the text recovery model comprising a coarse-grained predictor and a denoiser, inputting the ancient document image in the training set into the text recovery model; constructing a loss function of the coarse-grained predictor and the denoiser, introducing a training strategy of frequency domain decomposition to construct the loss function, training the text recovery model together with the loss function of the coarse-grained predictor and the denoiser, and obtaining a trained text recovery model;

[0009] S3: inputting the ancient document image into the trained text recovery model for denoising to obtain a preliminary recovered ancient document image;

[0010] S4: introducing an Enet network and a Tnet network, constructing a document image correction model based on the Enet network and the Tnet network, inputting the preliminary recovered ancient document image into the document image correction model, training the Enet network using supervised training and weakly supervised training, and training the Tnet network using self-supervised training, to obtain a trained document image correction model;

[0011] S5: inputting the preliminary recovered ancient document image into the trained document image correction model, learning the geometric structure features of the ancient document image through the Enet network, learning the local text deformation features of the ancient document image through the Tnet network, and obtaining a corrected ancient document image.

[0012] Further, all convolutional layers in the first probability diffusion model are replaced by the first residual block to obtain the coarse-grained predictor.

[0013] All convolutional layers in the second probability diffusion model are replaced by the second residual block, and a time embedding mechanism is introduced in each second residual block to obtain the denoiser.

[0014] Further, the first residual block and the second residual block have the same structure, each residual block comprising an input layer, a first activation function layer, a first convolutional layer, a second activation function layer, a regularization layer, a second convolutional layer and an output layer connected in sequence, and the input layer and the output layer are connected in a skip connection manner.

[0015] Further, a timing embedding mechanism is introduced in each second residual block, including:

[0016] Encode the high-dimensional vector of the time step set in the denoiser denoising process through a double-layer perception machine to generate modulation parameters, and use the modulation parameters to scale and offset the channels of the feature map after the first convolution layer processing and normalization in each second residual block.

[0017] Further, the loss functions of the coarse-grained predictor and the denoiser are constructed, including:

[0018] The loss function of the coarse-grained predictor is constructed, as shown in formula (1),

[0019] (1)

[0020] Wherein, is the loss function of the coarse-grained predictor, is the mathematical expectation, is the prediction result of the coarse-grained predictor, is the coarse-grained predictor, is the original image, is the original real clear image;

[0021] The loss function of the denoiser is constructed, as shown in formula (2),

[0022] (2)

[0023] Wherein, is the loss function of the denoiser, is the original un-noised clear image data, i.e. the original real clear image, is the denoiser, is the pre-defined signal-to-noise ratio coefficient, is the residual path, is the noise, is the time step, is the memory copy of

[0024] Further, a frequency domain decomposition training strategy is introduced to construct the loss function, which is trained together with the loss functions of the coarse-grained predictor and the denoiser to train the text restoration model, including:

[0025] The frequency domain decomposition training strategy is introduced, i.e. a low-frequency filter and a high-frequency filter are designed to separate the original real clear image into low-frequency and high-frequency information, as shown in formula (3),

[0026] (3)

[0027] where the low-frequency information is used to reconstruct the overall structure of the image, and the high-frequency information is used to reconstruct the features of the text strokes, respectively represent the low-pass and high-pass linear filter kernels, and the symbol represents the convolution operation, and by designing a complementary filter pair, the following is satisfied is the Dirac function, and separates the input image into low-frequency basic features and high-frequency detail features;

[0028] The loss function of the low-frequency basic features is constructed as shown in equation (4),

[0029] (4)

[0030] where denotes the mathematical expectation;

[0031] The loss function of the high-frequency detail features is constructed as shown in equation (5),

[0032] (5)

[0033] where is the noise state;

[0034] The total loss function is constructed as shown in equation (6),

[0035] (6)

[0036] where is the frequency weight, is the task weight; is the loss function of the coarse-grained predictor, is the loss function of the denoiser.

[0037] Further, the Enet network is trained using supervised training and weakly supervised training, including:

[0038] The preliminarily restored ancient document image is distorted by a set of real deformation fields to form synthetic data input to the Enet network for training. For the synthetic data input to the Enet network, a loss function for supervised learning is constructed to train the Enet network as shown in equation (7),

[0039] (7)

[0040] where is a function of extracting the deformation field boundary, is the real deformation field, is the synthetic deformation document image for training and its corresponding real deformation field​​ the deformation field generated by the Enet network;

[0041] For the real data input to the Enet network, that is, the preliminary recovered ancient document image, a loss function of weakly supervised learning is constructed to train the Enet network, as shown in formula (8),

[0042] (8)

[0043] wherein, is a deformation operation, used for deforming the mask according to the deformation field, is the inverse deformation field of a binary graph, containing the edge or contour information of the document, representing the document geometry of the deformed document, and the rectangular mask is the target shape of the image after completion of the deformation correction.

[0044] Further, the Tnet network is trained by using self-supervised training, including:

[0045] a self-supervised loss function is constructed, as shown in formula (9),

[0046] (9)

[0047] wherein, is a deformation operation, used for deforming the image according to the deformation field, and is the deformation field generated by the Tnet network, is a deformation disturbance, is the inverse deformation field of

[0048] Beneficial effects: the ancient document image correction method based on text recovery reconstruction is provided, a text recovery model is constructed based on a probability diffusion model, a coarse-grained predictor in the model focuses on global degradation, roughly processes the degradation of the whole text of the ancient book, recovers low-frequency information of the text, and then a denoiser finely processes complex text degradation and recovers high-frequency information such as text details, and the text recovery network can clearly reconstruct the text of the ancient book image.

[0049] The document image correction model is constructed based on the Enet network and the Tnet network, and the twin networks of the document image correction model correct the overall deformation of the text image and the text line distortion respectively. The design of the feedforward framework makes the correction network act on the ancient text image based on clearer and more complete ancient text, learns the text deformation features recovered by the text recovery, and effectively improves the performance of the correction. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0051] Figure 1 A flowchart of a text recovery and reconstruction-based ancient book document image correction method provided by the present application is shown in FIG. 1.

[0052] Figure 2 A whole framework diagram is shown in FIG. 2.

[0053] Figure 3 A residual block structure diagram is shown in FIG. 3.

[0054] Figure 4 A coarse-grained predictor structure diagram is shown in FIG. 4.

[0055] Figure 5 A coarse-grained predictor loss calculation process diagram is shown in FIG. 5.

[0056] Figure 6 A denoiser structure diagram is shown in FIG. 6.

[0057] Figure 7 A denoiser loss calculation process diagram is shown in FIG. 7.

[0058] Figure 8 A text recovery model whole loss function structure diagram is shown in FIG. 8.

[0059] Figure 9 An Enet network structure diagram is shown in FIG. 9.

[0060] Figure 10 An Enet network supervised training loss process diagram is shown in FIG. 10.

[0061] Figure 11 An Enet network weakly supervised training loss process diagram is shown in FIG. 11.

[0062] Figure 12 A Tnet network structure diagram is shown in FIG. 12.

[0063] Figure 13 A Tnet network self-supervised training loss process diagram is shown in FIG. 13.

[0064] Figure 14 A whole image recovery flowchart is shown in FIG. 14. DETAILED DESCRIPTION

[0065] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0066] The present embodiment provides a method for correcting ancient document image based on text recovery and reconstruction, as shown in the formula (I), comprising: Figure 1

[0067] S1: Obtain an ancient document image dataset to form a training set;

[0068] S2: Construct a first residual block and a second residual block, introduce two probability diffusion models, construct a text recovery model based on the probability diffusion model, the first residual block, the second residual block and a time embedding technology, the text recovery model comprising a coarse-grained predictor and a denoiser, input the ancient document image in the training set to the text recovery model; construct a loss function of the coarse-grained predictor and the denoiser, introduce a training strategy of frequency domain decomposition to construct the loss function, train the text recovery model together with the loss function of the coarse-grained predictor and the denoiser, and obtain a trained text recovery model;

[0069] S3: Input the ancient document image to the trained text recovery model for denoising to obtain a preliminary recovered ancient document image;

[0070] S4: Introduce an Enet network and a Tnet network, construct a document image correction model based on the Enet network and the Tnet network, input the preliminary recovered ancient document image to the document image correction model, train the Enet network by using supervised training and weakly supervised training, train the Tnet network by using self-supervised training, and obtain a trained document image correction model;

[0071] S5: Input the preliminary recovered ancient document image to the trained document image correction model, learn the geometric structure features of the ancient document image through the Enet network, learn the local text deformation features of the ancient document image through the Tnet network, and obtain a corrected ancient document image.

[0072] Specifically, as shown in the formula (II), the method comprises the following steps: Figure 2 ​As shown, the present application consists of two models, namely a text restoration model and a document image correction model. The text restoration model provides high-quality input for the document image correction model. The document image correction model corrects the output of the text restoration model, and finally obtains a clear text and non-deformed text line ancient document image. The text restoration model and the document image correction model are organically integrated. The text restoration model reconstructs the text of the ancient document image, and the residual generation reconstruction obtains clear enhanced ancient text. The subsequent correction task provides a clearer and higher quality text line deformation feature space, thereby improving the correction effect of complex ancient document images.

[0073] The text restoration model is based on a probability diffusion model and adopts a U-Net network structure. Through the synergistic action of a coarse-grained predictor and a denoiser, the text details are gradually restored. The coarse-grained predictor introduces a residual block to replace a convolutional layer and is responsible for extracting multi-scale features. The denoiser combines a time embedding technology to gradually optimize the image quality in the diffusion process.

[0074] The document image correction model adopts a double-branch encoder-decoder structure. It includes an Enet network that learns the geometric structure features of the ancient document image, a global correction that corrects the geometric deformation of the ancient document image, a Tnet network that learns the local text deformation features of the ancient document image, and a local correction that corrects the text distortion of the ancient document.

[0075] In specific embodiments, the scheme for obtaining an ancient document image dataset to form a training set is:

[0076] An actual clear ancient document image is obtained. The ancient document image is subjected to noise addition and text distortion to obtain a processed ancient image dataset for training.

[0077] In specific embodiments, the first residual block and the second residual block are constructed, two probability diffusion models are introduced, and a text restoration model is constructed based on the probability diffusion model, the first residual block, the second residual block, and the time embedding technology. The text restoration model includes a coarse-grained predictor and a denoiser. The ancient document image in the training set is input into the text restoration model. The loss function of the coarse-grained predictor and the denoiser is constructed. The training strategy of frequency domain decomposition is introduced to construct the loss function. The text restoration model is trained together with the loss function of the coarse-grained predictor and the denoiser to obtain the trained text restoration model.

[0078] As shown in Figure 3 The first residual block and the second residual block are constructed. The first residual block and the second residual block have the same structure. Each residual block includes an input layer, a first activation function layer, a first convolutional layer, a second activation function layer, a regularization layer, a second convolutional layer, and an output layer connected in sequence. The input layer and the output layer are connected in a skip connection.

[0079] As Figure 4 shown in FIG. 1, all convolutional layers in the first probability diffusion model are replaced with the first residual block to obtain a coarse-grained predictor;

[0080] Specifically, the coarse-grained predictor is a model of U-Net structure, which introduces a residual block to replace the convolution module in the original network for downsampling and upsampling, aiming to extract better features to generate a preliminary denoising result in the degraded document image. Its main task is to restore the low-frequency information of the image, such as the text content and text structure of the ancient book, but it may not be able to well restore the high-frequency information of the ancient book, such as the stroke edge and character boundary.

[0081] The encoder part of the coarse-grained predictor contains an initial convolutional layer and multiple residual modules. The initial convolutional layer uses a 3x3 convolutional kernel with a step size and padding of 1, followed by a Swish activation function, which preliminarily extracts features from the input ancient document image and maps them to a higher-dimensional feature space. The residual modules gradually expand the number of channels from 32 to 64 and 128. Each residual module has two convolutional layers, which are connected by a Swish activation function and a Dropout layer for nonlinear transformation and regularization processing, ensuring gradual deep feature extraction. The residual module also contains a skip connection that directly adds the input and output, alleviating the gradient vanishing problem and stabilizing the model training. This design of the encoder gradually extracts higher-level semantic information, reducing the complexity of the calculation, and the encoder finally outputs a high-dimensional feature map containing the global structure and context information of the ancient book, providing a compact global feature representation for the decoder.

[0082] The decoder part is composed of multiple modules that gradually upsample, with the number of channels gradually decreasing from 128 to 64 and 32. Each upsampling module expands the resolution of the feature map by one through transposed convolution, and then further refines the features through the residual block. In order to preserve more detailed information, the skip connection mechanism is introduced into the decoder, which splices the feature maps of the corresponding layers of the encoder and the decoder, supplementing high-resolution information to the feature information, effectively alleviating the gradient vanishing problem and stabilizing the model training process. Each stage of the decoder gradually restores the resolution and reduces the number of channels, and refers to the size of the input ancient document image to continuously restore the feature map to the size of the image. Finally, a 3x3 convolutional layer is constructed in the coarse-grained predictor to map the feature map output by the decoder to the same size as the input ancient document image, generating a preliminary denoised ancient book. This convolutional layer finally adjusts and integrates the feature map, ensuring that the output ancient document image is consistent with the target result in terms of global structure and details.

[0083] The main task of the coarse-grained predictor is to preliminarily restore a clean image from a degraded ancient document image, mainly focusing on pixel-level restoration. The blurred text ancient document image is input into the coarse-grained predictor to obtain a preliminary clean text image, and the loss is calculated with the real clear text image. To measure the difference between the prediction result of the coarse-grained predictor and the real image, the loss function used is the pixel-level mean square error, and the loss is as shown in Figure 5 The loss function of the coarse-grained predictor is as shown in formula (10),

[0084] (10)

[0085] wherein, is the loss function of the coarse-grained predictor, is the mathematical expectation, is the prediction result of the coarse-grained predictor, is the coarse-grained predictor, is the original image, is the original real clear image; by minimizing the pixel-level mean square error, the prediction result is as close to the real image as possible, and since the mean square error loss function mainly focuses on the difference at the pixel level, the coarse-grained predictor can effectively restore the low-frequency information of the image;

[0086] As shown in Figure 6 , all convolutional layers in the second probability diffusion model are replaced with the second residual block, and a temporal embedding mechanism is introduced in each second residual block to obtain a denoiser.

[0087] Wherein, as shown in Figure 3 , the temporal embedding mechanism is a high-dimensional vector of time steps set in the denoising process of the denoiser, which is encoded by a double-layer perception machine to generate a modulation parameter. The modulation parameter is used to scale and offset the channels of the feature map after the first convolutional layer processing and normalization in each second residual block.

[0088] In particular, the denoiser is implemented based on a diffusion model, aiming to recover the high-frequency details of the image through a step-by-step denoising process. The denoiser further refines the denoising effect based on the initial prediction results. It also adopts a U-Net architecture similar to the encoder structure of the coarse-grained predictor; the encoder structure consists of an initial convolutional layer and multiple residual modules. The initial convolutional layer uses a 3x3 convolutional kernel with a stride and padding of 1, followed by a Swish activation function. The residual modules of the encoder gradually expand the number of channels from the initial 32 to 64 and 128. Each residual module contains two convolutional layers, with a Swish activation function and a Dropout layer in between for non-linear transformation and regularization. When the number of channels increases, the encoder reduces the size of the feature map by half through downsampling operations, gradually extracting higher-level semantic information. The encoder finally outputs a high-dimensional feature map containing the global structure and context information of the ancient book, providing a compact global feature representation for the decoder;

[0089] The decoder part gradually restores the resolution of the feature map through multiple upsampling modules. Each upsampling module introduces a transpose convolution to expand the size of the feature map by a factor of two, and enters a residual block to further refine the features. The decoder introduces a skip connection mechanism to retain more detailed information, concatenating the feature maps of the corresponding layers of the encoder and the decoder to supplement high-frequency information while effectively alleviating the gradient vanishing problem and stabilizing the training process of the model. The number of channels in the decoder decreases from 128 to 64 and 32 layer by layer, and a 3x3 convolutional layer is constructed before output to map the feature map output by the decoder to the same size as the input ancient book document image, generating a high-quality denoising result. Unlike the coarse-grained predictor, the denoiser embeds time in the residual block to introduce time step information during the diffusion process, capturing the temporal dynamic characteristics of the degraded text in the ancient book document image. The time step information is encoded as a high-dimensional vector, which is fused with the feature map to enable the model to better understand the distribution of noise at different time steps. The fusion of the time step embedding refers to the fusion of the high-dimensional vector obtained by encoding a specific time step in the diffusion model denoising process with the feature map of the decoder network layer. This fusion process is usually implemented through "feature modulation": the high-dimensional vector of the time step will pass through a small neural network to generate a set of modulation parameters (such as scaling factors and offsets), which are directly applied to the feature map to scale and offset each channel of the feature map, dynamically injecting time information into the calculation of each layer to guide the model to take different actions at different denoising stages;

[0090] The denoiser network can choose to predict the original data or the added noise, and the two prediction methods can be converted into each other by mathematical transformation in the unconditional generation task, and they are equivalent. However, when the conditional generation mechanism is introduced, especially when the channel splicing conditional injection method is adopted, the existing equivalence relationship is broken. The existing method usually adopts the strategy of predicting the added noise to maintain the diversity of the generated samples, but this approach limits the utilization of conditional information by the model, so that the denoiser can only learn from the noisy data. In contrast, the scheme of directly predicting the original data, although inferior to predicting noise in terms of maintaining the diversity of generated samples, on the contrary, fully utilizes the conditional information and fully uses it for the supervised learning of the model, which can improve the generation quality of the key initial step in the reverse process. The text restoration task of ancient document image has a high requirement for the generation quality of the model. In this embodiment, the training strategy of predicting the original data is adopted to minimize the distance between the model prediction distribution and the real posterior distribution to achieve the optimization goal. The loss function of the denoiser is shown in formula (11),

[0091] (11)

[0092] wherein, is the loss function of the denoiser, is the original clear image data without noise, i.e., the original real clear image, is the denoiser, is the pre-defined signal-to-noise ratio coefficient, is the residual path, is the noise, is the time step, is the memory copy of , which is stored separately, is the memory copy of , which is stored separately, The gradient of the loss function flows only through the residual path and is finally transmitted to the coarse-grained predictor ; the gradient propagation path is controlled, and the conditional information is effectively integrated to more fully utilize the conditional information to guide the generation process, meet the high-fidelity requirement of the ancient document image text restoration task, and ensure better reconstruction performance. After the coarse-grained predictor and the denoiser complete joint training, a completely deterministic reverse denoising process can be obtained by analyzing and deducing the formula, as shown in formula (12),

[0093] (12)

[0094] The original data is gradually recovered by an iterative denoising process in reverse, specifically, given the current noise state and time step , the reverse process computes the state of the previous step by the following deterministic transformation :

[0095] (13)

[0096] where, denotes the residual result predicted by the denoiser network, and are predefined signal-to-noise ratio coefficients in the diffusion process, and the denoiser loss is shown as Figure 7 ;

[0097] To optimize the generated effect, a training strategy of frequency domain decomposition is introduced to train the model, that is, a specific linear filter is designed to separate the image residual into low-frequency and high-frequency information. The low-frequency information is mainly responsible for reconstructing the overall structure of the ancient document image, while the high-frequency information focuses on reconstructing the relatively detailed features such as text strokes. This frequency separation is shown in equation (14),

[0098] (14)

[0099] where, and represent the low-pass and high-pass linear filter kernels, respectively, and the symbol represents convolution operation, by designing a complementary filter pair, satisfying , is the Dirac function, which separates the input image into low-frequency basic features and high-frequency detailed features. This decomposition strictly satisfies the frequency energy conservation, the low-frequency path uses the Gaussian kernel to capture the overall structure of the document, and the high-frequency path uses the Laplace operator to extract the edge details of the strokes. The two features are input into different sub-networks for targeted processing, aiming to improve the denoiser's ability to recover high-frequency information lost in the coarse-grained predictor, while the coarse-grained predictor ensures the minimum task burden of the denoiser, and both have different advantages. The low-frequency basic features of the document image are mainly responsible for reconstructing the overall structure of the document image, including the macro-structure information such as page layout and text area distribution, so the loss function of the low-frequency basic features constructed is shown in equation (15),

[0100] (15)

[0101] where, denotes the mathematical expectation;

[0102] focuses on recovering high-frequency detailed features such as sharp edges of text strokes and local texture details, and by introducing the training strategy of frequency domain decomposition, In each step of iteration, not only the noise condition is relied on , but also the structural prior from the coarse-grained predictor is fused, so as to focus more on the repair of high-frequency residual signals and the loss function of high-frequency detail features, as shown in equation (16),

[0103] (16)

[0104] wherein, is the noise state;

[0105] As shown in equation (18), considering that Figure 8 and can complement each other in model training, an end-to-end joint optimization training strategy is adopted to strengthen the cooperation between the two sub-networks. The text restoration model shares the loss weight gradient, so that the structure generated can perceive the pressure of feedback on the restoration of details in real time, and adjust the low-frequency information structure of its output according to the feedback, to generate a structure that better meets the needs of the high-frequency denoising stage. A unified loss function of the text restoration model is defined, as shown in equation (17),

[0106] (17)

[0107] wherein, is the frequency weight, which determines the relative importance of high-frequency and low-frequency loss, is the task weight, which adjusts the proportion of coarse-grained and fine-grained optimization; is the loss function of the coarse-grained predictor, is the loss function of the denoiser.

[0108] In the present scheme, the coarse-grained predictor can gradually extract and restore the feature information of the degraded text of the ancient book, clearly reconstruct the degraded text in the ancient book document image, and effectively complete the preliminary task of text restoration. The denoiser introduces a time embedding mechanism, which improves the modeling ability of the denoiser for complex degradation patterns of ancient book text, so that it performs better in denoising non-uniform noise or dynamic noise. The loss function constructed through the deterministic backpropagation mechanism guided by the condition improves the visual quality of the restoration of the ancient book document image on the basis of ensuring the integrity of the structure of the ancient book document.

[0109] In specific embodiments, the scheme of inputting the ancient book document image into the trained text restoration model for denoising to obtain the preliminarily restored ancient book document image is as follows:

[0110] ​​The ancient document image is input into the trained text recovery model for denoising, is processed by the coarse-grained predictor, and a preliminary clean document is obtained. The preliminary clean document output by the coarse-grained predictor is finally generated into a clean document by the denoiser.

[0111] In specific embodiments, an Enet network and a Tnet network are introduced, a document image correction model is constructed based on the Enet network and the Tnet network, the preliminary recovered ancient document image is input into the document image correction model, the Enet network is trained by supervised training and weakly supervised training, and the Tnet network is trained by self-supervised training. The scheme for obtaining the trained document image correction model is:

[0112] The document image correction model is constructed based on the Enet network and the Tnet network, wherein the Enet network is the first subnetwork in the document image correction model, and the architecture thereof adopts a full convolutional encoder-decoder structure. The network adds coordinate information to the input distorted ancient document image, performs downsampling and upsampling through the encoder and the decoder respectively, and the whole network aims to generate a global deformation field through edge information to correct the global geometric deformation of the document image. The Enet network structure is as shown in Figure 9 The encoder adopts a full convolutional residual structure, and contains an initial convolutional layer and 5 residual modules. Each residual module gradually extracts the features of the input image and performs 2 times downsampling. The decoder is composed of 4 modules which gradually perform 2 times upsampling, and is used for gradually restoring the deep features output by the encoder into a higher size feature map to generate a global deformation field of the ancient document image.

[0113] The training of the Enet network is divided into two modes: supervised training and weakly supervised training.

[0114] On the synthetic data, the training mode of the Enet network is supervised learning. The synthetic data refers to the artificially generated distorted document image and its corresponding real deformation field. The supervised learning is to give a synthetic distorted document image and its corresponding real deformation field, and then generate a deformation field through the Enet network, so that the boundary elements thereof match the real deformation field. The loss function is as shown in formula (18),

[0115] (18)

[0116] wherein, is a function for extracting the boundary of the deformation field, and the extracted boundary element represents the boundary coordinates of the document image in the input image, is the real deformation field, is the synthetic distorted document image and its corresponding real deformation field for training the deformation field generated by the Enet network; after matching the boundary elements, the internal elements of the deformation field are derived from the boundary elements by linear interpolation, the deformed document image is corrected to a regular rectangular shape, and the Enet network supervised training loss is as shown in formula (18); Figure 10

[0117] On real data, the training mode of the Enet network is weakly supervised learning. Real data is actually photographed or scanned in real life. The deformation document image does not have a corresponding real deformation field like synthetic data . Therefore, the introduction of weakly supervised training enables these real data to guide the learning of the Enet network through some easily accessible auxiliary information. Given a real-world deformed document image and its segmentation mask, a deformation field is generated by the Enet network. In order to ensure that the generated deformation field has a reasonable geometric deformation structure and avoid learning only simple scaling transformation, a cycle consistency design is introduced. Specifically, the cycle consistency requires that the deformation field generated by the Enet network not only maps the segmentation mask of the input image to the target rectangular mask, but also maps the rectangular mask back to the original segmentation mask through the inverse deformation field. This bidirectional mapping constraint ensures the geometric reasonableness of the deformation field, and the loss function is as shown in formula (19),

[0118] (19)

[0119] wherein, is a deformation operation for deforming the mask according to the deformation field, is the inverse deformation field of , the segmentation mask is a binary image containing the edge or contour information of the document, representing the document geometric shape of the deformed document, and the rectangular mask is the target shape of the image after completing the deformation correction; through this cycle consistency design, the Enet network can generate an accurate deformation field that conforms to the distortion of the document itself, thereby effectively correcting the geometric distortion of the ancient document image and ensuring the rationality of the design of this correction method. The weakly supervised training loss of the Enet network is as shown in formula (18); Figure 11

[0120] The Tnet network is the second sub-network in the document image correction model. Its structure is similar to that of the Enet network, also adopting a full convolutional encoder-decoder structure, but their weights are different. The input of the Tnet network is the ancient document image corrected globally by the Enet network. After the geometric deformation correction of the document image, a local deformation field is further generated to correct the local text distortion of the ancient document image. The structure of the Tnet network is as shown in formula (20); Figure 12

[0121] ​​​The Tnet network pays more attention to the deformation of the local text of the ancient book, generates a deformation field of the text area, and combines the global deformation field output by the Enet network, so as to realize more fine deformation adjustment. The Tnet network generates a deformation image pair by applying a known deformation disturbance to the input image, and completes self-supervised training;

[0122] The training mode of the Tnet network is self-supervised learning. The specific representation of self-supervised learning in the network is to generate a deformation image by applying a known deformation disturbance to the input image, and train the network by using the relationship between the deformation image and the input image. A known deformation disturbance is applied to the input image to generate a disturbed image. The deformation disturbance applied is to move a point in a randomly selected document area on the document image in a random direction. The training target of the Tnet network is to minimize the self-supervised loss function as shown in formula (20),

[0123] (20)

[0124] wherein, is a deformation operation for deforming an image according to a deformation field, and is a deformation field generated by the Tnet network, is a deformation field applied to the original input image, is a deformation field applied to the deformation image, is a deformation disturbance, is the inverse deformation field of

[0125] The design of the cycle consistency also enables the Tnet network to generate more reasonable and accurate deformation fields, thereby effectively correcting the local texture distortion of the image, and the self-supervised training loss is as shown in formula (21). Figure 13

[0126] In the scheme, the document image correction model is composed of the Enet network and the Tnet network, and the clear reconstructed text of the ancient book document image is globally corrected and locally corrected, respectively, so that the accuracy of the ancient book document image with text blur, stroke breakage and text line deformation in document deformation correction can be effectively improved.

[0127] In specific embodiments, the preliminarily restored ancient book document image is input into the trained document image correction model, the geometric structure features of the ancient book document image are learned by the Enet network, the local text deformation features of the ancient book document image are learned by the Tnet network, and the scheme for obtaining the corrected ancient book document image is as follows:

[0128] The overall structure is as shown in Figure 14 ​As shown, the image-text restoration and reconstruction module, i.e., the text restoration model, clearly reconstructs the text information of ancient documents, eliminating text degradation in the image of the ancient document. The output of the text restoration and reconstruction module is used as the input of the image correction module, i.e., the document image correction model, to learn the deformation features of the reconstructed clear text and correct the complex deformations of the ancient document image. The text correction model performs a global-to-local correction on the clear text image. Throughout the process, the correction model learns more local clear text information from the clear and complete text, enabling it to more accurately identify deformations in the ancient document image, improving the correction effect, and ultimately obtaining a complete image of the ancient document with clear and intact text content, without distortion. This provides another powerful technical support for the digital preservation of ancient document images.

[0129] In summary, this method effectively restores and reconstructs the text blurring and stroke fading issues in degraded ancient texts, clearly reconstructing the text deformation features of the ancient documents. The document image correction model can easily learn these text deformation features and can even uncover more refined deformation features. The text restoration model clearly reconstructs the text in the ancient text image, while the document image correction model's dual sub-networks correct the overall text image deformation and text line distortion, respectively. This feedforward framework design allows the correction network to operate on a clearer and more complete ancient text image, learning the text deformation features reconstructed from the text restoration model. This effectively improves the correction performance while solving the problems of text degradation and deformation in ancient texts, resulting in a high-quality output.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for correcting an ancient document image based on text recovery and reconstruction, characterized in that, The method comprises the following steps: S1: obtaining a set of ancient document image data to form a training set; S2: constructing a first residual block and a second residual block, introducing two probability diffusion models, constructing a text restoration model based on the probability diffusion model, the first residual block, the second residual block and a time embedding technology, the text restoration model comprising a coarse-grained predictor and a denoiser, inputting the ancient document image in the training set into the text restoration model, constructing a loss function of the coarse-grained predictor and the denoiser, introducing a training strategy of frequency domain decomposition to construct the loss function, training the text restoration model together with the loss function of the coarse-grained predictor and the denoiser, and obtaining a trained text restoration model; wherein the first residual block is used to replace all convolutional layers in the first probability diffusion model to obtain the coarse-grained predictor; the second residual block is used to replace all convolutional layers in the second probability diffusion model, and a time embedding mechanism is introduced in each second residual block to obtain the denoiser; the time embedding mechanism introduced in each second residual block comprises: encoding a high-dimensional vector of a time step set in the denoising process of the denoiser through a double-layer perception machine to generate a modulation parameter, and using the modulation parameter to scale and offset the channel of a feature map processed by a first convolutional layer and normalized in each second residual block; a training strategy of frequency domain decomposition is introduced, that is, a low-frequency filter and a high-frequency filter are designed to separate the original real clear image into low-frequency and high-frequency information, as shown in formula (3), （3） where low frequency information is used to reconstruct the overall structure of the image, and high frequency information is used to reconstruct the features of the text strokes, and denote low-pass and high-pass linear filter kernels, respectively, and the symbol represents convolution operation, by designing a pair of complementary filters satisfying , is the Dirac function, separates the input image into low frequency base features and high frequency detail features; a loss function of low-frequency basic features is constructed, as shown in formula (4), （4） wherein denotes the mathematical expectation; a loss function of high-frequency detail features is constructed, as shown in formula (5), （5） wherein is a noise state; a total loss function is constructed, as shown in formula (6), （6） wherein, is a frequency weight, is a task weight; is a loss function for the coarse-grained predictor, is a loss function for the denoiser; S3: inputting the ancient document image into the trained text restoration model for denoising to obtain a preliminarily restored ancient document image; S4: introducing an Enet network and a Tnet network, constructing a document image correction model based on the Enet network and the Tnet network, inputting the preliminarily restored ancient document image into the document image correction model, training the Enet network by using supervised training and weakly supervised training, and training the Tnet network by using self-supervised training to obtain a trained document image correction model; S5: inputting the preliminarily restored ancient document image into the trained document image correction model, learning the geometric structure features of the ancient document image through the Enet network, learning the local text deformation features of the ancient document image through the Tnet network, and obtaining a corrected ancient document image.

2. The method according to claim 1, wherein, The first residual block and the second residual block have the same structure, each residual block comprises an input layer, a first activation function layer, a first convolutional layer, a second activation function layer, a regularization layer, a second convolutional layer and an output layer connected in sequence, and the input layer and the output layer are connected in a skip connection manner. 3.The ancient book document image correction method based on text recovery and reconstruction of claim 1, wherein, The loss function of the coarse-grained predictor and the denoiser comprises: the loss function of the coarse-grained predictor is constructed, as shown in formula (1), （1） wherein, is a loss function for the coarse-grained predictor, is a mathematical expectation, is a prediction result of the coarse-grained predictor, is a coarse-grained predictor, is an original image, is an original real sharp image; the loss function of the denoiser is constructed, as shown in formula (2), （2） wherein, is a loss function of the denoiser, is the original unnoisy clean image data, i.e. the original real clean image, is the denoiser, is a predefined signal-to-noise ratio coefficient, is a residual path, is a noise, is a time step, is a memory copy of 4. The method of claim 1, wherein the method further comprises: the Enet network is trained by using supervised training and weakly supervised training, comprising: The preliminarily restored ancient document image is distorted by a set of real deformation fields to form synthetic data input to the Enet network for training, and for the synthetic data input to the Enet network, a loss function of supervised learning is constructed to train the Enet network, as shown in formula (7), （7） wherein, is a function to extract the deformation field boundary, is the real deformation field, is a synthetic deformed document image for training and its corresponding real deformation field deformation field generated through the Enet network; For the real data input to the Enet network, that is, the preliminarily restored ancient document image, a loss function of weak supervised learning is constructed to train the Enet network, as shown in formula (8), （8） wherein, is a warping operation for warping the mask according to the warping field, is is the inverse warping field of is a binary map containing edge or contour information of the document, representing the document geometry of the warped document, the rectangular mask is the target shape of the document after completion of the warp correction.

5. The method of claim 4, wherein the method further comprises: The Tnet network is trained by self-supervised training, including: A self-supervised loss function is constructed, as shown in formula (9), （9） wherein, is a deformation operation for deforming the image according to a deformation field, and is a deformation field generated by the Tnet network, is a deformation field applied to the original input image, is a deformation field applied to the deformed image, is a deformation perturbation, is is an inverse deformation field of

Images