A historical document restoration method and system based on implicit interpolation network enhancement

Abstract

The application discloses a historical document restoration method and system based on implicit interpolation network enhancement, and the method comprises the following steps: acquiring a damaged historical document image, a position mask image for identifying a region to be restored, and a content guide image for guiding the semantic content of the restoration; constructing a historical document restoration model based on a denoising diffusion probability model; training the historical document restoration model by using a training data set containing pairs of damaged images and intact images; inputting the damaged historical document image to be restored, the position mask image and the content guide image into the historical document restoration model, and outputting a restored historical document image. The application significantly improves the accuracy of the character content of the restoration region and the naturalness of the visual boundary, and realizes high-fidelity digital restoration of complex degraded documents such as ancient books, inscriptions and rubbing sheets.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a historical document restoration method and system based on implicit interpolation network enhancement. BACKGROUND

[0002] As an important carrier of cultural heritage, historical documents are prone to degradation phenomena such as ink erosion, paper damage, and character loss during long-term preservation. Therefore, physical restoration is needed. However, direct physical restoration of real historical documents carries high risks, and once the operation is failed, it may cause permanent damage and irreversible cultural loss. In contrast, digital restoration technology can avoid physical risks and provide visual references and decision-making basis for subsequent restoration through non-contact processing.

[0003] Existing image restoration methods are mostly based on convolutional neural networks or generative adversarial networks, which have made progress in improving overall image quality, but have obvious shortcomings in maintaining text stroke details. These methods often over-smooth details, resulting in blurred and unclear edges of restored text strokes, especially for complex character structures. In addition, due to the lack of historical document data, the model lacks generalization ability and is difficult to adapt to diverse degradation scenarios.

[0004] In recent years, diffusion models have shown great potential in image generation, but their direct application to historical document restoration lacks sensitivity to details.

[0005] Therefore, the present application is proposed. SUMMARY

[0006] The present application provides a historical document restoration method based on implicit interpolation network enhancement, which includes:

[0007] S101, obtaining a damaged historical document image , a position mask image identifying the area to be repaired , and a content guide image guiding the semantic content of the repair ;

[0008] S102, constructing a historical document restoration model based on a denoising diffusion probability model; wherein the historical document restoration model contains an encoder and a decoder, and an implicit interpolation network is embedded in the feature reconstruction path of the decoder for enhancing the image features generated by the decoder, and the enhanced features are fused with the original features through residual connection;

[0009] S103, training the historical document restoration model using a training data set containing pairs of damaged images and intact images;

[0010] S104, inputting the damaged historical document image to be repaired , a position mask image , and a content guide image to the trained historical document repair model, and outputting a repaired historical document image .

[0011] Preferably, at each step of denoising sampling, the noise image of the current time step , the damaged historical document image , the content guide image , and the position mask image are spliced in the channel dimension to form a composite condition tensor, which is input to the historical document repair model.

[0012] The historical document repair model adopts a classifier-agnostic guidance strategy, and the predicted results are weighted and synthesized by setting a content guide scale factor and an image structure preservation scale factor, so as to constrain the repair results to align with the target character semantics and maintain the coherence with the image structure of the undamaged area.

[0013] The diffusion model sampler is used to update the image state step by step according to the weighted and synthesized predicted results, and after multiple iterations, a repaired historical document image is output .

[0014] Preferably, the historical document repair model is based on a Diffusion Transformer architecture, and its working process includes:

[0015] The input composite condition tensor is divided into multiple image blocks, and each image block is mapped to a feature vector to form an initial token sequence.

[0016] The encoder performs multi-level feature extraction on the initial token sequence, down-samples the token sequence through block merging operation, and outputs a feature pyramid containing different scales.

[0017] The decoder is composed of multiple layers of up-sampling blocks corresponding to the number of layers of the encoder, and the features are reconstructed by the multiple layers of up-sampling blocks. In the up-sampling blocks, the features of the current scale are enhanced by an implicit interpolation network, and the enhanced features are fused with the skip connection features of the corresponding scale from the encoder to gradually reconstruct the repaired image.

[0018] Preferably, the implicit interpolation network enhances feature details by a coordinate-based neural rendering technique, and its specific implementation includes the following steps:

[0019] receive an image feature tensor from the up-sampling block, and obtain normalized spatial coordinates of each pixel point in the image feature tensor to form a coordinate-feature pair;

[0020] map the normalized spatial coordinates to a high-dimensional continuous space by using a preset spatial encoding function to form a high-dimensional coordinate vector; wherein the spatial encoding function is a periodic sine encoding function, which is used to map the input low-dimensional coordinates to a high-dimensional space to better represent the high-frequency details of the image;

[0021] perform a channel dimension splicing or addition operation on the encoded high-dimensional coordinate vector and the corresponding image feature tensor, and input the fused result into a parameterized multi-layer perception network;

[0022] perform a nonlinear transformation on the fused result by the multi-layer perception network to output a detail enhancement feature vector;

[0023] superimpose the detail enhancement feature vector on the original decoder feature through residual connection, and output the enhanced feature as the enhanced feature for the next level of up-sampling block to perform image reconstruction, thereby improving the recovery quality of the historical document character stroke edge and complex texture.

[0024] Preferably, a staged progressive strategy is adopted when training the historical document restoration model, including:

[0025] In the semantic feature learning stage, a character perception loss is used to train to preferentially learn the semantic features of the characters, and the formula is: , wherein, represents the i-th layer feature map of the VGG network, L is the total number of layers, represents the target image, is the layer weight of the i-th layer, is a binary mask identification of the damaged area, and represents element-wise multiplication, and the layer weight is set to 1, so that the model focuses on understanding the character content and avoids optimizing the pixel-level details too early, thereby reducing blurring or distortion;

[0026] In the pixel-level reconstruction optimization stage, a diffusion loss is added for joint training to optimize the pixel-level reconstruction accuracy and balance the semantic preservation and pixel accuracy; the total loss of this stage is , wherein, ;

[0027] In the boundary fine-tuning and detail enhancement stage, a boundary consistency loss is added for fine-tuning to optimize the boundary naturalness and global visual coherence of the repaired area; the total loss of this stage is: ; wherein, .

[0028] Preferably, the content guidance image is generated according to the text content provided by the user is a binary image of a single channel, whose size is consistent with the historical document image , and it encodes the semantic information of the characters as the conditional input of the historical document inpainting model, ensuring that the restored characters are correct in content;

[0029] The position mask image is generated according to the position information provided by the user is a single-channel image that identifies the pixel positions that need to be inpainted, providing spatial prior for the historical document inpainting model, so that the inpainting focuses only on the target area.

[0030] The embodiment of the present application also provides a historical document inpainting system based on implicit interpolation network enhancement, which comprises:

[0031] An image acquisition unit is configured to acquire a damaged historical document image , a position mask image identifying the area to be inpainted , and a content guidance image guiding the semantic content of the inpainting .

[0032] A model construction unit is configured to construct a historical document inpainting model based on a denoising diffusion probability model; wherein the historical document inpainting model comprises an encoder and a decoder, and an implicit interpolation network is embedded in the feature reconstruction path of the decoder, for enhancing the image features generated by the decoder, and fusing the enhanced features with the original features through residual connection.

[0033] A model training unit is configured to train the historical document inpainting model using a training data set comprising pairs of damaged images and intact images;

[0034] An inpainting unit is configured to input the damaged historical document image , the position mask image , and the content guidance image to be inpainted to the trained historical document inpainting model, and output the inpainted historical document image .

[0035] Compared with the prior art, the present application has the following advantages:

[0036] 1. By creatively embedding the implicit interpolation network, a technique suitable for modeling continuous signals, into the decoding process of the discrete diffusion model, and using its coordinate mapping capability, the recovery of high-frequency details such as character strokes is explicitly enhanced.

[0037] 2. A phased progressive training strategy is designed, and a boundary consistency loss is introduced, systematically solving the problem of unnatural transition of the repaired area edge, ensuring the visual coherence of the repair result and the original document, and making the background repair transition natural and unobtrusive. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is the flowchart of the historical document repair method based on implicit interpolation network enhancement provided by the first embodiment of the present application;

[0039] Figure 2 and Figure 3 is the schematic diagram of the historical document image provided by the embodiment of the present application; , the position mask image and the content guide image

[0040] Figure 4 is the architecture diagram of the historical document repair model provided by the first embodiment of the present application;

[0041] Figure 5 is the module detail diagram of the implicit interpolation network provided by the first embodiment of the present application;

[0042] Figure 6 is the schematic diagram of embedding the implicit interpolation network between the decoders provided by the embodiment of the present application;

[0043] Figure 7 is the repair effect comparison diagram provided by the embodiment of the present application; Wherein, a-e is the damaged historical document image before repair; f-j is the historical document image after repair;

[0044] Figure 8 is the schematic diagram of the training early stopping mechanism provided by the embodiment of the present application;

[0045] Figure 9 is the structural schematic diagram of the historical document repair system based on implicit interpolation network enhancement provided by the second embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0047] In order to better understand the technical solutions of the present application, the embodiments of the present application will be described in detail below with reference to the drawings. ​

[0048] It should be noted that the described embodiments are merely a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0049] The terms used in the embodiments of the present application are merely for the purpose of describing the specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0050] It should be understood that the term "and / or" used herein merely describes an association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects.

[0051] Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if it is determined" or "if (a stated condition or event) is detected" can be interpreted as "when it is determined" or "in response to determining" or "when (a stated condition or event) is detected" or "in response to detecting (a stated condition or event)".

[0052] The "first / second" mentioned in the embodiments is merely to distinguish similar objects, and does not represent a specific order of the objects. Understandably, the "first / second" can be interchanged in a specific order or sequence as permitted. It should be understood that the objects distinguished by "first / second" can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0053] Please refer to Figure 1 The first embodiment of the present application provides a historical document repair method based on implicit interpolation network enhancement, which can be executed by a historical document repair device based on implicit interpolation network enhancement (hereinafter referred to as repair device), in particular, by one or more processors in the repair device to implement the following steps:

[0054] S101, obtaining a damaged historical document image, a position mask image identifying a to-be-repaired region, and a content guide image guiding the semantic content of the repair.

[0055] In the embodiment, the repairing device can be an electronic device equipped with a processor with a computer program of the repairing method and the computer program can be executed, such as a computer, a smart phone, a smart tablet, a workstation, etc., which are not limited herein.

[0056] In the embodiment, as shown in Figure 2 and Figure 3 , the system first receives a damaged historical document image input by a user The historical document image may be an image of a complex degraded document such as an ancient book or a rubbing of an inscription. It is assumed that the shape is 3xHxW, where 3 represents an RGB channel, H is the height of the image, and W is the width of the image.

[0057] After obtaining the historical document image , the user also needs to label the damaged area on the damaged historical document image and the content of the damaged area.

[0058] First, according to the position information provided by the user, a single-channel position mask image is generated, which has a shape of 1xHxW.

[0059] Specifically, the user provides a bounding box of the damaged area , , The system initializes a black image with the same size as , and then fills the damaged area with white pixels according to the parsed position information to obtain a mask area. Morphological operations (erosion, kernel size 3x3) are applied to smooth the mask boundary of the mask area, reduce the sawtooth effect, and verify whether the mask area is visually consistent with the damaged area of If consistent, the required position mask image is generated , which identifies the pixel positions that need to be repaired and provides spatial prior for the model, so that the repair focuses only on the target area.

[0060] Then, according to the text content provided by the user, a single-channel content guide image is generated, which also has a shape of 1xHxW.

[0061] Specifically, the system parses the text content input by the user, identifies the character sequence and the expected layout (font type, size and line spacing, which uses the common font of historical documents by default). Subsequently, the text content is drawn onto a blank canvas using a graphics library (OpenCV) to generate a historical document image Consistent single-channel image. Text regions are shown in black pixels (value 0) and background in white pixels (value 255), forming a binary mask. This image encodes the semantic information of the characters, which is used as the conditional input of the historical document restoration model, ensuring that the restored characters are correct in content.

[0062] where if the user provides a text location hint (e.g., start coordinates), the system renders the text to the specified location; otherwise, it renders the text centered by default and calibrates it with OCR assistance (aligns the remaining text with ).

[0063] S102, constructing a historical document restoration model based on a denoising diffusion probability model; wherein the historical document restoration model comprises an encoder and a decoder, and an implicit interpolation network is embedded in a feature reconstruction path of the decoder, for enhancing the image features generated by the decoder, and fusing the enhanced features with the original features through a residual connection.

[0064] S103, training the historical document restoration model using a training data set comprising pairs of damaged images and intact images.

[0065] S104, inputting the damaged historical document image to be restored, the position mask image and the content guide image into the trained historical document restoration model, and outputting the restored historical document image.

[0066] In this embodiment, the historical document restoration model is constructed based on a denoising diffusion probability model, which includes an encoder-decoder architecture based on Diffusion Transformer and a time step embedding mechanism as shown in Figure 4 . The encoder consists of 4 layers of down-sampling blocks, and the output channel numbers are 256, 512, 768, and 1024 in sequence; the decoder is symmetrical with the encoder and contains 4 layers of up-sampling blocks, which gradually reconstruct the features. The time step embedding adopts a position encoding method, and the formula is:

[0067]

[0068] where t is the time step of the diffusion process, is the frequency base, = , i from 0 to d / 2, d is the embedding dimension 64. is added to the self-attention layer or the feedforward network of the Transformer block, and the conditioning is realized through feature concatenation or addition operation.

[0069] When the historical document restoration model receives , , After that, the model generates a noisy image at each time step t of the denoising loop The input image is then augmented by applying random affine transformations (rotation, scaling, translation, and flipping) to create a batch of images The augmented images are then concatenated along the channel dimension to form an 8-channel composite condition tensor, where: is the 3-channel noisy image generated by the diffusion process. The noisy image is generated by: where is the Gaussian noise, is the noise schedule coefficient.

[0070] The concatenated composite condition tensor is then passed through an encoder-decoder architecture for forward propagation. First, the composite condition tensor is split into fixed-size image patches using the Patch Embedding operation, and each patch is mapped to a token by a linear projection and added with learnable positional encoding to generate an initial token sequence with shape Nxd, where N is the number of patches and d is the feature dimension.

[0071] The encoder consists of 4 hierarchical Transformer blocks, each containing multi-head self-attention (MSA), feed-forward network (FFN), and residual connection. The token sequence is progressively downsampled (by patch merging operation) to output a multi-scale feature pyramid where corresponds to features with different resolutions (the number of channels increases to 256, 512, 768, and 1024, respectively). These features capture information from local textures to global semantics.

[0072] The decoder path: The decoder is symmetric to the encoder and contains 4 upsampling blocks. Before processing, each block first enhances the details using an implicit interpolation network (INR).

[0073] As shown in Figure 5 and Figure 6 , the implicit interpolation network maps the pixel coordinates to a high-dimensional space, uses an MLP to predict the RGB values, and then performs a 1x1 convolution operation to change the 3 channels to the input channels and fuse them with the original features: The decoder also integrates a skip connection to fuse features of the corresponding scale from the encoder feature pyramid, ensuring the consistency of multi-scale information.

[0074] Throughout the process, temb is injected into the attention mechanism of each Transformer block, while the content-guided image and the binary mask image ​​​​As a conditional token, it is integrated into the decoder layer through a cross-attention mechanism to control the spatial and semantic consistency of the repair region.

[0075] The model outputs the predicted noise tensor (shape and...) (Same as above), then the diffusion scheduler (DPM-Solver++) calculates the update amount based on the current time step t, and updates the image state step by step. The denoising formula is:

[0076] in It is noise in the model prediction. It is the variance parameter. It's random noise. This process is repeated iteratively (1000 steps), and the final output is a repaired image. And it is mapped to the range [0,1] through numerical normalization.

[0077] In this embodiment, as Figure 5 As shown, the core function of the implicit interpolation network is to establish a continuous mapping relationship from two-dimensional pixel coordinates to three-dimensional RGB colors through a parameterized multilayer perceptron (MLP). Specifically, the input of the implicit interpolation network includes two parts: one is the image feature tensor E∈ extracted by the shared encoder. Secondly, there are the normalized spatial coordinates of each pixel. To enhance the model's ability to recover high-frequency details, the coordinate information is first mapped to a high dimension by a periodic spatial encoder, with the functional form P(x) = [sin(x), cos(x),...,sin( x), cos( x)], where the hyperparameter L=3 is used to control the frequency. Finally, a three-layer MLP (128 hidden layer dimensions) is responsible for decoding the input that fuses high-dimensional coordinates and image features.

[0078] To generate smooth and continuous results, the final RGB value s∈R3 of each pixel is obtained by weighted averaging of the predicted values ​​of its four nearest neighbor feature points (satisfying... =1) The fusion process is equivalent to a kind of implicit neural interpolation, enabling the model to adaptively learn shared features of different injury types. Specific implementation details are as follows:

[0079] Create a coordinate generation module for:

[0080]

[0081] ② Use torch.meshgrid to generate multi-dimensional grid coordinates, shape (h, w, 2) , ) coordinates, 2 represents and two dimensions).

[0082] Create a position encoding module to map low-dimensional coordinates to high-dimensional space and enhance the network's ability to model high-frequency information. For the input coordinates X∈ , encode as P(x) = [sin(x), cos(x),...,sin( x), cos( x)], the experimental setting L = 3, so the original 2-dimensional feature is mapped to 4L-dimensional.

[0083] Create a multi-layer perceptron prediction module as the core prediction network of INR, which maps input features to RGB output. Its structure is: the input channel dimension is the number of channels output by the upper decoder (dim), and the output dimension is 3 (corresponding to RGB three channels); the hidden layer dimension is specified as [128, 128, 128]; the structure of each layer is: fully connected layer + ReLU activation function, and the last layer has no activation function. Forward propagation flattens the input tensor to (-1, dim), reshapes it to the original input shape after passing through the multi-layer network (retaining all dimensions except the last dimension), and the output shape is (original shape, 3).

[0084] Create a local integrated prediction module to improve prediction accuracy through local integration of multiple offset coordinates, and the core steps are as follows:

[0085] ① Feature expansion, use F.unfold to perform 3x3 convolution kernel expansion (padding 1) on the input feature feat (shape (B, dim, h, w)), get q_feat (B, dimx9, h, w), that is, each position contains 3x3 neighborhood features;

[0086] ​②Offset coordinate generation, generate 4 offset coordinates (vx, vy) e {(-1, -1), (-1, 1), (1, -1), (1, 1)}, offset amount = 1 / h, =1 / w (pixel half-width under normalized coordinates); =1 / w (pixel half-width under normalized coordinates);

[0087] ③Feature and coordinate matching, generate the grid coordinates feat_coord of the feature (shape (B, 2, h, w)), which corresponds to the unfolded feature feat after flattening, position encode the feature coordinates feat_coord, and concatenate the original coordinates and the encoding results (same as the coordinate encoding in the forward propagation);

[0088] ④Calculate the relative coordinates, calculate the relative coordinates of the query coordinates coord and the feature coordinates feat_coord, and normalize them by the feature size: rel_coord[:, :, 0] = (coord_x - feat_coord_x) x h, rel_coord[:, :, 1] = (coord_y - feat_coord_y) x w;

[0089] ⑤Input feature concatenation, concatenate the unfolded neighborhood feature (q_feat), the relative coordinates (rel_coord), and the pixel interval (rel_cell) as the input of the MLP;

[0090] ⑥Local ensemble fusion, predict RGB for each of the 4 offset coordinates to get 4 prediction results preds, calculate the weight of each offset (inversely proportional to the area of the relative coordinates): area = |rel_coord_x x rel_coord_y|, normalize the weight and fuse the prediction results: ret =

[0091] ⑦Output reshaping, reshape the fusion result to (B, 3, h, w), i.e., the RGB image format.

[0092] Finally, combine the above several modules, input the image feature, and generate a complete RGB prediction result

[0093] The steps are:

[0094] (1) Generate coordinates coord: generate coordinates with image size (h, w) through the coordinate generation module, shape (B, h x w, 2) (B is the batch size).

[0095] (2) Generate pixel interval cell: represents the size of each pixel in the normalized coordinates, cell[:,0] = 2 / h, cell[:,1] = 2 / w, the shape is the same as coord.

[0096] (3) Coordinate encoding: apply position encoding to coord, concatenate the original coordinates and the encoding results, the shape is (B, h x w, 2+4L).

[0097] (4) Call the prediction module method, input the features, the encoded coordinates and the cell, and return the RGB residual prediction result.

[0098] Wherein, the specific repair comparison is shown in Figure 7 , and Figure 7 The left a-e are the input damaged historical document images before repair; and the right f-j are the historical document images after repair, it can be seen that the embodiment realizes effective repair of the damaged historical document image.

[0099] Some preferred embodiments of the application will be further described below.

[0100] In this embodiment, in order to ensure the visual coherence of the repair result and the original document and make the background repair transition natural and not conspicuous, a phased progressive training strategy is designed in the training process of the model.

[0101] Specifically, in the training stage, a multi-scale loss function is used for optimization, and clean oracle bone image is used as an intact image and synthetic noise image (including images with complex noise characteristics and images with relatively simple noise characteristics ) as damaged images to constitute a training data set for training. The historical document repair model is updated during the training process, and the parameters are optimized through back propagation until the output image reaches the preset quality standard. The specific loss function and training mechanism and the progressive strategy are as follows:

[0102] Wherein, the loss function includes:

[0103] Diffusion loss: L1 norm loss is used to ensure pixel-level accuracy, and the formula is:

[0104] Wherein is the repair image output by the model, is the target image. This loss dominates the global structure reconstruction.

[0105] Character perception loss: VGG-19 network-based feature alignment loss, focusing on the damaged area to improve semantic consistency. The formula is:

[0106] Wherein, represents the i-th layer feature map of VGG network, L is the total number of layers, represents the target image, is the layer weight of the i-th layer, is a binary mask to identify the damaged area (value 1 represents damaged pixels), and represents element-wise multiplication. This loss ensures that the character strokes and texture details are consistent with the target.

[0107] Boundary consistency loss: constrain the boundary smoothness between the repaired area and the undamaged area, and reduce artifacts. The formula is:

[0108] where, is the gradient operator (such as the Sobel filter), and the loss calculation is limited to the vicinity of the boundary by the mask (1- ).

[0109] where, the training mechanism and progressive strategy include:

[0110] First stage (semantic feature learning stage): only use character perception loss , enabled in the first 50 rounds, to make the model preferentially learn high-level semantic features (such as the overall shape of the character), and the total loss formula is: .

[0111] where, in this stage, the layer weight is set to 1, so that the model focuses on understanding the content of the character, avoiding the premature optimization of pixel-level details, thereby reducing blurring or distortion

[0112] Second stage (pixel-level reconstruction optimization stage): add diffusion loss , enabled in the 50th-100th round, to balance semantic and pixel-level optimization. The total loss formula is: .

[0113] Third stage (boundary fine-tuning and detail enhancement stage): add boundary consistency loss, in the 100th-200th round, all losses are fully enabled, and the total loss is , to ensure that the boundary loss serves as a late regularization term, fine-tuning the details without interfering with the main training.

[0114] Data augmentation: construct degradation-clean sample pairs: use existing historical document datasets and simulate physical degradation in a random manner (such as simulating ink diffusion, stains, local missing, etc.) to construct paired data for training.

[0115] Optimizer and scheduling: use AdamW optimizer (learning rate 1e-4, weight decay 1e-5) combined with cosine annealing scheduling, train for 200 rounds, and the early stopping mechanism is based on the PSNR index of the validation set, with a "patience" value of 20, as shown in Figure 7 .

[0116] Compared with the prior art, the embodiment has at least the following advantages:

[0117] 1. By embedding the implicit interpolation network, a technique suitable for modeling continuous signals, into the decoding process of the discrete diffusion model, the recovery of high-frequency details such as character strokes is explicitly enhanced using the coordinate mapping capability of the implicit interpolation network.

[0118] 2. A phased progressive training strategy is designed, and a boundary consistency loss is introduced to systematically solve the problem of unnatural transition at the edge of the repaired area, ensuring the visual coherence of the repaired result with the original document and making the background repair transition natural and unobtrusive.

[0119] Referring to Figure 9 , the second embodiment of the present application also provides a historical document repair system based on implicit interpolation network enhancement, which comprises:

[0120] An image acquisition unit 210 is configured to acquire a damaged historical document image , a position mask image for identifying the area to be repaired , and a content guide image for guiding the semantic content of the repair .

[0121] A model construction unit 220 is configured to construct a historical document repair model based on a denoising diffusion probability model; wherein the historical document repair model comprises an encoder and a decoder, and an implicit interpolation network is embedded in the feature reconstruction path of the decoder, for enhancing the image features generated by the decoder, and fusing the enhanced features with the original features through residual connection;

[0122] A model training unit 230 is configured to train the historical document repair model using a training data set comprising pairs of damaged images and intact images;

[0123] A repair unit 240 is configured to input the damaged historical document image , the position mask image , and the content guide image to be repaired to the trained historical document repair model, and output a repaired historical document image .

[0124] The above describes the preferred embodiments of the present application. It should be noted that for those skilled in the art of historical document repair technology, without departing from the core principles of the present application (such as diffusion model architecture, implicit neural representation enhancement, and multi-scale training strategy), a number of improvements and refinements can be made, which are also considered within the scope of protection of the present application.

Claims

1. A historical document restoration method based on implicit interpolation network enhancement, characterized in that, Comprising: S101, acquire a damaged historical document image , a position mask image identifying a region to be repaired , and a content guide image guiding content semantics of the repair ; S102, constructing a historical document restoration model based on a denoising diffusion probability model; wherein the historical document restoration model contains an encoder and a decoder, and an implicit interpolation network is embedded in the feature reconstruction path of the decoder for enhancing the image features generated by the decoder, and the enhanced features are fused with the original features through residual connection; wherein the implicit interpolation network enhances feature details through a coordinate-based neural rendering technique, which includes the following steps: receiving an image feature tensor from an up-sampling block of the decoder, and obtaining the normalized spatial coordinates of each pixel point in the image feature tensor to form a coordinate-feature pair; using a pre-set spatial encoding function to map the normalized spatial coordinates to a high-dimensional continuous space to form a high-dimensional coordinate vector; wherein the spatial encoding function is a periodic sine encoding function for mapping low-dimensional coordinates to high-dimensional space to better represent the high-frequency details of the image; the encoded high-dimensional coordinate vector and the corresponding image feature tensor are spliced or added in the channel dimension, and the fused result is input into a parameterized multi-layer perception network; the multi-layer perception network performs nonlinear transformation on the fused result to output a detail enhancement feature vector; the detail enhancement feature vector is superimposed with the original decoder feature through residual connection, and the output is used as the enhanced feature for image reconstruction by the next level of the up-sampling block of the decoder, thereby improving the restoration quality of the edges and complex textures of historical document characters. S103, training the historical document restoration model using a training data set containing pairs of damaged images and intact images; S104, inputting the damaged historical document image to be repaired , a position mask image and a content guide image to the trained historical document repair model, and outputting a repaired historical document image .

2. The historical document restoration method based on implicit interpolation network enhancement according to claim 1, characterized in that: At each step of the denoising sampling, a noisy image for the current time step is generated by adding noise to the previous time step's image with the impaired historical document image , a content guide image , and a position mask image are concatenated in the channel dimension to form a composite condition tensor, which is input to the historical document restoration model; The historical document restoration model adopts a classifier-independent guidance strategy, and the predicted results are weighted and synthesized by setting a content guidance scale factor and an image structure preservation scale factor to constrain the restoration results to align with the target character semantics and maintain the coherence with the image structure of the undamaged area. The image state is updated step by step according to the weighted synthesized prediction result by using a diffusion model sampler, and after multiple iterations, the repaired historical document image is output .

3. The historical document restoration method based on implicit interpolation network enhancement according to claim 2, characterized in that: The historical document restoration model is based on the Diffusion Transformer architecture, and its working process includes: Divide the input complex condition tensor into multiple image blocks, and map each image block to a feature vector to form an initial token sequence; The encoder performs multi-level feature extraction on the initial token sequence, down-samples the token sequence through block merging operation, and outputs a feature pyramid containing different scales; The decoder is composed of multiple layers of up-sampling blocks corresponding to the number of layers of the encoder, and the features are reconstructed through the multiple layers of up-sampling blocks, and in the up-sampling blocks, the current scale features are enhanced through the implicit interpolation network, and the enhanced features are fused with the corresponding scale jump connection features from the encoder to gradually reconstruct the restoration image.

4. The method of claim 1, wherein, The historical document restoration model is trained by adopting a staged progressive strategy, including: In the semantic feature learning stage, a character-aware loss is used is trained to preferentially learn the semantic features of the characters, with the formula: ; wherein, represents the i-th layer feature map of the VGG network, L is the total number of layers, represents the target image, is the layer weight of the i-th layer, is a binary mask identifying the damaged area, ⊙ represents element-wise multiplication, and the layer weight is set to 1, so that the model focuses on understanding the content of the characters and avoids prematurely optimizing pixel-level details, thereby reducing blurring or distortion. In the pixel-level reconstruction optimization stage, a diffusion loss is added Joint training is performed to optimize the pixel-level reconstruction accuracy and balance the semantic preservation and pixel accuracy. The total loss in this stage is ; In the boundary fine-tuning and detail enhancement stage, the boundary consistency loss is added The fine-tuning is performed to optimize the boundary naturalness and global visual coherence of the repaired region; the total loss of this stage is: .

5. The historical document restoration method based on implicit interpolation network enhancement according to claim 1, characterized in that: The content-guided image According to the user-provided text content generation, the content-guided image It is a binary image of a single channel, and its size is consistent with the historical document image It encodes the semantic information of the character as the conditional input of the historical document restoration model, ensuring that the restored character is correct in content. The position mask image According to the user-provided position information generation, the position mask image Is a single-channel image, the size of which is consistent with the historical document image Identifies the pixel positions that need to be repaired, providing a spatial prior for the historical document repair model, so that the repair focuses only on the target area.

6. A historical document restoration system based on implicit interpolation network enhancement, characterized in that, including: An image acquisition unit is configured to acquire a damaged historical document image , a position mask image identifying a region to be repaired , and a content guide image guiding content semantics of a repair ; A model construction unit is configured to construct a historical document restoration model based on a denoising diffusion probability model; wherein the historical document restoration model comprises an encoder and a decoder, and an implicit interpolation network is embedded in a feature reconstruction path of the decoder, used to enhance the image features generated by the decoder, and the enhanced features are fused with the original features through residual connection; wherein the implicit interpolation network enhances feature details through a coordinate-based neural rendering technique, and its specific implementation includes the following steps: receiving an image feature tensor of an up-sampling block from the decoder, and obtaining the normalized spatial coordinates of each pixel point in the image feature tensor to form a coordinate-feature pair; a preset spatial encoding function is used to map the normalized spatial coordinates to a high-dimensional continuous space to form a high-dimensional coordinate vector; wherein the spatial encoding function is a periodic sine encoding function, which is used to map the input low-dimensional coordinates to a high-dimensional space to better represent the high-frequency details of the image; the encoded high-dimensional coordinate vector and the corresponding image feature tensor are spliced or added in the channel dimension, and the fused result is input into a parameterized multi-layer perception network; the multi-layer perception network performs nonlinear transformation on the fused result to output a detail enhancement feature vector; the detail enhancement feature vector is superimposed with the original decoder feature through residual connection, and the output is used as the enhanced feature for the next level of up-sampling block of the decoder to reconstruct the image, thereby improving the recovery quality of the historical document character stroke edge and complex texture; A model training unit is configured to train the historical document restoration model using a training data set comprising damaged image and intact image pairs; A repairing unit for repairing the damaged historical document image to be repaired , a position mask image , and a content guide image Input into the trained historical document repairing model, output the repaired historical document image .

Images