Document restoration method and device based on structure-oriented diffusion model and OCR perception loss, and equipment

Abstract

This application relates to a document restoration method, apparatus, and device based on a structure-guided diffusion model and OCR perceptual loss. The method includes generating an alpha mask from an input document image using a segmentation network embedded with a Markov attention mechanism; generating a preliminary restored image from the input document image and the alpha mask using a structure-guided latent diffusion restoration network; and generating a final restored image from the preliminary restored image using an enhancement network constrained by OCR perceptual loss. This application accurately generates the alpha mask using a Markov attention segmentation network, preserving the texture and strokes under the seal; it employs a structure-guided latent diffusion network to restore geometric continuity under edge constraints and utilizes a biomimetic activation function to retain subtle details; finally, it uses an enhancement network constrained by OCR perceptual loss to ensure high fidelity of the restoration result at both the pixel and semantic levels.

Description

Technical Field

[0001] This application relates to the field of computer vision and image processing technology, and in particular to a document restoration method, apparatus and device based on structure-guided diffusion model and OCR perceptual loss. Background Technology

[0002] In critical sectors such as finance, law, and government, a large number of paper documents require digitization for archiving, retrieval, and automated analysis. However, these document images are often covered by various seals (such as official seals and interlocking seals), handwritten annotations, or stains. These markings not only affect visual appeal but also obscure key textual information such as signatures, amounts, and dates, severely compromising the structural integrity of the document. This leads to a significant decrease in the recognition accuracy of subsequent optical character recognition systems, hindering the automation and efficiency of business processes.

[0003] Currently, solutions to such problems are mainly divided into methods based on traditional image processing and methods based on deep learning. Traditional image processing methods often rely on setting thresholds in specific color spaces (such as HSV, YCbCr) to separate the red stamp area. These methods heavily depend on human experience and fixed parameters, and perform poorly in scenes with similar colors (such as grayscale stamps in black and white photocopies) or complex textured backgrounds. Furthermore, the segmentation edges often appear jagged, failing to accurately preserve the original information under semi-transparent areas. Deep learning-based methods primarily use generative adversarial networks for end-to-end image inpainting. However, when faced with large-area and structurally complex occlusions, traditional generative models often struggle to maintain the inherent geometric structure of the document (e.g., causing broken table lines and misaligned text lines). The generated text areas are also prone to "ghosting" phenomena such as blurred strokes, character overlap, or semantic distortion, making the inpainted results visually unnatural and failing to meet the practical application requirements in terms of machine readability.

[0004] Therefore, designing a document restoration method that can simultaneously achieve pixel-level visual naturalness, structural-level geometric coherence, and semantic-level machine readability has become a core technological bottleneck that urgently needs to be overcome in this field. The SDO-Net architecture proposed in this invention aims to systematically solve the above problems by integrating the high-fidelity generation capability of the latent diffusion model, precise mask control based on physical priors, and semantic-level optimization perceptualized by OCR, thereby achieving high-quality, high-fidelity restoration of occluded documents. Summary of the Invention

[0005] Based on this, this application provides a document restoration method based on a structure-guided diffusion model and OCR perceptual loss, characterized by comprising: Based on the input document image, an alpha mask is generated by a segmentation network with an embedded Markov attention mechanism. Based on the input document image and the alpha mask, a preliminary repair image is generated through a structure-guided latent diffusion repair network; Based on the preliminary repaired image, the final repaired image is generated through an enhanced network with OCR-perceived loss constraints.

[0006] Optionally, the step of generating an alpha mask based on the input document image using a segmentation network embedding a Markov attention mechanism includes: Based on the input document image, perform multimodal input tensor construction to enhance the feature separation between the region to be repaired and the background content; Based on the multimodal input tensor, a segmentation network with embedded Markov attention mechanism is used to perform probability state transition calculations for the region to be repaired, generating an initial segmentation result; Based on the initial segmentation result, continuous attention map iterative smoothing is performed to generate the alpha mask.

[0007] Optionally, the step of generating a preliminary repair image based on the input document image and the alpha mask using a structure-guided latent diffusion inpainting network includes: Based on the input document image, perform document structure edge feature extraction to generate an edge map condition; Based on the edge map conditions and the alpha mask, a controlled denoising process is performed in the latent space through the structure-guided latent diffusion repair network to generate an initial reconstruction result; Based on the initial reconstruction results, a biomimetic activation function is executed to generate the preliminary repaired image.

[0008] Optionally, generating the final restored image based on the preliminary restored image using an enhanced network with OCR-perceived loss constraints includes: Based on the preliminary repaired image, the final repaired image is generated by performing super-resolution refinement and semantic consistency optimization processing through an enhanced network that integrates a moving window attention mechanism and OCR semantic consistency constraints. During the optimization process of the enhanced network, the semantic consistency of its output is evaluated based on a predefined perceptual loss function; The perceptual loss function The predicted feature map output by the enhanced network is calculated. The ground truth feature map of the real document Defined by the square of the Euclidean distance between them, specifically: .

[0009] Optionally, the step of generating an initial segmentation result by performing probability state transition calculation of the region to be repaired through a segmentation network embedded with a Markov attention mechanism based on the multimodal input tensor includes: Based on the multimodal input tensor, the pixel state transition matrix is ​​constructed through the embedded Markov attention mechanism to model the probability propagation relationship of the region to be repaired between pixels; Based on the pixel state transition matrix and the initial attention map, perform matrix multiplication-based attention map iterative update to generate a smooth attention map; The attention map is iteratively updated, and the pixel state transition matrix is ​​denoted as... Matrix elements Represents pixels The probability of the area to be repaired per pixel The impact of the initial attention map , For the number of iterations, Indicates the first The attention map after the next iteration has the following iteration relationship: .

[0010] Optionally, the step of performing biomimetic activation function processing based on the initial reconstruction result to generate the preliminary repaired image includes: Based on the feature tensors in the initial reconstruction results, a biomimetic activation function is applied to the specified activation layer for processing. The biomimetic activation function is set as follows: For input feature values, The preset activation threshold, This is the scaling factor; When input value When, the output is ; When input value When, the output is .

[0011] Optionally, based on the preliminary repaired image, the step of performing super-resolution refinement and semantic consistency optimization processing through an enhanced network integrating a moving window attention mechanism and OCR semantic consistency constraints to generate the final repaired image includes: Based on the preliminary repaired image, self-attention computation is performed within non-overlapping local windows at multiple levels of the enhancement network; Regular movement of the local windows is performed between adjacent levels to achieve global information interaction across windows.

[0012] This application also provides a document repair device based on a structure-guided diffusion model and OCR perceived loss, the device comprising: The Markov attention segmentation module is used to generate an alpha mask based on the input document image by using a segmentation network that embeds a Markov attention mechanism. The structure-guided diffusion inpainting module is used to generate a preliminary inpainted image based on the input document image and the alpha mask through a structure-guided latent diffusion inpainting network. The OCR perception enhancement and repair module is used to generate the final repaired image based on the preliminary repaired image through an enhancement network constrained by OCR perception loss.

[0013] Optionally, the structure-guided diffusion repair module further includes: The structural edge extraction module is used to extract document structural edge features based on the input document image and generate edge map conditions. The latent diffusion denoising module is used to perform a controlled denoising process in the latent space through the latent diffusion model based on the edge map conditions and the Alpha mask to generate an initial reconstruction result. The biomimetic activation optimization module is used to generate the preliminary repaired image by processing the initial reconstruction results through a biomimetic activation function.

[0014] This application also provides an electronic device for implementing any of the document restoration methods based on the structure-guided diffusion model and OCR perceived loss, including: The processor is used to execute the complete computation and control process, starting from the input document image, generating an alpha mask through a segmentation network with an embedded Markov attention mechanism, generating a preliminary restored image through a structure-oriented latent diffusion restoration network, and finally generating the final restored image through an enhancement network constrained by OCR perceptual loss. The memory is used to store the pre-trained network model parameters necessary to implement the complete process, intermediate data during processing, and the final repaired image. The pre-trained network model parameters include at least the weight parameters of the segmentation network with embedded Markov attention mechanism, the weight parameters of the structure-oriented latent diffusion repair network, and the weight parameters of the OCR perceptual loss constraint enhancement network.

[0015] The beneficial effects of this application are as follows: By embedding a segmentation network based on the Markov attention mechanism of ink diffusion physics and combining it with multimodal input, accurate segmentation and Alpha mask generation of semi-transparent and blurred edge stamps are achieved, effectively preserving the original paper texture and weak character stroke information under the stamp; and through a structure-oriented latent diffusion repair network, the continuity of geometric structures such as document table lines and text lines is forcibly maintained under the guidance of ControlNet edge conditions, while using a biomimetic activation function to retain seemingly noisy weak stroke negative activation signals, achieving visual naturalness and structural integrity of the repaired area; and through an enhancement network constrained by OCR perceptual loss, combined with a moving window attention mechanism and feature space distance supervision of a frozen pre-trained OCR network, the generation network is forced to prioritize the recovery of topological structural features that are crucial for character recognition, achieving high fidelity of the repaired document at both the pixel and semantic levels. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings required in the description of the embodiments or the prior art are briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a document restoration method based on a structure-guided diffusion model and OCR sensing loss according to a specific embodiment of this application is shown. Figure 2 This diagram illustrates the architecture of a document restoration method based on a structure-guided diffusion model and OCR-perceived loss according to a specific embodiment of this application. Figure 3 This diagram illustrates a document repair apparatus based on a structure-guided diffusion model and OCR-sensing loss according to a specific embodiment of this application. Detailed Implementation

[0018] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0019] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0020] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0021] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0022] This application proposes a document restoration method based on a structure-guided diffusion model and OCR perceptual loss to address the challenges of intelligent removal and content reconstruction of occlusions such as stamps and stains in document images, achieving dual restoration at the pixel and semantic levels. Based on the semi-transparency of stamps in documents, the need to preserve geometric structure, and machine readability requirements, this application constructs a cascaded deep neural network architecture. Through three tightly coupled processing stages, it achieves progressive restoration from local to global and from pixel to semantic levels. The first stage designs an improved U-Net segmentation network (IMG-Net), innovatively embedding a Markov attention mechanism (MAM) in the bottleneck layer. This mechanism simulates the continuous diffusion and penetration physical process of ink in paper fibers. By constructing and iteratively calculating pixel state transition matrices, the attention map can smoothly fill the tiny gaps inside the stamp, thereby achieving precise focusing on the edges of the semi-transparent stamp and generating a continuous alpha mask that preserves the underlying texture. The second stage employs a Latent Diffusion Model (LDM) as the generation base and introduces ControlNet as a structure guidance branch. By extracting Canny edge features from the unoccluded parts of the document, the geometric layout is locked, and structure control signals are gradually injected into the diffusion process through zero-convolutional layers, forcing the repaired area to maintain geometric continuity. Simultaneously, a biomimetic activation function (BReLU) is used in the key activation layers of the diffusion network to effectively retain the negative activation values ​​of weak strokes misjudged as noise, preventing excessive smoothing of fine strokes during denoising. The third stage introduces an OCR perceptual loss constraint mechanism. The moving window attention mechanism of the Swin Transformer architecture is used to perform super-resolution refinement of the repaired image, and a frozen pre-trained OCR network is used as a supervisor to calculate the distance between the repaired image and the ground truth image in the high-dimensional semantic feature map of the OCR feature space. This loss is incorporated into the total loss function for backpropagation, thereby prioritizing the recovery of topological structural features crucial for OCR recognition. Furthermore, this application constructs a high-fidelity synthetic data engine, simulating the subtractive mixing optical properties of ink through a combination of alpha blending and positive overlay, and introducing ink penetration simulation and aging / wearing mechanisms to generate highly diverse and realistic training data. Experimental results show that the SDO-Net proposed in this application can effectively remove stamps while significantly improving the visual naturalness, structural integrity, and OCR recognition accuracy of the restored area, achieving high-fidelity restoration of document images.

[0023] Example 1 like Figure 1 The diagram shown is a flowchart of a document repair method based on a structure-guided diffusion model and OCR perceived loss according to an embodiment of this application. The method specifically includes the following: S100 generates an alpha mask based on the input document image through a segmentation network that embeds a Markov attention mechanism.

[0024] Specifically, to enhance the model's ability to distinguish between the stamp and the document background, the system first converts the input RGB document image to the CIELAB color space. This space decouples brightness and color information, and utilizes the red-green and blue-yellow axis components, which better align with human visual perception, providing richer color contrast cues for subsequent processing. Subsequently, the system performs Canny edge detection on the image to extract the document's geometric contour information. The original RGB image, the CIELAB-converted images of each channel, and the edge detection map together constitute a multimodal input tensor. This tensor integrates color, brightness, and structural information, laying a robust perceptual foundation for subsequent segmentation. The segmentation network employs an improved U-Net architecture, the core of which lies in embedding a Markov attention mechanism in the bottleneck layer. This mechanism simulates the continuous diffusion and penetration of ink in paper fibers in the physical world. Its key is learning a pixel state transition matrix, where each element defines the influence strength of the "stamp attribute" probability of any pixel on another adjacent pixel. During inference, the network generates a coarse attention map based on initial local feature predictions. Then, iterative matrix multiplication is performed using this state transition matrix. Each iteration propagates and smooths the attention values ​​spatially, gradually filling the tiny gaps inside the seal caused by stroke spacing, while simultaneously sharpening and refining the transition boundary between the seal and the background. Through this local-to-global attention focusing process, the network ultimately outputs a continuous and accurate alpha mask. This mask not only clearly defines the area to be repaired, but more importantly, the continuity of its probability values ​​preserves any faint paper texture and subtle stroke information that may exist beneath the seal in subsequent repair stages, avoiding information loss caused by simple binarization segmentation.

[0025] S200: Based on the input document image and the alpha mask, a preliminary repair image is generated through a structure-guided latent diffusion repair network.

[0026] Specifically, to implement geometric constraints on the generation process, the system first extracts structural edge features from the input document image. Edge detection algorithms such as Canny are typically used to obtain an edge map that mainly contains unoccluded table borders, text line outlines, and other layout dividing lines. This edge map serves as the structural condition guiding subsequent generation processes. The restoration process employs a latent diffusion model, which encodes the high-resolution image into a low-dimensional, efficient latent space using a variational autoencoder, and performs the denoising diffusion process within this space, thus achieving a balance between generation quality and computational efficiency. To accurately integrate the structural conditions into the generation process, the system introduces ControlNet as a control branch. ControlNet replicates the weights of the pre-trained diffusion model encoder as a trainable copy and connects to the decoding path of the main diffusion model through zero-convolutional layers. During inference, the extracted edge map conditions are input into ControlNet, and its extracted structural features are progressively and controllably injected into each level of the main diffusion model decoder through zero-convolutional layers. This forces the model to follow the geometric layout of the original document at each step of denoising and generating new content, ensuring that the newly generated restoration area maintains strict continuity with the original image in terms of table line extension, text line alignment, and other aspects. Furthermore, considering the gentle gradient changes between text and background, or between subtle strokes and stamp color differences in document images, a biomimetic activation function was employed in a specific activation layer of the diffusion network. This function maintains a linear response to strong signals exceeding a threshold, while using a smooth exponential function to map weak signals below the threshold. This allows the model to preserve and amplify those seemingly noisy but actually important details—weak negative or low-amplitude activation signals—thus avoiding over-smoothing of subtle textures during denoising and reconstruction. This significantly improves the visual refinement and realism of the restored area, ultimately outputting a preliminary restored image with complete structure and clear texture.

[0027] S300, based on the preliminary repaired image, the final repaired image is generated through an enhanced network with OCR-perceived loss constraints.

[0028] Specifically, the system first enhances the resolution and details of the initially repaired image. For this, an enhancement network based on the SwinTransformer architecture is employed. The SwinTransformer, by introducing a hierarchical design and a moving window attention mechanism, effectively addresses the high computational complexity of the standard Transformer model when processing high-resolution images. It computes self-attention within non-overlapping local windows and regularly moves the window position between adjacent network layers, thus achieving cross-window information interaction. This enables efficient modeling of global dependencies in the image, making it suitable for super-resolution reconstruction and detail refinement. This network processes the initially repaired image, outputting a visually higher-quality image for evaluation. However, visual fidelity alone is insufficient to guarantee the accuracy of text recognition. Therefore, the system introduces a crucial OCR perceptual loss as a semantic constraint. This mechanism utilizes an OCR network pre-trained on a large text image dataset with frozen weights as a fixed semantic feature extractor. During optimization, the image output by the enhancement network and the real, unoccluded target image are input into this frozen OCR network to extract feature maps from its deep intermediate layers. These feature maps encode high-level semantic and topological information in the image that is crucial for character recognition. Subsequently, the Euclidean distance between the generated image feature map and the real image feature map is calculated. This distance is defined as the OCR perceptual loss. By incorporating this loss term into the overall training objective of the augmentation network and participating in backpropagation, the augmentation network is forced to not only reduce pixel-level errors when refining image details, but also to prioritize optimizing features that affect OCR recognition results, such as the continuity of character strokes, the spacing between characters, and the stability of text lines. Ultimately, this direct constraint in the semantic feature space enables the final restored image output by the augmentation network to achieve a high degree of semantic consistency with the original document while maintaining pixel fidelity, ensuring the usability and readability of the restored document.

[0029] In summary, this application addresses the shortcomings of traditional document restoration methods in handling semi-transparent stamps, preserving geometric structure, and ensuring semantic readability. It proposes a cascaded restoration scheme that deeply integrates physical priors and semantic constraints. First, to address the challenge of inaccurate segmentation caused by the similarity in color between the semi-transparent stamp and background text, and the resulting blurred edges, a Markov attention mechanism simulating the physical process of ink diffusion is innovatively introduced. This mechanism constructs a pixel state transition matrix and performs iterative calculations, enabling the model to infer a globally consistent stamp probability distribution from local features. This not only accurately delineates the stamp outline but also generates a continuously grading alpha mask, thus completely preserving the subtle original paper texture and text stroke information beneath the stamp. This lays a precise regional foundation for subsequent high-fidelity restoration and avoids information loss caused by binary masks. Second, to overcome the common problems of disrupting the original document layout during restoration (such as broken table lines and misaligned text lines) and smoothing out subtle strokes, a structure-oriented latent diffusion restoration network is constructed. By extracting and injecting document edge maps as geometric conditions, ControlNet control branches are used to force the generated content to be strictly aligned with the original image structure. Simultaneously, a biomimetic activation function is used instead of traditional ReLU to effectively preserve and enhance subthreshold feature signals representing fine strokes and textures. This design ensures that the repaired area appears visually natural and seamless, achieving high fidelity in both structural continuity and detail richness. Finally, to ensure that the repaired result is not only "visually realistic" but also "machine-readable," OCR perceptual loss is creatively introduced as a semantic-level optimization objective. By utilizing a frozen pre-trained OCR network to extract high-level semantic features and constraining the repaired image to align with the real unoccluded image in the feature space, the enhancement network is driven to prioritize the recovery of topological structural features crucial for text recognition during super-resolution reconstruction. This mechanism fundamentally improves the character recognition rate of the repaired document, achieving a leap from pixel-level repair to semantic-level fidelity, greatly enhancing the practical value of the solution.

[0030] As an optional implementation of this application, optionally, in step S100, an alpha mask is generated based on the input document image using a segmentation network embedding a Markov attention mechanism, including: 110. Based on the input document image, perform multimodal input tensor construction to enhance the feature separation between the region to be repaired and the background content.

[0031] Specifically, this step aims to provide more discriminative perceptual input for subsequent segmentation networks, addressing the feature confusion caused by similar colors (such as a red stamp and black text). The process begins with a color space conversion of the input RGB document image. The system transforms it from the standard RGB color model to the CIELAB color space. The CIELAB space decouples the L channel (description of brightness) from the a (red-green axis) and b (blue-yellow axis) channels (description of color), a design that better aligns with the human eye's perception of color differences. This conversion effectively separates the significant response of the red stamp in the a channel from the low-brightness features of the black text in the L channel, providing the model with clearer color contrast cues than the original RGB space. Simultaneously, the system performs parallel extraction of the document's geometric structure information, typically using the Canny edge detection algorithm to process the original image and generate a binary edge map. This map primarily captures structural edges such as unoccluded table borders and text outlines. Ultimately, the original RGB three-channel image, the converted CIELAB three-channel image, and the single-channel edge map are stitched together along the channel dimension to form a multimodal input tensor. This tensor integrates the original pixel information, perceived color, and document structure information to construct a robust joint feature representation. This greatly enhances the feature separation between the area to be repaired and the complex background content at the input level, laying a solid foundation for accurate segmentation.

[0032] 120. Based on the multimodal input tensor, the probability state transition calculation of the region to be repaired is performed through a segmentation network with embedded Markov attention mechanism to generate the initial segmentation result.

[0033] Specifically, this step is the core of intelligent mask generation, aiming to infer the initial probability distribution of the region to be repaired from multimodal features through an attention mechanism that simulates a physical diffusion process. The segmentation network adopts an improved U-Net architecture, where the encoder progressively downsamples to extract multi-scale semantic features, the decoder progressively upsamples to recover spatial details, and deep and shallow features are fused through skip connections. The core of the network processing lies in the Markov attention mechanism embedded in its bottleneck layer (such as...). Figure 2 As shown in the diagram, this mechanism does not directly output a segmentation map, but instead constructs a state transition model between pixels. Specifically, the network first generates an initial, possibly coarse, attention map through convolutional layers based on the multimodal input tensor. Each pixel value represents the initial probability that its location belongs to the region to be repaired. Simultaneously, the mechanism internally learns and constructs a global pixel state transition matrix. Elements in the matrix Pixels are defined quantitatively. The probability of the area to be repaired per pixel The system assesses the influence of the probability of the area to be repaired, an influence learned based on feature similarity and spatial proximity, simulating the physical principle of ink diffusion from a point to the surrounding area in a paper fiber network. After generating the initial attention map, the system performs an iterative update process based on matrix multiplication, with each iteration following a relational formula. This calculation essentially propagates and smooths the current attention probability distribution across the entire image's spatial domain, according to the rules defined by the state transition matrix. Through several iterations, the probability values ​​continuously diffuse and balance in space, effectively filling attention gaps caused by uneven stamp colors or blank spaces within strokes, and making the probability transitions in edge regions more natural. Ultimately, it outputs a more coherent initial segmentation result after smoothing through the state transition calculation.

[0034] 130. Based on the initial segmentation result, perform continuous attention map iterative smoothing to generate the Alpha mask.

[0035] Specifically, this step is a continuation of the state transition calculation in step 120 and a transformation of the final output. It aims to convert the iteratively converged attention probability map into a continuously grading alpha mask that can be directly used for subsequent repair. After several rounds of Markov attention iterations, the probability distribution in the initial segmentation result has reached a stable state, forming a spatially continuous and smooth attention map. Each pixel value in this map is a continuous probability between 0 and 1, rather than a binary selection. The system then outputs this continuous attention map directly as an alpha mask. In this mask, areas with probability values ​​close to 1 are highly certain to be the main body of the seal to be repaired, areas with probability values ​​close to 0 are the background, and at the boundary between the seal and the background, especially in semi-transparent areas, a continuous gradient transition from 1 to 0 is presented. This continuous alpha value has two advantages: firstly, it accurately depicts the semi-transparent characteristics of the seal edge, avoiding the harsh boundaries caused by binary masks; secondly, it preserves the visibility weight of the original background below the seal (such as paper texture and faint strokes). In the subsequent restoration stage, this alpha mask will be used to precisely define the content area that needs to be generated and ensure that the restored image can be naturally blended with the original background based on the alpha value, thereby preserving and restoring the original details and texture information of the underlying document to the greatest extent while removing the stamp.

[0036] As an optional implementation of this application, optionally, in step S200, based on the input document image and the alpha mask, a preliminary repair image is generated through a structure-guided latent diffusion repair network, including: 210. Based on the input document image, perform document structure edge feature extraction to generate edge map conditions.

[0037] Specifically, this step aims to provide strong geometric constraints for the image inpainting process, ensuring that the newly generated content strictly adheres to the original document's layout and preventing broken table lines or misaligned text lines. The system's starting point is the original input document image. To extract its stable geometric skeleton, edge detection algorithms such as Canny are typically used to process the image. This algorithm employs a series of operations, including Gaussian filtering for noise reduction, calculating gradient magnitude and direction, non-maximum suppression, and double-threshold hysteresis, ultimately outputting a binarized edge map. In this edge map, bright lines correspond to areas in the image where grayscale changes drastically. For document images, these areas mainly include: table borders, cell separators, upper and lower boundaries of text lines, paragraph breaks, and the outlines of illustrations. This edge map primarily extracts reliable structural information from clean, unoccluded areas of the document. Subsequently, this edge map is encoded as a conditional signal that the subsequent diffusion inpainting network can understand. During training and inference, this edge map is input as a structural condition to the structure-guided branch (such as ControlNet). This process ensures that the generation of the repair network is locked within the geometric framework defined by the edges of the original document, thereby forcing any newly generated pixel content in the mask area to be aligned with the existing table line extension direction and the text line baseline, ultimately generating edge map conditions. This provides an indispensable geometric blueprint for achieving visually seamless and structurally coherent repair.

[0038] 220. Based on the edge map conditions and the alpha mask, a controlled denoising process is performed in the latent space through the structure-guided latent diffusion repair network to generate an initial reconstruction result.

[0039] Specifically, this step is the core generation stage of the entire repair scheme. Within an efficient low-dimensional latent space, structural conditions and region masks are fused to perform controllable content synthesis. The network entity is based on a latent diffusion model (such as...). Figure 2As shown in the diagram, this model first uses a pre-trained variational autoencoder to compress a high-resolution image into a low-dimensional latent space, significantly reducing the computational complexity of the subsequent diffusion process. The denoising process takes place in a Markov chain that gradually evolves from pure random noise into the latent variables of the target image. The control in this step comes from two aspects: spatial region control, i.e., alpha mask, which explicitly indicates which regions in the latent space need to be regenerated (i.e., "repaired") and which regions need to be strictly preserved; and geometric structure control, i.e., edge map conditions. To achieve the latter, the network integrates ControlNet as a control branch. ControlNet is connected to the main UNet denoiser through zero-convolutional layers. In each denoising iteration, the edge map conditions are input into ControlNet, and the extracted multi-level structural features are progressively and adaptively injected into the corresponding decoding layers of the main UNet through zero-convolutional layers. This mechanism allows the denoising process to be aware of the geometric layout of the original document at each step. Therefore, combined with the region guidance of the mask, the denoising process of the diffusion model in the latent space is precisely constrained, generating new visual content only within specified regions based on a given structural blueprint. This process starts with complete noise, goes through dozens to hundreds of iterative denoising steps, and finally obtains a clean latent variable in the latent space corresponding to the repaired image. Then, the VAE decoder maps it back to the pixel space, thereby generating an initial reconstruction result that fills the mask area and strictly maintains the continuity with the surrounding structure.

[0040] 230. Based on the initial reconstruction results, perform biomimetic activation function processing to generate the preliminary repaired image.

[0041] Specifically, this step involves a fine-grained, non-linear enhancement of the reconstructed result tailored to the characteristics of the document image, aiming to recover and enhance subtle details that may have been over-smoothed during diffusion denoising. After the initial reconstruction result is generated in pixel space, its deeper features are further extracted and processed. In specific late-activation layers of the network, the system employs a biomimetic activation function instead of the traditional ReLU function, defined as: for input feature values Given a preset activation threshold and a scaling factor ,when When the function output is Maintain linear propagation of the strong activation signal; when When the function output is Its design principle originates from the simulation of the refractory period mechanism of biological neurons and is optimized for processing document images. In document restoration scenarios, the gradient changes in areas where stamps overlap with text, or the edges of strokes caused by light ink, are very weak, appearing as small negative or low positive values ​​in the feature space. Traditional ReLU functions directly truncate these signals below the threshold to zero, causing the loss of these subtle but crucial texture information, which visually manifests as thicker strokes, blurred edges, or loss of detail. In contrast, the biomimetic activation function effectively handles these signals below the threshold. The subthreshold signal, through an exponential function, provides a smooth, non-zero response, enabling the network to preserve and moderately amplify weak feature signals that characterize the subtle textures of paper, light ink marks, or stroke edges. By applying this function to the feature tensor of the initial reconstruction result, the network can effectively sharpen and enhance these easily lost details, thereby outputting a preliminary restored image that maintains overall structural coherence while offering richer and clearer local textures, significantly improving the visual refinement and realism of the restored result.

[0042] As an optional embodiment of this application, optionally, in step S300, based on the preliminary repaired image, a final repaired image is generated through an OCR-perceived loss-constrained enhancement network, including: 310. Based on the preliminary repaired image, the final repaired image is generated by performing super-resolution refinement and semantic consistency optimization processing through an enhanced network that integrates a moving window attention mechanism and OCR semantic consistency constraints.

[0043] Specifically, this step aims to improve the visual quality of the image and ensure the accuracy of its semantic content, representing the final optimization stage of the restoration process. The enhancement network is built on the Swin Transformer architecture, its core being the integration of a moving window attention mechanism for efficient processing of high-resolution images. The network receives the initially restored image as input and, during processing, first divides the image into multiple non-overlapping regular local windows. Within each window, the network performs standard self-attention computation, enabling the model to effectively model long-range dependencies between pixels within the window, thereby integrating local contextual information for detail enhancement and super-resolution reconstruction. To establish connections between different window regions, the network performs regular window movements between adjacent layers; for example, in the next layer, the window partition shifts by half a window size relative to the previous layer. This design makes non-overlapping windows in the previous layer overlap in the next layer, thus achieving cross-window information interaction. Through this hierarchical alternation of "intra-window computation—inter-window movement," the model can capture the global image context with linear computational complexity, making it ideal for fine-grained texture restoration and resolution enhancement of document images. Meanwhile, the entire network is driven by OCR semantic consistency constraints during the training phase. This makes the network's learning objective not only to minimize pixel-level errors, but more importantly, to minimize a predefined OCR perceptual loss. This loss forces the network to prioritize generating features beneficial to text recognition, such as clear stroke separation, correct character shapes, and stable text line layouts, when optimizing image visual quality. Therefore, this process is an end-to-end optimization driven by powerful global context modeling capabilities while incorporating semantic fidelity objectives, ultimately outputting a final restored image that significantly enhances both visual clarity and machine readability.

[0044] 320. During the optimization process of the enhanced network, the semantic consistency of its output is evaluated based on a predefined perceptual loss function.

[0045] Specifically, this step defines the core supervisory signals driving the augmentation network's semantic optimization training, ensuring that the repaired result aligns with the original document at a high-level semantic level, not just the pixel level. This process is activated during the network's training phase. The system pre-prepares an optical character recognition network (such as a CRNN or a Transformer-based recognizer) trained on a large, diverse text image dataset and freezes its weights. This OCR network acts as a fixed, general semantic feature extractor. In each training iteration of the augmentation network, the network's output image (predicted image) and the corresponding ground truth image (unobstructed target image) are simultaneously input into this frozen OCR network. Instead of using its final classification result, the system extracts feature maps from its deep intermediate layers. These feature maps encode high-level abstract information crucial for character recognition in the image, such as the topological structure of strokes, the spatial relationships of character components, and the overall shape of text lines. Let the feature map extracted from the ground truth image be... The feature map extracted from the predicted image is The semantic consistency is evaluated by calculating the squared Euclidean distance between the two high-dimensional feature maps, and the formula is as follows: This scalar value This is known as OCR perceptual loss, which quantifies the difference between the generated image and the real image in the OCR feature space. By using this loss function as part of the training objective and calculating the gradient through backpropagation, the update direction of the enhancement network parameters is guided. The network is thus forced to learn how to generate images that, in the eyes of the OCR network, have highly similar semantic features to the ground truth image. This ensures not only pixel-level restoration but also structural recognizability of the output text, achieving a leap from simply looking at the image to reading it correctly at the semantic level.

[0046] Furthermore, to ensure effective training across all the aforementioned steps, specific designs were implemented for data construction and training strategies. In terms of data construction, a high-fidelity synthetic data engine was built. This engine uses a stamp-free PDF document as the background and generated circular, elliptical, and square red stamps as foreground objects. To simulate realistic physical stamping effects, the engine abandons simple image overlay and instead employs a blending mode combining Alpha Blending and Multiply Blending to simulate the subtractive mixing optical properties of ink on paper. Specifically, the synthesis formula is defined as... The system incorporates ink penetration simulation, applying Gaussian blur and random noise to the seal edges to mimic ink diffusion within paper fibers. Furthermore, the data engine introduces aging and wear mechanisms, applying random morphological erosion and salt-and-pepper noise to seal images to simulate real-world seal degradation features such as ink loss, wear, or unclear imprinting, thus constructing a highly diverse and realistic training dataset. The training strategy employs a phased, progressive optimization approach. In the first phase, the Intelligent Mask Generation Network (IMG-Net) is independently pre-trained, utilizing the precise alpha channel from the synthesized data as a supervisory signal. Joint optimization using binary cross-entropy and Dice Loss enables it to peel seals from complex backgrounds. In the second phase, the structure-guided diffusion repair stage, the variational autoencoder (VAE) portion of the pre-trained Stable Diffusion model is frozen, and ControlNet pre-trained weights are loaded. Synthesized images with seals are used as conditional inputs, and images without seals are used as ground truth values ​​to fine-tune the U-Net backbone and ControlNet. In the third stage, end-to-end joint training is performed during the OCR semantic enhancement stage. The outputs of the first two stages are fed into the SwinIR network. During this process, the supervised network (CRAFT or CRNN) used to calculate the OCR perceptual loss is pre-trained on a large public document dataset (such as SynthText, ICDAR) with its parameters frozen to ensure it has general text feature extraction capabilities. In later training stages, the system will unlock some parameters from the second stage with a small learning rate. The joint fine-tuning is used to adapt to the gradient backpropagation of the OCR Loss.

[0047] Example 2 Based on the same principles as the aforementioned methods, a document restoration device based on a structure-guided diffusion model and OCR perceptual loss is also proposed, see [link to relevant documentation]. Figure 3 A document repair device 100 based on a structure-guided diffusion model and OCR perceived loss according to an embodiment of this disclosure includes: Markov attention segmentation module 110 is used to generate an alpha mask based on the input document image by using a segmentation network with embedded Markov attention mechanism; The structure-guided diffusion inpainting module 120 is used to generate a preliminary inpainted image based on the input document image and the alpha mask through a structure-guided latent diffusion inpainting network. The OCR perception enhancement and repair module 130 is used to generate a final repaired image based on the preliminary repaired image through an enhancement network constrained by OCR perception loss.

[0048] As an optional embodiment of this application, the structure-guided diffusion repair module 120 may further include: The structural edge extraction module 121 is used to perform document structural edge feature extraction based on the input document image and generate edge map conditions; The latent diffusion denoising module 122 is used to perform a controlled denoising process in the latent space through the latent diffusion model based on the edge map conditions and the Alpha mask to generate an initial reconstruction result. The biomimetic activation optimization module 123 is used to generate the preliminary repair image by processing the initial reconstruction result through a biomimetic activation function.

[0049] Obviously, those skilled in the art should understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the control methods described above. The modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Therefore, this application is not limited to any specific hardware and software combination.

[0050] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the control methods described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0051] Example 3 Furthermore, this application proposes an electronic device for implementing any of the document restoration methods based on the structure-guided diffusion model and OCR perceived loss, comprising: The processor is used to execute the complete computation and control process, starting from the input document image, generating an alpha mask through a segmentation network with an embedded Markov attention mechanism, generating a preliminary restored image through a structure-oriented latent diffusion restoration network, and finally generating the final restored image through an enhancement network constrained by OCR perceptual loss. The memory is used to store the pre-trained network model parameters necessary to implement the complete process, intermediate data during processing, and the final repaired image. The pre-trained network model parameters include at least the weight parameters of the segmentation network with embedded Markov attention mechanism, the weight parameters of the structure-oriented latent diffusion repair network, and the weight parameters of the OCR perceptual loss constraint enhancement network.

[0052] The electronic device of this disclosure includes a processor and a memory for storing processor-executable instructions. The processor is configured to implement, when executing the executable instructions, any of the document restoration methods based on the structure-guided diffusion model and OCR perceived loss described above.

[0053] It should be noted that the number of processors can be one or more. Furthermore, the electronic device in this embodiment may also include input devices and output devices. The processor, memory, input devices, and output devices can be connected via a bus or other means, without specific limitations herein.

[0054] The memory, serving as a computer-readable storage medium for automated fault handling and self-learning methods in modules, can be used to store software programs, computer-executable programs, and various modules, such as the program or module corresponding to the document restoration method based on the structure-guided diffusion model and OCR sensing loss in this embodiment of the present disclosure. The processor executes various functional applications and data processing of the electronic device by running the software programs or modules stored in the memory.

[0055] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A document restoration method based on a structure-guided diffusion model and OCR perceived loss, characterized in that, include: Based on the input document image, an alpha mask is generated by a segmentation network with an embedded Markov attention mechanism. Based on the input document image and the alpha mask, a preliminary repair image is generated through a structure-guided latent diffusion repair network; Based on the preliminary repaired image, the final repaired image is generated through an enhanced network with OCR-perceived loss constraints.

2. The document restoration method based on structure-guided diffusion model and OCR perceptual loss as described in claim 1, characterized in that, The step of generating an alpha mask based on the input document image using a segmentation network embedding a Markov attention mechanism includes: Based on the input document image, perform multimodal input tensor construction to enhance the feature separation between the region to be repaired and the background content; Based on the multimodal input tensor, a segmentation network with embedded Markov attention mechanism is used to perform probability state transition calculations for the region to be repaired, generating an initial segmentation result; Based on the initial segmentation result, continuous attention map iterative smoothing is performed to generate the alpha mask.

3. The document restoration method based on structure-guided diffusion model and OCR perceptual loss as described in claim 1, characterized in that, The step of generating a preliminary repair image based on the input document image and the alpha mask using a structure-guided latent diffusion inpainting network includes: Based on the input document image, perform document structure edge feature extraction to generate an edge map condition; Based on the edge map conditions and the alpha mask, a controlled denoising process is performed in the latent space through the structure-guided latent diffusion repair network to generate an initial reconstruction result; Based on the initial reconstruction results, a biomimetic activation function is executed to generate the preliminary repaired image.

4. The document restoration method based on structure-guided diffusion model and OCR perceived loss as described in claim 1, characterized in that, The step of generating a final restored image based on the preliminary restored image using an enhanced network with OCR-perceptual loss constraints includes: Based on the preliminary repaired image, the final repaired image is generated by performing super-resolution refinement and semantic consistency optimization processing through an enhanced network that integrates a moving window attention mechanism and OCR semantic consistency constraints. During the optimization process of the enhanced network, the semantic consistency of its output is evaluated based on a predefined perceptual loss function; The perceptual loss function The predicted feature map output by the enhanced network is calculated. The ground truth feature map of the real document Defined by the square of the Euclidean distance between them, specifically: 。 5. The document restoration method based on structure-guided diffusion model and OCR perceptual loss as described in claim 2, characterized in that, The step involves calculating the probabilistic state transition of the region to be repaired using a segmentation network embedded with a Markov attention mechanism based on the multimodal input tensor, generating an initial segmentation result, including: Based on the multimodal input tensor, the pixel state transition matrix is ​​constructed through the embedded Markov attention mechanism to model the probability propagation relationship of the region to be repaired between pixels; Based on the pixel state transition matrix and the initial attention map, perform matrix multiplication-based attention map iterative update to generate a smooth attention map; The attention map is iteratively updated, and the pixel state transition matrix is ​​denoted as... Matrix elements Represents pixels The probability of the area to be repaired per pixel The impact of the initial attention map , For the number of iterations, Indicates the first The attention map after the next iteration has the following iteration relationship: 。 6. The document restoration method based on structure-guided diffusion model and OCR perceptual loss as described in claim 3, characterized in that, The step of performing biomimetic activation function processing based on the initial reconstruction result to generate the preliminary repaired image includes: Based on the feature tensors in the initial reconstruction results, a biomimetic activation function is applied to the specified activation layer for processing. The biomimetic activation function is set as follows: For input feature values, The preset activation threshold, This is the scaling factor; When input value When, the output is ; When input value When, the output is .

7. The document restoration method based on structure-guided diffusion model and OCR perceived loss as described in claim 4, characterized in that, The step of generating the final restored image by performing super-resolution refinement and semantic consistency optimization processing on the preliminary restored image through an enhanced network integrating a moving window attention mechanism and OCR semantic consistency constraints includes: Based on the preliminary repaired image, self-attention computation is performed within non-overlapping local windows at multiple levels of the enhancement network; Regular movement of the local windows is performed between adjacent levels to achieve global information interaction across windows.

8. A document restoration device based on a structure-guided diffusion model and OCR perceived loss, the device comprising: The Markov attention segmentation module is used to generate an alpha mask based on the input document image by using a segmentation network that embeds a Markov attention mechanism. The structure-guided diffusion inpainting module is used to generate a preliminary inpainted image based on the input document image and the alpha mask through a structure-guided latent diffusion inpainting network. The OCR perception enhancement and repair module is used to generate the final repaired image based on the preliminary repaired image through an enhancement network constrained by OCR perception loss.

9. The document repair device based on structure-guided diffusion model and OCR perception loss according to claim 8, wherein the structure-guided diffusion repair module further comprises: The structural edge extraction module is used to extract document structural edge features based on the input document image and generate edge map conditions. The latent diffusion denoising module is used to perform a controlled denoising process in the latent space through the latent diffusion model based on the edge map conditions and the Alpha mask to generate an initial reconstruction result. The biomimetic activation optimization module is used to generate the preliminary repaired image by processing the initial reconstruction results through a biomimetic activation function.

10. An electronic device for implementing the document restoration method based on structure-guided diffusion model and OCR perceived loss as described in any one of claims 1 to 7, comprising: The processor is used to execute the complete computation and control process, starting from the input document image, generating an alpha mask through a segmentation network with an embedded Markov attention mechanism, generating a preliminary restored image through a structure-oriented latent diffusion restoration network, and finally generating the final restored image through an enhancement network constrained by OCR perceptual loss. The memory is used to store the pre-trained network model parameters necessary to implement the complete process, intermediate data during processing, and the final repaired image. The pre-trained network model parameters include at least the weight parameters of the segmentation network with embedded Markov attention mechanism, the weight parameters of the structure-oriented latent diffusion repair network, and the weight parameters of the OCR perceptual loss constraint enhancement network.

Images