Big model-oriented two-stage document image general repair method and system

Abstract

The invention discloses a double-stage document image general restoration method and system for a large model, and the method employs a double-stage restoration model for image restoration, and comprises a space correction stage and a pixel correction stage. In the spatial correction stage, the degraded image is processed through a double-branch structure of a layout branch and a text line branch; wherein the layout branch predicts a sparse two-dimensional mapping field and a three-dimensional coordinate grid to recover a global geometric structure, and the text line branch generates a local modulation signal based on text region features extracted by a pre-training text segmentation model; fusing global and local information, generating a dense two-dimensional mapping field, and outputting a space correction image by using reverse sampling operation; in a pixel correction stage, a spatial correction image and a gradient map thereof are used as conditions to be input into a denoising diffusion implicit model architecture to gradually recover pixel values. According to the method, general repair of various degradation problems is completed through double-stage repair, so that the content reading capability of various large models on document images is improved.

Description

Technical Field

[0001] This invention belongs to the technical field of image processing and restoration, and relates to a document image restoration technology. It mainly relates to a two-stage general document image restoration method and system for large models. Background Technology

[0002] With the widespread adoption of mobile devices and the rapid development of large-scale model technology, capturing document images using mobile phone cameras and then having them analyzed by large-scale language models (LLMs) or multimodal large-scale models (MLLMs) has become a common practice in daily office work, education, and research. However, digital document images captured in real-world scenarios are often affected by various forms of degradation, including text distortion and spatial misalignment caused by physical deformation due to paper bending or tilting, and problems such as shadows, overexposure, underexposure, blurring, and noise caused by uneven lighting. These degradations not only affect the human visual experience but also severely restrict the downstream large-scale models' ability to understand and analyze document content, posing a significant challenge to fields such as finance, law, and medicine that rely on accurate document understanding. Therefore, developing a document image restoration method specifically designed to enhance the reading capabilities of large-scale models is of urgent practical significance.

[0003] Currently, existing technologies mainly fall into two categories: task-specific document augmentation methods and unified document restoration frameworks. Regarding task-specific methods, techniques for handling spatial deformation primarily utilize deep neural networks to predict pixel deformation fields and correct geometric distortions through grid sampling. Some methods attempt to introduce text line features as additional guidance to enhance the correction effect in text regions. For pixel perturbations, existing methods such as shadow removal, illumination correction, and appearance enhancement typically focus only on specific degradation types, such as shadows, low contrast, bleed-through, or blur, and cannot handle complex scenes where multiple pixel perturbations coexist. These task-specific methods require training and deploying multiple independent models for different degradation types, which is not only resource-intensive but also struggles to cope with situations where document images simultaneously contain multiple degradations in real-world scenarios, and cannot perform collaborative optimization based on the input characteristics of large models.

[0004] In terms of unified restoration frameworks, recent studies have attempted to handle complex degradation patterns within a unified framework. These methods attempt to simultaneously address geometric distortion and illumination correction, achieving fine-grained adjustments by segmenting the entire image into small patches during testing. More recent methods consider utilizing dynamic cues to handle five different enhancement tasks within a single model with shared weights. However, these methods still have significant limitations: on the one hand, they often require multiple iterations of enhancement during inference and rely on manual specification of specific degradation types, severely limiting their convenience and automation in practical applications; on the other hand, they fail to systematically analyze and solve degradation problems from the essence of digital images, have limited ability to handle complex degradation patterns involving both spatial deformation and pixel perturbations, and completely ignore the impact of restoration results on the readability of large models. Particularly noteworthy is that the evaluation criteria of existing technologies mainly focus on visual quality metrics such as PSNR and SSIM, rather than the accuracy of large models in understanding the restored document. This means that even visually good restoration results may not effectively improve the readability of large models.

[0005] Furthermore, the lack of high-quality training data also hinders the development of document restoration techniques optimized for large models. Existing datasets either contain only a single type of degradation or lack diversity in the coupling of spatial deformation and pixel perturbations. More importantly, the lack of specialized datasets that use the improvement of large model reading capabilities as an evaluation criterion makes it difficult for researchers to systematically optimize restoration algorithms to adapt to the input requirements of large models.

[0006] In summary, the main problems faced by existing technologies include: (1) a lack of systematic classification and unified processing framework for spatial deformation and pixel perturbation; (2) an inability to effectively cope with the complex situation of multiple degradations coexisting in real-world scenarios; (3) a repair process that relies on multiple iterations and manual intervention, making it difficult to meet the real-time processing needs of large models; and (4) existing methods neglect the targeted optimization of repair results for the reading ability of large models, resulting in a serious disconnect between the evaluation system and the needs of large models. Therefore, there is an urgent need for a general repair method that can comprehensively and efficiently handle various types of document degradation and is specifically optimized to improve the reading ability of large models. Summary of the Invention

[0007] This invention addresses the problems existing in existing technologies by providing a two-stage universal document image restoration method and system for large models. It employs a two-stage restoration model for image restoration, including a spatial correction stage and a pixel correction stage. In the spatial correction stage, degraded images are processed through a two-branch structure of layout branch and text line branch. Specifically, the layout branch predicts a sparse two-dimensional mapping field and a three-dimensional coordinate grid to restore the global geometric structure, while the text line branch generates local modulation signals based on text region features extracted by a pre-trained text segmentation model. Global and local information are fused to generate a dense two-dimensional mapping field, and a spatially corrected image is output using backsampling. In the pixel correction stage, the spatially corrected image and its gradient map are used as conditional inputs to a denoising diffusion implicit model architecture to gradually restore pixel values. This invention achieves universal restoration of various degradation problems through two-stage restoration, thereby improving the content reading capabilities of various large models for document images.

[0008] To achieve the above objectives, the technical solution adopted by this invention is: a two-stage general document image restoration method for large models, which employs a two-stage restoration model for image restoration, including at least a spatial correction stage and a pixel correction stage.

[0009] The spatial correction stage involves processing the degraded image through a dual-branch spatial correction process. The dual branches include a layout branch and a text line branch. The layout branch is used to predict a sparse two-dimensional mapping field and a three-dimensional coordinate grid to recover the global geometric structure, while the text line branch is used to extract text region features and generate local modulation signals. The global and local information are fused to generate a dense two-dimensional mapping field, and the spatially corrected image is output using a backsampling operation.

[0010] In the pixel correction stage, the spatially corrected image and its gradient map output from the spatial correction stage are used as conditional inputs to the denoising diffusion implicit model, and a frequency domain analysis mechanism is introduced to perform pixel correction.

[0011] The models for the spatial correction stage and the pixel correction stage are trained independently on their respective preprocessed datasets and can be used interchangeably or in combination.

[0012] As an improvement of the present invention, the repair in the spatial correction stage includes the following steps:

[0013] S101: Input degraded document image Feature extraction via CNN encoder The extracted features The input is processed simultaneously into the layout branch and the text line branch;

[0014] S102: In the layout branch, the receptive field is enhanced by an inflated pyramid extractor, and sparse 2D mapping fields are predicted using a 2D decoder. And 3D decoder predicts sparse 3D mesh The 3D mesh represents the three-dimensional coordinates of each control point in the degraded image;

[0015] S103: In the text line branch, use a pre-trained text segmentation model to obtain text segmentation features. Text segmentation features are obtained through an attention mechanism. Features extracted in step S101 Integration, specifically:

[0016] ;

[0017] Where SA represents self-attention layer, CA represents cross-attention layer, and FFN represents feedforward network;

[0018] After optimizing the features using N self-attention decoders, the modulation features are obtained. The processing procedure for the j-th decoder is as follows:

[0019] ;

[0020] For sparse 2D mapping fields Perform bilinear interpolation upsampling, and then utilize the modulation features. Weighted modulation of the upsampled mapping field will transform the sparse 2D mapping field. With modulation characteristics Fusion, generating a dense 2D mapping field ;

[0021] S104: Perform a backsampling operation on the input image. Apply the dense 2D mapping field obtained in step S103 Generate spatially corrected images .

[0022] As another improvement of the present invention, the repair in the pixel correction stage includes the following steps:

[0023] S201: The image output from the spatial correction stage is processed using the Sobel operator. Calculate its gradient plot ;

[0024] S202: Transfer image With gradient plot As a conditional input based on a denoised diffusion implicit model architecture, a diffusion sampling sequence is defined during inference. Total number of steps , Set the original diffusion steps; initialize the noisy image. The noise is gradually reduced through the following iterative process:

[0025] ;

[0026] in For U-Net networks, Indicates model parameters.

[0027] As another improvement of the present invention, in the spatial correction stage, the model training loss function It consists of an L1 loss weighted sum of sparse two-dimensional mapping fields, three-dimensional meshes, and dense mapping fields, specifically:

[0028]

[0029] in Represents the predicted sparse 2D mapping field. Represents the predicted sparse 3D mesh. This represents the predicted dense 2D mapping field.

[0030] As another improvement of the present invention, in the pixel correction stage, the loss function of model training... It is composed of a weighted average of pixel-level L1 loss and temporal frequency switching loss, specifically:

[0031]

[0032] in This is the output image from the pixel correction stage. To compensate for the temporal frequency switching loss, a Fourier decoupler is used to separate the corrected image into low-frequency components. and high-frequency components And according to the diffusion time step Dynamically adjust weights:

[0033]

[0034]

[0035] in Indicates the total number of diffusion steps. and These correspond to the actual low-frequency and high-frequency components, respectively. Weights for low-frequency components. Weights for high-frequency components.

[0036] To achieve the above objectives, the present invention also adopts the following technical solution: a two-stage universal document image restoration system for large models, comprising at least a spatial correction module and a pixel correction module.

[0037] The spatial correction module is a dual-branch spatial correction module that repairs images using a two-stage repair model. The dual branches include a layout branch and a text line branch. The layout branch predicts a sparse two-dimensional mapping field and a three-dimensional coordinate grid to recover the global geometric structure. The text line branch generates local modulation signals based on text region features extracted by a pre-trained text segmentation model. In the spatial correction module, global and local information are fused to generate a dense two-dimensional mapping field, and a spatially corrected image is output using backsampling. The spatial correction module employs L1 loss to jointly optimize the prediction accuracy of the sparse two-dimensional mapping field, the three-dimensional grid, and the dense mapping field.

[0038] The pixel correction module takes the spatially corrected image and its gradient map as input conditions to perform pixel correction. The pixel correction module is based on a denoising diffusion implicit model architecture. The total loss is composed of a weighted sum of pixel-level L1 loss and temporal frequency switching loss. The temporal frequency switching loss separates the high and low frequency components of the image through a Fourier decoupler and dynamically activates the corresponding frequency band constraint according to the current diffusion time step.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) This invention proposes a two-stage document image restoration method and system, which systematically divides complex degradation into two types of problems: spatial deformation and pixel perturbation. By processing spatial correction first and then pixel correction, the interference and amplification effects of single-stage methods in processing composite degradation are avoided, significantly improving the stability and effect of restoration. At the same time, it provides the downstream large model with geometrically correct and visually high-quality input.

[0041] (2) The present invention designs a dual-branch spatial correction module, which predicts global 2D and 3D coordinates through layout branches to restore the overall geometric structure of the document. At the same time, it uses text line branches combined with text segmentation features to accurately correct local text region distortion. The two are adaptively fused through a weighted upsampling strategy, which solves the problem of coexistence of global and local spatial distortion and greatly improves the readability of text regions.

[0042] (3) The method of the present invention introduces a temporal frequency switching loss function in the pixel correction module. Based on the dynamic characteristics of the diffusion model sampling process, it adaptively adjusts the attention weight of high and low frequency components at different stages. In the first half stage, it prioritizes the restoration of low frequency background structure, and in the second half stage, it focuses on enhancing high frequency text details, effectively maintaining the natural appearance of the document and significantly improving the text clarity.

[0043] (4) The entire framework of the system of the present invention has highly modular characteristics. The two stages can be trained and replaced independently without the need for manual specification of degradation type or multiple iterations. It realizes "one-click" universal repair, greatly improves practicality and deployment efficiency, significantly improves the reading accuracy of various large models for complex degraded documents, and can also enhance the geometric consistency and visual quality of document images, providing reliable technical support for document intelligent processing systems. Attached Figure Description

[0044] Figure 1 This is a flowchart of the method of the present invention;

[0045] Figure 2 This is a diagram of the spatial correction module architecture in the method of the present invention. Detailed Implementation

[0046] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention.

[0047] Example 1

[0048] A two-stage general document image restoration system for large models, comprising at least a spatial correction module and a pixel correction module.

[0049] The spatial correction module: such as Figure 2 As shown, this is a dual-branch spatial correction module. The dual branches include a layout branch and a text line branch. The layout branch is used to predict the sparse two-dimensional mapping field and the three-dimensional coordinate grid to recover the global geometry. The text line branch generates a local modulation signal based on the text region features extracted by the pre-trained text segmentation model. In the spatial correction module, global and local information are fused to generate a dense two-dimensional mapping field, and a spatially corrected image is output using a backsampling operation. The spatial correction module uses L1 loss to jointly optimize the prediction accuracy of the sparse two-dimensional mapping field, the three-dimensional grid, and the dense mapping field.

[0050] The pixel correction module takes the spatially corrected image and its gradient map as input conditions to perform pixel correction. The pixel correction module is based on a denoising diffusion implicit model architecture. The total loss is composed of a weighted sum of pixel-level L1 loss and temporal frequency switching loss. The temporal frequency switching loss separates the high and low frequency components of the image through a Fourier decoupler and dynamically activates the corresponding frequency band constraint according to the current diffusion time step.

[0051] A two-stage general document image restoration method for large models, such as Figure 1As shown, using the above system, for complex degraded document images containing spatial deformations such as distortion, overexposure, underexposure, blurring, noise and other pixel disturbances, a two-stage repair process is used to complete the general repair of various degradation problems, thereby improving the ability of various large models to read the content of document images.

[0052] First, the VCDoc940 dataset was collected and created. Image pairs containing multiple pixel perturbations but without significant spatial distortion were selected from real-world document images. After eliminating geometric distortion using spatial alignment tools and combining this with manual verification, a dataset of 940 high-quality degraded-clean image pairs was formed. The constructed VCDoc940 dataset specifically focuses on the pixel perturbation problem. Through rigorous selection and spatial calibration, it provides 940 high-quality degraded-clean image pairs, providing a crucial data foundation for training pixel correction models optimized for large models.

[0053] Real-world document images were selected from the WarpDoc dataset as the primary resource. This dataset contains various coupled degradation modes and their corresponding high-quality reference images. The DocAligner tool was then used to eliminate spatial distortion in the selected images, ensuring that only pixel perturbation degradation was retained. After spatial alignment, further manual verification and cropping were performed, rigorously selecting 699 images with good spatial geometry. Simultaneously, 235 images with negligible geometric distortion and background cropping and resizing were selected from the RealDAE dataset. These images contain various pixel perturbations such as shadows and color differences but without significant spatial deformation. These two sets of images were merged to form the final VCDoc940 dataset, consisting of 940 pairs of document images covering various document types, including scientific papers, magazines, and envelopes. Each pair of images includes a degraded image and a corresponding clean reference image. All images were manually verified to ensure spatial geometric correctness and contain a rich variety of pixel perturbation types, including but not limited to shadows, overexposure, underexposure, blur, and noise, providing a high-quality data foundation for training the pixel correction module.

[0054] Then, various image datasets are preprocessed. The method of this invention uses a two-stage inpainting model for image inpainting, which includes at least a spatial correction stage and a pixel correction stage. Therefore, various image datasets are preprocessed for training the model at each stage.

[0055] For the spatial correction stage, image pairs with complex spatial deformations are generated using synthetic datasets. For example, the Doc3D and UVDoc datasets are merged to generate a synthetic training set containing 120,000 image pairs. Spatial distortion simulation is performed on each image to uniformly adjust the input image size to 672×448 pixels, ensuring that the training data covers diverse spatial deformation modes such as perspective distortion, curve bending, and folding.

[0056] For the pixel correction stage, the VCDoc940 dataset is used to ensure that the spatial geometry of the input image has been corrected, retaining only the pixel perturbation components. Spatial geometry verification is performed on each image, removing samples with significant spatial distortion. The images are then cropped into 288×288 pixel training blocks. Simultaneously, grayscale conversion and gradient calculation are used to extract the corresponding gradient maps as auxiliary conditions to balance the sampling of pixel perturbation types, ensuring that various degradation modes such as shadows, overexposure, underexposure, blur, and noise are evenly distributed in the training set.

[0057] During the validation phase, the DocUNet and DocReal benchmark datasets were quality-screened, and 20 images with unresolved backlighting and color shift issues were removed. The final test set consisted of 128 DocUNet images and 192 DocReal images. All test images were kept at their original resolution to evaluate the model's generalization ability in real-world scenarios.

[0058] The method of this invention employs a two-stage restoration model for image restoration, including a spatial correction stage and a pixel correction stage. In the spatial correction stage, the input degraded image is processed by a dual-branch spatial correction module (SRM). First, an encoder extracts features from the low-quality document image containing the background. Then, the layout branch in the dual-branch spatial correction module predicts a sparse two-dimensional mapping field and a three-dimensional coordinate grid to restore the global geometric structure. The text line branch generates a local modulation signal based on text region features extracted by a pre-trained text segmentation model. A weighted upsampling module fuses global and local information to generate a dense two-dimensional mapping field, and a backsampling operation is used to output the spatially corrected image. In the pixel correction stage, the spatially corrected image and its gradient map are used as conditional inputs to the pixel correction module (VCM), which progressively restores pixel values ​​based on a denoising diffusion implicit model (DDIM) architecture.

[0059] Specifically, such as Figure 2 As shown, in the layout branch of the spatial correction stage, the input complex degraded document image is first processed. The image is a schematic diagram of an English document, where the spatial shape of the document image and the text exhibit significant distortion. Features are extracted using a CNN encoder. Secondly, the extracted features Simultaneously, the inputs are processed in both the layout branch and the text line branch; in the layout branch, the receptive field is enhanced by an inflated pyramid extractor, and then a sparse 2D mapping field is predicted by a 2D decoder. And 3D decoder predicts sparse 3D mesh The 3D mesh represents the three-dimensional coordinates of each control point in the degraded image, which is used to enhance the model's understanding of the document's geometric deformation.

[0060] In the text line branch, a pre-trained text segmentation model is first used to obtain text segmentation features. Then, through the attention mechanism, and The process of fusion can be represented as follows:

[0061]

[0062] Where SA represents self-attention layer, CA represents cross-attention layer, and FFN represents feedforward network;

[0063] After further optimization of the features by two self-attention decoders, the modulation features are finally obtained. The processing procedure for the j-th decoder is as follows:

[0064] .

[0065] The sparse 2D mapping field is obtained through a weighted upsampling module. With modulation characteristics Fusion: First to Perform 8x bilinear interpolation upsampling to Resolution, and then utilize The upsampled mapping field is weighted and modulated to generate a dense 2D mapping field. .

[0066] Finally, the input image is processed by backsampling. application Generate spatially corrected images .like Figure 2 As shown, the document region in the image undergoes significant spatial correction. Through the aforementioned dual-branch collaborative mechanism, this spatial correction stage can simultaneously handle the spatial deformation of the global layout structure and the local text region, providing geometrically correct input for the subsequent pixel correction stage.

[0067] In the pixel correction stage, the image output from the spatial correction stage is... Calculate its gradient plot The gradient map, extracted using the Sobel operator, effectively suppresses background information and enhances the text region contours; subsequently, and As a Conditional Input Correction Module (VCM), this module is based on a denoising diffusion implicit model architecture; during inference, a diffusion sampling sequence is defined. Total number of steps , Set the original diffusion steps; initialize the noisy image. The noise is gradually reduced through the following iterative process:

[0068]

[0069] in For U-Net networks, Indicates model parameters.

[0070] During the model training process in the two stages mentioned above, the spatial correction module uses L1 loss to jointly optimize the prediction accuracy of sparse two-dimensional mapping fields, three-dimensional meshes, and dense mapping fields; the total loss of the pixel correction module is composed of pixel-level L1 loss and temporal frequency switching loss, wherein the frequency switching loss separates the high and low frequency components of the image through a Fourier decoupler and dynamically activates the corresponding frequency band constraints according to the current diffusion time step; the two-stage modules are trained independently on the corresponding preprocessing datasets, supporting module-level replacement and combination, forming an end-to-end but decoupled general repair framework.

[0071] First, the synthetic datasets Doc3D and UVDoc were merged to form a training set containing 120,000 pairs of images. All input images were uniformly adjusted to a resolution of 672×448 pixels to ensure that the training data covered various spatial deformation modes such as perspective distortion, curve bending and folding.

[0072] Define corresponding supervision labels for each component in the spatial correction module, including the true labels for the sparse 2D mapping field. Realistic labels for sparse 3D meshes and the real labels of dense 2D mapping fields The Adam optimizer is used for parameter updates, with an initial learning rate set at 0.5%. The batch size is 16, and a learning rate decay strategy is applied, multiplying the learning rate by a coefficient of 0.1 when the training reaches a preset number of epochs; the training loss function consists of three weighted L1 loss components:

[0073]

[0074] in Represents the predicted sparse 2D mapping field. Represents the predicted sparse 3D mesh. This represents the predicted dense 2D mapping field.

[0075] During training, the text line branch uses a pre-trained text segmentation model to extract text region features from the input image and fuses them with image features through self-attention and cross-attention mechanisms. The number of self-attention layers N is set to 2, which enables the model to simultaneously optimize the global layout structure and the spatial correction capability of local text regions.

[0076] For training the pixel correction module, the VCDoc940 dataset was first used as the training resource. Each spatially corrected image was cropped into a 288×288 pixel training block, and the corresponding gradient map was calculated as an auxiliary condition. Then, the Adam optimizer was used to optimize the parameters, with an initial learning rate of 100%. The batch size is 16; the total loss function of the pixel correction module is composed of a weighted average of pixel-level L1 loss and temporal frequency switching loss.

[0077]

[0078] The temporal frequency switching loss is separated into low-frequency components of the corrected image through a Fourier decoupler. and high-frequency components And according to the diffusion time step Dynamically adjust weights:

[0079]

[0080]

[0081] in Indicates the total number of diffusion steps. and These correspond to the actual low-frequency and high-frequency components, respectively.

[0082] In terms of training strategy, the temporal frequency switching loss is only activated in the second half of the training phase, allowing the model to first master basic pixel repair capabilities and then optimize frequency-aware characteristics; during training, the gradient map is calculated using the Sobel operator and used as a conditional input to guide the network to focus on text regions.

[0083] Test case

[0084] To demonstrate the effectiveness and superiority of the method of this invention, it was tested on the DocUNet and DocReal datasets. The DocUNet and DocReal datasets are two representative real-world benchmark datasets in the field of document image restoration. Since most existing methods handle image distortion and low-quality images separately, this test case compares the method of this invention with representative methods in various related fields. DocGeoNet and FTDR are image distortion removal methods, while GCDRNet, UDoc-GAN, and DocDiff are image surface enhancement methods. The method of this invention, combined with various models, was fed into the Qwen3-VL-2B multimodal large model for reading the restored document images. The difference between the recognized text and the results of real high-quality document images was determined using two metrics: Edit Distance (ED) and Character Error Rate (CER). The results are shown in Table 1 below.

[0085] Table 1

[0086]

[0087] Table 1 shows the evaluation results of the reading ability of the proposed method and existing mainstream document correction methods on two authoritative datasets, DocUNet and DocReal, including different combinations of DocGeoNet and FTDR, as well as the DocRes method. The evaluation selected edit distance (ED) and character error rate (CER) as the core indicators of the reading ability of the large model. The lower the values ​​of these two indicators, the more accurate the text recognition of the corrected document image and the higher the semantic restoration degree. As can be seen from Table 1, on the DocUNet dataset, the proposed method exhibits extremely high accuracy, with an ED value as low as 386.04, superior to DocRes' 391.02, and significantly lower than other combinations such as DocGeoNet+UDoc-GAN (490.27) and FTDR+UDoc-GAN (514.52). Regarding the CER indicator, the proposed method's indicator is 0.1754, which is basically on par with the best DocRes indicator of 0.1742, maintaining its performance in the first tier.

[0088] More notably, the robustness of this invention is fully demonstrated on the more challenging DocReal dataset, which contains complex real-world scenarios. This invention significantly reduces the ED value to 242.85, more than 20 points lower than the second-best DocGeoNet+DocDiff (265.14), and nearly 100 points lower than DocRes (340.27). Simultaneously, the CER value of this invention is only 0.2518, the only method in the table to break through 0.26, significantly better than DocRes' 0.3462 and the best-performing FTDR series method (0.2724). This indicates that this invention can better preserve the geometric structure and semantic information of the text when dealing with complex distortions and real-world interference. Therefore, the overall performance of this invention on both datasets is optimal compared to other methods and combinations, especially demonstrating a significant leading advantage in real-world complex scenarios, effectively improving the reading accuracy of downstream large-scale models.

[0089] The method of this invention, combined with various integrated models, was directly compared with real high-quality document images after restoration. Regarding geometric consistency, three commonly used metrics were used: Multi-Scale SSIM (MS-SSIM), Local Distortion (LD), and Aligned Distortion (AD). Regarding image quality, two commonly used metrics were used: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). The test results are shown in Table 2 below.

[0090] Table 2

[0091]

[0092] Table 2 presents the quantitative results of a direct comparison between the method of this invention and various mainstream combined models on the repaired document images and real high-quality document images, including different variants of DocGeoNet and FTDR, as well as the DocRes method. The evaluation is divided into two dimensions: geometric consistency and image quality. Geometric consistency includes MS-SSIM, LD, and AD metrics, while image quality includes PSNR and SSIM metrics, comprehensively measuring the smoothness and sharpness of the document correction. As shown in Table 2, this invention demonstrates significant performance advantages on both authoritative datasets. On the DocUNet dataset, the MS-SSIM of this invention is improved to 0.5803, far exceeding DocRes' 0.4736 and other combined models; simultaneously, the local distortion LD and alignment distortion AD, representing the degree of distortion, are significantly reduced to 6.1814 and 0.3113, respectively, significantly better than DocRes' 9.4692 and 0.4638, proving the accuracy of this invention in geometric correction.

[0093] In terms of image quality, the PSNR of this invention reaches 12.76, and the SSIM is reduced to 0.3251, both of which are the best in the group. The robustness of this invention is further verified on the more challenging DocReal dataset. Its LD value of 9.9674 is the only method in the table that is controlled below 10, significantly reducing distortion error compared to DocGeoNet+UDoc-GAN's 12.4542 and the highest value of the FTDR series 14.3886. Simultaneously, the PSNR value of this invention is improved to 13.87, MS-SSIM reaches 0.5826, AD is reduced to 0.2573, and SSIM is reduced to 0.2926. These five key indicators comprehensively outperform all comparison methods, including DocRes (LD: 11.8857, PSNR: 13.74), demonstrating that this invention can not only more perfectly recover document geometry when dealing with real-world complex scenes, but also preserve image details to the greatest extent. Therefore, the combined effect of this invention on the two datasets is optimal compared to other methods and combinations, fully demonstrating its dual effectiveness in geometric restoration and image enhancement.

[0094] In summary, this invention discloses a two-stage general document image restoration method and system for improving the reading ability of large models. It designs a dual-branch spatial correction module, which predicts the restored 2D and 3D coordinates of the degraded input image through layout branches to accurately restore the global spatial geometry. Simultaneously, the text line branch uses text segmentation maps to correct distortions within local text regions. A weighted upsampling strategy adaptively fuses the two outputs, thereby collaboratively addressing global and local spatial distortions. A pixel correction module is constructed using a diffusion model to improve visual quality and maintain the natural appearance of the document. Furthermore, considering the dynamic attention characteristics of the diffusion model to different frequency components during sampling, a temporal frequency switching loss function is proposed to provide precise guidance for the pixel correction module at different sampling stages. This invention significantly improves the reading ability of large models for complex degraded documents, highlighting its robustness in handling complex degradation patterns and its versatility in enhancing the capabilities of large models.

[0095] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A two-stage general document image restoration method for large models, characterized by: Image restoration employs a two-stage restoration model, which includes at least a spatial correction stage and a pixel correction stage. The spatial correction stage involves processing the degraded image through a dual-branch spatial correction process. The dual branches include a layout branch and a text line branch. The layout branch is used to predict a sparse two-dimensional mapping field and a three-dimensional coordinate grid to recover the global geometric structure, while the text line branch is used to extract text region features and generate local modulation signals. The global and local information are fused to generate a dense two-dimensional mapping field, and the spatially corrected image is output using a backsampling operation. In the pixel correction stage, the spatially corrected image and its gradient map output from the spatial correction stage are used as conditional inputs to the denoising diffusion implicit model, and a frequency domain analysis mechanism is introduced to perform pixel correction. The models for the spatial correction stage and the pixel correction stage are trained independently on their respective preprocessed datasets and can be used interchangeably or in combination.

2. The two-stage document image general restoration method for large models as described in claim 1, characterized in that: The repair process during the spatial correction phase includes the following steps: S101: Input degraded document image Feature extraction via CNN encoder The extracted features The input is processed simultaneously into the layout branch and the text line branch; S102: In the layout branch, the receptive field is enhanced by an inflated pyramid extractor, and sparse 2D mapping fields are predicted using a 2D decoder. And 3D decoder predicts sparse 3D mesh The 3D mesh represents the three-dimensional coordinates of each control point in the degraded image; S103: In the text line branch, use a pre-trained text segmentation model to obtain text segmentation features. Text segmentation features are obtained through an attention mechanism. Features extracted in step S101 Integration, specifically: ； Where SA represents self-attention layer, CA represents cross-attention layer, and FFN represents feedforward network; After optimizing the features using N self-attention decoders, the modulation features are obtained. The processing procedure for the j-th decoder is as follows: ； sparse 2D mapping field With modulation characteristics Fusion, generating a dense 2D mapping field ; S104: Perform a backsampling operation on the input image. Apply the dense 2D mapping field obtained in step S103 Generate spatially corrected images .

3. The two-stage document image general restoration method for large models as described in claim 2, characterized in that: In step S103, the sparse 2D mapping field is... With modulation characteristics Fusion, generating a dense 2D mapping field The specific method is as follows: first, for the sparse 2D mapping field... Perform bilinear interpolation upsampling, and then utilize the modulation features. The upsampled mapping field is then weighted and modulated.

4. The two-stage document image general restoration method for large models as described in claim 1, characterized in that: The repair process during the pixel correction stage includes the following steps: S201: The image output from the spatial correction stage is processed using the Sobel operator. Calculate its gradient plot ; S202: Transfer image With gradient plot As a conditional input based on a denoised diffusion implicit model architecture, a diffusion sampling sequence is defined during inference. Total number of steps , Set the original diffusion steps; initialize the noisy image. The noise is gradually reduced through the following iterative process: ； in For U-Net networks, Indicates model parameters.

5. The two-stage document image general restoration method for large models as described in claim 1, characterized in that: In the spatial correction phase, the model training loss function It consists of an L1 loss weighted average of sparse two-dimensional mapping fields, three-dimensional meshes, and dense mapping fields, specifically: ； in Represents the predicted sparse 2D mapping field. Represents the predicted sparse 3D mesh. This represents the predicted dense 2D mapping field.

6. The two-stage document image general restoration method for large models as described in claim 5, characterized in that: In the pixel correction stage, the loss function of model training It is composed of a weighted average of pixel-level L1 loss and temporal frequency switching loss, specifically: ； in This is the output image from the pixel correction stage. To compensate for the temporal frequency switching loss, a Fourier decoupler is used to separate the corrected image into low-frequency components. and high-frequency components And according to the diffusion time step Dynamically adjust weights: ； ； in Indicates the total number of diffusion steps. and These correspond to the actual low-frequency and high-frequency components, respectively. Weights for low-frequency components. Weights are assigned to high-frequency components.

7. A two-stage universal document image restoration system for large models, characterized in that: It includes at least a spatial correction module and a pixel correction module. The spatial correction module is a dual-branch spatial correction module, comprising a layout branch and a text line branch. The layout branch predicts the sparse two-dimensional mapping field and the three-dimensional coordinate grid to recover the global geometric structure, while the text line branch generates a local modulation signal based on text region features extracted by a pre-trained text segmentation model. The spatial correction module fuses global and local information to generate a dense two-dimensional mapping field and outputs a spatially corrected image using backsampling. The spatial correction module employs L1 loss to jointly optimize the prediction accuracy of the sparse two-dimensional mapping field, the three-dimensional grid, and the dense mapping field. The pixel correction module takes the spatially corrected image and its gradient map as input conditions to perform pixel correction. The pixel correction module is based on a denoising diffusion implicit model architecture. The total loss is composed of a weighted sum of pixel-level L1 loss and temporal frequency switching loss. The temporal frequency switching loss separates the high and low frequency components of the image through a Fourier decoupler and dynamically activates the corresponding frequency band constraint according to the current diffusion time step.

Images