Layout-guided document image adaptive enhancement method

Abstract

The invention discloses a layout-guided document image adaptive enhancement method, and mainly solves the problems of poor readability, loss of detail structures and poor matching performance of repaired texts due to neglecting of internal structural semantic differences of document images in the prior art. Comprising the following steps: 1) inputting a low-resolution LR document image, and obtaining a semantic segmentation mask graph M based on a deep learning detection segmentation technology; 2) performing feature extraction and up-sampling operation on the LR image by using an existing generator to obtain an intermediate feature; 3) constructing an L-SPADE module as a conditional feature modulator, generating a spatial adaptive modulation parameter graph by using M through network learning, and performing affine transformation on a normalized feature graph; and 4) training the image restoration generator embedded with the L-SPADE module, and finally obtaining a restored image.According to the method, the document image restoration precision can be improved, the restoration effect is effectively aligned with a downstream task, and the overall efficiency of document digital processing is remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and further relates to document image restoration and enhancement technology. Specifically, it is a layout-guided adaptive enhancement method for document images, which can be used for the digital restoration of old documents such as ancient books and archives, as well as for improving the quality of document photography on mobile devices. Background Technology

[0002] With the increasing demand for document digitization, recovering clear, readable, high-quality images from low-quality, degraded scanned or photographed document images has become a crucial step in information extraction and preservation. Document images have characteristics that significantly distinguish them from natural images: the content is mainly structured text, including semantically clear areas such as tables, titles, and illustrations, and the fidelity requirements for details such as character strokes and line edges are extremely high.

[0003] Current mainstream image super-resolution and restoration models are primarily designed for natural images, optimizing pixel-level similarity metrics (such as PSNR and SSIM) through deep learning networks. When applied to document images, these models often encounter problems such as blurred or overlapping text strokes, broken table lines, and imbalanced restoration intensity across different semantic regions. This results in restorations that appear "clear" but are "difficult to read," severely impacting the accuracy of subsequent tasks such as Optical Character Recognition (OCR). Patent application CN121073805A, entitled "An Adaptive Image Enhancement Method," discloses an adaptive image enhancement method based on a combination of an incomplete beta function and an improved martial arts learning optimization algorithm. This method dynamically selects optimization strategies through reinforcement learning, achieving automated and intelligent optimization of image enhancement parameters. While this method can improve image enhancement to some extent, its reliance on a global grayscale transformation function (incomplete beta function) essentially makes it a traditional grayscale mapping method. It lacks the ability to model image structure and semantic content, and it is not specifically optimized for the core document restoration goal of "improving text readability." Furthermore, the modulation strategy may not be suitable for characters and table structures that are extremely sensitive to detail.

[0004] Existing methods generally suffer from several drawbacks: image restoration models ignore the semantic structural differences within a document, applying the same restoration parameters to all regions and failing to focus on enhancing key areas such as text and tables; model optimization targets are mostly concentrated on overall visual fidelity, making it difficult to maintain the integrity and clarity of fine structures such as character strokes and table borders, which can easily lead to irreversible information loss; and evaluation and optimization rely solely on pixel-level metrics, failing to directly correlate with the performance of downstream tasks such as OCR recognition rate, resulting in restoration results that may not meet the needs of practical applications. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a layout-guided adaptive document image enhancement method and system. This aims to solve the problems of poor readability, loss of detailed structure, and mismatch between the overall restoration effect and downstream recognition tasks caused by existing natural image restoration models neglecting the structural semantic differences within the document (such as the different characteristics of text, tables, titles, and images). This invention integrates high-precision document layout analysis technology with layout-oriented adaptive feature modulation technology, enabling the restoration network to dynamically adjust the feature reconstruction strategy for different regions based on the semantic structure information of the document. This significantly improves the readability and detail fidelity of key areas such as text and tables while enhancing the overall visual quality of the image, ultimately improving the accuracy of document image restoration and achieving effective alignment between the restoration effect and downstream OCR recognition tasks, thus significantly improving the overall efficiency of document digitization processing.

[0006] To achieve the above objectives, the technical solution of the present invention includes the following steps:

[0007] (1) Input a low-resolution LR document image, identify the location and category of different semantic regions in the image based on deep learning detection and segmentation technology, and output a semantic segmentation mask image M;

[0008] (2) Based on the Real-ESRGAN generator G, the LR image is input into the generator G. After passing through the initial convolutional layer, the backbone feature extraction module composed of multiple dense residual modules RRDB, and the upsampling process, the intermediate features are obtained. ;

[0009] (3) Construct an L-SPADE module as a conditional feature modulator and insert it into the convolutional enhancement process after the upsampling network of the generator G; the L-SPADE module learns and generates a pair of spatial adaptive modulation parameter maps through the network based on the input semantic segmentation mask map M, which are used to perform affine transformation on the normalized feature map to generate the repaired super-resolution image. ;

[0010] (4) An image inpainting generator is constructed by generator G and L-SPADE module, and generative adversarial training is performed on it to obtain the optimal image inpainting generator. This generator is used to achieve adaptive enhancement of document images.

[0011] Compared with the prior art, the present invention has the following advantages:

[0012] First, this invention adopts the overall framework of "layout parsing-guided repair," which for the first time systematically integrates high-precision document layout analysis (obtained using DocLayout_YOLO) with deep image inpainting networks (such as Real-ESRGAN). This enables the repair process to obtain pixel-level semantic structure prior information (i.e., semantic mask M). This integration allows subsequent feature modulation to accurately distinguish and locate different functional areas such as text, tables, and images, thereby breaking the limitation of traditional methods that perform "one-size-fits-all" processing of document images and laying a key foundation for implementing regionally differentiated adaptive enhancement.

[0013] Secondly, this invention innovatively designs an L-SPADE (Layout-Adaptive Normalization) module and strategically embeds it into the upsampling path of the repair network. This module utilizes a semantic mask M to dynamically generate scaling factors (γ) and offset factors (β) that perfectly correspond to spatial locations and feature channels through a lightweight convolutional network. This design enables the repair network to perform pixel-by-pixel and channel-by-channel affine transformations on the normalized feature map, thereby achieving refined regional differentiation of the repair strategy. In text areas, the network can learn to enhance high-frequency edge features to sharpen strokes and reduce adhesion; in table areas, it can strengthen line continuity to repair breaks; and in image areas, it can promote smooth color transitions and texture restoration. This directly enhances the ability to restore details in key information areas of a document, making the repaired image not only "look clear" but, more importantly, "read accurately."

[0014] Third, this invention establishes a task-oriented evaluation system for document restoration. In the process of model optimization, in addition to traditional pixel-level fidelity indicators (such as PSNR and SSIM), downstream task performance (such as OCR recognition confidence) is explicitly included as part of the core evaluation criteria and optimization objectives. This system shifts the training direction of the model from simply pursuing pixel similarity for "visual clarity" to ensuring recognition accuracy for "functional usability". This drives the restoration network to learn feature representations that are more conducive to character recognition, effectively solving the problem of the disconnect between restoration results and final application requirements. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the overall implementation of the method of the present invention;

[0016] Figure 2 This is a schematic diagram of document layout features in this invention;

[0017] Figure 3 A schematic diagram of the basic architecture of an existing Real-ESRGAN generator;

[0018] Figure 4A schematic diagram showing the location of the L_SPADE module designed for this invention within the generator;

[0019] Figure 5 The simulation experiment effect diagram provided for the embodiment of the present invention. Detailed Implementation

[0020] The present invention will now be further described with reference to the accompanying drawings.

[0021] Example 1: Refer to Appendix Figure 1 The present invention proposes a layout-guided document image adaptive enhancement method, the specific implementation steps of which include the following:

[0022] Step 1) Input a low-resolution LR document image, identify the location and category of different semantic regions in the image based on deep learning detection and segmentation technology, and output a semantic segmentation mask map M; in this embodiment, the semantic segmentation mask map M is obtained through a document layout analysis network, and the mask categories include at least two of the following: body text, title, table, and image.

[0023] Step 2) Based on the Real-ESRGAN generator G, the LR image is input into the generator G. After passing through the initial convolutional layer, the backbone feature extraction module composed of multiple dense residual modules RRDB, and the upsampling process, intermediate features are obtained. In this embodiment, there are a total of 23 dense residual modules (RRDBs). The first third of the RRDBs are used to handle simple degradation, including noise and slight blur; the middle third of the RRDBs are used to handle complex degradation, including compression artifacts and blur; and the last third of the RRDBs are used to recover fine textures and structural edges. The RRDB blocks are connected by identity to form residual chains. Each of the above dense residual modules (RRDBs) contains 3 dense residual blocks (RDBs), and each RDB uses a dense connection pattern.

[0024] Step 3) Construct an L-SPADE module as a conditional feature modulator and insert it into the convolutional enhancement process after the upsampling network of the generator G; the L-SPADE module learns and generates a pair of spatial adaptive modulation parameter maps based on the input semantic segmentation mask map M through the network, which are used to perform affine transformation on the normalized feature map to generate the repaired super-resolution image. In this embodiment, the spatial adaptive modulation parameter map includes a scaling factor γ and an offset factor β. Where C, H, and W represent the number of channels, height, and width of the current feature map, respectively; specifically, it is obtained by using the semantic mask map M as input to the L-SPADE module, which is then processed by a lightweight two-layer convolutional network, ConvNetspade. The above-mentioned repaired super-resolution image Specifically, it receives intermediate features output by the upsampling network in the generator G through the L-SPADE module. Then, the generated scaling factor γ and offset factor β are used to adjust the intermediate features. A affine transformation is performed channel-by-channel and pixel-by-pixel to obtain the modulated feature map. Finally, after passing through the terminal convolutional layer, we obtain:

[0025] .

[0026] Step 4) An image inpainting generator is constructed using generator G and the L-SPADE module, and generative adversarial training is performed on it to obtain the optimal image inpainting generator. This generator is then used to achieve adaptive enhancement of document images. In this embodiment, during the training process in this step, a discriminator D is used to distinguish the generated images. With real high-quality document diagrams ;Constructed loss function At least including pixel reconstruction loss Perceived loss and combat losses .

[0027] Example 2: The overall implementation steps of the document image adaptive enhancement method proposed in this example are the same as in Example 1. Please refer to the appendix for details. Figures 1-4 The implementation process of this invention will be further described in detail with specific examples:

[0028] Step 1: Extract the document semantic layout mask

[0029] As attached Figure 2 As shown, the low-resolution (LR) document image to be repaired is input into a pre-trained document layout analysis network model (such as DocLayout_YOLO). This model performs semantic segmentation on the image using deep learning-based detection and segmentation techniques, identifying the location and category of different semantic regions in the image, and outputting a pixel-level semantic segmentation mask M. Each pixel value in mask M corresponds to a predefined semantic category, such as background, body text, title, table, image, etc. This step provides structured region prior knowledge for the subsequent repair process.

[0030] Step 2: Extract image depth features

[0031] The same LR images are input in parallel into a generator G for deep feature extraction. (See attached image.) Figure 3As shown, the generator G employs a two-stage architecture comprising a deep feature extraction backbone and a progressive upsampling network. This embodiment preferably uses an architecture based on the Real-ESRGAN generator. The LR image sequentially passes through an initial convolutional layer and a backbone network composed of multiple residual dense blocks (RRDBs) to extract multi-level deep feature maps. This step is the fundamental feature acquisition process for image inpainting.

[0032] The image above first passes through an initial convolutional layer for shallow feature extraction, resulting in an initial feature map. Next, the deep feature extraction backbone consists of multiple RRDB (Residual-in-Residual Dense Block) modules. Each RRDB contains three Residual Dense Blocks (RDBs), forming a three-layer nested residual learning structure. Each RDB employs a dense connection pattern, allowing each convolutional layer to access features from all previous layers, mitigating the gradient vanishing problem in deep networks. Real-ESRGAN typically uses a depth configuration of 23 RRDB blocks. The shallow RRDBs (first 1 / 3) primarily handle simple degradations such as noise and slight blur; the middle RRDBs (middle 1 / 3) mainly handle complex degradations such as compression artifacts and blur; and the deep RRDBs (last 1 / 3) are used to recover fine textures and structural edges. Identity connections are used between the RRDB blocks to form residual chains, ensuring smooth information flow.

[0033] In this embodiment, the sampling process in this step adopts a progressive strategy rather than a single large upsampling step. In each upsampling unit, the image features are first enhanced by a 3×3 convolution, increasing the number of channels to four times the original. Then, pixel rearrangement is used to convert the channel dimension into a spatial dimension. Finally, the LReLU activation function is used to provide non-linearity. For the 4x super-resolution task, a two-stage upsampling is adopted. Each upsampling step only needs to learn a simple mapping, reducing checkerboard artifacts.

[0034] Step 3: Layout adaptive feature modulation.

[0035] The semantic mask M obtained in step 1 is used as a conditional input and applied after the upsampling network of the generator G described in step 2. Specifically, this is achieved by constructing a layout-adaptive feature modulation module, L-SPADE. The L-SPADE module proposed in this invention, as a conditional feature modulator, is strategically inserted into the convolutional enhancement process after the upsampling network of the image inpainting generator, such as... Figure 4 As shown:

[0036] The L-SPADE module described above receives a semantic mask M, processes it through a lightweight convolutional network (ConvNetspade), and outputs spatially adaptive modulation parameters that match the size of the current layer's feature map, including a scaling factor γ and an offset factor β.

[0037]

[0038] in, C, H, and W represent the number of channels, height, and width of the current feature map, respectively. The key point is that the values ​​of γ and β depend on the semantic content of the input mask M; therefore, the network learns completely different modulation parameters for text regions and image regions.

[0039] Let the feature map of the current input L-SPADE module be... The generated γ and β are used to perform a channel-by-channel, pixel-by-pixel affine transformation on the normalized features to obtain the modulated feature map. :

[0040]

[0041] The technical benefits of this operation include: the values ​​of the modulation parameters γ and β are entirely determined by the semantic content of the input mask M. Therefore, the network can learn and apply differentiated feature enhancement strategies for different semantic regions. For example, in text regions, it can learn to increase γ to sharpen edges and stroke features; in image regions, it can learn appropriate β to smooth color transitions. This achieves a fundamental shift in the inpainting strategy from "globally uniform" to "locally adaptive."

[0042] Step 4: High-resolution image generation and adversarial training

[0043] After adaptive modulation with an embedded L-SPADE module, the feature map is processed by the terminal convolutional layer to output the repaired super-resolution image. .

[0044] The entire inpainting network is trained within a Generative Adversarial Network (GAN) framework. The discriminator D is used to distinguish the generated images. With real high-quality document diagrams Total loss function Typically includes:

[0045] Pixel reconstruction loss:

[0046] Perceived loss:

[0047] Combat losses:

[0048] By jointly optimizing these losses, the generator can output images that are both realistic and sharp.

[0049] In summary, through the above steps, especially the introduction of the L-SPADE adaptive feature modulation module guided by semantic layout mask as described in step 3, this invention successfully integrates the structured prior knowledge of the document into the image restoration process. This enables the restoration network to perform refined enhancement based on the semantic importance of different regions, ultimately obtaining a document restoration image that is superior in both overall visual quality and downstream OCR recognition performance, and has great application potential.

[0050] The effects of the present invention will be further explained below with reference to simulation experiments.

[0051] 1. Simulation conditions:

[0052] The simulation experiments of this invention were conducted in a high-performance computing server hardware environment equipped with two NVIDIA RTX 4090 GPUs and a software environment based on a remote development connection using Visual Studio Code (VSCode).

[0053] 2. Simulation content:

[0054] Dataset: The Publaynet document dataset is used, which contains approximately 40,000 document images with rich layout annotations (titles, text, tables, images, etc.).

[0055] Degradation Simulation: To simulate complex input conditions in real-world scenarios, a composite degradation process is applied to the high-resolution image to generate an LR image: ① Gaussian blur (σ=2.5); ② Downsampling (scaling factor r=2.5); ③ Adding Gaussian noise (δ=0.12); ④ JPEG compression (quality factor q=80).

[0056] Training details: Real-ESRGAN was used as the baseline model, with an L-SPADE module embedded. Training was conducted for 100 epochs, totaling 400,000 iterations, with a batch size of 6 per GPU, using the Adam optimizer (learning rate 1×10⁻⁻⁴). 4 β = [0.9, 0.99]). The loss function includes L1 pixel loss (weight 1.0), VGG-19-based perceptual loss (weight 1.0), and standard adversarial loss (weight 0.1). Training employs a distributed strategy with TensorBoard monitoring enabled, saving checkpoints every 2000 iterations. The discriminator uses a spectral normalized UNet architecture, providing pixel-level gradient feedback.

[0057] 3. Simulation results:

[0058] Appendix Figure 5The experimental results are visualized, showing that compared with the baseline model (Real-ESRGAN), the image restored by the method of this invention exhibits clearer character edges in the text line area, significantly reduces stroke adhesion, and has more coherent table lines.

[0059] To highlight the beneficial effects of the present invention, the following description is provided in conjunction with the table below: As shown in Table 1, the quantitative evaluation results on the test set show that the Real-ESRGAN+L_SPADE method proposed in this invention has achieved comprehensive and significant performance improvements in all indicators.

[0060] Table 1. Performance comparison of different methods on the test set

[0061]

[0062] In terms of pixel-level fidelity, the method of this invention demonstrates an improvement in PSNR (Peak Signal-to-Noise Ratio) from 24.81 dB to 25.03 dB, an increase of 0.22 dB, and SSIM (Structural Similarity Score) from 0.9291 to 0.9363, an increase of 0.0072. Particularly noteworthy is the breakthrough achieved in task-oriented performance metrics, with OCR recognition confidence significantly improved from 0.8627 to 0.9065, a relative improvement of 5.23%.

[0063] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0064] The above simulation analysis proves the correctness and effectiveness of the method proposed in this invention.

[0065] The parts of this invention not described in detail are common knowledge to those skilled in the art.

[0066] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Obviously, those skilled in the art, after understanding the content and principle of the present invention, may make various modifications and changes in form and detail without departing from the principle and structure of the present invention. However, these modifications and changes based on the concept of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A layout-guided document image adaptive enhancement method, characterized in that, Includes the following steps: (1) Input a low-resolution LR document image, identify the location and category of different semantic regions in the image based on deep learning detection and segmentation technology, and output a semantic segmentation mask image M; (2) Based on the Real-ESRGAN generator G, the LR image is input into the generator G. After passing through the initial convolutional layer, the backbone feature extraction module composed of multiple dense residual modules RRDB, and the upsampling process, the intermediate features are obtained. ; (3) Construct an L-SPADE module as a conditional feature modulator and insert it into the convolutional enhancement process after the upsampling network of the generator G; the L-SPADE module learns and generates a pair of spatial adaptive modulation parameter maps through the network based on the input semantic segmentation mask map M, which are used to perform affine transformation on the normalized feature map to generate the repaired super-resolution image. ; (4) An image inpainting generator is constructed by generator G and L-SPADE module, and generative adversarial training is performed on it to obtain the optimal image inpainting generator. This generator is used to achieve adaptive enhancement of document images.

2. The method according to claim 1, characterized in that: The semantic segmentation mask map M in step (1) is obtained through a document layout analysis network; the mask categories in the mask map include at least two of the following: body text, title, table, and image.

3. The method according to claim 1, characterized in that: The dense residual module RRDB in step (2) consists of 23 blocks. The first 1 / 3 of the RRDB is used to process simple degradation, including noise and slight blur. The middle 1 / 3 of the RRDB is used to process complex degradation, including compression artifacts and blur. The last 1 / 3 of the RRDB is used to restore fine textures and structural edges. The residual chains are formed by identity connections between the RRDB blocks.

4. The method according to claim 1 or 3, characterized in that: The dense residual module RRDB mentioned in step (2) contains 3 dense residual blocks RDB, and each RDB adopts a dense connection mode.

5. The method according to claim 1, characterized in that: The spatial adaptive modulation parameter map mentioned in step (3) includes a scaling factor γ and an offset factor β. Where C, H, and W are the number of channels, height, and width of the current feature map, respectively; specifically, it is obtained by taking the semantic mask map M as the input of the L-SPADE module and processing it through a lightweight two-layer convolutional network ConvNetspade.

6. The method according to claim 5, characterized in that: The repaired super-resolution image described in step (3) Specifically, it receives intermediate features output by the upsampling network in the generator G through the L-SPADE module. Then, the generated scaling factor γ and offset factor β are used to adjust the intermediate features. A affine transformation is performed channel-by-channel and pixel-by-pixel to obtain the modulated feature map. Finally, after passing through the terminal convolutional layer, we obtain: 。 7. The method according to claim 1, characterized in that: In step (4), during the generative adversarial training process, the discriminator D is used to distinguish the generated images. With real high-quality document diagrams ;Constructed loss function At least including pixel reconstruction loss Perceived loss and combat losses .

Images