An image geometric-shading suppression joint correction method and system based on a hybrid architecture deep network

Abstract

The application belongs to the field of image correction, and specifically discloses a kind of image geometric-shading suppression joint correction method and system based on hybrid architecture deep network, method includes as follows: image preprocessing, image block shading suppression, image integration splicing. Through the collaborative work of multi-scale deformable document correction and image block shading suppression, the visual quality of the document image and the accuracy of subsequent OCR extraction are effectively improved, the application is suitable for various image documents, and can also process the geometric and illumination distortion problems of the document image, realize automatic correction, thereby improve the efficiency of acceptance.

Description

Technical Field

[0001] This invention relates to the field of image correction technology, and specifically to an image geometry-shadow suppression joint correction method and system based on a hybrid architecture deep network. Background Technology

[0002] In the process of processing documents at the company, people often use smartphones to photograph documents. However, the photographed document images often suffer from geometric and lighting distortion problems. This is due to various physical deformations of the paper (such as folds, bends, or crumpling), camera position, and uneven lighting conditions. These distortions not only degrade the visual quality and readability of the document images but also negatively impact downstream multimedia applications such as automatic text recognition, polymorphic retrieval, content analysis, editing, and saving.

[0003] Early work on document image geometric correction relied on 3D reconstruction and auxiliary hardware or multi-view images to restore the 3D shape of the paper, which increased costs and limited applicability. Other methods assumed a parametric model of the document surface and optimized it by extracting specific features, but the optimization process was costly and oversimplified, resulting in poor performance. In recent years, deep learning has been introduced to address geometric and lighting distortion problems, but current methods often ignore the overall physical deformation of the paper and cannot effectively capture long-distance relationships, resulting in curved text areas remaining in the corrected document, or the inability to capture global context in lighting correction, leaving shadow areas in the corrected image. Therefore, a new image geometry-shadow suppression joint correction method is urgently needed to address the geometric distortion caused by shooting angle and paper curvature, as well as the shadow artifacts caused by uneven lighting during document image acquisition. Summary of the Invention

[0004] To address the issues of geometric distortion caused by shooting angle and paper curvature, as well as shadow artifacts caused by uneven lighting during document image acquisition, this invention provides a joint correction method and system for image geometry and shadow suppression based on a hybrid architecture deep network. It comprises two main modules: an adaptive non-uniform control point-guided multi-scale deformable document correction module and an image patch shadow suppression module. This invention can collaboratively process geometric and lighting distortions, significantly improving the visual quality of images and OCR recognition accuracy, thus solving the problems mentioned in the background section.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an image geometry-shadow suppression joint correction method based on a hybrid architecture deep network, comprising the following steps: S1. Image preprocessing; S2. Multi-scale Deformable Document Correction: The preprocessed background-removed image is input into a convolutional neural network (CNN) module to extract high-resolution local features. Then, a multi-head self-attention mechanism called Swin Transformer with a moving window is used to progressively downsample the hierarchical feature map and model the global context and long-range dependencies of the image. Features at different scales are fused at multiple scales. The fused feature map is input into an adaptive non-uniform control point sampling sub-network to predict the local complexity map of the image, dynamically generating a non-uniformly distributed control point grid. The generated non-uniform control points and their corresponding local features are then input into a differentiable control point prediction head to regress and obtain the target position of each control point. Finally, a differentiable smooth deformation sampling layer is used to generate a continuous sampling grid, and the geometrically corrected document image is obtained by resampling from the original image. S3. Image Patch Shadow Suppression: The geometrically corrected image is cropped into image patches with overlapping regions. Each image patch is processed by a convolutional layer to extract features, and then input into the SAN shadow adaptive normalization module. Shadow features are extracted through multi-scale dilated convolution, and the feature distribution of shadow and non-shadow regions is aligned using a pixel-level adaptive lighting model. Subsequently, the normalized features are input into the frequency domain-guided lighting reconstruction module. The Fast Fourier Attention (FFAT) module captures the global lighting pattern in the frequency domain, separates the spatial periodic artifacts in the shadow region, and reconstructs the pixel-level lighting correction effect through a lightweight decoder to obtain the corrected image patch. S4. Image Integration and Stitching: The output corrected image blocks are seamlessly stitched together using a multi-band fusion algorithm to obtain a complete document image that has been geometrically and illumination corrected.

[0006] Preferably, in step S1, the image preprocessing includes: taking the document material to be corrected as input and converting it into a uniform image format; downsampling the image to a fixed size using bilinear interpolation; predicting the confidence map of the foreground document region using a lightweight semantic segmentation network; obtaining a binary document region mask through threshold binarization; and removing the image background through element-wise matrix multiplication to obtain a foreground image containing only the document. The specific steps are as follows: S11. Take the document material to be corrected as input and convert it into a uniform image format; S12. Normalize the converted image by first unifying the color space to RGB, removing possible Alpha channels, and then downsampling to a fixed size through bilinear interpolation to obtain a standard-sized image S. S13. Use the lightweight semantic segmentation network DocSegNet to extract the foreground document region from the image S and output a binary mask M. S14. Perform element-wise multiplication between the obtained binary mask M and the standard-size image S to obtain the document foreground image after removing the background. The formula is expressed as .

[0007] Preferably, in the lightweight semantic segmentation network DocSegNet, the encoder adopts a MobileNetV3 backbone network based on depthwise separable convolution. A document prior attention module is introduced in the bottleneck layer of the encoder to generate a rectangular prior mask that matches the size of the feature map. Then, the prior is fused with deep semantic features through the attention mechanism. The decoder uses a progressive upsampling method and skips connections with the corresponding layer features of the encoder to restore edge details. In the final output segmentation head, a differentiable conditional random field (CRF) layer is used as a refinement module to perform edge correction on the coarse confidence map output by the network, so that the predicted document mask boundary is closer to the curved or wrinkled edges of the actual document, and finally outputs a binary mask M.

[0008] Preferably, step S2 specifically includes the following: S21. Local detail feature extraction: Extracting local details from the image The input consists of a convolutional neural network (CNN) module with four residual blocks. Each residual block contains two 3×3 convolutional layers, batch normalization, and ReLU activation with strides of 1, 2, 2, and 2, respectively. The output is a multi-scale local feature map. ; S22, Global Context Modeling: Integrating Feature Maps The input multi-head self-attention module based on a moving window models the global context and long-range dependencies of the image at different scales through progressive downsampling and window shifting mechanisms, and outputs multi-scale feature maps; S23. Multi-scale feature fusion: The output multi-scale feature maps are upsampled to a uniform size, concatenated along the channel dimension, and then fused using two 3×3 convolutional layers to obtain the fused feature map. ; S24. Adaptive Non-uniform Control Point Generation: This involves fusing feature maps... The input is an adaptive non-uniform control point sampling subnetwork. This subnetwork first estimates the local complexity map of the image using a lightweight complexity prediction head, and then generates a non-uniform control point grid based on the complexity map using a hierarchical sampling strategy. S25. Control point target position regression: The local features corresponding to the adaptive non-uniform control points are input into the control point prediction head. The prediction head contains two residual fully connected blocks. Each block contains: a fully connected layer, layer normalization, ReLU activation and Dropout. Finally, the target position coordinates of each control point are obtained through linear layer regression and the coordinates are normalized to the [0,1] interval. S26. Sampling Network Generation: A continuous sampling grid is generated based on the predicted control points and predefined source control points by transforming the TPS layer using differentiable thin-plate spline. Grid ; S27. Using bilinear interpolation, based on the sampling grid... Grid From the original image The pixel values ​​are resampled to obtain the geometrically corrected image C.

[0009] Preferably, in step S24, the generation of a non-uniform control point grid using a hierarchical sampling strategy based on the complexity map specifically includes: allocating denser control points in areas with severe curvature or dense text, while allocating sparser control points in flat areas, with the overall number of control points being fixed but the distribution being adaptively non-uniform.

[0010] Preferably, in step S26, the continuous sampling grid is generated by transforming the differentiable thin-plate spline TPS layer based on the predicted control points and the predefined source control points. Grid Specifically, it includes: S261. For the normalized coordinates of each pixel in the output image, first calculate its floating-point index in the sparse control point grid, locate the four corner control points of the grid cell, and obtain the corresponding displacement vector. S262. Calculate the local relative coordinates of the pixel with respect to the top-left control point of the cell; S263. Input the shared MLP to obtain four raw weight values, and normalize them into interpolated weights using Softmax. S264. The dense displacement of the current pixel is obtained by weighted summation. After the calculation is completed for all pixels, a complete high-resolution dense displacement field is obtained, and then a sampling grid is generated.

[0011] Preferably, step S3 specifically includes the following: S31. Image segmentation and positional encoding: The geometrically corrected image C is cropped into overlapping image blocks, resulting in a total of N image blocks. ; Append normalized coordinates of each image patch to the global image. As a learnable positional embedding, Indicates the relative position of the block's center point; S32, Image Patch Feature Extraction: Each image patch First, feature extraction is performed using three convolutional layers. The first layer has a 7x7 kernel, a stride of 2, and outputs 64 channels; the second layer has a 5x5 kernel, a stride of 1, and outputs 128 channels; the third layer has a 3x3 kernel, a stride of 1, and outputs 256 channels, resulting in a feature map. ; S33, SAN Shadow Adaptive Normalization Processing: This process transforms the feature map... The input shadow adaptive normalization module first extracts multi-scale shadow features using parallel multi-scale dilated convolution, and then concatenates them to obtain the shadow feature map. Then, a lightweight attention network is used to estimate the illumination parameters of each pixel. and The features within the local window are adaptively normalized; finally, channel-selective normalization is introduced, which adaptively selects the channels to be normalized using a learnable gating vector, and outputs the feature map. ; S34. Frequency-domain guided illumination reconstruction: The output feature map... The process involves reconstructing an illumination-corrected image patch. First, a Fast Fourier Attention (FFAT) module captures the global illumination pattern. Then, a learnable projection is performed on the feature map to obtain the query Q and key K, and cross-attention is calculated in the frequency domain. The frequency-domain enhanced features are fused with the original spatial domain features through a learnable gating mechanism. This fusion is then input into a lightweight decoder and passed through skip connections to preserve edge details. Finally, the illumination-corrected image patch is output. .

[0012] Preferably, step S4 specifically includes the following: S41. For each image block Construct a 5-layer Gaussian pyramid and the Pyramid of Laplace ; S42. Calculate the weight mask for each image patch based on its position in the global image. For high-frequency layers ( Narrowband fusion is used for the low-frequency layer; Broadband fusion is adopted, and the weight graph of each layer is normalized to ensure that the sum of the weights of the overlapping areas is 1; S43. Perform weighted fusion of each frequency band; S44. Rebuild from the highest level: From Upsampling and The fusion layers are added together, and so on, until... The final stitched image is obtained. .

[0013] On the other hand, to achieve the above objectives, the present invention also provides the following technical solution: an image geometry-shadow suppression joint correction system based on a hybrid architecture deep network, comprising the following modules: Image preprocessing module; The multi-scale deformable document correction module: The preprocessed background-removed image is input into a convolutional neural network (CNN) module to extract high-resolution local features. Then, a multi-head self-attention mechanism (Swin Transformer) with a moving window is used to progressively downsample the hierarchical feature map and model the global context and long-range dependencies of the image. Features at different scales are fused at multiple scales. The fused feature map is input into an adaptive non-uniform control point sampling sub-network to predict the local complexity map of the image, dynamically generating a non-uniformly distributed control point grid. The generated non-uniform control points and their corresponding local features are then input into a differentiable control point prediction head to regress and obtain the target position of each control point. Finally, a differentiable smooth deformation sampling layer generates a continuous sampling grid, and the geometrically corrected document image is obtained by resampling from the original image. Image patch shadow suppression module: The geometrically corrected image is cropped into image patches with overlapping regions. Each image patch is processed by a convolutional layer to extract features, and then input into the SAN shadow adaptive normalization module. The module extracts shadow features through multi-scale dilated convolution and uses a pixel-level adaptive lighting model to align the feature distribution of shadow and non-shadow regions. The normalized features are then input into the frequency domain-guided lighting reconstruction module. The module uses a Fast Fourier Attention (FFAT) module to capture the global lighting pattern in the frequency domain, separates the spatial periodic artifacts in the shadow region, and reconstructs the pixel-level lighting correction effect through a lightweight decoder to obtain the corrected image patch. Image integration and stitching module: The output corrected image blocks are seamlessly stitched together using a multi-band fusion algorithm to obtain a complete document image that has been geometrically and illumination corrected.

[0014] The beneficial effects of this invention are: 1) This invention patent achieves high-precision geometric and illumination joint correction of material images. In the geometric correction stage, it combines global dependency modeling with CNN local detail preservation and moving window self-attention mechanism, and accurately restores the physical deformation of the document through refined non-uniform control point prediction structure and multi-task loss function. In the illumination correction stage, it adopts block processing and multi-band fusion strategy for high-resolution images to effectively remove shadow artifacts and keep text edges clear. Each step is closely connected and end-to-end differentiable optimization, which ultimately significantly improves the visual quality of the image and the accuracy of OCR recognition. It is particularly suitable for the automated processing of large-size, complex background document images in IoT acceptance scenarios. 2) This invention integrates a hybrid coding architecture into the document image correction process and proposes the first document image correction framework based on the fusion of CNN and moving window multi-head self-attention mechanism. By introducing a hybrid coding architecture of hierarchical moving window self-attention and convolutional neural network in the geometric correction module, it not only uses CNN to preserve the local details of text strokes and page edges, but also uses the moving window attention mechanism to capture the global context and long-range dependencies of the image, accurately modeling the overall curvature trend and local wrinkle morphology of the document, making up for the shortcomings of traditional CNN in not being able to capture the relationship between distant pixels to fully describe the physical deformation of paper. 3) This invention patent addresses the geometric distortion correction problem by proposing a geometric correction method combining differentiable thin-plate spline transformation. The geometric correction module first extracts high-resolution local features through CNN, then establishes global dependencies on the hierarchical feature map using a multi-head self-attention module based on a moving window, and inputs multi-scale fused features into a differentiable control point prediction head to regress and obtain the target positions of non-uniformly distributed control points. Finally, a smooth sampling grid is generated through a differentiable deformation sampling layer, and the geometric correction result is obtained by resampling from the original image. This end-to-end differentiable design unifies and optimizes feature extraction and geometric transformation, avoiding problems such as incomplete correction and local distortion caused by ignoring the overall physical deformation of paper or using post-processing methods in existing methods. 4) The image patch shadow suppression module proposed in this invention further removes shadow artifacts after geometric correction. This module crops the geometrically corrected image into overlapping image patches and learns the global illumination pattern in the frequency domain through a Fast Fourier Attention Module (FFAT). It effectively separates the spatial periodic artifacts in the shadow area using frequency domain attention, without the need for explicit shadow mask prediction. The frequency domain enhancement features and spatial domain features are fused through learnable gating and then input into a lightweight decoder to reconstruct pixel-level illumination correction. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall framework of the image geometry-shadow suppression joint correction method based on a hybrid architecture deep network according to an embodiment of the present invention; Figure 2 This is a comparison image showing the background removal effect in an embodiment of the present invention; Figure 3 This is a control point prediction diagram according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the design of a differentiable deformable module according to an embodiment of the present invention. Figure 5 This is a schematic diagram of the multi-band fusion correction algorithm flow design in an embodiment of the present invention; Figure 6 This is a schematic diagram of the geometric-shading joint correction view of a student's homework image in an embodiment of the present invention; Figure 7 This is a schematic diagram of an image geometry-shadow suppression joint correction system module based on a hybrid architecture deep network, according to an embodiment of the present invention. In the diagram, 110 is the image preprocessing module; 120 is the multi-scale deformable document correction module; 130 is the image patch shadow suppression module; and 140 is the image integration and stitching module. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] This invention provides a technical solution: a joint correction method for image geometry and shadow suppression based on a hybrid architecture deep network. Taking student homework images as an example, a novel end-to-end framework called a student homework image converter is proposed. This converter consists of two main modules: an adaptive non-uniform control point-guided multi-scale deformable document correction module and an image patch shadow suppression module. It can collaboratively process geometric and illumination distortions, significantly improving the visual quality and OCR recognition accuracy of the image. The overall framework flow is as follows: Figure 1 As shown, the detailed steps include the following: 1. Obtain student assignment images: Retrieve student-uploaded assignment attachments from the teaching management system or assignment submission platform. Utilize RPA tools to write automated scripts that connect to the academic affairs system or learning platform, locate and download student-submitted assignment attachments based on their file names. Ensure the stability and accuracy of the retrieval process, handle potential anomalies (such as network timeouts, insufficient permissions, missing files, etc.) and log them.

[0018] 2. Unified Conversion of Attachment Formats: For all obtained assignment attachments, convert them to image formats using appropriate methods based on their type. Specifically, for PDF files, use the PyMuPDF library to render them page-by-page as high-resolution JPEG images; for Word documents, use LibreOffice's headless mode to convert them to PDF and then convert them to images again; for existing image formats (such as JPEG and PNG), retain them directly. Set the output resolution to 300 DPI during conversion to ensure the text is clearly legible.

[0019] 3. Preliminary Image Preprocessing: The converted image is normalized. First, the image color space is unified to RGB, and possible alpha channels are removed. Then, the image size is checked. If any dimension is less than 640 pixels, bicubic interpolation is used to enlarge the image to ensure the feature extraction effect in subsequent processing.

[0020] 4. Adaptive Image Downsampling: The preprocessed image is downsampled to obtain a standard-sized image S. The downsampling size is dynamically determined based on the original image resolution: if the longer side of the image is greater than 1024 pixels, it is scaled proportionally to a longer side of 1024 pixels; if the longer side is less than 288 pixels, it is scaled proportionally to a shorter side of 288 pixels; otherwise, the original size is maintained. The downsampling method uses bicubic interpolation with anti-aliasing to preserve edge sharpness.

[0021] 5. Prior-guided document region segmentation: Based on the image S obtained in step 4, the foreground document region is extracted using DocSegNet, a lightweight semantic segmentation network specifically designed for document images. The network structure of DocSegNet is as follows: the encoder uses a MobileNetV3 backbone network based on depthwise separable convolutions, the input image size is 288×288, and the output is a feature map with a stride of 32. A document prior attention module is introduced at the bottleneck layer of the encoder: first, a rectangular prior mask matching the size of the feature map is generated. (Based on a pre-defined rectangle in the central region of the image, with a size equal to 80% of the feature map), the prior information and deep semantic features are then fused using an attention mechanism: ; in Features of the bottleneck layer for The activation function and decoder employ progressive upsampling (2x, 4x, 8x, 16x) and skip connections with corresponding layer features from the encoder to restore edge details. In the final segmentation head, a differentiable Conditional Random Field (CRF) layer is used as a refinement module to correct the edges of the network's coarse confidence map, making the predicted document mask boundaries more closely match the curved or wrinkled edges of the actual document. The final output is a binary mask M, where 1 represents the document region and 0 represents the background.

[0022] 6: Loss function for segmentation network training: DocSegNet uses a joint loss function for training: ; in For pixel-level binary cross-entropy loss, Loss for boundary perception. This is the global IoU loss due to masking. Hyperparameter settings. , The Adam optimizer is used during training, with an initial learning rate of... Batch size 32, 200 iterations.

[0023] 7: Document Region Extraction. The binary mask M obtained in step 5 is multiplied element-wise with the original image S to obtain the document foreground image after background removal. : ; For holes in the mask, morphological closing operations (kernel size 5×5) are used to fill them, ensuring the integrity and continuity of the document area. Figure 2 The example shows a comparison before and after background removal, where... Figure 2 The left image is the original input image. Figure 2 The image on the right shows the effect after removing the background.

[0024] 8: Local detail feature extraction. Extracting local details from the image... The input consists of a convolutional neural network module with four residual blocks. Each residual block contains two 3×3 convolutional layers, batch normalization, and ReLU activation, with strides of 1, 2, 2, and 2, respectively. The output is a multi-scale local feature map, denoted as the lowest layer output. Preserve the strokes and edge details of the text.

[0025] 9: Global context modeling. This involves modeling the feature map. The input is a multi-head self-attention module (SwinTransformer Tiny version) based on a moving window. The network structure consists of 4 stages, with a window size of 7*7 and the number of multi-head attention heads [2, 4, 8, 16], and downsampling by a factor of 2 in each stage. Through progressive downsampling and window shifting mechanisms, the global context and long-range dependencies of the image are modeled at different scales, and multi-scale feature maps are output. The dimensions are 72*72*128, 36*36*256, 18*18*512, and 9*9*1024 respectively; 10: Multi-scale feature fusion. The multi-scale feature maps output from step 9 are upsampled to a uniform size of 36×36 (using bilinear interpolation). After concatenation along the channel dimension, feature fusion is performed through two 3×3 convolutional layers to obtain the fused feature map. .

[0026] 11: Adaptive non-uniform control point generation. This involves fusing feature maps. An adaptive non-uniform control point sampling subnetwork is input. This subnetwork first estimates the local complexity map of the image through a lightweight prediction head. The complexity is calculated based on prior gradient magnitude and text density, and the results are then fed into EAST for confidence evaluation. A non-uniform control point grid is generated using a hierarchical sampling strategy based on the complexity map: denser control points are allocated in high-complexity regions (such as densely texted areas or curved / wrinkled areas), while sparser control points are allocated in flat areas. The total number of control points is fixed at 256, but the distribution exhibits an adaptive non-uniform state, denoted as . .like Figure 3 As shown, the visualization results of the control point distribution are presented, where Figure 3 The image on the left is the original image. Figure 3 The middle image shows control points using a uniform grid (traditional method). Figure 3 The right figure shows the adaptive non-uniform control points generated by this invention. It can be seen that the control points are more dense in areas with dense text and curved areas.

[0027] 12: Control point target position regression: This involves retrieving the local features corresponding to adaptive non-uniform control points (from...). The input control point prediction head is obtained through bilinear sampling. The prediction head contains two residual fully connected blocks, each containing: a fully connected layer (outputting 256 dimensions), layer normalization, ReLU activation, and Dropout (ratio 0.1). Finally, linear regression is used to obtain the target position coordinates of 256 control points. The coordinates are normalized to the interval [0,1].

[0028] 13: Differentiable MLP-TPS deformable modules, such as Figure 4 As shown, predefined source control points and The corresponding adaptive grid coordinates (normalized to [0,1]). According to and A dense sampling grid is generated using a differentiable interpolation module based on MLP. The specific steps are as follows: 13.1 Normalized coordinates for each pixel in the output image First, calculate its floating-point index in the sparse control point grid, locate the four corner control points of the grid cell, and obtain the corresponding displacement vector. (The displacement vector is defined as) ).

[0029] 13.2 Calculate the local relative coordinates of the pixel with respect to the top-left control point of the cell. ; 13.3 The input is shared with an MLP (structure: 2D input → 32D hidden layer, ReLU → 4D output layer, no activation), resulting in four original weight values. These are then normalized to interpolated weights using Softmax. .

[0030] 13.4 Weighted summation yields the dense displacement of the current pixel: ; 13.5 After completing the calculations for all pixels, a complete high-resolution dense displacement field is obtained. This leads to the generation of a sampling grid: ; 14: Geometric Correction Image Generation: Using bilinear interpolation, the original image is generated based on a sampling grid. By resampling pixel values, a geometrically corrected image is obtained. Where H and W are the output image sizes (they are the same as the input S by default).

[0031] 15: Image Blocking and Position Encoding: The geometrically corrected image C is cropped into overlapping image blocks, each 128*128 pixels in size, with a stride of 112 pixels (overlap rate 12.5%), resulting in a total of N image blocks. Append normalized coordinates from the global image to each image patch. As a learnable positional embedding, This must indicate the relative position of the block's center point.

[0032] 16: Image patch feature extraction: for each image patch First, feature extraction is performed using three convolutional layers. The first layer has a 7x7 kernel, a stride of 2, and outputs 64 channels; the second layer has a 5x5 kernel, a stride of 1, and outputs 128 channels; the third layer has a 3x3 kernel, a stride of 1, and outputs 256 channels, resulting in a feature map. .

[0033] 17: SAN Shadow Adaptive Normalization Processing: This process transforms the image patch feature map output from step 16 into a single image patch feature map. The input shadow adaptive normalization module first extracts multi-scale shadow features using parallel multi-scale dilated convolution (dilation rates of 3, 6, and 12, outputting 128 channels), and then concatenates them to obtain the shadow feature map. The illumination parameters for each pixel are then estimated using a lightweight attention network. and Adaptive normalization is performed on the features within the local window (7*7): ; Finally, channel-selective normalization (CSNorm) is introduced, which adaptively selects the channels to be normalized using a learnable gating vector, and outputs a feature map: ; 18: Frequency-domain guided illumination reconstruction: Reconstruct the feature map output from step 17. Reconstruct the illuminated image patches. First, capture the global illumination pattern using a Fast Fourier Attention Module (FFAT): obtain the query by performing a learnable projection on the feature map. s and keys Calculate cross-attention in the frequency domain: ; in For two-dimensional fast Fourier transform, The process involves element-wise multiplication. Frequency domain enhancement features are fused with the original spatial domain features using a learnable gating mechanism. The resulting data is then input into a lightweight decoder (3 layers of transposed convolutions, channel count 256→128→64), followed by skip connections to preserve edge details. The final output is an illumination-corrected image patch. To maintain color consistency, correction is performed only in the luminance channel, while the chroma channel directly reuses the original image blocks.

[0034] 19: Multi-band fusion and stitching: Combine all corrected image blocks A multi-band fusion algorithm is used to seamlessly stitch the images together according to their overlapping regions, resulting in a complete illumination-corrected image R, as shown below. Figure 5 As shown, the specific steps are as follows: 19.1: For each image patch Construct a 5-layer Gaussian pyramid and the Pyramid of Laplace ( ,in (Based on the original size, downsampled by 2 times for each layer); 19.2: Calculate the weight mask for each image patch based on its position in the global image. (Based on distance transformation to block boundaries). For high-frequency layers ( Narrow-band fusion (weight transition band width 8 pixels) is used for the low-frequency layer. Broadband fusion (weight transition band width 32 pixels) is used. The weight maps of each layer are normalized to ensure that the sum of the weights in the overlapping areas is 1.

[0035] 19.3: Weighted fusion of various frequency bands: ; 19.4: Rebuild from the top floor: From Upsampling and The fusion layers are added together, and so on, until... The final stitched image is obtained. .

[0036] 20: The final corrected image The image is compressed and stored according to a preset format (JPEG format, quality factor 95) and saved to a specified database or file server. A processing log is generated simultaneously, recording the image source, processing time, correction parameters, and processing status for subsequent auditing and quality traceability. For OCR recognition tasks, the corrected image can be directly input into the downstream OCR engine for text recognition.

[0037] 21: The experiment collected 5280 student homework images to form a dataset, including handwritten and printed assignments from different subjects such as Chinese, mathematics, and English, covering various distortion types such as flatness, curvature, wrinkles, and local shadows. These images were divided into training, validation, and test sets in a 6:2:2 ratio. The experimental metrics used were SSIM and PSNR, calculated using the following formulas: ; ; ; in These are the local window pixel values ​​of the original image and the corrected image, respectively. They are respectively The mean, They are respectively standard deviation yes covariance, L=255, K1=0.01, K2=0.03. The experimental results are shown in Table 1: Table 1 Experimental Results ; The image showing the result of geometric-shading joint correction of student assignments, such as... Figure 6 As shown, in the comparative experiment ( Figure 6 As shown in Table 1, this method achieves optimal results across all metrics. Specifically, the method achieves an SSIM of 0.972, a PSNR of 28.7 dB, and an OCR accuracy of 96.8%, representing improvements of 2.2%, 1.2 dB, and 2.7 percentage points respectively compared to the second-best UDoc method. Compared to existing methods such as Hough transform, DocGeo, DewarpNet, and SwinIR, this method demonstrates significant advantages in structural fidelity, pixel-level reconstruction quality, and character recognition accuracy, validating its advanced nature and robustness in document image correction and enhancement tasks. This invention is applicable to various image documents and can also handle geometric and lighting distortion problems in document images. Through the collaborative work of a multi-scale deformable document correction module and an image patch shadow suppression module, the visual quality and OCR accuracy of document images are effectively improved.

[0038] Based on the same inventive concept as the above-described method embodiments, this application also provides an image geometry-shadow suppression joint correction system based on a hybrid architecture deep network. This system can achieve the functions provided by the above-described method embodiments, such as... Figure 7 As shown, the system includes the following modules: Image preprocessing module 110: Takes the document material to be corrected (such as PDF, Word, etc.) as input and converts it into a unified image format (such as PNG, JPG) using a third-party library or interface. First, the image is downsampled to a fixed size (e.g., 288×288) using bilinear interpolation. Then, a lightweight semantic segmentation network (such as an encoder-decoder structure based on MobileNetV3) predicts the confidence map of the foreground document region. After thresholding and binarization, a binary document region mask is obtained. The image background is removed through element-wise matrix multiplication to obtain a foreground image containing only the document. This preprocessing network is trained with binary cross-entropy loss to ensure segmentation accuracy.

[0039] The multi-scale deformable document correction module 120 inputs the preprocessed background-removed image into a convolutional neural network (CNN) module to extract high-resolution local features, preserving details of text strokes and page edges. Subsequently, a multi-head self-attention mechanism, the Swin Transformer, is used to progressively downsample the hierarchical feature map and model the global context and long-range dependencies of the image, accurately capturing the overall curvature trend and local wrinkle morphology of the document. Features at different scales are fused at multiple scales to obtain rich feature representations. The fused feature map is first input into an adaptive non-uniform control point sampling sub-network. This sub-network dynamically generates a non-uniformly distributed control point grid by predicting the local complexity map of the image. Specifically, denser control points are allocated in areas with severe curvature or dense text, while sparser control points are allocated in flat areas, thereby improving correction accuracy while reducing computational redundancy. Then, the generated non-uniform control points and their corresponding local features are input into a differentiable control point prediction head, and the target position of each control point is obtained through global average pooling and fully connected layer regression. Finally, a smooth sampling grid is calculated using a differentiable thin-plate spline transform (TPS) layer based on predicted control points and predefined source control points. The original image is then resampled using bilinear interpolation to obtain the geometrically corrected document image. This entire process achieves end-to-end differentiable optimization from feature extraction to geometric transformation, balancing local detail preservation with global deformation correction. Furthermore, adaptive control point distribution further focuses on complex deformation regions. During training, the L1 distance between the predicted and ground images is used as the loss function, supplemented by a control point regularization term to ensure grid smoothness. Image patch shadow suppression module 130: After geometric correction, document images may still contain sampling and shadow artifacts. The illumination correction converter aims to remove these shadows and improve OCR accuracy. The geometrically corrected image is cropped into image patches with overlapping regions. Each image patch has its features extracted by a convolutional layer and then input into the SAN shadow adaptive normalization module. Shadow features are extracted through multi-scale dilated convolution, and the feature distribution of shadow and non-shadow regions is aligned using a pixel-level adaptive illumination model. Subsequently, the normalized features are input into the frequency domain-guided illumination reconstruction module. The Fast Fourier Attention module (FFAT) captures the global illumination pattern in the frequency domain, separates the spatial periodic artifacts in the shadow region, and reconstructs the pixel-level illumination correction effect through a lightweight decoder to obtain the corrected image patch. During optimization, the transformer is trained by minimizing the L1 distance and VGG perceptual loss between the predicted image patch and the real image.

[0040] Image integration and stitching module 140: Seamlessly stitches together the output corrected image patches using a multi-band fusion algorithm to obtain a complete document image that has undergone geometric and illumination correction. Multi-band fusion decomposes the high-frequency details and low-frequency structure of the image patches using the Laplacian pyramid, providing narrow-band transitions for high frequencies to preserve sharpness and wide-band transitions for low frequencies to eliminate stitching artifacts, ultimately reconstructing a high-quality full image.

[0041] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0042] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0043] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0044] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0045] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A joint correction method for image geometry-shadow suppression based on a hybrid architecture deep network, characterized in that, Includes the following steps: S1. Image preprocessing; S2. Multi-scale Deformable Document Correction: The preprocessed background-removed image is input into a convolutional neural network (CNN) module to extract high-resolution local features. Then, a multi-head self-attention mechanism called Swin Transformer with a moving window is used to progressively downsample the hierarchical feature map and model the global context and long-range dependencies of the image. Features at different scales are fused at multiple scales. The fused feature map is input into an adaptive non-uniform control point sampling sub-network to predict the local complexity map of the image, dynamically generating a non-uniformly distributed control point grid. The generated non-uniform control points and their corresponding local features are then input into a differentiable control point prediction head to regress and obtain the target position of each control point. Finally, a differentiable smooth deformation sampling layer is used to generate a continuous sampling grid, and the geometrically corrected document image is obtained by resampling from the original image. S3, Image Patch Shadow Suppression: The geometrically corrected image is cropped into image patches with overlapping regions. Each image patch is processed by a convolutional layer to extract features, and then input into the SAN shadow adaptive normalization module. Shadow features are extracted through multi-scale dilated convolution, and the feature distribution of shadow areas and non-shadow areas is aligned using a pixel-level adaptive lighting model. The normalized features are then input into the frequency domain-guided illumination reconstruction module. The global illumination pattern is captured in the frequency domain by the Fast Fourier Attention Module (FFAT), and the spatial periodic artifacts in the shadow area are separated. The pixel-level illumination correction effect is reconstructed by the lightweight decoder to obtain the corrected image block. S4. Image Integration and Stitching: The output corrected image blocks are seamlessly stitched together using a multi-band fusion algorithm to obtain a complete document image that has been geometrically and illumination corrected.

2. The image geometry-shadow suppression joint correction method based on hybrid architecture deep networks according to claim 1, characterized in that: In step S1, the image preprocessing includes: taking the document material to be corrected as input and converting it into a uniform image format; downsampling the image to a fixed size using bilinear interpolation; predicting the confidence map of the foreground document region using a lightweight semantic segmentation network; obtaining a binary document region mask through threshold binarization; and removing the image background through element-wise matrix multiplication to obtain a foreground image containing only the document. The specific steps are as follows: S11. Take the document material to be corrected as input and convert it into a uniform image format; S12. Normalize the converted image by first unifying the color space to RGB, removing possible Alpha channels, and then downsampling to a fixed size through bilinear interpolation to obtain a standard-sized image S. S13. Use the lightweight semantic segmentation network DocSegNet to extract the foreground document region from the image S and output a binary mask M. S14. Perform element-wise multiplication between the obtained binary mask M and the standard-size image S to obtain the document foreground image after removing the background. The formula is expressed as .

3. The image geometry-shadow suppression joint correction method based on hybrid architecture deep networks according to claim 2, characterized in that: In the lightweight semantic segmentation network DocSegNet, the encoder adopts the MobileNetV3 backbone network based on depthwise separable convolution. A document prior attention module is introduced in the bottleneck layer of the encoder to generate a rectangular prior mask that matches the size of the feature map. Then, the prior is fused with deep semantic features through the attention mechanism. The decoder uses a progressive upsampling method and skips connections with the corresponding layer features of the encoder to restore edge details. In the final output segmentation head, a differentiable conditional random field (CRF) layer is used as a refinement module to perform edge correction on the coarse confidence map output by the network, so that the predicted document mask boundary is closer to the curved or wrinkled edges of the actual document, and finally outputs a binary mask M.

4. The image geometry-shadow suppression joint correction method based on hybrid architecture deep networks according to claim 1, characterized in that: Step S2 specifically includes the following: S21. Local detail feature extraction: Extracting local details from the image The input consists of a convolutional neural network (CNN) module with four residual blocks. Each residual block contains two 3×3 convolutional layers, batch normalization, and ReLU activation with strides of 1, 2, 2, and 2, respectively. The output is a multi-scale local feature map. ; S22, Global Context Modeling: Integrating Feature Maps The input multi-head self-attention module based on a moving window models the global context and long-range dependencies of the image at different scales through progressive downsampling and window shifting mechanisms, and outputs multi-scale feature maps; S23. Multi-scale feature fusion: The output multi-scale feature maps are upsampled to a uniform size, concatenated along the channel dimension, and then fused using two 3×3 convolutional layers to obtain the fused feature map. ; S24. Adaptive Non-uniform Control Point Generation: This involves fusing feature maps... The input is an adaptive non-uniform control point sampling subnetwork. This subnetwork first estimates the local complexity map of the image using a lightweight complexity prediction head, and then generates a non-uniform control point grid based on the complexity map using a hierarchical sampling strategy. S25. Control point target position regression: The local features corresponding to the adaptive non-uniform control points are input into the control point prediction head. The prediction head contains two residual fully connected blocks. Each block contains: a fully connected layer, layer normalization, ReLU activation and Dropout. Finally, the target position coordinates of each control point are obtained through linear layer regression and the coordinates are normalized to the [0,1] interval. S26. Sampling Network Generation: A continuous sampling grid is generated based on the predicted control points and predefined source control points by transforming the TPS layer using differentiable thin-plate spline. Grid ; S27. Using bilinear interpolation, based on the sampling grid... Grid From the original image The pixel values ​​are resampled to obtain the geometrically corrected image C.

5. The image geometry-shadow suppression joint correction method based on a hybrid architecture deep network according to claim 4, characterized in that: In step S24, the generation of a non-uniform control point grid based on a hierarchical sampling strategy according to the complexity map specifically includes: allocating denser control points in areas with severe curvature or dense text, while allocating sparse control points in flat areas, with the overall number of control points fixed but the distribution in an adaptive non-uniform state.

6. The image geometry-shadow suppression joint correction method based on hybrid architecture deep networks according to claim 4, characterized in that: In step S26, a continuous sampling grid is generated by transforming the differentiable thin-plate spline layer (TPS) based on the predicted control points and predefined source control points. Grid Specifically, it includes: S261. For the normalized coordinates of each pixel in the output image, first calculate its floating-point index in the sparse control point grid, locate the four corner control points of the grid cell, and obtain the corresponding displacement vector. S262. Calculate the local relative coordinates of the pixel with respect to the top-left control point of the cell; S263. Input the shared MLP to obtain four raw weight values, and normalize them into interpolated weights using Softmax. S264. The dense displacement of the current pixel is obtained by weighted summation. After the calculation is completed for all pixels, a complete high-resolution dense displacement field is obtained, and then a sampling grid is generated.

7. The image geometry-shadow suppression joint correction method based on hybrid architecture deep networks according to claim 1, characterized in that: Step S3 specifically includes the following: S31. Image segmentation and positional encoding: The geometrically corrected image C is cropped into overlapping image blocks, resulting in a total of N image blocks. ; Append normalized coordinates of each image patch to the global image. As a learnable positional embedding, Indicates the relative position of the block's center point; S32, Image Patch Feature Extraction: Each image patch First, feature extraction is performed using three convolutional layers. The first layer has a 7x7 kernel, a stride of 2, and outputs 64 channels; the second layer has a 5x5 kernel, a stride of 1, and outputs 128 channels; the third layer has a 3x3 kernel, a stride of 1, and outputs 256 channels, resulting in a feature map. ; S33, SAN Shadow Adaptive Normalization Processing: This process transforms the feature map... The input shadow adaptive normalization module first extracts multi-scale shadow features using parallel multi-scale dilated convolution, and then concatenates them to obtain the shadow feature map. ; The illumination parameters for each pixel are then estimated using a lightweight attention network. and Adaptive normalization is performed on the features within the local window; Finally, channel-selective normalization is introduced, which adaptively selects the channels to be normalized using a learnable gating vector, and outputs the feature map. ; S34. Frequency-domain guided illumination reconstruction: The output feature map... Reconstruct the illumination-corrected image patch; first, capture the global illumination pattern using the Fast Fourier Attention (FFAT) module: perform a learnable projection on the feature map to obtain the query Q and key K, and calculate the cross-attention in the frequency domain; Frequency domain enhancement features are fused with the original spatial domain features using learnable gating. The resulting image is then fed into a lightweight decoder via skip connections to preserve edge details, ultimately outputting an illumination-corrected image patch. .

8. The image geometry-shadow suppression joint correction method based on hybrid architecture deep networks according to claim 1, characterized in that: Step S4 specifically includes the following: S41. For each image block Construct a 5-layer Gaussian pyramid and the Pyramid of Laplace ; S42. Calculate the weight mask for each image patch based on its position in the global image. For high-frequency layers ( Narrowband fusion is used for the low-frequency layer; Broadband fusion is adopted, and the weight graph of each layer is normalized to ensure that the sum of the weights of the overlapping areas is 1; S43. Perform weighted fusion of each frequency band; S44. Rebuild from the highest level: From Upsampling and The fusion layers are added together, and so on, until... The final stitched image is obtained. .

9. A system for image geometry-shadow suppression joint correction method based on hybrid architecture deep networks according to any one of claims 1-8, characterized in that: Includes the following modules: Image preprocessing module (110); Multi-scale deformable document correction module (120): The preprocessed background-removed image is input into the convolutional neural network (CNN) module to extract high-resolution local features. Then, through the multi-head self-attention mechanism Swin Transformer with moving windows, the hierarchical feature map is downsampled stepwise and the global context and long-range dependencies of the image are modeled. Features at different scales are fused at multiple scales. The fused feature map is input into the adaptive non-uniform control point sampling sub-network to predict the local complexity map of the image, dynamically generating a non-uniformly distributed control point grid. Then, the generated non-uniform control points and their corresponding local features are input into the differentiable control point prediction head to regress and obtain the target position of each control point. Finally, a continuous sampling grid is generated through the differentiable smooth deformation sampling layer, and the geometrically corrected document image is obtained by resampling from the original image. Image patch shadow suppression module (130): The geometrically corrected image is cropped into image patches with overlapping regions. Each image patch is processed by a convolutional layer to extract features and then input into the SAN shadow adaptive normalization module. Shadow features are extracted through multi-scale dilated convolution and the feature distribution of shadow areas and non-shadow areas is aligned using a pixel-level adaptive illumination model. The normalized features are then input into the frequency domain-guided illumination reconstruction module. The global illumination pattern is captured in the frequency domain by the Fast Fourier Attention Module (FFAT), and the spatial periodic artifacts in the shadow area are separated. The pixel-level illumination correction effect is reconstructed by the lightweight decoder to obtain the corrected image block. Image integration and stitching module (140): Seamlessly stitches the output corrected image blocks using a multi-band fusion algorithm to obtain a complete document image that has been geometrically and illumination corrected.

Images