Robust document image geometric distortion correction method based on selective state space sequence modeling and related device

Abstract

The invention discloses a robust document image geometric distortion correction method and related device based on selective state space sequence modeling, and the method comprises the steps: carrying out the image enhancement and feature extraction of a distorted document image, and obtaining a group of original feature maps; each original feature map is planarized into a vector, and all feature vectors are used to construct a document sequence; using a selective state space sequence model to learn context information hidden in the document sequence, transforming all the document sequences subjected to context information hidden in the document sequence into updated feature maps, and encoding the updated feature maps to obtain final feature maps; and performing bilinear sampling by predicting reverse mapping through a decoder to obtain a corrected image. According to the corrected image, the straightness of the text lines can be kept, the alignment of the point of the starting text line and the end point can be ensured, and the method has very high robustness under various background environments, illumination conditions and distortion types such as curvature, folding, wrinkling and viewpoints.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a robust document image geometric distortion correction method based on selective state space sequence modeling and a related device. Background Art

[0002] Smartphones can provide a more flexible, portable, and direct option for digitizing document images than traditional flatbed scanners. However, document images captured this way often suffer from distortions due to unpredictable factors such as the document’s physical deformation, camera angle, contrast, and lighting conditions. These flaws make document images unsuitable for many formal assessment situations and can undermine subsequent processes such as automatic text recognition, analysis, retrieval, question answering, and editing.

[0003] Traditional 3D reconstruction-based methods often face limitations, including limited applicability to complex real-world deformations and high computational requirements during optimization. Among current deep learning-based methods, convolutional neural network-based methods are unable to capture long-range dependencies that are critical for modeling paper deformation. Although Transformer-based methods can capture global deformations to a greater extent, they still have some problems, such as the inability to accurately locate the position of the paper in the image and poor local details of the corrected image, such as text line bending and multi-line alignment. Summary of the invention

[0004] In order to overcome the technical problem of poor image correction effect in the prior art, the purpose of the present invention is to provide a robust document image geometric distortion correction method and related devices based on selective state space sequence modeling, which can achieve that the corrected image can not only maintain the straightness of the text lines, but also ensure that the starting point and the end point of the text line are aligned, and the method of the present invention has high robustness under various background environments, lighting conditions, and distortion types such as curvature, folding, wrinkling and viewpoint.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows: A robust document image geometric distortion correction method based on a selective state space sequence model comprises the following steps: Acquire a distorted document image; Perform image enhancement and feature extraction on the distorted document image to obtain a set of original feature maps; Flatten each original feature map into a vector and construct a document sequence from all feature vectors; Use the selective state space sequence model to learn the context information hidden in the document sequence, transform all document sequences with the context information hidden in the document sequence into updated feature maps, encode the updated feature maps, and obtain the final feature maps; The original feature map and the final feature map are dedistorted to obtain the text line segmentation map.

[0006] A further improvement of the present invention is that image enhancement and feature extraction are performed on the distorted document image to obtain the original feature map, including the following steps: image enhancement is performed on the distorted document image through geometric transformation and image filtering algorithm to obtain an enhanced document image.

[0007] A further improvement of the present invention is that the enhanced document image and the distorted document image are subjected to feature extraction by a convolutional neural network to obtain an original feature map.

[0008] A further improvement of the present invention is that before using the selective state space sequence model to learn the context information hidden in the document sequence, the following steps are performed: embedding the document sequence into a position encoding to obtain a position-aware feature map; and performing a gated linear projection operation on the position-aware feature map.

[0009] A further improvement of the present invention is that the updated feature map is encoded using stacked dilated convolutional layers.

[0010] A further improvement of the present invention is that the original feature map and the final feature map are subjected to image dewarping to obtain a text line segmentation map, comprising the following steps: adding the original feature map and the final feature map together, estimating a two-dimensional reverse mapping and a three-dimensional coordinate, and obtaining a backward mapping map; The backward mapping map is processed to obtain the text line segmentation map.

[0011] A further improvement of the present invention is that the backward mapping map is processed to obtain a text line segmentation map, comprising the following steps: According to the backward mapping graph, a reconstructed image is obtained by upsampling and backward mapping the distorted document image; Extract text lines from the reconstructed image and obtain a text line segmentation map.

[0012] A robust document image geometric distortion correction system based on a selective state space sequence model, comprising: A document image acquisition module, used for acquiring a distorted document image; An image enhancement and feature extraction module is used to perform image enhancement and feature extraction on the distorted document image to obtain a set of original feature maps; The document sequence construction module is used to flatten each original feature map into a vector and construct a document sequence from all feature vectors; A context information learning module is used to learn context information hidden in a document sequence using a selective state space sequence model, transform all document sequences with context information hidden in the document sequence into updated feature maps, encode the updated feature maps, and obtain a final feature map; The image dewarping module is used to dewarp the original feature map and the final feature map to obtain a text line segmentation map.

[0013] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the robust document image geometric distortion correction method based on a selective state space sequence model is implemented.

[0014] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the robust document image geometric distortion correction method based on a selective state space sequence model is implemented.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention uses the idea of ​​state space model to achieve the purpose of document image dewarping. Through a new sequence modeling and context information learning module, the context information between document tags and the non-local information of deep feature mapping can be well learned. A large number of experiments conducted on two widely used benchmark databases, DocUNet database and DIR300 database, show that the method proposed by the present invention can achieve the most advanced performance in both quantitative and qualitative aspects. The corrected image of the present invention can not only keep the text lines straight, but also ensure the alignment of the starting point and the ending point of the text line, and the method of the present invention has high robustness under various background environments, lighting conditions, and distortion types such as curvature, folding, wrinkling and viewpoint. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a robust document image geometric distortion correction method based on selective state space sequence modeling of the present invention; Figure 2 A comparison diagram of embodiments of the method of the present invention and other related methods on the DocUNet dataset; Figure 3 A comparison diagram of embodiments of the method of the present invention and other related methods on the DIR300 data set; Figure 4 The correction effect diagram of the method of the present invention on multiple document images with unknown and arbitrary distortions; Figure 5It is a structural schematic diagram of the robust document image geometric distortion correction system based on the selective state space sequence model of the present invention. DETAILED DESCRIPTION

[0017] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. The preferred embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thoroughly and comprehensively understood.

[0018] State Space Models (SSM) and its improved variants, the selective state space sequence models Mamba and Mamba2, have become an efficient backbone for long sequence modeling and contextual information learning. Compared with networks based on convolutional neural models designed to capture local dependencies, SSM is more powerful on long sequences; compared with Transformer networks that require the sequence length to be quadratic in complexity, SSM has higher computational efficiency and storage efficiency.

[0019] The present invention introduces the DocMamba network, a novel network based on Mamba2, which processes the entire image at the pixel level to estimate a coarse back-mapping for correction to reduce background interference and improve the readability of the document image. In particular, various pixel-level and geometric data augmentation operations are first performed, and an encoder based on a convolutional neural model is used to extract deep features of the distorted image. Then, a sequence modeling and context information learning module is designed to encode the contextual information and non-local information of the document image; then, a lightweight dual-head convolution is used as a decoder to estimate the coarse back-mapping. The final corrected image is obtained by upsampling the original distorted image for back-mapping.

[0020] like Figure 1 As shown in FIG. 1 , a robust document image geometric distortion correction method based on a selective state space sequence model includes three steps: document image enhancement and feature extraction, sequence modeling and context information learning, and robust document image dedistortion. The specific steps are as follows: 1) Document image enhancement and feature extraction: For document images with unknown and arbitrary distortion ,Firstly, the image enhancement process is carried out through geometric ,transformation and image filtering algorithm to obtain the enhanced ,document image.

[0021] These enhanced document images and document images with unknown and arbitrary distortions It is sent to the convolutional neural network for preliminary feature extraction and outputs a set of downsampled original feature maps .

[0022] 2) Sequence modeling and context information learning: For the sequence modeling and context information learning modules, each original feature map Flatten into a vector, and construct the document sequence from all these feature vectors. Embed the document sequence into position encoding to obtain a position-aware feature map .

[0023] Position-aware feature maps After the gated linear projection operation, the selective state space sequence model Mamba2 is used to learn the contextual information hidden in the document sequence. In the method of the present invention, there are three Mamba2 blocks, each of which contains 16 Mamba2 layers for sequence-to-sequence conversion. It is worth noting that the output of each Mamba2 block flows into a stacked dilated convolution block. The present invention uses different dilation rates to perform different levels of information integration.

[0024] Generally speaking, the selective state space sequence model is used to understand the contextual information hidden in the document image sequence, and the dilated convolution is designed to integrate non-local information. Through the above operations, the present invention transforms the contextual information document sequence hidden in the document sequence into an updated feature map, and further encodes the updated feature map using the stacked dilated convolution layer into the final feature map .

[0025] 3) Robust document image dewarping: The original feature map And the final feature map They are added together and sent to the decoder to estimate the two-dimensional reverse mapping and three-dimensional coordinates to obtain the backward mapping graph g and the three-dimensional coordinate graph w.

[0026] According to the backward mapping graph g, by Perform upsampling and backward mapping (bilinear sampling) to obtain the final corrected document image , that is, reconstruct the picture.

[0027] Extract text lines from the reconstructed image and obtain a text line segmentation map.

[0028] The visual contrast effect of the present invention is as follows Figure 2 and Figure 3As shown in the figure, the two images are from the DocUNet dataset and the DIR300 dataset respectively. The leftmost column is the original distorted image, the rightmost column is the result of geometric correction by the method of the present invention, and the other images in the middle are the results of other correction methods (including DewarpNet, PaperEdge, DocTr, FTAR, DocRes and DocGeoNet). By comparison, it can be seen that in the method of the present invention, the corrected image can not only keep the text lines straight, but also ensure that the starting point and the end point of the text line are aligned, and the removal effect of the background environment is also optimal.

[0029] Figure 4 More results of the present invention are displayed. The first row is 7 distorted pictures with arbitrary distortion on the DocUNet dataset and the DIR300 dataset. The second row is the result after geometric correction by the method of the present invention. It can be seen that the method of the present invention has strong robustness under various background environments, lighting conditions, and distortion types such as curvature, folding, wrinkles and viewpoint.

[0030] See also Figure 5 Another embodiment of the present invention provides a robust document image geometric distortion correction system based on a selective state space sequence model, comprising: A document image acquisition module, used for acquiring a distorted document image; An image enhancement and feature extraction module is used to perform image enhancement and feature extraction on the distorted document image to obtain a set of original feature maps; The document sequence construction module is used to flatten each original feature map into a vector and construct a document sequence from all feature vectors; A context information learning module is used to learn context information hidden in a document sequence using a selective state space sequence model, transform all document sequences with context information hidden in the document sequence into updated feature maps, encode the updated feature maps, and obtain a final feature map; The image dewarping module is used to dewarp the original feature map and the final feature map to obtain a text line segmentation map.

[0031] Another embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the robust document image geometric distortion correction method based on the selective state space sequence model when executing the computer program.

[0032] Another embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the robust document image geometric distortion correction method based on the selective state space sequence model is implemented.

[0033] The above description is only for the best embodiment of the present invention, but it should not be understood as limiting the claims. The present invention is not limited to the above embodiments, and its specific structure is allowed to be changed. However, all changes made within the protection scope of the independent claims of the present invention are within the protection scope of the present invention.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

Claims

1. A robust document image geometric distortion correction method based on selective state space sequence modeling, characterized in that: The steps include: Acquire a distorted document image; Perform image enhancement and feature extraction on the distorted document image to obtain a set of original feature maps; Flatten each original feature map into a vector and construct a document sequence from all feature vectors; Use the selective state space sequence model to learn the context information hidden in the document sequence, transform all document sequences with the context information hidden in the document sequence into updated feature maps, encode the updated feature maps, and obtain the final feature maps; The original feature map and the final feature map are dedistorted to obtain the text line segmentation map.

2. The method for robust document image geometric distortion correction based on selective state space sequence modeling according to claim 1, characterized in that: The method performs image enhancement and feature extraction on a distorted document image to obtain an original feature map, including the following steps: performing image enhancement on the distorted document image through geometric transformation and image filtering algorithm to obtain an enhanced document image.

3. The robust document image geometric distortion correction method based on selective state space sequence modeling according to claim 2 is characterized in that: The enhanced document image and the distorted document image are subjected to feature extraction using a convolutional neural network to obtain the original feature map.

4. The robust document image geometric distortion correction method based on selective state space sequence modeling according to claim 1, characterized in that: Before using the selective state-space sequence model to learn the contextual information hidden in the document sequence, the following steps are performed: embedding the document sequence into position encoding to obtain a position-aware feature map; and performing a gated linear projection operation on the position-aware feature map.

5. The robust document image geometric distortion correction method based on selective state space sequence modeling according to claim 1, characterized in that: The updated feature maps are encoded using stacked dilated convolutional layers.

6. The robust document image geometric distortion correction method based on selective state space sequence modeling according to claim 1, characterized in that: The original feature map and the final feature map are subjected to image dewarping to obtain a text line segmentation map, including the following steps: the original feature map and the final feature map are added together to estimate a two-dimensional reverse mapping and a three-dimensional coordinate to obtain a backward mapping map; The backward mapping map is processed to obtain the text line segmentation map.

7. The method for robust document image geometric distortion correction based on selective state space sequence modeling according to claim 6, characterized in that: Processing the backward mapping map to obtain a text line segmentation map includes the following steps: According to the backward mapping graph, a reconstructed image is obtained by upsampling and backward mapping the distorted document image; Extract text lines from the reconstructed image and obtain a text line segmentation map.

8. A robust document image geometric distortion correction system based on selective state space sequence modeling, characterized in that: include: A document image acquisition module, used for acquiring a distorted document image; An image enhancement and feature extraction module is used to perform image enhancement and feature extraction on the distorted document image to obtain a set of original feature maps; The document sequence construction module is used to flatten each original feature map into a vector and construct a document sequence from all feature vectors; A context information learning module is used to learn context information hidden in a document sequence using a selective state space sequence model, transform all document sequences with context information hidden in the document sequence into updated feature maps, encode the updated feature maps, and obtain a final feature map; The image dewarping module is used to dewarp the original feature map and the final feature map to obtain a text line segmentation map.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the robust document image geometric distortion correction method based on selective state space sequence modeling as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the robust document image geometric distortion correction method based on selective state space sequence modeling as described in any one of claims 1 to 7 is implemented.

Images