A robust document image geometric distortion rectification method based on selective state space sequence modeling and related device

Abstract

The application discloses a robust document image geometric distortion correction method based on selective state space sequence modeling and related devices, and the method comprises the following steps: image enhancement and feature extraction are performed on a document image with distortion to obtain a group of original feature maps; each original feature map is planarized into a vector, and all feature vectors are used to construct a document sequence; selective state space sequence model is used to learn context information hidden in the document sequence, all context information hidden in the document sequence is deformed into an updated feature map, the updated feature map is encoded to obtain a final feature map; and a bilinear sampling is performed through a decoder prediction reverse mapping to obtain a corrected image. The corrected image can not only keep the straightness of text lines, but also ensure the alignment of starting and ending points of the text lines, and the method has high robustness under various background environments, illumination conditions, distortion types such as curvature, folding, wrinkling and viewpoint.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a robust document image geometric distortion rectification method based on selective state space sequence modeling and a related device. BACKGROUND

[0002] Compared with traditional flatbed scanners, smartphones can provide a more flexible, portable and direct option for document image digitization. However, the document images taken in this way are often distorted due to unpredictable factors such as physical deformation of the document, camera angle, contrast and lighting conditions. These defects make the document images unsuitable for many formal evaluation situations and can damage subsequent processes such as automatic text recognition, analysis, retrieval, question answering and editing.

[0003] Traditional methods based on three-dimensional reconstruction usually face limitations, including limited applicability to complex real-world deformations and high computational requirements in the optimization process. Among the current deep learning-based methods, convolutional neural network-based methods cannot capture long-range dependencies that are crucial for modeling paper deformation. Although the Transformer-based method can capture global deformation to a greater extent, it still has some problems, such as being unable to accurately locate the position of the paper in the image and poor local details in the corrected image, such as text line bending and multi-line alignment. SUMMARY

[0004] To overcome the technical problem of poor image correction effect in the prior art, the purpose of the present application is to provide a robust document image geometric distortion rectification method based on selective state space sequence modeling and a related device, which can achieve a corrected image that not only maintains the straightness of text lines but also ensures the alignment of the starting and ending points of the text lines, and the method of the present application has high robustness under various background environments, lighting conditions and distortion types such as curvature, folding, wrinkling and viewpoint.

[0005] To achieve the above-mentioned purpose, the technical solution adopted by the present application is as follows:

[0006] A robust document image geometric distortion rectification method based on a selective state space sequence model, comprising the following steps:

[0007] Obtain a document image with distortion;

[0008] Perform image enhancement and feature extraction on the document image with distortion to obtain a set of original feature maps;

[0009] Flatten each original feature map into a vector, and construct a document sequence from all feature vectors;

[0010] The context information hidden in the document sequence is learned using a selective state space sequence model, the document sequence with the context information hidden therein is deformed into an updated feature map, the updated feature map is encoded to obtain a final feature map.

[0011] The original feature map and the final feature map are image unwarped to obtain a text line segmentation map.

[0012] The original feature map is obtained by performing image enhancement and feature extraction on the document image with distortion, comprising the following steps: performing image enhancement on the document image with distortion through geometric transformation and image filtering algorithm to obtain an enhanced document image.

[0013] The original feature map is obtained by performing feature extraction on the enhanced document image and the document image with distortion through a convolutional neural network.

[0014] Before the context information hidden in the document sequence is learned using the selective state space sequence model, the following steps are performed: embedding position coding on the document sequence to obtain a position-aware feature map; and performing a gated linear projection operation on the position-aware feature map.

[0015] The updated feature map is encoded using a stacked dilated convolutional layer.

[0016] The original feature map and the final feature map are image unwarped to obtain a text line segmentation map, comprising the following steps: adding the original feature map and the final feature map together to perform estimated two-dimensional inverse mapping and three-dimensional coordinates to obtain a backward mapping map.

[0017] The backward mapping map is processed to obtain a text line segmentation map.

[0018] The backward mapping map is processed to obtain a text line segmentation map, comprising the following steps:

[0019] According to the backward mapping map, the document image with distortion is up-sampled and backward mapped to obtain a reconstructed picture.

[0020] The text line is extracted from the reconstructed picture to obtain a text line segmentation map.

[0021] A robust document image geometric warping correction system based on a selective state space sequence model, comprising:

[0022] A document image acquisition module is configured to acquire a document image with distortion.

[0023] An image enhancement and feature extraction module is configured to perform image enhancement and feature extraction on the document image with distortion to obtain a set of original feature maps;

[0024] A document sequence construction module is configured to flatten each original feature map into a vector and construct a document sequence from all the feature vectors;

[0025] A context information learning module is configured to learn context information hidden in the document sequence using a selective state space sequence model, deform all the document sequences with the context information hidden therein into updated feature maps, encode the updated feature maps to obtain final feature maps;

[0026] An image unwarping module is configured to perform image unwarping on the original feature maps and the final feature maps to obtain text line segmentation maps.

[0027] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the robust document image geometric distortion correction method based on the selective state space sequence model when executing the computer program.

[0028] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the robust document image geometric distortion correction method based on the selective state space sequence model.

[0029] Compared with the prior art, the present application has the beneficial effects that:

[0030] The present application uses the idea of state space model to achieve the purpose of document image unwarping, and through a new sequence modeling and context information learning module, the context information between document marks and the non-local information of deep feature mapping can be well learned; a large number of experiments on two widely used benchmark databases, DocUNet database and DIR300 database, show that the method proposed in the present application can achieve the most advanced performance in quantitative and qualitative aspects. The corrected image of the present application can not only keep the text lines straight, but also ensure that the starting text line and the ending point are aligned, and the method of the present application has high robustness under various background environments, lighting conditions, and distortion types such as curvature, folding, wrinkling, and viewpoint. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 A flowchart of the robust document image geometric distortion correction method based on selective state space sequence modeling of the present application;

[0032] Figure 2The embodiment comparison chart of the method of the present application and other related methods on the DocUNet dataset;

[0033] Figure 3 The embodiment comparison chart of the method of the present application and other related methods on the DIR300 dataset;

[0034] Figure 4 The correction effect chart of the method of the present application on multiple document images with unknown and arbitrary distortion;

[0035] Figure 5 The structural schematic diagram of the robust document image geometric distortion correction system based on the selective state space sequence model of the present application. DETAILED DESCRIPTION

[0036] In order to facilitate the understanding of the present application, the present application will be described more fully below with reference to the accompanying drawings. The preferred embodiments of the present application are shown in the drawings. However, the present application can be realized in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0037] State space models (SSM) and their improved variants, selective state space sequence models Mamba and Mamba2, have become an efficient backbone for long sequence modeling and context information learning. Compared with convolutional neural model-based networks designed to capture local dependencies, SSM is more powerful on long sequences; compared with Transformer networks that require quadratic complexity of sequence length, SSM has higher computational efficiency and storage efficiency.

[0038] The present application introduces a DocMamba network, which is a new network based on Mamba2, which processes the entire image at the pixel level to estimate a rough backward mapping for correction to reduce background interference and improve the readability of the document image. In particular, first, various pixel-level and geometric data enhancement operations are performed, and a convolutional neural model-based encoder is used to extract deep features of the distorted image. Then, a sequence modeling and context information learning module is designed to encode the context information and non-local information of the document image; then, a lightweight double-head convolution is used as a decoder to estimate a rough inverse mapping. By upsampling the original distorted image backward mapping, the final corrected image is obtained.

[0039] As shown in Figure 1 A robust document image geometric distortion correction method based on a selective state space sequence model includes three steps: document image enhancement and feature extraction, sequence modeling and context information learning, and robust document image unwarping. The specific steps are as follows:

[0040] 1) Document image enhancement and feature extraction: For document images with unknown and arbitrary distortions , first, the image enhancement process is carried out through geometric transformation, image filtering algorithm, and the enhanced document image is obtained.

[0041] These enhanced document images and document images with unknown and arbitrary distortions are sent to the convolutional neural network for preliminary feature extraction, and a set of down-sampled original feature maps is output.

[0042] 2) Sequence modeling and context information learning: For the sequence modeling and context information learning module, each original feature map is flattened into a vector, and the document sequence is constructed from all these feature vectors. The position encoding of the document sequence is obtained .

[0043] After the position-aware feature map is subjected to a gated linear projection operation, the selective state space sequence model Mamba2 is used to learn the context information hidden in the document sequence. In the method of the present invention, there are three Mamba2 blocks, each containing 16 Mamba2 layers for sequence-to-sequence transformation. It is worth noting that the output of each Mamba2 block flows into a stacked dilated convolution block. The present invention uses different dilation rates to perform different levels of information integration.

[0044] In general, the selective state space sequence model is used to understand the context information hidden in the document image sequence, and the dilated convolution is designed to integrate non-local information. Through the above operations, the context information hidden in the document sequence is transformed into an updated feature map .

[0045] 3) Robust document image unwarping: The original feature map and the final feature map are added together and sent to the decoder to estimate the two-dimensional inverse mapping and the three-dimensional coordinates, obtaining the backward mapping map g and the three-dimensional coordinate map w.

[0046] According to the backward mapping map g, the document image with unknown and arbitrary distortions is subjected to up-sampling backward mapping (bilinear sampling), and the final corrected document image , i.e. the reconstructed picture, is obtained.

[0047] The text line is extracted from the reconstructed picture to obtain a text line segmentation map.

[0048] The visual contrast effect of the present application is shown in Figure 2 and Figure 3 , in which the pictures on the left column are the originally distorted pictures, the pictures on the right column are the results of the present application, and the pictures in the middle column are the results of other correction methods (including DewarpNet, PaperEdge, DocTr, FTAR, DocRes and DocGeoNet). By comparison, it can be seen that in the present application, the corrected image can not only keep the text lines straight, but also ensure that the starting and ending points of the text lines are aligned, and the removal effect of the background environment is also optimal.

[0049] Figure 4 is a more result display of the present application, the first row is 7 distorted pictures with distortion on the DocUNet dataset and the DIR300 dataset, and the second row is the result of the present application after geometric correction. It can be seen that the present application has strong robustness under various background environments, lighting conditions, and distortion types such as curvature, folding, wrinkling and viewpoint.

[0050] Referring to Figure 5 , another embodiment of the present application provides a robust document image geometric distortion correction system based on a selective state space sequence model, comprising:

[0051] A document image acquisition module is used to acquire a document image with distortion.

[0052] An image enhancement and feature extraction module is used to perform image enhancement and feature extraction on the document image with distortion to obtain a set of original feature maps.

[0053] A document sequence construction module is used to flatten each original feature map into a vector and construct a document sequence from all feature vectors.

[0054] A context information learning module is used to learn the context information hidden in the document sequence using a selective state space sequence model, deform all the context information hidden in the document sequence into updated feature maps, encode the updated feature maps to obtain final feature maps.

[0055] An image unwarping module is used to perform image unwarping on the original feature maps and the final feature maps to obtain a text line segmentation map.

[0056] Another embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the robust document image geometric distortion rectification method based on the selective state space sequence model.

[0057] Another embodiment of the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the robust document image geometric distortion rectification method based on the selective state space sequence model.

[0058] The above description is only for the best embodiments of the present application, but should not be understood as a limitation to the claims. The present application is not limited to the above embodiments, and the specific structure allows changes. Any changes within the scope of the independent claims of the present application are within the scope of the present application.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

Claims

1. A robust document image geometric skew correction method based on selective state space sequence modeling, characterized in that, The method comprises the following steps: obtaining a document image with distortion; performing image enhancement and feature extraction on the document image with distortion to obtain a set of original feature maps; flattening each original feature map into a vector and constructing a document sequence from all feature vectors; learning context information hidden in the document sequence using a selective state space sequence model, wherein the selective state space sequence model comprises three Mamba2 blocks, each Mamba2 block comprising 16 Mamba2 layers for sequence-to-sequence conversion, deforming all document sequences with hidden context information into updated feature maps, and encoding the updated feature maps using stacked dilated convolution layers to obtain final feature maps; performing image unwarping on the original feature maps and the final feature maps to obtain a text line segmentation map.

2. The robust document image geometric skew correction method based on selective state space sequence modeling according to claim 1, wherein, The method for performing image enhancement and feature extraction on the document image with distortion to obtain the original feature map comprises the following steps: performing image enhancement on the document image with distortion through geometric transformation and image filtering algorithms to obtain an enhanced document image.

3. The robust document image geometric skew correction method based on selective state space sequence modeling according to claim 2, wherein, The method for performing feature extraction on the enhanced document image and the document image with distortion using a convolutional neural network to obtain the original feature map.

4. The robust document image geometric skew correction method based on selective state space sequence modeling according to claim 1, wherein, Before learning the context information hidden in the document sequence using the selective state space sequence model, the following steps are performed: embedding position encoding on the document sequence to obtain a position-aware feature map; and performing a gated linear projection operation on the position-aware feature map.

5. The robust document image geometric skew correction method based on selective state space sequence modeling according to claim 1, wherein, The method for performing image unwarping on the original feature maps and the final feature maps to obtain the text line segmentation map comprises the following steps: adding the original feature maps and the final feature maps together, estimating two-dimensional inverse mapping and three-dimensional coordinates, and obtaining a backward mapping map; processing the backward mapping map to obtain the text line segmentation map.

6. The robust document image geometric skew correction method based on selective state space sequence modeling according to claim 5, wherein, The method for processing the backward mapping map to obtain the text line segmentation map comprises the following steps: According to the backward mapping map, performing up-sampling backward mapping on the document image with distortion to obtain a reconstructed picture; extracting text lines from the reconstructed picture to obtain the text line segmentation map.

7. A robust document image geometric warping rectification system based on selective state space sequence modeling, characterized in that, The method comprises the following steps: a document image acquisition module for acquiring a document image with distortion; an image enhancement and feature extraction module for performing image enhancement and feature extraction on the document image with distortion to obtain a set of original feature maps; a document sequence construction module for flattening each original feature map into a vector and constructing a document sequence from all feature vectors; a context information learning module for learning context information hidden in the document sequence using a selective state space sequence model, wherein the selective state space sequence model comprises three Mamba2 blocks, each Mamba2 block comprising 16 Mamba2 layers for sequence-to-sequence conversion, deforming all document sequences with hidden context information into updated feature maps, and encoding the updated feature maps using stacked dilated convolution layers to obtain final feature maps; an image unwarping module for performing image unwarping on the original feature maps and the final feature maps to obtain a text line segmentation map.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor executes the computer program to implement the robust document image geometric distortion rectification method based on selective state space sequence modeling according to any one of claims 1-6.

9. A computer readable storage medium, the computer readable storage medium storing a computer program, characterized in that: The computer program is executed by the processor to implement the robust document image geometric distortion rectification method based on selective state space sequence modeling according to any one of claims 1-6.

Images