Self-supervised methods, systems, devices, and storage media for correcting wrinkled document images.

Abstract

This invention discloses a self-supervised method, system, device, and storage medium for correcting wrinkled document images. These are corresponding solutions that can efficiently utilize large-scale unlabeled wrinkled document images for pre-training, thereby extracting more effective document deformation representations during the fine-tuning stage and improving the correction effect on deformed document images. Furthermore, it is applicable to document images of various languages ​​and document layouts, eliminating the need for specialized extraction and training of features such as text lines. The correction effect can be further improved by adding more unlabeled data for pre-training, enhancing the scalability of the solution. Moreover, compared to previous techniques based on a single neural network model, this invention, after two stages of training, can achieve the same document image correction effect while reducing computational overhead. Therefore, it can be integrated into smartphones and other smart devices with cameras, facilitating wider application in various scenarios.

Description

Technical Field

[0001] This invention relates to the field of image correction technology for wrinkled documents, and in particular to a self-supervised method, system, device and storage medium for image correction of wrinkled documents. Background Technology

[0002] With the widespread adoption of smartphones, document digitization has become more convenient than ever before. However, due to random factors such as document distortion, camera angle, and lighting conditions, document images captured by smartphones often exhibit various distortions. These distortions pose challenges to downstream visual tasks, such as automatic text recognition, content analysis, and intelligent question-answering systems. To overcome these problems, distorted document image correction has been actively researched over the past few decades.

[0003] Traditional solutions for correcting creased document images are typically based on 3D reconstruction techniques. These methods either rely on additional hardware or involve capturing multiple viewpoints around the creased document to reconstruct its three-dimensional shape, inevitably leading to numerous inconveniences. Constrained by high hardware costs and cumbersome shooting requirements, these techniques have failed to gain widespread adoption.

[0004] Most smartphones now have built-in document image correction algorithms. These algorithms typically detect the four straight line boundaries or four corner points of a document in a captured image to obtain the quadrilateral region contained in the document; then, they use projection transformation technology to map it onto a regular matrix image, thus completing the correction of the captured document image.

[0005] In recent years, with the development of deep learning, more and more methods in the field of wrinkled document image correction have adopted deep learning to achieve correction. Compared with traditional methods, deep learning-based methods have lower computational overhead. By training the model on a dataset of rendered, synthesized wrinkled documents, the model will have the ability to correct document wrinkles and possess strong generalization ability. Given a document image of any form as input, the trained model can immediately derive the coordinate mapping matrix corresponding to each pixel. Using this coordinate mapping matrix, the pixels of the actual document region in the input image are mapped to the empty image to be filled, thus achieving distortion correction of the wrinkled document image.

[0006] Overall, the existing solutions have the following main shortcomings:

[0007] (1) Existing methods for training neural networks for geometric correction mostly rely on limited attribute-labeled data to explicitly extract deformation representations, and cannot utilize a large amount of readily available unlabeled data to extract the deformation representations of the document itself. Therefore, a lot of extra effort and time is required to label the data, and the existing relevant labeled datasets are limited, which also restricts the research and improvement of wrinkled document correction algorithms. The corrected document image still has a certain degree of distortion.

[0008] (2) To improve the correction effect, existing network models have increased the model size and the number of parameters, resulting in excessively long training time and the possibility of overfitting. This leads to the problem that the model performs well on the training set but the correction effect is not ideal in actual use. Furthermore, the increase in size and number of parameters leads to longer time consumed in the correction inference process, affecting the user's actual user experience.

[0009] (3) Existing document correction algorithms built into smartphones are mostly applicable to regular quadrilateral document images without wrinkles. If the document being photographed has wrinkles, it will be impossible to form a regular quadrilateral in the image, thus making correction impossible. This drawback is a major limitation that prevents wrinkle document correction algorithms from being widely applied on mobile devices, even though document digitization on mobile devices is currently one of the most important application scenarios.

[0010] In view of this, the present invention is hereby proposed. Summary of the Invention

[0011] The purpose of this invention is to provide a self-supervised method, system, device, and storage medium for correcting wrinkled document images. It introduces self-supervised learning into wrinkled document correction and restoration applications. By pre-training on an unlabeled wrinkled document dataset and completing specific tasks, such as reconstructing and restoring randomly masked images, the network helps extract the inherent deformation representations of the document. In the subsequent fine-tuning stage, relevant weight data is imported to achieve better correction and restoration results. Furthermore, compared to existing methods, it has stronger generalization capabilities, is applicable to wrinkled document image correction in multilingual environments, and requires less computational overhead, facilitating its widespread adoption.

[0012] The objective of this invention is achieved through the following technical solution:

[0013] A self-supervised method for correcting wrinkled document images includes:

[0014] A wrinkled document image correction framework is constructed and trained. This framework includes a pre-trained framework and a fine-tuning framework. In the pre-training phase, the original wrinkled document image is input into the pre-training framework. After background removal and random masking, a reconstruction restorer composed of a first encoder and a first decoder outputs a reconstructed wrinkled document image. The reconstruction restorer is trained using a self-supervised approach. In the fine-tuning phase, the original wrinkled document image is input into the fine-tuning framework. After background removal, a coordinate mapping matrix composed of a second encoder and a second decoder predicts the coordinate mapping matrix. The coordinate mapping matrix is ​​trained based on the difference between the predicted and actual coordinate mapping matrices. After training, the fine-tuning framework is called the wrinkled document image correction model. The parameters of the second encoder in the coordinate mapping matrix are initialized using the parameters of the first encoder in the pre-trained reconstruction restorer.

[0015] The wrinkled document image to be corrected is input into the wrinkled document image correction model. After background removal processing, the coordinate mapping matrix is ​​predicted by the coordinate mapping matrix generator. Then, the predicted coordinate mapping matrix is ​​used to correct the wrinkled document image after background removal processing to obtain the corrected document image.

[0016] A self-supervised system for correcting wrinkled document images, comprising:

[0017] A framework construction and training unit is used to construct and train a wrinkled document image correction framework, which includes a pre-trained framework and a fine-tuning framework. In the pre-training phase, the original wrinkled document image is input into the pre-training framework, and after background removal and random masking, a reconstruction restorer composed of a first encoder and a first decoder outputs a reconstructed wrinkled document image. The reconstruction restorer is trained in a self-supervised manner. In the fine-tuning phase, the original wrinkled document image is input into the fine-tuning framework, and after background removal, a coordinate mapping matrix composed of a second encoder and a second decoder predicts the coordinate mapping matrix. The coordinate mapping matrix is ​​trained based on the difference between the predicted and actual coordinate mapping matrices. After training, the fine-tuning framework is called the wrinkled document image correction model. The parameters of the second encoder in the coordinate mapping matrix are initialized using the parameters of the first encoder in the pre-trained reconstruction restorer.

[0018] The wrinkled document image correction unit is used to input the wrinkled document image to be corrected into the wrinkled document image correction model, perform background removal processing, predict the coordinate mapping matrix through the coordinate mapping matrix generator, and then use the predicted coordinate mapping matrix to correct the wrinkled document image after background removal processing to obtain the corrected document image.

[0019] A processing device includes: one or more processors; and a memory for storing one or more programs.

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0021] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0022] As can be seen from the technical solution provided by the present invention, the present invention can fully and efficiently utilize a large-scale unlabeled dataset of wrinkled document images for pre-training, and then extract more effective document deformation representations in the subsequent fine-tuning stage, thereby improving the correction effect on deformed document images. Furthermore, the present invention is applicable to document images of various languages ​​and document layouts, without requiring specialized extraction and training for features such as text lines. Simultaneously, it can improve the subsequent correction effect by adding more unlabeled data for pre-training, thus enhancing the scalability of the solution. In addition, compared to previous techniques based on a single neural network model, the present invention, after two stages of training, can achieve the same document image correction effect while reducing computational overhead. Therefore, the present invention can be integrated into smartphones and other smart devices with cameras, facilitating its widespread use in a wider range of application scenarios. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A schematic diagram of the overall structure of a self-supervised wrinkled document image correction method provided in an embodiment of the present invention;

[0025] Figure 2 A schematic diagram of a self-supervised wrinkle document image correction system provided in an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0028] First, the following explanations are provided for the terms that may be used in this article:

[0029] The terms “including,” “comprising,” “containing,” “having,” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, “including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.)” should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0030] The following provides a detailed description of a self-supervised method, system, device, and storage medium for correcting wrinkled document images provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they should be performed according to conventional conditions in the art or conditions recommended by the manufacturer.

[0031] Example 1

[0032] This invention provides a self-supervised method for correcting wrinkled document images. By pre-training on a large-scale unlabeled dataset of wrinkled documents, the deformation features of the document in the wrinkled document image are extracted. This allows for more accurate correction and restoration in subsequent fine-tuning stages, resulting in a corrected document image with visual quality close to the original document. Simultaneously, the method minimizes the number of model parameters and computational complexity to improve operational efficiency during actual correction, facilitating widespread application. Figure 1 The diagram illustrates the overall structure of the above method, which mainly includes:

[0033] 1. Construct and train a wrinkled document image correction framework, which includes a pre-trained framework and a fine-tuning framework.

[0034] This invention employs a two-stage training scheme. The first stage is a pre-training stage, where the training object is the pre-trained framework. The parameters obtained after pre-training can be imported into the fine-tuning framework. The second stage is a fine-tuning stage, where the training object is the fine-tuning framework. The fine-tuning framework uses the parameters obtained after pre-training, and further training is performed on the input document image after background removal, achieving better correction and restoration results. Specifically:

[0035] (1) Pre-training phase.

[0036] In this embodiment of the invention, the original wrinkled document image is input into a pre-training framework, and after background removal and random masking are performed sequentially, the reconstructed wrinkled document image is output by a reconstruction restorer composed of a first encoder and a first decoder. The reconstruction restorer is trained in a self-supervised manner.

[0037] In this embodiment of the invention, the pre-training framework mainly includes: a background removal module, a random mask module, and a reconstruction restorer.

[0038] The background removal module removes the original wrinkled document image I through background removal processing. d The background area, outputting the wrinkled document image after background removal processing. e The original wrinkled document image can be a single-frame RGB wrinkled document image.

[0039] The random masking module performs random masking on the wrinkled document image after background removal, and outputs the masked image I. e The pixel content of the wrinkled document image after background removal is randomly masked according to a set ratio using a random mask. Figure 1 The example provided is a mask for three regions.

[0040] The reconstruction restorer, by encoding and decoding the masked image, outputs a reconstructed wrinkled document image I. r .

[0041] In this embodiment of the invention, the processing principle of the reconstruction restorer can refer to conventional technology, so it will not be described in detail. For example, it can be achieved through a self-attention mechanism.

[0042] In this embodiment of the invention, the wrinkled document image I after background removal is calculated using an optimized algorithm. e and the reconstructed image I r Self-supervised learning is performed using the deviation value; specifically, the reconstructed wrinkled document image I can be calculated. r Image of the wrinkled document after background removal processing I eThe L2 distance between the two regions is used as the first loss function to train the reconstruction restorer. In this process, the reconstruction restorer learns how to effectively extract the deformation representation of the visible region (i.e., the unmasked region) to reconstruct the pixels of the invisible region (i.e., the masked region).

[0043] During the pre-training stage, a large-scale unlabeled dataset of wrinkled documents can be used for pre-training, thereby better extracting the deformation features of documents in wrinkled document images.

[0044] (2) Fine-tuning stage.

[0045] In this embodiment of the invention, the original wrinkled document image is input into the fine-tuning frame. After background removal processing, a coordinate mapping matrix f is predicted by a coordinate mapping matrix consisting of a second encoder and a second decoder. Based on the predicted coordinate mapping matrix f and the actual coordinate mapping matrix f... gt The coordinate mapping matrix is ​​trained using the deviation values, wherein the parameters of the second encoder in the coordinate mapping matrix are initialized using the parameters (mainly weight parameters) of the first encoder in the pre-trained reconstruction restorer.

[0046] In this embodiment of the invention, the fine-tuning framework mainly includes: a background removal module, a coordinate mapping matrix, and a correction restorer.

[0047] The background removal module removes the background area of ​​the original wrinkled document image through background removal processing, and outputs the wrinkled document image I after background removal processing. e The fine-tuned framework and the pre-trained framework can use the same background removal module.

[0048] The coordinate mapping matrix generator predicts the coordinate mapping matrix by encoding and decoding the wrinkled document image after background removal.

[0049] The correction and restoration device uses a predicted coordinate mapping matrix to correct the wrinkled document image after background removal, obtaining a corrected document image; specifically: for the wrinkled document image after background removal, a bilinear interpolation algorithm is used ( Figure 1 The symbol S) is combined with the predicted coordinate mapping matrix to perform pixel mapping and obtain the corrected document image.

[0050] In this embodiment of the invention, the corrective restorer does not participate in the training during the fine-tuning phase.

[0051] The fine-tuning phase trains the coordinate mapping matrix generator based on the difference between the predicted coordinate mapping matrix and the actual coordinate mapping matrix. Specifically, this involves calculating the difference between the predicted coordinate mapping matrix f and the actual coordinate mapping matrix f. gtThe L1 distance between them is used as the second loss function to train the coordinate mapping matrix generator. During training, the actual coordinate mapping matrix f contained in the dataset can be used. gt Information is input as supervisory information into the fine-tuning framework. After training, the fine-tuned framework is called the wrinkled document image correction model.

[0052] 2. Input the wrinkled document image to be corrected into the wrinkled document image correction model. After background removal, predict the coordinate mapping matrix using a coordinate mapping matrix generator. Then, use the predicted coordinate mapping matrix to correct the wrinkled document image after background removal to obtain the corrected document image I. c .

[0053] The processing flow here is the collaborative processing flow of the background removal module, coordinate mapping matrix and correction restorer in the fine-tuning framework introduced earlier.

[0054] The method provided in this invention can efficiently utilize a large-scale unlabeled dataset of wrinkled document images for pre-training, thereby extracting more effective document deformation representations in the subsequent fine-tuning stage and improving the correction effect on deformed document images. Furthermore, this invention is applicable to document images of various languages ​​and document layouts, eliminating the need for specialized extraction and training of features such as text lines. It can also improve the correction effect by adding more unlabeled data for pre-training, thus enhancing the scalability of the solution. In addition, compared to previous neural network-based techniques, this invention, after two stages of training, can reduce computational overhead while achieving the same document image correction effect. Therefore, this invention can be integrated into smartphones and other smart devices with cameras, facilitating its wider application in various scenarios.

[0055] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, smartphone, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0056] Example 2

[0057] This invention also provides a self-supervised image correction system for wrinkled documents, which is mainly based on the method provided in the foregoing embodiments, such as... Figure 2 As shown, the system mainly includes:

[0058] A framework construction and training unit is used to construct and train a wrinkled document image correction framework, which includes a pre-trained framework and a fine-tuning framework. In the pre-training phase, the original wrinkled document image is input into the pre-training framework, and after background removal and random masking, a reconstruction restorer composed of a first encoder and a first decoder outputs a reconstructed wrinkled document image. The reconstruction restorer is trained using a self-supervised approach. In the fine-tuning phase, the original wrinkled document image is input into the fine-tuning framework, and after background removal, a coordinate mapping matrix composed of a second encoder and a second decoder predicts the coordinate mapping matrix. The coordinate mapping matrix is ​​trained and tuned based on the difference between the predicted and actual coordinate mapping matrices. After training, the fine-tuning framework is called the wrinkled document image correction model. The parameters of the second encoder in the coordinate mapping matrix are initialized using the parameters of the first encoder in the pre-trained reconstruction restorer.

[0059] The wrinkled document image correction unit is used to input the wrinkled document image to be corrected into the wrinkled document image correction model, perform background removal processing, predict the coordinate mapping matrix through the coordinate mapping matrix generator, and then use the predicted coordinate mapping matrix to correct the wrinkled document image after background removal processing to obtain the corrected document image.

[0060] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0061] Example 3

[0062] The present invention also provides a processing device, such as Figure 3 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0063] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0064] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0065] Input devices can be touchscreens, smartphones, image acquisition devices, physical buttons, or mice, etc.

[0066] The output device can be a display terminal;

[0067] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0068] Example 4

[0069] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0070] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0071] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A self-supervised method for correcting wrinkled document images, characterized in that, include: A wrinkled document image correction framework is constructed and trained. This framework includes a pre-trained framework and a fine-tuning framework. In the pre-training phase, the original wrinkled document image is input into the pre-training framework. After background removal and random masking, a reconstruction restorer composed of a first encoder and a first decoder outputs a reconstructed wrinkled document image. The reconstruction restorer is trained using a self-supervised approach. In the fine-tuning phase, the original wrinkled document image is input into the fine-tuning framework. After background removal, a coordinate mapping matrix composed of a second encoder and a second decoder predicts the coordinate mapping matrix. The coordinate mapping matrix is ​​trained based on the difference between the predicted and actual coordinate mapping matrices. After training, the fine-tuning framework is called the wrinkled document image correction model. The parameters of the second encoder in the coordinate mapping matrix are initialized using the parameters of the first encoder in the pre-trained reconstruction restorer. The wrinkled document image to be corrected is input into the wrinkled document image correction model. After background removal processing, the coordinate mapping matrix is ​​predicted by the coordinate mapping matrix generator. Then, the predicted coordinate mapping matrix is ​​used to correct the wrinkled document image after background removal processing to obtain the corrected document image.

2. The self-supervised method for correcting wrinkled document images according to claim 1, characterized in that, The pre-trained framework includes: a background removal module, a random masking module, and a reconstruction restorer; wherein: The background removal module removes the background area of ​​the original wrinkled document image through background removal processing and outputs the wrinkled document image after background removal processing. The random masking module performs random masking on the wrinkled document image after background removal and outputs the masked image. The reconstruction restorer outputs a reconstructed image of the wrinkled document by encoding and decoding the masked image.

3. A self-supervised method for correcting wrinkled document images according to claim 1 or 2, characterized in that, The method of training the reconstruction restorer using a self-supervised approach includes: The L2 distance between the reconstructed wrinkled document image and the wrinkled document image after background removal is calculated and used as the first loss function to train the reconstruction restorer.

4. The self-supervised method for correcting wrinkled document images according to claim 1, characterized in that, The fine-tuning framework includes: a background removal module, a coordinate mapping matrix, and a correction and restoration module; wherein: The background removal module removes the background area of ​​the original wrinkled document image through background removal processing and outputs the wrinkled document image after background removal processing. The coordinate mapping matrixer performs encoding and decoding processing on the wrinkled document image after background removal to predict the coordinate mapping matrix. The correction restorer uses a predicted coordinate mapping matrix to correct the wrinkled document image after background removal processing, and obtains the corrected document image; the correction restorer does not participate in the fine-tuning stage.

5. A self-supervised method for correcting wrinkled document images according to claim 1 or 4, characterized in that, The step of correcting the wrinkled document image after background removal using the predicted coordinate mapping matrix to obtain the corrected document image includes: For the wrinkled document image after background removal, a bilinear interpolation algorithm combined with a predicted coordinate mapping matrix is ​​used to perform pixel mapping to obtain the corrected document image.

6. A self-supervised method for correcting wrinkled document images according to claim 1 or 4, characterized in that, The process of training the coordinate mapping matrixer based on the difference between the predicted and actual coordinate mapping matrices includes: The L1 distance between the predicted coordinate mapping matrix and the actual coordinate mapping matrix is ​​calculated and used as a second loss function to train the coordinate mapping matrix generator.

7. A self-supervised image correction system for wrinkled documents, characterized in that, Based on the method described in any one of claims 1 to 6, the system comprises: A framework construction and training unit is used to construct and train a wrinkled document image correction framework, which includes a pre-trained framework and a fine-tuning framework. In the pre-training phase, the original wrinkled document image is input into the pre-training framework, and after background removal and random masking, a reconstruction restorer composed of a first encoder and a first decoder outputs a reconstructed wrinkled document image. The reconstruction restorer is trained in a self-supervised manner. In the fine-tuning phase, the original wrinkled document image is input into the fine-tuning framework, and after background removal, a coordinate mapping matrix composed of a second encoder and a second decoder predicts the coordinate mapping matrix. The coordinate mapping matrix is ​​trained based on the difference between the predicted and actual coordinate mapping matrices. After training, the fine-tuning framework is called the wrinkled document image correction model. The parameters of the second encoder in the coordinate mapping matrix are initialized using the parameters of the first encoder in the pre-trained reconstruction restorer. The wrinkled document image correction unit is used to input the wrinkled document image to be corrected into the wrinkled document image correction model, perform background removal processing, predict the coordinate mapping matrix through the coordinate mapping matrix generator, and then use the predicted coordinate mapping matrix to correct the wrinkled document image after background removal processing to obtain the corrected document image.

8. A processing apparatus, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 6.

9. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images