Document image correction method and device, equipment, storage medium and computer program product

Abstract

The invention relates to the technical field of image processing, and discloses a document image correction method, device and equipment, a storage medium and a computer program product.The method comprises the steps that a to-be-corrected document image is input into a document image correction model, a mask guiding module is arranged in the document image correction model, and the mask guiding module is used for guiding the to-be-corrected document image to be corrected; guiding the attention of the document image correction model to focus on a foreground region of the to-be-corrected document image through a mask guiding module, obtaining a distortion field, and correcting the to-be-corrected document image based on the distortion field to obtain a corrected document image; according to the application, the mask guide module is used for promoting the model to strengthen the perception of the geometrical characteristics of the foreground elements, so that the model preferentially pays attention to the foreground elements which are crucial for understanding the document content when processing the document image, the attention to background noise is reduced, and the user experience is improved. Therefore, the problem that foreground elements are easy to distort when a related method is used for processing complex geometric distortion is solved, the accuracy of a correction result is improved, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technologies, and in particular, to a method, device, equipment, storage medium, and computer program product for document image correction. Background Art

[0002] With the popularization of mobile devices and the improvement of built-in camera technologies, document digitization has gradually penetrated into people's daily lives. Compared with traditional scanning devices, smartphones and portable cameras provide a more convenient and flexible way to capture electronic documents. However, limited by the shooting angle and the physical state of the document, the captured images usually have problems such as perspective distortion and geometric deformation. These deformations not only affect the visual effect of the image but also cause significant interference to downstream tasks such as subsequent OCR (Optical Character Recognition) and document structure analysis. With the rapid development of deep learning methods, data-driven learnable methods have been introduced to solve the geometric correction problem of distorted document images.

[0003] However, when dealing with complex geometric distortions, foreground elements in related document image correction methods are prone to distortion, resulting in the defects of low correction accuracy and poor user experience. Summary of the Invention

[0004] The main purpose of the present application is to provide a method, device, equipment, storage medium, and computer program product for document image correction, aiming to solve the technical problem that in related document image correction methods, foreground elements are prone to distortion when dealing with complex geometric distortions, resulting in the defects of low correction accuracy and poor user experience.

[0005] To achieve the above purpose, the present application provides a method for document image correction, and the method for document image correction includes:

[0006] Input the document image to be corrected into a document image correction model, where a mask guidance module is set in the document image correction model;

[0007] Guide the attention of the document image correction model by the mask guidance module to focus on the foreground area of the document image to be corrected, and obtain a distortion deformation field;

[0008] Correct the document image to be corrected based on the distortion deformation field to obtain a corrected document image.

[0009] Optionally, a foreground segmentation module is also set in the document image correction model; before guiding the attention of the document image correction model by the mask guidance module to focus on the foreground area of the document image to be corrected and obtaining a distortion deformation field, it further includes:

[0010] The foreground segmentation module performs foreground segmentation on the document image to be corrected to obtain a predicted foreground mask;

[0011] Correspondingly, guiding the attention of the document image correction model to the foreground area of the document image to be corrected by the mask guiding module to obtain a distortion deformation field includes:

[0012] Based on the predicted foreground mask, guiding the attention of the document image correction model to the foreground area of the document image to be corrected by the mask guiding module to obtain a distortion deformation field.

[0013] Optionally, the foreground segmentation module performing foreground segmentation on the document image to be corrected to obtain a predicted foreground mask includes:

[0014] The foreground segmentation module performs foreground segmentation on the document image to be corrected to obtain a segmentation mask;

[0015] Applying a normalization operation with a smoothing coefficient to the segmentation mask to obtain per-pixel class probabilities;

[0016] Calculating the expected value of the class probabilities and calculating the predicted foreground mask of the document image to be corrected according to the expected value.

[0017] Optionally, a feature extraction module and a Transformer encoder are further provided in the document image correction model; before guiding the attention of the document image correction model to the foreground area of the document image to be corrected based on the predicted foreground mask to obtain a distortion deformation field, it further includes:

[0018] The feature extraction module extracts target features from the document image to be corrected;

[0019] The Transformer encoder encodes the target features to obtain a feature sequence;

[0020] Correspondingly, guiding the attention of the document image correction model to the foreground area of the document image to be corrected based on the predicted foreground mask to obtain a distortion deformation field includes:

[0021] Based on the feature sequence and the predicted foreground mask, guiding the attention of the document image correction model to the foreground area of the document image to be corrected by the mask guiding module to obtain a distortion deformation field.

[0022] Optionally, before inputting the document image to be corrected into the document image correction model, it further includes:

[0023] Obtain an undistorted original image and preprocess the undistorted original image to obtain training data;

[0024] Obtain the true backward deformation field during the preprocessing of the undistorted original image;

[0025] Using the true backward deformation field as the training target, train a document image correction model based on the training data to obtain a trained document image correction model;

[0026] Correspondingly, the inputting the document image to be corrected into the document image correction model includes:

[0027] Input the document image to be corrected into the trained document image correction model.

[0028] Optionally, the training data includes a distorted image, a distorted foreground, and distorted lines; the obtaining an undistorted original image and preprocessing the undistorted original image to obtain training data includes:

[0029] Obtain an undistorted original image and perform foreground and background segmentation on the undistorted original image to obtain a foreground mask;

[0030] Extract line elements from the undistorted original image to obtain line elements;

[0031] Apply the deformation field to the undistorted original image, the foreground mask, and the line elements respectively to obtain a distorted image, a distorted foreground, and distorted lines.

[0032] Optionally, the obtaining an undistorted original image and performing foreground and background segmentation on the undistorted original image to obtain a foreground mask includes:

[0033] Obtain an undistorted original image and perform foreground and background segmentation on the undistorted original image through a fine-tuned Hi-SAM model to obtain foreground elements, where the foreground elements include at least one of text, straight lines, and pictures;

[0034] Generate a foreground mask for the undistorted original image based on the foreground elements.

[0035] Optionally, the line elements include text lines and table lines; the extracting line elements from the undistorted original image to obtain line elements includes:

[0036] Detect text lines in the undistorted original image through a character recognition engine and use the mid-bisector of the text detection box as the text line;

[0037] Remove abnormal lines through a preset edge detection model and a preset straight line detection model to obtain table lines.

[0038] Optionally, before applying the deformation fields to the undistorted original image, the foreground mask, and the line elements respectively to obtain the distorted image, the distorted foreground, and the distorted lines, it further includes:

[0039] Obtain the true backward deformation field, and inversely solve the corresponding true forward deformation field based on the true backward deformation field;

[0040] Correspondingly, applying the deformation fields to the undistorted original image, the foreground mask, and the line elements respectively to obtain the distorted image, the distorted foreground, and the distorted lines includes:

[0041] Apply the true forward deformation field to the undistorted original image, the foreground mask, and the line elements respectively to obtain the distorted image, the distorted foreground, and the distorted lines.

[0042] Optionally, training the document image correction model based on the training data with the true backward deformation field as the training target to obtain the trained document image correction model includes:

[0043] Input the distorted image into the document image correction model to obtain the predicted backward deformation field output by the document image correction model;

[0044] Calculate the deformation field regression loss of the document image correction model based on the predicted backward deformation field and the true backward deformation field;

[0045] Obtain the segmentation mask when the document image correction model outputs the predicted backward deformation field, and calculate the mask loss of the document image correction model based on the segmentation mask and the distorted foreground;

[0046] Calculate the curvature consistency loss of the document image correction model with the distorted lines as the sampling control points;

[0047] Calculate the total loss of the document image correction model according to the deformation field regression loss, the mask loss, and the curvature consistency loss;

[0048] Adjust the model parameters of the document image correction model based on the total loss to obtain the trained document image correction model.

[0049] Optionally, calculating the curvature consistency loss of the document image correction model with the distorted lines as the sampling control points includes:

[0050] Perform uniform sampling with the distorted lines as the sampling control points to obtain a discrete point set;

[0051] Project the discrete point set onto the predicted backward deformation field and the true backward deformation field respectively to obtain the predicted control points and the true control points;

[0052] Calculate the discrete point curvature difference based on the predicted control points and the true control points to obtain the curvature consistency loss of the document image correction model.

[0053] In addition, to achieve the above object, the present application also proposes a document image correction device, which includes:

[0054] An image input module for inputting a document image to be corrected into the document image correction model, wherein a mask guidance module is provided in the document image correction model;

[0055] A model guidance module for guiding the attention of the document image correction model to the foreground area of the document image to be corrected through the mask guidance module to obtain a distortion deformation field;

[0056] A document correction module for correcting the document image to be corrected based on the distortion deformation field to obtain a corrected document image.

[0057] Optionally, a foreground segmentation module is further provided in the document image correction model; the document image correction device further includes:

[0058] A mask prediction module for performing foreground segmentation on the document image to be corrected through the foreground segmentation module to obtain a predicted foreground mask;

[0059] Correspondingly, the document correction module is further configured to guide the attention of the document image correction model to the foreground area of the document image to be corrected through the mask guidance module based on the predicted foreground mask to obtain a distortion deformation field.

[0060] Optionally, the mask prediction module is further configured to perform foreground segmentation on the document image to be corrected through the foreground segmentation module to obtain a segmentation mask; perform a normalization operation with a smoothing coefficient on the segmentation mask to obtain a per-pixel class probability; calculate the expected value of the class probability, and calculate the predicted foreground mask of the document image to be corrected according to the expected value.

[0061] Optionally, a feature extraction module and a Transformer encoder are further provided in the document image correction model; the document image correction device further includes:

[0062] A sequence extraction module for extracting target features from the document image to be corrected through the feature extraction module; encoding the target features through the Transformer encoder to obtain a feature sequence;

[0063] Accordingly, the document correction module is further configured to focus the attention of the document image correction model on the foreground area of the document image to be corrected based on the feature sequence and the predicted foreground mask through the mask guidance module, so as to obtain a distortion deformation field.

[0064] Optionally, the document image correction device further includes:

[0065] A model training module, configured to obtain an undistorted original image, preprocess the undistorted original image to obtain training data; obtain a true backward deformation field during the preprocessing of the undistorted original image; use the true backward deformation field as a training target, and train a document image correction model based on the training data to obtain a trained document image correction model;

[0066] Accordingly, the image input module is further configured to input the document image to be corrected into the trained document image correction model.

[0067] Optionally, the training data includes a distorted image, a distorted foreground, and a distorted line; the model training module is further configured to obtain an undistorted original image, perform foreground and background segmentation on the undistorted original image to obtain a foreground mask; extract line elements from the undistorted original image to obtain line elements; apply the deformation field to the undistorted original image, the foreground mask, and the line elements respectively to obtain a distorted image, a distorted foreground, and a distorted line.

[0068] In addition, to achieve the above object, the present application further provides a document image correction device, where the document image correction device includes a memory, a processor, and a document image correction program stored on the memory and executable on the processor, and the document image correction program is configured to implement the document image correction method as described above.

[0069] In addition, to achieve the above object, the present application further provides a storage medium, on which a document image correction program is stored, and when the document image correction program is executed by a processor, it implements the document image correction method as described above.

[0070] In addition, to achieve the above object, the present application further provides a computer program product, where the computer program product includes a document image correction program, and when the document image correction program is executed by a processor, it implements the document image correction method as described above.

[0071] One or more technical solutions proposed by the present application have at least the following technical effects:

[0072] In this application, it is disclosed that a document image to be corrected is input into a document image correction model. Among them, a mask guidance module is set in the document image correction model. Through the mask guidance module, the attention of the document image correction model is guided to focus on the foreground area of the document image to be corrected, a distortion deformation field is obtained, and the document image to be corrected is corrected based on the distortion deformation field to obtain a corrected document image. Since this application uses the mask guidance module to prompt the model to strengthen the perception of the geometric features of foreground elements, when the model processes document images, it gives priority to paying attention to foreground elements that are crucial for understanding document content and reduces the attention to background noise. Thus, the problem that foreground elements are prone to distortion when dealing with complex geometric distortions in related methods is solved, the accuracy of the correction result is improved, and the user experience is further enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The accompanying drawings here are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with this application, and are used together with the specification to explain the principles of this application.

[0074] To more clearly illustrate the technical solutions in the embodiments of this application or in the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0075] Figure 1 It is a schematic flowchart of the first embodiment of the document image correction method of this application;

[0076] Figure 2 It is a schematic flowchart of the second embodiment of the document image correction method of this application;

[0077] Figure 3 It is a schematic diagram of the document image correction model of an embodiment of the document image correction method of this application;

[0078] Figure 4 It is a schematic flowchart of the third embodiment of the document image correction method of this application;

[0079] Figure 5 It is a schematic diagram of the training of the document image correction model of an embodiment of the document image correction method of this application;

[0080] Figure 6 It is a schematic diagram of the module structure of the document image correction device in the embodiment of this application;

[0081] Figure 7 It is a schematic diagram of the device structure of the hardware operating environment involved in the document image correction method in the embodiment of this application.

[0082] The realization of the purpose of this application, its functional features and advantages will be further described in conjunction with embodiments with reference to the accompanying drawings. Detailed implementation manners

[0083] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of this application and are not used to limit this application.

[0084] To better understand the technical solutions of this application, the following will be described in detail in conjunction with the accompanying drawings of the specification and specific implementation manners.

[0085] With the popularization of mobile devices and the improvement of built-in camera technology, document digitization has gradually penetrated into people's daily lives. Compared with traditional scanning devices, smartphones and portable cameras provide a more convenient and flexible way to capture electronic documents. However, limited by the shooting angle and the physical state of the document, the captured images usually have problems such as perspective distortion and geometric deformation. These deformations not only affect the visual effect of the image but also cause significant interference to downstream tasks such as subsequent OCR (Optical Character Recognition) and document structure analysis. With the rapid development of deep learning methods, data-driven learnable methods have been introduced to solve the geometric correction problem of distorted document images.

[0086] However, when dealing with complex geometric distortions, the foreground elements of related document image correction methods are prone to distortion, resulting in the defects of low correction accuracy and poor user experience.

[0087] Therefore, to overcome the above defects, this application provides a solution, which includes: inputting the document image to be corrected into a document image correction model, where a mask guidance module is set in the document image correction model. The mask guidance module guides the attention of the document image correction model to focus on the foreground area of the document image to be corrected, obtaining a distortion deformation field, and correcting the document image to be corrected based on the distortion deformation field to obtain a corrected document image; because this application uses the mask guidance module to prompt the model to strengthen the perception of the geometric features of the foreground elements, when the model processes the document image, it gives priority to focusing on the foreground elements that are crucial for understanding the document content and reduces the attention to background noise, thus solving the problem that the foreground elements are prone to distortion when dealing with complex geometric distortions in related methods, improving the accuracy of the correction result, and further enhancing the user experience.

[0088] It should be noted that the execution subject of this embodiment can be a document image correction device with data processing, network communication, and program running functions, such as a server, a computer, a terminal device, etc., or other electronic devices that can achieve the same or similar functions. This embodiment does not limit this.

[0089] Based on this, the embodiments of the present application provide a method for correcting a document image. Referring to Figure 1 , Figure 1 is a schematic flowchart of the first embodiment of the document image correction method of the present application.

[0090] In the first embodiment, the document image correction method includes:

[0091] Step S10: Input the document image to be corrected into the document image correction model, where a mask guidance module is set in the document image correction model.

[0092] It should be understood that the document image to be corrected may refer to a digital document image with geometric deformations (such as bending, folding, perspective distortion), which can be captured by a mobile device or generated by a scanner. For example, a photo of a bent paper document taken by a mobile phone, etc. The document image correction model may refer to a model based on deep learning or image processing technology, which is used to correct a distorted document image into an image close to the original flat state. The mask guidance module may refer to a component in the document image correction model that can guide the model to pay more attention to the foreground elements (such as text, lines, and pictures, etc.) in the document, thereby improving the correction effect.

[0093] In a specific implementation, the document image to be corrected is input as an input to the trained document image correction model, and the document image correction model can analyze and process the input image.

[0094] Step S20: Through the mask guidance module, guide the attention of the document image correction model to focus on the foreground area of the document image to be corrected, and obtain a distortion deformation field.

[0095] It can be understood that the foreground area may refer to the area in the document image that contains important information (such as text, charts, lines, etc.), and these areas need to be focused on in the task of removing distortion. The distortion deformation field may refer to a vector field that describes the displacement required for each pixel point in the document image from the distorted state to the flat state, and is the basis for image correction.

[0096] In a specific implementation, the mask guidance module first obtains the predicted foreground mask of the document image to be corrected. The predicted foreground mask identifies the foreground regions in the document image to be corrected. Then, the mask guidance module uses the predicted foreground mask to guide the attention of the document image correction model, enabling the document image correction model to pay more attention to the foreground regions when processing the image (for example, by adding a term related to the predicted foreground mask to the loss function of the document image correction model, so as to ensure that the model learns the importance of the foreground regions during the training process). Under the guidance of the mask guidance module, through the key analysis of the foreground regions, the document image correction model can predict the distortion deformation field from the distorted state to the flat state. The distortion deformation field describes the direction and distance that each pixel point in the image needs to move.

[0097] Step S30: Correct the document image to be corrected based on the distortion deformation field to obtain the corrected document image.

[0098] It should be understood that the corrected document image can refer to the document image that has been processed by the document image correction model and restored to a state close to the original flat state.

[0099] In a specific implementation, the document image to be corrected is corrected through the distortion deformation field to obtain a document image close to the original flat state. The foreground elements such as text, straight lines, and pictures in this corrected image become clearer and neater, greatly improving the readability of the document.

[0100] In this embodiment, the mask guidance module prompts the model to strengthen the perception of the geometric features of the foreground elements, enabling the model to give priority to the foreground elements that are crucial for understanding the document content when processing the document image, and reducing the attention to background noise. Thus, the problem that the foreground elements are prone to distortion when dealing with complex geometric distortions in related methods is solved, the accuracy of the correction result is improved, and the user experience is further enhanced.

[0101] Refer to Figure 2 , Figure 2 which is the flowchart of the second embodiment of the document image correction method of this application. Based on the first embodiment shown above Figure 1 a second embodiment of the document image correction method of this application is proposed.

[0102] In the second embodiment, a foreground segmentation module is further provided in the document image correction model; before step S20, it further includes:

[0103] Step S11: Perform foreground segmentation on the document image to be corrected through the foreground segmentation module to obtain the predicted foreground mask.

[0104] Correspondingly, step S20 includes:

[0105] Step S20': Based on the predicted foreground mask, the mask guidance module guides the attention of the document image correction model to focus on the foreground area of the document image to be corrected, and a distortion deformation field is obtained.

[0106] It should be understood that, in order to improve the guiding effect of the mask guidance module, in this embodiment, a foreground segmentation module is further provided in the document image correction model. The foreground segmentation module performs foreground segmentation on the document image to be corrected to obtain a predicted foreground mask. Based on the predicted foreground mask, the mask guidance module guides the attention of the document image correction model to focus on the foreground area of the document image to be corrected, and a distortion deformation field is obtained.

[0107] Further, the step S11 includes: performing foreground segmentation on the document image to be corrected through the foreground segmentation module to obtain a segmentation mask; performing a normalization operation with a smoothing coefficient on the segmentation mask to obtain a per-pixel class probability; calculating the expected value of the class probability, and calculating the predicted foreground mask of the document image to be corrected according to the expected value.

[0108] Further, a feature extraction module and a Transformer encoder are further provided in the document image correction model. Before the step S20', it further includes: extracting target features from the document image to be corrected through the feature extraction module; encoding the target features through the Transformer encoder to obtain a feature sequence; correspondingly, the step S20' includes: based on the feature sequence and the predicted foreground mask, the mask guidance module guides the attention of the document image correction model to focus on the foreground area of the document image to be corrected, and a distortion deformation field is obtained.

[0109] For ease of understanding, reference is made to Figure 3 for illustration, but it does not limit the present application. Figure 3 It is a schematic diagram of a document image correction model for an embodiment of the document image correction method of the present application. As an example, Figure 5 in, it is assumed that the document image I to be corrected u is used as the input. The goal of document image undistortion is to predict the pixel-level displacement, that is, to predict the backward deformation field. In previous undistortion methods, pixels of text, line elements, and the background are all treated equally, lacking consideration of the correlation between foreground pixels in the undistortion task. In this embodiment, with the help of the generated distortion mask, the model is guided to pay more attention to foreground information to fit the application scenario of the undistortion task, that is, the readability and neatness of the text. As Figure 3 shown, the main body of the document image correction model consists of four parts, the CNNs feature extraction module (corresponding to Figure 3The CNN in it, namely the feature extraction module), the efficient Transformer Encoder (corresponding to Figure 3 the Efficient Transformer Encoder in it, namely the Transformer encoder), the lightweight foreground segmentation model (corresponding to Figure 3 the Lightweight Segmentation in it, namely the foreground segmentation module), and the mask-guided Transformer Decoder (corresponding to Figure 3 the Mask-Guided Transformer Decoder in it, namely the mask-guided module). In particular, the foreground detection model is connected as a plug-and-play module to the result output of the Transformer Encoder, and the predicted foreground mask is used for feature guidance of the Transformer Decoder.

[0110] The specific steps of inputting the document image to be corrected into the document image correction model to obtain the distortion deformation field output by the document image correction model are as follows:

[0111] 1. CNNs feature extraction module: Given a distorted document image I u ∈R H×W×3 , first adjust it to the model input image with H i =W i =1024 and C i =3, and then send it to the convolutional layer with a large kernel and multiple stacked residual layers. The feature extraction module extracts features from I u and represents them as In this embodiment, C is set to 256.

[0112] 2. Efficient Transformer Encoder: To improve the global correlation between features, this embodiment uses the vanilla Transformer to further encode the features. This encoder consists of three Transformer layers. Overlap Patch Embedding (i.e., corresponding to Figure 3 the Overlap Patch in it) is used between the Transformer layers. Non-overlap Patch Embedding will result in the loss of features of the text edge pixels. Therefore, Overlap Patch Embedding with a kernel size set to 3 and a stride set to 2 is used to reduce the computational amount of the underlying features. Finally, the Transformer Encoder will output three feature maps {E1, E2, E3}.

[0113] To reduce the complexity of the attention mechanism, this embodiment uses the spatial pooling window strategy on K and V.

[0114] 3. Lightweight foreground segmentation model: Taking the feature sequence {E1, E2, E3} as input, the foreground segmentation model aims to predict the binary mask of the foreground region using a lightweight network design. Specifically, in this embodiment, the feature channels of the multi-scale feature maps are first unified using a convolution with a kernel size of 1, and then the unified features are upsampled to Finally, the merged features are passed through multiple convolutions with a kernel size of 1 to obtain the foreground segmentation result In the foreground segmentation model, C seg is set to 2, corresponding to the foreground and background classes in the warped image respectively. During training, the L1 loss is used to supervise the segmentation mask M, and the ground truth uses the generated mask M gt . The mask loss L seg is calculated as follows:

[0115] L seg = ||M - M gt ||

[0116] To enable the foreground segmentation result M to be differentiably embedded into the subsequent decoder (i.e., corresponding to the Learnable Embeddings in Figure 3 ), this embodiment uses an integration method to calculate the expectation of the segmentation result M. In the key point detection task, this technique is also known as soft argmax. Specifically, this embodiment applies a softmax operation with a smoothing coefficient to the segmentation result M to obtain the per-pixel class probability. Subsequently, the expected value of the class probability is calculated to predict the mask value of the foreground Predict foreground mask is calculated as follows:

[0117]

[0118] where β is the smoothing coefficient, and softmax ensures that the sum of the elements of M is 1, so that when taking the expected value to estimate the mask value, it can be directly used as the probability density.

[0119] 4. Mask-guided Transformer Decoder: In this embodiment, the Transformer Decoder is used for mask guidance and feature extraction of the foreground region. The Transformer Decoder takes the feature sequence {E1, E2, E3} and the foreground mask as input and uses a fixed number of learnable embeddings Capture the distortion information of the distorted image layer by layer and output the distortion deformation field. The same as the number of decoder layers, this encoder is composed of three Transformer layers. The Transformer layers are connected by upsampling operations. Inside the Transformer layer, it is composed of mask-guided self-attention and encoder-decoder cross-attention. For the mask-guided self-attention, in this embodiment, the Embeddings D of the encoder i is used as the input of the attention, and attention will be serialized and unfolded to restrict the self-attention module to focus on the text area and line elements. For the encoder-decoder cross-attention, in this embodiment, the embeddings of the decoder are used as the query, and the embeddings of the encoder in the same layer are used as the key and value.

[0120]

[0121] CA(Q, K, V) = CrossAttention(D i-1 , E i , E i )

[0122] where MSA represents mask-guided self-attention and σ represents the scaling factor. seq represents sequence unfolding along the feature dimension. CA represents encoder-decoder cross-attention, D i-1 represents the output of the i-1 decoder layer, and E i represents the output of the i encoder layer. To reduce the complexity of the attention mechanism, in this embodiment, the K and V of the decoder are also reduced in the spatial dimension. Finally, in this embodiment, the upsampling methods in DocTr and GeoDewarpNet (i.e., Upsampling and Upsampling Head corresponding to Figure 3 ) are used to obtain a high-resolution distortion deformation field This upsampling method optimizes its weights based on the initialized grid coordinates to train learnable weights. The document image to be corrected is corrected through the distortion deformation field to obtain a document image close to the original flat state.

[0123] In this embodiment, the mask-guided module can achieve the following effects:

[0124] 1. Optimization objective focus: Redefine the optimization direction of the de-distortion task from the model architecture level and ingeniously integrate the mask guidance module into the model. Through the generated distortion mask, this module precisely guides the model to highly concentrate its attention on foreground information. When processing document images, the model preferentially focuses on text, straight lines, and picture areas that are crucial for understanding the document content, reduces attention to background noise, and greatly improves the model's attention and processing accuracy for foreground elements.

[0125] 2. Enhance regional discrimination ability: Effectively strengthen the model's ability to distinguish between readable regions and background regions. During the de-distortion process of document images, the mask guidance module enables the model to accurately identify and distinguish between foreground and background, avoiding interference or misprocessing of foreground elements when processing the background region, thereby significantly improving the quality of the corrected image, ensuring that foreground elements remain clear and complete after de-distortion, and greatly enhancing the readability of the document.

[0126] 3. Improve geometric perception performance: The mask guidance mechanism prompts the model to strengthen its perception of the geometric features of foreground elements. When facing complex deformations of document images, the model can more accurately capture the geometric shapes and position changes of foreground elements with the help of mask guidance, and thus more precisely restore the original forms of foreground elements during the de-distortion process, effectively solving the problem that foreground elements are prone to distortion when traditional methods handle complex geometric distortions, and improving the accuracy and reliability of the correction results.

[0127] In the document image correction model of this embodiment, a foreground segmentation module is also provided. The foreground segmentation module performs foreground segmentation on the document image to be corrected to obtain a predicted foreground mask. Based on the predicted foreground mask, the mask guidance module guides the attention of the document image correction model to focus on the foreground region of the document image to be corrected, obtaining a distortion deformation field, thereby being able to improve the guiding effect of the mask guidance module and further improving the accuracy of the correction result.

[0128] Refer to Figure 4 , Figure 4 which is a schematic flowchart of the third embodiment of the document image correction method of this application. Based on the above embodiments, the third embodiment of the document image correction method of this application is proposed.

[0129] In the third embodiment, before the step S10, it further includes:

[0130] Step S01: Obtain an undistorted original image and preprocess the undistorted original image to obtain training data.

[0131] It should be understood that, different from the classical problems in computer vision (such as image classification, object detection, image segmentation, etc.), the refined annotation of document bending correction is extremely difficult. The limited training data confines the model to a low generalization ability. Therefore, in order to overcome the above defects, in this embodiment, the original undistorted image can be directly preprocessed to obtain training data, and the document image correction model can be trained based on the training data to obtain the trained document image correction model.

[0132] It can be understood that the original undistorted image can refer to the original document image that has not been distorted or deformed, and the original undistorted image can be used as a reference or benchmark for subsequent processing or analysis. Preprocessing can refer to a series of operations or conversions performed on the data before it is input into the model, aiming to obtain accurate training samples. In a specific implementation, the preprocessing can include operations such as character-level foreground and background segmentation, line element extraction, and distortion field generation.

[0133] Furthermore, in order to improve the training effect of the model, in this embodiment, the training data includes a distorted image, a distorted foreground, and a distorted line. The step S01 includes: obtaining the original undistorted image, performing foreground and background segmentation on the original undistorted image to obtain a foreground mask; extracting line elements from the original undistorted image to obtain line elements; applying the deformation field to the original undistorted image, the foreground mask, and the line elements respectively to obtain a distorted image, a distorted foreground, and a distorted line.

[0134] It can be understood that foreground and background segmentation can refer to the process of dividing an image into a foreground and a background. In a document image, the foreground can refer to readable information such as text, lines, pictures, etc., while the background is the part other than the foreground. The foreground mask can refer to a binary image used to represent the foreground area. The foreground area is marked as white (or 1), and the background area is marked as black (or 0). Line elements in a document image can refer to straight elements such as text lines and table lines, which are crucial for maintaining the readability and structural integrity of the document. The deformation field can refer to the displacement mapping of each pixel in the image from the original position to the corrected position. The deformation field can be divided into a forward deformation field (Forward Mapping, FM) and a backward deformation field (Backward Mapping, BM). The forward deformation field represents the mapping relationship from the original undistorted image to the distorted image. The backward deformation field is opposite to the forward deformation field, and it represents the mapping relationship from the distorted image to the original undistorted image. The distorted image can refer to the distorted image obtained by applying the deformation field to the original undistorted image, which is used to simulate the distorted document in the real scene. The distorted foreground and the distorted line respectively refer to the distorted results obtained by applying the deformation field to the foreground mask and the line elements, which are used to train the image correction model.

[0135] In a specific implementation, an image segmentation technique (such as a fine-tuned Hi-SAM model) is used to segment the undistorted original image to obtain a foreground mask. A text line and line elements such as table lines are extracted using an OCR engine and an edge detection method. The deformation field is applied to the undistorted original image, the foreground mask, and the line elements respectively to obtain a distorted image, a distorted foreground, and distorted lines.

[0136] Further, in order to enable the model to capture richer key information in the document, avoid missing important content that may affect document understanding and processing, and comprehensively improve the perception ability of the document structure and content, in this embodiment, text, straight lines, and pictures are clearly defined as foreground elements. The obtaining of the undistorted original image and performing foreground and background segmentation on the undistorted original image to obtain a foreground mask includes: obtaining the undistorted original image, and performing foreground and background segmentation on the undistorted original image through a fine-tuned Hi-SAM model to obtain foreground elements, where the foreground elements include at least one of text, straight lines, and pictures; generating a foreground mask of the undistorted original image based on the foreground elements.

[0137] It should be understood that the Hi-SAM model can refer to an advanced image segmentation model that can be used for foreground and background segmentation of document images after fine-tuning, that is, to distinguish foreground elements (such as text, straight lines, pictures, etc.) and background regions in the document.

[0138] Further, in order to improve the accuracy of line element extraction, the line elements include text lines and table lines; the performing of line element extraction on the undistorted original image to obtain line elements includes: detecting text lines of the undistorted original image through a character recognition engine, and taking the bisector of the text detection box as the text line; removing abnormal lines through a preset edge detection model and a preset line detection model to obtain table lines.

[0139] In a specific implementation, this embodiment uses an OCR engine as a text line extractor and takes the bisector of the text detection box as the text straight line. For the table line elements existing in the document, this embodiment designs a straight line detection method for table types based on a preset edge detection model and a preset line detection model. This method masks the straight line results that are not horizontally and vertically distributed and filters out the repeatedly detected straight lines. Among them, the preset edge detection model can be a Canny edge detection algorithm model, and the preset line detection model can be an LSD straight line detection model.

[0140] Further, in order to improve the reliability of the generation of the distortion field, before applying the deformation field to the original undistorted image, the foreground mask, and the line elements respectively to obtain the distorted image, the distorted foreground, and the distorted lines, it further includes: obtaining a true backward deformation field, and inversely solving the corresponding true forward deformation field based on the true backward deformation field; correspondingly, applying the deformation field to the original undistorted image, the foreground mask, and the line elements respectively to obtain the distorted image, the distorted foreground, and the distorted lines includes: applying the true forward deformation field to the original undistorted image, the foreground mask, and the line elements respectively to obtain the distorted image, the distorted foreground, and the distorted lines.

[0141] For ease of understanding, the following is an example, but it does not limit the present application. As an example, assume that the training data includes a distorted image, a distorted foreground, and distorted lines. The preprocessing of the original undistorted image includes three operations: character-level foreground and background segmentation, line element extraction, and distortion field generation. Due to reasons such as image deformation and lighting effects, there are inevitable errors in directly extracting feature information from the distorted image. Therefore, in this embodiment, fine foreground elements are creatively extracted from the original undistorted image, and a randomly generated distortion field is applied to them to obtain accurate training samples.

[0142] 1. Character-level foreground and background segmentation: In order to obtain the character-level foreground and background segmentation result, in this embodiment, the Hi-SAM model is fine-tuned so that the Hi-SAM model supports segmenting the charts in the document image as the foreground. The fine-tuned Hi-SAM model has obtained sufficient effects for multi-language segmentation scenarios. Input the original undistorted image I r , and the foreground mask M r can be obtained.

[0143] 2. Line element extraction: In this embodiment, the OCR engine is used as the text line extractor, and the mid-bisector of the text detection box is used as the text line. For the table line elements existing in the document, in this embodiment, a straight line detection method for the table type is designed based on a preset edge detection model and a preset straight line detection model. This method masks the straight line results that are not horizontally and vertically distributed and filters out the repeatedly detected straight lines. Among them, the preset edge detection model can be the Canny edge detection algorithm model, and the preset straight line detection model can be the LSD straight line detection model.

[0144] 3. Distorted Field Generation: In this embodiment, the original real backward deformation field (Backward Mapping, BM) is obtained from DOC3D, and the corresponding real forward deformation field (Forward Mapping, FM) is inversely solved using the real backward deformation field. To obtain a rich deformation field, the original real backward deformation field is slightly cropped and randomly overlapped pairwise. Then, the corresponding real forward deformation field is inversely solved using the real backward deformation field. Finally, the obtained real forward deformation field is applied to the original undistorted image I r , foreground mask M r , and line elements respectively to obtain the distorted image I u , distorted foreground M gt and distorted line L gt , and the real backward deformation field BM is used as the training target.

[0145] The extraction and utilization of foreground information in this embodiment can achieve the following effects:

[0146] 1. Definition of Multi-Element Foreground: Breaking through the traditional limitation of only focusing on text lines, text, straight lines, and pictures are innovatively defined as foreground elements. This expansion enables the model to capture richer key information in the document, avoiding the omission of important content that may affect document understanding and processing, and comprehensively enhancing the perception ability of the document structure and content.

[0147] 2. Extraction Strategy of Original Undistorted Image: Since directly extracting element information from distorted pictures is prone to errors due to image deformation and lighting effects, this embodiment takes a unique approach and selects to perform fine foreground element extraction on the original undistorted image. Character-level foreground and background segmentation are achieved by fine-tuning the Hi-SAM model, and text lines and table line elements are extracted using the OCR engine and a customized straight line detection algorithm respectively, ensuring that the obtained foreground information is accurate and reliable, providing high-quality data support for subsequent model training.

[0148] 3. Generation and Application of True Value Labels: A special label preprocessing module is designed to generate corresponding true value labels (i.e., real backward deformation fields) based on the extracted foreground elements. These true value labels play a key role in the model training process, providing an accurate target orientation for the model to learn, guiding the model to effectively utilize foreground information for distortion removal operations, significantly enhancing the model's understanding and processing ability of the foreground region, and thus optimizing the distortion removal effect.

[0149] Step S02: Obtain the real backward deformation field during the preprocessing of the original undistorted image.

[0150] It should be understood that during the preprocessing process, the corresponding true backward deformation field is obtained by recording the applied backward warping field. Among them, the deformation field can refer to the displacement mapping of each pixel in the image from the original position to the corrected position. The deformation field can be divided into a forward deformation field (Forward Mapping, FM) and a backward deformation field (Backward Mapping, BM). The forward deformation field represents the mapping relationship from the original undistorted image to the distorted image. The backward deformation field is opposite to the forward deformation field and represents the mapping relationship from the distorted image to the original undistorted image.

[0151] Step S03: Using the true backward deformation field as the training target, train the document image correction model based on the training data to obtain the trained document image correction model.

[0152] Correspondingly, step S10 includes:

[0153] Step S10': Input the document image to be corrected into the trained document image correction model.

[0154] Furthermore, in order to improve the training effect of the document image correction model, in this embodiment, the document image correction model is trained based on the deformation field regression loss, the mask loss, and the curvature consistency loss. Step S03 includes: inputting the distorted image into the document image correction model to obtain the predicted backward deformation field output by the document image correction model; calculating the deformation field regression loss of the document image correction model based on the predicted backward deformation field and the true backward deformation field; obtaining the segmentation mask when the document image correction model outputs the predicted backward deformation field, and calculating the mask loss of the document image correction model based on the segmentation mask and the distorted foreground; calculating the curvature consistency loss of the document image correction model with the distorted line as the sampling control point; calculating the total loss of the document image correction model according to the deformation field regression loss, the mask loss, and the curvature consistency loss; adjusting the model parameters of the document image correction model based on the total loss to obtain the trained document image correction model.

[0155] It can be understood that the specific steps of inputting the distorted image into the document image correction model to obtain the predicted backward deformation field output by the document image correction model can be to input the distorted image into the document image correction model, where a mask guidance module is set in the document image correction model; the mask guidance module guides the attention of the document image correction model to focus on the foreground area of the distorted image to obtain the predicted backward deformation field. The subsequent steps of the mask guidance module guiding the attention of the document image correction model to focus on the foreground area of the distorted image to obtain the predicted backward deformation field are the same as the specific implementation content proposed in the second embodiment of this application and will not be elaborated here.

[0156] Further, in order to improve the accuracy of the curvature consistency loss, calculating the curvature consistency loss of the document image correction model with the twisted line as the sampling control point includes: uniformly sampling with the twisted line as the sampling control point to obtain a discrete point set; projecting the discrete point set onto the predicted backward deformation field and the true backward deformation field respectively to obtain predicted control points and true control points; calculating the discrete point curvature difference according to the predicted control points and the true control points to obtain the curvature consistency loss of the document image correction model.

[0157] It should be understood that for a document image with geometric deformation, a conventional detection model cannot effectively capture the distorted text lines and straight line elements. After experimentation, without end-to-end embedding of the model foreground prediction into the de-distortion model, complex geometric distortions will cause a sharp drop in the performance of the detection model. And it is extremely expensive to label foreground labels such as text lines for each distorted picture. Therefore, in this embodiment, this embodiment clearly takes text, straight lines, and pictures as foreground elements, and designs a label preprocessing module to generate true value labels for the corresponding elements. At the same time, how to efficiently utilize the generated foreground information is also the focus of this embodiment. The foreground region can represent the key distortion amplitude of the de-distorted picture. Therefore, this embodiment designs a mask guidance module to limit the model's perception ability within the readable region of the document. In addition, this embodiment also realizes that the conventional L1 loss supervision method cannot capture geometric distortions. This embodiment describes the geometric distortion of the distortion field through the curvature value of points, and designs a curvature consistency principle based on control points to supervise the deformation field prediction.

[0158] For ease of understanding, reference is made to Figure 5 for illustration, but it does not limit the present application. Figure 5 This is a schematic diagram of the training of the document image correction model for an embodiment of the document image correction method of the present application. As an example, Figure 5 in, assume the distorted image I u is used as the input. The goal of document image de-distortion is to predict the pixel-level displacement, that is, to predict the backward deformation field. In previous de-distortion methods, pixels of text, line elements, and the background were all treated equally, lacking consideration of the correlation between foreground pixels in the de-distortion task. In this embodiment, with the help of the generated distortion mask, the model is guided to pay more attention to the foreground information to fit the application scenario of the de-distortion task, that is, the readability and neatness of the text. As Figure 5 shown, the main body of the document image correction model consists of four parts: the CNNs feature extraction module (corresponding to the CNN in Figure 5 , that is, the feature extraction module), the efficient Transformer Encoder (corresponding to Figure 5The Efficient Transformer Encoder, i.e., the Transformer encoder), the lightweight foreground segmentation model (corresponding to Figure 5 the Lightweight Segmentation in Figure 5 , i.e., the foreground segmentation module) and the mask-guided Transformer Decoder (corresponding to

[0159] the Mask-Guided Transformer Decoder in

[0160] u ∈R H×W×3 , first adjust it to a model input image with H i =W i =1024 and C i =3, and then send it to a convolutional layer with a large kernel and multiple stacked residual layers. The feature extraction module extracts features from I u , denoted as In this embodiment, C is set to 256.

[0161] 2. Efficient Transformer Encoder: To improve the global correlation between features, this embodiment uses a vanilla Transformer to further encode the features. This encoder consists of three Transformer layers. Overlap patch embedding is used between the Transformer layers (i.e., corresponding to Figure 5 the Overlap Patch in

[0162] Non-overlapping patch embedding will result in the loss of features of text edge pixels. Therefore, overlap patch embedding with a kernel size of 3 is used, and the stride is set to 2 to reduce the computational amount of the underlying features. Finally, the Transformer Encoder will output three feature maps {E1, E2, E3}.

[0163] 3. Lightweight foreground segmentation model: Taking the feature sequence {E1, E2, E3} as input, the foreground segmentation model aims to predict the binary mask of the foreground region using a lightweight network design. Specifically, in this embodiment, the feature channels of the multi-scale feature maps are first unified using a convolution with a kernel size of 1, and then the unified features are upsampled to Finally, the merged features are passed through multiple convolutions with a kernel size of 1 to obtain the foreground segmentation result In the foreground segmentation model, C seg is set to 2, corresponding to the foreground and background categories in the warped image respectively. During training, the L1 loss is used to supervise the segmentation mask M, and the ground truth uses the generated mask M gt . The mask loss L seg is calculated as follows:

[0164]

[0165] To enable the foreground segmentation result M to be differentiably embedded into the subsequent decoder (i.e., corresponding to Figure 5 Learnable Embeddings in), this embodiment uses an integral method to calculate the expectation of the segmentation result M. In the keypoint detection task, this technique is also known as soft argmax. Specifically, this embodiment applies a softmax operation with a smoothing coefficient to the segmentation result M to obtain the per-pixel class probability. Subsequently, the expected value of the class probability is calculated to predict the mask value of the foreground Predict foreground mask is calculated as follows:

[0166]

[0167] where β is the smoothing coefficient, and softmax ensures that the sum of the elements of M is 1, so that when taking the expected value to estimate the mask value, it can be directly used as the probability density.

[0168] 4. Mask-guided Transformer Decoder: In this embodiment, the Transformer Decoder is used for mask guidance and feature extraction of the foreground region. The Transformer Decoder takes the feature sequence {E1, E2, E3} and the foreground mask as input, and uses a fixed number of learnable embeddings Capture the distortion information of the distorted image layer by layer and output the distortion deformation field. The same as the number of decoder layers, this encoder consists of three Transformer layers. The Transformer layers are connected by upsampling operations. Inside the Transformer layer, it is composed of mask-guided self-attention and encoder-decoder cross-attention. For mask-guided self-attention, in this embodiment, the Embeddings D of the encoder i is used as the input of the attention, and attention will be serialized and unfolded to limit the self-attention module to focus on the text area and line elements. For encoder-decoder cross-attention, in this embodiment, the embeddings of the decoder are used as the query, and the embeddings of the encoder in the same layer are used as the key and value.

[0169]

[0170] CA(Q, K, V) = CrossAttention(D i-1 , E i , E i )

[0171] where MSA represents mask-guided self-attention and σ represents the scaling factor. seq represents sequence unfolding along the feature dimension. CA represents encoder-decoder cross-attention, D i-1 represents the output of the i - 1 decoder layer, and E i represents the output of the i encoder layer. To reduce the complexity of the attention mechanism, in this embodiment, the K and V of the decoder are also reduced in the spatial dimension. Finally, in this embodiment, the upsampling methods in DocTr and GeoDewarpNet (i.e., Upsampling and Upsampling Head corresponding to Figure 5 ) are used to obtain the high-resolution predicted backward deformation field This upsampling method optimizes its weights based on the initialized grid coordinates to train learnable weights.

[0172] The steps to calculate the total loss of the document image rectification model are as follows:

[0173] The main model contains three optimization objectives, namely foreground mask loss, backward graph regression, and control line constraint. Among them, the foreground mask loss L seg is designed to supervise the generation of a differentiable foreground mask. The backward graph regression is to calculate the predicted backward deformation field The L1 distance from the true backward deformation field BM. However, for the task of document de - distortion, text lines and straight lines often occupy a very small part of the image area. This problem of unbalanced foreground - background distribution weakens the supervision ability of the L1 distance. At the same time, the L1 distance can guide the model to independently predict the approximate projection position of each point, but it is difficult to understand the geometric shape of straight - line elements. Thanks to the twisted line L extracted from the undistorted original image gt , this embodiment introduces the curvature invariance based on the control line to solve the above problems.

[0174] Specifically, for each extracted twisted line L gt , uniform sampling is performed every 4 pixels to generate a discrete point set P = {p i |p i ∈L gt , i = 1, 2,..., N}. Subsequently, the point set is projected onto the predicted backward deformation field and the true backward deformation field BM respectively, so as to obtain the predicted control points Cp and the true control points Cp gt . To reduce errors and ensure the differentiability of the process, this embodiment uses the method of bilinear interpolation for point mapping and projection, expressed as:

[0175]

[0176] where ω p is the bilinear interpolation weight calculated according to the relative position of p i in the grid landing point. N(p i ) represents the set of four neighborhood points of point p i in the grid. For the obtained mapped point set Cp = {(x i , y i )|i = 1, 2,..., N}, this embodiment uses the central - difference method to calculate the derivative of discrete points, and uses the forward and backward - difference methods for boundary points. The curvature of the control line is expressed as:

[0177]

[0178] where x i ′, x i ″, y i ′ and y i ″ represent the first - order discrete derivative and the second - order discrete derivative of the coordinate points respectively. To avoid the problem of gradient explosion or overflow during training, this embodiment alleviates it by adding a positive decimal ε. In this embodiment, ε is set to 0.0001. This embodiment uses constraints to expect that the curvature change trend at local discrete points is as close to the real situation as possible. This embodiment formulates the L k function as follows:

[0179]

[0180] Among them, and k i respectively represent the curvature values at the discrete point P of the predicted backward deformation field and the true backward deformation field BM. The final loss is defined as a linear combination of different losses:

[0181]

[0182] where L represents the total loss of the document image correction model, L1 represents the deformation field regression loss of the document image correction model, and L seg represents the mask loss of the document image correction model, and L k represents the curvature consistency loss of the document image correction model. represents the predicted backward deformation field, and BM represents the true backward deformation field.

[0183] In this embodiment, the curvature consistency based on control points can achieve the following effects:

[0184] 1. Innovation in modeling strong distortion regions: Pioneeringly introduce the principle of curvature consistency based on control points, and use curvature values to explicitly model the strong distortion regions in document images. By uniformly sampling each pixel of the extracted line elements, a discrete point set is obtained, and it is projected onto the predicted deformation field and the true deformation field to obtain predicted control points and true control points, and then the curvature values are calculated. This method can accurately describe the geometric distortion characteristics of the distortion field, provide more detailed and accurate geometric information for the model, and help the model better understand and process strong distortion regions.

[0185] 2. Solve the supervision problem: Effectively solve the deficiencies of traditional L1 distance supervision in the document undistortion task. On the one hand, aiming at the problem of unbalanced foreground and background distribution caused by the small proportion of text lines and straight lines in the image, by introducing curvature constraints based on control points, the supervision weight for foreground straight line elements is increased, making the model pay more attention to the deformation of these key regions; on the other hand, overcome the defect that the L1 distance is difficult to understand the geometric shape of straight line elements. The curvature consistency principle can guide the model to learn the geometric shape characteristics of straight line elements, ensure that the shape of straight line elements is accurately restored during the undistortion process, and improve the model's processing ability for straight line elements.

[0186] 3. Optimize the deformation field prediction: Based on the curvature consistency of the control points, a loss function is formulated to constrain the curvature change trend of the predicted deformation field and the real deformation field at the discrete points to be as close to the real situation as possible. During the model training process, this function serves as an additional supervision signal to guide the model to continuously adjust the prediction results, enabling the model to learn a distortion field geometry that is more in line with the real situation, thereby significantly improving the accuracy of the deformation field prediction, ultimately achieving a more accurate document image undistortion effect, and effectively enhancing the quality and reliability of the corrected image.

[0187] In this embodiment, the original undistorted image is directly preprocessed to obtain training data, and the document image correction model is trained based on the training data to obtain the trained document image correction model. Thus, only the original undistorted image is required to quickly perform model training and iteration.

[0188] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the document image correction method of this application. Based on this technical concept, more forms of simple transformations are within the protection scope of this application.

[0189] This application also provides a document image correction device. Please refer to Figure 6 , and the document image correction device includes:

[0190] An image input module 10 for inputting the document image to be corrected into the document image correction model, where a mask guidance module is set in the document image correction model;

[0191] A model guidance module 20 for guiding the attention of the document image correction model to the foreground area of the document image to be corrected through the mask guidance module to obtain a distortion deformation field;

[0192] A document correction module 30 for correcting the document image to be corrected based on the distortion deformation field to obtain a corrected document image.

[0193] The document image correction device provided by this application adopts the document image correction method in the above embodiment, which can solve the technical problem that in the related document image correction method, foreground elements are prone to distortion when dealing with complex geometric distortions, resulting in low correction accuracy and poor user experience. Compared with the prior art, the beneficial effects of the document image correction device provided by this application are the same as those of the document image correction method provided by the above embodiment, and other technical features in the document image correction device are the same as those disclosed in the method of the above embodiment, which will not be elaborated here.

[0194] The present application provides a document image correction device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the document image correction method in Embodiment 1 above.

[0195] Reference is made below to Figure 7 , which shows a schematic structural diagram of a document image correction device suitable for implementing the embodiments of the present application. The document image correction device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions, tablet computers), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The document image correction device shown is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0196] As Figure 7As shown, the document image correction device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which may perform various appropriate actions and processes according to a program stored in a ROM (Read Only Memory) 1002 or a program loaded from a storage device 1003 into a RAM (Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the document image correction device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, an LCD (Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the document image correction device to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows a document image correction device having various systems, it should be understood that it is not required to implement or have all the shown systems. More or fewer systems may be alternatively implemented or had.

[0197] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above functions defined in the methods of the embodiments disclosed in the present application are performed.

[0198] The document image correction device provided by the present application adopts the document image correction method in the above embodiment, and can solve the technical problem that in the related document image correction method, when dealing with complex geometric distortions, foreground elements are prone to distortion, resulting in low correction accuracy and poor user experience. Compared with the prior art, the beneficial effects of the document image correction device provided by the present application are the same as those of the document image correction method provided by the above embodiment, and other technical features in the document image correction device are the same as those disclosed in the method of the previous embodiment, and will not be elaborated here.

[0199] It should be understood that each part disclosed in this application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0200] As described above, the above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application, and all should be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0201] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the document image correction method in the above embodiments.

[0202] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), or flash memory, optical fibers, CD-ROM (CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0203] The above computer-readable storage medium can be included in the document image correction device; or it can exist separately without being assembled into the document image correction device.

[0204] The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by the document image correction device, the document image correction device is caused to execute the above document image correction method.

[0205] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The above-mentioned programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a LAN (Local Area Network) or a WAN (Wide Area Network), or it can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0206] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of the code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0207] The modules described in the embodiments of this application can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation on the unit itself in some cases.

[0208] The readable storage medium provided by this application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned document image correction method, which can solve the technical problem that in the related document image correction method, foreground elements are prone to distortion when dealing with complex geometric distortions, resulting in low correction accuracy and poor user experience. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by this application are the same as those of the document image correction method provided in the above embodiments, and will not be elaborated here.

[0209] This application also provides a computer program product, including a computer program, which when executed by a processor implements the document image correction method as described above.

[0210] The computer program product provided by this application can solve the technical problem that in the related document image correction method, foreground elements are prone to distortion when dealing with complex geometric distortions, resulting in low correction accuracy and poor user experience. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as those of the document image correction method provided in the above embodiments, and will not be elaborated here.

[0211] The above are only partial embodiments of this application, and do not limit the patent scope of this application. Any equivalent structural transformation made under the technical concept of this application by using the content of the specification and drawings of this application, or directly / indirectly applied in other related technical fields, is included in the patent protection scope of this application.

[0212] This application discloses A1. A document image correction method, the document image correction method includes:

[0213] Input the document image to be corrected into a document image correction model, wherein a mask guidance module is set in the document image correction model;

[0214] Through the mask guidance module, guide the attention of the document image correction model to focus on the foreground area of the document image to be corrected, and obtain a distortion deformation field;

[0215] Based on the distortion deformation field, correct the document image to be corrected to obtain a corrected document image.

[0216] A2. The document image correction method as described in A1, wherein a foreground segmentation module is also set in the document image correction model; before guiding the attention of the document image correction model to focus on the foreground area of the document image to be corrected through the mask guidance module and obtaining a distortion deformation field, it further includes:

[0217] The foreground segmentation module performs foreground segmentation on the document image to be corrected to obtain a predicted foreground mask;

[0218] Correspondingly, guiding the attention of the document image correction model by the mask guidance module to focus on the foreground area of the document image to be corrected to obtain a distortion deformation field includes:

[0219] Based on the predicted foreground mask, guiding the attention of the document image correction model by the mask guidance module to focus on the foreground area of the document image to be corrected to obtain a distortion deformation field.

[0220] A3. The document image correction method as described in A2, where the foreground segmentation module performs foreground segmentation on the document image to be corrected to obtain a predicted foreground mask, including:

[0221] The foreground segmentation module performs foreground segmentation on the document image to be corrected to obtain a segmentation mask;

[0222] Applying a normalization operation with a smoothing coefficient to the segmentation mask to obtain per-pixel class probabilities;

[0223] Calculating the expected value of the class probabilities and calculating the predicted foreground mask of the document image to be corrected according to the expected value.

[0224] A4. The document image correction method as described in A2, where a feature extraction module and a Transformer encoder are further provided in the document image correction model; before guiding the attention of the document image correction model by the mask guidance module based on the predicted foreground mask to focus on the foreground area of the document image to be corrected to obtain a distortion deformation field, it further includes:

[0225] The feature extraction module extracts target features from the document image to be corrected;

[0226] The Transformer encoder encodes the target features to obtain a feature sequence;

[0227] Correspondingly, guiding the attention of the document image correction model by the mask guidance module based on the predicted foreground mask to focus on the foreground area of the document image to be corrected to obtain a distortion deformation field includes:

[0228] Based on the feature sequence and the predicted foreground mask, guiding the attention of the document image correction model by the mask guidance module to focus on the foreground area of the document image to be corrected to obtain a distortion deformation field.

[0229] A5. The document image correction method according to any one of A1 to A4, before inputting the document image to be corrected into the document image correction model, further comprising:

[0230] Obtaining an undistorted original image and preprocessing the undistorted original image to obtain training data;

[0231] Obtaining the true backward deformation field during the preprocessing of the undistorted original image;

[0232] Using the true backward deformation field as the training target, training the document image correction model based on the training data to obtain a trained document image correction model;

[0233] Correspondingly, inputting the document image to be corrected into the document image correction model includes:

[0234] Inputting the document image to be corrected into the trained document image correction model.

[0235] A6. The document image correction method according to A5, wherein the training data includes a distorted image, a distorted foreground, and distorted lines; obtaining an undistorted original image and preprocessing the undistorted original image to obtain training data includes:

[0236] Obtaining an undistorted original image and performing foreground-background segmentation on the undistorted original image to obtain a foreground mask;

[0237] Performing line element extraction on the undistorted original image to obtain line elements;

[0238] Applying the deformation field to the undistorted original image, the foreground mask, and the line elements respectively to obtain a distorted image, a distorted foreground, and distorted lines.

[0239] A7. The document image correction method according to A6, obtaining an undistorted original image and performing foreground-background segmentation on the undistorted original image to obtain a foreground mask, including:

[0240] Obtaining an undistorted original image and performing foreground-background segmentation on the undistorted original image through a fine-tuned Hi-SAM model to obtain foreground elements, where the foreground elements include at least one of text, straight lines, and pictures;

[0241] Generating a foreground mask for the undistorted original image based on the foreground elements.

[0242] A8. The document image correction method according to A6, wherein the line elements include text lines and table lines; performing line element extraction on the undistorted original image to obtain line elements, including:

[0243] Detect the text lines of the undistorted original image through a character recognition engine, and use the bisector of the text detection box as the text line;

[0244] Remove abnormal lines through a preset edge detection model and a preset line detection model to obtain table lines.

[0245] A9. The document image correction method as described in A6. Before applying the deformation field to the undistorted original image, the foreground mask, and the line elements respectively to obtain a distorted image, a distorted foreground, and distorted lines, it further includes:

[0246] Obtain a true backward deformation field, and inversely solve the corresponding true forward deformation field based on the true backward deformation field;

[0247] Correspondingly, the step of applying the deformation field to the undistorted original image, the foreground mask, and the line elements respectively to obtain a distorted image, a distorted foreground, and distorted lines includes:

[0248] Apply the true forward deformation field to the undistorted original image, the foreground mask, and the line elements respectively to obtain a distorted image, a distorted foreground, and distorted lines.

[0249] A10. The document image correction method as described in A6. Using the true backward deformation field as the training target, training the document image correction model based on the training data to obtain a trained document image correction model includes:

[0250] Input the distorted image into the document image correction model to obtain a predicted backward deformation field output by the document image correction model;

[0251] Calculate the deformation field regression loss of the document image correction model based on the predicted backward deformation field and the true backward deformation field;

[0252] Obtain the segmentation mask when the document image correction model outputs the predicted backward deformation field, and calculate the mask loss of the document image correction model based on the segmentation mask and the distorted foreground;

[0253] Calculate the curvature consistency loss of the document image correction model with the distorted lines as sampling control points;

[0254] Calculate the total loss of the document image correction model according to the deformation field regression loss, the mask loss, and the curvature consistency loss;

[0255] Adjust the model parameters of the document image correction model based on the total loss to obtain a trained document image correction model.

[0256] A11. The document image correction method as described in A10, wherein calculating the curvature consistency loss of the document image correction model with the twisted curve as the sampling control point includes:

[0257] Uniformly sample with the twisted curve as the sampling control point to obtain a discrete point set;

[0258] Project the discrete point set onto the predicted backward deformation field and the true backward deformation field respectively to obtain predicted control points and true control points;

[0259] Calculate the discrete point curvature difference according to the predicted control points and the true control points to obtain the curvature consistency loss of the document image correction model.

[0260] The present application also discloses B12. A document image correction device, the document image correction device includes:

[0261] An image input module, configured to input a document image to be corrected into a document image correction model, wherein a mask guidance module is set in the document image correction model;

[0262] A model guidance module, configured to guide the attention of the document image correction model to focus on the foreground area of the document image to be corrected through the mask guidance module to obtain a distorted deformation field;

[0263] A document correction module, configured to correct the document image to be corrected based on the distorted deformation field to obtain a corrected document image.

[0264] B13. The document image correction device as described in B12, wherein a foreground segmentation module is further set in the document image correction model; the document image correction device further includes:

[0265] A mask prediction module, configured to perform foreground segmentation on the document image to be corrected through the foreground segmentation module to obtain a predicted foreground mask;

[0266] Correspondingly, the document correction module is further configured to guide the attention of the document image correction model to focus on the foreground area of the document image to be corrected through the mask guidance module based on the predicted foreground mask to obtain a distorted deformation field.

[0267] B14. The document image correction device as described in B13, wherein the mask prediction module is further configured to perform foreground segmentation on the document image to be corrected through the foreground segmentation module to obtain a segmentation mask; perform a normalization operation with a smoothing coefficient on the segmentation mask to obtain a per-pixel class probability; calculate the expected value of the class probability, and calculate the predicted foreground mask of the document image to be corrected according to the expected value.

[0268] B15. The document image correction device as described in B13, wherein a feature extraction module and a Transformer encoder are further provided in the document image correction model; the document image correction device further includes:

[0269] A sequence extraction module, configured to extract target features from the document image to be corrected through the feature extraction module; and encode the target features through the Transformer encoder to obtain a feature sequence.

[0270] Correspondingly, the document correction module is further configured to, based on the feature sequence and the predicted foreground mask, guide the attention of the document image correction model to the foreground area of the document image to be corrected through the mask guidance module, so as to obtain a distortion deformation field.

[0271] B16. The document image correction device as described in any one of B12 to B15, wherein the document image correction device further includes:

[0272] A model training module, configured to obtain an undistorted original image, preprocess the undistorted original image to obtain training data; obtain a true backward deformation field during the preprocessing of the undistorted original image; and use the true backward deformation field as a training target to train the document image correction model based on the training data to obtain a trained document image correction model.

[0273] Correspondingly, the image input module is further configured to input the document image to be corrected into the trained document image correction model.

[0274] B17. The document image correction device as described in B16, wherein the training data includes a distorted image, a distorted foreground, and distorted lines; the model training module is further configured to obtain an undistorted original image, perform foreground and background segmentation on the undistorted original image to obtain a foreground mask; extract line elements from the undistorted original image to obtain line elements; and apply the deformation field to the undistorted original image, the foreground mask, and the line elements respectively to obtain a distorted image, a distorted foreground, and distorted lines.

[0275] The present application also discloses C18. A document image correction device, including: a memory, a processor, and a document image correction program stored on the memory and executable on the processor, where when the document image correction program is executed by the processor, it implements the document image correction method as described above.

[0276] The present application also discloses D19. A storage medium, on which a document image correction program is stored, where when the document image correction program is executed by a processor, it implements the document image correction method as described above.

[0277] The present application also discloses E20, a computer program product, which includes a document image correction program. When the document image correction program is executed by a processor, it implements the document image correction method as described above.

Claims

1. A document image correction method, characterized in that: The document image correction method comprises: Inputting the document image to be corrected into a document image correction model, wherein the document image correction model is provided with a mask guidance module; Guiding the document image correction model to focus on the foreground area of ​​the document image to be corrected by the mask guiding module to obtain a distortion deformation field; The document image to be corrected is corrected based on the distortion deformation field to obtain a corrected document image.

2. The document image correction method according to claim 1, characterized in that: The document image correction model is also provided with a foreground segmentation module; the mask guiding module is used to guide the attention of the document image correction model to focus on the foreground area of ​​the document image to be corrected, and before obtaining the distortion deformation field, it also includes: Performing foreground segmentation on the document image to be corrected by the foreground segmentation module to obtain a predicted foreground mask; Correspondingly, guiding the attention of the document image correction model to focus on the foreground area of ​​the document image to be corrected by the mask guiding module to obtain the distortion deformation field includes: Based on the predicted foreground mask, the mask guiding module guides the attention of the document image correction model to focus on the foreground area of ​​the document image to be corrected, so as to obtain a distortion deformation field.

3. The document image correction method according to claim 2, characterized in that: The foreground segmentation module is used to perform foreground segmentation on the document image to be corrected to obtain a predicted foreground mask, including: Performing foreground segmentation on the document image to be corrected by the foreground segmentation module to obtain a segmentation mask; Applying a normalization operation with a smoothing coefficient to the segmentation mask to obtain pixel-by-pixel category probabilities; An expected value of the category probability is calculated, and a predicted foreground mask of the document image to be corrected is calculated according to the expected value.

4. The document image correction method according to claim 2, characterized in that: The document image correction model is also provided with a feature extraction module and a Transformer encoder; the document image correction model is guided by the mask guidance module based on the predicted foreground mask to focus on the foreground area of ​​the document image to be corrected, and before obtaining the distortion deformation field, it also includes: Extracting target features from the document image to be corrected by the feature extraction module; Encoding the target feature through the Transformer encoder to obtain a feature sequence; Accordingly, the step of guiding the attention of the document image correction model to focus on the foreground area of ​​the document image to be corrected through the mask guiding module based on the predicted foreground mask to obtain the distortion deformation field includes: Based on the feature sequence and the predicted foreground mask, the mask guiding module guides the attention of the document image correction model to focus on the foreground area of ​​the document image to be corrected, so as to obtain a distortion deformation field.

5. The document image correction method according to any one of claims 1 to 4, characterized in that: Before inputting the document image to be corrected into the document image correction model, the method further includes: Obtaining an undistorted original image, and preprocessing the undistorted original image to obtain training data; Obtaining a true backward deformation field during the preprocessing of the undistorted original image; Taking the true backward deformation field as a training target, training the document image correction model based on the training data to obtain a trained document image correction model; Accordingly, the step of inputting the document image to be corrected into the document image correction model includes: The document image to be corrected is input into the trained document image correction model.

6. The document image correction method according to claim 5, characterized in that: The training data includes a distortion map, a distortion foreground, and a distortion line; The obtaining of the undistorted original image and preprocessing of the undistorted original image to obtain training data includes: Obtaining an undistorted original image, and performing foreground and background segmentation on the undistorted original image to obtain a foreground mask; Extracting line elements from the original image without distortion to obtain line elements; The deformation field is applied to the undistorted original image, the foreground mask and the line element respectively to obtain a distorted image, a distorted foreground and a distorted line.

7. A document image correction device, characterized in that: The document image correction device comprises: An image input module, used for inputting a document image to be corrected into a document image correction model, wherein the document image correction model is provided with a mask guidance module; A model guiding module, used for guiding the document image correction model to focus on the foreground area of ​​the document image to be corrected through the mask guiding module to obtain a distortion deformation field; The document correction module is used to correct the document image to be corrected based on the distortion deformation field to obtain a corrected document image.

8. A document image correction device, characterized in that: The document image correction device includes: a memory, a processor, and a document image correction program stored in the memory and executable on the processor. When the document image correction program is executed by the processor, the document image correction method according to any one of claims 1 to 6 is implemented.

9. A storage medium, characterized in that: The storage medium stores a document image correction program, and when the document image correction program is executed by the processor, the document image correction method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product comprises a document image correction program, and when the document image correction program is executed by a processor, the document image correction method according to any one of claims 1 to 6 is implemented.

Images