Document image correction method and apparatus

Abstract

The specification provides a document picture correction method and device, wherein the document picture correction method comprises: obtaining a to-be-corrected picture and a corresponding reference picture, wherein the to-be-corrected picture and the reference picture are pictures corresponding to a document, the reference picture does not include handwritten content, and the printed content in the reference picture is consistent with the to-be-corrected picture; matching the to-be-corrected picture with the reference picture to determine a handwritten character region in the to-be-corrected picture; and rendering the handwritten character region in the corresponding rendering position in the reference picture to obtain a target picture after correction of the to-be-corrected picture. In this case, the to-be-corrected picture and the reference picture can be matched to extract the handwritten character region in the to-be-corrected picture, and the extracted handwritten character region can be rendered to the corresponding position on the reference picture, thereby achieving the purpose of correcting the document picture and ensuring the correction effect of the document picture.

Description

Technical Field

[0001] This specification relates to the field of image processing technology, and in particular to a document image correction method. This specification also relates to a document image correction apparatus, a computing device, and a computer-readable storage medium. Background Technology

[0002] In recent years, with the rapid development and iteration of computer technology, the internet industry, network infrastructure, and terminal devices, image processing technology has also developed rapidly. Various paper documents can be photographed into document images, facilitating the storage, management, and automated processing of large quantities of documents. However, user-uploaded document images often suffer from uneven lighting, shadows, wrinkles, noise, low light, and color casts, resulting in low efficiency and accuracy in subsequent automated processing of document images.

[0003] In existing technologies, pre-trained semantic segmentation models are often used to enhance user-uploaded document images, thereby effectively removing problems such as uneven lighting, shadows, wrinkles, noise, low light, and color cast.

[0004] However, some document images may have problems such as shape distortion, rotation, and translation due to the shooting angle. Beautifying document images using a pre-trained semantic segmentation model cannot solve these problems. In addition, beautification may also result in issues such as cutouts in the images and thick fonts, leading to poor document image correction results and affecting the efficiency and accuracy of subsequent automated processing of document images. Summary of the Invention

[0005] In view of this, embodiments of this specification provide a document image correction method. This specification also relates to a document image correction apparatus, a computing device, and a computer-readable storage medium, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this specification, a document image correction method is provided, comprising:

[0007] Obtain the image to be corrected and the corresponding reference image. Both the image to be corrected and the reference image are images corresponding to the document. The reference image does not include handwritten content, and the printed content in the reference image is consistent with the image to be corrected.

[0008] Match the image to be corrected with the reference image to determine the handwritten text area in the image to be corrected;

[0009] The handwritten text area is rendered in the corresponding rendering position in the reference image to obtain the target image after correction.

[0010] According to a second aspect of the embodiments of this specification, a document image correction apparatus is provided, comprising:

[0011] The acquisition module is configured to acquire the image to be corrected and the corresponding reference image. Both the image to be corrected and the reference image are images corresponding to the document. The reference image does not include handwritten content, and the printed content in the reference image is consistent with the image to be corrected.

[0012] The matching module is configured to match the image to be corrected with a reference image to determine the handwritten text region in the image to be corrected.

[0013] The rendering module is configured to render the handwritten text area at the corresponding rendering position in the reference image, thereby obtaining the target image after correction of the image to be corrected.

[0014] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising:

[0015] Memory and processor;

[0016] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the document image correction method described above.

[0017] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the document image correction method described above.

[0018] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the document image correction method described above.

[0019] The document image correction method provided in this specification involves obtaining an image to be corrected and a corresponding reference image. Both the image to be corrected and the reference image are images corresponding to the document. The reference image does not include handwritten content, and the printed content in the reference image is consistent with that in the image to be corrected. The image to be corrected is matched with the reference image to determine the handwritten text region in the image to be corrected. The handwritten text region is then rendered at the corresponding rendering position in the reference image to obtain the corrected target image. In this case, matching the image to be corrected and the reference image extracts the handwritten text region from the image to be corrected, and rendering the extracted handwritten text region at the corresponding position on the reference image. This completely solves problems such as document image shape distortion, rotation, and translation, as well as issues such as image cutouts and thick fonts that occur after beautifying the document based on a pre-trained semantic segmentation model. This achieves the purpose of correcting the document image, ensuring the correction effect and thus guaranteeing the efficiency and accuracy of subsequent automated processing of the document image. Attached Figure Description

[0020] Figure 1a This is a schematic diagram of the structure of a semantic segmentation model provided in one embodiment of this specification;

[0021] Figure 1b This is a schematic diagram of a first type of user-uploaded test paper image provided in one embodiment of this specification;

[0022] Figure 1c This is a schematic diagram of a second type of user-uploaded test paper image provided in one embodiment of this specification;

[0023] Figure 1d This is a schematic diagram of a third type of user-uploaded test paper image provided in one embodiment of this specification;

[0024] Figure 1e This is a schematic diagram of a fourth type of user-uploaded test paper image provided in one embodiment of this specification;

[0025] Figure 2a This is a schematic diagram of the first type of beautified exam paper image provided in one embodiment of this specification;

[0026] Figure 2b This is a schematic diagram of the second type of beautified exam paper image provided in one embodiment of this specification;

[0027] Figure 2c This is a schematic diagram of the third type of beautified exam paper image provided in one embodiment of this specification;

[0028] Figure 2d This is a schematic diagram of the fourth type of beautified exam paper image provided in one embodiment of this specification;

[0029] Figure 3This is a flowchart of a document image correction method provided in one embodiment of this specification;

[0030] Figure 4 This is a visual schematic diagram illustrating the process of extracting a handwritten text region according to an embodiment of this specification;

[0031] Figure 5 This is a flowchart illustrating a document image correction method applied in an educational setting, as provided in one embodiment of this specification.

[0032] Figure 6 This is a visual schematic diagram illustrating the processing procedure of a document image correction method provided in one embodiment of this specification;

[0033] Figure 7 This is a schematic diagram of the structure of a document image correction device provided in one embodiment of this specification;

[0034] Figure 8 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0035] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0036] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0037] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0038] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0039] U2Net: A two-layer nested U-shaped structure for salient object detection, where U represents the extraction and encoding of multi-scale features on the input. The nested U-shaped structure can more effectively extract multi-scale features within the image and multi-level features during the aggregation stage.

[0040] KeyNet model: A keypoint detection network that runs on a multi-scale representation of a full-size image and returns a response map containing keypoint scores for each pixel. Specifically, it can blur the input image at three scale levels with a blur and downsampling factor of 1.2, and then upsample, cascade, and feed the feature maps from all scale levels into a final convolutional filter to obtain the final response map.

[0041] HardNet is a CNN (Convolutional Neural Network) model used for image segmentation and object detection, characterized by its low memory footprint. With increased computing power and larger datasets, more complex networks can be trained. For real-time applications, the challenge lies in improving computational efficiency while reducing power consumption. HarDNet was proposed to address this challenge, seeking the optimal balance between the two.

[0042] Homography matrix: A homography matrix describes the correspondence between two images of a 3D point on the same plane in space. Homography matrices can be applied to image correction, image registration and stitching, viewpoint transformation, and calculating camera motion (rotation and translation) between two images.

[0043] RANSC: Random Sample Consensus Algorithm. It uses an iterative method to find the optimal parameter model in a dataset containing "outsiders". Points that do not conform to the optimal model are defined as "outsiders". It is widely used in image registration and stitching.

[0044] Perspective transformation: Essentially, it projects an image onto a new view plane.

[0045] Scale-invariant feature transform (SIFT) is a descriptor used in image processing. It possesses scale invariance and can detect key points in images. It is a local feature descriptor based on scale-space invariance, image scaling, rotation, and even affine transformation invariance. SIFT is invariant to image changes such as scale, rotation, and certain viewpoint and illumination variations, and it also exhibits very low discriminability.

[0046] It should be noted that the quality of photos of documents uploaded by users varies greatly, often exhibiting issues such as uneven lighting, shadows, wrinkles, noise, low light, and color cast. To better showcase the user's handwriting in the document, it is desirable to enhance the uploaded document images using image processing technology to achieve a cleaner document page. After enhancing the document images, issues such as uneven lighting, shadows, wrinkles, noise, low light, and color cast can be effectively removed.

[0047] In practical applications, the U2Net semantic segmentation model can be used to enhance the appearance of user-uploaded document images. Figure 1a This is a schematic diagram of the structure of a semantic segmentation model provided in one embodiment of this specification, which inputs user-uploaded document images into a pre-trained... Figure 1a The semantic segmentation model shown can obtain an enhanced document image. The label of this semantic segmentation model contains two categories: the first category is the background, represented by 0, and the second category is the foreground (including all text, numbers, letters, lines, tables, graphics, pictures, etc. in the document), represented by 1.

[0048] U2Net is a two-layer nested U-shaped structure, such as Figure 1a As shown, its top layer is a system consisting of 11 stages (such as...). Figure 1a The nested U-shaped structure is composed of cubes, with each stage filled by a well-configured RSU. Therefore, the nested U-structure can more effectively extract multi-scale features within a stage and multi-level features of the aggregation stage.

[0049] like Figure 1a As shown, the U2Net network consists of three parts: a six-level encoder, a five-level decoder, and a saliency map fusion model connected to the decoder and the final encoder. In the encoder stages En_1, En_2, En_3, and En_4, RSU structures of RSU-7, RSU-6, RSU-5, and RSU-4 are used, respectively. The preceding numbers, such as 7, 6, 5, and 4, represent the height L of the RSU, which is typically configured according to the spatial resolution of the input feature maps. In En_5 and En_6, the feature map resolution is relatively low, and further downsampling of these feature maps would lead to the loss of useful context. Therefore, in the RSU-5 and RSU-6 stages, RSU-4F is used, where F indicates that RSU is an extended version, replacing the merging and upsampling operations with extended convolutions. This means that all intermediate feature maps in RSU-4F have the same resolution as their input feature maps.

[0050] The decoding stage has a similar structure to the symmetric encoding stage in En_6. In De_5, the RSU-4F extended board is also used, similar to that used in the encoding stages of En_5 and En_6. Each decoder stage takes as input a concatenation of the upsampled feature maps from the previous stage and the feature maps from its symmetric encoder stage, see [link to De_5]. Figure 1a .

[0051] The final part is the saliency map fusion module, which generates saliency probability maps. The U2Net network first generates six output saliency probability maps from En_6, De_5, De_4, De_3, De_2, and De_1 using 3x3 convolutions and the sigmoid function. Then, the logistic graph of the output saliency map (convolutional output, before the sigmoid function) is upsampled to the same size as the input image and fused through a cascade operation. Finally, it is passed through a 1x1 convolutional layer and a sigmoid function to generate the final saliency probability map S. fuse .

[0052] The initial training parameters of the semantic segmentation model U2Net are as follows: the model is trained for 100,000 epochs (i.e., epoch_num = 100,000), the training batch size is 5 (batch_size_train = 5), the testing batch size is 1 (batch_size_val = 1), the model is saved once every 2,000 iterations (save_frq = 2,000), the model uses the Adam optimizer, the initial learning rate is 0.01 (lr = 0.01), the training sample size is 768, the training hardware is two NVIDIA V100 graphics cards for parallel training, and the model used for document image enhancement is the model that has been iterated 196,000 times.

[0053] For example, taking an educational scenario as an example, the document image is an exam paper image. Figures 1b-1e This is an illustration of a user-uploaded exam paper image. Figure 1b The exam paper image shown has uneven lighting. Figure 1c The exam paper image shown has a shadow. Figure 1d The exam paper image shown has wrinkles. Figure 1e The exam paper image shown has issues such as noise and low light. Figures 1b-1e The exam paper image shown is input into the trained semantic segmentation model above for beautification processing, which can yield the following results: Figures 2a-2d The image shown is a schematic diagram of an aesthetically pleasing exam paper.

[0054] It should be noted that, in addition to issues such as uneven lighting, shadows, wrinkles, noise, low light, and color cast, some document images may also have shape distortion, rotation, or translation issues due to the shooting angle. Beautification cannot solve these problems. Furthermore, for drawings within document images, beautification may result in issues such as cut-out images or thicker fonts.

[0055] Therefore, in this embodiment, the image to be corrected and the reference image are matched to extract the handwritten text region in the image to be corrected, and the extracted handwritten text region is rendered onto the corresponding position on the reference image. This completely solves the problems of document image shape distortion, rotation, and translation, as well as the problems of image cutouts and thick fonts that occur after beautifying the document based on the pre-trained semantic segmentation model. This achieves the purpose of correcting the document image, ensures the correction effect of the document image, and thus ensures the efficiency and accuracy of subsequent automated processing of the document image.

[0056] This specification provides a document image correction method, and also relates to a document image correction device, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0057] Figure 3 A flowchart of a document image correction method according to an embodiment of this specification is shown, which specifically includes the following steps:

[0058] Step 302: Obtain the image to be corrected and the corresponding reference image. Both the image to be corrected and the reference image are images corresponding to the document. The reference image does not include handwritten content, and the printed content in the reference image is consistent with the image to be corrected.

[0059] It should be noted that the image to be corrected is obtained by the user writing content on a blank document, obtaining the written document, and then photographing the written document. The reference image is a pre-entered image of a blank document from the image library. This reference image does not include the handwritten content, and the printed content in the reference image is consistent with the image to be corrected.

[0060] The document can refer to any document that requires users to write content in various application scenarios, such as test papers, questionnaires, and learning materials. For example, users write their answers on a test paper, take a picture of the completed document to obtain the image to be corrected, and use the corresponding blank test paper as a reference image; users answer questions on a questionnaire, take a picture of the completed document to obtain the image to be corrected, and use the corresponding blank questionnaire as a reference image; users take notes on learning materials, take a picture of the completed document to obtain the image to be corrected, and use the corresponding blank materials as a reference image.

[0061] In practical applications, the image to be corrected is uploaded by the user, while the reference image is obtained from an image library. Specifically, multiple blank document images are pre-entered and stored in the image library. Each blank document image carries a corresponding document identifier, which is used to identify the corresponding blank document image. This document identifier can be a document number, QR code, etc. After obtaining the image to be corrected, image recognition can be performed on the image to obtain the document identifier. Based on this document identifier, the corresponding blank document image is retrieved from the image library as the reference image.

[0062] In one optional implementation of this embodiment, besides directly using the document image uploaded by the user as the image to be corrected, the document image uploaded by the user can also be preliminarily beautified, and the beautified document image can be used as the image to be corrected, that is, to obtain the image to be corrected. The specific implementation process can be as follows:

[0063] Get the initial image uploaded by the user;

[0064] The initial image is corrected using a semantic segmentation model to obtain the image to be corrected.

[0065] Among them, the semantic segmentation model refers to the U2Net model, which is a two-layer nested U-shaped structure that can be used for salient object detection. The U represents the extraction and encoding of multi-scale features on the input. The nested U-shaped structure can more effectively extract multi-scale features and multi-level features in the aggregation stage of the image.

[0066] In practical applications, the initial training parameters of a semantic segmentation model can be as follows: the model is trained for 100,000 epochs (i.e., epoch_num = 100,000), the training batch size is 5 (batch_size_train = 5), the testing batch size is 1 (batch_size_val = 1), the model is saved every 2,000 iterations (save_frq = 2,000), the model uses the Adam optimizer, the initial learning rate is 0.01 (lr = 0.01), the training sample size is 768, and the training hardware is two NVIDIA V100 graphics cards for parallel training. For example, the semantic segmentation model used for document image enhancement could be a model that has undergone 196,000 iterations.

[0067] It should be noted that the initial image uploaded by the user can be input into the trained semantic segmentation model. This semantic segmentation model can output a preliminarily beautified document image. This preliminarily beautified document image may have slight distortion, thick fonts, and hollowed-out images. In order to solve problems such as shape distortion, rotation, and translation caused by shooting angle, as well as problems such as hollowed-out images and thick fonts that may appear after beautification, this preliminarily beautified document image can be used as the image to be corrected. Further correction is then performed on the preliminarily beautified document image to ensure the correction effect of the document image, thereby ensuring the efficiency and accuracy of subsequent automated processing of the document image.

[0068] Step 304: Match the image to be corrected with the reference image to determine the handwritten text area in the image to be corrected.

[0069] It should be noted that the image to be corrected and the reference image correspond to the same document. The image to be corrected includes handwritten content by the user, while the reference image is a blank document. This blank document is pre-entered into the image library and is not a photo uploaded by the user. In other words, the reference image is a high-quality and accurate blank document image. Therefore, the image to be corrected and the reference image can be matched to determine the handwritten text area in the image to be corrected. Subsequently, it can be re-rendered onto the reference image to achieve the final effect of correcting the document image.

[0070] In one optional implementation of this embodiment, the image to be corrected is matched with a reference image to determine the handwritten text region in the image to be corrected. The specific implementation process can be as follows:

[0071] Perform a perspective transformation on the image to be corrected to obtain the initial corrected image after the transformation.

[0072] Identify the damaged handwritten text regions in the initial corrected image;

[0073] Generate a corresponding handwritten text mask based on the damaged handwritten text area;

[0074] Based on the handwritten text mask, determine the handwritten text region in the initial calibration image.

[0075] It should be noted that, firstly, based on feature matching between the image to be corrected and the reference image, a perspective transformation is performed on the image to be corrected, transforming it to the coordinate system of the reference image to obtain the initial corrected image. Then, the damaged handwritten text regions in the initial corrected image are identified. These damaged handwritten text regions refer to the handwritten text regions obtained after removing edge marks of printed content; that is, these regions have lost some of their handwritten text content. Next, a corresponding handwritten text mask is generated based on the damaged handwritten text regions. Based on this mask, the handwritten text regions in the initial corrected image can be extracted. This facilitates the subsequent re-rendering of the extracted handwritten text regions to their corresponding positions in the reference image, achieving document image correction.

[0076] Furthermore, before matching the image to be corrected with the reference image to determine the handwritten text area in the image to be corrected, that is, before performing perspective transformation on the image to be corrected to obtain the initial corrected image after transformation, the dimensions of the image to be corrected and the reference image can be normalized to make them a unified set size. This set size includes width and length. The width can be set as a numerical value, and the length can be calculated. For example, the set size includes a width of 720, and the length is calculated using the following formula (1):

[0077] L = (720 * height) / width (1)

[0078] Where height is the length of the image to be corrected, and width is the width of the image to be corrected.

[0079] It should be noted that since feature analysis is required on the image to be calibrated and the reference image later, and feature analysis is often performed on grayscale images, the image to be calibrated and the reference image can be read in grayscale mode, that is, the image to be calibrated and the reference image are grayscale images. Then, the scale of the image to be calibrated and the reference image is changed to unify the size of the image to the specified set size, thereby speeding up the processing of subsequent algorithms.

[0080] In one optional implementation of this embodiment, perspective transformation can be performed on the image to be corrected through feature point matching, that is, perspective transformation is performed on the image to be corrected to obtain the initial corrected image after transformation. The specific implementation process is as follows:

[0081] Identify matching feature point pairs in the image to be corrected and the reference image;

[0082] Based on the feature point pairs, generate the homography matrix of the image to be corrected and the reference image;

[0083] A perspective transformation is performed on the image to be corrected using a homography matrix to obtain the initial corrected image after transformation.

[0084] Specifically, the feature point is a feature region obtained by dividing the image. For example, the image to be corrected and the reference image can be divided into a set number of feature regions. Each feature region is a feature point. The shape, size and other attribute parameters of different feature points can be the same or different.

[0085] In practical applications, feature points of the image to be corrected and the reference image can be matched to obtain feature point pairs. A feature point pair includes a first candidate feature point and a second candidate feature point. The first candidate feature point is a feature point in the image to be corrected, and the second candidate feature point is a feature point in the reference image. The first candidate feature point and the second candidate feature point correspond to each other, that is, the first candidate feature point and the second candidate feature point indicate the same feature region in different images.

[0086] Furthermore, perspective transformation essentially projects an image onto a new viewing plane. Therefore, after matching and obtaining feature point pairs, a homography matrix between the image to be corrected and the reference image can be generated based on the feature point pairs. This homography matrix can describe the correspondence between three-dimensional points on the same plane in space and the two images. Thus, the perspective transformation of the image to be corrected can be performed using this homography matrix to obtain the initial corrected image after transformation. In specific implementation, the homography matrix can be input into the perspective transformation function to obtain the coordinates of the transformed elements. Specifically, the initial corrected image after transformation can be obtained through the following formula (2):

[0087]

[0088] in, Represents the coordinates of element w after perspective transformation; Represents the homography matrix; This represents the coordinates of element W in the image to be corrected. The coordinates of each element after perspective transformation can be obtained using formula (2), thus obtaining the initial corrected image after transformation.

[0089] In this embodiment, matching feature point pairs in the image to be corrected and the reference image can be determined first. Then, based on the feature point pairs, a homography matrix of the image to be corrected and the reference image is generated. This homography matrix represents the correspondence between the two images. A perspective transformation is performed on the image to be corrected to obtain the transformed initial corrected image, thereby transforming the image to be corrected to the coordinate system of the reference image. This facilitates the extraction of handwritten text areas, which can then be re-rendered onto the reference image to achieve the final effect of correcting the document image. In an optional implementation of this embodiment, feature points and feature point descriptors in the image to be corrected and the reference image can be obtained, and then the image to be corrected and the reference image can be matched, that is, matching feature point pairs in the image to be corrected and the reference image can be determined. The specific implementation process can be as follows:

[0090] Obtain the first feature point and the first feature point descriptor of the image to be corrected, and obtain the second feature point and the second feature point descriptor of the reference image;

[0091] The first and second feature point descriptors are matched to determine the first candidate feature point in the image to be corrected and the second candidate feature point in the reference image.

[0092] Calculate the distance between the first candidate feature point and the second candidate feature point, and take the first candidate feature point and the second candidate feature point whose distance is less than the distance threshold as feature point pairs.

[0093] Specifically, feature point descriptors refer to scale-invariant feature transformations. These descriptors are scale-invariant and can detect key points in images.

[0094] It should be noted that the core problem of image matching is to match images of the same target under different times, resolutions, lighting conditions, and poses. Since feature point descriptors are invariant to image changes such as scale, rotation, and certain viewpoint and lighting changes, and feature point descriptors have very light discriminability, the image to be corrected and the reference image can be matched based on feature point descriptors.

[0095] In practical applications, the KeyNet model can be used to extract the first feature points of the image to be corrected and the second feature points of the reference image, while the HardNet model can be used to extract the first feature point descriptors of the image to be corrected and the second feature point descriptors of the reference image. Both the first and second feature point descriptors can be 32*32 pixels in size.

[0096] In practice, the first and second feature point descriptors can be matched to determine similar first and second candidate feature points in the image to be corrected and the reference image. That is, the first and second candidate feature points have similar features and may correspond to the same region. For example, circular feature points in the image to be corrected can be used as first candidate feature points, and circular feature points in the reference image can be used as second candidate feature points. Then, the Euclidean distance between each first and second candidate feature point is calculated. If the Euclidean distance is small enough, it indicates that the two candidate feature points are likely to correspond to the same region in different images. Therefore, first and second candidate feature points with a distance less than a threshold can be selected as feature point pairs.

[0097] In the embodiments of this specification, feature points and feature point descriptors in the image to be corrected and the reference image can be obtained, and the image to be corrected and the reference image can be matched to determine feature point pairs. This facilitates subsequent perspective transformation of the image to be corrected based on the feature point pairs, obtaining the transformed initial corrected image, thereby extracting the handwritten text area, which can then be re-rendered onto the reference image to achieve the final effect of correcting the document image.

[0098] In an optional implementation of this embodiment, when generating the homography matrix, the obtained feature point pairs can be optimized to remove noise points with low confidence. That is, based on the feature point pairs, a homography matrix of the image to be corrected and the reference image is generated. The specific implementation process can be as follows:

[0099] Based on the feature point pair and noise optimization algorithm, noise point pairs are removed from the feature point pairs to obtain optimized feature point pairs;

[0100] Based on the optimized feature point pairs, a homography matrix is ​​generated for the image to be corrected and the reference image.

[0101] Specifically, a homography matrix describes the correspondence between three-dimensional points on the same plane in space and two images. The size of a homography matrix can be 3*3. Noise optimization algorithms can refer to algorithms that can remove feature points with low confidence. For example, a noise optimization algorithm could be the Random Sample Consensus Algorithm (RANSC). RANSC uses an iterative method to find points in a dataset containing "outliers" whose optimal parameter model does not conform to the optimal model; these are defined as "outliers" and filtered out.

[0102] In practical applications, the obtained feature point pairs can be directly input into a homography matrix generation algorithm to obtain the corresponding homography matrix. This homography matrix generation algorithm refers to an algorithm that integrates various functions for calculating the homography matrix. Alternatively, the Random Sample Consensus Algorithm (RANSAC) can be integrated into the homography matrix generation algorithm. The obtained feature point pairs are input into the algorithm, and the RANSAC optimization algorithm within the algorithm first processes each feature point pair, filtering out noisy pairs to obtain optimized feature point pairs. Specifically, the RANSAC algorithm is used to find an optimal homography matrix H, with a matrix size of 3×3. The goal of RANSAC is to find the optimal parameter matrix that maximizes the number of data points satisfying the matrix. Typically, the elements of the third row and third class of the homography matrix H are normalized by setting the third row to 1.

[0103] In the embodiments of this specification, a noise optimization algorithm is used to remove noise point pairs from feature point pairs to obtain optimized feature point pairs. Based on these optimized feature point pairs, a homography matrix of the image to be corrected and the reference image is generated, which results in the generation of the homography matrix having the most feature point pairs and a high confidence level, thus ensuring the accuracy of the generated homography matrix.

[0104] In one optional implementation of this embodiment, the damaged handwritten text region can be extracted through binarization, that is, the damaged handwritten text region in the initial correction image can be determined. The specific implementation process is as follows:

[0105] Binarize the initial calibration image and the reference image separately;

[0106] Morphological erosion is performed on the binarized reference image to obtain a loss reference image;

[0107] The loss reference image is subtracted from the binarized initial correction image to obtain the lossy handwritten text region.

[0108] Specifically, image binarization involves setting the grayscale value of each pixel in an image to 0 or 255, thus presenting the entire image as a distinct black and white visual effect.

[0109] It should be noted that binarization can be performed on the initial correction image and the reference image separately. Pixels with values ​​less than the binarization threshold are set to 0, and pixels with values ​​not less than the threshold are set to 255. This sets the pixel values ​​corresponding to the handwritten text area to 0, while setting the pixel values ​​of other areas to 255. The binarization threshold can be predefined and can be set relatively high to recognize even faintly written handwritten text areas. For example, the binarization threshold could be set to 230.

[0110] Furthermore, morphology, or mathematical morphology, is a crucial research area in image processing. Morphology primarily extracts component information from images, which is often essential for representing and depicting the shape of an image and represents the most fundamental shape features used in image understanding. Morphological operations mainly include erosion and dilation. Erosion is one of the most basic morphological operations; it can eliminate boundary points in an image, causing the image to shrink inward along the boundaries, and can also remove portions smaller than a specified structural element. Erosion can be used to "shrink" or "thin" the foreground in a binary image, thereby achieving functions such as noise removal and element segmentation.

[0111] In practical applications, morphological erosion can be performed on the binarized reference image. Two erosion operations are performed, and the morphological descriptor is a 5*5 rectangle to obtain a loss reference image to eliminate edge marks of printed content.

[0112] In practical applications, the damaged handwritten text area can be calculated and determined using the following formula (3):

[0113] lost_write_words_img=Binary(warp_img)-Erode(Binary(reference_img))^2(3)

[0114] Among them, lost_write_words_img is the lossy handwritten text region, warp_img is the initial correction image, reference_img is the reference image, Binary refers to binarization, Erode refers to morphological erosion, and 2 indicates that two erosion operations are performed.

[0115] In the embodiments of this specification, a morphological erosion operation can be used to obtain a damaged handwritten text area to eliminate edge traces of printed content and achieve noise removal.

[0116] In one optional implementation of this embodiment, a corresponding mask can be constructed based on the connected components in the extracted lossy handwritten text region. That is, a corresponding handwritten text mask is generated based on the lossy handwritten text region. The specific implementation process can be as follows:

[0117] Morphological erosion is performed on the damaged handwritten text area to obtain the restored handwritten text area;

[0118] Perform a bitwise NOT operation on the recovered handwritten text region, and calculate the connected components of the recovered handwritten text region after the bitwise NOT operation to obtain the rectangular coordinates of each connected component;

[0119] Generate a corresponding handwritten text mask based on the size of the initial corrected image;

[0120] Based on the rectangular coordinates of each connected component, the handwritten text mask is filled with rectangles.

[0121] It should be noted that since the damaged handwritten text area is obtained by removing the edge traces of the printed content, some handwritten text areas may be incomplete. Therefore, a morphological erosion operation can be performed on the damaged handwritten text area to obtain the restored handwritten text area. The morphological descriptor is also a 5*5 rectangle.

[0122] Furthermore, connected components determine whether it's possible to reach another vertex from a given vertex in a graph; in other words, whether there's a path between any two vertices. Therefore, we can calculate the connected components of the recovered handwritten text region after bitwise inversion. These connected components indicate the coordinates of the bounding box of the recovered handwritten text region, thus pinpointing its location in the image. In practice, we can perform a bitwise inversion on the recovered handwritten text region and calculate its connected components, obtaining the rectangular coordinates of each connected component. These rectangular coordinates are the coordinates of the bounding box of the recovered handwritten text region.

[0123] In practical applications, the bitwise inversion result can be obtained using the following formula (4):

[0124]

[0125] Where q(i,j) is the result of bitwise inversion of 8 bits, and p(i,j) is the recovered handwritten text area.

[0126] In practice, a zero-matrix can be generated first as a handwritten text mask, with its size matching that of the initial calibration image—that is, a completely black image with the same size as the initial calibration image. Then, this completely black image is filled with rectangles. Specifically, the rectangle coordinates of each connected component region are the coordinates of the handwritten text region. Therefore, the rectangles indicated by the rectangle coordinates of each connected component region in the generated completely black image can be filled, and the pixels within these rectangles can be set to 255 to obtain the handwritten text mask. In other words, the handwritten text region in this mask is white, and the non-handwritten text region is black.

[0127] In addition, some noise points may be misidentified as handwritten text areas, such as shadows, impurities, and dirt spots. Therefore, noise areas with a rectangular area smaller than a set area threshold can be excluded. This set area threshold can be set based on experience, such as 100, to exclude noise areas with a rectangular area smaller than 100 in the connected component region.

[0128] In this embodiment, a zero matrix can be generated first as a handwritten text mask, i.e., a completely black image. Based on the rectangular coordinates of the connected components, the pixel values ​​in the corresponding areas of the completely black image are filled with 255, i.e., the rectangular areas of the connected components are set to white. The white area is the handwritten text area in the obtained handwritten text mask. Subsequently, the handwritten text mask can be directly applied to the initial correction image to extract the handwritten text area in the initial correction image. It can then be re-rendered onto the reference image to achieve the final effect of correcting the document image.

[0129] In one optional implementation of this embodiment, the handwritten text region in the initial correction image is determined based on the handwritten text mask. The specific implementation process can be as follows:

[0130] Based on the handwritten text mask, the initial calibration image is copied to obtain the handwritten text area.

[0131] It should be noted that the handwritten text area in the handwritten text mask is white, and the non-handwritten text area is black. Therefore, the initial calibration image can be copied directly based on the handwritten text mask, and the content corresponding to the white area in the initial calibration image can be extracted to obtain the handwritten text area in the initial calibration image.

[0132] Example, Figure 4 This is a visual schematic diagram illustrating the process of extracting a handwritten text region according to an embodiment of this specification, such as... Figure 4 As shown, taking an educational exam paper as an example, the image to be corrected is the beautified exam paper image, and the reference image is a blank exam paper obtained from the question bank. By performing perspective transformation on the beautified exam paper image using the blank exam paper, we can obtain the transformed initial corrected image. Then, we can extract the damaged handwritten text regions in the initial corrected image through binarization. Based on these damaged handwritten text regions, we can generate a corresponding handwritten text mask. Applying this handwritten text mask to the initial corrected image yields the corresponding handwritten text regions.

[0133] Step 306: Render the handwritten text area onto the corresponding rendering position in the reference image to obtain the target image after correction.

[0134] It should be noted that the handwritten text area is the content handwritten by the user in the image to be corrected. The handwritten text area can be directly rendered in the corresponding rendering position in the reference image to obtain the corrected target image. In other words, the area of ​​the user's handwritten content is extracted from the image to be corrected, and the extracted area is directly superimposed and rendered in the corresponding position in the reference image to obtain the corrected target image.

[0135] In one optional implementation of this embodiment, the handwritten text area is rendered at the corresponding rendering position in the reference image to obtain the corrected target image. The specific implementation process can be as follows:

[0136] Based on the area coordinates of the handwritten text region, determine the corresponding rendering coordinates in the reference image;

[0137] The handwritten text area is rendered at the rendering position indicated by the rendering coordinates to obtain the target image after correction.

[0138] It should be noted that the image to be corrected and the reference image are corresponding, except that the image to be corrected includes the user's handwritten content, while the reference image is blank. Therefore, based on the area coordinates of the handwritten text region, the corresponding rendering coordinates in the reference image can be determined, and the handwritten text region can be rendered at the rendering position indicated by the rendering coordinates to obtain the target image after correction.

[0139] The document image correction method provided in this specification involves obtaining an image to be corrected and a corresponding reference image. Both the image to be corrected and the reference image are images corresponding to the document. The reference image does not include handwritten content, and the printed content in the reference image is consistent with that in the image to be corrected. The image to be corrected is matched with the reference image to determine the handwritten text region in the image to be corrected. The handwritten text region is then rendered at the corresponding rendering position in the reference image to obtain the corrected target image. In this case, matching the image to be corrected and the reference image extracts the handwritten text region from the image to be corrected, and rendering the extracted handwritten text region at the corresponding position on the reference image. This completely solves problems such as document image shape distortion, rotation, and translation, as well as issues such as image cutouts and thick fonts that occur after beautifying the document based on a pre-trained semantic segmentation model. This achieves the purpose of correcting the document image, ensuring the correction effect and thus guaranteeing the efficiency and accuracy of subsequent automated processing of the document image.

[0140] The following is in conjunction with the appendix Figure 5 Taking the document image correction method provided in this manual as an example in an educational setting, this paper further explains the document image correction method. Figure 5 This specification illustrates a flowchart of a document image correction method applied in an educational setting, according to an embodiment of this specification. The method specifically includes the following steps:

[0141] Step 502: Obtain the image of the answer sheet uploaded by the student, identify the image of the answer sheet to obtain the test paper identifier, and obtain the corresponding blank test paper reference image from the test paper database based on the test paper identifier.

[0142] Step 504: Input the student's uploaded answer sheet image into the U2Net model for image enhancement to obtain the enhanced answer sheet image.

[0143] Step 506: Import the beautified answer sheet image and the blank answer sheet reference image in grayscale mode, scale the beautified answer sheet image and the blank answer sheet reference image, and unify the beautified answer sheet image and the blank answer sheet reference image to the set size.

[0144] Step 508: Use the KeyNet model to extract feature points from the beautified answer sheet image and the blank answer sheet reference image, and use the HardNet model to extract feature point descriptors from the beautified answer sheet image and the blank answer sheet reference image. Based on the feature point descriptors, match the feature points in the beautified answer sheet image and the blank answer sheet reference image to obtain feature point pairs.

[0145] Step 510: Use the feature point pair to generate the homography matrix of the beautified answer sheet image and the blank answer sheet reference image.

[0146] In particular, the homography matrix for generating the beautified answer sheet image and the blank answer sheet reference image can be optimized using the RANSC algorithm.

[0147] Step 512: Use homography matrix to perform perspective transformation on the beautified answer sheet image to obtain the initial corrected image after transformation.

[0148] Step 514: Perform binarization processing on the initial calibration image and the blank test paper reference image respectively. Set the pixels with a value less than 230 in the initial calibration image and the blank test paper reference image to 0, and set the pixels with a value not less than 230 to 255.

[0149] Step 516: Perform two morphological erosion operations on the binarized blank test paper reference image to obtain a loss reference image. Subtract the loss reference image from the binarized initial correction image to obtain the lossy handwritten text region.

[0150] Step 518: Perform another morphological erosion operation on the damaged handwritten text area obtained in the previous step to obtain the restored handwritten text area.

[0151] Step 520: Perform a bitwise inversion operation on the recovered handwritten text region, and calculate the connected components of the recovered handwritten text region after the bitwise inversion operation to obtain the rectangular coordinates of each connected component; generate an all-zero matrix with the same size as the initial corrected image, and fill the corresponding rectangular region in the all-zero matrix with 255 according to the rectangular coordinates of each connected component, and exclude noise regions with a rectangular area of ​​less than 100 pixels in the connected component region.

[0152] Step 522: Based on the handwritten text mask generated in the previous step, copy the initial calibration image, keeping only the pixel value of the handwritten text mask at 255, to obtain the handwritten text area.

[0153] Step 524: Based on the area coordinates of the handwritten text region, determine the corresponding rendering coordinates in the blank test paper reference image, render the handwritten text region at the rendering position indicated by the rendering coordinates, and obtain the corrected target test paper image.

[0154] Example, Figure 6 This is a visual schematic diagram illustrating the processing procedure of a document image correction method provided in one embodiment of this specification, such as... Figure 6 As shown, the system retrieves the student's uploaded answer sheet image and a blank exam paper reference image, enhances the answer sheet image to obtain an enhanced answer sheet image, determines the handwritten text area based on the enhanced answer sheet image and the blank exam paper reference image, and renders the handwritten text area onto the blank exam paper reference image to obtain a rendered image.

[0155] The document image correction method provided in this specification can match the beautified answer sheet image with a blank answer sheet reference image, extract the handwritten text area from the beautified answer sheet image, and render the extracted handwritten text area onto the corresponding position on the blank answer sheet reference image. This completely solves problems such as image distortion, rotation, and translation, as well as problems such as image cutouts and thick fonts that occur after beautifying the answer sheet image based on a pre-trained semantic segmentation model. This achieves the purpose of correcting the answer sheet image, ensuring the correction effect, and thus ensuring the efficiency and accuracy of subsequent automated processing of the answer sheet image.

[0156] Corresponding to the above method embodiments, this specification also provides embodiments of a document image correction device. Figure 7 A schematic diagram of a document image correction device according to an embodiment of this specification is shown. Figure 7 As shown, the device includes:

[0157] The acquisition module 702 is configured to acquire the image to be corrected and the corresponding reference image. The image to be corrected and the reference image are both images corresponding to the document. The reference image does not include handwritten content, and the printed content in the reference image is consistent with the image to be corrected.

[0158] Matching module 704 is configured to match the image to be corrected with a reference image to determine the handwritten text region in the image to be corrected;

[0159] The rendering module 706 is configured to render the handwritten text area at the corresponding rendering position in the reference image to obtain the target image after correction.

[0160] Optionally, the matching module 704 is further configured as follows:

[0161] Perform a perspective transformation on the image to be corrected to obtain the initial corrected image after the transformation.

[0162] Identify the damaged handwritten text regions in the initial corrected image;

[0163] Generate a corresponding handwritten text mask based on the damaged handwritten text area;

[0164] Based on the handwritten text mask, determine the handwritten text region in the initial calibration image.

[0165] Optionally, the matching module 704 is further configured as follows:

[0166] Identify matching feature point pairs in the image to be corrected and the reference image;

[0167] Based on the feature point pairs, generate the homography matrix of the image to be corrected and the reference image;

[0168] A perspective transformation is performed on the image to be corrected using a homography matrix to obtain the initial corrected image after transformation.

[0169] Optionally, the matching module 704 is further configured as follows:

[0170] Obtain the first feature point and the first feature point descriptor of the image to be corrected, and obtain the second feature point and the second feature point descriptor of the reference image;

[0171] The first and second feature point descriptors are matched to determine the first candidate feature point in the image to be corrected and the second candidate feature point in the reference image.

[0172] Calculate the distance between the first candidate feature point and the second candidate feature point, and take the first candidate feature point and the second candidate feature point whose distance is less than the distance threshold as feature point pairs.

[0173] Optionally, the matching module 704 is further configured as follows:

[0174] Based on the feature point pair and noise optimization algorithm, noise point pairs are removed from the feature point pairs to obtain optimized feature point pairs;

[0175] Based on the optimized feature point pairs, a homography matrix is ​​generated for the image to be corrected and the reference image.

[0176] Optionally, the matching module 704 is further configured as follows:

[0177] Binarize the initial calibration image and the reference image separately;

[0178] Morphological erosion is performed on the binarized reference image to obtain a loss reference image;

[0179] The loss reference image is subtracted from the binarized initial correction image to obtain the lossy handwritten text region.

[0180] Optionally, the matching module 704 is further configured as follows:

[0181] Morphological erosion is performed on the damaged handwritten text area to obtain the restored handwritten text area;

[0182] Perform a bitwise NOT operation on the recovered handwritten text region, and calculate the connected components of the recovered handwritten text region after the bitwise NOT operation to obtain the rectangular coordinates of each connected component;

[0183] Generate a corresponding handwritten text mask based on the size of the initial corrected image;

[0184] Based on the rectangular coordinates of each connected component, the handwritten text mask is filled with rectangles.

[0185] Optionally, the matching module 704 is further configured as follows:

[0186] Based on the handwritten text mask, the initial calibration image is copied to obtain the handwritten text area.

[0187] Optionally, the acquisition module 702 is further configured as follows:

[0188] Get the initial image uploaded by the user;

[0189] The initial image is corrected using a semantic segmentation model to obtain the image to be corrected.

[0190] Optionally, the rendering module 706 is further configured as follows:

[0191] Based on the area coordinates of the handwritten text region, determine the corresponding rendering coordinates in the reference image;

[0192] The handwritten text area is rendered at the rendering position indicated by the rendering coordinates to obtain the target image after correction.

[0193] The document image correction device provided in this specification can match the image to be corrected with a reference image, extract the handwritten text region in the image to be corrected, and render the extracted handwritten text region onto the corresponding position on the reference image. This completely solves problems such as document image shape distortion, rotation, and translation, as well as problems such as image cutouts and thick fonts that occur after beautifying the document based on a pre-trained semantic segmentation model. It achieves the purpose of correcting document images, ensures the correction effect of document images, and thus ensures the efficiency and accuracy of subsequent automated processing of document images.

[0194] The above is a schematic scheme of a document image correction device according to this embodiment. It should be noted that the technical solution of this document image correction device and the technical solution of the document image correction method described above belong to the same concept. For details not described in detail in the technical solution of the document image correction device, please refer to the description of the technical solution of the document image correction method described above.

[0195] Figure 8 A structural block diagram of a computing device according to an embodiment of this specification is shown. The components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0196] The computing device 800 also includes an access device 840, which enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 840 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0197] In one embodiment of this specification, the above-described components of the computing device 800 and Figure 8 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 8 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0198] The computing device 800 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 800 can also be a mobile or stationary server.

[0199] The processor 820 is used to execute the following computer-executable instructions to implement the steps of the document image correction method described above.

[0200] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the document image correction method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the document image correction method described above.

[0201] An embodiment of this specification also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, are used to implement the steps of the document image correction method described above.

[0202] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the document image correction method described above. Details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the document image correction method described above.

[0203] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the document image correction method described above.

[0204] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the document image correction method described above belong to the same concept. Details not described in detail in the computer program's technical solution can be found in the description of the document image correction method described above.

[0205] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0206] Computer instructions include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0207] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this specification is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this specification. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this specification.

[0208] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0209] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. These embodiments have been selected and specifically described in this specification to better explain the principles and practical applications of this specification, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method of document image correction, characterized by, The method comprises: acquiring a picture to be corrected and a corresponding reference picture, wherein the picture to be corrected and the reference picture are pictures corresponding to a document, the reference picture does not include handwritten content, and printed content in the reference picture is consistent with the picture to be corrected; performing perspective transformation on the picture to be corrected to obtain an initial corrected picture after transformation; performing binarization processing on the initial corrected picture and the reference picture respectively; performing a morphological erosion operation on the binarized reference picture to obtain a loss reference picture; subtracting the loss reference picture from the binarized initial corrected picture to obtain a loss handwritten text area; performing a morphological erosion operation on the loss handwritten text area to obtain a restored handwritten text area, performing a bitwise inversion operation on the restored handwritten text area, and calculating connected components of the restored handwritten text area after the bitwise inversion operation to obtain rectangular coordinates of each connected component, generating a corresponding handwritten text mask based on the size of the initial corrected picture, and performing rectangular filling on the handwritten text mask according to the rectangular coordinates of each connected component; and determining a handwritten text area in the initial corrected picture according to the handwritten text mask. rendering the handwritten text area in a corresponding rendering position in the reference picture to obtain a target picture after correction of the picture to be corrected.

2. The document picture correction method according to claim 1, characterized by, The method comprises: determining a matched feature point pair in the picture to be corrected and the reference picture; generating a homography matrix of the picture to be corrected and the reference picture according to the feature point pair; performing perspective transformation on the picture to be corrected by using the homography matrix to obtain the initial corrected picture after transformation.

3. The document picture correction method according to claim 2, characterized in that, The method comprises: acquiring a first feature point and a first feature point descriptor of the picture to be corrected, and acquiring a second feature point and a second feature point descriptor of the reference picture; matching the first feature point descriptor and the second feature point descriptor to determine a first candidate feature point in the picture to be corrected and a second candidate feature point in the reference picture; calculating distances of the first candidate feature point and the second candidate feature point, and taking the first candidate feature point and the second candidate feature point with distances less than a distance threshold as the feature point pair.

4. The document picture correction method according to claim 2, characterized by, The method comprises: removing noise points in the feature point pair according to the feature point pair and a noise point optimization algorithm to obtain an optimized feature point pair; generating a homography matrix of the picture to be corrected and the reference picture according to the optimized feature point pair.

5. The document picture correction method of claim 1, wherein, The method comprises: copying the initial corrected picture according to the handwritten text mask to obtain the handwritten text area.

6. The document picture correction method according to any one of claims 1 to 5, characterized in that, The method comprises: acquiring an initial picture uploaded by a user; performing an initial correction operation on the initial picture by using a semantic segmentation model to obtain the picture to be corrected.

7. The document picture correction method according to any one of claims 1 to 5, characterized by, The rendering of the handwritten character region in the corresponding rendering position in the reference picture obtains a target picture after correction of the picture to be corrected. According to the region coordinates of the handwritten character region, corresponding rendering coordinates are determined in the reference picture; The handwritten character region is rendered at the rendering position indicated by the rendering coordinates, and a target picture after correction of the picture to be corrected is obtained.

8. A document image correction apparatus characterized by comprising: The device comprises: An acquisition module configured to acquire a picture to be corrected and a corresponding reference picture, wherein the picture to be corrected and the reference picture are pictures corresponding to a document, the reference picture does not include handwritten content, and the printed content in the reference picture is consistent with the picture to be corrected; A matching module configured to perform perspective transformation on the picture to be corrected to obtain an initial corrected picture after transformation; perform binarization processing on the initial corrected picture and the reference picture respectively; perform a morphological erosion operation on the binarized reference picture to obtain a loss reference picture; subtract the loss reference picture from the binarized initial corrected picture to obtain a loss handwritten character region; perform a morphological erosion operation on the loss handwritten character region to obtain a restored handwritten character region, perform a bitwise NOT operation on the restored handwritten character region, and calculate the connected components of the restored handwritten character region after the bitwise NOT operation to obtain rectangular coordinates of each connected component, generate a corresponding handwritten character mask based on the size of the initial corrected picture, and perform rectangular filling on the handwritten character mask according to the rectangular coordinates of each connected component; and determine a handwritten character region in the initial corrected picture according to the handwritten character mask. A rendering module configured to render the handwritten character region in the corresponding rendering position in the reference picture to obtain a target picture after correction of the picture to be corrected.

9. A computing device comprising: a memory and a processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions to realize the steps of the document picture correction method in any one of claims 1-7.

10. A computer readable storage medium storing computer instructions, which, when executed by a processor, realize the steps of the document picture correction method in any one of claims 1-7.

Images