Document image rectification method and apparatus, and storage medium and computer device

Abstract

The present application relates to the technical field of image processing. Disclosed are a document image rectification method and apparatus, and a storage medium and a computer device. The method comprises: acquiring a document image to be rectified, and cropping the borders of the document image to be rectified, so as to obtain an initial document image; on the basis of the initial document image, searching a preset standard document library for a standard document image corresponding to the initial document image; by means of a pre-trained document rectification model and on the basis of image features of the standard document image, rectifying image features of the initial document image, so as to obtain image mapping parameters of the initial document image; and on the basis of the image mapping parameters, performing warping processing on the initial document image, so as to obtain a rectified target document image. The method can reduce the training difficulty of a document rectification model and the collection difficulty of sample data and improve the rectification effect of a document image.

Description

Document image correction method and device, storage medium and computer device

[0001] The present application claims priority from the Chinese patent application No. 202411079065.X filed on August 7, 2024, and entitled "Document image correction method and device, storage medium and computer device", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of image processing, in particular to a document image correction method and device, a storage medium and a computer device. BACKGROUND

[0003] In the related products of AI (Artificial Intelligence) technology, especially in the applications focusing on document processing, document correction technology is playing a crucial role. Document correction can restore the distorted document image caused by shooting angle, uneven lighting or other factors to its original shape. The corrected image not only improves the visual effect, but also improves the accuracy of subsequent processing steps, such as improving the accuracy of OCR (Optical Character Recognition).

[0004] At present, the mainstream document correction method is mainly based on deep learning algorithm. This kind of method can realize the correction of image through complex neural network model, and the optimization strategy mainly focuses on two aspects: one is to refine the model structure to enhance the correction ability; the other is to improve the quality of training data to promote the learning of model.

[0005] However, with the complication of model structure, the calculation burden of model is greatly increased, which leads to the prolongation of model inference time. Moreover, in order to train a high-performance document correction model, a large amount of high-quality and diversified labeled data is usually required, however, the collection and preparation of these data often consumes time and effort, and costs a lot, which limits the development speed and application range of the model. In addition, even if the optimized model, for some extreme cases of document image, such as severe distortion or shadow obstruction, the existing document correction method is still difficult to achieve ideal correction effect, thereby affecting the reliability and user experience of document correction.

[0006] SUMMARY

[0007] Therefore, the embodiments of the present application provide a document image correction method and device, a storage medium and a computer device, which mainly aim to solve the technical problems of large training difficulty of document correction model and poor document correction effect.

[0008] According to an aspect of the present application, a document image correction method is provided, which comprises:

[0009] obtaining a document image to be corrected, performing frame clipping on the document image to be corrected to obtain an initial document image;

[0010] searching for a standard document image corresponding to the initial document image from a preset standard document library based on the initial document image;

[0011] correcting image features of the initial document image based on image features of the standard document image by using a pre-trained document correction model to obtain image mapping parameters of the initial document image;

[0012] performing deformation processing on the initial document image based on the image mapping parameters to obtain a target document image after correction.

[0013] According to another aspect of the present application, a document image correction device is provided, which comprises:

[0014] an image clipping module configured to obtain a document image to be corrected, perform frame clipping on the document image to be corrected to obtain an initial document image;

[0015] an image searching module configured to search for a standard document image corresponding to the initial document image from a preset standard document library based on the initial document image;

[0016] an image recognition module configured to correct image features of the initial document image based on image features of the standard document image by using a pre-trained document correction model to obtain image mapping parameters of the initial document image;

[0017] an image correction module configured to perform deformation processing on the initial document image based on the image mapping parameters to obtain a target document image after correction.

[0018] According to still another aspect of the present application, a storage medium having a computer program stored thereon is provided, wherein the program is executed by a processor to implement the above-mentioned document image correction method.

[0019] According to yet another aspect of the present application, a computer device is provided, which comprises a storage medium, a processor and a computer program stored in the storage medium and executable on the processor, wherein the processor implements the above-mentioned document image correction method when executing the program.

[0020] By the technical solution, the document image correction method and device, the storage medium and the computer device provided by the embodiment of the application can more accurately identify and correct the complex deformation and distortion in the initial document image by using the standard document image as the reference image of the document image to be corrected and using the document correction model to perform feature matching and mapping calculation, so that the details such as the text and graphics in the original image can be completely reserved, thereby avoiding the problems such as blurring and distortion that may be introduced by the traditional method, and improving the correction accuracy of the document. In addition, the document correction model used in the method can be trained by using a conventional model structure and training data, without the need for a complex network architecture or a large amount of specific annotation data, thereby reducing the cost and time of model training. Moreover, since the document correction model learns the image mapping relationship between the deformed image and the standard image, the document correction model can learn more rich and accurate correction strategies, thereby enhancing the generalization ability of the model and the prediction ability of the image mapping relationship.

[0021] The above description is only a summary of the technical solutions of the application. In order to enable the technical means of the application to be more clearly understood, the application can be implemented in accordance with the content of the description, and in order to enable the above and other purposes, features and advantages of the application to be more apparent and easy to understand, the following specific embodiments of the application are described. BRIEF DESCRIPTION OF DRAWINGS

[0022] The drawings described herein are used to provide further understanding of the application, and form a part of the application. The schematic embodiments of the application and the description thereof are used to explain the application, and do not constitute an improper limitation on the application. In the drawings:

[0023] FIG. 1 shows a flowchart of a document image correction method provided by an embodiment of the application;

[0024] FIG. 2 shows a flowchart of another document image correction method provided by an embodiment of the application;

[0025] FIG. 3 shows a page diagram of a document image to be corrected provided by an embodiment of the application;

[0026] FIG. 4 shows a page diagram of a standard document image provided by an embodiment of the application;

[0027] FIG. 5 shows a page diagram of a target document image provided by an embodiment of the application;

[0028] FIG. 6 shows a structural diagram of a document image correction device provided by an embodiment of the application;

[0029] FIG. 7 shows a structural diagram of another document image correction device provided by an embodiment of the application. DETAILED DESCRIPTION

[0030] Hereinafter, the present application will be described in detail with reference to the accompanying drawings and in conjunction with embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0031] At present, the document correction technology generally relies on deep learning algorithms, and the core optimization strategy of the algorithm mainly focuses on the improvement of model architecture and the improvement of training data quality. However, complex model structure may slow down the inference speed of the model, thereby limiting the application of the document correction function on mobile platforms; and high-quality training data is costly and difficult to fully cover all application scenarios, especially in the case of extreme deformation, it is often difficult to achieve a relatively ideal correction effect.

[0032] In view of the above problems, the present application proposes a document image correction scheme for a specific scene. Among them, the specific scene refers to the document correction scene suitable for the "standard template" characteristics, such as education assistance (test papers, teaching materials, exercise books, etc.), medical records (medical records, examination reports, etc.), legal documents (contracts, judgments, etc.), financial documents (invoicing, receipts, etc.), and the like. In these scenarios, the document type is relatively fixed, so the pre-defined standard gallery can be used to compare and correct the document image provided by the user, which not only improves the efficiency and accuracy of the correction and enhances the correction ability in the case of extreme deformation, but also effectively simplifies the training process of the model, reduces the dependence of the model on high-quality sample data, and enables the correction model to achieve excellent correction performance through the conventional structure and sample data set.

[0033] In one embodiment, the core idea of the above-mentioned document correction method is shown in FIG. 1, a document image correction method is provided, and the method is applied to a computer device such as a server or a client, including the following steps:

[0034] Step 101, obtaining a document image to be corrected, and performing frame cutting on the document image to be corrected to obtain an initial document image.

[0035] Among them, the document image refers to a digital image obtained by scanning or photographing a paper document containing text, graphics and other information. Among them, the document image may be deformed or distorted due to factors such as shooting angle and light condition. Frame cutting refers to removing the excess edge part from the original document image, such as removing the excess background, frame, etc. during scanning or photographing, to obtain an initial document image that focuses more on the document content.

[0036] Specifically, the computer device can collect the document image to be corrected through a terminal device such as a scanner, a digital camera, a smart mobile device, and the like, or can obtain the document image to be corrected through a network or an electronic document (such as a PDF file) and the like. The extracted document image can be distorted, inclined, or irregular due to reasons such as an incorrect shooting angle, a page wrinkle, uneven light, and the like. Further, the computer device can identify the boundary in the document image through an image processing technology such as an edge detection algorithm, a contour recognition algorithm, a page content extraction model, and the like, and then cut off all regions outside the identified boundary to only keep the part containing the document content, thereby obtaining an initial document image. The boundary of the document image can be a white edge of the document, a shadow area, or an area with a significant color change. This step can reduce background noise and unnecessary processing areas in the document image, so that the document image is easier to search.

[0037] In step 102, based on the initial document image, a standard document image corresponding to the initial document image is searched in a preset standard document library.

[0038] The standard document library refers to a database containing standard document images of various types and formats. The standard document images in the standard document library are usually high-quality, non-deformed, and standardized images. The standard document images can be stored in various forms. For example, for each standard document in the standard document library, the document image itself and the image features, text content, and features of the identification object (such as a two-dimensional code, a bar code, or an encoded image, or a specific identification such as a document title, an identification, and the like) corresponding to the document image can be stored. By storing various information of the standard document image in the standard document library, the search efficiency and accuracy of the standard document image can be improved. In this embodiment, the standard document library can continuously receive new documents and automatically update the new documents to the standard library, thereby improving the search ability for new documents.

[0039] Specifically, the computer device can compare the initial document image with the images in the standard document library through a feature matching algorithm (such as SIFT, SURF, etc.) or a deep learning model (such as a convolutional neural network CNN), and calculate the similarity or distance between the initial document image and the standard document image in the standard document library through the feature matching algorithm or the deep learning model, so as to search for the standard image that matches the initial image most as the standard document image corresponding to the initial document image according to the comparison result of the similarity or distance. For example, referring to FIG. 3, a document image to be corrected is provided, and the initial document image obtained after the frame is cropped can be used as a basic file, and the standard document image corresponding to the initial document image is searched in the standard document library, as shown in FIG. 4. As shown in FIGS. 3 and 4, FIG. 3 is an original document image captured and uploaded by a user, which not only contains document information, but also contains background information and user handwriting trace information, while the standard document image only contains the same document information as that in FIG. 3. In addition, FIG. 3 is also a high-quality, non-deformed and standardized image.

[0040] In step 103, the image features of the initial document image are corrected based on the image features of the standard document image through the pre-trained document correction model, and the image mapping parameters of the initial document image are obtained.

[0041] The document correction model refers to a deep learning model trained by a large number of standard document images and corresponding deformed images, which can learn the mapping relationship from the deformed image to the standard image, and can use a convolutional neural network (CNN), a generative adversarial network (GAN) or other types of deep learning architecture. The image features refer to the features in the image that can be used to distinguish and identify different objects or contents, such as edges, textures, shapes, etc. The image mapping parameters refer to the deformation parameters required to convert the initial document image into an image in a standard state, such as echo point coordinates, pixel mapping relationships, etc.

[0042] Specifically, the computer device can input the initial document image and the standard document image into the document correction model, and the document correction model can analyze the image features of the two images, such as edges, corner points, textures, etc., and calculate the image mapping parameters required to convert the initial document image into a document in a standard state. The image mapping parameters can include echo point coordinates, pixel mapping relationships, etc., wherein the image mapping parameters can deform the initial document image into a layout and shape that can match the standard document image.

[0043] In step 104, the initial document image is deformed based on the image mapping parameters to obtain a corrected target document image.

[0044] Specifically, the computer device can use image deformation techniques such as affine transformation algorithms, perspective transformation algorithms, and the like, to perform deformation processing on the initial document image according to the calculated image mapping parameters. During the deformation processing, each pixel or point in the initial document image can be adjusted in position according to the mapping parameters to eliminate distortion and tilt in the image, thereby obtaining a corrected target document image. The target document image can visually present the correct direction and shape, and the content remains clear and intact.

[0045] It should be noted that the image mapping parameters of the initial document image predicted by the computer device refer to the mapping relationship between all feature points in the initial document image and the feature points on the corrected document, which includes the original content in the document page and the trace features left by the user on the document page, such as the user's written content, etc. The document correction model can make the model obtain the appearance of the image in the ideal state (i.e., the standard document image) by inputting the initial document image and the corresponding standard document image, which helps the model learn how to align the feature points in the initial document image with the feature points in the standard document image, thereby improving the accuracy and robustness of the correction. At the same time, the model only uses the standard document image as a reference for the recovery of the initial document image in terms of text form and image form, and the correction target of the model is still the initial document image itself. Thus, the model will only correct the points or pixels in the initial document image to make the initial document image recover to a more ideal state. For example, as shown in FIGS. 3-5, FIG. 3 is a document image to be corrected, and by performing a bounding box cropping operation, an initial document image can be obtained, FIG. 4 is a standard document image corresponding thereto, and FIG. 5 is a target document image obtained by performing correction processing on the initial document image using the standard document image. As can be seen from FIGS. 3-5, the document correction model can not only recover the document content in FIG. 3, but also recover the trace information in the document, thereby ensuring the lossless display of the document content and improving the quality of the document image and the user's experience.

[0046] The technical scheme provided by the embodiment can more accurately identify and correct complex deformation and distortion in the initial document image by using a standard document image as a reference image of the document image to be corrected and using a document model to perform feature matching and mapping calculation, so that details such as text and graphics in the original image can be completely preserved, thereby avoiding problems such as blurring and distortion that can be introduced by traditional methods, and improving the correction accuracy of the document. In addition, the document correction model used in the above method can be trained by using a conventional model structure and training data, without the need for a complex network architecture or a large amount of specific labeled data, thereby reducing the cost and time of model training. Moreover, since the document correction model learns the image mapping relationship between the deformed image and the standard image, the document correction model can learn more rich and accurate correction strategies, thereby enhancing the generalization ability of the model and the prediction ability of the image mapping relationship.

[0047] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to completely describe the specific implementation process of the embodiment, another document image correction method is provided, as shown in FIG. 2, which includes the following steps:

[0048] In step 201, a sample document image and a standard document image corresponding to the sample document image are obtained, and a neural network model is trained based on the sample document image and the standard document image to obtain a document correction model.

[0049] Specifically, before model training, the computer device can first collect a large number of sample document images and their corresponding standard document images as a training data set. The sample document images can come from various scanning devices or photographing devices, and therefore, the sample document images can have various deformations and noises, such as distortion, tilt, perspective transformation, and the like. In contrast, the standard document image is the version of the sample document image that has been accurately corrected, and is usually a rectangular document image with correct content layout. In addition, a part of the samples in the training data set can be set to lack a standard document image. In the embodiment, an image with the same size as the sample document image can be generated for the sample lacking the standard document image, and the background color of the image can be a uniform color, for example, all set to white. This sample processing method can ensure that the model can learn to correct the document content as much as possible without a standard image reference, and place the corrected content on a white background, thereby simulating a standardized output.

[0050] Further, the computer device can train the selected neural network model by using the processed sample document image and its corresponding standard document image or corresponding white background image. During the model training process, the model can learn to extract features from the input sample document image and predict the corresponding corrected image, i.e., the model can try to minimize the difference between the predicted image and the standard image. This process can involve iterative steps such as forward propagation to calculate the loss, backward propagation to update the model parameters, and the like, until the model performance reaches a preset standard, i.e., the training of the document correction model is completed.

[0051] By setting sample document images containing various deformations and noises, the embodiment can enable the document correction model to learn how to extract key information from these complex scenarios and effectively correct the document image, so as to more accurately restore the original appearance of the document when processing document images in actual applications, and improve the accuracy and efficiency of information extraction. In addition, for samples lacking standard document images, the embodiment introduces a white background image as a substitute standard, so that the model can learn how to handle such special cases during the training process. This processing method not only reduces the difficulty and cost of data collection, but also significantly improves the correction ability of the model for such samples. In actual applications, when encountering similar document images without standard references, the model can correct them to a relatively standard state based on the learned knowledge, thereby expanding the application range and practicality of the model.

[0052] Step 202, obtaining a document image to be corrected, performing frame cropping on the document image to be corrected to obtain an initial document image.

[0053] Specifically, the implementation process of step 202 can refer to step 101, which will not be repeated here.

[0054] In an optional embodiment, the document image to be corrected can be frame cropped by at least one of the following methods, and in addition, the two methods can be used in combination, which is not limited here.

[0055] Step 202-1, by using a pre-trained contour recognition model, extracting a document page feature from the document image, and based on the document page feature, identifying the position of the document page in the document image, and performing frame cropping on the document image according to the position of the document page to obtain an initial document image.

[0056] In this embodiment, the computer device can use a pre-trained contour recognition model for the bounding box cropping. First, the contour recognition model can be trained on a large number of images annotated with document page features through deep learning techniques, so as to learn to recognize features such as page edges and corner points unique to document images. Then, when processing the document image to be corrected, the contour recognition model can extract these features and determine the exact position of the document page based on them. Subsequently, the computer device can crop the document image according to the recognized document page position, thereby removing the background or irrelevant content outside the page to obtain the initial document image.

[0057] Step 202-2, image enhancement processing is performed on the document image to be corrected, and contour information in the document image after image enhancement processing is recognized through an edge detection algorithm and a contour finding algorithm. The document image is cropped based on the contour information to obtain the initial document image.

[0058] In this embodiment, the computer device can also perform bounding box cropping in combination with image enhancement, edge detection, and contour finding algorithms. First, the computer device can perform image enhancement processing on the document image, such as contrast enhancement, noise suppression, etc., to improve image quality and make edge information clearer. Then, an edge detection algorithm (such as the Canny edge detector) can be applied to identify edge points in the image, which usually correspond to the boundaries of the document page. Next, a contour finding algorithm (such as the Hough transform) can be used to extract complete contour lines from the edge points, which are the boundaries of the document page. Finally, the document image can be cropped based on the recognized contour information to obtain the initial document image.

[0059] It should be noted that steps 202-1 and 202-2 can be used separately or in combination to improve the accuracy and robustness of the bounding box cropping. For example, step 202-1 can be used for preliminary cropping, and step 202-2 can be used for fine-tuning the results, etc., which is not limited in this embodiment.

[0060] This embodiment can remove unnecessary edge parts in the document image by performing bounding box cropping on the document image to be corrected, making the object of subsequent correction processing more concentrated, thereby reducing the computational load of image processing and improving the efficiency of image processing. Moreover, bounding box cropping helps to more accurately locate the position of the document page, thereby providing more accurate input for subsequent correction processing. Especially when the document image has a complex background or deformation, bounding box cropping can remove irrelevant external interference factors, so that the correction model can focus more on the document content itself, thereby improving the accuracy of correction.

[0061] In step 203, a standard document image corresponding to the initial document image is searched from the preset standard document library based on the initial document image.

[0062] Specifically, the implementation process of step 203 can refer to step 102, which will not be repeated here.

[0063] In an optional embodiment, the standard document image corresponding to the initial document image can be searched by at least one of the following methods, and the three methods can also be used in combination, which will not be specifically limited here.

[0064] In step 203-1, image feature extraction is performed on the initial document image, and similarity comparison is performed between the extracted image features and the image features of each standard document image in the standard document library, and based on the similarity comparison result of the image features, the standard document image corresponding to the initial document image is determined.

[0065] In the embodiment, the computer device can search the standard document image by using a global image feature comparison method. Specifically, the computer device can first extract features from the initial document image by using an image feature extraction technique to generate a feature descriptor representing the global characteristics of the image. Then, the image features of each standard document image in the standard document library can be traversed (the image features of the standard document images in the standard document library can be extracted when searching, or can be extracted when the standard image is stored in the library, and stored with the standard document image). Subsequently, the computer device can calculate the similarity between the feature descriptors of the initial document image and the standard document library by using a feature matching algorithm. Finally, based on the similarity comparison result, the standard document image with the highest similarity is selected as the standard document image corresponding to the initial document image.

[0066] In step 203-2, a key region is extracted from the initial document image, and similarity comparison is performed between the image features of the key region and the image features of the key region of each standard document image in the standard document library, and based on the similarity comparison result of the image features of the key region, the standard document image corresponding to the initial document image is determined, wherein the key region includes a coding pattern and / or a specific identifier.

[0067] In the embodiment, the computer device can search the standard document images by using a key region feature comparison method. Specifically, based on the fact that a document image often contains some specific key regions, such as a two-dimensional code, a bar code, a company logo, a specific text box, and the like, these regions usually play an important role in the unique identification of the document. Therefore, the image features of these key regions can be accurately located and extracted from the initial document image. Subsequently, only the image features of the key regions of the initial document image and the key regions of the standard document images in the standard document library are compared for similarity, so as to determine the standard document image corresponding to the initial document image. This method can focus more on the core features of the document, and improve the accuracy and efficiency of the search.

[0068] In step 203-3, the text information in the initial document image is extracted, and the text information is compared for similarity with the text information of each standard document image in the standard document library. Based on the similarity comparison result of the text information, the standard document image corresponding to the initial document image is determined.

[0069] In the embodiment, the computer device can also search the standard document images by using a text information comparison method. Specifically, for a document image containing a large amount of text information, the text content in the initial document image can also be extracted by using an optical character recognition technology. Then, the extracted text information is compared with the text information of each standard document image in the standard document library. The comparison of the text content can be based on a simple string matching algorithm, or a more complex natural language processing technology can be used to evaluate the semantic similarity between the texts. Finally, based on the similarity comparison result of the text information, the standard document image most matching the initial document image is determined.

[0070] It should be noted that in actual applications, one or more of the above search methods can be combined according to the specific characteristics and needs of the document image to achieve a better search effect. Corresponding to the above-mentioned multiple image search methods, the standard document library can store at least one kind of information of the standard document images, including the standard document images themselves and the image features, the image features of the key regions, and the text information corresponding to the standard document images, and the like, so as to improve the search efficiency and search accuracy of the standard document images.

[0071] By adopting multiple search strategies, including global image feature comparison strategy, key area feature comparison strategy and text information comparison strategy, the embodiment can flexibly select one or more image search strategies according to specific application scenarios, thereby significantly improving the search efficiency and search accuracy of standard document images. In addition, when facing complex and variable document images, such as scenes with severe deformation, occlusion and noise, by combining multiple search methods, the feature information in the document image can be more comprehensively captured, thereby improving the adaptability and robustness of the system to various types of document images. Even in the case of missing or blurred feature information, effective matching can be achieved through other feature information.

[0072] Step 204, when no standard document image corresponding to the initial document image is searched, a reference image with the same size as the initial document image and a preset background color is generated, and the reference image is taken as the standard document image corresponding to the initial document image.

[0073] Specifically, when no standard document image corresponding to the initial document image is searched, the computer device can obtain the width and height information of the initial document image, thereby determining the size of the reference image. Then, the computer device can determine a preset background color for the reference image, wherein the preset color is usually a general and non-interfering color, such as white (#FFFFFF), light gray (#F5F5F5) and the like, so as to facilitate subsequent processing. Subsequently, the computer device can create a new image file according to the size of the initial document image and the preset background color, and the new image file has the same size as the initial document image, and all pixels of the new image file are set to the preset background color. After generating the reference image, it can be taken as the standard document image corresponding to the initial document image. In the subsequent processing process, the reference image can be used to replace the missing standard document image for document correction and the like.

[0074] By generating a reference image as a substitute standard document image when no standard document image corresponding to the initial document image is searched, the embodiment can ensure the continuity of the document processing process. This processing method avoids interruption or failure of processing due to the lack of standard document images, and improves the stability and reliability of the system. In addition, by using the reference image as the standard document image corresponding to the initial document image, the document correction model can be used to correct the initial document image compatible with the reference image, thereby providing a flexible and effective solution for the case where the corresponding standard document image cannot be found.

[0075] Step 205, by the pre-trained document correction model, the image features of the initial document image are corrected based on the image features of the standard document image, to obtain the image mapping parameters of the initial document image.

[0076] Specifically, the implementation process of step 205 can refer to step 103, which will not be repeated here.

[0077] In an optional implementation, the document correction model can obtain the image mapping parameters of the initial document image by: inputting the image features of the initial document image and the image features of the standard document image into the pre-trained document correction model, and performing feature fusion on the image features of the initial document image and the image features of the standard document image by the document correction model, and then predicting the mapping relationship between the initial document image and the standard document image based on the fused image features by the document correction model to obtain the image mapping parameters of the initial document image.

[0078] Specifically, the computer device can input the image features of the initial document image and the standard document image into the pre-trained document correction model, which can include edge information, texture features, shape features, and the like. Among them, the document correction model has been trained on a large number of document image pairs with correct mapping relationship, and has learned how to predict the mapping relationship between images based on image features. Therefore, inside the model, the model will perform feature fusion processing on the input image features, that is, fuse the two sets of features into a unified feature representation for subsequent processing. Then, the model can predict the mapping relationship between the initial document image and the standard document based on this fused feature representation. This mapping relationship can be pixel-level, that is, the correspondence between each pixel point in the two images, or a coarse-grained mapping based on key points or pixel blocks. Finally, the image mapping parameters of the initial document image can be calculated according to the predicted mapping relationship. These parameters can be echo point coordinates, that is, the coordinate positions where specific key points in the initial document image should be located after correction, or pixel mapping relationship, that is, the positions where each pixel point or pixel block in the initial document image should be mapped to the document image after correction.

[0079] The present embodiment can accurately predict the mapping relationship between the initial document image and the standard document by pre-training the document correction model to correct the initial document image, thereby achieving accurate correction of the initial document image. In addition, the present embodiment can select a more appropriate correction method according to actual needs by correcting based on key points or correcting based on pixel mapping, so that the document correction method can be applied to different types of document images and different correction demand scenarios.

[0080] In an optional implementation, the image mapping parameters include echo point coordinates or pixel mapping relationship, and the image mapping parameters of the initial document image can be obtained by any of the following methods.

[0081] Step 205-1, by the document correction model, based on the fused image features, predicting the coordinate positions of the key points in the initial document image to the key points on the standard document image, obtaining the key point coordinates of the initial document image, and performing interpolation transformation processing based on the key point coordinates to obtain the echo point coordinates of the initial document image.

[0082] In the present embodiment, the document correction model can obtain the image mapping parameters of the initial document image by using a prediction method based on key point coordinates. Specifically, the document correction model can extract key information from the fused image features and analyze and predict the coordinate positions of a series of key points in the initial document image on the document in the standard state. The key points can include document corner points, edge intersection points, positioning points of various text contents in the document, etc. Then, the model can perform interpolation transformation processing on the predicted key points. Interpolation transformation is a mathematical method that can be used to estimate the coordinates of other positions in the image based on known key point coordinates. In the present embodiment, the key point coordinates can be used as control points to calculate the positions of multiple points in the initial document image after correction by using interpolation algorithms such as bilinear interpolation, bicubic interpolation, etc., thereby obtaining the echo point coordinates.

[0083] The present embodiment can calculate the mapping relationship between the initial document image and the document in the standard state by using the prediction of key point coordinates and interpolation transformation algorithms. The above method has small calculation amount and strong robustness, and is more suitable for document images in which key points are easy to extract and stable. In addition, the application of interpolation transformation also guarantees the smoothness and continuity of the mapping results, thereby improving the quality of the corrected image.

[0084] Step 205-2, by the document correction model, based on the fused image features, predicting the mapping relationship of the pixel blocks in the initial document image to the pixel blocks in the standard document image, or predicting the mapping relationship of the pixel points in the initial document image to the pixel points in the standard document image, obtaining the pixel mapping relationship of the initial document image.

[0085] In this embodiment, the document correction model can also adopt a method of predicting the mapping relationship between pixels or pixel blocks to obtain the image mapping parameters of the initial document image. Specifically, the document correction model can learn and predict the mapping relationship between each pixel or pixel block in the initial document image and the corresponding pixel or pixel block in the document in the standard state based on the fused image features. This mapping relationship can be a point-to-point accurate mapping or a coarse-grained mapping based on pixel blocks. Point-to-point mapping requires the model to predict each pixel point, which has high accuracy but high computational complexity; the mapping based on pixel blocks simplifies the problem by considering the overall relationship between pixel blocks, which can improve computational efficiency.

[0086] By predicting the mapping relationship between pixels or pixel blocks, this embodiment can achieve more fine and accurate correction of the initial document image. Compared with the key point-based method, this method has higher accuracy and flexibility, and is especially suitable for application scenarios that require high correction results. At the same time, its computational complexity is relatively high, and it requires more powerful computing resources and longer processing time. Therefore, in actual application, the specific document correction model can be adaptively selected according to actual needs and resource conditions.

[0087] Step 206, based on the image mapping parameters, performing deformation processing on the initial document image to obtain a corrected target document image.

[0088] Specifically, the computer device can select a corresponding deformation algorithm according to the type of image mapping parameters, such as echo point coordinates or pixel mapping relationship. Specifically, if the image mapping parameters are echo point coordinates, image affine transformation or perspective transformation algorithms can be used to calculate the transformation matrix from the key points in the initial document image to the target position (i.e. echo point coordinates), so as to realize the deformation processing of the entire image. This image deformation method is relatively simple and can be applied to most document correction scenarios. Further, if the image mapping parameters are pixel mapping relationships, grid-based deformation algorithms such as Thin Plate Spline (TPS) or Radial Basis Function (RBF) interpolation algorithms can be used to calculate the new positions of all pixel points on the entire image according to the given mapping relationship between pixel points or pixel blocks, thereby realizing accurate deformation of the image.

[0089] In this embodiment, by deforming the initial document image, the processed target document image can accurately reflect the mapping relationship between the initial document image and the standard state document, while maintaining the clarity and readability of the image. In this embodiment, the target document image can also be post-processed, such as smoothing filtering, sharpening enhancement, etc., to further improve the quality of the image.

[0090] In this embodiment, the initial document image is deformed based on the image mapping parameters, which can accurately correct the initial document image and eliminate its original deformation, tilt, etc. Moreover, the automatic correction process can be completed without human intervention, greatly improving the efficiency of document processing. At the same time, since the correction process is based on the pre-trained model and image mapping parameters, it has high stability and reliability. The corrected target document image also has better visual effect and readability, making the text, graphics and other information in the document more clear and readable.

[0091] Step 207, the corrected target document image is subjected to text completion processing and format correction processing to obtain a corrected target document image, and the corrected target document image is output.

[0092] Specifically, the computer device can detect the text content in the corrected target document image, and when it detects that the text content in the target document image is missing, it can complete the missing text content in the target document image based on the text content in the standard document image, and then use the pre-trained format correction model to correct the format of the target document image with the completed text content to obtain a format-corrected target document image. In this embodiment, when the text content in the target document image is not missing, the format correction model can be used to correct the format of the target document image directly to make the format of the target document image more uniform and standardized, thereby improving the visual effect of the document.

[0093] In this embodiment, the computer device can detect the text content of the corrected target document image through optical character recognition (OCR) technology or deep learning-based text detection algorithms. This process can identify the text regions and their specific content in the image, providing a basis for subsequent text completion processing. In the case of missing text content in the target document image, such as partial text being obscured, blurred, or missing, the missing text can be completed based on the text content in the standard document image. During the completion process, the text at the corresponding position in the standard document image can be extracted and fused into the missing area of the target document image through image processing techniques such as font matching, color adjustment, etc., to restore the complete text content. After completing the text completion, the target document image can be processed for format correction using a pre-trained format correction model. The format correction model, by pre-learning the specification standards of various document formats, can identify format errors in the document, such as inconsistent line spacing, inconsistent font size, alignment errors, etc., and automatically adjust and correct these errors to ensure that the document is visually cleaner and more standardized, facilitating subsequent reading and processing. Finally, the target document image after text completion and format correction can be output for further use or saving by the user.

[0094] Through text completion processing, this embodiment can effectively restore the missing text content in the target document image, ensuring the integrity and accuracy of the document information. Moreover, through format correction processing, it can automatically identify and correct format errors in the document, making the document more visually clean and standardized, reducing misunderstandings or errors caused by format problems. In addition, both the text completion and format correction processes can be completed based on automated algorithms and models without any manual intervention, greatly improving the efficiency and accuracy of document processing and reducing labor costs and time costs.

[0095] The technical solution provided by this embodiment can more accurately identify and correct complex deformations and distortions in the initial document image by using a standard document image as a reference image for the document image to be corrected and using a document model for feature matching and mapping calculation, ensuring that the text, graphics, and other detailed information in the original image are completely preserved, improving the correction accuracy of the document. Moreover, the above method can further improve the standardization and readability of the document image through text completion and format correction operations, thereby improving the visual effect of the document image. In addition, the document correction model used in the above method can be trained using conventional model structures and training data without complex network architectures or large amounts of specific labeled data, reducing the cost and time of model training. Moreover, the trained document correction model can also correct the initial document image for which no standard document image is found, thereby improving the applicability of the model.

[0096] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties. In addition, the labels corresponding to the steps in the above embodiments only play an identification role and do not limit the execution order of the steps. The execution order of the steps in each embodiment can be set according to the actual situation.

[0097] Further, as a specific implementation of the method shown in FIGS. 1 to 5, the embodiment of the present application provides a document image correction device, as shown in FIG. 6, which comprises:

[0098] The image cropping module 31 can be configured to obtain a document image to be corrected, and crop a frame of the document image to be corrected to obtain an initial document image.

[0099] The image searching module 32 can be configured to search a standard document image corresponding to the initial document image from a preset standard document library based on the initial document image.

[0100] The image correction module 33 can be configured to correct image features of the initial document image based on image features of the standard document image through a pre-trained document correction model to obtain image mapping parameters of the initial document image.

[0101] The image processing module 34 can be configured to perform deformation processing on the initial document image based on the image mapping parameters to obtain a target document image after correction.

[0102] In a specific application scenario, the image cropping module 31 can be configured to extract a document page feature from the document image through a pre-trained contour recognition model, and identify a position of a document page of the document image based on the document page feature. The document image is cropped based on the position of the document page to obtain an initial document image.

[0103] In a specific application scenario, the image cropping module 31 can be configured to perform image enhancement processing on the document image to be corrected, and identify contour information in the document image after image enhancement processing through an edge detection algorithm and a contour searching algorithm. The document image is cropped based on the contour information to obtain an initial document image.

[0104] In a specific application scenario, the image search module 32 can be configured to perform image feature extraction on the initial document image, and perform similarity comparison between the extracted image features and image features of each standard document image in the standard document library, and determine the standard document image corresponding to the initial document image based on the similarity comparison result of the image features.

[0105] In a specific application scenario, the image search module 32 can be configured to extract a key region from the initial document image, and perform similarity comparison between image features of the key region and image features of a key region of each standard document image in the standard document library, and determine the standard document image corresponding to the initial document image based on the similarity comparison result of the image features of the key region, wherein the key region includes a coding pattern and / or a specific identifier.

[0106] In a specific application scenario, the image search module 32 can be configured to extract text information from the initial document image, and perform similarity comparison between the text information and text information of each standard document image in the standard document library, and determine the standard document image corresponding to the initial document image based on the similarity comparison result of the text information.

[0107] In a specific application scenario, as shown in FIG. 7, the above device further includes an image generation model 35, which can be configured to generate a reference image with the same size as the initial document image and a preset background color when no standard document image corresponding to the initial document image is searched out, and take the reference image as the standard document image corresponding to the initial document image.

[0108] In a specific application scenario, the image correction module 33 can be configured to input the image features of the initial document image and the image features of the standard document image into the pre-trained document correction model, perform feature fusion on the image features of the initial document image and the image features of the standard document image through the document correction model, and predict the mapping relationship between the initial document image and the standard document image based on the fused image features through the document correction model, to obtain the image mapping parameters of the initial document image.

[0109] In a specific application scenario, the image mapping parameter comprises echo point coordinates; the image correction module 33 can be configured to, based on the fused image features, predict, by the document correction model, coordinate positions of key points in the initial document image to key points on the standard document image, obtain key point coordinates of the initial document image, and perform interpolation transformation processing based on the key point coordinates to obtain echo point coordinates of the initial document image.

[0110] In a specific application scenario, the image mapping parameter comprises pixel mapping relationship; the image correction module 33 can be configured to, based on the fused image features, predict, by the document correction model, mapping relationship of a pixel block in the initial document image to a pixel block in the standard document image, or predict mapping relationship of a pixel in the initial document image to a pixel in the standard document image, to obtain pixel mapping relationship of the initial document image.

[0111] In a specific application scenario, as shown in FIG. 7, the device further comprises an image post-processing model 36, which can be configured to detect the text content in the corrected target document image; when detecting that the text content in the target document image is missing, based on the text content in the standard document image, the missing text content in the target document image is completed; the target document image with completed text content is subjected to document format correction processing by a pre-trained format correction model to obtain a target document image after format correction.

[0112] It should be noted that other corresponding descriptions of the functions of the document image correction device provided in the embodiments of the present application can refer to the corresponding descriptions in the methods of FIGS. 1 to 5, which will not be described here.

[0113] The embodiments of the present application also provide a computer device, which can be a personal computer, a server, a network device, etc. The computer device comprises a bus, a processor, a memory, and a communication interface, and can further comprise an input / output interface and a display device. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is used to store position information. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement the steps in the method embodiments.

[0114] Those skilled in the art can understand that the structure of the computer device described above is only part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components, or combine certain components, or have a different arrangement of components.

[0115] In an embodiment, a computer readable storage medium is provided, which can be non-volatile or volatile, and has stored thereon a computer program, which, when executed by a processor, implements the steps in the above method embodiments.

[0116] In an embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps in the above method embodiments.

[0117] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0118] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0119] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A document image correction method, wherein, The method comprises: obtaining a document image to be corrected, performing frame cutting on the document image to be corrected to obtain an initial document image; based on the initial document image, searching for a standard document image corresponding to the initial document image in a preset standard document library; based on the image features of the standard document image, correcting the image features of the initial document image through a pre-trained document correction model to obtain image mapping parameters of the initial document image; based on the image mapping parameters, performing deformation processing on the initial document image to obtain a target document image after correction.

2. The method of claim 1, wherein, The frame cutting on the document image to be corrected to obtain an initial document image comprises: extracting document page features in the document image through a pre-trained contour recognition model, and identifying the position of the document page in the document image based on the document page features; performing frame cutting on the document image based on the position of the document page to obtain an initial document image.

3. The method of claim 1 or 2, wherein, The frame cutting on the document image to be corrected to obtain an initial document image comprises: performing image enhancement processing on the document image to be corrected, and identifying contour information in the document image after image enhancement processing through an edge detection algorithm and a contour search algorithm; based on the contour information, performing frame cutting on the document image to obtain an initial document image.

4. The method of claim 1, wherein, The searching for a standard document image corresponding to the initial document image in a preset standard document library based on the initial document image comprises: extracting image features of the initial document image, and performing similarity comparison between the extracted image features and image features of each standard document image in the standard document library; based on the similarity comparison result of the image features, determining the standard document image corresponding to the initial document image.

5. The method of claim 1 or 4, wherein, The searching for a standard document image corresponding to the initial document image in a preset standard document library based on the initial document image comprises: extracting a key area in the initial document image, and performing similarity comparison between image features of the key area and image features of a key area of each standard document image in the standard document library; based on the similarity comparison result of the image features of the key area, determining the standard document image corresponding to the initial document image, wherein the key area includes an encoding pattern and / or a specific identifier.

6. The method of claim 1 or 4, wherein, The searching for a standard document image corresponding to the initial document image in a preset standard document library based on the initial document image comprises: extracting text information in the initial document image, and performing similarity comparison between the text information and text information of each standard document image in the standard document library; based on the similarity comparison result of the text information, determining the standard document image corresponding to the initial document image.

7. The method of claim 1, wherein, The method further comprises: generate a reference image with the same size as the initial document image and a preset background color as a standard document image corresponding to the initial document image when no standard document image corresponding to the initial document image is searched out.

8. The method of claim 1, wherein, The image mapping parameter of the initial document image is obtained by correcting, by the pre-trained document correction model, image features of the initial document image based on image features of the standard document image, including: inputting the image features of the initial document image and the image features of the standard document image into the pre-trained document correction model; performing feature fusion on the image features of the initial document image and the image features of the standard document image by the document correction model; predicting, by the document correction model, a mapping relationship between the initial document image and the standard document image based on the fused image features to obtain the image mapping parameter of the initial document image.

9. The method of claim 8, wherein, The image mapping parameter includes a reflection point coordinate; the image mapping parameter of the initial document image is obtained by predicting, by the document correction model, a mapping relationship between the initial document image and the standard document image based on the fused image features, including: predicting, by the document correction model, a coordinate position of a key point in the initial document image to a key point on the standard document image based on the fused image features to obtain a key point coordinate of the initial document image; performing interpolation transformation processing based on the key point coordinate to obtain a reflection point coordinate of the initial document image.

10. The method of claim 8, wherein, The image mapping parameter includes a pixel mapping relationship; the image mapping parameter of the initial document image is obtained by predicting, by the document correction model, a mapping relationship between the initial document image and the standard document image based on the fused image features, including: predicting, by the document correction model, a mapping relationship between a pixel block in the initial document image and a pixel block in the standard document image, or predicting a mapping relationship between a pixel point in the initial document image and a pixel point in the standard document image based on the fused image features to obtain a pixel mapping relationship of the initial document image. The method further includes:

11. The method of claim 1, wherein, detecting the text content in the corrected target document image; when it is detected that the text content in the target document image is missing, completing the missing text content in the target document image based on the text content in the standard document image; performing document format correction processing on the target document image with the completed text content by a pre-trained format correction model to obtain a target document image after format correction. The device includes:

12. A document image correction apparatus, wherein, an image cropping module configured to obtain a document image to be corrected, perform border cropping on the document image to be corrected, and obtain an initial document image; an image searching module configured to search for a standard document image corresponding to the initial document image in a preset standard document library based on the initial document image; ​ An image correction module is configured to correct image features of the initial document image based on image features of the standard document image through a pre-trained document correction model to obtain image mapping parameters of the initial document image. An image processing module is configured to perform deformation processing on the initial document image based on the image mapping parameters to obtain a target document image after correction.

13. A storage medium having stored thereon a computer program, wherein The computer program, when executed by a processor, implements the method of any one of claims 1-11.

14. A computer device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein, The processor, when executing the computer program, implements the method of any one of claims 1-11.

Images