Method for correcting a document image, device therefor and electronic equipment

Abstract

The application provides a document image correction method, a device thereof and an electronic device. The method comprises: obtaining a document image; inputting the document image into an image iteration module to obtain a plurality of preliminary document images; constructing a three-dimensional information prediction model, inputting each preliminary document image into the three-dimensional information prediction model to obtain three-dimensional structure information of each preliminary document image, and integrating the three-dimensional structure information of each preliminary document image to obtain three-dimensional structure information of the document image; constructing an image correction model, determining a transformation matrix according to at least the preliminary document image, the three-dimensional structure information of the document image and the image correction model, and correcting the document image by using the transformation matrix to obtain a target document image. The method improves the training efficiency of the model through the image iteration module, the recognition effect of the document image by the transformation matrix based on the three-dimensional structure information is good, and thus the problems of slow recognition speed and poor effect of the document image in the prior art are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a document image correction method, a document image correction device, a computer readable storage medium and an electronic device. BACKGROUND

[0002] The document photographed under different illuminations, different devices and different angles appears distortion phenomenon. In a more common case, due to the deformation of paper, the transformation of camera position and other real condition restrictions, the photograph obtained by photographing is very difficult for subsequent automatic information extraction and content analysis, and even the first step of OCR (Optical Character Recognition) recognition is rejected. At the same time, in a general scene, the background of the device for collecting documents is changing, resulting in document stacking, so that the target document to be processed is blurred in the background, and thus the automatic recognition error is large.

[0003] Therefore, there is an urgent need for a document image correction method which is fast and has good recognition effect.

[0004] The above information disclosed in the background section is only used to enhance the understanding of the background of the technology described herein, therefore, the background section can contain certain information which is not known to those skilled in the art as prior art in the country. SUMMARY

[0005] The main purpose of the present application is to provide a document image correction method, a document image correction device, a computer readable storage medium and an electronic device, so as to solve the problem of slow recognition speed and poor effect of the prior art document image.

[0006] In order to achieve the above purpose, according to one aspect of the present application, a document image correction method is provided, comprising: acquiring a document image, the document image being an image containing a document; inputting the document image into an image iteration module to obtain a plurality of preliminary document images, the image iteration module being used for iterating the document image; constructing a three-dimensional information prediction model, inputting each of the preliminary document images into the three-dimensional information prediction model to obtain three-dimensional structure information of each of the preliminary document images, and integrating the three-dimensional structure information of each of the preliminary document images to obtain the three-dimensional structure information of the document image; constructing an image correction model, determining a transformation matrix according to at least the preliminary document image, the three-dimensional structure information of the document image and the image correction model, and correcting the document image by using the transformation matrix to obtain a target document image.

[0007] Optionally, the image correction model is constructed, and a transformation matrix is determined according to the preliminary document image, the three-dimensional structure information of the document image, and the image correction model, including: setting a plurality of reference points on the preliminary document image; inputting the preliminary document image and the three-dimensional structure information of the document image into the image correction model to obtain a relative position coordinate change of each reference point on a corrected image; and calculating the transformation matrix according to the relative position coordinate change.

[0008] Optionally, the document image is corrected by using the transformation matrix to obtain a target document image, including: projecting the document image to a predetermined plane based on the transformation matrix to obtain the target document image.

[0009] Optionally, the document image is obtained, including: obtaining an initial document image, and determining a number of document regions and areas of the document regions in the initial document image based on a segmentation positioning network, the initial document image being an image containing a document; in a case where the number of the document regions is greater than a first predetermined value, sorting the document regions according to the area from large to small, and extracting a predetermined number of the document regions to form the one-to-one corresponding document image.

[0010] Optionally, the initial document image is obtained, including: obtaining an initial image, and determining whether the initial image contains a document; in a case where the initial image does not contain the document, obtaining a new initial image; in a case where the initial image contains the document, determining an area of the document region in the initial image, and determining the initial image in which the area of the document region is greater than a predetermined area as the initial document image.

[0011] Optionally, before the document image is input into the image iteration module, the method further includes: performing edge expansion processing on the document image, the edge expansion processing being to expand four edges of the document image respectively in all directions, a set of two opposite edges of the document image being expanded by a second predetermined value, and another set of two opposite edges being expanded by a third predetermined value, and the expanded area being filled with predetermined pixels.

[0012] Optionally, the three-dimensional information prediction model is a convolutional neural network model.

[0013] According to another aspect of the present application, there is provided a document image correction apparatus, comprising an obtaining unit, an iteration unit, a prediction unit and a correction unit, wherein the obtaining unit is configured to obtain a document image, the document image being an image containing a document; the iteration unit is configured to input the document image into an image iteration module to obtain a plurality of preliminary document images, the image iteration module being configured to iterate the document image; the prediction unit is configured to construct a three-dimensional information prediction model, input each of the preliminary document images into the three-dimensional information prediction model to obtain three-dimensional structure information of each of the preliminary document images, and integrate the three-dimensional structure information of each of the preliminary document images to obtain the three-dimensional structure information of the document image; and the correction unit is configured to construct an image correction model, determine a transformation matrix according to at least the preliminary document images, the three-dimensional structure information of the document image and the image correction model, and correct the document image using the transformation matrix to obtain a target document image.

[0014] According to yet another aspect of the present application, there is provided a computer-readable storage medium, comprising a stored program, wherein the program, when executed, controls a device in which the computer-readable storage medium is located to perform any of the document image correction methods.

[0015] According to still another aspect of the present application, there is provided an electronic device, comprising one or more processors, a memory, a display apparatus and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs comprise a program for performing any of the document image correction methods.

[0016] With the technical scheme, the correction method of the document image comprises the following steps: obtaining a document image, the document image being an image containing a document; inputting the document image into an image iteration module to obtain a plurality of preliminary document images, the image iteration module being configured to iteratively process the document image; constructing a three-dimensional information prediction model, inputting each of the preliminary document images into the three-dimensional information prediction model to obtain three-dimensional structure information of each of the preliminary document images, and integrating the three-dimensional structure information of each of the preliminary document images to obtain the three-dimensional structure information of the document image; constructing an image correction model, determining a transformation matrix based on at least the preliminary document images, the three-dimensional structure information of the document image, and the image correction model, and correcting the document image by using the transformation matrix to obtain a target document image. The method can reduce the complexity of the model by pre-processing the document image into a plurality of preliminary document images, inputting the plurality of preliminary document images into the three-dimensional information prediction model and the image correction model, thereby improving the training efficiency of the model. The transformation matrix obtained based on the three-dimensional structure information can improve the recognition effect of the document image, thereby solving the problems of slow recognition speed and poor effect of the document image in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which form a part of the present description, illustrate the present application and together with the written description serve to explain the principles of the present application. In the drawings:

[0018] Figure 1 A flowchart of a correction method of a document image according to an embodiment of the present application is shown;

[0019] Figure 2 A structural diagram of a segmentation positioning network according to an embodiment of the present application is shown;

[0020] Figure 3 A structural diagram of a Unet network according to an embodiment of the present application is shown;

[0021] Figure 4 A flowchart of a correction device of a document image according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0023] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application, so that those skilled in the art can better understand the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0025] It should be understood that when an element (such as a layer, film, region, or substrate) is described as "on" another element, it can be directly on the other element, or there can be an intermediate element. Moreover, in the specification and claims, when an element is described as "connected" to another element, it can be "directly connected" to the other element, or "connected" to the other element through a third element.

[0026] For ease of description, the following describes some nouns or terms related to the embodiments of the present application:

[0027] Convolutional Neural Networks (CNN): a class of feedforward neural networks containing convolutional computation and having deep structure, is one of the representative algorithms of deep learning.

[0028] Mask R-CNN: an instance segmentation algorithm that can be used for "object detection", "object instance segmentation", and "object key point detection".

[0029] Document page extraction work: for document extraction work, the commonly used technical solution is to detect the four corner points of the document, then predict the approximate position of the document area according to the straight line between the points, and then manually adjust.

[0030] As described in the background, the prior art document image recognition speed is slow and the effect is poor, in order to solve the above problems, in a typical embodiment of the present application, a document image correction method, a document image correction device, a computer readable storage medium and an electronic device are provided.

[0031] According to the embodiments of the present application, a document image correction method is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from here.

[0032] Figure 1 is a flowchart of the document image correction method according to the embodiments of the present application. As shown in Figure 1 the method comprises the following steps:

[0033] Step S101, obtaining a document image, the document image is an image containing a document;

[0034] In another embodiment of the present application, the initial document image is obtained, including: obtaining an initial image, and determining whether the initial image contains a document; in the case that the initial image does not contain a document, obtaining a new initial image; in the case that the initial image contains a document, determining the area of the document region in the initial image, and determining that the initial image with the area of the document region greater than a predetermined area is the initial document image. Specifically, the initial image can be a picture taken by the user in any scene, when the initial image does not include a document, a new initial image is reacquired, the user can be prompted to retake an image with a document, when the initial image contains a document, in order to improve the subsequent recognition effect of the document, the initial image with the area of the document region greater than a predetermined area is determined for recognition.

[0035] In practical applications, if it is detected that the area of the document region is less than 10% of the area of the initial image, a prompt information is given that the document region is too small, prompting the user to retake; if the area of the document region is between 10% and 50% of the area of the initial image, the user can be prompted to adjust the area of the detection frame to select the document region; if the area of the document region is more than 50% of the area of the initial image, it can be considered that the document region has been detected, and subsequent processing is performed on this basis.

[0036] In order to accurately locate the document area and improve the effect of document recognition and correction, in another embodiment of the application, the document image is obtained, including: obtaining an initial document image, and determining the number of document areas in the initial document image and the area of each document area based on a segmentation positioning network, wherein the initial document image is an image containing a document; in the case that the number of document areas is greater than a first predetermined value, each document area is sorted in descending order of area size, and a predetermined number of document areas are extracted to form a one-to-one corresponding document image.

[0037] Specifically, there can be multiple document areas in the initial document image. The document areas are predicted according to the segmentation positioning network, and the prediction result includes four vertices of the minimum bounding rectangle of the document area, the pixel-level contour of the document area, and the area of the detected area. Each of the document areas is sorted in descending order of area size, and a predetermined number of document areas are extracted to form a one-to-one corresponding document image. In actual application, the top two document areas in terms of area size can be selected for document page extraction to obtain a document image, which can further improve the effect of subsequent correction of the document image.

[0038] In actual application, the segmentation positioning network can use an improved Mask-RCNN (Mask Region Convolution Neural Network) neural network, in which a ResNet50+FPN structure with a channel number of 128 is used for feature extraction, as shown in Figure 2 .

[0039] In step S102, the document image is input into an image iteration module to obtain a plurality of preliminary document images, and the image iteration module is used to iterate the document image.

[0040] The image iteration module can shorten the training time of the subsequent three-dimensional information prediction model and the image correction model, thereby greatly improving the efficiency of the entire document correction. In actual application, the step of iteratively processing the image in the model training stage is extracted separately, and the pckl format data is saved offline. Then, the dataloader for training the model only needs to directly download the pckl format data, and then randomly select the size of the document area to send to the model for training. At the same time, multiple data can be randomly fused during each loading, so that the data amount is greatly improved and the data complexity is increased, providing feasibility for the generalization ability and applicable scenarios of the model. The image iteration module can improve the training efficiency by 5 to 10 times.

[0041] In order to avoid the edge of the document image being distorted and the information of the document being lost, in another embodiment of the present application, before the document image is input into the image iteration module, the method further comprises: performing edge expansion processing on the document image, wherein the edge expansion processing is to expand the four edges of the document image respectively, and the two opposite edges of the document image are expanded by a second predetermined value, the other two opposite edges are expanded by a third predetermined value, and the expanded area is filled with predetermined pixels. Specifically, the top, bottom, left and right of the picture are adaptively expanded, for example, the top edge is expanded by 5% of the height of the picture, the bottom edge is expanded by 5% of the height of the picture, and the left and right edges are similarly expanded.

[0042] In step S103, a three-dimensional information prediction model is constructed, each of the prepared document images is input into the three-dimensional information prediction model, and three-dimensional structure information of each of the prepared document images is obtained. The three-dimensional structure information of each of the prepared document images is integrated to obtain the three-dimensional structure information of the document image.

[0043] In another embodiment of the present application, the three-dimensional information prediction model is a convolutional neural network model. The three-dimensional information prediction model is similar to a generative model, and therefore requires a deep network structure and good results. Therefore, the Unet structure can be used for network structure design, as shown in the following figure. Figure 3 As shown in the figure, the prepared document image is 3 channels, and the height and width are 256x256 pixels. The convolution operation is indicated by a thick arrow. In addition to the convolution in the dashed box at the bottom, the left half is the Unet downsampling structure, the convolution is 4x4 stride=2, and the right half is the Unet upsampling structure, the convolution is also 4x4 stride=2. The thin arrow in the middle of the left and right halves indicates a concat connection. The left part is concatenated with the right part, and then a convolution layer is used to change the number of channels to the same number of channels as the left level. For example, in the 512x16x16 level, the right half is first upsampled to 512x16x16 from 1024x8x8, and then concatenated with the left half 512x16x16 to form (512+512)x16x16, i.e., 1024x16x16 features, and then a convolution layer is used to reduce the number of channels to 512 to obtain the 512x16x16 of the right half. Then, the next upsampling is performed and then connected with the left side. The entire network performs the above operations from the bottom to the top.

[0044] In actual application, before the model is trained, the data needs to be preprocessed, and the rendered picture area is extracted as much as possible.

[0045] In step S104, an image correction model is constructed, a transformation matrix is determined according to at least the prepared document image, the three-dimensional structure information of the document image, and the image correction model, and the document image is corrected by using the transformation matrix to obtain a target document image.

[0046] The image correction model can have the same network structure as the three-dimensional information prediction model, and in order to increase the correction effect of the image correction model, a loop function of a document region can be added to the model for back propagation processing in a training stage.

[0047] In order to conveniently and quickly calculate the transformation matrix, in another embodiment of the present application, the image correction model is constructed, the transformation matrix is determined according to at least the prepared document image, the three-dimensional structure information of the document image, and the image correction model, and the transformation matrix is determined by comprising the following steps: a plurality of reference points are set on the prepared document image; the prepared document image and the three-dimensional structure information are input into the image correction model to obtain a relative position coordinate change of each reference point on the corrected image; and the transformation matrix is calculated according to the relative position coordinate change.

[0048] In another embodiment of the present application, the document image is corrected by using the transformation matrix to obtain a target document image, and the method comprises the following steps: the document image is projected onto a predetermined plane based on the transformation matrix to obtain the target document image. Projecting the document image onto the predetermined plane based on the transformation matrix can ensure the correction effect and improve the usability of the document.

[0049] In the method for correcting the document image, first, a document image is obtained, the document image is an image containing a document; then, the document image is input into an image iteration module to obtain a plurality of prepared document images, the image iteration module is used for iterating the document image; then, a three-dimensional information prediction model is constructed, each prepared document image is input into the three-dimensional information prediction model to obtain three-dimensional structure information of each prepared document image, and the three-dimensional structure information of each prepared document image is integrated to obtain the three-dimensional structure information of the document image; finally, an image correction model is constructed, a transformation matrix is determined according to at least the prepared document image, the three-dimensional structure information of the document image, and the image correction model, and the document image is corrected by using the transformation matrix to obtain a target document image. The method can reduce the model complexity by pre-processing the document image into a plurality of prepared document images, inputting the plurality of prepared document images into the three-dimensional information prediction model and the image correction model, thereby improving the training efficiency of the model, the transformation matrix obtained based on the three-dimensional structure information can improve the recognition effect of the document image, and thus the problem of slow recognition speed and poor effect of the document image in the prior art is solved.

[0050] The embodiment of the present application further provides a document image correction device. It should be noted that the document image correction device of the embodiment of the present application can be used to execute the method for correcting a document image provided by the embodiment of the present application. The document image correction device provided by the embodiment of the present application is introduced as follows.

[0051] Figure 4 FIG. 1 is a schematic diagram of a document image correction device according to the embodiment of the present application. As shown in the figure, the device comprises an acquisition unit 10, an iteration unit 20, a prediction unit 30 and a correction unit 40, wherein: Figure 4

[0052] The acquisition unit 10 is configured to acquire a document image, wherein the document image is an image containing a document.

[0053] The iteration unit 20 is configured to input the document image into an image iteration module to obtain a plurality of preliminary document images, wherein the image iteration module is configured to iterate the document image.

[0054] The prediction unit 30 is configured to construct a three-dimensional information prediction model, input each of the preliminary document images into the three-dimensional information prediction model to obtain three-dimensional structure information of each of the preliminary document images, and integrate the three-dimensional structure information of each of the preliminary document images to obtain the three-dimensional structure information of the document image.

[0055] The correction unit 40 is configured to construct an image correction model, determine a transformation matrix according to at least the preliminary document images, the three-dimensional structure information of the document image of the document image and the image correction model, and correct the document image by using the transformation matrix to obtain a target document image.

[0056] ​The correction device of the document image comprises: an obtaining unit configured to obtain a document image, the document image being an image containing a document; an iteration unit configured to input the document image into an image iteration module to obtain a plurality of preliminary document images, the image iteration module being configured to iterate the document image; a prediction unit configured to construct a three-dimensional information prediction model, input each preliminary document image into the three-dimensional information prediction model to obtain three-dimensional structure information of each preliminary document image, and integrate the three-dimensional structure information of each preliminary document image to obtain the three-dimensional structure information of the document image; and a correction unit configured to construct an image correction model, determine a transformation matrix according to the preliminary document image, the three-dimensional structure information of the document image, and the image correction model, and correct the document image by using the transformation matrix to obtain a target document image. The device can reduce the complexity of the model by pre-processing the document image into a plurality of preliminary document images, inputting the plurality of preliminary document images into the three-dimensional information prediction model and the image correction model, thereby improving the training efficiency of the model. The transformation matrix obtained based on the three-dimensional structure information can improve the recognition effect of the document image, thereby solving the problems of slow recognition speed and poor effect of the document image in the prior art.

[0057] The image iteration module can shorten the training time of the subsequent three-dimensional information prediction model and the image correction model, thereby greatly improving the efficiency of the entire document correction. In practical applications, the step of iteratively processing the image in the model training stage is extracted separately, and the pckl format data is saved offline. Then, the dataloader for training the model only needs to directly download the pckl format data, and then directly randomly select the size of the document region and input it into the model for training. At the same time, multiple data can be randomly fused during each loading, which greatly improves the data volume and increases the complexity of the data, thereby providing feasibility for the generalization ability and application scenarios of the model. The image iteration module can improve the training efficiency by 5 to 10 times.

[0058] In order to calculate the transformation matrix conveniently and quickly, in another embodiment of the present application, the correction unit comprises a setting module, an input module, and a calculation module. The setting module is configured to set a plurality of reference points on the preliminary document image. The input module is configured to input the preliminary document image and the three-dimensional structure information of the document image into the image correction model to obtain the relative position coordinate change of each reference point on the corrected image. The calculation module is configured to calculate the transformation matrix according to the relative position coordinate change.

[0059] In yet another embodiment of the present application, the correction unit comprises a projection module, which is configured to project the document image onto a predetermined plane based on the transformation matrix to obtain the target document image. Projecting the document image onto a predetermined plane based on the transformation matrix can ensure the correction effect and improve the usability of the document.

[0060] To accurately locate the document region and improve the correction effect of the document recognition, in another embodiment of the present application, the obtaining unit comprises an obtaining module and an extracting module. The obtaining module is configured to obtain an initial document image, and determine the number of document regions in the initial document image and the area of each document region based on a segmentation and positioning network. The initial document image is an image containing a document. The extracting module is configured to sort each document region in descending order of area size when the number of document regions is greater than a first predetermined value, and extract a predetermined number of document regions to form a one-to-one corresponding document image.

[0061] Specifically, there can be multiple document regions in the initial document image. The segmentation and positioning network predicts the document regions, and the prediction result includes four vertices of the minimum bounding rectangle of the document region, the pixel-level contour of the document region, and the area of the detected region. Each document region is sorted in descending order of area size, and a predetermined number of document regions are extracted to form a one-to-one corresponding document image. In practical applications, the top two document regions in terms of area size can be selected for document page extraction to obtain a document image, which can further improve the correction effect of the document image.

[0062] In practical applications, the segmentation and positioning network can use an improved Mask-RCNN (Mask Region Convolution Neural Network) neural network, in which the feature extraction layer uses a ResNet50+FPN structure with a channel number of 128, as shown in Figure 2 .

[0063] In another embodiment of the present application, the obtaining module comprises a first obtaining sub-module, a second obtaining sub-module, and a determining sub-module, wherein the first obtaining sub-module is configured to obtain an initial image and determine whether the initial image contains a document; the second obtaining sub-module is configured to obtain a new initial image when the initial image does not contain a document; and the determining sub-module is configured to determine the area of the document region in the initial image when the initial image contains a document, and determine the initial image in which the area of the document region is greater than a predetermined area as the initial document image. Specifically, the initial image can be a picture taken by a user in any scene. When the initial image does not contain a document, a new initial image is obtained again, and the user can be prompted to take a picture with a document again. When the initial image contains a document, in order to improve the recognition effect of the subsequent document, the initial image in which the area of the document region is greater than a predetermined area is determined for recognition.

[0064] In actual application, when it is detected that the area of the document region is less than 10% of the area of the initial image, a prompt information is given that the area of the document region is too small, and the user is prompted to take a picture again. When the area of the document region is between 10% and 50% of the area of the initial image, the user can be prompted to adjust the area of the detection frame to select the document region. When the area of the document region is greater than 50% of the area of the initial image, it can be considered that the document region has been detected, and subsequent processing is performed on this basis.

[0065] In order to avoid the loss of information of the document image due to the possible distortion of the edge of the document image, in another embodiment of the present application, before the document image is input into the image iteration module, the device further comprises a processing unit, and the processing unit is configured to perform edge expansion processing on the document image. The edge expansion processing is to expand the four edges of the document image respectively in all directions. A second predetermined value is added to each of a group of two opposite edges of the document image, a third predetermined value is added to the other group of two opposite edges, and the expanded area is filled with predetermined pixels. Specifically, the top, bottom, left and right of the picture are adaptively expanded, for example, the height of the picture area above is expanded by 5%, the height of the picture area below is expanded by 5%, and the left and right are expanded similarly.

[0066] In another embodiment of the present application, the three-dimensional information prediction model is a convolutional neural network model. The three-dimensional information prediction model is similar to a generative model, and therefore requires a deep network structure and has a good effect. Therefore, the Unet structure can be used for network structure design, for example, Figure 3As shown, the prepared document image is 3 channels, with a height and width of 256*256 pixels, and the convolution operation is indicated by the thick arrow, wherein, except for the convolution in the dashed box at the bottom, the left half is the Unet downsampling structure, the convolution is 4*4 stride=2, the right half is the Unet upsampling structure, and the convolution is also 4*4 stride=2, the thin arrow in the middle of the left and right halves indicates the concat connection, the left part is concatenated with the right part, and then a convolution layer is used to change the channel number back to the same channel number as the left level. For example, for the level of 512*16*16, the right half is first upsampled from 1024*8*8 to 512*16*16, then concatenated with the left part 512*16*16 features to form (512+512)*16*16, that is, 1024*16*16 features, and then a convolution layer is used to reduce the channel number to 512 to obtain the right half 512*16*16, and then the next upsampling is performed and then connected with the left part. The entire network performs the above operations from the bottom to the top.

[0067] In actual application, before training the model, the data needs to be preprocessed, and the rendered picture area is cut out as large as possible.

[0068] The image correction model can have the same network structure as the three-dimensional information prediction model, and in order to increase the correction effect of the image correction model, a loop function of the document area can be added to the model for back propagation processing in the training stage.

[0069] The above-mentioned document image correction device includes a processor and a memory, and the acquisition unit, the iteration unit, the prediction unit and the correction unit are all stored in the memory as program units, and the above-mentioned program units stored in the memory are executed by the processor to realize the corresponding functions.

[0070] The processor includes a core, and the core retrieves the corresponding program unit from the memory. The core can be set to one or more, and the existing technical problems of slow document image recognition speed and poor effect can be solved by adjusting the core parameters.

[0071] The memory can include a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory includes at least one memory chip.

[0072] The embodiment of the present application provides a computer readable storage medium, and the above-mentioned computer readable storage medium includes a stored program, wherein, when the above-mentioned program runs, the device where the above-mentioned computer readable storage medium is located executes the above-mentioned document image correction method.

[0073] The embodiment of the present application provides a processor used for running a program, wherein the processor is used for executing the document image correction method when the program is running.

[0074] The embodiment of the present application provides a device, the device comprising a processor, a memory, and a program stored in the memory and capable of running on the processor, wherein the processor implements at least the following steps when the program is running:

[0075] In step S101, a document image is acquired, wherein the document image is an image containing a document.

[0076] In step S102, the document image is input into an image iteration module to obtain a plurality of preliminary document images, wherein the image iteration module is used for iterating the document image.

[0077] In step S103, a three-dimensional information prediction model is constructed, each preliminary document image is input into the three-dimensional information prediction model to obtain three-dimensional structure information of each preliminary document image, and the three-dimensional structure information of each preliminary document image is integrated to obtain the three-dimensional structure information of the document image.

[0078] In step S104, an image correction model is constructed, a transformation matrix is determined according to at least the three-dimensional structure information of the preliminary document image, the three-dimensional structure information of the document image, and the image correction model, and the document image is corrected by using the transformation matrix to obtain a target document image.

[0079] The device in the present application can be a server, a PC, a PAD, a mobile phone, or the like.

[0080] The present application further provides a computer program product, which is adapted to execute the program with at least the following method steps when executed on a data processing device:

[0081] In step S101, a document image is acquired, wherein the document image is an image containing a document.

[0082] In step S102, the document image is input into an image iteration module to obtain a plurality of preliminary document images, wherein the image iteration module is used for iterating the document image.

[0083] In step S103, a three-dimensional information prediction model is constructed, each preliminary document image is input into the three-dimensional information prediction model to obtain three-dimensional structure information of each preliminary document image, and the three-dimensional structure information of each preliminary document image is integrated to obtain the three-dimensional structure information of the document image.

[0084] Step S104, constructing an image correction model, determining a transformation matrix according to the above-mentioned pre-prepared document image, the above-mentioned three-dimensional structure information of the above-mentioned document image and the above-mentioned image correction model, and correcting the above-mentioned document image by using the above-mentioned transformation matrix to obtain a target document image.

[0085] In the above-mentioned embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0086] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the above-mentioned units can be a logical function division, and in actual implementation, there can be another division way, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0087] The units described as separate components above can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed to multiple units. According to actual needs, part or all of the units can be selected to achieve the purpose of the present embodiment scheme.

[0088] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0089] The integrated unit described above, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described above in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0090] From the above description, it can be seen that the above-described embodiments of the present application achieve the following technical effects:

[0091] 1) In the above document image correction method, first, a document image is obtained, the document image being an image containing a document; then, the document image is input into an image iteration module to obtain a plurality of preliminary document images, the image iteration module being used to iterate the document image; then, a three-dimensional information prediction model is constructed, each preliminary document image is input into the three-dimensional information prediction model to obtain three-dimensional structure information of each preliminary document image, and the three-dimensional structure information of each preliminary document image is integrated to obtain the three-dimensional structure information of the document image; finally, an image correction model is constructed, a transformation matrix is determined according to at least the preliminary document image, the three-dimensional structure information of the document image, and the image correction model, and the document image is corrected using the transformation matrix to obtain a target document image. This method can reduce the complexity of the model by using an image iteration module to pre-process the document image into a plurality of preliminary document images, and then inputting the plurality of preliminary document images into a three-dimensional information prediction model and an image correction model, thereby improving the training efficiency of the model. The transformation matrix obtained based on the three-dimensional structure information makes the recognition effect of the document image better, thereby solving the problem of slow recognition speed and poor effect of the document image in the prior art.

[0092] 2) The correction device of the document image, wherein the document image is obtained by the acquisition unit, the document image is an image containing a document; the document image is input into an image iteration module by the iteration unit to obtain a plurality of preliminary document images, the image iteration module is used for iterating the document image; a three-dimensional information prediction model is constructed by the prediction unit, each preliminary document image is input into the three-dimensional information prediction model to obtain three-dimensional structure information of each preliminary document image, and the three-dimensional structure information of each preliminary document image is integrated to obtain the three-dimensional structure information of the document image; an image correction model is constructed by the correction unit, a transformation matrix is determined according to at least the preliminary document image, the three-dimensional structure information of the document image and the image correction model, and the document image is corrected by using the transformation matrix to obtain a target document image. The device can reduce the model complexity by using the image iteration module to pre-process the document image into a plurality of preliminary document images, inputting the plurality of preliminary document images into the three-dimensional information prediction model and the image correction model, thereby improving the training efficiency of the model. The transformation matrix obtained based on the three-dimensional structure information can improve the recognition effect of the document image, thereby solving the problems of slow recognition speed and poor effect of the document image in the prior art.

[0093] The above only describes the preferred embodiments of the present application and is not used to limit the present application. The present application can have various changes and modifications for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of correcting a document image, characterized by, The method comprises the following steps: acquiring a document image, the document image being an image containing a document; inputting the document image into an image iteration module to obtain a plurality of preliminary document images, the image iteration module being configured to iterate the document image; constructing a three-dimensional information prediction model, inputting each of the preliminary document images into the three-dimensional information prediction model to obtain three-dimensional structure information of each of the preliminary document images, and integrating the three-dimensional structure information of each of the preliminary document images to obtain the three-dimensional structure information of the document image; constructing an image correction model, determining a transformation matrix based on at least the preliminary document images, the three-dimensional structure information of the document image, and the image correction model, and correcting the document image using the transformation matrix to obtain a target document image, the step of iterating the document image in the model training stage is extracted separately, the pckl format data is generated offline and saved, the dataloader of the trained model directly downloads the pckl format data, the size of the document region is randomly selected to be sent into the model for training, and a plurality of pieces of the pckl format data are fused during each loading.

2. The method of claim 1, wherein, constructing an image correction model, determining a transformation matrix based on at least the preliminary document images, the three-dimensional structure information of the document image, and the image correction model, comprising: setting a plurality of reference points on the preliminary document images; inputting the preliminary document images and the three-dimensional structure information of the document image into the image correction model to obtain a relative position coordinate change of each of the reference points on the corrected image; calculating the transformation matrix based on the relative position coordinate change.

3. The method of claim 1, wherein, correcting the document image using the transformation matrix to obtain a target document image, comprising: projecting the document image to a predetermined plane based on the transformation matrix to obtain the target document image.

4. The method of claim 1, wherein, acquiring a document image, comprising: acquiring an initial document image, and determining the number of document regions and the area of each of the document regions in the initial document image based on a segmentation positioning network, the initial document image being an image containing a document; when the number of the document regions is greater than a first predetermined value, sorting the document regions in descending order of area, and extracting a predetermined number of the document regions to form a one-to-one corresponding document image.

5. The method of claim 4, wherein, acquiring an initial document image, comprising: acquiring an initial image, and determining whether the initial image contains a document; when the initial image does not contain a document, acquiring a new initial image; when the initial image contains a document, determining the area of the document region in the initial image, and determining that the initial image with an area of the document region greater than a predetermined area is the initial document image.

6. The method according to any one of claims 1 to 5, characterized in that, Before the document image is input into the image iteration module, the method further comprises: The document image is subjected to edge expansion processing, which is to expand the four edges of the document image respectively in all directions, each of a group of two opposite edges of the document image is expanded by a second predetermined value, and each of another group of two opposite edges is expanded by a third predetermined value, and the expanded area is filled with predetermined pixels.

7. The method according to any one of claims 1 to 5, characterized in that, The three-dimensional information prediction model is a convolutional neural network model.

8. A document image correction apparatus characterized by comprising: Comprise: An acquisition unit configured to acquire a document image, the document image being an image containing a document; An iteration unit configured to input the document image into an image iteration module to obtain a plurality of preliminary document images, the image iteration module being configured to iterate the document image; A prediction unit configured to construct a three-dimensional information prediction model, input each of the preliminary document images into the three-dimensional information prediction model to obtain three-dimensional structure information of each of the preliminary document images, and integrate the three-dimensional structure information of each of the preliminary document images to obtain the three-dimensional structure information of the document image; A correction unit configured to construct an image correction model, determine a transformation matrix according to at least the preliminary document images, the three-dimensional structure information of the document image, and the image correction model, and correct the document image using the transformation matrix to obtain a target document image, The device is also configured to extract the step of iterating the document image in the model training stage, generate pckl format data offline, save the pckl format data, and directly download the pckl format data by a dataloader of the trained model, randomly select the size of the document area to send to the model for training, and perform data fusion on multiple pieces of the pckl format data at each loading.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored program, wherein the program controls the device where the computer readable storage medium is located to execute the document image correction method of any one of claims 1 to 7 when the program is running.

10. An electronic device, comprising: Comprise: One or more processors, memories, display devices, and one or more programs, wherein the one or more programs are stored in the memories and configured to be executed by the one or more processors, and the one or more programs comprise a program for executing the document image correction method of any one of claims 1 to 7.

Images