Document image correction method and device, and training method and device of correction model

Abstract

The application provides a document image correction method and device, a training method and device of a document image correction model, a computer device, and a storage medium, and solves the problem of low OCR accuracy caused by poor quality of a to-be-detected document image in the prior art. The document image correction method comprises the following steps: determining a denoising image of a document image, a deflection angle of the document image, and corner point coordinates of a document in the document image; the deflection angle is used to indicate a deflection angle of the document in the document image; determining a correction matrix based on the corner point coordinates, the deflection angle, and a preset standard document; and determining an output image based on the correction matrix, the denoising image, and the deflection angle.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a document image correction method and apparatus, a document image correction model training method and apparatus, a computer device, and a storage medium. Background Technology

[0002] Optical Character Recognition (OCR) refers to the process of using electronic devices to examine characters printed on paper, determining their shape by detecting dark and light patterns, and then translating the shape into computer text using character recognition methods. OCR technology typically requires high-quality images of the documents being inspected. However, with the widespread use of smart devices and the improvement in their camera quality, paper documents can now be converted into electronic versions simply by taking a photo. In smart device photography scenarios, the difficulty in precisely controlling the shooting angle and framing often results in poor image quality, often containing noise, perspective distortion, and tilt, leading to a decrease in OCR accuracy. Summary of the Invention

[0003] In view of this, embodiments of this application provide a document image correction method and apparatus, a document image correction model training method and apparatus, a computer device, and a storage medium to solve the problem of low OCR accuracy caused by poor quality of the document image to be detected in related technologies.

[0004] The first aspect of this application provides a document image correction method, comprising: determining a denoised image of the document image, a deflection angle, and corner coordinates of the document in the document image, wherein the deflection angle is used to indicate the deflection angle of the document in the document image; determining a correction matrix based on the deflection angle, corner coordinates, and a preset standard document; and determining an output image based on the denoised image, the deflection angle, and the correction matrix.

[0005] As can be seen, in this embodiment, the denoised image, the deflection angle, and the corner coordinates of the document image are determined. A correction matrix is ​​then determined based on the corner coordinates, the deflection angle, and a preset standard document. The output image is then determined based on the correction matrix, the denoised image, and the deflection angle. The denoised image is used to remove noise from the document image, the deflection angle is used to correct document tilt, and the correction matrix is ​​used to correct document distortion. Therefore, by comprehensively considering the denoised image, the deflection angle, and the correction matrix to determine the output image, the problems of noise, tilt, and perspective distortion in the document image can be solved simultaneously, improving the quality of the document image and thus increasing the accuracy of subsequent OCR processing. Furthermore, using the document image correction method provided in this embodiment to preprocess the document image before performing OCR reduces the quality requirements for the document image, thereby increasing its applicability to input images and making it more universal.

[0006] A second aspect of this application provides a training method for a document image correction model, comprising: determining a denoised image of the document image, a deflection angle, and corner coordinates of the document in the document image, wherein the document image is labeled with noise labels, deflection angle labels, and corner coordinate labels of the document; determining a first loss function based on the denoised image and noise labels; determining a second loss function based on corner coordinates and corner coordinate labels; determining a third loss function based on deflection angle and deflection angle labels; determining a total loss function based on the first loss function, the second loss function, and the third loss function; and adjusting the parameters of the document image correction model based on the total loss function.

[0007] As can be seen, in this embodiment, the denoised image and deflection angle of the document image are first determined, along with the corner coordinates of the document. The document image is labeled with noise, deflection angle, and corner coordinate labels. Then, a first loss function is determined based on the denoised image and noise labels, a second loss function is determined based on the deflection angle and its label, and a third loss function is determined based on the corner coordinates and their labels. Finally, a total loss function is determined based on the first, second, and third loss functions. The parameters of the document image correction model are then adjusted based on the total loss function. When preprocessing the document image using this correction model, a correction matrix can be determined based on the deflection angle and corner coordinates output by the correction model. The denoised image, deflection angle, and correction matrix are then combined to determine the output image. The denoised image is used to remove noise from the document image, the deflection angle is used to correct document tilt, and the correction matrix is ​​used to correct document distortion. Therefore, the output image determined by combining the denoised image, the deflection angle, and the correction matrix can simultaneously solve the problems of noise, tilt, and perspective distortion in document images, improve the quality of document images, and thus improve the accuracy of subsequent OCR processing.

[0008] A third aspect of this application provides a document image correction device, comprising: a first determining module for determining a denoised image of a document image, a deflection angle, and corner coordinates of a document in the document image, wherein the deflection angle is used to indicate the deflection angle of the document in the document image; a second determining module for determining a correction matrix based on the deflection angle, corner coordinates, and a preset standard document; and a third determining module for determining an output image based on the denoised image, the deflection angle, and the correction matrix.

[0009] A fourth aspect of this application provides a training apparatus for a document image correction model, comprising: a first determining module for determining a denoised image of a document image, a deflection angle, and corner coordinates of the document in the document image, wherein the document image is labeled with noise labels, deflection angle labels, and corner coordinate labels of the document; a second determining module for determining a first loss function based on the denoised image and noise labels, a second loss function based on corner coordinates and corner coordinate labels, and a third loss function based on deflection angle and deflection angle labels; a third determining module for determining a total loss function based on the first loss function, the second loss function, and the third loss function; and an adjusting module for adjusting the parameters of the document image correction model based on the total loss function.

[0010] The fifth aspect of this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executed by the processor. When the processor executes the computer program, it implements the steps of the document image correction method provided in any of the above embodiments and / or the steps of the training method of the document image correction model provided in any of the above embodiments.

[0011] The sixth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the document image correction method provided in any of the above embodiments and / or the steps of the training method for the document image correction model provided in any of the above embodiments. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of a document image provided for an embodiment of this application.

[0013] Figure 2 This is an architectural diagram of a document image correction model provided in an embodiment of this application.

[0014] Figure 3 A flowchart of a document image correction method provided in an embodiment of this application.

[0015] Figure 4 This is a schematic diagram illustrating the process of determining the correction matrix according to an embodiment of this application.

[0016] Figure 5This is a schematic diagram illustrating the correction process of a document image provided in an embodiment of this application.

[0017] Figure 6 A flowchart of the training method for the document image correction model provided in the first embodiment of this application.

[0018] Figure 7 A flowchart of the training method for the document image correction model provided in the second embodiment of this application.

[0019] Figure 8 This is a structural block diagram of a document image correction device provided in an embodiment of this application.

[0020] Figure 9 This is a structural block diagram of a training device for a document image correction model provided in an embodiment of this application.

[0021] Figure 10 This is a structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0022] Financial companies such as banks and securities firms generate a large amount of data in their business scenarios, including cards, receipts, and forms. In actual operations, this data needs to be entered into the system. Currently, in OCR scenarios, text on some common standard documents (ID cards, bank cards, business licenses) can be accurately recognized, such as document images obtained through scanning. However, as mentioned in the background, some document images are of poor quality, resulting in lower OCR accuracy.

[0023] Figure 1 This is a schematic diagram of document images provided for embodiments of this application. For example... Figure 1 As shown, in the current business scenario, the smart terminal needs to photograph paper documents to obtain electronic versions. During the photographing process, due to improper control of the shooting angle and range, the actual shooting effect is as follows... Figure 1 As shown. See also Figure 1 The paper document 110 is significantly distorted and tilted in the electronic document image 120. The edges of the paper document 110 are not fully displayed, and there is a background area Q around the paper document 110. Under these circumstances, directly performing OCR processing on the electronic document image 120 will be difficult to achieve good recognition results.

[0024] In view of this, embodiments of this application provide a document image correction method and apparatus, a document image correction model training method and apparatus, a computer device, and a storage medium. The document image correction method can be executed by the document image correction apparatus provided in this application embodiment, or by a smart terminal device integrating the document image correction apparatus. The document image correction model training method can be executed by the document image correction model training apparatus provided in this application embodiment, or by a smart terminal device integrating the document image correction apparatus training apparatus. At least one of the document image correction apparatus and the document image correction model training apparatus can be implemented in hardware or software. The smart terminal can be a smartphone, tablet computer, PDA, laptop computer, smart wearable device, or other device equipped with a processor and camera, possessing photo-taking and image processing functions.

[0025] According to the document image correction method and apparatus, document image correction model training method and apparatus, computer equipment, and storage medium provided in this application embodiment, a denoised image, a deflection angle, and corner coordinates of the document in the document image are determined. A correction matrix is ​​determined based on the corner coordinates, the deflection angle, and a preset standard document. An output image is determined based on the correction matrix, the denoised image, and the deflection angle. The denoised image is used to remove noise from the document image, the deflection angle is used to correct document tilt, and the correction matrix is ​​used to correct document distortion. Therefore, by comprehensively considering the denoised image, the deflection angle, and the correction matrix to determine the output image, the problems of noise, tilt, and perspective distortion in the document image can be solved simultaneously, improving the quality of the document image and thus increasing the accuracy of subsequent OCR processing. At the same time, using the document image correction method provided in this embodiment to preprocess the document image before performing OCR can reduce the quality requirements of the document image, thereby increasing the applicability to the input image and making it more universal.

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] Figure 2 This is an architectural diagram of a document image correction model provided in an embodiment of this application. Figure 3 A flowchart illustrating a document image correction method provided in an embodiment of this application. See also... Figure 2 and Figure 3 As shown, the document image correction method 300 includes:

[0028] Step S310: Determine the denoised image of the document image, the deflection angle, and the corner coordinates of the document in the document image.

[0029] Step S320: Determine the correction matrix based on the deflection angle, corner coordinates, and preset standard document.

[0030] Step S330: Determine the output image based on the denoised image, the deflection angle, and the correction matrix.

[0031] In step S310, the document image can be an electronic document image captured by a portable camera or camcorder. Generally, when taking photos using a portable camera or camcorder, due to the influence of shooting angle, ambient light, and framing, the captured document image will inevitably contain some noise and distortion. The document type in the document image can be a card, ticket, paper, display screen, etc.

[0032] See Figure 2 The document image correction model 10 includes three branches: a noise reduction branch 11, an angle detection branch 12, and an image regression branch 13.

[0033] The denoising branch 11 outputs a denoised image. A denoised image is the image obtained after denoising the document image. Denoising removal removes noise from the document image.

[0034] Specifically, first, features are extracted from the document image to obtain multiple feature representation vectors. This feature extraction step can be implemented using an encoder. The encoder includes multiple convolutional layers, which perform convolution operations on the input image to obtain multiple feature representation vectors. Second, a prediction image is determined based on these multiple feature representation vectors. For example, a scaling transformation is performed on the multiple feature representation vectors to obtain the prediction image. The scaling transformation operation can be implemented using at least one convolutional layer. Third, a denoised image is determined based on the prediction image; the denoised image and the document image have the same dimensions. For example, the prediction image can be directly used as the denoised image.

[0035] The skew angle detection branch 12 outputs the skew angle, which indicates the tilt angle of the document within the document image. This tilt angle is typically characterized by the offset angle between the document's reading direction and the top-down view of the document image. The reading direction refers to the direction in which the text lines are arranged from top to bottom when written linearly from left to right. In practice, a direction indicator can be set to indicate the document's reading direction. This indicator can be a line or column of text, any edge of the document, or any straight line within the document, etc. Figure 1Taking the document image shown as an example, the top-view direction L1 of the document image is vertically downward. The direction index is defined as the left edge of the document. The actual direction L2 of the left edge in the document image is offset by 10° clockwise relative to the top-view direction L1, which indicates a deflection angle of 10°. It should be understood that when the actual direction L2 is offset by 10° counterclockwise relative to the top-view direction L1, it indicates a deflection angle of -10°.

[0036] Specifically, firstly, features are extracted from the document image to obtain multiple feature representation vectors. This feature extraction process can be implemented using the same encoder as the feature extraction process in the denoising branch 11. Secondly, angle classification is performed based on the multiple feature representation vectors to obtain the deflection angle. The deflection angle detection branch 12 fully connects the multiple feature vectors output by the encoder and inputs them into the softmax layer to classify the multiple feature vectors by angle and output the deflection angle of the document image. In one embodiment, the deflection angle output by the deflection angle detection branch 12 is an 8*1 dimensional vector, where each dimension represents the probability of a preset angle, including 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°.

[0037] Image regression branch 13 includes a corner localization unit, which outputs the corner coordinates of the document in the document image. Specifically, firstly, features are extracted from the document image to obtain multiple feature representation vectors. This feature extraction process can be implemented using the same encoder as the feature extraction processes in denoising branch 11 and corner detection branch 12. Secondly, a predicted image is constructed based on the multiple feature representation vectors output by the encoder. In this embodiment, the predicted image is a denoised image, meaning that image regression branch 13 further shares the same convolutional layer with denoising branch 11 to output the predicted image. Thirdly, the corners of the document in the denoised image are located to obtain the corner coordinates of the document. Specifically, the edge probability map of the document in the document image can be determined first. The edge probability map includes the document's bounding box and the probability that each pixel on the bounding box belongs to an edge point. The edge probability map can be obtained through an edge detection model. Then, the four vertices of the document are determined from the edge probability map. For example, line detection is performed on the bounding boxes of the document in the edge probability map to obtain a set of lines based on the bounding boxes. The intersection points between each pair of lines in the set are calculated. The intersection points are roughly located in different regions. For each region, the centroid of the intersection points in that region is obtained as the vertex of the document.

[0038] In step S320, refer to Figure 2 The image regression branch 13 also includes a correction matrix determination unit, which determines the correction matrix based on the deflection angle output by the deflection detection branch 12, the corner coordinates output by the image regression branch 13, and the preset standard document.

[0039] Figure 4This is a schematic diagram illustrating the process of determining the correction matrix according to an embodiment of this application. (See attached diagram.) Figure 4 The preset standard document is an electronic document that meets the output requirements. When the standard document is in the document image, its reading direction indicates the top-down direction L1 of the document image. The size of the standard document is defined by four standard corner points, whose coordinates are known and denoted as A, B, C, and D. The process of determining the correction matrix includes: first, rotating the preset standard document based on the deflection angle to make the reading direction of the preset standard document consistent with the reading direction of the document, thus obtaining the target document. Figure 1 Taking the document image shown as an example, if the deflection angle output by the angle detection branch 12 is 10°, then the standard document should be rotated 10° clockwise, and the standard corner points A, B, C, and D should be transformed into the target corner points A0, B0, C0, and D0 respectively, thus transforming the standard document into the target document. Then, a correction matrix is ​​determined based on the coordinates of the target corner points and the corner point coordinates output by the image regression branch 13. This correction matrix is ​​used to normalize each corner point a, b, c, and d of the document to the target corner points A0, B0, C0, and D0, respectively. For example, an affine transformation matrix can be determined based on the corner point coordinates and the target corner point coordinates, serving as the correction matrix. The process of determining the affine transformation matrix can be implemented using conventional techniques, which will not be detailed here.

[0040] In this embodiment, the reading direction of the target document and the document in the document image is consistent, ensuring that the difference between the document in the document image and the target document is minimal. Therefore, the correction matrix determined based on the corner coordinates of the target document and the corner coordinates of the document in the document image has good accuracy.

[0041] Continue reading according to step S330. Figure 2 The document image correction model 10 also includes a correction module 14, which determines the output image based on the correction matrix, the denoised image, and the deflection angle.

[0042] Specifically, Figure 5 This is a schematic diagram illustrating the document image correction process according to an embodiment of this application. (See attached image.) Figure 5 Assuming Figure 1 The document image 120 shown is the denoised image. First, the denoised image is corrected based on the correction matrix to obtain an intermediate image. Specifically, a dot product operation can be performed between the denoised image and the correction matrix to normalize the corner points of the document in the denoised image to... Figure 4 The target documents A0, B0, C0, and D0 are shown, and intermediate images are obtained, as follows. Figure 5As shown. Next, the intermediate image is corrected based on the deflection angle to obtain the output image. Specifically, the reading direction of the noise-free corrected image is rotated by the opposite number of the deflection angle, i.e., 10° clockwise, to obtain the output image. The size and reading direction of the output image are consistent with the standard image. It can be seen that in this embodiment, the denoised image is corrected sequentially by the correction matrix and the deflection angle, i.e., corrected first and then rotated. This execution order depends on the correction matrix. Since the reading direction of the target document and the document image is consistent, correcting first can normalize the corner points of the document to the corner points of the target document, i.e., perform deformation correction within a small angle range, ensuring correction accuracy.

[0043] According to the document image correction method provided in this application, a denoised image, a deflection angle, and corner coordinates of the document in the document image are determined. A correction matrix is ​​then determined based on the corner coordinates, the deflection angle, and a preset standard document. Finally, an output image is determined based on the correction matrix, the denoised image, and the deflection angle. The denoised image is used to remove noise from the document image, the deflection angle is used to correct document tilt, and the correction matrix is ​​used to correct document distortion. Therefore, by comprehensively considering the denoised image, the deflection angle, and the correction matrix to determine the output image, the problems of noise, tilt, and perspective distortion in the document image can be solved simultaneously, improving the quality of the document image and thus increasing the accuracy of subsequent OCR processing. Furthermore, using the document image correction method provided in this embodiment to preprocess the document image before performing OCR reduces the quality requirements for the document image, thereby increasing its applicability to input images and making it more universal.

[0044] This application also provides a method for training a document image correction model. Figure 6 A flowchart of the training method for the document image correction model provided in the first embodiment of this application. Figure 6 The training method shown can be used for training. Figure 2 The document image correction model is shown, and it is executed. Figure 3 The document image correction method shown is 300. Combined with... Figure 2 and Figure 6 As shown, training method 600 includes:

[0045] Step S610: Determine the denoised image of the document image, the deflection angle, and the corner coordinates of the document in the document image. The document image is labeled with noise labels, deflection angle labels, and corner coordinate labels of the document.

[0046] Document images can be electronic images of documents captured by portable cameras or camcorders. Generally speaking, when taking photos using portable cameras or camcorders, due to the influence of shooting angle, ambient light, and framing, the captured document images will more or less contain some noise and distortion. The document types in the document images can be cards, tickets, papers, displays, etc.

[0047] The document images are labeled with noise tags, which indicate the noise in the document images. Specifically, under basically unchanged environmental conditions, multiple images are taken by the same camera from the same perspective, and the average of these multiple images is taken. This average image is the noise tag, and any one of the multiple images can be used as the document image.

[0048] The document images are labeled with deflection angle tags, which indicate the angle of deviation between the reading direction of the document and the top-down view of the document image. Figure 1 Taking the document image shown as an example, if the actual left edge of the document in the image points to L2, which is offset clockwise by 10° relative to L1, then the deflection angle label is 10°. In one embodiment, the deflection angle output by the deflection angle detection branch 12 is an 8*1 dimensional vector, where each dimension represents the probability of a preset angle. The preset angles include 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°. It should be understood that in this case, the deflection angle label should be selected from one of the preset angles according to the rounding principle. For example, Figure 1 The deflection angle label for the document image shown should be 0°.

[0049] The document image also includes corner coordinate labels, which indicate the actual positions of the document's corners in the image coordinate system. (Still using...) Figure 1 Taking the document image shown as an example, the corner coordinates are labeled a, b, c, and d. Points a and d are virtual points that can be determined using a bounding box detection model.

[0050] like Figure 2As shown, after the document image is input into the document image correction model 10, the denoising branch 11 outputs a denoised image, the angle detection branch 12 outputs the deflection angle, and the image regression branch 13 outputs the corner coordinates of the document in the document image. Specifically, the encoder extracts features from the document image to obtain multiple feature representation vectors. The denoising branch 11 performs a scale transformation on the multiple feature representation vectors to obtain a denoised image, which has the same dimension as the document image. The angle detection branch 12 fully connects the multiple feature vectors output by the encoder and inputs them into a softmax layer to classify the multiple feature vectors by angle and output the deflection angle of the document image. The image regression branch 13 locates the corners of the document in the denoised image to obtain the corner coordinates of the document. In this case, the denoising branch 11, the angle detection branch 12, and the image regression branch 13 share the same encoder, and the denoising branch 11 and the image regression branch 13 further share the same convolutional layer to output the denoised image, reducing algorithm redundancy. See [link to documentation] for the specific execution process of each branch. Figure 3 The embodiments shown are not described in detail here.

[0051] Step S620: Determine the first loss function based on the denoised image and noise labels.

[0052] Step S630: Determine the second loss function based on the deflection angle and the deflection angle label.

[0053] Step S640: Determine the third loss function based on the corner coordinates and corner coordinate labels.

[0054] Step S650: Determine the total loss function based on the first loss function, the second loss function, and the third loss function.

[0055] Step S660: Adjust the parameters of the document image correction model based on the total loss function.

[0056] According to the training method of the document image correction model provided in this application embodiment, firstly, the denoised image and deflection angle of the document image, as well as the corner coordinates of the document in the document image, are determined. The document image is labeled with noise labels, deflection angle labels, and corner coordinate labels. Then, a first loss function is determined based on the denoised image and noise labels, a second loss function is determined based on the deflection angle and deflection angle labels, and a third loss function is determined based on the corner coordinates and corner coordinate labels. Finally, a total loss function is determined based on the first, second, and third loss functions, and then the parameters of the document image correction model are adjusted using the total loss function. When preprocessing the document image using this document image correction model, a correction matrix can be determined based on the deflection angle and corner coordinates of the document output by the document image correction model. Then, the output image is determined by combining the denoised image, deflection angle, and correction matrix. Since denoised images can remove noise from document images, tilt angles can correct document tilt, and correction matrices can correct document distortion, combining denoised images, tilt angles, and correction matrices to determine the output image can simultaneously solve the problems of noise, tilt, and perspective distortion in document images, improve document image quality, and thus improve the accuracy of subsequent OCR processing.

[0057] In this embodiment, the noise reduction branch 11 is a fully convolutional network, and the first loss function determined in step S320 can be the cross-entropy loss function. The skew angle detection branch 12 is a classification network, and the second loss function determined in step S330 can be the cross-entropy loss function. The image regression branch 13 is a regression network, and the third loss function determined in step S340 can be the mean squared error loss function or the mean absolute error loss function.

[0058] Figure 7 A flowchart illustrating the training method for the document image correction model provided in the second embodiment of this application. Figure 7 As shown, in this embodiment, step S650 is specifically executed as follows:

[0059] Step S710: Adjust the first loss function based on the weight coefficients, where the weight coefficients are proportional to the number of iterations.

[0060] Step S720: Based on the adjusted first loss function, second loss function, and third loss function, determine the total loss function. Specifically, the total loss function is determined as the first loss function multiplied by the weight coefficient, plus the second and third loss functions.

[0061] Since the labels of the skew angle detection branch 12 and the image regression branch 13 have high accuracy, the weights of the second and third loss functions in the total loss function can both be set to 1. However, the label accuracy of the noise branch 11 is low, so directly setting the weight of the first loss function in the total loss function to 1 would result in low detection accuracy for the document image correction model 10. Therefore, the weight of the first loss function is set to increase with the number of iterations; that is, from the start to the end of training, the weight of the noise reduction branch 11 gradually increases from 0 to 1. In this case, as the noise reduction effect of the noise reduction branch 11 improves, the weight of the first loss function gradually increases, thereby improving the detection accuracy of the document image correction model 10.

[0062] In one embodiment, the weighting coefficient depends on the ratio of the current iteration number to the maximum iteration number. For example, the weighting coefficient... Where E represents the current iteration number, E max This represents the maximum number of iterations in the training process. Determining the weight coefficients based on the ratio of the current iteration number to the maximum iteration number ensures that the change pattern of the weight coefficients matches the improvement in the noise reduction effect of the noise reduction branch 11, thereby ensuring that the document image correction model 10 achieves better detection accuracy.

[0063] This application also provides a document image correction device. Figure 8 This is a structural block diagram of a document image correction device provided in an embodiment of this application. Figure 8 As shown, the document image correction device 80 includes a first determining module 81, a second determining module 82, and a third determining module 83. The first determining module 81 determines the denoised image of the document image, the deflection angle, and the corner coordinates of the document within the document image. The second determining module 82 determines a correction matrix based on the deflection angle, corner coordinates, and a preset standard document. The third determining module 83 determines the output image based on the denoised image, the deflection angle, and the correction matrix.

[0064] According to the document image correction device provided in this application embodiment, a denoised image, a deflection angle, and corner coordinates of the document in the document image are determined. A correction matrix is ​​determined based on the corner coordinates, the deflection angle, and a preset standard document. An output image is determined based on the correction matrix, the denoised image, and the deflection angle. The denoised image is used to remove noise from the document image, the deflection angle is used to correct document tilt, and the correction matrix is ​​used to correct document distortion. Therefore, by comprehensively considering the denoised image, the deflection angle, and the correction matrix to determine the output image, the problems of noise, tilt, and perspective distortion in the document image can be solved simultaneously, improving the quality of the document image and thus increasing the accuracy of subsequent OCR processing. Furthermore, using the document image correction method provided in this embodiment to preprocess the document image before performing OCR reduces the quality requirements for the document image, thereby increasing its applicability to input images and making it more universal.

[0065] In one embodiment, the first determining module 81 is specifically used for: extracting features from the document image to obtain multiple feature representation vectors; performing angle classification based on the multiple feature representation vectors to obtain the deflection angle; determining a predicted image based on the multiple feature representation vectors; and determining a denoised image and corner coordinates based on the predicted image.

[0066] When the predicted image and the document image have the same dimensions, the steps to determine the denoised image and corner coordinates based on the predicted image can be performed as follows: determine that the predicted image is a denoised image; locate the corners of the document in the denoised image and obtain the corner coordinates.

[0067] According to the document image correction device provided in this embodiment, the first determining module uses multiple feature vectors output by an encoder to simultaneously determine the predicted image, deflection angle, and corner coordinates, and the predicted image is also used to determine the denoised image and corner coordinates, thereby reducing algorithm redundancy.

[0068] In one embodiment, the second determining module 82 is specifically used to: rotate a preset standard document based on a deflection angle so that the reading direction of the preset standard document is consistent with the reading direction of the document, thereby obtaining a target document; and determine a correction matrix based on the corner coordinates and the target corner coordinates of the target document.

[0069] The step of rotating a preset standard document based on a deflection angle to make the reading direction of the preset standard document consistent with the reading direction of the document, and obtaining the target document, can be performed as follows: rotate the direction marker of the preset standard document by a deflection angle to obtain the target document, where the direction marker indicates the reading direction.

[0070] The steps for determining the correction matrix based on the corner coordinates and the target corner coordinates of the target document can be performed as follows: determine the affine transformation matrix based on the corner coordinates and the target corner coordinates, and use it as the correction matrix.

[0071] According to the document image correction device provided in this embodiment, a correction matrix is ​​determined based on the target document and the document in the document image. The offset angle between the target document and the document in the document image is small, that is, the correction matrix is ​​determined within a small angle range to ensure that the correction matrix has good accuracy.

[0072] In one embodiment, the third determining module 83 is specifically used to: correct the denoised image based on the correction matrix to obtain an intermediate image; and correct the intermediate image based on the deflection angle to obtain an output image.

[0073] The step of correcting the denoised image based on the correction matrix to obtain the intermediate image can be performed as follows: multiply the correction matrix and the denoised image by a dot product to obtain the intermediate image.

[0074] The steps to correct the intermediate image based on the deflection angle and obtain the output image can be performed as follows: rotate the intermediate image by the opposite number of the deflection angle to obtain the output image.

[0075] According to the document image correction device provided in this embodiment, the denoised image is corrected sequentially by a correction matrix and an offset angle, i.e., correction is performed first and then rotation is performed. This execution order depends on the correction matrix. Since the target document and the document in the document image are aligned in the same reading direction, correction first can normalize the corner points of the document to the corner points of the target document, i.e., deformation correction is performed within a small angle range, ensuring correction accuracy.

[0076] This application also provides a training device for a document image correction model. Figure 9 This is a structural block diagram of a training device for a document image correction model provided in one embodiment of this application. Figure 9 As shown, the training device 90 includes: a first determining module 91, a second determining module 92, a third determining module 93, and an adjustment module 94. The first determining module 91 determines the denoised image of the document image, the deflection angle, and the corner coordinates of the document in the document image. The document image is labeled with noise labels, deflection angle labels, and corner coordinate labels. The second determining module 92 determines a first loss function based on the denoised image and noise labels, a second loss function based on the corner coordinates and corner coordinate labels, and a third loss function based on the deflection angle and deflection angle labels. The third determining module 93 determines a total loss function based on the first, second, and third loss functions. The adjustment module 94 adjusts the parameters of the document image correction model based on the total loss function.

[0077] According to the training apparatus for the document image correction model provided in this application embodiment, the denoised image and deflection angle of the document image, as well as the corner coordinates of the document in the document image, are first determined. The document image is labeled with noise labels, deflection angle labels, and corner coordinate labels. Then, a first loss function is determined based on the denoised image and noise labels, a second loss function is determined based on the deflection angle and deflection angle labels, and a third loss function is determined based on the corner coordinates and corner coordinate labels. Finally, a total loss function is determined based on the first, second, and third loss functions, and the parameters of the document image correction model are adjusted according to the total loss function. When preprocessing the document image using this document image correction model, the output image can be determined based on the denoised image, deflection angle, and corner coordinates of the document output by the document image correction model. This is because the denoised image is used to remove noise from the document image, the deflection angle is used to correct document tilt, and the correction matrix is ​​used to correct document distortion. Therefore, the output image determined by combining the noise-reduced image, the deflection angle, and the corner coordinates of the document can simultaneously solve the problems of noise, tilt, and perspective distortion in the document image, improve the quality of the document image, and thus improve the accuracy of subsequent OCR processing.

[0078] In one embodiment, the first determining module 91 is specifically used for: extracting features from the document image to obtain multiple feature representation vectors; performing angle classification based on the multiple feature representation vectors to obtain the deflection angle; determining a predicted image based on the multiple feature representation vectors; and determining a denoised image and corner coordinates based on the predicted image.

[0079] When the predicted image and the document image have the same dimensions, the steps to determine the denoised image and corner coordinates based on the predicted image can be performed as follows: determine that the predicted image is a denoised image; locate the corners of the document in the denoised image and obtain the corner coordinates.

[0080] According to the training device of the document image correction model provided in this embodiment, the first determining module uses multiple feature vectors output by an encoder to simultaneously determine the predicted image, the deflection angle and the corner coordinates, and the predicted image is used to simultaneously determine the denoised image and the corner coordinates, thereby reducing algorithm redundancy.

[0081] In one embodiment, the second determining module 92 is specifically configured to: calculate a first cross-entropy loss function based on the denoised image and noise labels; calculate a second cross-entropy loss function based on corner coordinates and corner coordinate labels; and calculate a mean squared error loss function based on deflection angles and deflection angle labels.

[0082] In one embodiment, the third determining module 93 is specifically used to: adjust the first loss function based on the weight coefficients, and determine the total loss function based on the adjusted first loss function, second loss function, and third loss function. Specifically, the total loss function is determined to be the product of the first loss function and the weight coefficients, plus the second and third loss functions.

[0083] According to the training apparatus for the document image correction model provided in this embodiment, the weight of the first loss function is set to increase with the number of iterations; that is, from the start to the end of training, the weight of the denoising branch gradually increases from 0 to 1. In this case, as the denoising effect of the denoising branch improves, the weight of the first loss function gradually increases, thereby improving the detection accuracy of the document image correction model.

[0084] It should be noted that the training device for the document image correction model provided in the various embodiments of this application and the training method for the document image correction model provided in the embodiments of this application belong to the same application concept. The document image correction device provided in the various embodiments of this application and the document image correction method provided in the embodiments of this application belong to the same application concept. Technical details not described in detail in this device embodiment can be found in the method embodiment provided in this application, and will not be repeated here.

[0085] This application also provides a computer device. Figure 10 This is a structural block diagram of a computer device provided in one embodiment of this application. Figure 10 As shown, the computer device 100 includes: a memory 101, a processor 102, and a computer program stored on the memory 101 and executed by the processor 102. When the processor 102 executes the computer program, it implements the steps of the document image correction method provided in any of the above embodiments and / or the steps of the training method of the document image correction model provided in any of the embodiments.

[0086] Memory 101 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory includes random access memory (RAM) and / or cache memory, etc. Non-volatile memory includes read-only memory (ROM), hard disk, flash memory, etc. Memory 101 may also store a mapping table of special characters and pronunciation categories.

[0087] The processor 102 may be a processing unit with data processing capabilities and / or instruction execution capabilities, such as a central processing unit (CPU).

[0088] Computer programs can be written in one or more programming languages. Programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C. A computer program can execute entirely on computer device 100, partially on computer device 60 and partially on a server, or as a standalone software package.

[0089] In one embodiment, the computer device 100 further includes an input device 103 and an output device 104, which are respectively connected to the processor 102. The input device 103 may be a camera for capturing document images. The input device 103 may also be a keyboard, mouse, etc. The output device 104 can output various information to the outside, including outputting images. The output device 104 may be a monitor, speaker, printer, etc.

[0090] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the document image correction method provided in any of the above embodiments and / or the steps of the training method for the document image correction model provided in any of the above embodiments.

[0091] Computer-readable storage media can take the form of any combination of one or more readable media. A readable storage medium can be any of the following forms: electrical, magnetic, optical, electromagnetic, infrared, semiconductor, or a combination thereof. Examples of readable storage media include: hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, etc.

[0092] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A document image correction method, characterized in that, include: The denoised image of the document image, the deflection angle, and the corner coordinates of the document in the document image are determined by a pre-trained document image correction model. The deflection angle is used to indicate the tilt angle of the document in the document image. The preset standard document is rotated based on the aforementioned deflection angle to obtain the target document; Based on the corner coordinates and the target corner coordinates of the target document, determine the correction matrix; The output image is determined based on the denoised image, the deflection angle, and the correction matrix.

2. The document image correction method according to claim 1, characterized in that, The determination of the denoised image of the document image, the deflection angle, and the corner coordinates of the document in the document image includes: Feature extraction is performed on the document image to obtain multiple feature representation vectors; The deflection angle is obtained by performing angle classification based on the multiple feature representation vectors. The predicted image is determined based on the multiple feature representation vectors; The denoised image and the corner coordinates are determined based on the predicted image.

3. The document image correction method according to claim 2, characterized in that, The predicted image and the document image have the same dimensions; determining the denoised image and the corner coordinates based on the predicted image includes: The predicted image is determined to be the denoised image; The corner points of the document in the denoised image are located to obtain the corner point coordinates.

4. The document image correction method according to any one of claims 1-3, characterized in that, The reading direction of the rotated preset standard document is consistent with the reading direction of the document.

5. The document image correction method according to any one of claims 1-3, characterized in that, The step of determining the output image based on the denoised image, the deflection angle, and the correction matrix includes: The denoised image is corrected based on the correction matrix to obtain an intermediate image; The intermediate image is corrected based on the deflection angle to obtain the output image.

6. The document image correction method according to any one of claims 1-3, characterized in that, Also includes: Determine the denoised image of the document image, the deflection angle, and the corner coordinates of the document in the document image, wherein the document image is labeled with noise labels, deflection angle labels, and corner coordinate labels of the document; A first loss function is determined based on the denoised image and the noise label; A second loss function is determined based on the corner coordinates and the corner coordinate labels; A third loss function is determined based on the deflection angle and the deflection angle label; The total loss function is determined based on the weight coefficients of the first loss function, the first loss function, the second loss function, and the third loss function; the weight coefficients are proportional to the number of iterations of the document image correction model. The parameters of the document image correction model are adjusted based on the total loss function.

7. A document image correction device, characterized in that, include: The first determining module determines the denoised image of the document image, the deflection angle, and the corner coordinates of the document in the document image through a pre-trained document image correction model. The deflection angle is used to indicate the deflection angle of the document in the document image. The second determining module is used to rotate the preset standard document based on the deflection angle to obtain the target document; Based on the corner coordinates and the target corner coordinates of the target document, determine the correction matrix; The third determining module is used to determine the output image based on the denoised image, the deflection angle, and the correction matrix.

8. The document image correction device according to claim 7, characterized in that, The first determining module is further configured to determine the denoised image of the document image, the deflection angle, and the corner coordinates of the document in the document image, wherein the document image is labeled with noise labels, deflection angle labels, and corner coordinate labels of the document; The second determining module is further configured to determine a first loss function based on the denoised image and the noise label, determine a second loss function based on the corner coordinates and the corner coordinate label, and determine a third loss function based on the deflection angle and the deflection angle label; The third determining module is further configured to determine the total loss function based on the weight coefficients of the first loss function, the first loss function, the second loss function, and the third loss function; the weight coefficients are proportional to the number of iterations of the document image correction model; The device further includes an adjustment module for adjusting the parameters of the document image correction model based on the total loss function.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executed by the processor, characterized in that, When the processor executes the computer program, it implements the steps of the document image correction method as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the document image correction method as described in any one of claims 1 to 6.

Images