Document correction methods, apparatus, computer equipment and storage media

Abstract

This application relates to a document correction method, apparatus, computer device, and storage medium. The method includes: acquiring an image to be corrected; performing layer-by-layer downsampling on the image to be corrected to obtain downsampled image features at different scales; extracting image features of the image to be corrected based on a trained deep learning neural network, and determining the pixel offset matrix of the image to be corrected based on the image features; the deep learning neural network includes a cascaded first UET network and a second UET network; normalizing the output of the first UET network, concatenating the normalized features with the output of the first UET network, and using the concatenated features as the input of the second UET network; mapping the image to be corrected based on the pixel offset matrix to obtain the corrected image. The above method does not rely on environmental parameters, requires only one processing step to obtain the pixel offset matrix, and has a simple processing procedure with low computational cost.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a document correction method, apparatus, computer device, and storage medium. Background Technology

[0002] With the development of technology, technologies have emerged to convert paper documents into electronic documents. By converting paper documents into electronic documents, processing can be shifted from offline to online, improving processing efficiency and facilitating long-distance transmission and archiving. Converting paper documents into electronic documents typically involves scanning or photographing. However, during the scanning and photographing process, image distortion or tilting may occur. In such cases, the distorted or tilted document images need to be corrected.

[0003] Related techniques utilize the source image before correction and camera environment parameters as prior knowledge to calculate the vertical and horizontal distortion coefficients, ultimately correcting the source image. This method relies on learning the settings of environmental parameters, resulting in a complex processing procedure and high computational cost. Summary of the Invention

[0004] Therefore, it is necessary to provide a document correction method, apparatus, computer device, and storage medium that do not depend on environmental parameter settings to address the aforementioned technical problems.

[0005] A document correction method, the method comprising:

[0006] Obtain the image to be corrected;

[0007] Based on the trained and determined deep learning neural network, the image features of the image to be corrected are extracted, and the pixel offset matrix of the image to be corrected is determined according to the image features of the image to be corrected; the deep learning neural network includes a cascaded first UET network and a second UET network; after normalizing the output of the first UET network, the normalized features are concatenated with the output of the first UET network, and the concatenated features are used as the input of the second UET network;

[0008] The image to be corrected is mapped based on the pixel offset matrix to obtain the corrected image of the image to be corrected.

[0009] A document correction device, the device comprising:

[0010] The acquisition module is used to acquire the image to be corrected.

[0011] A pixel offset matrix determination module is used to extract image features of the image to be corrected based on a trained deep learning neural network, and to determine the pixel offset matrix of the image to be corrected based on the image features of the image to be corrected; the deep learning neural network includes a cascaded first UET network and a second UET network; after normalizing the output of the first UET network, the normalized features are concatenated with the output of the first UET network, and the concatenated features are used as the input of the second UET network;

[0012] The mapping module is used to map the image to be corrected based on the pixel offset matrix to obtain the corrected image of the image to be corrected.

[0013] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0014] Obtain the image to be corrected;

[0015] Based on the trained and determined deep learning neural network, the image features of the image to be corrected are extracted, and the pixel offset matrix of the image to be corrected is determined according to the image features of the image to be corrected; the deep learning neural network includes a cascaded first UET network and a second UET network; after normalizing the output of the first UET network, the normalized features are concatenated with the output of the first UET network, and the concatenated features are used as the input of the second UET network;

[0016] The image to be corrected is mapped based on the pixel offset matrix to obtain the corrected image of the image to be corrected.

[0017] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0018] Obtain the image to be corrected;

[0019] Based on the trained and determined deep learning neural network, the image features of the image to be corrected are extracted, and the pixel offset matrix of the image to be corrected is determined according to the image features of the image to be corrected; the deep learning neural network includes a cascaded first UET network and a second UET network; after normalizing the output of the first UET network, the normalized features are concatenated with the output of the first UET network, and the concatenated features are used as the input of the second UET network;

[0020] The image to be corrected is mapped based on the pixel offset matrix to obtain the corrected image of the image to be corrected.

[0021] The aforementioned document correction method, apparatus, computer equipment, and storage medium extract image features from the acquired image to be corrected based on a trained deep learning neural network. Based on the extracted image features, a pixel offset matrix of the image to be corrected is determined. This pixel offset matrix is ​​then used to map the image to be corrected, resulting in the corrected image. The deep learning neural network consists of two cascaded UET networks. The output of the first UET network is normalized, and the normalized result is concatenated with the output of the first UET network. The concatenated feature is used as the input to the second UET network. The output of the second UET network is the output of the deep learning neural network, i.e., the pixel offset matrix. This method uses a deep learning neural network composed of two cascaded UET networks to extract image features from the image to be corrected and determines the corresponding pixel offset matrix based on these features. The trained deep learning neural network can effectively learn the offset between the distorted image to be corrected and the corresponding reference image, thereby generating the pixel offset matrix of the image to be corrected. Then, pixel mapping is performed on the image to be corrected based on the pixel offset matrix to finally obtain the corrected image. This method does not depend on environmental parameters, and the pixel offset matrix can be obtained in just one process. The process is simple and the computation is small. Attached Figure Description

[0022] Figure 1 This is a diagram illustrating the application environment of a document correction method in one embodiment.

[0023] Figure 2 This is a flowchart illustrating a document correction method in one embodiment;

[0024] Figure 3 This is a schematic diagram of the structure of two cascaded UET networks in a specific embodiment;

[0025] Figure 4 This is a flowchart illustrating a document correction method in another embodiment;

[0026] Figure 5(1) is a schematic diagram of the image to be corrected in a specific embodiment;

[0027] Figure 5(2) is a schematic diagram of the corrected image in a specific embodiment;

[0028] Figure 6 This is a flowchart illustrating the training process of a deep learning neural network in one embodiment;

[0029] Figure 7(1) is a schematic diagram of a distorted sample image constructed in a specific embodiment;

[0030] Figure 7(2) is a schematic diagram of the distorted sample image constructed in another specific embodiment;

[0031] Figure 8 This is a schematic diagram of the image to be corrected, the corrected image, and the image results obtained from each smoothing process in a specific embodiment.

[0032] Figure 9 This is a structural block diagram of a document correction device in one embodiment;

[0033] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0035] The document correction method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network.

[0036] In some embodiments, the application environment may involve both terminal 102 and server 104. Server 104 acquires the image to be corrected input through terminal 102, performs layer-by-layer downsampling on the image to be corrected, and then performs layer-by-layer upsampling on the image features of the last downsampled layer. It merges the image features belonging to the same layer in the downsampling and upsampling to obtain a pixel offset matrix of the image to be corrected. This pixel offset matrix is ​​then used to map the image to be corrected to obtain the corresponding corrected image. Furthermore, server 104 can also feed the corrected image back to terminal 102. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0037] In other embodiments, after the terminal 102 obtains the image to be corrected, it performs downsampling, upsampling and merging operations on the image to obtain the pixel offset matrix of the image to be corrected, and then uses the pixel offset matrix to map the image to be corrected to obtain the corrected image.

[0038] In one embodiment, such as Figure 2 As shown, a document correction method is provided, which is applied to... Figure 1The following explanation uses the server as an example, including steps S210 to S230.

[0039] Step S210: Obtain the image to be corrected.

[0040] The image to be corrected is the image that needs to be corrected. In one embodiment, the image to be corrected includes a distorted image, a tilted image, and so on.

[0041] Step S220: Extract image features of the image to be corrected based on the trained and determined deep learning neural network, and determine the pixel offset matrix of the image to be corrected according to the image features of the image to be corrected; the deep learning neural network includes a cascaded first UET network and a second UET network; after normalizing the output of the first UET network, the normalized features are concatenated with the output of the first UET network, and the concatenated features are used as the input of the second UET network.

[0042] In this embodiment, the UET network includes a preset number of downsampling modules and a preset number of upsampling modules. The preset number of downsampling modules downsample the input image / input image features layer by layer to obtain downsampled images at different scales. The preset number of upsampling modules upsample the downsampled image features output by the last downsampling module layer by layer to obtain upsampled image features at different scales. Upsampled image features and downsampled image features belonging to the same scale are then connected across layers to obtain cross-layer connected image features. The scales of the image features output by the downsampling and upsampling modules correspond one-to-one.

[0043] Subsampling, also known as downsampling, has two main purposes: to make the image fit the size of the display area and to generate a thumbnail of the corresponding image. The principle of subsampling is as follows: For an image I of size M*N, it is downsampled by a factor of s, resulting in a resolution image of size (M / s)*(N / s), where s is the common divisor of M and N. If considering a matrix image, this means transforming the image within an s*s window of the original image into a single pixel, and the value of this pixel is the average value of all pixels within the window. In one embodiment, the final image feature output by layer-by-layer downsampling is an image feature of a preset size, for example, a 1*1 image feature.

[0044] In one embodiment, the structure of any downsampling module in the UET network can be completely identical or not completely identical. In one embodiment, the structures of the downsampling modules are the same; the downsampling module includes a leaky linear correction unit, a convolutional unit, and a batch normalization unit; further, in one specific embodiment, the convolutional unit of the downsampling module has a kernel size of 4 and a stride of 2.

[0045] The main purpose of upsampling is to enlarge the original image so that it can be displayed on a higher resolution display device. The principle of upsampling is that image enlargement almost always uses interpolation methods, that is, inserting new elements between pixels of the original image using a suitable interpolation algorithm.

[0046] In one embodiment, the structure of any layer upsampling module in the UET network can be completely identical or not completely identical. In one embodiment, the structures of the upsampling modules are the same, and the upsampling module includes a linear correction unit, a deconvolution unit, a batch normalization unit, and a merging unit; further, in one specific embodiment, the kernel size of the deconvolution unit is 4, and the stride is 2.

[0047] In this embodiment, after upsampling the downsampled image features at the last scale layer by layer, upsampled image features at different scales are obtained. Furthermore, each scale of the upsampled feature image corresponds to each scale of the downsampled image features.

[0048] Furthermore, after obtaining the downsampled image features and upsampled image features at each scale, each upsampling module also performs cross-layer concatenation between the upsampled image features output by this layer and the downsampled image features belonging to the same scale, thereby obtaining cross-layer concatenated image features. In one embodiment, cross-layer concatenation of downsampled image features and upsampled image features at the same scale can be implemented in any way.

[0049] Furthermore, in one embodiment, after connecting downsampled and upsampled image features belonging to the same scale across layers, the method further includes: interval correction of the connected image features. In one embodiment, an activation function can be used to perform interval correction on the connected image features. In a specific embodiment, a hyperbolic tangent unit (tanh) is used to perform interval correction on the merged output to obtain an output between [-1, 1].

[0050] like Figure 3 The diagram shown is a schematic representation of the structure of a deep learning neural network consisting of two cascaded UET networks in a specific embodiment.

[0051] Step S230: Map the image to be corrected based on the pixel offset matrix to obtain the corrected image of the image to be corrected.

[0052] In this embodiment, the corrected image is the image output after the image to be corrected has been corrected. After obtaining the pixel offset matrix of the image to be corrected, the pixel offset matrix is ​​used to perform pixel-by-pixel mapping on the image to be corrected to obtain the corrected image corresponding to the image to be corrected.

[0053] In one embodiment, such as Figure 4 As shown, before mapping the image to be corrected based on the pixel offset matrix to obtain the corrected image, the method further includes step S410: smoothing the pixel offset matrix to obtain a smoothed offset matrix; in this embodiment, mapping the image to be corrected based on the pixel offset matrix to obtain the corrected image includes step S420: mapping the image to be corrected based on the smoothed offset matrix to obtain the corrected image.

[0054] Smoothing, also known as filtering, or combined as smoothing filtering, is a spatial domain filtering technique that enhances low frequencies. Its purposes are twofold: blurring and noise reduction. In this embodiment, the purpose of smoothing the pixel offset matrix is ​​to eliminate noise. In one embodiment, smoothing includes mean filtering, median filtering, Gaussian filtering, and bilateral filtering, among other methods.

[0055] Further, in one embodiment, smoothing the pixel offset matrix to obtain a smoothed offset matrix includes: performing Gaussian smoothing on the pixel offset matrix with a window of a first preset size to obtain a Gaussian smoothed offset matrix; and performing average smoothing on the Gaussian smoothed offset matrix with a window of a second preset size to obtain a smoothed offset matrix.

[0056] Gaussian smoothing is a linear smoothing filter suitable for eliminating Gaussian noise and widely used in image processing noise reduction. Simply put, Gaussian filtering is a weighted averaging process across the entire image; the value of each pixel is obtained by weighted averaging of its own value and the values ​​of other pixels in its neighborhood. The conventional operation of Gaussian filtering is to scan each pixel in the image using a template (or convolution, mask), and replace the value of the center pixel of the template with the weighted average gray value of the pixels in the neighborhood defined by the template. In this embodiment, a window of a first preset size is used to smooth the pixel offset matrix. Further, in a specific embodiment, the horizontal window size used in Gaussian smoothing is h / 2, and the vertical window size is w / 3, where h and w represent the height and width of the image to be corrected, respectively.

[0057] In one specific embodiment, the horizontal window size in the second preset window of the average smoothing is h / 2, and the vertical window size is w / 3, where h and w represent the height and width of the image to be corrected, respectively. Furthermore, in one embodiment, the Gaussian smoothing offset matrix can be subjected to multiple average smoothing operations; repeated operations can yield better restoration results for the corrected image.

[0058] In this embodiment, after smoothing the pixel offset matrix, the image to be corrected is mapped to remove as much noise as possible caused by the downsampling and upsampling process in the pixel offset matrix. This avoids problems such as local discontinuities and a subjective artificial feel in the corrected matrix when using the pixel offset matrix for correction. When using the smoothed pixel offset matrix for correction, the text in the corrected image is close to horizontal, without local distortion, and the subjective evaluation by the human eye is not artificial. Figure 5(1) shows a schematic diagram of the image to be corrected, and Figure 5(2) shows the corrected image output after correcting the image to be corrected in a specific embodiment.

[0059] The document correction method described above extracts image features from the acquired image to be corrected based on a trained deep learning neural network. It then determines the pixel offset matrix of the image to be corrected based on these features, and finally maps this pixel offset matrix to the image to obtain the corrected image. The deep learning neural network consists of two cascaded UET networks. The output of the first UET network is normalized, and then concatenated with the output of the first UET network. The concatenated features are used as the input to the second UET network. The output of the second UET network is the output of the deep learning neural network, i.e., the pixel offset matrix. This method uses a deep learning neural network composed of two cascaded UET networks to extract image features from the image to be corrected and determines the corresponding pixel offset matrix based on these features. The trained deep learning neural network can effectively learn the offset between the distorted image to be corrected and the corresponding reference image, thereby generating the pixel offset matrix of the image to be corrected. Finally, pixel mapping is performed on the image to be corrected based on this pixel offset matrix to obtain the corrected image. This method does not depend on environmental parameters, and the pixel offset matrix can be obtained in just one process. The process is simple and the computation is small.

[0060] In one embodiment, such as Figure 6 As shown, the training process of a deep learning neural network includes steps S610 to S640.

[0061] Step S610: Input the distorted sample image into the preset deep learning neural network framework.

[0062] Deep learning (Machine Learning, ML) is a new research direction in the field of machine learning. Machine learning is an interdisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0063] Step S620: Obtain the predicted pixel offset matrix of the distorted sample image output by the preset deep learning neural network framework.

[0064] The preset deep learning neural network framework is pre-defined, and the deep learning neural network is obtained by training the preset deep learning neural network. In one embodiment, the preset deep learning neural network framework includes two cascaded UET networks. The UET network includes a preset number of downsampling modules and a preset number of upsampling modules connected in sequence. Each downsampling module includes a leaky linear correction unit, a convolution unit, and a batch normalization unit. Each upsampling module includes a linear correction unit, a deconvolution unit, a batch normalization unit, and a merging unit. The merging unit performs cross-scale concatenation between the upsampling image features of its layer and the downsampling image features of the corresponding scale to obtain cross-scale concatenated image features.

[0065] In another embodiment, the merging unit can also be set independently of the upsampling module. That is, the upsampling module only includes a linear correction unit, a deconvolution unit, and a batch normalization unit. After the upsampling module outputs the upsampled image features, the merging unit performs cross-scale connection on the upsampled image features and the downsampled image features of the same scale to obtain cross-scale connected image features.

[0066] Furthermore, in another embodiment, the UET network further includes a hyperbolic tangent unit after the upsampling module. The hyperbolic tangent unit is used to perform interval correction on cross-scale connectivity features, correcting the output of the merging unit to the interval [-1, 1].

[0067] In another embodiment, the preset deep learning neural network includes two cascaded UET networks. The output of the first UET network and the upsampled image features of the last scale are concatenated and used as the input of the second UET network.

[0068] Cascading two UET networks as a deep learning neural network framework can better learn the complex mapping relationships between sample image groups during training, enabling the trained deep learning neural network to achieve better learning results.

[0069] Step S630: Determine the loss function value based on the predicted pixel offset matrix and the preset loss function.

[0070] A loss function, or cost function, maps the values ​​of a random event or its related random variables to non-negative real numbers to represent the "risk" or "loss" of that random event. In machine learning, loss functions are commonly used for model parameter estimation, determining the model's performance, and then adjusting the model during training based on the loss function. In this embodiment, the loss functions used in the training of the deep learning neural network include: single-pixel prediction loss, adjacent pixel offset difference loss, and complete transformation loss.

[0071] Input the predicted pixel offset matrix corresponding to the distorted sample image output by the preset deep learning neural network framework into the preset loss function to obtain the corresponding loss function value.

[0072] Step S640: Train the preset deep learning neural network framework according to the loss function value until a deep learning neural network is obtained.

[0073] After obtaining the predicted pixel offset matrix, the loss function value can be determined based on the difference between the predicted pixel offset matrix and the true pixel offset matrix. This loss function value is then used to train a pre-defined deep learning neural network until the deep learning neural network is obtained. The training termination condition for the pre-defined deep learning neural network can be set according to the actual situation.

[0074] Furthermore, in one embodiment, the single-pixel prediction loss includes: the distance between the predicted pixel offset matrix and the corresponding pixel offset values ​​in the true pixel offset matrix; the true pixel offset matrix is ​​determined by the distorted sample image and the corresponding normal sample image.

[0075] In one embodiment, the distorted sample image used to train the preset deep learning neural network is obtained by distorting a normal sample image. Therefore, the true pixel offset matrix of the distorted sample image can be determined based on the distorted sample image and the normal sample image.

[0076] Further, the distance between corresponding pixel offset values ​​in the predicted pixel offset matrix and the true pixel offset matrix includes: the distance between pixel offset values ​​at the same position in the predicted pixel offset matrix and the true pixel offset matrix. Even further, in one embodiment, the distance between two pixel offset values ​​can be calculated in any way; for example, calculating the L1 distance between two pixel offset values.

[0077] In one embodiment, the adjacent pixel offset difference loss includes: the distance between two adjacent pixel offset values ​​in the predicted pixel offset matrix.

[0078] In real-world scenarios, pixels that are close to each other are likely to have similar offset directions, meaning that there is a possibility of local clustering of offsets. Therefore, in this embodiment, the distance between the offset values ​​of any two adjacent pixels in the predicted pixel matrix is ​​calculated as the adjacent pixel offset difference loss term.

[0079] In one embodiment, the complete change loss includes: shifting the predicted pixel offset matrix in a preset direction to obtain a shifted offset matrix; calculating the residual matrix by comparing the corresponding elements in the predicted pixel offset matrix and the shifted offset matrix; and calculating the sum of squares of all elements in the residual matrix as the complete change loss.

[0080] To further enhance the local clustering characteristics of the offset, this embodiment also introduces a complete change loss. The difference between the predicted pixel matrix after the position shift and the predicted pixel matrix is ​​calculated and denoted as the residual matrix in this embodiment. Then, the sum of squares of each element in the residual matrix is ​​calculated as the complete change loss.

[0081] In this embodiment, a loss function including single-pixel prediction loss, adjacent pixel offset difference loss, and complete change loss is used to train the preset deep learning neural network framework, which can improve the model learning effect.

[0082] In one embodiment, constructing a distorted sample image includes: acquiring a normal target object image; randomly selecting a target distortion function from a set of distortion functions; mapping the normal target object image based on the target distortion function to obtain a corresponding distorted target object image; and combining the distorted target object image with a random background to obtain a distorted sample image.

[0083] The target object refers to the object used for offsetting; it can be any type of object, such as a document, table, etc. A normal target object image represents an image of the target object without distortion or tilt.

[0084] A set of distortion functions is used to distort a normal target object image to obtain a distorted target object image corresponding to the normal target image. In one embodiment, the set of distortion functions includes a sequence of distortion functions and a sequence of folding functions; randomly selecting a target distortion function from the set of distortion functions includes: randomly selecting a function from the sequence of distortion functions and the sequence of folding functions in sequence as the target distortion function.

[0085] The distortion function and the folding function are two forces that exert an effect on pixels in an image. In one embodiment, these forces cause the position of the affected pixels to change in the new image. The distortion function and the folding function can be set according to the actual situation.

[0086] After distorting the normal target object image using the selected target distortion function, a distorted target object image corresponding to the normal target object image can be obtained. Since a large portion of distorted images in real-world scenes are obtained by photographing the target object, and the photographing scene also contains other objects, i.e., background image information, in this embodiment, after constructing the distorted target object image, the distorted target object image is combined with the background image to obtain a distorted sample image.

[0087] Furthermore, the distortion sample image can be obtained by combining the distorted target object image with a random background in the following way: randomly extract a background image from the background image database, and paste the distorted target object image onto the selected background image to obtain the distorted sample image; understandably, in this embodiment, the size of the background image is larger than the size of the distorted sample image.

[0088] In this embodiment, a distortion image of the target object is obtained by selecting a distortion function and a folding function to process the normal target object image. Then, the distorted image is randomly combined with the background to obtain an image as a distorted sample image for training a deep learning neural network, which can increase the generalization ability of the data.

[0089] In one embodiment, mapping a normal target object image based on a target distortion function to obtain a corresponding distorted target object image includes: selecting a transformation center point in the normal target object image; randomly generating an offset source vector passing through the transformation center point; calculating the pixel distance between each pixel in the normal target object image and the offset source vector; determining the target function value of each pixel distance based on the target distortion function; and offsetting each pixel in the normal target object image using the corresponding pixel distance target function value to obtain the distorted target object image.

[0090] Figure 7 illustrates a distorted sample image constructed using the method described above in a specific embodiment. In another embodiment, a generative adversarial network can also be used to construct and generate a corresponding distorted sample image based on a normal sample image.

[0091] In this embodiment, by selecting a distortion function and a folding function, and using the selected distortion function and folding function to distort the image of a normal target object, a distorted image of the target object can be obtained. Then, by randomly combining the distorted image with the background, a distorted sample image can be constructed, which is used to train a preset deep learning neural network framework. This simplifies the process of constructing training sample data and reduces manpower and improves processing efficiency compared to manually constructing distorted sample images in the traditional method.

[0092] This application also provides an application scenario in which the above-described document correction method is applied. Specifically, in this application scenario, the document correction method is applied to the correction of underwriting document images. In other embodiments, the above-described document correction method can also be applied to the correction of other images. Underwriting refers to insurance underwriting, which is the process by which an insurer, based on a comprehensive understanding and verification of the information of the insured object, assesses and classifies insurable risks, and then decides whether to underwrite and under what conditions. During the underwriting process, underwriters will provide different underwriting conditions according to different risk categories of the insured object to ensure business quality and the stability of insurance operations. Underwriting documents refer to the document data generated during the underwriting process; in one embodiment, underwriting documents include: insurance-related documents issued by the insurance company to the insured, medical examination forms and receipts provided by the hospital when the insured undergoes a medical examination, etc. When the above-described document correction method is applied to the scenario of underwriting documents, a good correction rate can be obtained on at least three types of heterogeneous underwriting documents.

[0093] Based on the UET network as the basic network structure, a preset deep learning neural network framework is constructed; specifically, two UET networks are cascaded together as the preset deep learning neural network framework.

[0094] The first UET network consists of multiple downsampling modules, multiple upsampling modules, and a hyperbolic tangent unit. Each downsampling module performs layer-by-layer downsampling on the input image to obtain upsampled image features at different scales. Each upsampling module performs layer-by-layer upsampling on the downsampled image features at the last scale to obtain upsampled image features at different scales, and merges the downsampled and upsampled image features at the same scale to obtain merged image features. Finally, the hyperbolic tangent unit performs interval adjustment on the merged image features, correcting them to the [-1, 1] interval.

[0095] For the second UET network: the output of the hyperbolic tangent unit of the first UET network and the upsampled image features at the last scale are concatenated to obtain the input of the second UET network. The second UET network performs layer-by-layer downsampling of the input using multiple downsampling modules, and then performs layer-by-layer upsampling using multiple upsampling modules. The downsampled and upsampled image features at the same scale are merged to obtain the merged image features. The hyperbolic tangent unit of the second UET network is then used to correct the merged image features to the [-1,1] interval. In one embodiment, the first and second UET networks have the same structure.

[0096] After constructing the pre-defined deep learning neural network framework, it needs to be trained using sample images. Unlike other common deep learning tasks, the current task is difficult to generate training data using common manual image annotation methods: annotators need to manually create distortions in real images, take photos to obtain the distorted images, and simultaneously obtain the offset matrix of the distortion transformation. This method requires a large amount of manpower, and the mapping relationship between normal and distorted images is a complex non-linear transformation, which is difficult to obtain accurately. In view of this, based on relevant knowledge of computer graphics, the following process is designed to construct distorted samples:

[0097] Design a set of distortion functions G = {{d}, {c}}, where {d} is a sequence of distortion functions and {c} is a sequence of folding functions. For each input normal target object image, randomly select several functions from {d} and several functions from {c}. Apply the selected distortion and / or folding functions sequentially to each pixel in the normal target object image to map each pixel onto a blank image to construct a distorted image. The implementation of these two function sequences is as follows: First, randomly select a point p = (p_x, p_y) on the normal target object image as the transformation center, and simultaneously randomly generate an offset source vector v = {v_x, v_y} passing through this point. With the transformation center p as the core, calculate the distance d from each pixel (x, y) in the entire normal target object image to the offset vector v, and then offset it to (f(d) + x, f(d) + y). The influence function f(d) can be either a distortion function or a folding function. For a sequence of distortion functions, f(d) is a direct proportional function of the variable d; for a sequence of folding functions, g(d) is an inverse proportional function of the variable d. The specific form of the influence function can be adjusted based on the specific scenario. While generating the distorted target object image, the pixel offset matrix of the normal target object image is simultaneously recorded. Each element in the offset matrix is ​​the offset (Δx, Δy) from a pixel in the normal target object image to a pixel in the distorted target object image. Finally, the generated distorted target object image is pasted onto a random background to obtain distorted sample images, which enhance the generalization ability of the data.

[0098] The pre-defined deep learning neural network framework is trained using the constructed distorted sample image: The distorted sample image is input into the pre-defined deep learning neural network framework, the predicted pixel offset matrix of the distorted sample image is output, the predicted pixel offset matrix is ​​compared with the real pixel offset matrix to determine the loss function value, and then the pre-defined deep learning neural network framework is trained to obtain the deep learning neural network.

[0099] The loss functions used in training include 1) single-pixel prediction loss, 2) adjacent pixel offset difference loss, and 3) complete transformation loss. The single-pixel prediction loss directly calculates the L1 distance between each pixel offset d_r in the prediction matrix and the true offset d_g. The adjacent pixel offset difference loss calculates the L1 distance between pairwise pixel offsets d_r1 and d_r2, taking into account the local clustering characteristics of offsets in real-world scenes (pixels in close proximity have a higher probability of similar offset directions). The complete transformation loss function calculates the residual matrix between adjacent rows / columns of the input offset matrix (the predicted pixel offset matrix minus the predicted pixel offset matrix shifted one column / row to the right / down), and sums the squared distances of each element in the residual matrix as the output. The sum of these three losses is used as the final loss function for training the deep learning neural network in this embodiment.

[0100] After training, a well-trained deep learning neural network is obtained. The acquired image to be corrected is input into the well-trained neural network model, and the image features are extracted through two cascaded UET networks, outputting the pixel offset matrix of the image to be corrected.

[0101] The corrected image can be obtained by mapping the pixel offset matrix M to the image to be corrected pixel by pixel using map(D,M). However, due to the complexity of the learning task, pixel-by-pixel regression often contains noise, which often reveals discontinuities in local areas in the restored image, resulting in a noticeable artificial feel in subjective evaluation. To solve these problems, the output pixel offset matrix is ​​smoothed before mapping the image again using map(D,smooth(M)). Specifically, Gaussian smoothing is applied first, followed by multiple average smoothing operations to obtain a smoothed offset matrix. Then, the smoothed offset matrix is ​​used to map the image to be corrected pixel by pixel to obtain the corrected image. The horizontal window size for Gaussian smoothing is h / 2, and the vertical window size is w / 3, where h and w are the height and width of the image. The horizontal window size for average smoothing is h / 2, and the vertical window size is w / 3. Gaussian smoothing effectively smooths local noise, and repeating the average smoothing operation several times achieves good restoration results. In the final output image, the text is nearly horizontal, the output image has no local distortion, and the subjective evaluation by the human eye is not artificial. Figure 8 The image shown is a schematic diagram of the image to be corrected, the corrected image, and the image results obtained from each smoothing process in a specific embodiment.

[0102] In the above method, a deep learning neural network is used to output a pixel offset matrix of the image to be corrected. After smoothing the pixel offset matrix, the distortion of the image to be corrected is restored based on the smoothed pixel offset matrix, which can obtain a high-quality corrected image. The corrected image has no local distortion, and the text in it is close to horizontal, which improves the quality of the corrected image.

[0103] This application also provides an application scenario in which the above-described document correction method is applied. Specifically, after applying the above-described document correction method to correct the image to be corrected, a corrected image is obtained, which can better ensure image quality. Furthermore, performing optical character recognition on the obtained corrected image can improve the effect of optical character recognition and subsequent information extraction, providing strong technical support for optical character recognition and information extraction.

[0104] Furthermore, in another embodiment, the above-described document correction method is integrated into an optical character recognition network, thereby enabling the deep learning network to achieve the entire end-to-end process and reducing the overhead of intermediate result conversion processing.

[0105] It should be understood that although the steps in the flowcharts of any of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of any of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0106] In one embodiment, such as Figure 9 As shown, a document correction device is provided. This device can be a software module, a hardware module, or a combination of both integrated into a computer device. Specifically, the device includes: an acquisition module 910, a pixel offset matrix determination module 920, and a mapping module 930, wherein:

[0107] The acquisition module 910 is used to acquire the image to be corrected;

[0108] The pixel offset matrix determination module 920 is used to extract image features of the image to be corrected based on the trained deep learning neural network, and determine the pixel offset matrix of the image to be corrected based on the image features of the image to be corrected; the deep learning neural network includes a cascaded first UET network and a second UET network; after normalizing the output of the first UET network, the normalized features are concatenated with the output of the first UET network, and the concatenated features are used as the input of the second UET network;

[0109] The mapping module 930 is used to map the image to be corrected based on the pixel offset matrix to obtain the corrected image of the image to be corrected.

[0110] The aforementioned document correction device extracts image features from the acquired image to be corrected based on a trained deep learning neural network. It then determines the pixel offset matrix of the image to be corrected based on these features and maps the image to obtain the corrected image. The deep learning neural network consists of two cascaded UET networks. The output of the first UET network is normalized, and the normalized result is concatenated with the output of the first UET network. The concatenated feature is used as the input to the second UET network. The output of the second UET network is the output of the deep learning neural network, i.e., the pixel offset matrix. This device uses a deep learning neural network composed of two cascaded UET networks to extract image features from the image to be corrected and determines the corresponding pixel offset matrix based on these features. The trained deep learning neural network can effectively learn the offset between the distorted image to be corrected and the corresponding reference image, thereby generating the pixel offset matrix of the image to be corrected. Finally, pixel mapping is performed on the image to be corrected based on the pixel offset matrix to obtain the corrected image. This method does not depend on environmental parameters, and the pixel offset matrix can be obtained in just one process. The process is simple and the computation is small.

[0111] In one embodiment, the above-mentioned apparatus further includes: a smoothing processing module, used to smooth the pixel offset matrix to obtain a smoothed offset matrix; in this embodiment, the mapping module 930 of the above-mentioned apparatus is specifically used to: map the image to be corrected based on the smoothed offset matrix to obtain a corrected image.

[0112] In one embodiment, the smoothing module of the above-mentioned device includes: a Gaussian smoothing unit, configured to: perform Gaussian smoothing on the pixel offset matrix with a window of a first preset size to obtain a Gaussian smoothed offset matrix; and an average smoothing unit, configured to perform average smoothing on the Gaussian smoothed offset matrix with a window of a second preset size to obtain a smoothed offset matrix.

[0113] In one embodiment, the above-described apparatus further includes a training module, configured to: input a distorted sample image into a preset deep learning neural network framework; obtain a predicted pixel offset matrix of the distorted sample image output by the preset deep learning neural network framework; determine a loss function value based on the predicted pixel offset matrix and a preset loss function; and train the preset deep learning neural network framework according to the loss function value until a deep learning neural network is obtained.

[0114] In one embodiment, the preset loss function in the above-described device includes: single-pixel prediction loss, adjacent pixel offset difference loss, and complete change loss.

[0115] In one embodiment, the single-pixel prediction loss includes: the distance between the predicted pixel offset matrix and the corresponding pixel offset values ​​in the true pixel offset matrix; the true pixel offset matrix is ​​determined by the distorted sample image and the corresponding normal sample image.

[0116] In one embodiment, the adjacent pixel offset difference loss includes: the distance between two adjacent pixel offset values ​​in the predicted pixel offset matrix.

[0117] In one embodiment, the complete change loss includes: shifting the predicted pixel offset matrix in a preset direction to obtain a shifted offset matrix; calculating the residual matrix by comparing the corresponding elements in the predicted pixel offset matrix and the shifted offset matrix; and calculating the sum of squares of all elements in the residual matrix as the complete change loss.

[0118] In one embodiment, the above apparatus further includes: a distorted sample image construction module, comprising: a normal image acquisition unit for acquiring a normal target object image; a function selection unit for randomly selecting a target distortion function from a set of distortion functions; a mapping unit for mapping the normal target object image based on the target distortion function to obtain a corresponding distorted target object image; and a background combination unit for combining the distorted target object image with a random background to obtain a distorted sample image.

[0119] In one embodiment, the mapping unit of the above-described apparatus includes: a center point selection subunit, configured to select a transformation center point in a normal target object image; a vector generation subunit, configured to randomly generate an offset source vector passing through the transformation center point; a distance calculation subunit, configured to calculate the pixel distance between each pixel in the normal target object image and the offset source vector; a function value determination subunit, configured to determine the target function value of each pixel distance based on the target distortion function; and an offset processing subunit, configured to offset each pixel in the normal target object image using the target function value corresponding to the pixel distance, to obtain a distorted target object image.

[0120] For specific limitations regarding the document correction device, please refer to the limitations of the document correction method above, which will not be repeated here. Each module in the aforementioned document correction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the computer device in hardware form or independently of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0121] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a document correction method.

[0122] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0123] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0124] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0125] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0126] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0127] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A document correction method, characterized in that, The method includes: Obtain the image to be corrected; The deep learning neural network, based on training, extracts image features from the image to be corrected and determines the pixel offset matrix of the image to be corrected based on these features. The deep learning neural network includes a cascaded first and second UET networks. The output of the first UET network is normalized, and the normalized features are concatenated with the output of the first UET network. The concatenated features are then used as the input of the second UET network. The outputs of both the first and second UET networks are subjected to interval correction using hyperbolic tangent units. The pixel offset matrix describes the offset between the image to be corrected and the corresponding reference image. The pixel offset matrix is ​​Gaussian smoothed using the horizontal and vertical window sizes within a first preset window size to obtain a Gaussian smoothed offset matrix, thereby eliminating Gaussian noise. The horizontal window size is determined based on the height of the image to be corrected, and the vertical window size is determined based on the width of the image to be corrected. The Gaussian smoothed offset matrix is ​​then averaged multiple times using the horizontal and vertical window sizes within a second preset window size to obtain a smoothed pixel offset matrix. The image to be corrected is mapped based on the smoothed pixel offset matrix to obtain the corrected image of the image to be corrected.

2. The document correction method according to claim 1, characterized in that, The training process of the deep learning neural network includes: Input the distorted sample image into a preset deep learning neural network framework; Obtain the predicted pixel offset matrix of the distorted sample image output by the preset deep learning neural network framework; The loss function value is determined based on the predicted pixel offset matrix and the preset loss function. The preset deep learning neural network framework is trained based on the loss function value until the deep learning neural network is obtained.

3. The document correction method according to claim 2, characterized in that, The preset loss function includes: single-pixel prediction loss, adjacent pixel offset difference loss, and complete change loss.

4. The document correction method according to claim 3, characterized in that, Includes at least one of the following: First item, The single-pixel prediction loss includes: the distance between the predicted pixel offset matrix and the corresponding pixel offset values ​​in the true pixel offset matrix; the true pixel offset matrix is ​​determined by the distorted sample image and the corresponding normal sample image. The second item, The adjacent pixel offset difference loss includes: the distance between the offset values ​​of two adjacent pixels in the predicted pixel offset matrix; The third item, The complete change loss includes: shifting the predicted pixel offset matrix in a preset direction to obtain a shifted offset matrix; calculating a residual matrix by comparing the corresponding elements in the predicted pixel offset matrix and the shifted offset matrix; and calculating the sum of squares of all elements in the residual matrix as the complete change loss.

5. The document correction method according to claim 2, characterized in that, The construction of the distorted sample image includes: Acquire images of normal target objects; Randomly select a target warp function from the set of warp functions; The normal target object image is mapped based on the target distortion function to obtain the corresponding distorted target object image; The distorted target object image is combined with a random background to obtain the distorted sample image.

6. The document correction method according to claim 5, characterized in that, The step of mapping the normal target object image based on the target distortion function to obtain the corresponding distorted target object image includes: Select a transformation center point in the normal target object image; Randomly generate an offset source vector passing through the transformation center point; Calculate the pixel distance between each pixel in the normal target object image and the offset source vector; The target function value for each pixel distance is determined based on the target distortion function. For each pixel in the normal target object image, offset by the target function value of the corresponding pixel distance to obtain the distorted target object image.

7. A document correction device, characterized in that, The device includes: The acquisition module is used to acquire the image to be corrected. A pixel offset matrix determination module is used to extract image features of the image to be corrected based on a trained deep learning neural network, and to determine the pixel offset matrix of the image to be corrected based on the image features. The deep learning neural network includes a cascaded first UET network and a second UET network. After normalizing the output of the first UET network, the normalized features are concatenated with the output of the first UET network, and the concatenated features are used as the input of the second UET network. The outputs of both the first and second UET networks are interval-corrected using hyperbolic tangent units. The pixel offset matrix is ​​used to describe the offset between the image to be corrected and the corresponding reference image. The smoothing module includes a Gaussian smoothing unit and an average smoothing unit; The Gaussian smoothing unit is used to perform Gaussian smoothing on the pixel offset matrix with a horizontal window size and a vertical window size within a window of a first preset size to obtain a Gaussian smoothed offset matrix to eliminate Gaussian noise. The horizontal window size is determined based on the height of the image to be corrected, and the vertical window size is determined based on the width of the image to be corrected. The average smoothing unit is used to perform multiple average smoothings on the Gaussian smoothing offset matrix using the horizontal window size and the vertical window size within a window of the second preset size, to obtain a smoothed pixel offset matrix. The mapping module is used to map the image to be corrected based on the smoothed pixel offset matrix to obtain the corrected image of the image to be corrected.

8. The document correction device according to claim 7, characterized in that, The device also includes a training module; The training module is used to input a distorted sample image into a preset deep learning neural network framework; obtain the predicted pixel offset matrix of the distorted sample image output by the preset deep learning neural network framework; determine the loss function value based on the predicted pixel offset matrix and a preset loss function; and train the preset deep learning neural network framework according to the loss function value until the deep learning neural network is obtained.

9. The document correction device according to claim 8, characterized in that, The preset loss function includes: single-pixel prediction loss, adjacent pixel offset difference loss, and complete change loss.

10. The document correction device according to claim 9, characterized in that, The single-pixel prediction loss includes: the distance between the predicted pixel offset matrix and the corresponding pixel offset values ​​in the true pixel offset matrix; the true pixel offset matrix is ​​determined by the distorted sample image and the corresponding normal sample image.

11. The document correction device according to claim 9, characterized in that, The adjacent pixel offset difference loss includes the distance between the offset values ​​of two adjacent pixels in the predicted pixel offset matrix.

12. The document correction device according to claim 9, characterized in that, The complete change loss includes: shifting the predicted pixel offset matrix in a preset direction to obtain a shifted offset matrix; calculating a residual matrix by comparing the corresponding elements in the predicted pixel offset matrix and the shifted offset matrix; and calculating the sum of squares of all elements in the residual matrix as the complete change loss.

13. The document correction device according to claim 8, characterized in that, The device also includes a distorted sample image construction module; The distorted sample image construction module includes: The normal image acquisition unit is used to acquire images of normal target objects; The function selection unit is used to randomly select a target twisting function from the set of twisting functions; The mapping unit is used to map the normal target object image based on the target distortion function to obtain the corresponding distorted target object image; The background combination unit is used to combine the distorted target object image with a random background to obtain the distorted sample image.

14. The document correction device according to claim 13, characterized in that, The mapping unit includes: The center point selection subunit is used to select the transformation center point in the normal target object image; A vector generation subunit is used to randomly generate an offset source vector passing through the transformation center point; The distance calculation subunit is used to calculate the pixel distance between each pixel in the normal target object image and the offset source vector; The function value determination subunit is used to determine the target function value of each pixel distance based on the target distortion function; The offset processing subunit is used to offset each pixel in the normal target object image by the target function value of the corresponding pixel distance to obtain the distorted target object image.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

16. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images