Document image processing method and apparatus, storage medium, and electronic device

Abstract

This application discloses a document image processing method, apparatus, storage medium, and electronic device. The method includes: acquiring an initial document image; performing image block processing using a document image enhancement model to obtain initial image block data; performing image enhancement processing on each initial image block data to obtain reference enhanced image block data; determining first reference enhanced image block data of the quality check failure type and second reference enhanced image block data of the quality check success type; performing local supplementary enhancement processing on the first reference enhanced image block data to obtain third reference enhanced image block data; performing sliding window fusion processing on the second and third reference enhanced image block data to obtain a target enhanced document image; and performing character recognition processing to obtain a target character sequence corresponding to the initial document image and a target character confidence level corresponding to the target character sequence. Thus, this application improves the character recognition accuracy of the initial document image.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a document image processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] With the widespread adoption of digital office work and mobile internet, acquiring document images via scanners or mobile phone cameras has become an integral part of daily work and life. However, in actual acquisition processes, due to factors such as equipment limitations, uneven ambient lighting, camera shake, inaccurate focusing, and lossy compression, the quality of acquired document images is often poor. These document images may suffer from various degradation problems, including blurriness, noise, geometric distortion (such as perspective tilt), low resolution, and compression artifacts (such as JPEG banding). These image quality issues not only affect the human reading experience but also significantly reduce the accuracy of optical character recognition systems.

[0003] Therefore, for those skilled in the art, how to improve the accuracy of character recognition in document images is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] This application provides a document image processing method, apparatus, computer storage medium, and electronic device. The technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide a document image processing method, the method comprising:

[0006] An initial document image is obtained, and the initial document image is processed by a document image enhancement model to obtain data for each initial image block.

[0007] The document image enhancement model is used to perform image enhancement processing on each of the initial image block data to obtain each reference enhanced image block data. From each of the reference enhanced image block data, a first reference enhanced image block data of the quality check failure type and a second reference enhanced image block data of the quality check success type are determined. The first reference enhanced image block data is subjected to local supplement enhancement processing to obtain a third reference enhanced image block data. Based on the second reference enhanced image block data and the third reference enhanced image block data, a sliding window fusion processing is performed to obtain the target enhanced document image.

[0008] The target enhanced document image is processed by the document image enhancement model to obtain the target character sequence corresponding to the initial document image and the target character confidence level corresponding to the target character sequence.

[0009] In some possible implementations, the document image enhancement model includes a residual sampling module, and the step of performing image enhancement processing on each of the initial image patch data through the document image enhancement model to obtain each reference enhanced image patch data includes:

[0010] Determine the current sampling step number corresponding to the residual sampling module, and determine the sampling weight parameter corresponding to the current sampling step;

[0011] Based on the sampling weight parameters, the residual sampling module is used to perform image enhancement processing on each of the initial image block data to obtain each basic enhanced image block data.

[0012] Determine whether the current sampling step number is greater than or equal to the preset sampling step number;

[0013] If the current sampling step is less than the preset sampling step, then the next sampling step corresponding to the current sampling step is obtained, the next sampling step is used as the current sampling step, the basic enhanced image block data is used as the initial image block data, and the steps of determining the sampling weight parameter corresponding to the current sampling step, and using the residual sampling module to perform image enhancement processing on each of the initial image block data based on the sampling weight parameter to obtain each basic enhanced image block data are executed.

[0014] If the current sampling step number is greater than or equal to the preset sampling step number, then the basic enhanced image block data corresponding to the current sampling step number are used as the reference enhanced image block data.

[0015] In some possible implementations, the document image enhancement model further includes a scoring network module and a character recognition module, and the method further includes:

[0016] During the model training process of the residual sampling module, the first sample augmented image patch data generated by the initial residual sampling module in the k-th sampling step is obtained, and the second sample augmented image patch data generated by the initial residual sampling module in the k+1-th sampling step is determined based on the preset residual update formula.

[0017] The preset residual update formula satisfies the following formula:

[0018] in, For the second sample, enhance the image patch data. Enhance the image patch data for the first sample. Step size, The drift coefficient, The strength of the scoring network module, For the scoring network module, The semantic guidance strength weights are for the k-th sampling step. The frequency channel weights for the k-th sampling step. The loss gradient of the character recognition module is... Inject control weights into the frequency domain regularization of the k-th sampling step. The frequency domain loss gradient;

[0019] The third sample enhanced image block data generated by the undistilled residual sampling module in the sampling process of step k+1 is obtained. The distillation loss corresponding to the third sample enhanced image block data and the second sample enhanced image block data is determined by the first loss calculation formula. Based on the distillation loss, the semantic guidance strength weight, the frequency channel weight, and the frequency domain regularization injection control weight, the model parameters of the initial residual sampling module are adjusted to obtain the residual sampling module.

[0020] The first loss calculation formula satisfies the following formula:

[0021] in, For distillation losses, For image residual alignment loss, for The weight, For intermediate statistical matching loss, for The weight, To display the gradient preservation loss of the character recognition module, for The weights;

[0022] The second loss calculation formula satisfies the following equation:

[0023]

[0024] in, This refers to the first enhanced image patch inference data output by the initial residual sampling module from the initial image patch data of the sample. This refers to the second enhanced image patch inference data output by the undistilled residual sampling module from the initial image patch data of the sample. It is an L1 norm;

[0025] The third loss calculation formula, obtained through the third loss calculation formula, satisfies the following formula:

[0026]

[0027] in, For Fourier transform operators, Indicates the range of the spectrum. For image gradient operators, Constructing operators for histograms The KL divergence is used to measure the difference in gradient distributions. As weight, As weight, It is an L1 norm;

[0028] The fourth loss calculation formula satisfies the following equation:

[0029]

[0030] in, express and The maximum mean difference This refers to the character recognition module.

[0031] In some possible implementations, the method further includes:

[0032] During the training of the scoring network module, sample noisy document images and noise data distribution gradient labels corresponding to the sample noisy document images are obtained. An initial scoring network module is used to identify the predicted noise data distribution gradient corresponding to the sample noisy document images. The fifth loss calculation formula is used to determine the score matching loss corresponding to the predicted noise data distribution gradient and the noise data distribution gradient labels. Based on the score matching loss, the model parameters of the initial scoring network module are adjusted to obtain the scoring network module.

[0033] The fifth loss calculation formula satisfies the following formula:

[0034]

[0035] in, The score matching loss, As weight, The result is obtained through the sixth loss calculation formula. It is obtained through the seventh loss calculation formula;

[0036] The sixth loss calculation formula satisfies the following formula:

[0037]

[0038] in, The sample noisy document image, Through Noise samples were added to obtain the sample. The noise-free document image corresponding to the sample noisy document image. The noise data distribution gradient label, Indicates time Observed Under the premise of noise-free image The probability distribution, For the noise sample, This is the initial score network module. The input is and , The output is the gradient of the predicted noise data distribution. Indicates time step The relevant weighting function, The square operation represents the L2 norm. Represents all possible Seeking expectations;

[0039] The seventh loss calculation formula satisfies the following formula:

[0040] in, For Fourier transform operators, Indicates a high-frequency mask. This represents element-wise multiplication. Describing the L1 norm, As weight, It is a set of JPEG ring band indexes. Extract the k-th frequency component from the image after Fourier transform.

[0041] In some possible implementations, the method further includes:

[0042] During the training process of the character recognition module, sample augmented document images and character sequence labels corresponding to the sample augmented document images are acquired. Based on the initial character recognition module, the sample augmented document images are processed to obtain predicted character data. The character recognition loss corresponding to the predicted character data and the character sequence labels is determined by the recognition loss calculation formula. Based on the character recognition loss, the model parameters of the initial character recognition module are adjusted to obtain the character recognition module.

[0043] The formula for calculating the recognition loss satisfies the following formula:

[0044]

[0045] in, The character recognition loss, For continuous time-series classification loss, For attention deficit, For differentiable editable distance loss, It is obtained by recursion using the recursive formula for differentiable editable distance. , , They are respectively , , The weights;

[0046] The identification loss formula is obtained by recognizing the following formula:

[0047]

[0048] in, Enhance the document image of the sample. It is a candidate alignment path that can be collapsed into an "I". For path The symbol in frame t, P () indicates that the symbol for frame t is The probability, T For path Length, This indicates that the product is obtained by independent multiplication along the time axis. Path probability, It is all that can be folded into The set of paths The character sequence label.

[0049] In some possible implementations, the document image enhancement model further includes a character stroke prior module and an image degradation module, and the method further includes:

[0050] During the training process of the document image enhancement model, the comprehensive model loss of the initial document image enhancement model is determined by a comprehensive loss calculation formula, and the model parameters of the initial document image enhancement model are adjusted based on the comprehensive model loss to obtain the document image enhancement model.

[0051] The formula for calculating the overall loss satisfies the following equation:

[0052]

[0053] in, The loss of the comprehensive model is... The score matching loss of the scoring network module is... The character recognition loss of the character recognition module. The character shape loss of the character stroke prior module. The stroke loss of the aforementioned character stroke prior module. The character component loss of the aforementioned character stroke prior module. The image degradation loss of the image degradation module is... For frequency domain loss, To perceive loss.

[0054] In some possible implementations, the document image enhancement model includes an image degradation module, wherein determining first reference enhanced image block data of the quality check failure type and second reference enhanced image block data of the quality check success type from each of the reference enhanced image block data includes:

[0055] Character recognition processing is performed on each of the reference enhanced image block data to obtain a predicted character sequence for each of the reference enhanced image block data and a predicted character confidence level corresponding to the predicted character sequence;

[0056] The image degradation module is used to perform image degradation processing on each of the reference enhanced image block data to obtain reference degraded image block data, and the reference back projection error between the reference degraded image block data and the initial image block data is determined.

[0057] Determine the character error evaluation value corresponding to the predicted character sequence;

[0058] Based on the predicted character confidence, the reference back projection error, and the character error evaluation value, the quality assessment process of each reference enhanced image block data is performed to obtain the quality check result of each reference enhanced image block data.

[0059] Based on the quality inspection results, a first reference enhanced image block data of the quality inspection failure type and a second reference enhanced image block data of the quality inspection success type are determined.

[0060] In some possible implementations, the method further includes:

[0061] During the training process of the image degradation module, a sample ideal document image and the real document image corresponding to the sample ideal document image are obtained. The image degradation calculation formula of the initial image degradation module is used to perform image degradation processing on the sample ideal document image to obtain the observed degraded document image.

[0062] The image degradation calculation formula satisfies the following formula:

[0063]

[0064] in, For the observed degraded document image, For all degradation parameters in the initial image degradation model, The ideal document image for the sample. Is the factor downsampling, For spatial fuzzy kernel, For convolution operations, For composite noise parameters, This is a low-frequency light generation module. For superimposed paper texture and / or transparent texture; Used to simulate the distortion that occurs in image signals during digital imaging and storage;

[0065] The image degradation loss corresponding to the real document image and the observed degraded document image is determined based on the eighth loss formula. The initial image degradation model is then adjusted based on the degradation loss to obtain the image degradation model. The eighth loss formula satisfies the following formula:

[0066]

[0067] in, The image degradation loss, The actual document image, As weight, Used for evaluation and Image similarity between It is an L1 norm.

[0068] Secondly, embodiments of this application provide a document image processing apparatus, the apparatus comprising:

[0069] The image segmentation module is used to acquire an initial document image and perform image segmentation processing on the initial document image through a document image enhancement model to obtain data for each initial image block.

[0070] The image enhancement module is used to perform image enhancement processing on each of the initial image block data through the document image enhancement model to obtain each reference enhanced image block data, determine a first reference enhanced image block data of the quality check failure type and a second reference enhanced image block data of the quality check success type from each of the reference enhanced image block data, perform local supplement enhancement processing on the first reference enhanced image block data to obtain a third reference enhanced image block data, and perform sliding window fusion processing on the second reference enhanced image block data and the third reference enhanced image block data to obtain a target enhanced document image;

[0071] The character recognition module is used to perform character recognition processing on the target enhanced document image through the document image enhancement model to obtain the target character sequence corresponding to the initial document image and the target character confidence level corresponding to the target character sequence.

[0072] Thirdly, embodiments of this application provide a computer storage medium having multiple instructions adapted for loading and executing the methods described above by a processor.

[0073] Fourthly, embodiments of this application provide an electronic device, which may include: a memory and a processor; wherein the memory stores a computer program adapted to be loaded by the memory and to execute the above-described method.

[0074] The beneficial effects of the technical solutions provided in this application include at least the following:

[0075] The document image processing method provided in this application embodiment acquires an initial document image, then performs image block processing on the initial document image using a document image enhancement model to obtain initial image block data, and performs image enhancement processing on each initial image block data using the same model to obtain reference enhanced image block data. From the reference enhanced image block data, a first reference enhanced image block data of the quality check failure type and a second reference enhanced image block data of the quality check success type are determined. The first reference enhanced image block data undergoes local supplementation enhancement processing to obtain a third reference enhanced image block data. Based on the second and third reference enhanced image block data, a sliding window fusion processing is performed to obtain a target enhanced document image. In this way, the quality of the initial document image is repaired or enhanced, thereby obtaining a target enhanced document image with high image quality. Then, the target enhanced document image is processed by the document image enhancement model to obtain the target character sequence corresponding to the initial document image and the target character confidence level corresponding to the target character sequence. This achieves accurate target character sequence recognition based on a high-quality target enhanced document image, improving the character recognition accuracy of the initial document image. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0077] Figure 1 This is a schematic flowchart of a document image processing method provided in an embodiment of this application;

[0078] Figure 2 This is a schematic flowchart of an image enhancement processing method based on a residual sampling module provided in an embodiment of this application;

[0079] Figure 3This is a flowchart illustrating the first model training method provided in the embodiments of this application;

[0080] Figure 4 This is a flowchart illustrating the second model training method provided in the embodiments of this application;

[0081] Figure 5 This is a flowchart illustrating the third model training method provided in the embodiments of this application;

[0082] Figure 6 This is a flowchart illustrating a document enhancement image quality inspection method provided in an embodiment of this application;

[0083] Figure 7 This is a flowchart illustrating the fourth model training method provided in the embodiments of this application;

[0084] Figure 8 This is a schematic diagram of the structure of a document image processing device provided in an embodiment of this application;

[0085] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0086] To make the inventive objectives, features, and advantages of the embodiments of this application more apparent and understandable, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0087] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this application, it should be noted that, unless otherwise expressly specified and limited, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0088] The present application will now be described in detail with reference to specific embodiments.

[0089] In one embodiment, such as Figure 1 As shown, a document image processing method is proposed. This method can be implemented using a computer program and can run on a document image processing device based on the von Neumann architecture. This computer program can be integrated into applications or run as a standalone utility application.

[0090] Specifically, the document image processing method includes:

[0091] S101, Obtain the initial document image, and use the document image enhancement model to perform image block processing on the initial document image to obtain the data of each initial image block.

[0092] The initial document image can be a two-dimensional digital image stored in pixel form, obtained by photographing or scanning a paper document using imaging devices such as scanners, copiers, digital cameras, or terminal cameras. The main content of the initial document image is a user-readable document, which can be a standard formatted document generated by office software, or it can be a ticket, contract, book page, exam paper, business card, invoice, report, etc. Specifically, the document can include layout elements such as text, tables, images, seals, and handwritten annotations.

[0093] In this embodiment, the initial document image may be a low-quality document image. This may be due to factors such as device limitations, uneven ambient lighting, camera shake, inaccurate focus, and lossy compression, resulting in the user capturing an initial document image of poor quality.

[0094] The initial document image can be a printed document image, a handwritten annotation document image, or a document image containing tables and / or barcodes.

[0095] Document image enhancement models are pre-trained deep learning models that integrate multiple image processing functions. The built-in image processing functions of document image enhancement models may include, but are not limited to, document image preprocessing functions, document image restoration (or enhancement) processing functions, and character recognition functions based on the restored document images.

[0096] The initial image patch data includes the initial image data corresponding to each initial image patch. The initial image patch data may include, but is not limited to, standard pixel values, text probability data, and content type labels. The initial image patch is an image patch obtained by sliding window slicing of the standard observation document image. For the interpretation of text probability data, content type labels, and standard observation document images, please refer to the description below this step, which will not be elaborated here.

[0097] In one embodiment, the initial document image can be downloaded from the cloud. Alternatively, the initial document image can be obtained from a user terminal, which can provide the user with an image upload function, allowing the user to instruct the user terminal to upload the local initial document image to the electronic device.

[0098] After acquiring the initial document image, it is input into the document image enhancement model. The model performs pixel normalization, geometric shaping, image content type recognition, and sliding window segmentation on the initial document image to obtain initial image block data. The initial image block data can be obtained through the document image preprocessing function of the document image enhancement model.

[0099] Understandably, pixel normalization processing specifically includes: if the initial document image is a color image, then convert the initial document image to a standard color document image in the standard sRGB color space; if the initial document image is a grayscale image, then cancel the color space conversion processing of the initial document image and determine that the initial document image is a grayscale document image; map the pixel values ​​of the standard color document image or grayscale document image to the [0,1] interval to obtain normalized pixel values.

[0100] Geometric shaping may include: determining the shorter side length of the width and height of the initial document image; if the shorter side length is less than a preset pixel, enlarging the initial document image according to a preset ratio so that the shorter side length of the enlarged initial document image reaches the preset pixel, thus obtaining a standard-sized initial document image. During the enlargement process, the sharpness and aspect ratio of the initial document image are maintained. For example, the preset pixel can be set to 512 pixels. Multiple corner points of the paper are located in the standard-sized initial document image; the rotation angle or perspective distortion of the paper is calculated based on the coordinates of the corner points; the homography matrix and its inverse are calculated from the standard-sized initial document image to a standard rectangular view; and these homography matrices are used as geometric parameters. Furthermore, distortion shaping is performed on the paper area in the standard-sized initial document image to obtain a standard observation document image. The standard observation document image is an image of the distorted paper area cropped from the standard-sized initial document image after distortion shaping of the paper area in the standard-sized initial document image. The standard observation document image is a standard reference image with unchanged document content but a standard format, compared to the initial document image.

[0101] In the embodiments of this application, symbols are used for ease of description. Represents geometric parameters.

[0102] Image content type recognition processing may include: performing semantic segmentation on a standard observation document image to obtain the probability that each pixel in the standard observation document image belongs to text, and also obtaining the content type label corresponding to each pixel block of a preset size in the standard observation document image. The content type label may include, but is not limited to, tables, formulas, barcodes or QR codes, and backgrounds. For example, the preset size pixel block may be set to a 32×32 pixel block.

[0103] The sliding window slicing process can include: slicing a standard observation document image into multiple image blocks using a preset window pixel size and preset overlap pixels; synchronizing text probability data corresponding to each image block based on the probability that each pixel in the standard observation document image belongs to text, where the text probability data includes the probability that each pixel in the image block belongs to text; and synchronizing the content type label corresponding to each image block based on the content type label corresponding to each pixel block of a preset size in the standard observation document image. Furthermore, it can also synchronize the standard pixel value corresponding to each image block based on normalized pixel values, where the standard pixel value includes the normalized pixel value of each pixel in the image block.

[0104] For example, the preset window pixel size can be set to 512×768, and the preset overlap pixels can be set to 32.

[0105] S102, image enhancement processing is performed on each initial image block data through a document image enhancement model to obtain each reference enhanced image block data. From each reference enhanced image block data, a first reference enhanced image block data of the quality check failure type and a second reference enhanced image block data of the quality check success type are determined. Local supplementary enhancement processing is performed on the first reference enhanced image block data to obtain a third reference enhanced image block data. Based on the second reference enhanced image block data and the third reference enhanced image block data, sliding window fusion processing is performed to obtain the target enhanced document image.

[0106] It is understandable that an initial image patch data is the initial image data corresponding to an initial image patch, and a reference enhanced image patch data is the enhanced image data corresponding to an enhanced image patch. The enhanced image patch is the initial image patch after enhancement processing.

[0107] Image enhancement processing means that in the process of denoising the initial image block, it also repairs the strokes and structure of blurred fonts in the initial image block.

[0108] Specifically, the document image enhancement model includes a residual sampling module, which is pre-trained. The residual sampling module performs image enhancement processing by progressively performing multiple rounds of image enhancement based on initial image patch data to obtain reference enhanced image patch data. After obtaining the reference enhanced image patch data, a quality check is performed on each reference enhanced image patch data to obtain a quality check result. The quality check result can include a success type and a failure type. The first reference enhanced image patch data is determined to have failed the quality check, and the second reference enhanced image patch data is determined to have succeeded in the quality check.

[0109] Understandably, the quality inspection process may include: comprehensively evaluating the backprojection error, character recognition sequence confidence, and character recognition error rate corresponding to each reference enhanced image patch to obtain the quality inspection result for each reference enhanced image patch. Here, the backprojection error refers to the error between the initial image patch and the corresponding degraded image patch of the reference enhanced image patch. It is understood that the initial image patch is obtained based on a standard observed document image, and thus can naturally serve as the standard reference image patch for backprojection.

[0110] The third reference enhanced image block data is obtained by performing local supplementary enhancement processing on the first reference enhanced image block data using the residual sampling module. This local supplementary enhancement processing can be achieved by the residual sampling module performing at least one additional round of image enhancement processing on the first reference enhanced image block data to obtain the third reference enhanced image block data.

[0111] Specifically, during the sliding window fusion process, at least one pair of reference enhanced image block data with overlapping pixels is determined from the first reference enhanced image block data and the third reference enhanced image block data. From each pair of reference enhanced image block data with overlapping pixels, first overlapping region pixel data and second overlapping region pixel data located in the overlapping region are determined. For example, image block A and image block B are pixel blocks with overlapping regions. The reference enhanced image block data includes image block data of image block A and image block data of image block B. The image block data of image block A is the first reference enhanced image block data, and the image block data of image block B is the third reference enhanced image block data. First overlapping region pixel data located in the overlapping region can be determined from the first reference enhanced image block data, and second overlapping region pixel data located in the overlapping region can be determined from the third reference enhanced image block data.

[0112] The first enhanced image block, corresponding to the reference enhanced image block data including the pixel data of the first overlapping region, is input into a pre-trained character recognition module for character recognition processing. This yields the first character confidence score of the character sequence corresponding to the first enhanced image block. The mean of the first character confidence scores is then calculated to obtain the average first character confidence score for the first enhanced image block. Similarly, the second enhanced image block, corresponding to the reference enhanced image block data including the pixel data of the second overlapping region, is input into the pre-trained character recognition module for character recognition processing. This yields the second character confidence score of the character sequence corresponding to the second enhanced image block. The mean of the second character confidence scores is then calculated to obtain the average second character confidence score for the second enhanced image block.

[0113] The mean confidence scores of the first character and the mean confidence scores of the second character are summed to obtain a sum. The ratio of the mean confidence score of the first character to the sum is determined to obtain the first overlap weight corresponding to the pixel data of the first overlapping region. Based on the first overlap weight, the second overlap weight corresponding to the pixel data of the second overlapping region is determined. The sum of the first overlap weight and the second overlap weight is 1. The pixel data of the first overlapping region and the pixel data of the second overlapping region are weighted based on the first overlap weight and the second overlap weight to obtain the target pixel data corresponding to the overlapping region.

[0114] A Poisson fusion process is performed on all target pixel data and a pair of overlapping reference enhanced image patches corresponding to all target pixel data to obtain the target enhanced document image. The target enhanced document image is a high-quality enhanced image patch obtained after quality checking, which is then fused to obtain a high-quality document image containing complete document content. The enhanced image patch is an image patch obtained by enhancing the initial image patch, which is an image patch obtained by sliding window slicing of the standard observed document image. Compared to the initial document image, the standard observed document image is a standard reference image with unchanged document content but a standard format. Therefore, compared to the standard observed document image, the target enhanced document image is a document image with unchanged document content in a standard format and higher text clarity; the target enhanced document image is a document image of higher image quality.

[0115] S103, the target enhanced document image is processed by the document image enhancement model to obtain the target character sequence corresponding to the initial document image and the target character confidence level corresponding to the target character sequence.

[0116] Specifically, the document image enhancement module also includes a character recognition module. This module is pre-trained and implements character recognition functionality within the document image enhancement module. The character recognition module performs character recognition processing on the target enhanced document image to obtain the target character sequence corresponding to the initial document image and the target character confidence level corresponding to the target character sequence.

[0117] The document image processing method provided in this application embodiment acquires an initial document image, then performs image block processing on the initial document image using a document image enhancement model to obtain initial image block data, and performs image enhancement processing on each initial image block data using the same model to obtain reference enhanced image block data. From the reference enhanced image block data, a first reference enhanced image block data of the quality check failure type and a second reference enhanced image block data of the quality check success type are determined. The first reference enhanced image block data undergoes local supplementation enhancement processing to obtain a third reference enhanced image block data. Based on the second and third reference enhanced image block data, a sliding window fusion processing is performed to obtain a target enhanced document image. In this way, the quality of the initial document image is repaired or enhanced, thereby obtaining a target enhanced document image with high image quality. Then, the target enhanced document image is processed by the document image enhancement model to obtain the target character sequence corresponding to the initial document image and the target character confidence level corresponding to the target character sequence. This achieves accurate target character sequence recognition based on a high-quality target enhanced document image, improving the character recognition accuracy of the initial document image.

[0118] The above Figure 1 The illustrated embodiments describe specific implementations of the document image processing method of this application. For example, during execution... Figure 1 For details regarding step S102, "obtaining reference enhanced image block data by performing image enhancement processing on each initial image block data using a document image enhancement model," please refer to [link / reference]. Figure 2 The diagram shows a flowchart of an image enhancement processing method based on a residual sampling module.

[0119] Specifically, such as Figure 2 As shown, the image enhancement processing method based on the residual sampling module includes the following steps:

[0120] S201, Determine the current sampling step number corresponding to the residual sampling module.

[0121] S202, determine the sampling weight parameter corresponding to the current sampling step.

[0122] The residual sampling model is obtained by distilling a pre-trained, undistilled residual sampling module. The undistilled residual sampling module is a multi-step, complex "teacher" diffusion model, while the residual sampling module is a short-step, efficient "student" model. Through distillation, the residual sampling module retains the core advantages of the undistilled residual sampling module (denoising and repairing blurred fonts) while significantly reducing the image enhancement processing time in practical applications. This allows the residual sampling module to be deployed in server environments with high latency requirements, and even on mobile devices with limited computing power.

[0123] It is understandable that the principle of the residual sampling module is to obtain reference enhanced image block data by performing multiple rounds of image enhancement processing based on the initial image block data through a small number of sampling steps. In each sampling step to achieve image enhancement processing, the weight parameters used by the residual sampling module for image enhancement processing are also different. Therefore, the current sampling step number of the residual sampling module can be determined first.

[0124] Different sampling steps correspond to different sampling weight parameters, which can be learned during the training of the residual sampling module. The sampling step number and the corresponding sampling weight parameters can be stored in a sampling mapping table. Therefore, after determining the current sampling step number, the sampling weight parameters corresponding to the current sampling step number can be obtained from the sampling mapping table. Specifically, the sampling weight parameters can include semantic guidance strength weights, frequency channel weights, and frequency domain regularization injection control weights.

[0125] S203, based on the sampling weight parameters, the residual sampling module is used to perform image enhancement processing on each initial image block data to obtain each basic enhanced image block data.

[0126] In one embodiment, the residual sampling module uses an image enhancement calculation formula to obtain basic enhanced image patch data, and the image enhancement calculation formula satisfies the following formula:

[0127]

[0128] in, Based on enhanced image patch data, For initial image patch data, Step size, The drift coefficient (of the probability flow ODE) To score the strength of the network module, For the scoring network module, The semantic guidance strength weight is the sampled weight at step m. The frequency channel weights for the m-th sampling step. The loss gradient of the character recognition module. Inject control weights into the frequency domain regularization of the m-th sampling step. Let m be the frequency domain loss gradient, and m be the current sampling step number. , , These are the sampling weight parameters.

[0129] Specifically, The first gradient update formula, obtained from the first gradient update calculation formula, satisfies the following formula:

[0130]

[0131] in, This represents the text probability data corresponding to the initial image patch data. This represents element-wise multiplication. Used to control the background leakage ratio , This represents the perceptual loss based on LPIPS or the perceptual loss based on VGG. According to... Figure 1 The relevant description in the illustrated embodiment states that the text probability data includes the probability that each pixel in the initial image block corresponding to the initial image block data belongs to text.

[0132] For example, as m increases, the aforementioned semantic guidance strength weights... It decreases linearly from the initial value of 0.2 to 0.05.

[0133] For example, the number of sampling steps m can be 1-10, where m is a positive number.

[0134] Optional, regarding the above Figure 1In the embodiment shown, the local supplementary enhancement processing mentioned in step S102 can also be performed by the residual sampling module on the first reference enhanced image block data for an additional 1-2 steps based on the image enhancement calculation formula in this step, so as to obtain the third reference enhanced image block data.

[0135] S204, determine whether the current sampling step number is greater than or equal to the preset sampling step number.

[0136] Understandably, the preset number of sampling steps is used to decide whether to terminate the enhancement process of the residual sampling module. When the current number of sampling steps is greater than or equal to the preset number of sampling steps, the decision can be made to terminate the enhancement process of the residual sampling module.

[0137] S205, if the current sampling step number is less than the preset sampling step number, then obtain the next sampling step number corresponding to the current sampling step number, use the next sampling step number as the current sampling step number, and use the basic enhanced image block data as the initial image block data.

[0138] Understandably, when the current sampling step number is less than the preset sampling step number, the multi-round enhancement process of the residual sampling module is not yet complete and the next round of enhancement processing needs to be performed. Therefore, the next sampling step number corresponding to the current sampling step number can be obtained. For example, if the current sampling step number is step m, the next sampling step number is step m+1. The next sampling step number is used as the current sampling step number, and the process returns to steps S202-S204. The sampling weight parameters corresponding to the current sampling step number are determined again, and the residual sampling module is used again to perform image enhancement processing on each initial image block data based on the sampling weight parameters to obtain each basic enhanced image block data. Finally, it is determined whether the current sampling step number is greater than or equal to the preset sampling step number to decide whether to end the multi-round enhancement process of the residual sampling module.

[0139] In this way, after one round of image enhancement, the base enhanced image block data obtained in that round is then subjected to another round of image enhancement processing. This process can gradually denoise the initial document image containing noise, while also making the strokes and structure of the blurred fonts in the initial image blocks clearer and clearer.

[0140] S206, if the current sampling step number is greater than or equal to the preset sampling step number, then the basic enhanced image block data corresponding to the current sampling step number shall be used as the reference enhanced image block data.

[0141] Understandably, when the current sampling step number is greater than or equal to the preset sampling step number, it indicates that the multi-round enhancement process of the residual sampling module has been completed. The basic enhanced image block data obtained by the residual sampling module in the current sampling step number is the high-quality image data obtained after denoising and repairing blurred fonts. The basic enhanced image block data corresponding to the current sampling step number are the basic enhanced image block data obtained by the residual sampling module in the current sampling step number. Therefore, the basic enhanced image block data corresponding to the current sampling step number are used as the reference enhanced image block data. The reference enhanced image block data is the image data of the high-quality enhanced image block obtained after denoising and repairing blurred fonts. Among them, compared with the initial image block corresponding to the initial image block data, the high-quality enhanced image block has less noise and the font of the high-quality enhanced image block is clearer.

[0142] Optionally, in another embodiment, when performing step S204: determining whether the current sampling step number is greater than or equal to the preset sampling step number, online early stop judgment processing can also be performed. The document image enhancement module also includes an image degradation module, and the online early stop judgment processing can include: using the image degradation module to perform image degradation processing on each reference enhanced image block data to obtain reference degraded image block data, determining the reference back projection error between the reference degraded image block data and the initial image block data; determining the current character error evaluation value corresponding to the current predicted character sequence, the current predicted character sequence is obtained by using the character recognition module to perform character recognition processing on the reference enhanced image block data; obtaining the previous sampling step number corresponding to the current sampling step number, obtaining the previous reference degraded image block data corresponding to the previous sampling step number, the previous reference degraded image block data is obtained by the image degradation module performing image degradation processing on the previous reference enhanced image block data corresponding to the previous sampling step number, obtaining the previous reference back projection error corresponding to the previous sampling step number. The previous reference back projection error is the error between the previous reference degraded image block data and the initial image block data. The previous predicted character sequence corresponding to the previous sampling step is obtained, along with the previous character error evaluation value corresponding to the previous predicted character sequence. The previous predicted character sequence is obtained by performing character recognition processing on the previous reference enhanced image block data using a character recognition module. If the number of consecutive steps for which the first inequality corresponding to the reference back projection error and the previous reference back projection error holds true is equal to a preset step threshold, and the number of consecutive steps for which the second inequality corresponding to the current character error evaluation value and the previous character error evaluation value holds true is equal to a preset step threshold, then the basic enhanced image block data corresponding to the current sampling step is used as the reference enhanced image block data. Otherwise, step S205 is executed, and steps S202-S204 are returned to the execution steps.

[0143] The first inequality satisfies the following formula:

[0144]

[0145] in, This indicates the projection error from the previous reference back. Indicates the reference projection error. The value can be set to

[0146] The back projection error calculation formula satisfies the following equation:

[0147]

[0148] in, This indicates the image degradation module. This represents the reference degraded image patch data output by the image degradation module. This represents the initial image patch data. This represents the L1 norm.

[0149] The second inequality satisfies the following formula:

[0150]

[0151] in, This indicates the error evaluation value of the previous character. This indicates the current character error evaluation value. The value can be set to .

[0152] Optionally, the preset step threshold can be set to 2.

[0153] Optionally, the current character error evaluation value can be obtained by calculating the edit distance between the predicted character sequence corresponding to the previous step (i.e., the previous predicted character sequence) and the current predicted character sequence corresponding to the current sampling step.

[0154] Thus, by performing online early stopping judgment processing, the residual sampling module can be terminated in advance to perform image enhancement processing for subsequent steps, thereby saving latency.

[0155] In the image enhancement processing method provided in this application embodiment, the image enhancement calculation formula represents the image generation process of enhancing a low-quality image to obtain a high-quality image, by using the loss gradient of the character recognition module. By directly injecting it into the image generation process, the enhancement process is directly driven by the ultimate goal of "improving readability". The model will actively repair or generate the strokes and structures that are most conducive to correct recognition, overcoming the problem that general enhancement models may damage the semantics of text, thereby significantly reducing the character error rate of the subsequent character recognition module.

[0156] The above Figure 2 The illustrated embodiment describes an image enhancement processing method based on a residual sampling module. For an exemplary description of the training process of the residual sampling module, please refer to [link to relevant documentation]. Figure 3 The flowchart of the first model training method is shown.

[0157] Specifically, the first model training method is the training method for the residual sampling module, and the first model training method includes:

[0158] S301, during the model training process of the residual sampling module, the first sample enhanced image block data generated by the initial residual sampling module in the k-th sampling step is obtained, and the second sample enhanced image block data generated by the initial residual sampling module in the k+1-th sampling step is determined based on the preset residual update formula.

[0159] It is understandable that the preset residual update formula satisfies the following formula:

[0160] in, Enhance the image patch data for the second sample. Enhance the image patch data for the first sample. Step size, The drift coefficient of the probability flow ODE. To score the strength of the network module, For the scoring network module, The semantic guidance strength weights are for the k-th sampling step. The frequency channel weights for the k-th sampling step. The loss gradient of the character recognition module. Inject control weights into the frequency domain regularization of the k-th sampling step. This represents the frequency domain loss gradient.

[0161] Optional, The second gradient update formula is obtained from the second gradient update formula, which satisfies the following formula:

[0162]

[0163] in, This represents the text probability data corresponding to the initial image patch data of the first sample. This represents element-wise multiplication. Used to control the background leakage ratio , This represents the perceptual loss based on LPIPS or the perceptual loss based on VGG. According to... Figure 1The relevant description in the illustrated embodiment states that the text probability data includes the probability that each pixel in the first sample initial image block corresponding to the first sample initial image block data belongs to text.

[0164] It is understandable that the initial image patch data of the first sample is the input data of the initial residual sampling module in the first sampling process. The initial residual sampling module is the training residual sampling module that is initialized, and the parameters in the training residual sampling module can be set with initial values ​​during the initialization process.

[0165] Understandably, the above second gradient update formula is designed to strengthen the loss gradient of the character recognition module in the text region while weakening the background region.

[0166] For example, in the process of using printed document image data as sample data for model training, the above... It can be set to 0.1. During the process of using handwritten annotation document image data as sample data for model training, the above... It can be set to 0.25. During model training using document image data containing tables and / or barcodes as sample data, the above... It can be set to 0.1.

[0167] Optional, It is obtained through the following formula:

[0168]

[0169] in, For Fourier transform operators, Indicates a high-frequency mask. This represents element-wise multiplication. Describing the L1 norm, As weight, It is a set of JPEG ring band indexes. Extract the k-th frequency component from the image after Fourier transform.

[0170] Understandable, This includes indices of specific frequency components associated with JPEG ringing artifacts, corresponding to frequency regions prone to artifacts during JPEG compression. The first term in the above formula brings the statistical characteristics of the high-frequency components closer to the real document, while the second term suppresses JPEG ringing artifacts.

[0171] For example, in the process of using printed document image data as sample data for model training, the above... It can be set to 0.01. During model training using document image data containing tables and / or barcodes as sample data, the above... It can be set to 0.02.

[0172] S302, obtain the third sample enhanced image block data generated by the undistilled residual sampling module in the sampling process of step k+1, determine the distillation loss corresponding to the third sample enhanced image block data and the second sample enhanced image block data using the first loss calculation formula, and perform model parameter adjustment processing on the initial residual sampling module based on the distillation loss, semantic guidance intensity weight, frequency channel weight, and frequency domain regularization injection control weight to obtain the residual sampling module.

[0173] The formula for calculating the first loss satisfies the following equation:

[0174] in, For distillation losses, For image residual alignment loss, for The weight, For intermediate statistical matching loss, for The weight, To display the gradient preservation loss of the character recognition module, for The weight.

[0175] The second loss calculation formula satisfies the following equation:

[0176]

[0177] in, This is the first enhanced image patch inference data output by the initial residual sampling module from the initial image patch data of the sample. This is the second enhanced image patch inference data output by the undistilled residual sampling module from the initial image patch data of the sample. It is an L1 norm.

[0178] The third loss calculation formula satisfies the following equation:

[0179]

[0180] in, For Fourier transform operators, Indicates the range of the spectrum. For image gradient operators, Constructing operators for histograms The KL divergence is used to measure the difference in gradient distributions. As weight, As weight, It is an L1 norm.

[0181] The fourth loss calculation formula satisfies the following equation:

[0182]

[0183] in, express and The maximum mean difference This indicates the character recognition module.

[0184] Understandably, the undistilled residual sampling module is a multi-step, complex "teacher" diffusion model that has been trained, while the residual sampling module is a fewer-step, more efficient "student" model. Distillation is used to obtain the residual sampling module, allowing it to retain the core advantages of the undistilled residual sampling module: denoising and repairing blurred fonts.

[0185] In the first model training method of this application, the preset residual update formula represents the image generation process of generating a denoised image from a noisy image, by using the loss gradient of the character recognition module. By directly injecting it into the image generation process, the enhancement process is directly driven by the ultimate goal of "improving readability". The model will actively repair or generate the strokes and structures that are most conducive to correct recognition, overcoming the problem that general enhancement models may damage the semantics of text, thereby significantly reducing the character error rate of the subsequent character recognition module.

[0186] It is understandable that the scoring network module, character recognition module, and residual sampling module are all sub-networks that play different functions in the document image enhancement model.

[0187] For an example, regarding the training process of the scoring network module and the character recognition module, please refer to [link / reference] respectively. Figure 4 The second model training method shown and Figure 5 The flowchart of the third model training method is shown.

[0188] Specifically, Figure 4 The second model training method shown is the training method for the scoring network module, which includes:

[0189] S401, During the training of the scoring network module, the sample noisy document image and the noise data distribution gradient label corresponding to the sample noisy document image are obtained, and the initial scoring network module is used to identify the predicted noise data distribution gradient corresponding to the sample noisy document image.

[0190] S402, the fifth loss calculation formula is used to determine the score matching loss corresponding to the gradient of the predicted noise data distribution and the gradient label of the noise data distribution. Based on the score matching loss, the model parameters of the initial score network module are adjusted to obtain the score network module.

[0191] It is understandable that the initial score network module is the score network module to be trained that is initialized. During the initialization process, the parameters in the score network module to be trained can be set to initial values.

[0192] The fifth loss calculation formula satisfies the following formula:

[0193]

[0194] in, For score matching loss, As weight, The result is obtained through the sixth loss calculation formula. It is obtained through the seventh loss calculation formula.

[0195] The sixth loss calculation formula satisfies the following formula:

[0196]

[0197] in, For sample noisy document images, Through Noise samples were added to obtain the sample. For the noiseless document image corresponding to the sample noisy document image, Gradient labels for the distribution of noisy data. Indicates time Observed Under the premise of noise-free image The probability distribution, For the noise sample, For the initial scoring network module, The input is and , The output is the gradient of the predicted noise data distribution. Indicates time step The relevant weighting function, The square operation represents the L2 norm. Represents all possible Seeking expectations.

[0198] The seventh loss calculation formula satisfies the following formula:

[0199] in, For Fourier transform operators, Indicates a high-frequency mask. This represents element-wise multiplication. Describing the L1 norm, As weight, It is a set of JPEG ring band indexes. Extract the k-th frequency component from the image after Fourier transform.

[0200] Specifically, the scoring network module plays a crucial role in recovering high-quality images from noisy images. The score function fitting formula is obtained by fitting the score function, and the score function fitting formula satisfies the following formula:

[0201]

[0202] in, Indicates the current noise level The logarithmic density gradient.

[0203] Optional, It can be obtained through the forward diffusion formula, which satisfies the following formula:

[0204]

[0205] in, This represents sample noisy document images at different noise levels. Represents standard Brownian motion. Indicates the drift coefficient. Indicates noise intensity. The forward diffusion formula represents the... Adding noise results in a noisy image. The process.

[0206] In the second model training method of this application embodiment, the scoring network module is trained so that it can indicate the direction of "towards a high-probability document appearance" during the image enhancement process.

[0207] Specifically, Figure 5 The third model training method shown is the training method for the character recognition module, which includes:

[0208] S501: During the training process of the character recognition module, the sample augmented document image and the character sequence label corresponding to the sample augmented document image are obtained. Based on the initial character recognition module, the sample augmented document image is processed to obtain the predicted character data.

[0209] S502: The character recognition loss corresponding to the predicted character data and character sequence label is determined by the recognition loss calculation formula. Based on the character recognition loss, the model parameters of the initial character recognition module are adjusted to obtain the character recognition module.

[0210] It is understandable that the initial character recognition module is the character recognition module to be trained that is initialized. During the initialization process, the parameters in the character recognition module to be trained can be set to initial values.

[0211] The recognition loss calculation formula satisfies the following formula:

[0212]

[0213] in, For character recognition loss, For continuous time-series classification loss, For attention deficit, For differentiable editable distance loss, It is obtained by recursion using the recursive formula for differentiable editable distance. , , They are respectively , , The weight.

[0214] The identification loss sub-calculation formula is obtained by recognizing the following formula:

[0215]

[0216] in, Enhance document images for samples. It is a candidate alignment path that can be collapsed into an "I". For path The symbol in frame t, P () indicates that the symbol for frame t is The probability, T For path Length, This indicates that the product is obtained by independent multiplication along the time axis. Path probability, It is all that can be folded into The set of paths For character sequence labels.

[0217] Specifically, The recursive formula for the differentiable editable distance is obtained by recursion, and the recursive formula for the differentiable editable distance satisfies the following formula:

[0218]

[0219]

[0220] It is the front Each visual step is aligned to the front. The soft cost of each target character; It is the current matching cost (negative log-likelihood). It is a balancing factor; Control the smoothness. This recursion can be backpropagated to give a smooth edit distance gradient.

[0221] During the recursive process using the differentiable editable distance recursive formula, the soft edit matrix can be obtained. The elements stored in the soft edit matrix are... When the recursion reaches the bottom right element of the soft edit matrix... hour, Where T represents the row number of the bottom-right element of the soft-edit matrix, and T represents the column number of the bottom-right element of the soft-edit matrix.

[0222] Optionally, during the process of using handwritten annotation-type document image data as sample data for model training, the above... It can be set to 0.3.

[0223] In the third model training method provided in this application embodiment, the character recognition accuracy of the character recognition module can be improved by training the character recognition module.

[0224] In addition to a scoring network module, a character recognition module, and a residual sampling module, the document image enhancement model of this application may also include a character stroke prior module and an image degradation module. The document image processing method of this application may also include:

[0225] During the training process of the document image enhancement model, the comprehensive loss of the initial document image enhancement model is determined by the comprehensive loss calculation formula. Based on the comprehensive loss, the model parameters of the initial document image enhancement model are adjusted to obtain the document image enhancement model.

[0226] The formula for calculating the overall loss satisfies the following:

[0227]

[0228] in, To summarize the model loss, The score matching loss is the score matching loss of the scoring network module. The character recognition loss of the character recognition module. For the character shape loss of the character shape stroke prior module, The stroke loss is for the character stroke prior module. For the character component loss of the character stroke prior module, Image degradation loss for the image degradation module. For frequency domain loss, To perceive loss.

[0229] Specifically, The character shape loss calculation formula satisfies the following equation:

[0230]

[0231] in, This represents the first sample augmented document image based on the first sample initial document image. This represents the ideal glyph diagram for the initial document image of the first sample. This represents the stroke direction and skeleton corresponding to the first sample augmented document image. This represents the stroke direction and skeleton of an ideal character shape. This represents the stroke field network, and the output of the stroke field network can be denoted as: , Representing an image The direction of the strokes, Representing an image The skeleton.

[0232] Understandably, the text content identified from the initial document image of the first sample is rendered into sharp glyphs using a standard digital font library. Then, geometric transformations are applied to precisely match the position, size, and deformation of the blurred characters in the original image, resulting in an ideal glyph image aligned with the actual layout of the initial document image of the first sample. The expression for the ideal glyph image can be written as: ,in, Represented as a differentiable glyph renderer, This represents the text content identified from the initial document image of the first sample. This represents style deformation parameters (including font family, weight, slant, and thin plate spline (TPS) control points). express Figure 1 Geometric parameters in the illustrated embodiment.

[0233] Specifically, The stroke loss calculation formula, obtained from the stroke loss calculation formula, satisfies the following formula:

[0234]

[0235] in, express Medium pixel The direction of the strokes at that location; express Medium pixel The direction of the strokes at that location; It is a binary function used to focus attention on pixels with strokes.

[0236] Specifically, The character component loss calculation formula is obtained from the character component loss calculation formula, which satisfies the following formula:

[0237]

[0238] in, It is a character A collection of components, Indicates extraction component The direction of the strokes and the skeleton.

[0239] Specifically, The image degradation loss calculation formula is obtained from the image degradation loss calculation formula, which satisfies the following formula:

[0240]

[0241] in, For the image degradation module The degraded image obtained by image degradation processing. The standard observed document image corresponding to the initial document image of the first sample. As weight, Used for evaluation and Image similarity between It is an L1 norm.

[0242] Optionally, the character stroke prior module may include a stroke field network and a differentiable character shape renderer. When training the stroke field network, it can be based on... , and Weighted processing is performed to determine the character shape comprehensive loss, and then the character shape comprehensive loss is used to update the model parameters of the stroke field network to obtain the trained stroke field network.

[0243] For example, in the process of using printed document image data as sample data for model training, the above comprehensive loss calculation formula... It can be set to 0.2. It can be set to 0.1. When using handwritten annotation document image data as sample data for model training, the above comprehensive loss calculation formula... It can be set to 0.3. It can be set to 0.05. During model training using document image data containing tables and / or barcodes as sample data, the above comprehensive loss calculation formula... It can be set to 0.15. It can be set to 0.25.

[0244] Specifically, in the process of adjusting the model parameters of the initial document image enhancement model based on the comprehensive model loss to obtain the document image enhancement model, the goal of training is to achieve the following: ,in,

[0245] These represent the model parameters of the scoring network module. This represents the model parameters of the character recognition module. This represents the model parameters of the differentiable glyph renderer. This represents the model parameters of the image degradation module.

[0246] The weighted process of the aforementioned comprehensive calculation formula runs through the entire backpropagation process of the end-to-end training of the document image enhancement model and takes effect at each step of gradient descent. Thus, by achieving joint training through this comprehensive calculation formula, all prior information (including the scoring network module based on the diffusion model and the character stroke prior module) and constraints jointly influence the model's learning in the same training process. This integrates multi-scale priors and conclusions, enabling the model to have a deep understanding of the morphology of characters (character shape, orientation, skeleton). Even in cases of severe blurring or incompleteness, it can recover characters with regular structures and clear details, with particularly significant effects on complex character systems such as Chinese characters.

[0247] For example, in execution Figure 1 In the illustrated embodiment, step S102, "determining from each of the reference enhanced image block data the first reference enhanced image block data of the quality check failure type and the second reference enhanced image block data of the quality check success type," can be found in the following details. Figure 6 The diagram shows a flowchart of the document enhancement image quality inspection method.

[0248] Specifically, such as Figure 6 As shown, the document enhancement image quality inspection method includes the following steps:

[0249] S601, Perform character recognition processing on each reference enhanced image block data to obtain the predicted character sequence and the predicted character confidence corresponding to each reference enhanced image block data;

[0250] S602, The image degradation module is used to perform image degradation processing on each reference enhanced image block data to obtain reference degraded image block data, and the reference back projection error between the reference degraded image block data and the initial image block data is determined;

[0251] S603, Determine the character error evaluation value corresponding to the predicted character sequence;

[0252] S604, based on the predicted character confidence, reference back projection error and character error evaluation value, the quality assessment processing of each reference enhanced image block data is performed to obtain the quality check results of each reference enhanced image block data;

[0253] S605, based on the quality inspection results, determine the first reference enhanced image block data of the quality inspection failure type and the second reference enhanced image block data of the quality inspection success type.

[0254] In step S601, a character recognition module is used to perform character recognition processing on each reference enhanced image block data to obtain the predicted character sequence and the corresponding predicted character confidence for each reference enhanced image block data. For details on the training process of the character recognition module, please refer to [link to relevant documentation]. Figure 5 The third model training method shown will not be described in detail here.

[0255] In step S602, the formula for calculating the reference back projection error satisfies the following formula:

[0256]

[0257] in, Indicates the reference projection error. This indicates the image degradation module. This represents the reference degraded image patch data output by the image degradation module. This represents the initial image patch data. This represents the L1 norm.

[0258] In step S603, it can be understood that the reference enhanced image patch data is obtained by the residual sampling module through the enhancement processing corresponding to the m-th sampling step, and the character error evaluation value can be obtained by using the historical predicted character sequence corresponding to the (m-1)-th sampling step as a temporary ground truth and calculating the edit distance with the predicted character sequence. The historical predicted character sequence corresponding to the (m-1)-th sampling step is obtained by the character recognition module performing character recognition on the enhanced image patch data obtained by the residual sampling module based on the (m-1)-th sampling step.

[0259] In step S604, a preset character confidence threshold, a preset back projection error threshold, and a preset character error rate threshold are obtained. For each reference augmented image block data, if the predicted character confidence is greater than or equal to the preset character confidence threshold, the reference back projection error is less than or equal to the preset back projection error threshold, and the character error evaluation value is less than or equal to the preset character error rate threshold, a quality check result of the successful quality check type corresponding to the reference augmented image block data can be generated; otherwise, a quality check result of the quality check type corresponding to the reference augmented image block data is generated.

[0260] Thus, checking the reference back projection error can assess whether the reference enhanced image patch data and the initial image patch data meet physical consistency, and checking the character error evaluation value can assess whether the reference enhanced image patch data and the initial image patch data meet semantic consistency. Only when the predicted character confidence, reference back projection error, and character error evaluation value all meet the corresponding thresholds can the quality check of the reference enhanced image patch data be determined to have passed, and thus proceed to the subsequent sliding window fusion process to ensure the enhancement quality of the target enhanced document image.

[0261] If the reference enhanced image patch data fails the quality check, it must undergo local supplementary enhancement processing using the residual sampling module before proceeding to the subsequent sliding window fusion process. Optionally, the residual sampling module can perform an additional 1-2 steps of image enhancement calculation processing on the reference enhanced image patch data that fails the quality check to improve its enhancement quality.

[0262] For an example, regarding the training process of the image degradation module, please refer to [link / reference]. Figure 7 The flowchart of the fourth model training method is shown.

[0263] Specifically, this fourth model training method is the training method for the image degradation module, and this fourth model training method includes:

[0264] S701: During the training process of the image degradation module, the sample ideal document image and the real document image corresponding to the sample ideal document image are obtained. The image degradation calculation formula of the initial image degradation module is used to perform image degradation processing on the sample ideal document image to obtain the observed degraded document image.

[0265] S702: Based on the eighth loss calculation formula, determine the image degradation loss corresponding to the real document image and the observed degraded document image, and adjust the model parameters of the initial image degradation model based on the degradation loss to obtain the image degradation model.

[0266] The sample ideal document image and the real document image are document images with different resolutions and different visual characteristics, collected based on the same sample document content. The sample ideal document image can be obtained by scanning the sample document content with a high-definition scanner (resolution 300-600 dpi), or by vector PDF rendering of the sample document content. The sample document image is a high-resolution image, and its visual characteristics may include, but are not limited to, no motion / defocus blur, no compression block effect, uniform illumination, no fingerprints, no bleed-through, and no noise. The real document image is a document image obtained by photographing the document content in a real environment. The real document image is a low-resolution image, and its visual characteristics may include, but are not limited to, motion / defocus blur, loss of detail, noise, uneven illumination, fingerprints, bleed-through, and compression block effect.

[0267] It is understandable that the initial image degradation module is the initialization module for the image degradation to be trained. During the initialization process, the parameters in the image degradation module for the image degradation to be trained can be set to initial values.

[0268] Specifically, the image degradation calculation formula satisfies the following formula:

[0269]

[0270] in, To observe degraded document images, For all degradation parameters in the initial image degradation model, For the ideal document image of the sample, Is the factor downsampling, For spatial fuzzy kernel, For convolution operations, For composite noise parameters, This is a low-frequency light generation module. For superimposed paper texture and / or transparent texture; Used to simulate the distortion that occurs in image signals during digital imaging and storage.

[0271] Specifically, the eighth loss calculation formula satisfies the following formula:

[0272]

[0273] in, For image degradation loss, Images of real documents As weight, Used for evaluation and Image similarity between It is an L1 norm.

[0274] For example, the parameter configuration in the trained image degradation module can be as follows: the spatial blur kernel includes the motion blur kernel length, motion blur angle, and defocus blur radius. The motion blur kernel length can be 5-15 pixels, the motion blur angle can be -90° to +90°, and the defocus blur radius can be 1-3 pixels. The kernel noise parameters can include the Gaussian noise standard deviation and the Poisson noise parameter. The Gaussian noise standard deviation can be 0-25, and the Poisson noise parameter can be 0-20. The low-frequency illumination generation module can include the low-frequency illumination perturbation cutoff frequency. The low-frequency illumination perturbation cutoff frequency can be 1 / 256. The transparency of the bleed effect can be 0-0.2, and the JPEG compression quality factor can be 40-80.

[0275] Thus, by training the image degradation module using the fourth model training method, it is possible to ensure that the reference enhanced image patch data can be back-projected to approximate the initial image patch data. This ensures that the reference enhanced image patch data can achieve a physical closed loop from the enhanced image back-projection to the low-quality observation image, thereby ensuring that the enhancement result (i.e. the target enhanced document image) is physically credible and based on evidence, effectively suppressing the arbitrary "transformation" of the model.

[0276] The following will combine Figure 8 This application provides a detailed description of the document image processing apparatus provided in its embodiments. It should be noted that... Figure 8 The document image processing apparatus shown is used to execute this application. Figures 1-7 The methods shown in the embodiments are for illustrative purposes only, illustrating the parts relevant to the embodiments of this application. For specific technical details not disclosed, please refer to this application. Figures 1-7 The example shown.

[0277] Please see Figure 8 This diagram illustrates the structure of a document image processing apparatus according to an embodiment of this application. The document image processing apparatus 1 can be implemented as all or part of the apparatus through software, hardware, or a combination of both. According to some embodiments, the document image processing apparatus 1 includes an image segmentation module 11, an image enhancement module 12, and a character recognition module 13, specifically used for:

[0278] Image segmentation module 11 is used to acquire an initial document image and perform image segmentation processing on the initial document image through a document image enhancement model to obtain initial image block data;

[0279] Image enhancement module 12 is used to perform image enhancement processing on each of the initial image block data through the document image enhancement model to obtain each reference enhanced image block data, determine a first reference enhanced image block data of the quality check failure type and a second reference enhanced image block data of the quality check success type from each of the reference enhanced image block data, perform local supplement enhancement processing on the first reference enhanced image block data to obtain a third reference enhanced image block data, and perform sliding window fusion processing on the second reference enhanced image block data and the third reference enhanced image block data to obtain a target enhanced document image;

[0280] The character recognition module 13 is used to perform character recognition processing on the target enhanced document image through the document image enhancement model to obtain the target character sequence corresponding to the initial document image and the target character confidence level corresponding to the target character sequence.

[0281] Optionally, the image enhancement module 12 is specifically used for:

[0282] Determine the current sampling step number corresponding to the residual sampling module, and determine the sampling weight parameter corresponding to the current sampling step;

[0283] Based on the sampling weight parameters, the residual sampling module is used to perform image enhancement processing on each of the initial image block data to obtain each basic enhanced image block data.

[0284] Determine whether the current sampling step number is greater than or equal to the preset sampling step number;

[0285] If the current sampling step is less than the preset sampling step, then the next sampling step corresponding to the current sampling step is obtained, the next sampling step is used as the current sampling step, the basic enhanced image block data is used as the initial image block data, and the steps of determining the sampling weight parameter corresponding to the current sampling step, and using the residual sampling module to perform image enhancement processing on each of the initial image block data based on the sampling weight parameter to obtain each basic enhanced image block data are executed.

[0286] If the current sampling step number is greater than or equal to the preset sampling step number, then the basic enhanced image block data corresponding to the current sampling step number are used as the reference enhanced image block data.

[0287] Optionally, the document image enhancement model further includes a scoring network module and a character recognition module, and the document image processing device 1 is further used for:

[0288] During the model training process of the residual sampling module, the first sample augmented image patch data generated by the initial residual sampling module in the k-th sampling step is obtained, and the second sample augmented image patch data generated by the initial residual sampling module in the k+1-th sampling step is determined based on the preset residual update formula.

[0289] The preset residual update formula satisfies the following formula:

[0290] in, For the second sample, enhance the image patch data. Enhance the image patch data for the first sample. Step size, The drift coefficient, The strength of the scoring network module, For the scoring network module, The semantic guidance strength weights are for the k-th sampling step. The frequency channel weights for the k-th sampling step. The loss gradient of the character recognition module is... Inject control weights into the frequency domain regularization of the k-th sampling step. The frequency domain loss gradient;

[0291] The third sample enhanced image block data generated by the undistilled residual sampling module in the sampling process of step k+1 is obtained. The distillation loss corresponding to the third sample enhanced image block data and the second sample enhanced image block data is determined by the first loss calculation formula. Based on the distillation loss, the semantic guidance strength weight, the frequency channel weight, and the frequency domain regularization injection control weight, the model parameters of the initial residual sampling module are adjusted to obtain the residual sampling module.

[0292] The first loss calculation formula satisfies the following formula:

[0293] in, For distillation losses, For image residual alignment loss, for The weight, For intermediate statistical matching loss, for The weight, To display the gradient preservation loss of the character recognition module, for The weights;

[0294] The second loss calculation formula satisfies the following equation:

[0295]

[0296] in, This refers to the first enhanced image patch inference data output by the initial residual sampling module from the initial image patch data of the sample. This refers to the second enhanced image patch inference data output by the undistilled residual sampling module from the initial image patch data of the sample. It is an L1 norm;

[0297] The third loss calculation formula, obtained through the third loss calculation formula, satisfies the following formula:

[0298]

[0299] in, For Fourier transform operators, Indicates the range of the spectrum. For image gradient operators, Constructing operators for histograms The KL divergence is used to measure the difference in gradient distributions. As weight, As weight, It is an L1 norm;

[0300] The fourth loss calculation formula satisfies the following equation:

[0301]

[0302] in, express and The maximum mean difference This refers to the character recognition module.

[0303] Optionally, the document image processing device 1 is also used for:

[0304] During the training of the scoring network module, sample noisy document images and noise data distribution gradient labels corresponding to the sample noisy document images are obtained. An initial scoring network module is used to identify the predicted noise data distribution gradient corresponding to the sample noisy document images. The fifth loss calculation formula is used to determine the score matching loss corresponding to the predicted noise data distribution gradient and the noise data distribution gradient labels. Based on the score matching loss, the model parameters of the initial scoring network module are adjusted to obtain the scoring network module.

[0305] The fifth loss calculation formula satisfies the following formula:

[0306]

[0307] in, The score matching loss, As weight, The result is obtained through the sixth loss calculation formula. It is obtained through the seventh loss calculation formula;

[0308] The sixth loss calculation formula satisfies the following formula:

[0309]

[0310] in, The sample noisy document image, Through Noise samples were added to obtain the sample. The noise-free document image corresponding to the sample noisy document image. The noise data distribution gradient label, Indicates time Observed Under the premise of noise-free image The probability distribution, For the noise sample, This is the initial score network module. The input is and , The output is the gradient of the predicted noise data distribution. Indicates time step The relevant weighting function, The square operation represents the L2 norm. Represents all possible Seeking expectations;

[0311] The seventh loss calculation formula satisfies the following formula:

[0312] in, For Fourier transform operators, Indicates a high-frequency mask. This represents element-wise multiplication. Describing the L1 norm, As weight, It is a set of JPEG ring band indexes. Extract the k-th frequency component from the image after Fourier transform.

[0313] Optionally, the document image processing device 1 is also used for:

[0314] During the training process of the character recognition module, sample augmented document images and character sequence labels corresponding to the sample augmented document images are acquired. Based on the initial character recognition module, the sample augmented document images are processed to obtain predicted character data. The character recognition loss corresponding to the predicted character data and the character sequence labels is determined by the recognition loss calculation formula. Based on the character recognition loss, the model parameters of the initial character recognition module are adjusted to obtain the character recognition module.

[0315] The formula for calculating the recognition loss satisfies the following formula:

[0316]

[0317] in, The character recognition loss, For continuous time-series classification loss, For attention deficit, For differentiable editable distance loss, It is obtained by recursion using the recursive formula for differentiable editable distance. , , They are respectively , , The weights;

[0318] The identification loss formula is obtained by recognizing the following formula:

[0319]

[0320] in, Enhance the document image of the sample. It is a candidate alignment path that can be collapsed into an "I". For path The symbol in frame t, P () indicates that the symbol for frame t is The probability, T For path Length, This indicates that the product is obtained by independent multiplication along the time axis. Path probability, It is all that can be folded into The set of paths The character sequence label.

[0321] Optionally, the document image enhancement model further includes a character stroke prior module and an image degradation module, and the document image processing device 1 is further used for:

[0322] During the training process of the document image enhancement model, the comprehensive model loss of the initial document image enhancement model is determined by a comprehensive loss calculation formula, and the model parameters of the initial document image enhancement model are adjusted based on the comprehensive model loss to obtain the document image enhancement model.

[0323] The formula for calculating the overall loss satisfies the following equation:

[0324]

[0325] in, The loss of the comprehensive model is... The score matching loss of the scoring network module is... The character recognition loss of the character recognition module. The character shape loss of the character stroke prior module. The stroke loss of the aforementioned character stroke prior module. The character component loss of the aforementioned character stroke prior module. The image degradation loss of the image degradation module is... For frequency domain loss, To perceive loss.

[0326] Optionally, the document image enhancement model includes an image degradation module, and the image enhancement module 12 is specifically used for:

[0327] Character recognition processing is performed on each of the reference enhanced image block data to obtain a predicted character sequence for each of the reference enhanced image block data and a predicted character confidence level corresponding to the predicted character sequence;

[0328] The image degradation module is used to perform image degradation processing on each of the reference enhanced image block data to obtain reference degraded image block data, and the reference back projection error between the reference degraded image block data and the initial image block data is determined.

[0329] Determine the character error evaluation value corresponding to the predicted character sequence;

[0330] Based on the predicted character confidence, the reference back projection error, and the character error evaluation value, the quality assessment process of each reference enhanced image block data is performed to obtain the quality check result of each reference enhanced image block data.

[0331] Based on the quality inspection results, a first reference enhanced image block data of the quality inspection failure type and a second reference enhanced image block data of the quality inspection success type are determined.

[0332] Optionally, the document image processing device 1 is further configured to: during the training process of the image degradation module, acquire a sample ideal document image and a real document image corresponding to the sample ideal document image, and perform image degradation processing on the sample ideal document image using the image degradation calculation formula of the initial image degradation module to obtain the observed degraded document image;

[0333] The image degradation calculation formula satisfies the following formula:

[0334]

[0335] in, For the observed degraded document image, For all degradation parameters in the initial image degradation model, The ideal document image for the sample. Is the factor downsampling, For spatial fuzzy kernel, For convolution operations, For composite noise parameters, This is a low-frequency light generation module. For superimposed paper texture and / or transparent texture; Used to simulate the distortion that occurs in image signals during digital imaging and storage;

[0336] The image degradation loss corresponding to the real document image and the observed degraded document image is determined based on the eighth loss formula. The initial image degradation model is then adjusted based on the degradation loss to obtain the image degradation model. The eighth loss formula satisfies the following formula:

[0337]

[0338] in, The image degradation loss, The actual document image, As weight, Used for evaluation and Image similarity between It is an L1 norm.

[0339] Please refer to Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include one or more of the following components: a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, the memory 120, the input device 130, and the output device 140 can be connected via the bus 150.

[0340] Processor 110 may include one or more processing cores. Processor 110 connects to various parts of the electronic device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by calling data stored in memory 120. Optionally, processor 110 may be implemented using at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). Processor 110 may integrate one or more of a central processing unit (CPU), graphics processing unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 110 and may be implemented separately using a communication chip.

[0341] The memory 120 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 120 may include a non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (e.g., touch function, sound playback function, image playback function, etc.), instructions for implementing the various method embodiments described below, etc. The operating system may be the Android system, including systems deeply developed based on the Android system, the iOS system developed by Apple Inc., including systems deeply developed based on the iOS system, or other systems.

[0342] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to establish data communication between the third-party applications and the operating system. This would allow the operating system to obtain the current scenario information of the third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0343] The input device 130 is used to receive input instructions or data, and includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 140 is used to output instructions or data, and includes, but is not limited to, a display device and a speaker. In one example, the input device 130 and the output device 140 can be combined, and the input device 130 and the output device 140 can be a touch display screen.

[0344] The touch display screen can be designed as a full-screen, curved screen, or irregularly shaped screen. It can also be designed as a combination of a full-screen and a curved screen, or a combination of an irregularly shaped screen and a curved screen; however, this application does not limit the specific design of the touch display screen.

[0345] In addition, those skilled in the art will understand that the structure of the electronic device shown in the above figures does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the electronic device may also include radio frequency circuits, input units, sensors, audio circuits, Wireless Fidelity (WiFi) modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.

[0346] In some embodiments, Figure 9 In the illustrated electronic device, the processor 110 can be used to call a program for a document image processing method stored in the memory 120, and specifically perform the following operations:

[0347] An initial document image is obtained, and the initial document image is processed by a document image enhancement model to obtain data for each initial image block.

[0348] The document image enhancement model is used to perform image enhancement processing on each of the initial image block data to obtain each reference enhanced image block data. From each of the reference enhanced image block data, a first reference enhanced image block data of the quality check failure type and a second reference enhanced image block data of the quality check success type are determined. The first reference enhanced image block data is subjected to local supplement enhancement processing to obtain a third reference enhanced image block data. Based on the second reference enhanced image block data and the third reference enhanced image block data, a sliding window fusion processing is performed to obtain the target enhanced document image.

[0349] The target enhanced document image is processed by the document image enhancement model to obtain the target character sequence corresponding to the initial document image and the target character confidence level corresponding to the target character sequence.

[0350] This application also provides a computer-readable storage medium storing at least one instruction that is executed by a processor to implement the document image processing method as described in the above embodiments.

[0351] This application also provides a computer program product that stores at least one instruction, which is loaded and executed by the processor to implement the document image processing method described in the above embodiments.

[0352] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0353] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method of document image processing, characterized by, The method comprises: obtaining an initial document image, performing image block processing on the initial document image through a document image enhancement model to obtain initial image block data; performing image enhancement processing on each of the initial image block data through the document image enhancement model to obtain reference enhanced image block data, determining first reference enhanced image block data of a quality check failure type and second reference enhanced image block data of a quality check success type from the reference enhanced image block data, performing local supplementary enhancement processing on the first reference enhanced image block data to obtain third reference enhanced image block data, and performing sliding window fusion processing based on the second reference enhanced image block data and the third reference enhanced image block data to obtain a target enhanced document image; performing character recognition processing on the target enhanced document image through the document image enhancement model to obtain a target character sequence corresponding to the initial document image and a target character confidence corresponding to the target character sequence; The document image enhancement model comprises an image degradation module, and the determination of the first reference enhanced image block data of the quality check failure type and the second reference enhanced image block data of the quality check success type from the reference enhanced image block data comprises: performing character recognition processing on each of the reference enhanced image block data to obtain a predicted character sequence for each of the reference enhanced image block data and a predicted character confidence corresponding to the predicted character sequence; performing image degradation processing on each of the reference enhanced image block data using the image degradation module to obtain reference degraded image block data, determining a reference back-projection error between the reference degraded image block data and the initial image block data, determining a character error evaluation value corresponding to the predicted character sequence, performing quality evaluation processing on each of the reference enhanced image block data based on the predicted character confidence, the reference back-projection error and the character error evaluation value to obtain a quality check result of each of the reference enhanced image block data, and determining the first reference enhanced image block data of the quality check failure type and the second reference enhanced image block data of the quality check success type based on the quality check result.

2. The method of claim 1, wherein, The document image enhancement model comprises a residual sampling module, and the image enhancement processing of the initial image block data through the document image enhancement model comprises: determining a current sampling step number corresponding to the residual sampling module and a sampling weight parameter corresponding to the current sampling step number; performing image enhancement processing on each of the initial image block data using the residual sampling module based on the sampling weight parameter to obtain basic enhanced image block data; determining whether the current sampling step number is greater than or equal to a preset sampling step number; If the current sampling step number is less than the preset sampling step number, a next sampling step number corresponding to the current sampling step number is obtained, the next sampling step number is taken as the current sampling step number, the basic enhanced image block data is taken as the initial image block data, and the step of determining a sampling weight parameter corresponding to the current sampling step number is performed, and each initial image block data is subjected to image enhancement processing by the residual sampling module based on the sampling weight parameter to obtain each basic enhanced image block data; If the current sampling step number is greater than or equal to the preset sampling step number, each basic enhanced image block data corresponding to the current sampling step number is taken as each reference enhanced image block data.

3. The method of claim 2, wherein, The document image enhancement model further comprises a score network module and a character recognition module, and the method further comprises: In the model training process of the residual sampling module, first sample enhanced image block data generated by an initial residual sampling module in a kth step sampling process is obtained, and second sample enhanced image block data generated by the initial residual sampling module in a (k+1) th step sampling process is determined based on a preset residual update formula; The preset residual update formula satisfies the following formula: wherein, enhancing image block data for the second sample, enhancing image block data for the first sample, is a step size, is a drift coefficient, is a strength of the scoring network module, is the scoring network module, is a semantic guidance strength weight sampled at the k-th step, is a frequency channel weight sampled at the k-th step, is a loss gradient of the character recognition module, is a frequency domain regular injection control weight sampled at the k-th step, is a frequency domain loss gradient; Third sample enhanced image block data generated by an un-distilled residual sampling module in the (k+1) th step sampling process is obtained, a distillation loss corresponding to the third sample enhanced image block data and the second sample enhanced image block data is determined by using a first loss calculation formula, and the initial residual sampling module is subjected to model parameter adjustment processing based on the distillation loss, the semantic guidance intensity weight, the frequency channel weight and the frequency domain regular injection control weight to obtain the residual sampling module; The first loss calculation formula satisfies the following formula: wherein, is a distillation loss, is an image residual alignment loss, is a weight of, is an intermediate statistical matching loss, is a weight of, is a display character recognition module gradient preservation loss, is a weight of. The second loss calculation formula satisfies the following formula: wherein, is first enhanced image block inference data output by the initial residual sampling module for sample initial image block data, is second enhanced image block inference data output by the un-distilled residual sampling module for the sample initial image block data, is an L1 norm; The third loss calculation formula satisfies the following formula: wherein, is a Fourier transform operator, denotes taking the spectral magnitude, is an image gradient operator, is a histogram construction operator, is a KL divergence for measuring the difference in gradient distribution, is a weight, is a weight, is an LI norm; The fourth loss calculation formula is obtained by the following formula: wherein, denotes and the maximum mean discrepancy, denotes the character recognition module, denotes the loss gradient of the character recognition module with respect to the loss gradient of the character recognition module with respect to the loss gradient of the character recognition module with respect to the loss gradient of the character recognition module with respect to 4. The method of claim 3, wherein, The method further comprises: In the training process of the score network module, sample noise document images and noise data distribution gradient labels corresponding to the sample noise document images are obtained, a predicted noise data distribution gradient corresponding to the sample noise document images is recognized by using an initial score network module, a score matching loss corresponding to the predicted noise data distribution gradient and the noise data distribution gradient labels is determined by using a fifth loss calculation formula, and the initial score network module is subjected to model parameter adjustment processing based on the score matching loss to obtain the score network module; The fifth loss calculation formula satisfies the following formula: wherein, is the score matching loss, is a weight, is obtained by a sixth loss calculation formula, is obtained by a seventh loss calculation formula; The sixth loss calculation formula satisfies the following formula: wherein, is the sample noisy document image, is the noise-free document image corresponding to the sample noisy document image, is obtained by adding a noise sample to is the noise-free document image corresponding to the sample noisy document image, is the noise data distribution gradient label, denotes the probability distribution of the noise-free image given that the noisy image is observed at time is the probability distribution of the noise-free image given that the noisy image is observed at time is the noise sample, is the initial score network module, is the input of the initial score network module, and , is the output of the predicted noise data distribution gradient, denotes a weighting function related to the time step , denotes a square operation of the L2 norm, denotes an expectation over all possible . The seventh loss calculation formula satisfies the following formula: wherein, is a Fourier transform operator, denotes a high frequency mask, denotes an element-wise multiplication operation, denotes an LI norm, is a weight, is a set of JPEG ring band frequency indices, extracts the k-th frequency component from the Fourier transformed image, denotes a sample enhanced document image.

5. The method of claim 3, wherein, The method further comprises: In the training process of the character recognition module, sample enhanced document images and character sequence labels corresponding to the sample enhanced document images are obtained, predicted character data are obtained by performing character recognition processing on the sample enhanced document images based on an initial character recognition module, a character recognition loss corresponding to the predicted character data and the character sequence labels is determined by using an identification loss calculation formula, and the initial character recognition module is subjected to model parameter adjustment processing based on the character recognition loss to obtain the character recognition module; The identification loss calculation formula satisfies the following formula: wherein, is the character recognition loss, is the continuous time classification loss, is the attention loss, is the differentiable edit distance loss, is obtained by recursively applying the differentiable edit distance recursive formula, , , are weights for , , , respectively. The recognition loss sub-computation is obtained by identifying a loss sub-computation that satisfies the following equation: wherein, enhancing the document image for the sample, is a candidate alignment path that can be folded into I, is a path is a symbol at the t-th frame, P () represents the probability that the symbol at the t-th frame is , T is a path is the length of the path represents the path probability along the time axis independently multiplied to , is a set of paths that can be folded into , is the character sequence label.

6. The method of claim 3, wherein, The document image enhancement model further comprises a glyph stroke prior module and an image degradation module, and the method further comprises: In the training process of the document image enhancement model, a comprehensive loss calculation formula is used to determine a comprehensive model loss of an initial document image enhancement model, and a model parameter adjustment process is performed on the initial document image enhancement model based on the comprehensive model loss to obtain the document enhancement image model. The comprehensive loss calculation formula satisfies the following formula: wherein, is the score matching loss of the score network module, is the score matching loss of the score network module, is the character recognition loss of the character recognition module, is the character loss of the character part prior module, is the stroke loss of the character part prior module, is the character loss of the character part prior module, is the image degradation loss of the image degradation module, is the frequency domain loss, is the perceptual loss.

7. The method of claim 1, wherein, The method further includes: In the training process of the image degradation module, a sample ideal document image and a real document image corresponding to the sample ideal document image are obtained, an image degradation calculation formula of an initial image degradation module is used to perform image degradation processing on the sample ideal document image to obtain an observed degraded document image; The image degradation calculation formula satisfies the following formula: in, For the observed degraded document image, For all degradation parameters in the initial image degradation model, The ideal document image for the sample. Is the factor downsampling, For spatial fuzzy kernel, For convolution operations, For composite noise parameters, This is a low-frequency light generation module. For superimposed paper texture and / or transparent texture; Used to simulate the distortion that occurs in image signals during digital imaging and storage; An image degradation loss corresponding to the real document image and the observed degraded document image is determined based on an eighth loss calculation formula, and a model parameter adjustment process is performed on the initial image degradation model based on the degradation loss to obtain the image degradation model, and the eighth loss calculation formula satisfies the following formula: wherein, is the image degradation loss, is the real document image, is a weight, for assessing the image similarity between and is the L1 norm.

8. A computer storage medium, characterized in that The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded and executed by the processor to perform the method of any one of claims 1-7.

9. An electronic device, comprising: Including: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded and executed by the processor to perform the method of any one of claims 1-7.

Images