A text image repair method, device and storage medium

Abstract

The present invention discloses a text image repair method and device thereof, and belongs to the technical field of image text recognition. The method comprises a distribution difference parameter acquisition step, a reconstruction loss parameter acquisition step, and an OCR loss parameter acquisition step; the distribution difference parameter acquisition step, the reconstruction loss parameter acquisition step, and the OCR loss parameter acquisition step are repeated until the total loss function constructed by the distribution difference parameter, the reconstruction loss parameter, and the OCR loss parameter converges; the text image to be repaired is input into the image repair model to reconstruct the text image, and a repaired text image is obtained. The present invention improves the recognition accuracy of OCR by repairing the image. In addition, the repaired image can be recognized by humans, and the result has stronger visualization and interpretability.

Description

Technical Field

[0001] The present invention belongs to the technical field of image text recognition, and in particular relates to a text image repair method and device, and a computer-readable storage medium. Background Art

[0002] Paper documents such as contracts and invoices will wear out during storage, or character pixels in the obtained text images will be missing due to lighting, occlusion, etc. during shooting or scanning, making the text difficult to recognize, thereby reducing the recognition accuracy of OCR (Optical Character Recognition).

[0003] There are currently many image restoration algorithms, but most of these image restoration algorithms are aimed at natural images, such as photos, and there is no special application research on text images. As a result, when conventional image restoration algorithms are used to repair text images, there will be errors in the restoration results, such as 1. Different semantics: the restoration becomes other words; 2. Pseudo-Chinese characters: the restoration results look like Chinese characters, but they are not meaningful Chinese characters. Summary of the invention

[0004] The purpose of the present invention is to provide a text image repair method and device, and a storage medium, which can take into account both image quality and semantics during repair, thereby improving machine readability and further improving the accuracy of character recognition in text images.

[0005] In order to solve the above-mentioned technical problems, the present invention specifically adopts the following technical solutions:

[0006] A first aspect of the present invention provides a text image repair method, comprising:

[0007] Distribution difference parameter acquisition step: input the defective image sample and the original image sample corresponding to the defective image sample into the encoder of the image restoration model to obtain the source domain covariance matrix of the original image sample and the target domain covariance matrix of the defective image sample; calculate the distribution difference parameter LossDA using the source domain covariance matrix and the target domain covariance matrix, and optimize the image restoration model using the distribution difference parameter LossDA;

[0008] Reconstruction loss parameter acquisition steps: input the defective image sample into the image restoration model to obtain the reconstructed text image, and calculate the reconstruction loss parameter LossREC; use the reconstruction loss parameter LossREC to optimize the image restoration model;

[0009] OCR loss parameter acquisition steps: input the reconstructed text image into the OCR model for text recognition, and calculate the OCR loss parameter LossOCR; use the OCR loss parameter LossOCR to optimize the image restoration model;

[0010] Repeat the steps of obtaining the distribution difference parameter, obtaining the reconstruction loss parameter, and obtaining the OCR loss parameter until the total loss function constructed by the distribution difference parameter, the reconstruction loss parameter, and the OCR loss parameter converges;

[0011] The text image to be repaired is input into the image repair model to reconstruct the text image to obtain a repaired text image.

[0012] As an improvement, the method for obtaining the distribution difference parameter LossDA includes:

[0013] Using the formula: Calculate the distribution difference parameter; where LossDA is the distribution difference parameter, F is the Frobenius norm, d is the feature dimension, CovS is the source domain covariance matrix, and CovT is the target domain covariance matrix.

[0014] As an improvement, the method for obtaining the reconstruction loss parameter includes:

[0015] Using the formula: Calculate the reconstruction loss parameter; where LossREC is the reconstruction loss parameter, y i is the true value, is the predicted value, n is the number of samples, and M is the mask.

[0016] As an improvement, the method for obtaining the OCR loss parameter includes:

[0017] The OCR loss function is calculated using the formula: LossOCR = L(RealText, PredictedText); where LossOCR is the OCR loss parameter, L is the loss function used in OCR model training, RealText is the real text, and PredictedText is the predicted text.

[0018] As an improvement, the method of constructing the total loss function includes using the formula

[0019] LossTOTAL=α×LossOCR+β×LossDA+γ×LossREC

[0020] Construct the total loss parameter; where LossTOTAl is the total loss parameter, LossOCR is the OCR loss parameter, LossDA is the distribution difference parameter, LossREC is the reconstruction loss function, and α, β, and γ are the preset weight coefficients.

[0021] A second aspect of the present invention is to provide a text image restoration system, which is used to implement the above-mentioned text image restoration method, comprising:

[0022] A distribution difference parameter acquisition module is used to input the defective image sample and the original image sample corresponding to the defective image sample into the encoder of the image restoration model to obtain the source domain covariance matrix of the original image sample and the target domain covariance matrix of the defective image sample; the distribution difference parameter LossDA is calculated using the source domain covariance matrix and the target domain covariance matrix, and the image restoration model is optimized using the distribution difference parameter LossDA;

[0023] The reconstruction loss parameter acquisition module is used to input the defective image sample into the image restoration model to obtain the reconstructed text image and calculate the reconstruction loss parameter LossREC; the image restoration model is optimized using the reconstruction loss parameter LossREC;

[0024] The OCR loss parameter acquisition module is used to input the reconstructed text image into the OCR model for text recognition and calculate the OCR loss parameter LossOCR; the OCR loss parameter LossOCR is used to optimize the image restoration model;

[0025] A total loss function convergence module, used to repeat the distribution difference parameter acquisition step, the reconstruction loss parameter acquisition step, and the OCR loss parameter acquisition step until the total loss function constructed by the distribution difference parameter, the reconstruction loss parameter, and the OCR loss parameter converges;

[0026] The text image restoration module is used to input the text image to be restored into the image restoration model to perform text image reconstruction to obtain a restored text image.

[0027] The present invention also provides a computer program, which can implement the above-mentioned text image repair method when being executed.

[0028] A third aspect of the present invention is to provide a storage device, wherein a computer program is stored in the storage device, and when the computer program is executed, the above-mentioned text image repair method can be implemented.

[0029] A fourth aspect of the present invention is to provide a computer system, comprising a memory and a processor; a computer program is stored in the memory, and when the computer program is executed by the processor, the above-mentioned text image repair method can be implemented.

[0030] The benefit of the present invention lies in that: the present invention optimizes the image restoration model by introducing the distribution difference parameter LossDA as a constraint condition to find a feature space that can well present both visual features and semantic features to achieve collaborative optimization, thereby aligning the semantic features with the visual features and reducing the generation of artifacts by combining domain adaptation technology.

[0031] The purpose of the present invention is to align the semantic features and visual features of the defective image sample with the original image sample by a domain adaptive method. Due to the lack of information in the defective image, the distribution of the features extracted by the encoder will be inconsistent with the feature distribution of the original image. However, simply by reconstructing the loss parameter LossREC for training, the model does not recover the text recognition features (or semantic recognition) well. After adding the OCR loss parameter LossOCR, due to the inconsistency between text recognition and visual features, a conflict between LossREC and LossOCR occurs, which will cause the problem of losing one while gaining the other, resulting in the invalidity of LosOCR (i.e., the semantic information is not restored, or the accuracy is not high), or the invalidity of LosREC (i.e., the visual information is not restored or the accuracy is not high), thereby reducing the readability of the machine. That is to say, in order to improve machine readability, it is necessary to overcome the conflict between the two. The present invention adjusts the distribution difference parameter LossDA in combination with domain adaptation to effectively balance image quality and OCR performance, that is, by finding a common feature space that integrates visual quality and OCR recognition features, and the image reconstructed based on the common feature space meets the optimization goals and standards, so as to eliminate the typical conflicts found in multi-objective optimization tasks, thereby improving machine readability and further improving the accuracy of character recognition in text images. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, each element or part is not necessarily drawn according to the actual scale. Obviously, the drawings described below are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without paying creative labor.

[0033] Figure 1 is a flowchart of a text image repair method according to an exemplary embodiment of the present invention;

[0034] Figure 2 It is a principle diagram of the text image restoration method of the present invention;

[0035] Figure 3 This is a comparison diagram of the effects of the present invention and other models;

[0036] Figure 4 It is a structural principle diagram of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0038] Herein, suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present invention, and have no specific meanings by themselves. Therefore, "module", "component" or "unit" can be used mixedly.

[0039] In this document, the terms "upper", "lower", "inner", "outer", "front", "back", "one end", "the other end" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0040] In this document, unless otherwise clearly specified and limited, the terms "installed", "provided with", "connected", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0041] Herein, "and / or" includes any and all combinations of one or more of the listed related items. Herein, "plurality" means two or more, that is, it includes two, three, four, five, etc.

[0042] In this article, "visual features" refer to the comprehensive information of the original text image, including text semantics, text background, font type, size, color, etc. It reflects the complete and comprehensive information of the original document.

[0043] In this article, "semantic features" refer to: carriers that can express and be received and carry specific meaning information. For example, a text skeleton is an information carrier that can be read and understood by humans. In the process of OCR model recognition, the features that can help the OCR model to correctly recognize text are semantic features. Unlike skeletons, data-driven models such as OCR are difficult to interpret as black box models, and there is no explicit representation for such semantic features. In this technology, semantic features are not limited to OCR models that use model skeleton information or key point information as the basis for text recognition.

[0044] Existing text image restoration models can use reconstruction loss functions or OCR loss functions to optimize the models. Among them, using reconstruction loss functions such as MSE (Mean Squared Error) loss function as the main loss function (loss parameter) to train image restoration models is a common method. This method uses mean square error as the optimization target to minimize pixel difference, so that the pixel error of the result generated by the image restoration model is minimized, thereby ensuring the visual quality of the image. However, this training method has defects for target restoration such as documents, and the generated / restored images may be pseudo-text or other text.

[0045] Using the OCR loss function can improve the recognition rate of the repaired document image in OCR (Optical Character Recognition) and improve the machine recognition rate of text. However, its disadvantage is that it is easy to cause artifacts, thus affecting the image quality. The discriminant features of machine-recognized text may be inconsistent with the discriminant features of human text, which leads to the generation of artifacts. This artifact can improve OCR's recognition of text, but it does not conform to the judgment of the human eye, that is, it reduces the image quality, for example, causing many noise points to appear in the obtained image.

[0046] That is, there is a conflict between the OCR loss function and the MSE loss function, which makes it difficult to train the text image restoration model. There may be performance deviations and inclinations between semantic recognition performance and image quality. For example, although the image quality is improved, pseudo-text appears; or, although the accuracy of text recognition is improved, the image quality is reduced.

[0047] In order to solve the above technical problems, Figure 1 As shown, the present invention provides a text image repair method, and its specific steps include:

[0048] S1. Distribution difference parameter acquisition step: input the defective image sample and the original image sample corresponding to the defective image sample into the encoder of the image restoration model to obtain the source domain covariance matrix of the original image sample and the target domain covariance matrix of the defective image sample; use the source domain covariance matrix and the target domain covariance matrix to calculate the distribution difference parameter LossDA, and use the distribution difference parameter LossDA to optimize the image restoration model.

[0049] The image restoration model in the present invention includes an encoder and a decoder (the image restoration model itself is actually not the object of the present invention, and models such as CNN and Uformer can be selected and are not limited to). When training the model, a sample image pair is input to the model, that is, a defective image sample and an original image sample corresponding to the defective image sample. The original image sample and the defective image sample are encoded by the encoder, and two corresponding feature matrices (the feature matrix includes visual features and semantic features) are output, namely the source domain covariance matrix and the target domain covariance matrix.

[0050] In the present invention, see Figure 2 , the difference between the source domain covariance matrix and the target domain covariance matrix is ​​calculated through the distribution difference parameter LossDA. Then the distribution difference parameter LossDA is used as a constraint to align the target domain covariance matrix with the source domain covariance matrix, that is, by minimizing the distribution difference parameter LossDA, the distribution difference of encoder feature extraction is solved.

[0051] The difference between the target domain covariance matrix and the source domain covariance matrix can be regarded as the missing part of the defective image sample compared with the original image sample. During the alignment process, the feature mapping from the defective image sample to the original image sample is completed, which helps the model to train and makes the model encoder more robust to feature extraction.

[0052] More specifically, the method for obtaining the distribution difference parameter LossDA in the present invention includes:

[0053] Using the formula: Calculate the distribution difference parameter; where LossDA is the distribution difference parameter, F is the Frobenius norm, d is the feature dimension, that is, the dimension of the output feature of the hidden layer of the neural network, CovS is the source domain covariance matrix, and CovT is the target domain covariance matrix.

[0054] S2, reconstruction loss parameter acquisition step: input the defective image sample into the image restoration model to obtain the reconstructed text image, and calculate the reconstruction loss parameter LossREC; use the reconstruction loss parameter LossREC to optimize the image restoration model.

[0055] Reconstruction loss parameters are often used to optimize image inpainting models. Image inpainting models calculate the pixel-by-pixel distance between the predicted image and the original image to help the model learn and generate images that follow the distribution of the original image. Reconstruction loss parameters focus on graphical features or visual features.

[0056] In this implementation, see Figure 2 When the defective image sample in the image sample pair is input into the above-mentioned image restoration model, a reconstructed text image will be obtained. The reconstruction loss function is calculated using the reconstructed text image and the original image sample in the image sample pair. Specifically, the method for obtaining the reconstruction loss parameter includes:

[0057] Using the formula: Calculate the reconstruction loss parameter; where LossREC is the reconstruction loss parameter, y i is the true value, is the prediction value, n is the number of samples, and M is the mask, marking the degraded area of ​​the image. Here, the degraded area is 1 and the non-degraded area is 0. In this way, the model only calculates the prediction value error of the damaged area, so that the model pays more attention to the damaged area of ​​the image.

[0058] After obtaining the reconstruction loss parameter LossREC, continue to use it to optimize the above image restoration model.

[0059] S3, OCR loss parameter acquisition step: input the reconstructed text image into the OCR model for text recognition, and calculate the OCR loss parameter LossOCR; use the OCR loss parameter LossOCR to optimize the image restoration model.

[0060] The OCR model in the present invention is pre-trained and in a frozen state, that is, the OCR model is no longer optimized. The reconstructed text image output by the image restoration model is input into the OCR model for text recognition, and the OCR loss parameter LossOCR is calculated using the result of text recognition, i.e., the predicted text and the real text (i.e., the original image). The OCR loss parameter focuses on text recognition.

[0061] Specifically, the method for obtaining the OCR loss parameter includes:

[0062] The OCR loss function is calculated using the formula: LossOCR = L(RealText, PredictedText); where LossOCR is the OCR loss parameter, L is the loss function used in OCR model training, RealText is the real text, and PredictedText is the predicted text.

[0063] After calculating the OCR loss function, it is used to optimize the above image restoration model.

[0064] S4. Repeat the distribution difference parameter acquisition step, the reconstruction loss parameter acquisition step, and the OCR loss parameter acquisition step until the total loss function constructed by the distribution difference parameter, the reconstruction loss parameter, and the OCR loss parameter converges.

[0065] The total loss parameter is a balanced fusion of the individual loss parameters including the distribution difference parameter LossDA, the reconstruction loss parameter LossREC and the OCR loss parameter LossOCR.

[0066] In the present invention, the total loss function is:

[0067] LossTOTAl=α×LossOCR+β×LossDA+γ×LossREC; where LossTOTAl is the total loss parameter, LossOCR is the OCR loss parameter, LossDA is the distribution difference parameter, LossREC is the reconstruction loss function, and α, β, and γ are weight coefficients to balance the effect of each loss parameter.

[0068] By repeatedly executing the distribution difference parameter acquisition step, the reconstruction loss parameter acquisition step, and the OCR loss parameter acquisition step, the total loss function converges. Convergence means that the function gradually approaches a certain value. When the independent variable is infinitely close to a given value, the output of the function is also close to a specific value. In other words, function convergence means that the function is infinitely close to a certain limit value or stable state.

[0069] After the total loss function converges, the above image restoration model is considered to have been trained.

[0070] The model optimization of the present invention introduces distribution difference parameters on the basis of the existing reconstruction loss function and OCR loss function, and realizes collaborative optimization by finding a feature space that meets visual requirements and semantic requirements, that is, aligning semantic features with visual features in the reconstructed image, and reducing artifacts generated by optimizing only visual features and pseudo-words generated by optimizing only semantic features by combining domain adaptation technology, that is, by minimizing the feature distribution of domain adaptation technology, the feature domain of the damaged image and the feature of the complete image are helped to align, improve information recovery, and thus improve the quality of the final repaired image. Domain adaptation refers to adapting a model or algorithm from one domain to another very different domain, and strengthening the generalization performance of the model by reducing data distribution or feature distribution, so as to achieve better performance when testing or applying in a domain different from the training data. That is, for problems in different domains of the training set and the test set, the model is adapted from the training set domain to the test domain through some certain technologies and methods, thereby improving the generalization ability and performance of the model.

[0071] Prior art Chinese patent application CN115050032A discloses a domain adaptation text image recognition method based on feature alignment and entropy regularization, the method comprising: converting an input text line image into an implicit representation feature sequence through an encoder, converting and outputting an initial predicted text sequence and a predicted probability of each character in the predicted text sequence through a decoder, calculating the source domain cross entropy loss using the decoder's conversion output and the source domain sample text label, and minimizing the solution to obtain an initial recognition model; inputting text line images with text labels in the source domain and text line images without text labels in the target domain into the initial recognition model, calculating the total loss function of domain adaptation learning, and minimizing the solution to obtain a final recognition model; inputting the target domain text line image to be tested into the final recognition model, and obtaining a final predicted text sequence output by the decoder.

[0072] Although the above-mentioned prior art also involves solving the problem of domain transfer through domain adaptation technology, it mainly focuses on the problem of inconsistent data domains in the use scenarios of training data and test data, which will lead to a decrease in OCR recognition performance. The present invention uses domain adaptation during the training process, allowing the OCR model to perform text recognition by utilizing the common features of training data and target data.

[0073] Although the present invention also adopts domain adaptation, the purpose of the present invention is to align the semantic features and visual features of the defective image sample with the original image sample through the method of domain adaptation. Due to the lack of information in the defective image, the distribution of the features extracted by the encoder will be inconsistent with the feature distribution of the original image. However, simply training through the reconstruction loss parameter lossREC, the model does not recover the text recognition features well. After adding the OCR loss parameter lossOCR, due to the inconsistency between text recognition and visual features, a conflict between lossREC and lossOCR occurs, and finally lossOCR is invalid (semantic information is not restored), or lossREC is invalid (visual information is not restored). In order to overcome the conflict, the present invention adjusts the distribution difference parameter lossDA in combination with domain adaptation.

[0074] In addition, the present invention adopts an anchor-free domain adaptation strategy, that is, no source domain or target domain sample label information is used (there is no need to add corresponding text labels to the original image and the damaged image in advance as in the existing training process, which greatly reduces the workload) and aims to adapt between domains by only using the feature distribution difference between the source domain and the target domain. In the prior art, domain adaptation is usually performed by data reconstruction to improve the performance of downstream reconstruction tasks. Although this can ensure the integrity of information in the model, thereby improving the model's ability to repair damaged images, the reconstruction task will anchor the feature space, thereby limiting the generalization of the model's feature domain and reducing the possibility of enhancing the data domain, that is, anchoring a feature space, so that the image output by the model may only meet the visual standard. The original image is not optimized for OCR recognition, so there are differences in the features related to OCR recognition because it belongs to a different field from the image used in the OCR model training. Therefore, the original image cannot access the optimized feature space. The model is freed from the need to reconstruct the original image, so that it can tend to the feature space defined by OCR, thereby enhancing the recognition performance of the generated image (including the original image).

[0075] S5. Input the text image to be repaired into the image repair model to reconstruct the text image to obtain a repaired text image.

[0076] The effectiveness of the present invention is verified by comparison with other models below. The goal of the present invention is to evaluate the impact and performance of various methods, each of which has a different loss function setting, representing their respective optimization goals. Three methods were considered: the baseline method (i.e., only using LossREC for optimization, i.e., control group 1), focusing on visual quality optimization; the OCR enhancement method (i.e., using LossREC and LossOCR for optimization, i.e., control group 2), targeting both machine readability and visual quality; the method provided by the present invention (i.e., the experimental group) aims to prevent the shortcomings of the OCR enhancement method, i.e., the decline in visual quality, while improving machine readability. Specifically, referring to Table 1 below, damaged images with different masking area ratios are input into the three methods respectively, and the corresponding repair results are obtained respectively. By comparison, it can be seen that the output of the model optimized by the present invention balances the image (especially single-word image) output and semantic recognizability compared with the above two models, greatly increasing the machine readability.

[0077] Table 1 Comparison of partially repaired text images obtained by the experimental group and control group 1 and control group 2

[0078]

[0079] As shown in Table 1 above, the method of the experimental group of the present invention improves the machine readability: in all masking ratios, the ACC value of the experimental group is greater than that of the control group 1 and the control group II, and, compared with the undamaged original image (i.e., the masking ratio is 0%), the ACC also increases by about 1%.

[0080] (93.44%-92.47%=0.97%).

[0081] As shown in Table 1 above, compared with the control group 1, although the peak signal-to-noise ratio SSIM has decreased (for example, compared with the undamaged original image (i.e., the masking ratio is 0%), it has decreased from 1 to 0.07), but it is negligible. This is because the overall reduction of pixel values ​​will cause an unmeasurable decrease in the peak signal-to-noise ratio PSNR. In addition, see Figure 3 It can be seen that the images in the experimental group are darker than the original images, but this does not mean that the degradation of visual quality will lead to the loss of key information to reduce machine readability.

[0082] As shown in Table 1, when the masked area ratio is greater than 0%, the experimental group enhances the robustness of the model, and as the masked area ratio increases, the improvement of ACC becomes more obvious. In addition, compared with experimental group 2, the experimental group effectively reduces the noise in the repaired image in control group 2, such as Figure 3 As shown, the gap in image quality is narrowing.

[0083] Specifically, in the process of image restoration using the method of this embodiment, the weight coefficients in the function are: α = 0.01, β = 0.1, γ = 1. Of course, the weight coefficients can be adjusted according to actual conditions.

[0084] like Figure 4 As shown, the present invention also provides a text image repair device, which is used to implement the above text image repair method, including

[0085] A distribution difference parameter acquisition module is used to input the defective image sample and the original image sample corresponding to the defective image sample into the encoder of the image restoration model to obtain the source domain covariance matrix of the original image sample and the target domain covariance matrix of the defective image sample; the distribution difference parameter LossDA is calculated using the source domain covariance matrix and the target domain covariance matrix, and the image restoration model is optimized using the distribution difference parameter LossDA;

[0086] The reconstruction loss parameter acquisition module is used to input the defective image sample into the image restoration model to obtain the reconstructed text image and calculate the reconstruction loss parameter LossREC; the image restoration model is optimized using the reconstruction loss parameter LossREC;

[0087] The OCR loss parameter acquisition module is used to input the reconstructed text image into the OCR model for text recognition and calculate the OCR loss parameter LossOCR; the OCR loss parameter LossOCR is used to optimize the image restoration model;

[0088] A total loss function convergence module, used to repeat the distribution difference parameter acquisition step, the reconstruction loss parameter acquisition step, and the OCR loss parameter acquisition step until the total loss function constructed by the distribution difference parameter, the reconstruction loss parameter, and the OCR loss parameter converges;

[0089] The text image restoration module is used to input the text image to be restored into the image restoration model to perform text image reconstruction to obtain a restored text image.

[0090] The present invention also provides a computer program, which can implement the above-mentioned text image repair method when being executed.

[0091] The present invention also provides a storage device, in which a computer program is stored. When the computer program is executed, the above-mentioned text image repair method can be implemented.

[0092] The present invention also provides a computer system, comprising a memory and a processor; a computer program is stored in the memory, and when the computer program is executed by the processor, the above-mentioned text image repair method can be implemented.

[0093] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0094] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a computer terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0095] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A text image restoration method, characterized in that: include: Distribution difference parameter acquisition step: input the defective image sample and its corresponding original image sample into the encoder of the image restoration model to obtain the source domain covariance matrix CovS of the original image sample and the target domain covariance matrix CovT of the defective image sample; calculate the distribution difference parameter LossDA based on the source domain covariance matrix and the target domain covariance matrix, and use the distribution difference parameter LossDA to optimize the image restoration model; Reconstruction loss parameter acquisition step: inputting the defective image sample into the image restoration model to obtain the reconstructed text image, and calculating the reconstruction loss parameter LossREC; optimizing the image restoration model based on the reconstruction loss parameter LossREC; OCR loss parameter acquisition step: inputting the reconstructed text image obtained by the image restoration model into a pre-trained and frozen OCR model for text recognition, and calculating the OCR loss parameter LossOCR; optimizing the image restoration model based on the OCR loss parameter LossOCR; Repeat the distribution difference parameter acquisition step, the reconstruction loss parameter acquisition step, and the OCR loss parameter acquisition step until a total loss function constructed based on the distribution difference parameter, the reconstruction loss parameter, and the OCR loss parameter converges to obtain a trained image restoration model; The text image to be repaired is input into the image repair model to reconstruct the text image to obtain a repaired text image.

2. A text image repair method according to claim 1, characterized in that: The method for obtaining the distribution difference parameter LossDA includes: Using the formula: Calculate the distribution difference parameter; where LossDA is the distribution difference parameter, F is the Frobenius norm, d is the feature dimension, CovS is the source domain covariance matrix, and CovT is the target domain covariance matrix.

3. A text image repair method according to claim 1, characterized in that: The method for obtaining the reconstruction loss parameter includes: Using the formula: Calculate the reconstruction loss parameter; where LossREC is the reconstruction loss parameter, y is the true value, is the predicted value and M is the mask.

4. A text image repair method according to claim 1, characterized in that The method for obtaining the OCR loss parameter includes: The OCR loss function is calculated using the formula: LossOCR = L(RealText, PredictedText); where LossOCR is the OCR loss parameter, L is the loss function used in OCR model training, RealText is the real text, and PredictedText is the predicted text.

5. A text image repair method according to claim 1, characterized in that The method of constructing the total loss function includes using the formula: LossTOTAl=α×LossOCR+β×LossDA+γ×LossREC constructs the total loss parameter; among them, LossTOTAl is the total loss parameter, LossOCR is the OCR loss parameter, LossDA is the distribution difference parameter, LossREC is the reconstruction loss function, and α, β, and γ are preset weight coefficients.

6. The text image repair method according to any one of claims 1 to 5, characterized in that: The defective image samples in the target domain have no text labels.

7. The text image repair method according to any one of claims 1 to 5, characterized in that: The original image samples in the source domain have no text labels.

8. A text image repair device, used to implement the text image repair method according to any one of claims 1 to 7, characterized in that: include: A distribution difference parameter acquisition module is used to input the defective image sample and the original image sample corresponding to the defective image sample into the encoder of the image restoration model to obtain the source domain covariance matrix of the original image sample and the target domain covariance matrix of the defective image sample; The source domain covariance matrix and the target domain covariance matrix are used to calculate the distribution difference parameter LossDA, and the image restoration model is optimized using the distribution difference parameter LossDA; The reconstruction loss parameter acquisition module is used to input the defective image sample into the image restoration model to obtain the reconstructed text image and calculate the reconstruction loss parameter: LossREC; the image restoration model is optimized using the reconstruction loss parameter: LossREC; The OCR loss parameter acquisition module is used to input the reconstructed text image into the OCR model for text recognition and calculate the OCR loss parameter LossOCR; the OCR loss parameter LossOCR is used to optimize the image restoration model; A total loss function convergence module, used to repeat the distribution difference parameter acquisition step, the reconstruction loss parameter acquisition step, and the OCR loss parameter acquisition step until the total loss function constructed by the distribution difference parameter, the reconstruction loss parameter, and the OCR loss parameter converges; The text image restoration module is used to input the text image to be restored into the image restoration model to perform text image reconstruction to obtain a restored text image.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed, the text image repair method described in any one of claims 1 to 7 can be implemented.

10. A computer system, characterized in that: It comprises a memory and a processor; the memory stores a computer program, and when the computer program is executed by the processor, the text image repair method described in any one of claims 1 to 7 can be implemented.

Images