Method for extracting document characters and watermarks in watermark document image and related equipment

Abstract

The invention discloses a method for extracting document characters and watermarks in a watermark document image and related equipment. The method comprises the following steps: inputting an original picture into a watermark elimination module in an extraction model to obtain a watermark-eliminated picture; inputting the watermark-eliminated picture into a character restoration module in the extraction model to obtain a document character picture; and meanwhile, inputting the original picture into a watermark extraction module in the extraction model to obtain a shading watermark picture. In the process, shading watermarks are removed through a low-frequency feature repairing model in the watermark eliminating module, and low-frequency features are repaired. On the basis, in order to further recover the text content in the document picture, the watermark-eliminated picture is input into a high-frequency feature repairing model of a text main body in a text repairing module, so that subsequent accurate OCR (Optical Character Recognition) is facilitated. And meanwhile, the high-frequency feature of the watermark main body is repaired by extracting the high-frequency feature repairing model of the watermark main body in the watermark module, which is a diffusion model, so that the subsequent watermark character recognition is facilitated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a method for extracting document text and watermarks from a watermarked document image and related equipment. BACKGROUND

[0002] With the continuous enhancement of copyright protection awareness and information protection awareness, many pictures or files nowadays will add watermarks. For example, in card and form document pictures, watermark and background texture interference phenomena are very common. However, in the process of card OCR recognition, the common underprint watermark often becomes a big problem. Especially when the text watermark appears in the key position of the card (for example Figure 1 The underprint watermark of the "motor vehicle registration certificate" covers multiple key information), or overlaps with the text of the card itself (for example, adding a text watermark on the "ID number" of the ID copy), which will seriously affect the accuracy of OCR recognition, and further adversely affect the downstream intelligent processing tasks such as key information extraction, text mining and speech synthesis. Therefore, removing the underprint watermark interference in the document image is crucial to improve the accuracy of OCR recognition and optimize the performance of downstream intelligent processing tasks.

[0003] At present, some models based on convolutional neural networks or adversarial neural networks usually use regression prediction methods, but the pictures generated by these models often have the problems of blurred text or artifacts, which are difficult to effectively repair the text content. Therefore, the present application proposes a method for extracting text and watermarks from a watermarked document image, which aims to solve the problem of unclear text content in the generated picture to improve the accuracy of subsequent OCR recognition. SUMMARY

[0004] The present application provides a method for extracting document text and watermarks from a watermarked document image and related equipment. After using a low-frequency feature restoration model to restore low-frequency structural features, two diffusion models are used to restore the high-frequency features of the main text and the underprint watermark, respectively. By restoring the main text, the problem of unclear text content in the generated picture can be solved.

[0005] In a first aspect, the present application provides a method for extracting document text and watermarks from a watermarked document image, which comprises:

[0006] obtaining an original picture of a watermarked document, the original picture being a picture of a document with an underprint watermark;

[0007] The original picture is input into a watermark removal module in the extraction model to obtain a watermark removal picture, the watermark removal module comprising a trained low-frequency feature repair model, the low-frequency feature repair model adopting a non-linear activation network, and training data of the watermark removal model being low-frequency feature historical document pictures and low-frequency feature training document pictures;

[0008] The watermark removal picture is input into a text repair module in the extraction model to obtain a document text picture, the document text picture being a picture after the watermark removal picture is repaired with text content, the text repair module comprising a trained high-frequency feature repair model of a text body, the high-frequency feature repair model of the text body being a diffusion model, and training data of the high-frequency feature repair model of the text body being high-frequency feature historical document pictures and high-frequency feature training document pictures;

[0009] The original picture is input into an extraction watermark module in the extraction model to obtain a background watermark picture, the extraction watermark module comprising a trained high-frequency feature repair model of a watermark body, the high-frequency feature repair model of the watermark body being a diffusion model, and training data of the high-frequency feature repair model of the watermark body being high-frequency feature historical background watermark and high-frequency feature training background watermark.

[0010] Optionally, the training process of the extraction model comprises:

[0011] A historical data set is obtained, the historical data set comprising historical document pictures with background watermark, low-frequency feature historical document pictures, high-frequency feature historical document pictures and high-frequency feature historical background watermark;

[0012] A loss function LL of the low-frequency feature repair model is obtained according to the low-frequency feature historical document pictures;

[0013] A loss function LN of the high-frequency feature repair model of the text body is obtained according to the high-frequency feature historical document pictures;

[0014] A loss function LH of the high-frequency feature repair model of the watermark body is obtained according to the high-frequency feature historical background watermark;

[0015] The high-frequency feature training document pictures and the high-frequency feature training background watermark are combined to generate a reconstructed picture;

[0016] A mean square error of the historical document pictures with background watermark and the reconstructed picture is calculated to obtain a loss function L1;

[0017] The loss function LL, the loss function LN, the loss function LH, the loss function L1 and a loss function L2 of a text recognition model are summed to obtain a total loss, the text recognition model being used to optimize a text body detail in the high-frequency feature training document pictures;

[0018] The learnable parameters in the training extraction model are updated reversely according to the total loss.

[0019] Optionally, the loss function LL of the low-frequency feature restoration model is obtained according to the low-frequency feature historical document picture, and the loss function LL comprises:

[0020] The historical document picture with the background watermark is input into the low-frequency feature restoration model to generate a low-frequency feature training document picture.

[0021] The mean square error of the low-frequency feature training document picture and the low-frequency feature historical document picture is calculated to obtain the loss function LL.

[0022] Optionally, the loss function LN of the high-frequency feature restoration model of the text subject is obtained according to the high-frequency feature historical document picture, and the loss function LN comprises:

[0023] The low-frequency feature training document picture is input into the high-frequency feature restoration model of the text subject to generate a high-frequency feature training document picture.

[0024] The mean square error of the high-frequency feature training document picture and the high-frequency feature historical document picture is calculated to obtain the loss function LN.

[0025] Optionally, the process of using the text recognition model comprises:

[0026] The high-frequency feature training document picture is input into the text recognition model to obtain predicted text.

[0027] The predicted text is compared with the real text in the high-frequency feature historical document picture to optimize the text subject details in the high-frequency feature training document picture.

[0028] Optionally, the loss function LH of the high-frequency feature restoration model of the watermark subject is obtained according to the high-frequency feature historical background watermark, and the loss function LH comprises:

[0029] The historical document picture with the background watermark is input into the high-frequency feature restoration model of the watermark subject to generate the high-frequency feature training background watermark.

[0030] The mean square error of the high-frequency feature training background watermark and the high-frequency feature historical background watermark is calculated to obtain the loss function LH.

[0031] Optionally, the historical data set is obtained, and the historical data set comprises:

[0032] The background watermark and the text watermark are generated respectively, and the background watermark and the text watermark are used to obtain the background watermark.

[0033] The background watermark is added to the clean historical document picture to obtain the historical document picture with the background watermark.

[0034] The clean historical document picture is subjected to low-pass filtering image processing to obtain a low-frequency feature historical document picture;

[0035] The clean historical document picture is subjected to high-pass filtering image processing to obtain a high-frequency feature historical document picture;

[0036] The watermark is subjected to high-pass filtering image processing to obtain a high-frequency feature historical watermark.

[0037] In a second aspect, the application provides a device for extracting document text and watermark from a watermark document image, the device comprising:

[0038] An acquisition unit is configured to acquire an original picture of a watermark document, the original picture being a picture of a watermark document with a watermark;

[0039] A watermark elimination unit is configured to input the original picture into a watermark elimination module in an extraction model to obtain a watermark-eliminated picture, the watermark elimination module comprising a trained low-frequency feature repair model, the low-frequency feature repair model adopting a non-linear activation network, and training data of the watermark elimination model being a low-frequency feature historical document picture and a low-frequency feature training document picture;

[0040] A text repair unit is configured to input the watermark-eliminated picture into a text repair module in the extraction model to obtain a document text picture, the document text picture being a picture of the watermark-eliminated picture after repair of text content, the text repair module comprising a trained high-frequency feature repair model of a text body, the high-frequency feature repair model of the text body being a diffusion model, and training data of the high-frequency feature repair model of the text body being a high-frequency feature historical document picture and a high-frequency feature training document picture;

[0041] An extraction watermark unit is configured to input the original picture into an extraction watermark module in the extraction model to obtain a watermark picture, the extraction watermark module comprising a trained high-frequency feature repair model of a watermark body, the high-frequency feature repair model of the watermark body being a diffusion model, and training data of the high-frequency feature repair model of the watermark body being a high-frequency feature historical watermark and a high-frequency feature training watermark.

[0042] Optionally, the device comprises:

[0043] The acquisition unit is further configured to acquire a historical data set, the historical data set comprising a historical document picture with a watermark, a low-frequency feature historical document picture, a high-frequency feature historical document picture, and a high-frequency feature historical watermark;

[0044] The obtaining unit is configured to obtain a loss function LL of the low-frequency feature repair model according to the low-frequency feature historical document picture;

[0045] The obtaining unit is further configured to obtain a loss function LN of the high-frequency feature restoration model of the text body according to the high-frequency feature historical document picture;

[0046] The obtaining unit is further configured to obtain a loss function LH of the high-frequency feature restoration model of the watermark body according to the high-frequency feature historical background watermark;

[0047] The synthesizing unit is configured to synthesize the high-frequency feature training document picture and the high-frequency feature training background watermark to generate a reconstructed picture;

[0048] The obtaining unit is further configured to obtain a loss function L1 by calculating a mean square error of the historical document picture with the background watermark and the reconstructed picture;

[0049] The obtaining unit is further configured to obtain a total loss by summing the loss function LL, the loss function LN, the loss function LH, the loss function L1, and a loss function L2 of a text recognition model, the text recognition model being configured to optimize details of the text body in the high-frequency feature training document picture;

[0050] The training unit is configured to reversely update learnable parameters in the training extraction model according to the total loss.

[0051] Optionally, when the obtaining unit is configured to obtain the loss function LL of the low-frequency feature restoration model according to the low-frequency feature historical document picture, the obtaining unit is specifically configured to:

[0052] input the historical document picture with the background watermark into the low-frequency feature restoration model to generate a low-frequency feature training document picture;

[0053] obtain the loss function LL by calculating a mean square error of the low-frequency feature training document picture and the low-frequency feature historical document picture.

[0054] Optionally, when the obtaining unit is configured to obtain the loss function LN of the high-frequency feature restoration model of the text body according to the high-frequency feature historical document picture, the obtaining unit is specifically configured to:

[0055] input the low-frequency feature training document picture into the high-frequency feature restoration model of the text body to generate a high-frequency feature training document picture;

[0056] obtain the loss function LN by calculating a mean square error of the high-frequency feature training document picture and the high-frequency feature historical document picture.

[0057] Optionally, the apparatus further comprises:

[0058] The obtaining unit is further configured to input the high-frequency feature training document picture into the text recognition model to obtain predicted text;

[0059] The optimization unit is configured to compare the predicted text with the real text in the high-frequency feature historical document picture to optimize the text main body details in the high-frequency feature training document picture.

[0060] Optionally, the obtaining unit is configured to obtain the loss function LH of the high-frequency feature restoration model of the watermark main body according to the high-frequency feature historical watermark, and specifically configured to:

[0061] input the historical document picture with the watermark into the high-frequency feature restoration model of the watermark main body to generate the high-frequency feature training watermark;

[0062] calculate the mean square error of the high-frequency feature training watermark and the high-frequency feature historical watermark to obtain the loss function LH.

[0063] Optionally, the obtaining unit is configured to obtain the historical data set, and specifically configured to:

[0064] generate the watermark and the text respectively, and obtain the watermark according to the watermark and the text;

[0065] add the watermark on the clean historical document picture to obtain the historical document picture with the watermark;

[0066] perform low-pass filtering image processing on the clean historical document picture to obtain the low-frequency feature historical document picture;

[0067] perform high-pass filtering image processing on the clean historical document picture to obtain the high-frequency feature historical document picture;

[0068] perform high-pass filtering image processing on the watermark to obtain the high-frequency feature historical watermark.

[0069] In a third aspect, the present application provides an electronic device, which comprises a memory and a processor:

[0070] The memory is configured to store a computer program;

[0071] The processor is configured to execute the method provided in the first aspect according to the computer program.

[0072] In a fourth aspect, the present application provides a computer readable storage medium, which is configured to store a computer program, and the computer program is configured to execute the method provided in the first aspect.

[0073] Therefore, the present application has the following beneficial effects:

[0074] The application provides a method for extracting document text and watermark of a watermark document image. First, an original picture of a watermark document is obtained, which is a picture of the watermark document with a background watermark; the original picture is input into a watermark elimination module in an extraction model to obtain a watermark-eliminated picture, the watermark elimination module comprising a trained low-frequency feature repair model, and the low-frequency feature repair model adopts a nonlinear activation network; then, the watermark-eliminated picture is input into a text repair module in the extraction model to obtain a document text picture, which is a picture of the watermark-eliminated picture after the text content is repaired, and the text repair module comprises a trained high-frequency feature repair model of a text body, and the high-frequency feature repair model of the text body is a diffusion model; at the same time, the original picture is input into an watermark extraction module in the extraction model to obtain a background watermark picture, and the watermark extraction module comprises a trained high-frequency feature repair model of a watermark body, and the high-frequency feature repair model of the watermark body is a diffusion model. In this process, the low-frequency feature repair model removes the background watermark and repairs the low-frequency features, restores the structural content of the original picture, and obtains the watermark-eliminated picture. On this basis, in order to further restore the text content in the document picture, the watermark-eliminated picture is input into the diffusion model, i.e., the high-frequency feature repair model of the text body, to repair the high-frequency features of the text body and restore the text content in the document picture, thereby obtaining the document text picture, which is helpful for subsequent accurate OCR recognition. At the same time, the high-frequency feature repair model of the watermark body is used to repair the high-frequency features of the watermark body and restore the text content of the watermark, which is helpful for subsequent watermark text recognition. BRIEF DESCRIPTION OF DRAWINGS

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0076] Figure 1 An example diagram of a motor vehicle registration certificate in the embodiments of the present application;

[0077] Figure 2 A flowchart of an embodiment of a method for extracting document text and watermark of a watermark document image in the embodiments of the present application;

[0078] Figure 3 An example diagram of an original picture of a watermark document in the embodiments of the present application;

[0079] Figure 4 An example diagram of a picture after low-frequency feature repair in the embodiments of the present application;

[0080] Figure 5Fig. 1 is an example of a main text picture after high-frequency feature repair in an embodiment of the present application;

[0081] Figure 6 Fig. 2 is an example of a watermark picture after extraction of the underprint in an embodiment of the present application;

[0082] Figure 7 Fig. 3 is a flowchart of another embodiment of a method for extracting document text and watermark from a watermark document image in an embodiment of the present application;

[0083] Figure 8 Fig. 4 is a structural diagram of an apparatus for extracting document text and watermark from a watermark document image in an embodiment of the present application;

[0084] Figure 9 Fig. 5 is a structural diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0085] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art belong to the scope of protection of the present application.

[0086] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0087] Currently, convolutional neural network (CNN) or generative adversarial network (GAN) models are commonly used for processing watermark document pictures. However, the output pictures generated by these models often have fuzzy text and artifacts, which makes it difficult to effectively repair the text content. When trying to remove the underprint watermark in a document image, the original text information becomes unclear, which leads to loss of information. Although diffusion models are also used to process watermark document pictures, using only diffusion models also faces challenges in the task of removing underprint watermark, such as difficulty in distinguishing foreground text from background watermark, fuzzy text, appearance of artifacts, and underprint and watermark residue. Therefore, this method also cannot accurately obtain the document content in the watermark picture.

[0088] In the embodiments of the present application, by establishing a model architecture with two parts of low-frequency feature restoration and high-frequency feature restoration, the overall structure and content of the watermark document picture are first restored, the watermark is removed, and then the high-frequency feature restoration model of the text body is used to restore the text details in the document picture; at the same time, the high-frequency feature restoration model of the watermark body is used to restore the watermark in the document picture.

[0089] In specific implementation, the method may, for example, include: first, obtaining an original picture of a watermark document, the original picture being a picture of a document with a watermark; inputting the original picture into a watermark removal module in an extraction model to obtain a watermark removal picture, the watermark removal module including a trained low-frequency feature restoration model, the low-frequency feature restoration model being a non-linear activation network; then inputting the watermark removal picture into a text restoration module in the extraction model to obtain a document text picture, the document text picture being a picture of the watermark removal picture after restoration of text content, the text restoration module including a trained high-frequency feature restoration model of a text body, the high-frequency feature restoration model of the text body being a diffusion model; and simultaneously inputting the original picture into an extraction watermark module in the extraction model to obtain a watermark picture, the extraction watermark module including a trained high-frequency feature restoration model of a watermark body, the high-frequency feature restoration model of the watermark body being a diffusion model.

[0090] As can be seen, the method provided by the embodiments of the present application uses a low-frequency feature restoration model to restore low-frequency structural features, and then uses two diffusion models to restore high-frequency features of the text body and the watermark, respectively, and through targeted restoration of the text body, the problems of unclear text content in the picture and the inability of the diffusion model to distinguish between foreground text and background watermark are solved.

[0091] Finally, the text content in the document picture is restored, so as to solve the problem of unclear text information in the watermark removal process in the background technology, and to avoid information loss.

[0092] To facilitate understanding of the specific implementation of the method for extracting document text and watermark from a watermark document image provided by the embodiments of the present application, the following will be described with reference to the accompanying drawings.

[0093] It should be noted that the subject of implementing the method for extracting document text and watermark from a watermark document image can be a device for extracting document text and watermark from a watermark document image provided by the embodiments of the present application, and the device for extracting document text and watermark from a watermark document image can be carried in an electronic device or a functional module of an electronic device. The electronic device in the embodiments of the present application can be any device capable of implementing the method for extracting document text and watermark from a watermark document image in the embodiments of the present application, for example, an Internet of Things (IoT) device.

[0094] Figure 2 A flowchart of a method for extracting document text and watermarks from a watermarked document image is provided for embodiments of the present application. The method can be applied to an apparatus for extracting document text and watermarks from a watermarked document image, which may, for example, be an apparatus for extracting document text and watermarks from a watermarked document image 800 as shown in Figure 8 The apparatus for extracting document text and watermarks from a watermarked document image may, for example, be integrated into functional modules in an electronic device 900 as shown in Figure 9

[0095] As shown in Figure 2 The method includes the following S201-S204:

[0096] S201: Obtain an original picture of a watermarked document, the original picture being a picture of a watermarked document with a background watermark.

[0097] To implement the extraction of document text and watermarks from a watermarked document, first, an original picture of a watermarked document with a background watermark is obtained, and then the original picture is input into a watermark removal module in an extraction model to obtain a watermark-removed picture, and the watermark-removed picture is input into a text repair module in the extraction model to obtain a document text picture; at the same time, the original picture is input into an watermark extraction module in the extraction model to obtain a background watermark picture. Therefore, the original picture of the watermarked document as shown in Figure 3 is obtained by S201, which makes a preparation for obtaining the document text picture and the background watermark picture subsequently.

[0098] S202: Input the original picture into a watermark removal module in the extraction model to obtain a watermark-removed picture, the watermark removal module including a trained low-frequency feature repair model, the low-frequency feature repair model using a non-linear activation network, and the training data of the watermark removal model being low-frequency feature historical document pictures and low-frequency feature training document pictures.

[0099] In the embodiments of the present application, the extraction of document text and watermarks from a watermarked document image is completed by using an extraction model, which includes a watermark removal module, a text repair module, and a watermark extraction module.

[0100] Therefore, as an example, S202 can include: inputting the original picture into a watermark removal module in the extraction model, the watermark removal module being used to remove the background watermark and repair low-frequency features in the original picture to restore the structural content of the original picture, and obtaining a watermark-removed picture as shown in Figure 4 ​The low-frequency feature repaired picture shown is a watermark-removed picture. The watermark-removed module includes a trained low-frequency feature repair model, which uses a nonlinear activation free network (NAFN). The network architecture does not use a nonlinear activation function, effectively reducing the computational load and achieving excellent results in image repair. The network architecture also uses a UNet-based network architecture, which is a regression prediction-based image repair method designed to restore the structural features of the document image.

[0101] It should be noted that, for logical consistency, the specific content of training the watermark-removed model will be described in the part of the system after S204, which will not be expanded here.

[0102] S203: inputting the watermark-removed picture into the text repair module of the extraction model to obtain a document text picture, the document text picture being a picture after the watermark-removed picture is repaired for text content, the text repair module including a trained high-frequency feature repair model of a text body, the high-frequency feature repair model of the text body being a diffusion model, and training data of the high-frequency feature repair model of the text body being high-frequency feature historical document pictures and high-frequency feature training document pictures.

[0103] As an example, S203 can include: on the basis of removing the underprint watermark and restoring the structural content of the original picture by S202 described above, continuing to input the watermark-removed picture into the text repair module of the extraction model, the text repair module being used to repair the text details in the document picture and restore the high-frequency text features in the watermark-removed picture to obtain a high-frequency feature repaired text body picture shown as Figure 5 The high-frequency feature repaired text body picture shown is a document text picture, which is a picture after the watermark-removed picture is repaired for text content. The text repair module includes a trained high-frequency feature repair model of a text body, which uses a diffusion model, wherein the watermark-removed picture is input as a condition of the diffusion model, and Gaussian noise is restored to a corresponding picture without an underprint watermark document. Since the repair of the document picture does not require diversity of image generation, in order to speed up the iteration speed of the diffusion model, the present application embodiment uses a sampling operator named DPM-solver, which can achieve a relatively good image repair effect after 10-20 iterations.

[0104] It should be noted that, for logical consistency, the specific content of training the high-frequency feature repair model of the text body will be described in the part of the system after S204, which will not be expanded here.

[0105] The embodiment of the present application proposes a progressive document image restoration method. The method first restores the low-frequency structural features of the image by using the watermark elimination module, and then further restores the high-frequency text features based on the watermark elimination picture using a diffusion model. This method effectively solves the problem of text blur or artifacts in previous methods.

[0106] S204: input the original picture into the extraction watermark module in the extraction model to obtain the underprint watermark picture, the extraction watermark module includes the trained high-frequency feature restoration model of the watermark subject, the high-frequency feature restoration model of the watermark subject is a diffusion model, and the training data of the high-frequency feature restoration model of the watermark subject is the high-frequency feature historical underprint watermark and the high-frequency feature training underprint watermark.

[0107] As an example, S204 can include: inputting the original picture into the extraction watermark module in the extraction model, the extraction watermark module being used to restore the high-frequency features of the underprint watermark, i.e., to restore the text content of the underprint watermark, to obtain the underprint watermark picture extracted after high-frequency feature restoration as shown in Figure 6 , i.e., the underprint watermark picture. Wherein the extraction watermark module includes the trained high-frequency feature restoration model of the watermark subject, the high-frequency feature restoration model of the watermark subject adopts a diffusion model, wherein the original picture is input as a condition of the diffusion model, and the Gaussian noise is restored to the corresponding underprint watermark original picture.

[0108] It should be noted that the execution order of S202-S203 and S204 is not limited in the embodiment of the present application, i.e., S202-S203 can be executed first and then S204 can be executed, or S204 can be executed first and then S202-S203 can be executed, or S202-S203 and S204 can be executed simultaneously.

[0109] It should be noted that the extraction model is used to extract the document text and the underprint watermark in the present application, wherein the extraction model includes three modules, i.e., the watermark elimination module, the text restoration module and the extraction watermark module, and the three modules also include corresponding models. For the three models, the loss function of the entire network architecture needs to be calculated, and the parameters of the model are updated by the gradient descent method, so that the extraction model has the ability to extract the document text and the underprint watermark. The training process of the extraction model is described in detail as follows:

[0110] (1) Prepare training data: obtain a historical data set, the historical data set including a historical document picture with underprint watermark, a low-frequency feature historical document picture, a high-frequency feature historical document picture and a high-frequency feature historical underprint watermark.

[0111] The specific process of obtaining the historical data set can include: generating a background and a text watermark respectively, and obtaining a background watermark according to the background and the text watermark; adding the background watermark to a clean historical document picture to obtain a historical document picture with a background watermark; performing low-pass filtering image processing on the clean historical document picture to obtain a low-frequency feature historical document picture; performing high-pass filtering image processing on the clean historical document picture to obtain a high-frequency feature historical document picture; and performing high-pass filtering image processing on the background watermark to obtain a high-frequency feature historical background watermark.

[0112] (2) Obtain a loss function corresponding to each model: ①Loss function LL of the low-frequency feature restoration model: input the historical document picture with the background watermark into the low-frequency feature restoration model to generate a low-frequency feature training document picture; calculate the mean square error of the low-frequency feature training document picture and the low-frequency feature historical document picture to obtain the loss function LL. ②Loss function LN of the high-frequency feature restoration model of the text main body: input the low-frequency feature training document picture into the high-frequency feature restoration model of the text main body to generate a high-frequency feature training document picture; calculate the mean square error of the high-frequency feature training document picture and the high-frequency feature historical document picture to obtain the loss function LN. ③Loss function LH of the high-frequency feature restoration model of the watermark main body: input the historical document picture with the background watermark into the high-frequency feature restoration model of the watermark main body to generate a high-frequency feature training background watermark; calculate the mean square error of the high-frequency feature training background watermark and the high-frequency feature historical background watermark to obtain the loss function LH.

[0113] (3) Calculate the reconstruction loss: the model architecture in the embodiment introduces a reconstruction loss function, specifically, the high-frequency feature training document picture and the high-frequency feature training background watermark are combined to generate a reconstructed picture, the mean square error of the historical document picture with the background watermark and the reconstructed picture is calculated to obtain the loss function L1 of the reconstruction, so that the output of the model can imitate the original historical document picture with the background watermark as much as possible, and the document text picture and the background watermark picture can be extracted and removed at the same time.

[0114] (4) Obtain the loss function of the text recognition model: in the training stage, the text recognition model is introduced in the embodiment, the high-frequency feature training document picture is input into the text recognition model to obtain predicted text; the predicted text is compared with the real text in the high-frequency feature historical document picture, and a connection time sequence classification (CTC) loss function, i.e. the loss function L2, is used, and the high-frequency feature training document picture is further optimized in the gradient back propagation mode, so as to improve the accuracy of subsequent OCR recognition. The text recognition model in the embodiment adopts an open source PaddleOCR, and the text recognition engine can extract the coordinates and text content of the text line.

[0115] (5) Training according to the total loss: summing the loss function LL, the loss function LN, the loss function LH, the loss function L1 and the loss function L2 of the character recognition model to obtain the total loss; and updating the learnable parameters in the extraction model according to the total loss by the method of gradient descent, to achieve the effect of removing the background watermark.

[0116] It can be seen that, by the regression prediction method of the convolutional neural network in the low-frequency feature restoration model, the background watermark is removed and the low-frequency feature is restored in the embodiment of the application, the structure content of the original picture is recovered, and the watermark-removed picture is obtained. On this basis, in order to further recover the text content in the document picture, the high-frequency feature of the text subject is restored by inputting the watermark-removed picture into the high-frequency feature restoration model of the text subject, which is a diffusion model, to restore the text content in the document picture, so as to obtain the document text picture, which is helpful for subsequent accurate OCR recognition. At the same time, by the high-frequency feature restoration model of the watermark subject, the high-frequency feature of the watermark subject is restored, which is helpful for subsequent watermark text recognition. And in this process, the extraction model in the embodiment of the application further includes a character recognition model, that is, the text feature is restored by using the character recognition model, and the potential text stroke deformation or loss is restored, so as to obtain the optimized document text picture, thereby improving the accuracy of subsequent OCR recognition. At the same time, the high-frequency feature restoration model of the watermark subject is used to recover the background watermark text content in the original picture, which is helpful for subsequent watermark text recognition.

[0117] In order to make the extraction model training method provided by the embodiment of the application more clear and easy to understand, the following will be combined with Figure 7 The extraction model training method is described.

[0118] As shown in Figure 7 , the embodiment can include S701-S710:

[0119] S701: acquiring a historical data set, the historical data set including a historical document picture with a background watermark, a low-frequency feature historical document picture, a high-frequency feature historical document picture and a high-frequency feature historical background watermark.

[0120] As an example, S701 can include: generating a texture (randomly controlling the color and transparency of the texture) and a text watermark (randomly controlling the size and rotation angle of the text) respectively, and obtaining a texture watermark according to the texture and the text watermark; adding the texture watermark on a clean historical document picture (randomly controlling the transparency of the texture watermark, that is, controlling the Alpha channel in RGBA), to obtain a historical document picture with a texture watermark; performing low-pass filtering image processing on the clean historical document picture to obtain a low-frequency feature historical document picture; performing high-pass filtering image processing on the clean historical document picture to obtain a high-frequency feature historical document picture; and performing high-pass filtering image processing on the texture watermark to obtain a high-frequency feature historical texture watermark.

[0121] S702: inputting the historical document picture with the texture watermark into a low-frequency feature restoration model to generate a low-frequency feature training document picture.

[0122] In order to generate a document picture with removed texture watermark and restored low-frequency features, the historical document picture with the texture watermark is input into the low-frequency feature restoration model to generate a low-frequency feature training document picture. In the embodiment of the present application, the low-frequency feature restoration model is based on a regression prediction image restoration method, which aims to restore the structural features of the document image.

[0123] S703: inputting the low-frequency feature training document picture into a high-frequency feature restoration model of a text body to generate a high-frequency feature training document picture.

[0124] Since the low-frequency feature training document picture generated in the above may have a blurred text or artifact, in order to solve this problem, the low-frequency feature training document picture is input into the high-frequency feature restoration model of the text body to further restore the text details in the document picture, to generate a clearer document picture, that is, a high-frequency feature training document picture.

[0125] S704: inputting the historical document picture with the texture watermark into a high-frequency feature restoration model of a watermark body to generate a high-frequency feature training texture watermark.

[0126] At the same time, in order to generate a texture watermark picture, the historical document picture with the texture watermark is input into the high-frequency feature restoration model of the watermark body to generate a high-frequency feature training texture watermark. The high-frequency feature restoration model of the watermark body can restore the text details of the watermark to generate a clear texture watermark.

[0127] S705: calculating the mean square error of the low-frequency feature training document picture and the low-frequency feature historical document picture to obtain a loss function LL.

[0128] In order to train each model in the extraction model, the loss function of the entire network architecture needs to be calculated, and the parameters of the model are updated by the method of gradient descent, so that the extraction model has the ability to remove the underprint watermark. The loss function involved in the embodiments of the present application includes the following parts: one is the mean square error (MSE) loss function LL of the low-frequency feature restoration model; two is the mean square error loss function LN of the high-frequency feature restoration model of the text body; three is the mean square error loss function LH of the high-frequency feature restoration model of the watermark body; four is the mean square error loss function L1 between the reconstructed picture and the historical document picture with underprint watermark; five is the connection time sequence classification (CTC) loss function L2 between the recognized text content and the real text content. The loss functions of these parts are needed in the subsequent steps.

[0129] S706: Calculate the mean square error of the high-frequency feature training document picture and the high-frequency feature historical document picture to obtain the loss function LN.

[0130] S707: Calculate the mean square error of the high-frequency feature training underprint watermark and the high-frequency feature historical underprint watermark to obtain the loss function LH.

[0131] S708: Calculate the mean square error between the historical document picture with underprint watermark and the reconstructed picture to obtain the loss function L1, wherein the reconstructed picture is generated by synthesizing the high-frequency feature training document picture and the high-frequency feature training underprint watermark.

[0132] S709: Sum the loss function LL, the loss function LN, the loss function LH, the loss function L1 and the loss function L2 of the text recognition model to obtain the total loss, wherein the text recognition model is used to optimize the text body details in the high-frequency feature training document picture.

[0133] The use process of the text recognition model in the embodiments of the present application includes: inputting the high-frequency feature training document picture into the text recognition model to obtain the predicted text; comparing the predicted text with the real text in the high-frequency feature historical document picture to optimize the text body details in the high-frequency feature training document picture and alleviate the problems of text stroke errors, blurring, etc. It should be noted that the text recognition model does not participate in model training and is a fixed model parameter.

[0134] S710: Update the learnable parameters in the training extraction model in reverse according to the total loss.

[0135] The total loss function of the entire extraction model in the embodiments of the present application is L = LL + LN + LH + L1 + L2, and the parameters of the three models (low-frequency feature restoration model, high-frequency feature restoration model of text body and high-frequency feature restoration model of watermark body) are updated by the method of gradient descent, so as to achieve the effect of removing the underprint watermark.

[0136] The embodiment provides a method for extracting watermark document image document text and watermark. After low-frequency structure features are recovered by using a regression prediction-based method, high-frequency features of subject text and underprint watermark are repaired by using two diffusion models respectively, foreground text and background watermark cannot be distinguished in the existing diffusion model, and the problem can be solved by targeted subject recovery text. Furthermore, a text recognition model is used to further optimize text subject details in the document picture, and problems such as text stroke error and blur are relieved, so that unclear text information in the background technology in the watermark removal process is solved, and information loss is avoided.

[0137] Referring to Figure 8 The embodiment of the present application provides a device 800 for extracting watermark document image document text and watermark, which comprises:

[0138] The acquisition unit 801 is configured to acquire an original picture of a watermark document, wherein the original picture is a picture of an underprint watermark document;

[0139] The watermark elimination unit 802 is configured to input the original picture into a watermark elimination module in an extraction model to obtain a watermark elimination picture, wherein the watermark elimination module comprises a trained low-frequency feature repair model, the low-frequency feature repair model adopts a nonlinear activation network, and training data of the watermark elimination model comprises low-frequency feature historical document pictures and low-frequency feature training document pictures;

[0140] The text repair unit 803 is configured to input the watermark elimination picture into a text repair module in the extraction model to obtain a document text picture, wherein the document text picture is a picture after the watermark elimination picture is repaired with text content, the text repair module comprises a trained high-frequency feature repair model of text subject, the high-frequency feature repair model of text subject is a diffusion model, and training data of the high-frequency feature repair model of text subject comprises high-frequency feature historical document pictures and high-frequency feature training document pictures;

[0141] The watermark extraction unit 803 is configured to input the original picture into an extraction watermark module in the extraction model to obtain an underprint watermark picture, wherein the extraction watermark module comprises a trained high-frequency feature repair model of watermark subject, the high-frequency feature repair model of watermark subject is a diffusion model, and training data of the high-frequency feature repair model of watermark subject comprises high-frequency feature historical underprint watermark and high-frequency feature training underprint watermark.

[0142] Optionally, the device 800 comprises:

[0143] The acquisition unit 801 is further configured to acquire a historical data set, wherein the historical data set comprises historical document pictures with underprint watermark, low-frequency feature historical document pictures, high-frequency feature historical document pictures and high-frequency feature historical underprint watermark.

[0144] obtain a loss function LL of the low-frequency feature restoration model according to the low-frequency feature historical document picture;

[0145] obtain a loss function LN of the high-frequency feature restoration model of the text subject according to the high-frequency feature historical document picture;

[0146] obtain a loss function LH of the high-frequency feature restoration model of the watermark subject according to the high-frequency feature historical background watermark;

[0147] combine the high-frequency feature training document picture and the high-frequency feature training background watermark to generate a reconstructed picture;

[0148] obtain a loss function L1 of the low-frequency feature historical document picture with the background watermark and the reconstructed picture;

[0149] obtain a total loss by summing the loss function LL, the loss function LN, the loss function LH, the loss function L1, and a loss function L2 of a text recognition model used to optimize the text subject details in the high-frequency feature training document picture;

[0150] update the learnable parameters in the training extraction model in a reverse direction according to the total loss.

[0151] Optionally, when the obtaining unit is configured to obtain the loss function LL of the low-frequency feature restoration model according to the low-frequency feature historical document picture, the obtaining unit is specifically configured to:

[0152] input the low-frequency feature historical document picture with the background watermark into the low-frequency feature restoration model to generate a low-frequency feature training document picture;

[0153] obtain the loss function LL by calculating a mean square error of the low-frequency feature training document picture and the low-frequency feature historical document picture.

[0154] Optionally, when the obtaining unit is configured to obtain the loss function LN of the high-frequency feature restoration model of the text subject according to the high-frequency feature historical document picture, the obtaining unit is specifically configured to:

[0155] input the low-frequency feature training document picture into the high-frequency feature restoration model of the text subject to generate a high-frequency feature training document picture;

[0156] obtain the loss function LN by calculating a mean square error of the high-frequency feature training document picture and the high-frequency feature historical document picture.

[0157] Optionally, the apparatus 800 further includes:

[0158] The obtaining unit is further configured to input the high-frequency feature training document picture into a character recognition model to obtain predicted characters.

[0159] The optimization unit is configured to compare the predicted characters with real characters in the high-frequency feature historical document picture to optimize character main body details in the high-frequency feature training document picture.

[0160] Optionally, when the obtaining unit is configured to obtain a loss function LH of the high-frequency feature repair model of the watermark main body according to the high-frequency feature historical watermark, the obtaining unit is specifically configured to:

[0161] input the historical document picture with the watermark into the high-frequency feature repair model of the watermark main body to generate the high-frequency feature training watermark;

[0162] calculate a mean square error of the high-frequency feature training watermark and the high-frequency feature historical watermark to obtain the loss function LH.

[0163] Optionally, when the obtaining unit 801 is configured to obtain the historical data set, the obtaining unit 801 is specifically configured to:

[0164] generate the watermark and the character watermark respectively, and obtain the watermark according to the watermark and the character watermark;

[0165] add the watermark on the clean historical document picture to obtain the historical document picture with the watermark;

[0166] perform low-pass filtering image processing on the clean historical document picture to obtain a low-frequency feature historical document picture;

[0167] perform high-pass filtering image processing on the clean historical document picture to obtain a high-frequency feature historical document picture;

[0168] perform high-pass filtering image processing on the watermark to obtain a high-frequency feature historical watermark.

[0169] It should be noted that the specific implementation manner of the device 800 for extracting the document characters and the watermark of the document image and the effects achieved can be referred to the related descriptions in the above Figure 2 or Figure 7 the method provided, which will not be described here.

[0170] Embodiments of the present application further provide an electronic device 900, as shown in the accompanying drawings, the device 900 comprises a memory 901 and a processor 902: Figure 9

[0171] The memory 901 is configured to store a computer program.

[0172] The processor 902 is configured to execute the above Figure 2 or​Figure 7 The method provided.

[0173] In addition, the present application also provides a computer readable storage medium, the computer readable storage medium is used for storing a computer program, the computer program is used for executing Figure 2 or Figure 7 The method provided.

[0174] From the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment methods can be implemented by means of software plus a general hardware platform. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as a read-only memory (English: read-only memory, ROM) / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments.

[0175] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, they are described more simply, and the relevant parts can be referred to the part of the description of the method embodiments. The device embodiments described above are only schematic, and the modules shown as separate components can or can not be physically separated, and the components shown as modules can or can not be physical modules, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, some or all of the modules can be selected to achieve the goal of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0176] The above description is only an exemplary embodiment of the present application, and is not intended to limit the protection scope of the present application.

Claims

1. A method for extracting document text and watermark from a watermarked document image, characterized in that: include: Obtaining an original image of the watermark document, wherein the original image is an image of the document with a shading watermark; Inputting the original image into a watermark removal module in an extraction model to obtain a watermark-removed image, wherein the watermark removal module includes a trained low-frequency feature restoration model, wherein the low-frequency feature restoration model uses a non-nonlinear activation network, and the training data of the watermark removal model are low-frequency feature historical document images and low-frequency feature training document images; Inputting the watermark-removed image into the text repair module in the extraction model to obtain a document text image, wherein the document text image is the image after the watermark-removed image has its text content repaired, the text repair module includes a trained high-frequency feature repair model for the text body, the high-frequency feature repair model for the text body is a diffusion model, and the training data for the high-frequency feature repair model for the text body is a high-frequency feature historical document image and a high-frequency feature training document image; The original image is input into the watermark extraction module in the extraction model to obtain a shading watermark image. The watermark extraction module includes a trained high-frequency feature repair model of the watermark body. The high-frequency feature repair model of the watermark body is a diffusion model. The training data of the high-frequency feature repair model of the watermark body is a high-frequency feature historical shading watermark and a high-frequency feature training shading watermark.

2. The method according to claim 1, characterized in that The training process of the extraction model includes: Acquire a historical data set, the historical data set including a historical document image with a shading watermark, the low-frequency feature historical document image, the high-frequency feature historical document image, and the high-frequency feature historical shading watermark; Obtaining a loss function LL of the low-frequency feature restoration model according to the low-frequency feature historical document image; Obtaining a loss function LN of a high-frequency feature restoration model of the text body according to the high-frequency feature historical document image; According to the high-frequency feature historical shading watermark, a loss function LH of a high-frequency feature restoration model of the watermark body is obtained; Synthesize the high-frequency feature training document image and the high-frequency feature training shading watermark to generate a reconstructed image; Calculate the mean square error between the historical document image with the shading watermark and the reconstructed image to obtain a loss function L1; Summing the loss function LL, the loss function LN, the loss function LH, the loss function L1, and the loss function L2 of the text recognition model to obtain a total loss, wherein the text recognition model is used to optimize the text body details in the high-frequency feature training document image; The learnable parameters in the extraction model are reversely updated and trained according to the total loss.

3. The method according to claim 2, characterized in that The step of obtaining the loss function LL of the low-frequency feature restoration model according to the low-frequency feature historical document image includes: Inputting the historical document image with the shading watermark into the low-frequency feature restoration model to generate the low-frequency feature training document image; The mean square error between the low-frequency feature training document image and the low-frequency feature historical document image is calculated to obtain the loss function LL.

4. The method according to claim 2, characterized in that The step of obtaining the loss function LN of the high-frequency feature restoration model of the text body based on the high-frequency feature historical document image includes: Inputting the low-frequency feature training document image into the high-frequency feature repair model of the text body to generate the high-frequency feature training document image; The mean square error between the high-frequency feature training document image and the high-frequency feature historical document image is calculated to obtain the loss function LN.

5. The method according to claim 4, characterized in that The process of using the text recognition model includes: Inputting the high-frequency feature training document image into the text recognition model to obtain predicted text; The predicted text is compared with the real text in the high-frequency feature historical document image to optimize the text body details in the high-frequency feature training document image.

6. The method according to claim 2, characterized in that The step of obtaining the loss function LH of the high-frequency feature restoration model of the watermark subject according to the high-frequency feature historical shading watermark comprises: Inputting the historical document image with the shading watermark into the high-frequency feature restoration model of the watermark body to generate the high-frequency feature training shading watermark; The mean square error between the high-frequency feature training shading watermark and the high-frequency feature historical shading watermark is calculated to obtain the loss function LH.

7. The method according to any one of claims 2 to 6, characterized in that: The obtaining of the historical data set includes: Generating a shading pattern and a text watermark respectively, and obtaining a shading pattern watermark according to the shading pattern and the text watermark; Adding the shading watermark to a clean historical document image to obtain the historical document image with the shading watermark; Performing low-pass filtering on the clean historical document image to obtain the low-frequency feature historical document image; Performing high-pass filtering on the clean historical document image to obtain the high-frequency feature historical document image; The shading watermark is subjected to high-pass filtering image processing to obtain the high-frequency characteristic historical shading watermark.

8. A device for extracting document text and watermark from a watermarked document image, characterized in that: include: An acquisition unit, configured to acquire an original image of a watermark document, wherein the original image is an image of a document with a shading watermark; a watermark removal unit, configured to input the original image into a watermark removal module in an extraction model to obtain a watermark-removed image, wherein the watermark removal module includes a trained low-frequency feature restoration model, the low-frequency feature restoration model employing a non-linear activation network, and the training data for the watermark removal model are low-frequency feature historical document images and low-frequency feature training document images; a text restoration unit, configured to input the watermark-removed image into a text restoration module in the extraction model to obtain a document text image, wherein the document text image is the image obtained by restoring the text content of the watermark-removed image, the text restoration module including a trained high-frequency feature restoration model for the text body, the high-frequency feature restoration model for the text body being a diffusion model, and the training data for the high-frequency feature restoration model for the text body being high-frequency feature historical document images and high-frequency feature training document images; The watermark extraction unit is used to input the original image into the watermark extraction module in the extraction model to obtain a shading watermark image. The watermark extraction module includes a trained high-frequency feature repair model of the watermark body. The high-frequency feature repair model of the watermark body is a diffusion model. The training data of the high-frequency feature repair model of the watermark body is a high-frequency feature historical shading watermark and a high-frequency feature training shading watermark.

9. An electronic device, characterized in that The device includes a memory and a processor, and the processor is configured to execute a program stored in the memory to run the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method according to any one of claims 1 to 7.

Images