Document denoising method, apparatus, device, and medium

Abstract

The present disclosure provides a document denoising method, which can be applied to the field of artificial intelligence technology. The method comprises: based on a coding-decoding structure fusion network model, using an encoding module to perform feature compression on an input first document image; and using a decoding module to perform image restoration on the output of the encoding module and output a second document image, wherein the second document image is used as the input of an optical character recognition system. The encoding module comprises at least one convolution layer, in each convolution layer, a region generation network is used to generate a search box on the input feature map of the convolution layer through a sliding window, and each search box is classified, the classification comprises two categories of redundancy and non-redundancy; a feature extraction sub-network is used to extract features of the search box classified into the non-redundancy category, to obtain the output feature map of the convolution layer. The present disclosure also provides a document denoising device, equipment, storage medium and program product.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically to a document denoising method, apparatus, device, medium, and program product. Background Technology

[0002] Currently, when documents are scanned, faxed, copied, and printed, external environmental factors can easily cause the original form of the document to become blurred, and due to these distortions, it becomes impossible to record them in a more permanent digital form. Noise in documents severely hinders the performance of optical character recognition systems, as well as subsequent digitization and analysis.

[0003] Existing image recognition methods often focus more on removing noise from images (such as natural landscapes) rather than written documents. Meanwhile, although deep neural networks have superior performance in image processing and computer vision applications, from low-level denoising to high-level recognition, deblurring, super-resolution, image inpainting, and restoring original images from compressed images, their performance is poor in document denoising tasks, and they still need improvement when dealing with document degradation caused by natural and human factors. Summary of the Invention

[0004] In view of the above problems, this disclosure provides document denoising methods, apparatus, devices, media, and program products for improving document image quality to enhance text recognition efficiency.

[0005] A first aspect of this disclosure provides a document denoising method. The method includes: inputting a first document image into a trained fusion network model, wherein the fusion network model includes an encoding module and a decoding module; performing feature compression on the first document image using the encoding module; and using the decoding module to restore the output of the encoding module and output a second document image. The encoding module includes at least one convolutional layer, wherein the convolutional layer includes a region generation network and a feature extraction sub-network; wherein performing feature compression on the first document image using the encoding module includes: using the region generation network to generate search boxes on the input feature map of the convolutional layer through a sliding window, and classifying each search box, the classification including redundant and non-redundant categories; and using the feature extraction sub-network to extract features from the search boxes classified as non-redundant, obtaining the output feature map of the convolutional layer.

[0006] According to embodiments of this disclosure, classifying each search box using the region generation network includes classifying each search box based on a nonmaximum suppression algorithm.

[0007] According to an embodiment of this disclosure, the step of using the feature extraction subnetwork to extract features from search boxes classified into non-redundant categories includes: using a Region of Interest (ROI) subnetwork to divide the search boxes classified into non-redundant categories into parts of the same size as the dimension of the output feature map, and using the maximum value of each part as the output of each position in the output feature map.

[0008] According to an embodiment of this disclosure, the feature compression of the first document image using the encoding module includes: through dense connections, each convolutional layer in the encoding module, except for the last convolutional layer, inputs the output of that convolutional layer into all subsequent convolutional layers.

[0009] According to embodiments of this disclosure, the fusion network model further includes a hollow spatial pyramid pooling module. Before using the decoding module to perform image reconstruction on the output of the encoding module, the method further includes: using the output of the encoding module as the input of the hollow spatial pyramid pooling module to perform feature fusion using the hollow spatial pyramid pooling module; and inputting the output of the hollow spatial pyramid pooling module into the decoding module.

[0010] According to an embodiment of this disclosure, the fusion network model further includes a conditional random field. After using the decoding module to restore the image from the output of the encoding module, the method further includes: using the output of the decoding module as the input of the conditional random field, processing it with the conditional random field, and then outputting the second document image.

[0011] According to embodiments of this disclosure, the method further includes: recognizing text in the second document image using the optical character recognition system.

[0012] A second aspect of this disclosure provides a document denoising apparatus. The apparatus includes a fusion network model. The fusion network model receives an input first document image, processes the first document image, and outputs a second document image, wherein the second document image serves as input to an optical character recognition system. Specifically, the fusion network model includes an encoding module and a decoding module. The encoding module performs feature compression on the first document image, and the decoding module performs image restoration on the output of the encoding module and outputs the second document image. More specifically, the encoding module includes at least one convolutional layer, wherein the convolutional layer includes a region generation network and a feature extraction sub-network. The region generation network generates search boxes on the input feature map of the convolutional layer using a sliding window and classifies each search box, including redundant and non-redundant categories. The feature extraction sub-network extracts features from the search boxes classified as non-redundant to obtain the output feature map of the convolutional layer.

[0013] According to embodiments of this disclosure, the apparatus further includes an optical character recognition system. The optical character recognition system is used to recognize text in the second document image.

[0014] A third aspect of this disclosure provides an electronic device. The electronic device includes one or more processors and a memory. The memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0015] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods described above.

[0016] A fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0017] The above one or more embodiments have the following advantages or beneficial effects: they can at least partially solve the problem that existing optical character recognition methods cannot effectively handle document degradation caused by natural and human factors, and thus can achieve text image enhancement processing on document image data with document degradation caused by natural and human factors, thereby improving the efficiency of text recognition in document images. Attached Figure Description

[0018] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 The illustrations depict application scenarios of document denoising methods, apparatuses, devices, media, and program products according to embodiments of the present disclosure.

[0020] Figure 2 A flowchart illustrating a document denoising method according to an embodiment of the present disclosure is shown schematically.

[0021] Figure 3 This illustration schematically shows the processing procedure of the convolutional layer in the encoding module of a document denoising method according to an embodiment of the present disclosure;

[0022] Figure 4 The illustration schematically shows the process of denoising a document image using a fusion network model in another embodiment of this disclosure;

[0023] Figure 5 The training process of the fusion network model in the embodiments of this disclosure is illustrated schematically;

[0024] Figure 6 This schematic diagram illustrates a structural block diagram of a document denoising apparatus according to an embodiment of the present disclosure;

[0025] Figure 7 An embodiment of the present disclosure is illustrated schematically. Figure 6 The diagram shows the structural block diagram of the fusion network model in the document denoising device; and

[0026] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a document denoising method according to embodiments of the present disclosure. Detailed Implementation

[0027] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0029] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0030] When expressions such as "at least one of A, B, and C" are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having only A, only B, only C, A and B, A and C, B and C, and / or systems having A, B, and C, etc.). The terms "first," "second," etc., used herein are for distinction only and have no limiting meaning, and the number of any elements in the accompanying drawings is for illustrative purposes only and not for limitation.

[0031] Document denoising is one of the most challenging tasks in computer vision. Millions of documents remain to be digitized, but issues such as document degradation caused by natural and human factors make this task extremely difficult.

[0032] In view of this, embodiments of the present disclosure provide a document denoising method, apparatus, device, medium, and program product. According to embodiments of the present disclosure, a first document image input therein can be processed using a fusion network model based on an encoding / decoding structure, and then a second document image with clearer text images can be output. Embodiments of the present disclosure can be used to recover from problems such as distortion caused by document aging, creases left on photocopied pages, random black patches, and slightly visible text, and improve image quality and the performance of optical character recognition systems.

[0033] Figure 1 The illustration schematically depicts application scenarios of document denoising methods, apparatus, devices, media, and program products according to embodiments of this disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0034] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as a medium for providing a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0035] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0036] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0037] Server 105 can be a server that provides various services. Server 105 can be configured with a fusion network model to improve the clarity of text in document images.

[0038] Users can upload document images to server 105 via terminal devices 101, 102, and 103. Server 105 can process the uploaded document images and feed the processing results back to terminal devices 101, 102, and 103. For example, server 105 can use its fusion network model to sharpen the document image and feed the sharpened image back to terminal devices 101, 102, and 103. Alternatively, in some embodiments, when server 105 is equipped with an optical character recognition system, server 105 can use the optical character recognition system to perform text recognition on the document image processed by the fusion network model and then feed the text recognition results back to terminal devices 101, 102, and 103.

[0039] It should be noted that the document denoising method provided in this disclosure embodiment can generally be executed by server 105. Accordingly, the document denoising apparatus, device, medium, and program product provided in this disclosure embodiment can generally be located in server 105. The document denoising method provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103, and / or server 105. Accordingly, the document denoising apparatus, device, medium, and program product provided in this disclosure embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103, and / or server 105.

[0040] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0041] The following will be based on Figure 1 The described scene, through Figures 2-4 The document denoising method according to embodiments of this disclosure will be described in detail. It should be noted that the sequence numbers of each operation in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0042] Figure 2 A flowchart illustrating a document denoising method according to an embodiment of the present disclosure is shown schematically.

[0043] like Figure 2 As shown, the document denoising method according to this embodiment may include operations S210 to S230.

[0044] First, in operation S210, the first document image is input into the trained fusion network model.

[0045] Then, in operation S220, the first document image is processed using the fusion network model.

[0046] The fusion network model includes an encoding module and a decoding module. Specifically, operation S220 involves: using the encoding module in the fusion network model to perform feature compression on the first document image, and using the decoding module in the fusion network model to restore the image from the output of the encoding module and output the second document image.

[0047] Next, in operation S230, the second document image output by the fusion network model is obtained. This second document image can be used as input to an optical character recognition system.

[0048] In one embodiment, the output of the fusion network model can be connected to the input of an optical character recognition system, thereby enabling the optical character recognition system to recognize text in a second document image.

[0049] In this embodiment of the disclosure, the encoding module may include at least one improved convolutional layer, wherein each of the at least one improved convolutional layer may include a region generation network and a feature extraction sub-network, and the data processing procedure of each convolutional layer may be as follows: Figure 3 As shown.

[0050] Figure 3 The illustration schematically shows the processing procedure of the convolutional layer in the encoding module of a document denoising method according to an embodiment of the present disclosure.

[0051] like Figure 3 As shown, the processing procedure of the above convolutional layer in the encoding module may include operation S301 and operation S302.

[0052] First, in operation S301, a Region Proposal Network (RPN) is used to generate search boxes on the input feature map of the convolutional layer through a sliding window, and each search box is classified into two categories: redundant and non-redundant.

[0053] The RPN network generates search boxes using a sliding window, ensuring that a search box can be generated for each feature. The RPN network then performs classification and regression operations on each search box. This allows for the removal of redundant search boxes during subsequent feature extraction and output, processing only those that are classified as non-redundant, thus addressing the redundancy problem caused by an excessive number of candidate regions during feature compression.

[0054] In one embodiment, the RPN network can classify each search box based on a nonmaximum suppression algorithm, thereby eliminating highly overlapping regions.

[0055] Then, in step S302, the feature extraction sub-network is used to extract features from the search boxes classified into non-redundant categories, resulting in the output feature maps of the convolutional layers. After the RPN network filters out redundant search boxes, the feature extraction network is used to extract features from the remaining search boxes.

[0056] In one embodiment, feature extraction can be performed using a Region of Interest (ROI) subnetwork. Specifically, the ROI subnetwork can be used to divide search boxes classified into non-redundant categories into portions of the same size as the output feature map, and the maximum value of each portion can be used as the output for each location in the output feature map. Alternatively, in other embodiments, the weighted average of the pixels in each portion can be used as the output for each location in the output feature map.

[0057] The feature extraction subnetwork extracts features from the non-redundant search box, enabling the reuse of feature maps in the convolutional network and ensuring that important features are not ignored.

[0058] According to one embodiment of this disclosure, the encoding module can also use dense connections (DenseNet) to input the output of each convolutional layer in the encoding module, except for the last convolutional layer, into all subsequent convolutional layers. In this way, from any layer to all subsequent layers in the encoding module, candidate regions can be extracted using RPN, and dense connections can be directly utilized to ensure the extraction of important features and minimize information loss. This enhances feature reuse, allowing information to be better preserved in the fusion network model. Consequently, the fusion network model can build deeper network models and achieve better performance with fewer parameters and computational costs.

[0059] According to embodiments of this disclosure, the fusion network model, based on the basic structures of the encoding and decoding modules, may further include a hollow spatial pyramid pooling module and a conditional random field (CRF). The input-output relationship between the hollow spatial pyramid pooling module and the CRF and the encoding and decoding modules can be referenced... Figure 4 The illustration.

[0060] Figure 4 The illustration schematically shows the denoising process of a document image using a fusion network model in another embodiment of this disclosure.

[0061] like Figure 4As shown, in the above operation S220, when denoising the document image, the output of the encoding module can be first input into the dilated spatial pyramid pooling module. The dilated spatial pyramid pooling module performs feature fusion on the output of the encoding module, and then inputs it into the decoding module. By using the dilated spatial pyramid pooling module to extract features from different angles and then re-aggregate them, the detailed information is fused again, further improving the denoising performance of the document image.

[0062] Furthermore, the image output by the decoding module can be first input into a conditional random field (CRF). After constraining the output sequence of the decoding module using the CRF, the final second document image is output. The CRF determines whether two pixels in the image output by the decoding module belong to the same class by calculating the similarity between them, thereby refining the classification of the extracted results. The result obtained after CRF post-processing is used as the final output of the fusion network model.

[0063] This embodiment improves the basic structure of the encoding and decoding modules, enhances the feature fusion of document images during document denoising, and further simplifies the precision of feature extraction through conditional random fields, thereby improving the clarity of text in the output second document image. This effectively solves the problem of difficulty in recovering document complexity and detail information caused by external environments such as noise and blur, and improves document denoising performance.

[0064] In the application of fusion network models Figures 2-4 Before performing document denoising, the fusion network model needs to be built and trained. The training process of the fusion network model in one embodiment can be found in [reference needed]. Figure 5 The illustration.

[0065] Figure 5 The training process of the fusion network model in the embodiments of this disclosure is illustrated schematically.

[0066] Combination Figure 5 The training process of the fusion network model mainly includes the following steps 1) to 5).

[0067] Step 1) Obtain various types of document image datasets and preprocess the datasets to ensure the smooth progress of model training and testing tasks.

[0068] Step 2) Divide the preprocessed document image dataset into a training set and a test set according to a preset 8:2 ratio.

[0069] Step 3) Construct a fusion network model based on the encoding / decoding structure. The basic model of the encoding / decoding structure includes an encoding module and a decoding module, and this basic model can be improved.

[0070] In one embodiment, the improvement described above could be by introducing an RPN network and dense connections into the encoding module. This allows for the extraction of important features and ensures minimal information loss from any layer to all subsequent layers by using the RPN to extract candidate regions and directly leveraging dense connections.

[0071] In another embodiment, the above improvement can also be achieved by using a hollow spatial pyramid pooling module after the encoding module to extract and re-aggregate features from different angles to fuse detailed information again, thereby improving the overall performance of the model.

[0072] In another embodiment, considering that after the decoding module, the model obtains a sequence of labeled scores for the input sequence, but the output labels at this point do not adequately consider the correlation between the preceding and following sequences, a conditional random field can be used to constrain the output sequence of the decoding module after the decoding module, and then the final output of the model can be obtained.

[0073] Step 4) Update the parameters of the neural network model based on the loss value output by the fusion network model.

[0074] Step 5) Use the test set to evaluate the trained improved network model, and combine the evaluation metrics to obtain the test results of the fusion model.

[0075] In step 1) above, various document datasets of different types are acquired. To meet this requirement, the NoisyOffice and PubLayNet public datasets are collected and downloaded to support the training and testing of this experiment. However, since the PubLayNet dataset is large, a portion of the dataset can be randomly selected for training and testing in this experiment. Furthermore, since this dataset is relatively clear, a portion of the dataset is re-acquired by artificially simulating a poor external environment. Then, noise is added to another portion of the dataset using data noise enhancement methods. The processed datasets are then organized and constructed to form a new document dataset for this experiment. Finally, to ensure the consistency of the input model's size, all images in the dataset are scaled or enlarged to a uniform size ratio.

[0076] In step 2) above, 80% of the samples in the constructed document image dataset can be randomly selected as the training set, and the remaining 20% ​​of the samples can be used as the test set for training and testing.

[0077] In step 3) above, the main framework of the fusion network model can use the U-Net basic model. Its main structure can be divided into two parts: an encoding module and a decoding module, with skip connections (or shortcut connections) from the encoding module to the decoding module. The encoding module is composed of a series of stacked convolutional blocks, and its purpose is to compress all the important information or features of the document into a feature vector, which can be used to reconstruct the document and eliminate noise in the process. The decoding module is composed of a series of stacked deconvolutional blocks, and its main purpose is to reconstruct the original image using the latent vector generated by the encoding module, while eliminating noise in the reconstruction process.

[0078] To mitigate gradient vanishing and enhance feature propagation, this embodiment of the present disclosure uses an RPN network and dense connections instead of the original convolutional blocks for feature extraction in each convolutional layer of the encoding module. This allows the RPN network to generate search boxes through a sliding window, ensuring that a search box is generated for each feature. Classification and regression operations are then performed on each search box, and highly overlapping regions are eliminated using algorithms such as non-maximum suppression, addressing the redundancy problem caused by an excessive number of candidate regions. Furthermore, dense connections connect each layer to all preceding layers along the feature dimension, enhancing feature reuse and allowing information (during forward propagation) or gradients (during backward propagation) to be better preserved throughout the network. This enables the training of deeper network models and the achievement of superior performance with fewer parameters and lower computational costs.

[0079] In each convolutional layer, after redundant search boxes are filtered out by the RPN network, the remaining search boxes can be used for feature extraction using the ROI sub-network. The ROI network divides each region to be processed into equal-sized parts (the number of which is the same as the output dimension), finds the maximum value of each part, and copies these maximum values ​​to the output. This ensures the processing speed of the network model, saves training time, and ensures that important features are not ignored by reusing the feature maps in the convolutional network, thereby improving the weight values.

[0080] Assuming each convolutional layer's input feature map has c channels, and each convolutional layer produces a new feature map with c channels, during the network's forward propagation, each convolutional layer can receive the outputs of all previous convolutional layers. The output of the nth convolutional layer can be expressed as follows:

[0081]

[0082] Where δ represents the activation function; This represents a convolution operation with a kernel size of k×k.

[0083] After feature extraction by the encoding module, the dilated spatial pyramid pooling module can unify the convolutional features of images at any scale into the same dimension. In the network structure, it is located before the fully connected layer and after the last pooling layer. The advantage of dilated spatial pyramid pooling is that it is insensitive to the size and aspect ratio of the input image, avoids cropping and deformation operations on the original image, and outputs a fixed-length feature vector.

[0084] Next, the decoding module is used to recover the spatial location of each pixel through upsampling. At the same time, in order to preserve the feature information output by different coding layers, skip connections are used to fuse the feature information with the output feature map obtained by the next layer decoding module after upsampling. The fused feature map is used to recover the spatial location information of the image. The additional features are used to improve the overall performance of the model.

[0085] Finally, the output of the decoding module is used as input data for the CRF (Conditional Random Field). The CRF determines whether two pixels in the image output by the decoding module belong to the same class by calculating the similarity between them, thereby refining the classification of the extracted results. The result obtained after CRF post-processing is used as the final output of the fusion network model.

[0086] In step 4) above, the loss value of the model training is obtained by using the cross-entropy loss function. The calculation of the loss function is as follows:

[0087]

[0088] Where J(θ) is the partial derivative of the parameter θ; y (i) It can be 0 or 1, where the specific value is determined by comparing each pixel of the image finally output by the fusion network model with the target image that can be used for text recognition prepared in advance for the output image in step 2) above.

[0089] In step 5) above, the structural similarity index (SSIM) is used as the evaluation metric for training the network model. The specific formula is as follows:

[0090]

[0091] Where x and y represent two images, and μ and δ represent different features. μ represents the variance of x. x c1 represents the mean, and c2 represents the deviation.

[0092] By training the fusion network model through steps 1) to 5) above, the fusion network model can use encoding and decoding modules to perform text and image enhancement processing on document image data that has been degraded due to natural and human factors, and solve the problem that complex and detailed information in document images is difficult to recover due to external environmental factors such as noise and blur.

[0093] Based on the document denoising methods of the above embodiments, this disclosure also provides a document denoising apparatus. The following will be combined with... Figure 6 and Figure 7 The device is described in detail.

[0094] Figure 6 A schematic block diagram of a document denoising apparatus 600 according to an embodiment of the present disclosure is shown.

[0095] like Figure 6 As shown, according to embodiments of the present disclosure, the document denoising apparatus 600 may include a fusion network model 610. According to other embodiments of the present disclosure, the document denoising apparatus 600 may further include an optical character recognition system 620. The document denoising apparatus 600 can perform the aforementioned references. Figures 2-4 The method described.

[0096] The fusion network model 610 receives a first document image as input, processes the first document image, and outputs a second document image, which serves as the input to the optical character recognition system. In one embodiment, the fusion network model 610 can perform the operation S220 described above.

[0097] The optical character recognition system 620 is used to receive a second document image output by the fusion network model 610 and recognize the text in the second document image.

[0098] Figure 7 An embodiment of the present disclosure is illustrated schematically. Figure 6 The diagram shows the structural block diagram of the fusion network model 610 in the document denoising device 600.

[0099] like Figure 7 As shown, the fusion network model 610 according to this embodiment may include an encoding module 611, a hole space pyramid pooling module 612, a decoding module 613, and a conditional random field 614.

[0100] The encoding module 611 is used to perform feature compression on the first document image and output the processing result to the hollow space pyramid pooling module 612.

[0101] The void space pyramid pooling module 612 is used to further fuse features from the output of the encoding module 611, and then output it to the decoding module 613.

[0102] The decoding module 613 is used to perform image reconstruction on the output of the void space pyramid pooling module 612, obtain preliminary reconstructed image data, and transmit it to the conditional random field 614.

[0103] The Conditional Random Field 614 is used to process the output of the decoding module 613 and output the second document image.

[0104] According to embodiments of the present disclosure, the encoding module 611 may include at least one convolutional layer, wherein each of the at least one convolutional layer includes a region generation network 6111 and a feature extraction sub-network 6112.

[0105] The region generation network 6111 is used to generate search boxes on the input feature map of the convolutional layer through a sliding window, and classify each search box into two categories: redundant and non-redundant. In one embodiment, the region generation network 6111 can perform the operation S301 described above.

[0106] The feature extraction subnetwork 6112 is used to extract features from the search boxes classified into non-redundant categories to obtain the output feature map of the convolutional layer. In one embodiment, the feature extraction subnetwork 6112 can perform the operation S302 described above.

[0107] According to embodiments of this disclosure, the fused network model 610 can be implemented using the methods described above. Figure 5 The training process shown is used for training, and after training, it can be applied to the document denoising method of this disclosure embodiment.

[0108] According to embodiments of this disclosure, any multiple modules among the fusion network model 610, optical character recognition system 620, encoding module 611, hollow spatial pyramid pooling module 612, decoding module 613, conditional random field 614, region generation network 6111, and feature extraction sub-network 6112 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functionality of one or more of these modules can be combined with at least some of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the following components—fusion network model 610, optical character recognition system 620, encoding module 611, hole space pyramid pooling module 612, decoding module 613, conditional random field 614, region generation network 6111, and feature extraction sub-network 6112—can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), programmable logic array (PLA), system-on-a-chip, system-on-a-substrate, system-on-package, application-specific integrated circuit (ASIC), or any other reasonable method of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three methods. Alternatively, at least one of the following components—fusion network model 610, optical character recognition system 620, encoding module 611, hole space pyramid pooling module 612, decoding module 613, conditional random field 614, region generation network 6111, and feature extraction sub-network 6112—can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0109] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a document denoising method according to embodiments of the present disclosure.

[0110] like Figure 8 As shown, an electronic device 800 according to an embodiment of this disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0111] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0112] According to embodiments of this disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0113] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0114] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0115] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0116] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0117] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0118] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0119] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0121] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0122] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A document denoising method, comprising: The first document image is input into the trained fusion network model, wherein the fusion network model includes an encoding module and a decoding module; The first document image is feature compressed using the encoding module; and The decoding module restores the output of the encoding module to an image and outputs a second document image, which is used as the input of the optical character recognition system. The encoding module includes at least one convolutional layer, wherein the convolutional layer includes a region generation network and a feature extraction sub-network; wherein the feature compression of the first document image using the encoding module includes: The region generation network is used to generate search boxes on the input feature map of the convolutional layer through a sliding window, and each search box is classified into two categories: redundant and non-redundant. The search boxes classified as redundant are removed, and only the search boxes classified as non-redundant are provided to the feature extraction subnetwork. The feature extraction subnetwork is used to extract features from the search boxes classified into non-redundant categories to obtain the output feature map of the convolutional layer.

2. The method according to claim 1, wherein, The classification of each search box using the region generation network includes: Each search box is classified based on a nonmaximum suppression algorithm.

3. The method according to claim 1, wherein, The step of using the feature extraction subnetwork to extract features from search boxes classified into non-redundant categories includes: The search boxes classified into non-redundant categories are divided into parts of the same size as the dimension of the output feature map using the Region of Interest (ROI) subnetwork, and the maximum value of each part is used as the output of each position in the output feature map.

4. The method according to claim 1, wherein, The step of using the encoding module to perform feature compression on the first document image includes: Through dense connections, each convolutional layer in the encoding module, except for the last convolutional layer, inputs its output into all subsequent convolutional layers.

5. The method according to any one of claims 1 to 4, wherein, The fusion network model further includes a hollow spatial pyramid pooling module. Before using the decoding module to perform image reconstruction on the output of the encoding module, the method further includes: The output of the encoding module is used as the input of the dilated spatial pyramid pooling module, and feature fusion is performed using the dilated spatial pyramid pooling module; and The output of the void space pyramid pooling module is input into the decoding module.

6. The method according to claim 5, wherein, The fusion network model further includes a conditional random field. After using the decoding module to reconstruct the image from the output of the encoding module, the method further includes: The output of the decoding module is used as the input of the conditional random field, and the second document image is output after processing by the conditional random field.

7. The method according to claim 1, wherein, The method further includes: The optical character recognition system is used to identify the text in the second document image.

8. A document denoising apparatus, comprising: A fusion network model is used to receive a first document image as input, process the first document image, and output a second document image, wherein the second document image is used as input to an optical character recognition system; The fusion network model includes an encoding module and a decoding module. The encoding module is used for feature compression of the first document image; and The decoding module is used to restore the image from the output of the encoding module and then output the second document image. The encoding module includes at least one convolutional layer, wherein the convolutional layer includes a region generation network and a feature extraction sub-network; The region generation network is used to: generate search boxes on the input feature map of the convolutional layer through a sliding window, and classify each search box, including two categories: redundant and non-redundant; and remove the search boxes classified as redundant, and only provide the search boxes classified as non-redundant to the feature extraction subnetwork. The feature extraction subnetwork is used to extract features from search boxes classified into non-redundant categories to obtain the output feature map of the convolutional layer.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method of any one of claims 1 to 7.

11. A computer program product comprising computer program instructions that, when executed by a processor, implement the method of any one of claims 1 to 7.

Images