A text image deblurring method based on deep learning

Abstract

The application discloses a text image deblurring method based on deep learning, relates to the field of computer image recognition, and mainly solves the problem of realizing deblurring of blurred text images to obtain clear text images through a deep learning mode. The method comprises the following steps: step 1, inputting a pair of blurred-clear text image datasets into a deep residual neural network for training; step 2, continuously iteratively training until objective evaluation indexes gradually converge, and obtaining a text image deblurring model; and step 3, inputting a blurred text image into the deep residual neural network, processing the blurred text image through the deblurring model, and obtaining a corresponding deblurred text image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer image recognition, more particularly to the technical field of a text image deblurring method based on deep learning. BACKGROUND

[0002] Image deblurring is a classic task in the field of low-level computer vision, which has been widely applied in theory and practice, such as target detection, medical imaging, OCR, public place monitoring, face detection, etc. Due to the low imaging pixels of the acquisition device or the shaking of the device during image acquisition, the image is blurred, which increases the inconvenience of the corresponding image recognition task.

[0003] Text images usually contain words, letters, symbols, tables, etc. When using mobile devices to scan and obtain documents or take screenshots, it is very likely to cause the occurrence of content blurring, noise superposition and other problems, so that the obtained text images cannot actually play a role.

[0004] Traditional image deblurring represents the task as an inverse filtering problem, in which the blurred image is modeled as the result of convolution with a blur kernel, either spatially invariant or spatially varying. Some early methods assume that the blur kernel is known and use classical image deconvolution algorithms to recover the sharp image.

[0005] In recent years, with the development of deep learning technology, the application in the field of image deblurring has been greatly improved. For example, the existing technology, a high-resolution image restoration method based on Transformer: Restormer, mainly includes the following steps:

[0006] Step S1: Collect pairs of clear-blurred images and input them into a generative adversarial network for training;

[0007] Step S2: Input the data set into the neural network and continuously iterate until convergence to obtain a deblurring model with good objective evaluation indicators;

[0008] Step S3: Input the blurred image into the neural network architecture and use the deblurring model obtained in step S2 to obtain the deblurred image

[0009] The overall framework of the Restormer network architecture is a u-net structure with global residual connection. First, low-order features F0 are extracted through 3*3 convolution; then, the encoder-decoder coding result composed of 4-stage transformer is obtained, wherein the up-sampling and down-sampling are realized by pixel unshuffle and shuffle respectively; then, refinement is performed through transformer; finally, the original channel number is restored through 3*3 convolution, and the output is obtained by adding the residual of the original image.

[0010] The coding and decoding modules in the network framework are all Transformer Blocks, which mainly include two modules GDFN (Gated Deconvolutional Feedforward Network) module and MDTA (Multi-Head Attention) module. The GDFN module performs controlled feature conversion, i.e. suppresses features with less information and only allows useful information to pass through the network hierarchy. The MDTA module can aggregate local and non-local pixel interactions and is efficient enough to process high-resolution images.

[0011] However, the current deep learning model architecture usually has very high network depth and very large model parameters, and is mostly trained on multiple GPUs, which makes it difficult to apply to actual applications. In addition, in order to pursue better objective evaluation indicators, the essential key problem of deblurring is ignored, and the subjective effect of the restored image is not very good. Therefore, it has great research significance and application value to study a simpler, less parameter, faster training speed and better subjective feeling deblurring method. SUMMARY

[0012] The purpose of the present application is to provide a text image deblurring method based on deep learning, which realizes the deblurring of blurred text images to obtain clear text images through deep learning.

[0013] The present application specifically adopts the following technical solutions to achieve the above purpose:

[0014] A text image deblurring method based on deep learning, comprising the following steps:

[0015] Step 1: input a pair of blurred-clear text image dataset into a deep residual neural network for training;

[0016] Step 2: continuously iterate the training until the objective evaluation indicators gradually converge, and obtain a text image deblurring model;

[0017] Step 3: input the blurred text image into the deep residual neural network, process it through the deblurring model, and obtain the corresponding deblurred text image.

[0018] As an optional technical solution, in step 1, the pair of blurred-clear text images are trained and predicted using the Textlmage dataset.

[0019] As an optional technical solution, the Textlmage dataset contains 1000 text images of different languages and different sizes of text, images, tables, equations, charts, and each text image has a resolution of more than 3500*5000. The 1000 images are divided according to the training test ratio of 9:1, 900 images are used for deblurring model training, and 100 images are used for image deblurring test.

[0020] As an optional technical solution, in step 1, the deep residual neural network adopts the encoding-decoding network structure of Unet architecture, which contains residual convolution encoding-decoding modules and different depth feature fusion modules. The first, second and third layers of the encoding part all adopt 1 block residual convolution module, and the fourth layer adopts 10 blocks of residual convolution module. At the bottom of the network architecture, 1 convolution module is used. The first, second, third and fourth layers of the decoding part all adopt 1 block residual convolution module.

[0021] As an optional technical solution, the residual convolution module is used to extract image features, wherein the encoding part contains an FFT2d module, and its formula is:

[0022]

[0023] Where X[N] is a sequence of N numbers, X[k] represents the frequency spectrum with frequency wk=2πk / N, and j represents the imaginary unit.

[0024] As an optional technical solution, the simplified channel attention module in the residual convolution module has the formula:

[0025] SCA(X)=X*W pool (x) (2)

[0026] Where SCA represents using a simplified channel attention to extract features from input features X, W pool represents using a 1x1 convolution layer to perform global pooling on the feature map to extract features.

[0027] As an optional technical solution, the SimleGate module in the residual convolution module uses 1x1 convolution operation to realize information exchange between channels, and its formula is:

[0028] SimpleGate(X, Y)=X⊙Y (3)

[0029] Where X and Y represent feature maps of the same size, and represents the product sum of corresponding elements of X and Y.

[0030] As an optional technical solution, the different depth feature fusion module is used for enhancing feature fusion when different scale features are connected, and the formula of the different depth feature fusion module is:

[0031]

[0032] Among them, is the feature fusion module of the nth layer, is the first layer feature, is the second layer feature, is the third layer feature, and ↑ represents up sampling.

[0033] As an optional technical solution, in step 2, when the deblurring model is trained, the loss function used contains L1 pixel loss and multi-scale structural similarity loss MS_SSIM, and the calculation formula of the loss function is:

[0034]

[0035] Among them, represents the total loss, and alpha represents a hyperparameter, which is 0.24 in this example, represents the MS_SSIM loss, is the Gaussian kernel for calculating the Mth scale in MS_SSIM, represents the L1 loss.

[0036] As an optional technical solution, when the image deblurring model is trained, an AdamW optimizer is used to optimize each network parameter of the image deblurring model, and first-order momentum and second-order momentum and weight decay are used to dynamically adjust the learning rate of each parameter.

[0037] The beneficial effects of the present application are as follows:

[0038] 1. The network model used in the prior art has many parameters, and when the input image size is a 255x255 resolution image, the total amount of model parameters of the prior art Restormer model is about 46.18M, while the model parameters proposed in the present application are only 8.7M;

[0039] 2. The network model used in the prior art has high computational complexity, and when the input image size is a 255x255 resolution image, the computational complexity of the prior art Restormer model is about 247.43Gmac, while the model parameters proposed in the present application are only 8.72Gmac;

[0040] 3. The prior art requires many devices, and the cost is high in practical application. The Restormer model in the prior art needs 8 Nvidia V100 graphics cards for parallel training for about 4 days when training the same size data set, while the model proposed in the application only needs 1 Nvidia V100 graphics card for single-card training for about 2.5 days to achieve the same image objective evaluation index.

[0041] 4. The prior art is difficult to implement in practical engineering application. The deblurring model trained by the Restormer model in the prior art is about 240Mb, which needs to occupy a large space in actual deployment, and it is difficult to deploy and implement application when deployed to mobile devices. The model trained in the application only occupies about 34.9Mb, which can be more conveniently transplanted to portable mobile devices in actual deployment, which is conducive to the engineering practical application of the technology and plays a higher engineering value. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a flowchart of the method of the application.

[0043] Figure 2 is a network structure diagram of the method of the application.

[0044] Figure 3 is a convolution residual block in the network architecture of the method of the application.

[0045] Figure 4 is a different depth feature fusion module in the network architecture of the method of the application.

[0046] Figure 5 is the blurred text image to be input in embodiment 1.

[0047] Figure 6 is the image after deblurring the image by the method described in embodiment 1. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, not all the embodiments. The components of the embodiments of the application described and shown in the drawings can be arranged and designed in various different configurations.

[0049] Therefore, the following detailed description of the embodiments of the application provided in the drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the application.

[0050] Embodiment

[0051] A deep learning-based text image deblurring method, referring to Figure 1 , comprising the following steps:

[0052] Step S1: input a pair of blurred-clear text image dataset into a deep residual neural network for training;

[0053] In this embodiment, the pair of blurred-clear text images are trained and predicted using the Textlmage dataset, and the experimental equipment is an NVIDIA TESLA V100. The Textlmage dataset contains 1000 text images of different languages and different sizes of text, images, tables, equations, and charts, and the resolution of each text image exceeds 3500*5000. The 1000 images are divided according to the training test ratio of 9:1, 900 images are used for deblurring model training, and 100 images are used for image deblurring test.

[0054] The deep residual neural network architecture is as shown in Figure 2 , which adopts a coding-decoding network structure of Unet architecture, including residual convolution coding-decoding modules and different depth feature fusion modules, Figure 2 The left half of the deep network architecture, i.e. the coding part, the first, second and third layers all adopt 1 residual convolution module, and the fourth layer adopts 10 residual convolution modules; at the bottom of the network architecture, 1 convolution module is adopted; the right half of the network architecture, i.e. the decoding part, the first, second, third and fourth layers all adopt 1 residual convolution module.

[0055] The residual convolution module (as shown in Figure 3 ) is used to extract image features, wherein the upper left half part contains an FFT2d (Fourier convolution) module, and its formula is:

[0056]

[0057] Where X[N] is a sequence of N complex numbers, X[k] represents the frequency spectrum with wk=2πk / N, and j represents the imaginary unit.

[0058] The simplified channel attention (SCA) module in the residual convolution module (as shown in Figure 3 ) has the formula:

[0059] SCA(X) = X*W pool (x)...(2)

[0060] Wherein, SCA represents the early feature of the input feature X using the simplified channel attention, W pool represents the global pooling of the feature map using the 1x1 convolution layer.

[0061] The SimleGate module in the residual convolution module (as shown in Figure 3 The formula of the SimleGate module is:

[0062] SunpleGate(X, Y) = x Y...(3)

[0063] Wherein, X and Y represent feature maps of the same size, and represents the product sum of the corresponding elements of X and Y.

[0064] The different depth feature fusion module (as shown in Figure 4 The different depth feature fusion module is used to enhance the feature fusion when connecting different scale features, and the formula of the different depth feature fusion module is:

[0065]

[0066] Wherein, is the feature fusion module of the nth layer, is the first layer feature, is the second layer feature, is the third layer feature, and ↑ represents up sampling

[0067] Step S2: through continuous iteration training, until the objective evaluation index gradually converges, to obtain the deblurring model of the text image.

[0068] When training the deblurring model, the loss function used includes L1 pixel loss and multi-scale structural similarity loss MS_SSIM, and the calculation formula of the loss function is:

[0069]

[0070] Wherein, represents the overall loss, and α represents a hyperparameter, which is 0.24 in this example, represents the MS_SSIM loss, is the Gaussian kernel for calculating the Mth scale in MS_SSIM, represents the L1 loss.

[0071] When training the image deblurring model, the AdamW optimizer is used to optimize each network parameter of the image deblurring model, and the first order momentum and the second order momentum and weight decay are used to dynamically adjust the learning rate of each parameter.

[0072] Step S3: input the blurred text image into the deep residual neural network, process it through the deblurring model, and obtain the corresponding deblurred text image.

[0073] This example will Figure 5 The deblurred image recovered after inputting the deep residual neural network and deblurring processing is Figure 6 , which is more clear and identifiable than the text texture in the original image, has better actual visual sensory effect, and has higher use value. The high efficiency of the method is proved by the evaluation indexes peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). The deep residual neural network is compared with the most advanced methods Convolutional Neural Networks for Direct Text Deblurring (TextDeblur), Simple Baselines for Image Restoration (NAFNet), and Restormer: Efficient Transformer for High-Resolution Image Restoration (Restormer). The experimental results are shown in Table 1, and the results are test results under the same experimental conditions. The deblurring performance, model size, and running time of the deep residual neural network all exceed those of the existing most advanced methods.

[0074] Table 1

[0075] Method PSNR / SSIM Model size (MB) Running time (s) TextDeblur 25.3572 / 0.8758 60.5 4.7674 Restormer 28.7976 / 0.9739 99.9 2.3998 NAFNet 29.5637 / 0.9725 272 1.8691 ours 31.1527 / 0.9854 34.9 0.8735

[0076] The above only describes the preferred embodiments of the present application and does not limit the present application. Any modification, equivalent replacement, and improvement within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A deep learning-based text image deblurring method, characterized in that, The method comprises the following steps: Step 1: input a pair of fuzzy-clear text image dataset into a deep residual neural network for training; Step 2: through continuous iteration training until the objective evaluation index gradually converges, a text image deblurring model is obtained; Step 3: input a fuzzy text image into the deep residual neural network, and process the fuzzy text image through the deblurring model to obtain a corresponding deblurred text image; In the step 1, the deep residual neural network adopts an encoding-decoding network structure of Unet architecture, and comprises a residual convolution encoding-decoding module and a different depth feature fusion module; the first, second and third layers of the encoding part each adopt one residual convolution module, and the fourth layer adopts ten residual convolution modules; at the bottom of the network architecture, one convolution module is adopted; the first, second, third and fourth layers of the decoding part each adopt one residual convolution module; The residual convolution module is used for extracting image features, wherein the encoding part comprises an FFT2d module, and the formula of the FFT2d module is: （1） Wherein, X[N] is a sequence of N complex numbers, X[k] represents a frequency spectrum with a frequency of wk=2πk / N, and j represents an imaginary unit; In the step 2, when the deblurring model is trained, the loss function used comprises an L1 pixel loss and a multi-scale structural similarity loss MS_SSIM, and the calculation formula of the loss function is: (5) wherein, denotes the overall loss, denotes a hyperparameter, which in the present example takes the value 0.24, denotes the MS_SSIM loss, is a Gaussian kernel that computes the M-th scale in the MS_SSIM, denotes the L1 loss.

2. The method of claim 1, wherein the method is based on deep learning. In the step 1, the pair of fuzzy-clear text images are trained and predicted by using a TextImage dataset.

3. The method of claim 2, wherein the method further comprises: The TextImage dataset comprises 1000 text images of different languages and different sizes of text, images, tables, equations and charts, and the resolution of each text image is more than 3500*5000; the 1000 images are divided according to a training test ratio of 9:1, 900 images are used for deblurring model training, and 100 images are used for image deblurring test.

4. The method of claim 1, wherein the method is based on deep learning. The simplified channel attention module in the residual convolution module has the formula: （2） wherein SCA represents using a simplified channel attention to pre-features of the input feature X, represents using a 1x1 convolutional layer to globally pool the feature map pre-feature.

5. The method of claim 1, wherein the method is based on deep learning. The SimleGate module in the residual convolution module realizes information exchange between channels by using a 1x1 convolution operation, and the formula is: （3） Wherein, X and Y represent feature maps of the same size, and represents the product sum of corresponding elements of X and Y.

6. The method of claim 1, wherein the method is based on deep learning. The different depth feature fusion module is used for enhancing feature fusion when connecting different scale features, and the formula of the different depth feature fusion module is: （4） wherein, is a feature fusion module of the n-th layer, is a first layer feature, is a second layer feature, is a third layer feature, and ↑ denotes up-sampling.

7. The method of claim 1, wherein the method is based on deep learning. When the image deblurring model is trained, an AdamW optimizer is used to optimize each network parameter of the image deblurring model, and the first-order momentum and the second-order momentum and weight decay are used to dynamically adjust the learning rate of each parameter.

Images