A natural scene text erasure method based on multi-stage neural network

Abstract

The present invention discloses a method for erasing text in natural scenes based on a multi-stage neural network, and the steps include: 1. collecting natural scene text images and establishing a natural scene text erasure data set; 2. constructing a multi-stage text erasure network; 3. running the multi-stage text erasure network; 4. calculating the difference between the erasure result of each input image and the corresponding annotated true result according to the loss function, and using the back propagation algorithm to supervise the training of the multi-stage text erasure network; 5. using the trained multi-stage text erasure network to perform erasure processing on any natural scene image with text. The present invention adopts a multi-stage neural network learning method, which can perform multiple erasure processing on the text in the natural scene and merge these results together to form the final erasure result, thereby improving the text erasure effect.

Description

Technical Field

[0001] The present invention relates to the field of image text removal, and in particular to a natural scene text erasing method based on a multi-stage neural network. Background Art

[0002] There are currently two main methods for erasing text in natural scenes: text erasure methods based on traditional algorithms and text erasure methods based on deep learning. The text erasure method based on deep learning is much better than the text erasure method based on traditional algorithms in terms of both erasure effect and robustness. However, the effect of the text erasure method based on deep learning still has a lot of room for improvement. In some common natural scenes, complex text is difficult to completely remove. For example, pictures with small text areas and pictures where the text appearance is very different from the surrounding scene cannot be effectively removed by the existing text erasure method based on deep learning. Summary of the invention

[0003] The present invention aims to solve the deficiencies of the above-mentioned prior art and proposes a natural scene text erasure method based on a multi-stage neural network, so as to perform multiple erasure processes on the text in the natural scene and merge the results together, thereby improving the text erasure effect.

[0004] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0005] The invention provides a method for erasing text in natural scenes based on a multi-stage neural network, which comprises the following steps:

[0006] Step 1: Collect a set of text pictures of real natural scenes, denoted as X, where the text picture set X contains n pictures X1, X2, ..., X i , ..., X n , where Xi represents the i-th text picture; the text picture set X is manually erased to obtain a text-free label picture set denoted as Y; the label picture set Y includes n label pictures Y1, Y2, ..., Y i , ..., Y n ; Among them, Y i Represents the i-th label picture;

[0007] Step 2: Construct a multi-stage text erasing network, including: a high-pixel image erasing module HE, a medium-pixel image erasing module ME, a low-pixel image erasing module LE, and a multi-stage image fusion module MF;

[0008] The high-pixel image erasing module HE includes: a high-resolution trunk network and two low-resolution branch networks; wherein the high-resolution trunk network contains 3f downsampling networks, the low-resolution branch network 1 contains f upsampling networks and f downsampling networks, and the low-resolution branch network 2 contains 2f upsampling networks;

[0009] The mid-pixel picture erasing module ME includes: m encoding modules E, m decoding modules D and k-layer convolutional networks C; wherein each encoding module and decoding module includes h convolutional networks;

[0010] The low-pixel image erasing module LE includes: m encoding modules E', m decoding modules D' and k-layer convolutional networks C'; wherein the encoding module E' and the decoding module D' each include h convolutional networks;

[0011] The multi-stage image fusion module MF comprises: i-layer long short-term memory network LSTM;

[0012] Step 3: Run the multi-stage text erasing network:

[0013] Step 3.1: Run the high-pixel image erasing module HE;

[0014] The i-th text image x i The high-resolution image is input into the high-pixel image erasing module HE, and is sequentially processed by the convolution of the f-layer downsampling network of the high-resolution backbone network to obtain a high-resolution feature map F with a size of C×H×W. i , where C is the number of channels of the input image, H and W are the height and width of the input image respectively; then the high-resolution feature map F i Input into two f-layer downsampling networks respectively, and the corresponding size is The feature map F i1 And the size is The feature map F i2 ;

[0015] The feature map F i1 Input into the low-resolution branch network 1, perform f upsampling processes and f downsampling processes respectively, and obtain the upsampled feature map F′ with a size of C×H×W i1 and size is The downsampled feature map F″ i1 ;

[0016] The feature map F i2 Input into the low-resolution branch network 2, and perform two f-layer upsampling processes to obtain an upsampled feature map F′ of size C×H×W i2 and size is The upsampled feature map F″ i2 ;

[0017] Finally, the high-resolution feature map Fi is upsampled to the feature map F′ i1 And the upsampled feature map F′ i2 Add the result to get the feature map F of the first stage output with the size of C×H×W i，stage1 ;

[0018] Step 3.2: Running pixel image erasing module ME;

[0019] For the i-th text image x i Divide into two equal parts and get The left bipartite graph x i，left and the right bipartite graph x i，right Then input into the middle pixel picture erasing module ME; the left bipartite image x i，left and the right bipartite graph x i，right After being processed by m encoding modules E in turn, the size is The left encoded feature map F i，en_left and the right-encoded feature map F of the same size i，en_right ;

[0020] The left encoded feature map F i，en_left and the right encoded feature map F i，en_right The images are concatenated along their third dimension to give a size of The encoding feature map F i，encoder ;

[0021] The encoding feature map F i，encoder Input into m decoding modules D to obtain a decoding feature map F of size 3C×H×W i，decoder ;

[0022] The decoding feature map F i，aecoder Input into the k-layer convolutional network C for convolution transformation to obtain the feature map F of the second stage output with different dimensions C×H×W i，stage2 ;

[0023] Step 3.3: Run the low-pixel image erasing module LE;

[0024] For the i-th text image x i Divide into four equal parts, and get The four quadrants of x i，upleft 、x i，upright 、x i，downleft and x i，downright Then input into the low pixel image erasing module LE; four quadrant images x i，upleft 、x i，upright 、x i，downleft and x i，downrightAfter being processed by m encoding modules E' in sequence, the corresponding sizes are The encoding feature map F i，en_upleft 、F i，en_upright 、F i，en_downleft and F i，en_downright ;

[0025] The four encoded feature maps F i，en_upleft 、F i，en_upright 、F i，en_downleft and F i，en_downright The image is concatenated along the second and third dimensions to obtain a size of The encoding feature map F′ i，encoder ;

[0026] The encoding feature map F′ i，encoder Input into m decoding modules D' for processing, and obtain the decoding feature map F' of size 3C×H×W i，decoder ;

[0027] The decoding feature map F′ deroder The k-layer convolutional network C' is input sequentially for convolution transformation to obtain the feature map F of the third stage output with different dimensions C×H×W. i，stage3 ;

[0028] Step 3.4: Run the multi-stage image fusion module MF;

[0029] The feature maps F output by the three stages i，stage1 、F i，stage2 and F i，stage3 The images are input into the multi-stage image fusion module MF together, and after being processed by the j-layer long short-term memory LSTM network, the j-th layer outputs a fusion feature map F with a size of C×H×W. i，LSTM , the fusion feature map F i，LSTM After a layer of convolutional network scale change, the text erasure result with a size of 3×H×W is obtained.

[0030] Step 4: Perform supervised training on the multi-stage text erasing network;

[0031] Step 4.1: Calculate the loss function of the text image set X of natural scenes

[0032] Using formula (1) and formula (2), we can establish the loss function Loss char and Loss edge :

[0033]

[0034]

[0035] In equations (1) and (2), Δ represents the Laplace transform;

[0036] Step 4.2: Train a multi-stage text erasing network;

[0037] The back propagation algorithm is used to iteratively train the parameters in the multi-stage text erasure network and calculate the overall loss function Loss total =Loss char +λLoss edge , until the iteration reaches the iteration threshold, and finally the trained text erasure network is obtained; where λ represents the weight parameter;

[0038] Step 5: Use the trained multi-stage text erasure network to process any natural scene image with text to obtain a natural scene image without text after erasure.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. Compared with the general text erasing method, the multi-stage text erasing network of the present invention has a significant improvement in erasing effect, especially for small text images and complex background images.

[0041] 2. The present invention can have a strong erasing capability for small text areas while retaining the non-text background information of the image through the high-pixel image erasing module HE, thereby achieving a better text erasing effect.

[0042] 3. The present invention utilizes the medium pixel image erasing module ME and the low pixel image erasing module LE to effectively learn the location information of larger text and extract rich background texture features around the text as a supplement to the high pixel image erasing module HE.

[0043] 4. The present invention fuses the erasure results of the three stages together through a multi-stage image fusion module MF. The image fusion method can fuse the erasure results of different stages together, remove the wrong parts of the results of each stage, retain the correct results of erasure, and finally generate a text-free natural scene image. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of the use of the natural scene text erasing method based on the multi-stage neural network of the present invention;

[0045] Figure 2 It is a network structure diagram of the natural scene text erasing method based on a multi-stage neural network of the present invention. DETAILED DESCRIPTION

[0046] In this embodiment, Figure 1As shown, a method for erasing text in natural scenes based on a multi-stage neural network is to obtain image erasing results in three stages by training three different neural networks, and then fuse the output results of the three stages to obtain a final natural scene image without text. Specifically, the method includes the following steps:

[0047] Step 1: Collect natural scene text images and construct a text erasure dataset. Preprocess the collected natural scene images with text. First, scale them into a fixed-size image set with a width of W and a height of H, and then normalize the scaled image set, that is, divide each pixel value on the image by 255, so that the pixel value of the image is normalized to between 0 and 1, which is convenient for the network to learn. The normalized natural scene text image set is used as a training data set and is denoted as X. The text image set X contains n images x1, x2, ..., x i , …, x n , where x i Denotes the i-th text image. The collected natural scene text images are erased by manual labeling to create label data, which is recorded as Y. The label image set Y contains n label images y1, y2, ..., y i , ..., y n ; Among them, y i Represents the i-th label image.

[0048] Step 2: Construct a multi-stage text erasing network, which includes four modules: high-pixel image erasing module HE, medium-pixel image erasing module ME, low-pixel image erasing module LE, and multi-stage image fusion module MF;

[0049] The high-pixel image erasing module HE includes: a high-resolution backbone network and two low-resolution branch networks; wherein the high-resolution backbone network contains 3f downsampling networks, the low-resolution branch network 1 contains f upsampling networks and f downsampling networks, and the low-resolution branch network 2 contains 2f upsampling networks;

[0050] The pixel image erasing module ME includes: m encoding modules E, m decoding modules D and k-layer convolutional networks C. Among them, the encoding module E and the decoding module D both include h-layer convolutional networks;

[0051] The low-pixel image erasing module LE includes: m encoding modules E', m decoding modules D' and k-layer convolutional networks C'; wherein the encoding module E' and the decoding module D' each include h convolutional networks;

[0052] The multi-stage image fusion module MF includes: i-layer long short-term memory network LSTM;

[0053] Step 3: Run the multi-stage text erasing network, such as Figure 2 As shown:

[0054] Step 3.1: Run the high-pixel image erasing module HE

[0055] The i-th text image x i The input high-pixel image is sent to the HE module, and then it is processed by the convolution of the f-layer downsampling network of the high-resolution backbone network to obtain a high-resolution feature map F with a size of C×H×W. i , where C is the number of channels of the input image, H and W are the height and width of each input image; then the high-resolution feature map F i Input into two f-layer downsampling networks respectively, and the corresponding size is The feature map F i1 And the size is The feature map F i2 ;

[0056] Feature map F i1 Input into the low-resolution branch network 1, perform f upsampling processes and f downsampling processes respectively, and obtain the upsampled feature map F′ with a size of C×H×W i1 and size is The downsampled feature map F″ i1 ;

[0057] Feature map F i2 Input into the low-resolution branch network 2, and perform two f-layer upsampling processes to obtain an upsampled feature map F′ of size C×H×W i2 and size is The upsampled feature map F″i2;

[0058] Finally, the high-resolution feature map Fi is upsampled to the feature map F′ i1 And the upsampled feature map F′ i2 Add the result to get the feature map F of the first stage output with the size of C×H×W i，stage1 ;

[0059] Multi-branch fusion generates the first stage output feature map F i，stage1 The steps make each low-resolution branch network feature information gradually added to the high-resolution backbone network, thereby effectively maintaining the high-resolution feature information. The high-resolution backbone network and low-resolution branch network connected in parallel and the multiple multi-scale fusion methods can obtain rich high-resolution feature information. The high-pixel image erasing module HE can always maintain the high-resolution feature information on the backbone network. No feature compression is performed in the backbone network, so there is no information loss. This network structure can effectively improve the erasing effect of pixel-level text erasing tasks.

[0060] Step 3.2: Running pixel image erasing module ME

[0061] For the i-th text image x i Divide into two equal parts and get The left bipartite graph x i，left and the right bipartite graph x i，right Then input into the middle pixel image erasing module ME; left bipartite image x i，left and the right bipartite graph x i，right After being processed by m encoding modules E in turn, the size is The left encoded feature map F i，en_left and the right-encoded feature map F of the same size i，en_right ;

[0062] The left encoded feature map F i，en_left and the right encoded feature map F i，en_right The images are concatenated along their third dimension to give a size of The encoding feature map F i，encoder ;

[0063] Encoded feature map F i，encoder Input into m decoding modules D to obtain a decoding feature map F of size 3C×H×W i，decoder ;

[0064] Decoding feature map F i，decoder Input into the k-layer convolutional network C for convolution transformation to obtain the feature map F of the second stage output with different dimensions C×H×W i，stage2 ;

[0065] The mid-pixel image erasing module ME uses the bisection of the input image for training, which can effectively reduce the size of the model and reduce the amount of model calculation. Using the bisection image for training will result in a smaller feature map, which has better erasing capabilities for larger text.

[0066] Step 3.3: Run the low-pixel image erasing module LE;

[0067] For the i-th text image x i Divide into four equal parts, and get The four quadrants of x i，upleft 、x i，upright 、x i，downleft and x i，downright Then input into the low-pixel image erasing module LE; four quadrant images x i，upleft 、x i，upright 、x i，downleft and x i，downright After being processed by m encoding modules E' in sequence, the corresponding sizes are The encoding feature map F i，en-upleft 、F i，en-upright 、F i，en-downleft and F i，en-downright ;

[0068] The four encoded feature maps F i，en-upleft 、F i，en-upright 、F i，en_downleft and F i，en_downright The image is concatenated along the second and third dimensions to obtain a size of The encoding feature map F′ i，encoder ;

[0069] Encoded feature map F′ i，encoder Input into m decoding modules D' for processing, and obtain the decoding feature map F' of size 3C×H×W i，decoder ;

[0070] Decoding feature map F′ decoder The k-layer convolutional network C' is input sequentially for convolution transformation to obtain the feature map F of the third stage output with different dimensions C×H×W. i，stage3 ;

[0071] The low-pixel image erasing module HE uses the quadrants of the input image for training, which has a smaller amount of computation and a faster training speed than the medium-pixel image erasing module ME.

[0072] Step 3.4: Run the multi-stage image fusion module MF;

[0073] The feature maps F output by the three stages i，stage1 、F i，stage2 and F i，stage3 The images are input into the multi-stage image fusion module MF together, and after being processed by the j-layer long short-term memory LSTM network, the j-th layer outputs a fusion feature map F with a size of C×H×W. i，LSTM , fusion feature map F i，LSTM After a layer of convolutional network scale change, the text erasure result with a size of 3×H×W is obtained.

[0074] Based on the multi-stage text erasure network method, multiple text erasure networks can be trained, and a similar ensemble learning method can be used to obtain the final text erasure model. The outputs of the three stages in the multi-stage text erasure network represent three different text erasure results F i，stage1 、F i，stage2 and F i，stage3 The final output will refer to the erasing results of these three stages at the same time. The correct parts of the three-stage erasure results are gathered together, and the wrong parts are removed at the same time, so as to generate the final result. In this embodiment, an LSTM network is used as the multi-stage image fusion module MF. The LSTM network can effectively process data with sequence changes, control the transmission state through the gate state, remember the effective information that needs to be remembered for a long time, and forget the unimportant information. Therefore, the three different text erasure F i，stage1 、F i，stage2 and F i，stage3 Input into the multi-stage image fusion module MF to generate the final erasing result

[0075] Step 4: Perform supervised training on the multi-stage text erasing network;

[0076] Step 4.1: Calculate the loss function of the text image set X of natural scenes;

[0077] Using formula (1) and formula (2), we can establish the loss function Loss char and Loss edge :

[0078]

[0079]

[0080] In equations (1) and (2), Δ represents Laplace transform.

[0081] Step 4.2: Train a multi-stage text erasing network;

[0082] The back propagation algorithm is used to iteratively train the parameters in the multi-stage text erasure network and calculate the overall loss function Loss total =Loss char +λLoss edge , until the iteration reaches the iteration threshold, and finally the trained text erasure network is obtained; where λ represents the weight parameter;

[0083] Step 5: Use the trained multi-stage text erasure network to process any natural scene image with text to obtain a natural scene image without text after erasure.

Claims

1. A natural scene text erasing method based on a multi-stage neural network, which is characterized by the following steps: Step 1: Collect a collection of text pictures of real natural scenes, denoted as , the text picture collection Contains n pictures ,in, Represents the i-th text picture; the text picture collection The set of labeled images without text obtained by manual erasure is recorded as ; The label picture collection Contains n labeled images ;in, Represents the i-th label picture; Step 2: Construct a multi-stage text erasure network, including: a high-pixel image erasure module HE, a medium-pixel image erasure module ME, a low-pixel image erasure module LE and a multi-stage image fusion module MF; The high-pixel image erasing module HE includes: a high-resolution trunk network and two low-resolution branch networks; wherein the high-resolution trunk network contains 3f downsampling networks, the low-resolution branch network 1 contains f upsampling networks and f downsampling networks, and the low-resolution branch network 2 contains 2f upsampling networks; The mid-pixel picture erasing module ME comprises: m encoding modules E, m decoding modules D and k layers of convolutional networks C; wherein each encoding module and decoding module comprises h convolutional networks; The low-pixel image erasing module LE includes: m encoding modules E', m decoding modules D' and k-layer convolutional networks C'; wherein the encoding module E' and the decoding module D' each include h convolutional networks; The multi-stage image fusion module MF includes: j layers of long short-term memory network LSTM; Step 3: Run the multi-stage text erasing network: Step 3.1: Run the high-pixel image erasing module HE; The i-th text picture The high-pixel image is input into the HE module, and then processed by the convolution of the f-layer downsampling network of the high-resolution backbone network to obtain a size of High-resolution feature map , where C is the number of channels of the input image, H and W are the height and width of the input image respectively; then the high-resolution feature map Input into two f-layer downsampling networks respectively, and the corresponding size is Feature map And the size is Feature map ; The feature map Input into the low-resolution branch network 1, perform f upsampling and f downsampling respectively, and obtain a size of Upsampled feature map and size is Downsampled feature map ; The feature map Input to the low-resolution branch network 2, and perform two f-layer upsampling processes to obtain a size of Upsampled feature map and size is Upsampled feature map ; Finally, the high-resolution feature map , upsampled feature map And upsampled feature maps Add the two together to get the size The feature map of the first stage output ; Step 3.2: Running pixel image erasing module ME; For the i-th text picture Divide into two equal parts and get The left bipartite graph of Right bipartite graph Then input into the middle pixel picture erasing module ME; the left bipartite picture Right bipartite graph After being processed by m encoding modules E in turn, the size is The left encoded feature map and the right encoded feature map of the same size ; The left encoded feature map and right encoded feature map The images are concatenated along their third dimension to give a size of The encoding feature map of ; The encoding feature map Input into m decoding modules D, and get the size Decoding feature map of ; The decoding feature map Input into the k-layer convolutional network C for convolution transformation to obtain different dimensions C The feature map of the second stage output ; Step 3.3: Run the low-pixel image erasing module LE; For the i-th text picture Divide into four equal parts, and get Four quadrants of , , Then input into the low pixel image erasing module LE; four quadrants , , After being processed by m encoding modules E' in sequence, the corresponding sizes are The encoding feature map of , , and ; The four encoded feature maps , , and The image is concatenated along the second and third dimensions to obtain a size of The encoding feature map of ; The encoding feature map Input to m decoding modules D' for processing, and the size is Decoding feature map of ; The decoding feature map Input k layers of convolutional network C' in sequence to perform convolution transformation to obtain different dimensions The feature map output by the third stage ; Step 3.4: Run the multi-stage image fusion module MF; The feature maps output by the three stages , and The image is input into the multi-stage image fusion module MF together, and after being processed by the j-layer long short-term memory LSTM network, the j-th layer outputs a size of Fusion feature map , the fusion feature map After a layer of convolutional network scale change, the size is The text erasing result ; Step 4: Perform supervised training on the multi-stage text erasing network; Step 4.1: Calculate the loss function of the text image set X of natural scenes Use equations (1) and (2) to establish the loss function respectively and : (1) (2) In formula (1) and formula (2), represents the Laplace transform; Step 4.2: Train a multi-stage text erasing network; Use the back propagation algorithm to iteratively train the parameters in the multi-stage text erasure network and calculate the overall loss function , until the iteration reaches the iteration threshold, and finally the trained text erasure network is obtained; among them, represents the weight parameter; Step 5: Use the trained multi-stage text erasure network to process any natural scene image with text to obtain a natural scene image without text after erasure.

Images