An image restoration method for incomplete scene text

Abstract

The present invention discloses an image restoration method for incomplete scene text. In order to make better use of text structure information, the present invention first separates the background from the text. Under the joint guidance of structural information and semantic information, the restoration process only repairs the text structure towards the most likely complete text, reducing the impact of background restoration on the text structure, thereby improving the recognition accuracy of subsequent incomplete scene text. The training process of the incomplete scene text image restoration model is divided into two steps: the incomplete text extraction module and the incomplete text reconstruction module are trained separately, and the incomplete text extraction module and the incomplete text reconstruction module are trained jointly, which greatly improves the restoration effect. In addition, the present invention constructs an incomplete scene text dataset for end-to-end training of the proposed incomplete scene text image restoration model, so that the incomplete scene text model can restore the incomplete text to the corresponding complete text.

Description

Technical Field

[0001] The invention relates to an image restoration method for incomplete scene text, and belongs to the technical field of image restoration. Background Art

[0002] Applications of scene text recognition include traffic sign recognition, text transcription, human-computer interaction, etc. In recent years, with the rapid development of deep learning, scene text recognition technology has made great progress. However, for incomplete scene text, it is impossible to accurately recognize it using existing scene text recognition methods. In natural scenes, although incomplete scene text is not as widespread as complete text, it still exists in some actual scenes, such as invoices after the seal is removed, container numbers, road identifiers, image reflections, etc. If the incomplete scene text is not correctly recognized in these scenes, it may cause some economic losses and even threaten personal safety in autonomous driving.

[0003] Existing image restoration methods can generally be divided into two branches: image restoration methods based on defect position prior masks and blind image restoration methods. In the image restoration method based on defect position prior masks, the input of the deep restoration network is the image I of the defect position prior mask. m and the original incomplete image I s The repair network first passes the defect location prior mask image I m To limit the area that needs to be repaired, and then use the original incomplete image I s The structural information and semantic information of the intact part of the image are used to repair the defective position, so that there is no obvious transition between the non-defective area and the defective area, and the whole image is harmonious and natural. In the blind image restoration method, only the original defective image I is input to the restoration algorithm. s , the image restoration algorithm first predicts the defective area, and then uses the original defective image I s The texture information and semantic information of the complete part are used to repair the incomplete position, and the incomplete image whose position cannot be known in advance can be well repaired.

[0004] In many scenarios, the incomplete position prior mask based restoration method can obtain better and more reliable restoration results than the blind image restoration method. However, since the incomplete position prior mask cannot be known in advance in some special scenarios, different restoration methods need to be selected according to the task scenario. For the incomplete scene text recognition task, the overall task is for the machine to automatically recognize the text quickly and accurately for later work. Therefore, it is impossible to provide an incomplete position prior mask for each character. Therefore, only the blind image restoration method can be used to repair the incomplete scene text to obtain a complete text image, and then use the existing scene text recognition method for recognition. However, the existing blind image restoration method will simultaneously repair the background and text structure of the incomplete position, causing the background restoration to blur the structure of the text, further affecting the accuracy of text recognition. Summary of the invention

[0005] In order to solve the deficiencies of the prior art, the present invention provides an image restoration algorithm for incomplete scene text. Compared with the existing image restoration algorithm, in order to better utilize the text structure information and separate the background from the text, the restoration process of the present invention only repairs the text structure towards the most likely complete text under the joint guidance of the structural information and the semantic information, thereby reducing the impact of the background restoration on the text structure, thereby improving the recognition rate of the incomplete scene text; at the same time, the image restoration model of the incomplete scene text is trained using the incomplete scene text dataset, so that the incomplete scene text model can restore the incomplete text to the corresponding complete text.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A method for image restoration of incomplete scene text, which separates the text and the background before restoration, reduces the influence of background restoration on the text structure during the restoration process, and restores the separated incomplete text towards the corresponding complete text under the guidance of the text structure and text semantic information; at the same time, a data set of incomplete scene text is constructed for training an image restoration model of incomplete scene text, so that the incomplete scene text model can restore the incomplete text into the corresponding complete text. The method comprises the following steps:

[0008] S1. Construct an incomplete scene text dataset, which includes incomplete scene text images I st , incomplete text image without background T et and the complete text image without background T t ;

[0009] S2. Construct an incomplete scene text image restoration model, which consists of two modules: an incomplete text extraction module and an incomplete text reconstruction module. The incomplete text reconstruction module takes the output of the incomplete text extraction module as input and repairs the incomplete text towards the corresponding complete text according to the guidance of the text structure and text semantic information.

[0010] S3, the incomplete scene text image I st Input into the incomplete text extraction module to separate the background and text, and obtain the output incomplete text image without background O et , calculate the incomplete text image T without background in the original dataset et With O et The loss between is used to constrain the training of the incomplete text extraction module;

[0011] S4, the incomplete text reconstruction module uses the output of the incomplete text extraction module et As input, the incomplete text is repaired to obtain a complete text image without background O t , calculate the complete text image T without background in the original dataset t With O t The loss between is used to constrain the training of the incomplete text reconstruction module;

[0012] S5, dividing the training process of the incomplete scene text image restoration model into two steps: separate training of the incomplete text extraction module and the incomplete text reconstruction module, and joint training of the incomplete text extraction module and the incomplete text reconstruction module;

[0013] S6. The final output of the incomplete scene text image restoration method is different from the original incomplete scene text image I st The corresponding complete text image without background O t .

[0014] Furthermore, step S1 specifically includes:

[0015] For the basic surface layer, text pixels and non-text pixels are marked by 1 and 0 respectively. For the incomplete block surface layer, it is filled with 0 value. The complete text surface layer is defined as S com ∈R h×w , the surface of the incomplete text is S incom ∈R h×w , the surface of the incomplete block is M∈R h×w , the expression generated by the incomplete text surface is as follows:

[0016] S incom =S com ⊙M (1)

[0017] Among them, ⊙ represents the Hadamard product;

[0018] For the generated incomplete text surface, the scene text image synthesis technology is used to add the background image and text color, and the image parameters such as saturation and transparency are set to finally synthesize the incomplete scene text image I st ; Among them, S com is the complete text image without background T t , S incom is a fragmented text image without background T et .

[0019] Furthermore, step S2 specifically includes:

[0020] The incomplete text extraction module is an encoder-decoder with a fully convolutional network structure, without any fully connected layers in the middle. For the encoder, it is first constructed by connecting five combination blocks with ELU activation functions in series. The combination block consists of a residual block and an ELU activation layer. The residual block consists of a convolutional layer, an ELU activation layer, and a convolutional layer. The encoder is used to extract the I st Encoding is performed by downsampling the image size to I through the convolution layer only. st The decoder is composed of four combination blocks with the same configuration as the encoder, two upsampling layers, a convolutional layer with ELU, and a single pure convolutional layer to form a single-channel feature map. Finally, the image containing only the text area is obtained through the Sigmod function. For the incomplete scene text extraction module, except for the output layer whose activation function is Sigmod, the activation functions of other layers are all ELU.

[0021] The incomplete text reconstruction module is also an encoder-decoder with a fully convolutional network structure, without any fully connected layers in the middle. For the encoder, it is first constructed by connecting ten combination blocks with ELU activation functions in series. The combination blocks here are the same as those in the incomplete text extraction module mentioned above. The encoder is used to et Encoding is performed by downsampling the image size to O only through the convolution layer et The decoder is composed of four combination blocks with the same configuration as the encoder, two upsampling layers, and a single convolutional layer to form a single-channel feature map. Finally, the Tanh function is used to obtain a single-channel background-free complete text image O t In the convolution layer, the convolution kernel expansion method is used to increase the receptive field of the convolutional neural network. For the incomplete scene text reconstruction module, except for the output layer activation function which is Tanh, the activation functions of other layers are ELU.

[0022] Furthermore, step S3 specifically includes:

[0023] For the incomplete text extraction module, two losses are used for constraints. The first loss is the binary cross entropy loss L bce, the second loss is L1 loss, the specific formula is:

[0024] L(T et ,O et )=α1L bce +α2L1 (2)

[0025] L bce =-λ1∑T et ·log(O et )-λ2∑(1-T et )·log(1-O et ) (3)

[0026] L1=||T et -O et || (4)

[0027] Among them, α1 and α2 are the binary cross entropy loss function L bce and the balance coefficient of the L1 loss function, λ1 and λ2 are the penalty coefficients for the loss of text and non-text regions.

[0028] Furthermore, step S4 specifically includes:

[0029] For the incomplete text reconstruction module, two losses are used for constraints. The first loss is the reconstruction loss L rec , the second loss is the semantic loss L sem , the specific formula is:

[0030] L(T t ,O t )=ω1L rec +ω2L sem (5)

[0031] Among them, ω1 and ω2 are the balance coefficients of reconstruction loss and semantic loss;

[0032] Reconstruction loss L rec Including L1 loss and binary cross entropy loss L bce , the specific formula is:

[0033] L rec =θ1L1+θ2L bce (6)

[0034] L1=||T t -O t || (7)

[0035] L bce =-λ1∑T t ·log(O t )-λ2∑(1-T t )·log(1-Ot ) (8)

[0036] Among them, θ1 and θ2 are L1 loss and binary cross entropy loss L bce The balance coefficient, λ1 and λ2 are the penalty coefficients for the loss of text area and non-text area;

[0037] Semantic loss L sem Including the perceptual loss L per and style loss L style , the specific formula is as follows:

[0038] L sem =β1L pre +β2L style (9)

[0039]

[0040]

[0041] Among them, φ i It is the feature map output by the relu1_1, relu2_1, relu3_1, relu4_1, and relu5_1 layers of the VGG19 network pre-trained on the ImageNet dataset; N i is the total number of pixels in the feature map of the i-th layer; G is the Gram matrix, G(F) = FF T ∈R n×n ; β1 and β2 are the balance coefficients of perceptual loss and style loss.

[0042] Furthermore, step S5 specifically includes:

[0043] When the number of iterations is less than or equal to m, the incomplete text extraction module and the incomplete text reconstruction module are trained separately, and their respective losses are used for constraints; when the number of iterations is greater than m, the incomplete text extraction module and the incomplete text reconstruction module are trained jointly, and the losses of the two modules are combined to constrain the two modules together. The expression of the joint loss function is as follows:

[0044] L=σ1L(T et ,O et )+σ2L(T t ,O t ) (12)

[0045] Among them, σ1 and σ2 are the balance coefficients of the incomplete text extraction module and the incomplete text reconstruction module.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] 1. Compared with the prior art, the present invention proposes an image restoration method specifically for incomplete text, which includes two parts: incomplete text extraction and incomplete text reconstruction. The method explicitly separates the text from the background and restores the structure of the separated incomplete text, thereby reducing the phenomenon of text structure blurring caused by background restoration and improving the recognition accuracy of the restored incomplete scene text.

[0048] 2. Compared with the prior art, the present invention uses semantic loss in the text reconstruction module. When the text loss in the image is serious, the incomplete text can be repaired according to the text semantics to obtain a complete text structure;

[0049] 3. The present invention constructs an incomplete scene text dataset for end-to-end training of the proposed incomplete scene text repair method to improve the effect of incomplete scene text repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flowchart of implementing incomplete scene text image restoration according to an embodiment of the present invention;

[0051] Figure 2 is a flow chart of generating an incomplete scene text image according to an embodiment of the present invention;

[0052] Figure 3 It is an overall structural diagram of the incomplete scene text image restoration network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention is further described below in conjunction with the accompanying drawings. The specific embodiments described herein are only used to more clearly illustrate the present invention and are not intended to limit the scope of protection of the present invention. Figure 1 , Figure 2 , Figure 3 The present invention discloses an image repair method for incomplete scene text, comprising the following steps:

[0054] Step S1 specifically includes:

[0055] The process of generating incomplete text is as follows Figure 2 As shown in the figure, for the base surface layer, text pixels and non-text pixels are marked by 1 and 0 respectively. For the incomplete block surface layer, it is filled with 0 value. The complete text surface layer is defined as S com ∈R h×w , the surface of the incomplete text is S incom ∈R h×w , the surface of the incomplete block is M∈R h×w , the expression generated by the incomplete text surface is as follows:

[0056] S incom =S com ⊙M (13)

[0057] Among them, ⊙ represents the Hadamard product;

[0058] For the generated incomplete text surface, the scene text image synthesis technology is used to add the background image and text color, and the image parameters such as saturation and transparency are set to finally synthesize the incomplete scene text image I st , where S com is the complete text image without background T t , S incom is a fragmented text image without background T et ;

[0059] The incomplete blocks are regular rectangles and irregular polygons. For irregular polygons, the convex hull algorithm based on circles and rectangles is used to generate them. First, n∈(8,50) vertices are selected from the inner areas of rectangles and circles, and then the convex hull is solved using the Graham scanning algorithm to construct irregular polygons.

[0060] Through the above process, an image I containing incomplete scene text is generated st and the incomplete text image T without background et , the complete text image without background T t The dataset contains three types of images as one data pair, with a total of 100,000 pairs, which are used to train the incomplete text image restoration algorithm. The ratio of the training set to the test set is 4:2.

[0061] Step S2 specifically includes:

[0062] The incomplete scene text restoration model consists of two modules: the incomplete text extraction module ( Figure 3 The upper part of the box) and the incomplete text reconstruction module ( Figure 3 in the lower part of the box).

[0063] Step S3 specifically includes:

[0064] For the incomplete text extraction module, two losses are used for constraints. The first loss is the binary cross entropy loss L bce , the second loss is L1 loss, the specific formula is:

[0065] L(T et ,O et )=α1L bce +α2L1 (14)

[0066] L bce =-λ1∑T et ·log(O et )-λ2∑(1-T et )·log(1-O et ) (15)

[0067] L1=||T et -O et || (16)

[0068] Among them, α1 and α2 are the balance coefficients of the binary cross entropy loss function and the L1 loss function, which are set to 10.0 and 1.0 respectively. λ1 and λ2 are the penalty coefficients of the text area and non-text area loss, which are set to 2.0 and 1.0 respectively within 10 iterations; and set to 1.0 and 1.0 respectively after 10 iterations.

[0069] Step S4 specifically includes:

[0070] For the incomplete text reconstruction module, two losses are used for constraints. The first loss is the reconstruction loss L rec , the second loss is the semantic loss L sem The specific formula is:

[0071] L(T t ,O t )=ω1L rec +ω2L sem (17)

[0072] Among them, ω1 and ω2 are the balance coefficients of reconstruction loss and semantic loss, which are set to 10.0 and 10.0 respectively;

[0073] Reconstruction loss L rec Including L1 loss and binary cross entropy loss L bce , the specific formula is:

[0074] L rec =θ1L1+θ2L bce (18)

[0075] L1=||T t -O t || (19)

[0076] L bce =-λ1∑T t ·log(O t )-λ2∑(1-T t )·log(1-O t ) (20)

[0077] Among them, θ1 and θ2 are L1 loss and binary cross entropy loss L bce The balance coefficients are set to 5.0 and 3.0 respectively. The setting method of λ1 and λ2 is the same as that in the above step S3;

[0078] Semantic loss L semIncluding the perceptual loss L per and style loss L style , the specific formula is as follows:

[0079] L sem =β1L per +β2L style (twenty one)

[0080]

[0081]

[0082] Among them, φ i is the feature map output by the relu1_1, relu2_1, relu3_1, relu4_1, and relu5_1 layers of the VGG19 network pre-trained on the ImageNet dataset; N i is the total number of pixels in the feature map of the i-th layer; G is the Gram matrix, G(F) = FF T ∈R n×n ; β1 and β2 are the balance coefficients of perceptual loss and style loss, which are set to 1.0 and 1.0 respectively.

[0083] Step S5 specifically includes:

[0084] The training strategy of the incomplete scene text repair model is as follows Figure 1 As shown in the figure, when the number of iterations is less than or equal to 10, the incomplete text extraction module and the incomplete text reconstruction module are trained separately, and their respective losses are used for constraints; when the number of iterations is greater than 10, the incomplete text extraction module and the incomplete text reconstruction module are trained jointly, and the joint loss is used to constrain the two modules. The expression of the joint loss function is as follows:

[0085] L=σ1L(T et ,O et )+σ2L(T t ,O t ) (twenty four)

[0086] Among them, σ1 and σ2 are set to 1.0 and 2.0 respectively.

[0087] The above description is merely a preferred embodiment of the present invention, which is only used to further illustrate the present invention and cannot be used to limit the protection scope of the present invention. Those skilled in the art may make some improvements and modifications to the present invention based on the contents of the above invention, and these improvements and modifications all fall within the protection scope of the present invention.

Claims

1. An image restoration method for incomplete scene text, characterized by: Before repairing, the text and background are separated to reduce the impact of background repair on the text structure during the repair process. Under the guidance of text structure and text semantic information, the separated incomplete text is repaired towards the most likely complete text; at the same time, an incomplete scene text dataset is constructed to train an incomplete scene text image repair model so that the incomplete scene text model can repair the incomplete text into the corresponding complete text. The method includes the following steps: S1. Construct an incomplete scene text dataset, which includes incomplete scene text images I st , incomplete text image without background T et and the complete text image without background T t ; S2. Construct an incomplete scene text image restoration model. The model consists of two modules, namely, an incomplete text extraction module and an incomplete text reconstruction module. The incomplete text reconstruction module takes the output of the incomplete text extraction module as input and repairs the incomplete text towards the corresponding complete text according to the guidance of the text structure and text semantic information. Step S2 specifically includes: The incomplete text extraction module is an encoder-decoder with a fully convolutional network structure, without any fully connected layers in the middle. For the encoder, it is first constructed by connecting five combination blocks with ELU activation functions in series. The combination block consists of a residual block and an ELU activation layer. The residual block consists of a convolutional layer, an ELU activation layer, and a convolutional layer. The encoder is used to extract the I st Encoding is performed by downsampling the image size to I through the convolution layer only. st The decoder is composed of four combination blocks with the same configuration as the encoder, two upsampling layers, a convolutional layer with ELU, and a single pure convolutional layer to form a single-channel feature map. Finally, the image containing only the text area is obtained through the Sigmod function. For the incomplete scene text extraction module, except for the output layer whose activation function is Sigmod, the activation functions of other layers are all ELU. The incomplete text reconstruction module is also an encoder-decoder with a fully convolutional network structure, without any fully connected layers in the middle. For the encoder, it is first constructed by connecting ten combination blocks with ELU activation functions in series. The combination blocks here are the same as those in the incomplete text extraction module mentioned above. The encoder is used to et Encoding is performed by downsampling the image size to O only through the convolution layer et The decoder is composed of four combination blocks with the same configuration as the encoder, two upsampling layers, and a single convolutional layer to form a single-channel feature map. Finally, the Tanh function is used to obtain a single-channel background-free complete text image O t ,For the incomplete scene text reconstruction module, except for the output layer whose activation function is Tanh, the activation functions of other layers are ELU; The convolution kernel expansion method is used in the convolution layer to increase the receptive field of the convolutional neural network; S3, the incomplete scene text image I st The image is input into the incomplete text extraction module to separate the background and text, and the output incomplete text image without background is obtained. et , calculate the incomplete text image T without background in the original dataset et With O et The loss between is used to constrain the training of the incomplete text extraction module. Step S3 specifically includes: For the incomplete text extraction module, two losses are used for constraints. The first loss is the binary cross entropy loss L bce , the second loss is L1 loss, the specific formula is: L(T et ,OH et )=α1L bce +α2L1 L bce =-λ1∑T et ·log(O et )-λ2Σ(1-T et )·log(1-O et ) L1=||T et -O et || Among them, α1 and α2 are the binary cross entropy loss function L bce and the balance coefficient of the L1 loss function, λ1 and λ2 are the penalty coefficients for the loss of text areas and non-text areas; S4, the incomplete text reconstruction module uses the output of the incomplete text extraction module et As input, the incomplete text is repaired to obtain a complete text image without background O t , calculate the complete text image T without background in the original dataset t With O t The loss between is used to constrain the training of the incomplete text reconstruction module. Step S4 specifically includes: For the incomplete text reconstruction module, two losses are used for constraints. The first loss is the reconstruction loss L rec , the second loss is the semantic loss L sem , including the perceptual loss L per and style loss L style , the specific formula is: L(T t ,O t )=ω1L rec +ω2L sem Among them, ω1 and ω2 are the balance coefficients of reconstruction loss and semantic loss; Reconstruction loss L rec Including L1 loss and binary cross entropy loss L bce , the specific formula is: L rec =θ1L1+θ2L bce L1=||T t -O t || L bce =-λ1ΣT t ·log(O t )-λ2Σ(1-T t )·log(1-O t ) Among them, θ1 and θ2 are L1 loss and binary cross entropy loss L bce The balance coefficient, λ1 and λ2 are the penalty coefficients for the loss of text area and non-text area; Semantic loss L sem Including the perceptual loss L per and style loss L style , the specific formula is as follows: L sem =β1L per +β2L style Among them, φ i It is the feature map output by the relu1_1, relu2_1, relu3_1, relu4_1, and relu5_1 layers of the VGG19 network pre-trained on the ImageNet dataset; N i is the total number of pixels in the feature map of the i-th layer; G is the Gram matrix, G(F) = FF T ∈R n×n ; β1 and β2 are the balance coefficients of perceptual loss and style loss; S5, the training process of the incomplete scene text image restoration model is divided into two steps: the incomplete text extraction module and the incomplete text reconstruction module are trained separately, and the incomplete text extraction module and the incomplete text reconstruction module are trained jointly. Step S5 specifically includes: When the number of iterations is less than or equal to m, the incomplete text extraction module and the incomplete text reconstruction module are trained separately, and their respective losses are used for constraints; when the number of iterations is greater than m, the incomplete text extraction module and the incomplete text reconstruction module are trained jointly, and the losses of the two modules are combined to constrain the two modules together. The expression of the joint loss function is as follows: L=σ1L(T et ,O et )+σ2L(T t ,O t ) Among them, σ1 and σ2 are the balance coefficients of the incomplete text extraction module and the incomplete text reconstruction module; S6. The final output of the incomplete scene text image restoration method is different from the original incomplete scene text image I st The corresponding complete text image without background O t .

2. The image restoration method for incomplete scene text according to claim 1, characterized in that: Step S1 specifically includes: For the base surface layer, text pixels and non-text pixels are marked by 1 and 0 respectively; for the incomplete block surface layer, it is filled with 0 value, and the complete text surface layer is defined as S com ∈R h×w , the surface of the incomplete text is S incom ∈R h×w , the surface of the incomplete block is M∈R h×w , the expression generated by the incomplete text surface is as follows: S incom =S com ⊙M Among them, ⊙ represents the Hadamard product; For the generated incomplete text surface, the scene text image synthesis technology is used to add the background image and text color, and the saturation and transparency image parameters are set to finally synthesize the incomplete scene text image I st , where S com is the complete text image without background T t , S incom is a fragmented text image without background T et .

Images