A two-stage scene text erasing method based on text segmentation

Abstract

The application discloses a two-stage scene text erasing method based on text segmentation, and the method comprises the following steps: constructing a data set: arranging and collecting existing text segmentation data sets and scene text erasing data sets; scene text segmentation: segmenting scene text by using a trained multi-scale attention text segmentation network, so that a text image is obtained; and scene text erasing: performing scene text erasing by using a trained text erasing network, and generating a real text-free image. The two-stage training method can effectively erase the text in the scene text image, and can selectively erase the text, and can be applied to the fields of image privacy protection, watermark removal, scene text editing and the like, and provides a more real background image.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology and artificial intelligence, and in particular, it is a two-stage scene text erasure method based on text segmentation. Background Technology

[0002] Scene text is an indispensable part of our lives, both in the real world and the virtual world, and it is closely related to people's lives and production. Besides public information such as news, newspapers, and job advertisements, scene text also contains a large amount of private or sensitive information, such as transaction records, communication content, credit information, account passwords, property information, and whereabouts. Protecting personal privacy from criminals is crucial, as it directly relates to personal safety and property security. Currently, the most common method for protecting privacy is to directly delete sensitive information from images and text, but this requires significant manual labor. With the development of artificial intelligence, natural scene text erasure based on deep learning has become an emerging research field.

[0003] Text erasure is similar to image restoration, but it is much more difficult. This is because text erasure requires not only refilling the text areas but also protecting the non-text areas to prevent them from being mistakenly identified as text and erased. Therefore, text erasure is more complex than image restoration.

[0004] In recent years, deep learning-based methods for text erasure in natural scenes can be mainly divided into one-stage methods and two-stage methods.

[0005] One-stage text erasure methods use end-to-end techniques, meaning they complete the task using a single network framework. For example, Nakamura et al., in "Scene text eraser," proposed a neural network based on a single-scale sliding window, using image transformation to quickly process image cropping while preserving non-text regions. Liu et al., in "EraseNet: End-to-end text removal in the wild," trained a text detection network and a background inpainting network in parallel, introducing a coarse-to-fine architecture and an additional segmentation head to aid in text localization, resulting in EraseNet. However, one-stage methods do not effectively distinguish between text and non-text regions, resulting in the problem of erasing non-text areas.

[0006] Two-stage methods treat text detection and background inpainting as upstream and downstream tasks. First, text regions in the image are detected. Then, the masks of these text regions are input into a background inpainting network for inpainting. For example, Tursun et al. proposed MTRNet in "MTRNet: A Generic Scene Text Eraser," which uses manually provided text masks to improve the detection network for text localization, achieving controlled text erasure. Tang et al. proposed predicting the strokes of characters in word images in "Stroke-based scene text erasing using synthetic data for training," and then inputting both the strokes and the image into an image inpainting network to generate a text-free image. However, inaccurate text detection still exists, and if a rectangular text mask is used, some non-text regions are still erased.

[0007] Natural scene text images often suffer from variations in font, arbitrary text direction, or complex backgrounds, making it difficult for existing methods to accurately identify text and non-text regions, thus affecting the erasure effect. Furthermore, accurately obtaining the corresponding text mask is also a significant challenge; failure to obtain a complete text mask may lead to the opposite effect. Therefore, this invention proposes a two-stage scene text erasure method based on text segmentation to address the shortcomings of existing technologies. Summary of the Invention

[0008] The primary objective of this invention is to provide a two-stage scene text erasure method based on text segmentation. This method utilizes text masks obtained from text segmentation, which can effectively erase text in scene text images and selectively erase text. It can be applied to multiple fields such as image privacy protection, watermark removal, and scene text editing, providing more realistic background images.

[0009] Based on this, the present invention is implemented in the following manner:

[0010] A two-stage scene text erasure method based on text segmentation includes the following steps:

[0011] Dataset construction: The existing text segmentation dataset and scene text erasure dataset are organized and collected, and precise text masks are manually annotated for the scene text erasure dataset. Using these text masks, more natural scene images containing text of different sizes are synthesized through scene text size editing technology.

[0012] Scene text segmentation: Construct a multi-scale attention text segmentation network, input the preprocessed training set into the multi-scale attention text segmentation network for training, and use the trained multi-scale attention text segmentation network to segment scene text to obtain text images;

[0013] Scene text erasure: Construct a text erasure network based on generative adversarial network. Input the labeled scene text erasure dataset containing text masks into the text erasure network for training. Finally, use the trained text erasure network to perform scene text erasure and generate realistic text-free images.

[0014] This invention employs a two-stage approach. Constructing the dataset is a prerequisite for network training, and the multi-scale attention text segmentation network and the text erasure network are trained separately. After training, a natural scene text image is first input into the multi-scale attention text segmentation network model, and the output is the segmented text image. Then, this image, along with the natural scene text image, is fed into the generator network of the text erasure network. The output of the generator network is the final prediction result.

[0015] Further improvements are made by constructing the dataset using the following method: collecting and organizing existing text segmentation datasets and scene text erasure datasets, and manually annotating the scene text erasure dataset with accurate text masks using LabelMe software.

[0016] Furthermore, background images with limited textures are collected, and using existing text masks, OpenCV and image synthesis methods are employed to synthesize more natural scene text images of various text sizes to increase the diversity of backgrounds and text.

[0017] Furthermore, the dataset is divided into training, validation, and test sets. The training set is preprocessed, and image pairs are randomly selected from the training dataset. These image pairs are then transformed, including rotation, scaling, and reduction.

[0018] Further improvements are made in the scene text segmentation stage. The multi-scale attention text segmentation network consists of three main modules: a text information enhancement module, a text segmentation module, and a dual-perception decoder. The overall structure is designed based on the classic encoder-decoder structure, and its input is a natural scene text image.

[0019] Furthermore, the text information enhancement module introduces higher-level text semantic information and provides text attention maps as prior knowledge to facilitate text recognition. By making the model focus more on the text regions in the image rather than the background, it can not only reduce the interference of similar textures in the background on text segmentation, but also reduce the computational cost of the network.

[0020] Furthermore, in the feature encoding stage, the proposed text segmentation module is used to model complex text segmentation features by accommodating both global and local features. It assigns equal attention weights to global text to reactivate text with rare textures, as they have strong semantic associations with the dominant text. In addition, it adapts to irregular multi-scale scene text to eliminate the interference of background noise, thereby capturing more effective local features.

[0021] Furthermore, in the decoding stage, pixel aggregation into a specific category is achieved through an upsampling process. However, small deviations in the depth feature map may lead to inaccurate or distorted text segmentation results, especially in scene text with complex shapes. To address this issue, we propose a dual-perception decoder whose parameters can be dynamically adjusted to adapt to spatial and channel content.

[0022] A further improvement is that the total loss function of the multi-scale attention text segmentation network consists of three loss functions, as shown in Equation (1):

[0023] L final =L sem +αL tri +βL tv (1)

[0024] Where α and β represent the weights of each loss function;

[0025] The image input to the network for training is x. After the network extracts features and passes them through an activation function, the probability of each pixel corresponding to each category is obtained, which is the predicted segmentation map x. sem The predicted segmentation map is derived from the true segmentation map x. gt Supervision is performed to obtain the cross-entropy loss function, as shown in formula (2).

[0026] L sem =CrossEntropy(x sem x gt (2)

[0027] Unlike other segmentation tasks, text edge boundaries are crucial to the effectiveness of text segmentation, so here L tri This indicates that using text edge boundaries to perform a weighted cross-entropy loss can further improve text accuracy. The WCE operation is a cross-entropy loss between x and y with weights w. tri This is the binary mapping value of the text boundary region in the image, w at the text boundary. tri =1, otherwise 0.

[0028] L tri =WCE(x out xgt w tri (3)

[0029]

[0030] Furthermore, considering the diversity of text styles and the numerous complex strokes in multilingual texts, we have incorporated L... tv The loss is used to suppress background noise, making the text strokes in the segmentation result coherent and smooth, as shown in formula (5):

[0031]

[0032] A further improvement is that the text erasure network is a model built on generative adversarial network technology, consisting of a generator and a discriminator.

[0033] Furthermore, the generator utilizes AOT blocks to aggregate multi-scale contextual features to improve the ability to capture long-distance features and rich structural details, while the discriminator uses Soft Mask-Guided PatchGAN to better distinguish between the generated parts and the original image, while also helping the generator synthesize fine-grained textures.

[0034] Furthermore, the AOT block adopts a three-step strategy of splitting, transforming, and aggregating, which does not introduce additional model parameters and computational costs compared to the standard residual block.

[0035] A further improvement is that the loss of the text erasure network consists of four parts: reconstruction loss, style loss, perceptual loss, and the adversarial loss of the proposed SM-PatchGAN, as shown in Equation (6):

[0036]

[0037] Where, λ rec , λ sty , λ per , λ adv Represents the weights of each loss function;

[0038] The reconstruction loss is to directly calculate the pixel error between the image x generated by the network and the real image, as shown in formula (7):

[0039] L rec =||xG(x⊙(1-m), m)||1 (7)

[0040] Style loss is calculated by comparing the SSIM of the image x generated by the network with that of the real image. Perceptual loss is calculated by comparing the L1 loss of each layer of feature maps of the image x generated by the network and the real image through a VGG19 model pre-trained on ImageNet. The formulas are as follows:

[0041]

[0042]

[0043] The proposed SM-PatchGAN uses a least-squares loss for its adversarial loss, as shown in Equation (10):

[0044]

[0045] The beneficial effects of this invention are as follows: After expanding the dataset, the multi-scale attention text segmentation network can accurately segment multilingual scene text images containing different scales. Using the segmented text images, the text erasure network can generate more realistic high-resolution non-text image predictions and can erase selected text. At the same time, the method of this invention is simple and convenient and has strong practicality. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the training process of the method of the present invention.

[0047] Figure 2 This is a schematic diagram of the testing process of the method of the present invention.

[0048] Figure 3 This is a flowchart illustrating the text segmentation network of the present invention.

[0049] Figure 4 This is a flowchart illustrating the text erasure network of the present invention.

[0050] Figure 5 This is a flowchart illustrating the AOT block of the present invention. Detailed Implementation

[0051] To make it easier for people to understand the technical means, purpose and effects of the present invention, specific embodiments will be described below.

[0052] Figure 1 and Figure 2 The training and testing process of this embodiment is illustrated, and a two-stage scene text erasure method based on text segmentation is proposed, including the following steps:

[0053] Dataset construction: The existing text segmentation dataset and scene text erasure dataset are organized and collected, and precise text masks are manually annotated for the scene text erasure dataset. Using these text masks, more natural scene images containing text of different sizes are synthesized through scene text size editing technology.

[0054] Scene text segmentation: Construct a multi-scale attention text segmentation network, input the preprocessed training set into the multi-scale attention text segmentation network for training, and use the trained multi-scale attention text segmentation network to segment scene text to obtain text images;

[0055] Scene text erasure: Construct a text erasure network based on generative adversarial network. Input the labeled scene text erasure dataset containing text masks into the text erasure network for training. Finally, use the trained text erasure network to perform scene text erasure and generate realistic text-free images.

[0056] The specific method for constructing the dataset is as follows:

[0057] We collected and organized existing text segmentation datasets and scene text erasure datasets, such as TextSeg, TextLogo3K, and ICDAR2013, and used LabelMe software to manually annotate the scene text erasure datasets with accurate text masks.

[0058] Furthermore, background images with limited textures are collected, and using existing text masks, OpenCV and image synthesis methods are employed to synthesize more natural scene text images of various text sizes, thereby increasing the diversity of backgrounds and text.

[0059] The dataset is divided into training, validation and test sets. The training set is preprocessed. Image pairs are randomly selected from the training dataset and transformed, including rotation, scaling and reduction.

[0060] The training set is used to train the text segmentation network and the text erasure network, the validation set is used to evaluate performance and adjust the model's hyperparameters, and finally the test set is used for the final evaluation of the model.

[0061] Figure 3 This is a flowchart illustrating the text segmentation network of the present invention. Its input is a natural scene text image, and its output is the segmented text image.

[0062] The multi-scale attention text segmentation network consists of three main modules: a text information enhancement module, a text segmentation module, and a dual-sensor decoder. The overall structure is designed based on the classic encoder-decoder structure.

[0063] The text information enhancement module introduces higher-level text semantic information and provides text attention maps as prior knowledge to facilitate text recognition. By making the model focus more on the text regions in the image rather than the background, it can not only reduce the interference of similar textures in the background on text segmentation, but also reduce the computational cost of the network.

[0064] In the feature encoding stage, the proposed text segmentation module is used to model complex text segmentation features by accommodating global and local features. It assigns equal attention weights to global text to reactivate text with rare textures because they have strong semantic associations with the dominant text. In addition, it adapts to irregular multi-scale scene text to eliminate the interference of background noise, thereby capturing more effective local features.

[0065] In the decoding stage, pixel aggregation into a specific category is achieved through an upsampling process. However, small deviations in the depth feature map may lead to inaccurate or distorted text segmentation results, especially in scene text with complex shapes. To address this issue, we propose a dual-perception decoder whose parameters can be dynamically adjusted to adapt to spatial and channel content.

[0066] The total loss function of the multi-scale attention text segmentation network consists of three loss functions, as shown in Equation (1):

[0067] L final =L sem +αL tri +βL tv (1)

[0068] Where α and β represent the weights of each loss function.

[0069] The image input to the network for training is x. After the network extracts features and passes them through an activation function, the probability of each pixel corresponding to each category is obtained, which is the predicted segmentation map x. sem The predicted segmentation map is derived from the true segmentation map x. gt Supervision is performed to obtain the cross-entropy loss function, as shown in formula (2).

[0070] L sem =CrossEntropy(x sem x gt (2)

[0071] Unlike other segmentation tasks, text edge boundaries are crucial to the effectiveness of text segmentation, so here L tri This indicates that using text edge boundaries to perform a weighted cross-entropy loss can further improve text accuracy. The WCE operation is a cross-entropy loss between x and y with weights W, where w... triThis is the binary mapping value of the text boundary region in the image, w at the text boundary. tri =1, otherwise 0.

[0072] L tri =WCE(x out x gt w tri (3)

[0073]

[0074] Furthermore, considering the diversity of text styles and the numerous complex strokes in multilingual texts, we have incorporated L... tv The loss is used to suppress background noise, making the text strokes in the segmentation result coherent and smooth, as shown in formula (5):

[0075]

[0076] Optimization is performed using the gradient descent algorithm.

[0077] Figure 4 This is a flowchart illustrating the text erasure network of the present invention. Its inputs are a scene text image and a text mask, and its output is a generated textless scene text image.

[0078] Text erasure networks are models built on generative adversarial networks, consisting of a generator and a discriminator.

[0079] The generator utilizes AOT blocks to aggregate multi-scale contextual features to improve the ability to capture long-distance features and rich structural details, while the discriminator uses Soft Mask-Guided PatchGAN to better distinguish between the generated parts and the original image, while also helping the generator synthesize fine-grained textures.

[0080] Figure 5 This is a flowchart of the AOT block of the present invention. The AOT block adopts a three-step strategy of splitting, transforming and aggregating. Compared with the standard residual block, it does not introduce additional model parameters and computational costs.

[0081] The loss of the text erasure network consists of four parts: reconstruction loss, style loss, perceptual loss, and the adversarial loss of the proposed SM-PatchGAN, as shown in Equation (6):

[0082]

[0083] Where, λ rec , λ sty , λ per , λ adv Represents the weights of each loss function;

[0084] The reconstruction loss is to directly calculate the pixel error between the image x generated by the network and the real image, as shown in formula (7):

[0085] L rec =||xG(x⊙(1-m), m)||1 (7)

[0086] Style loss is calculated by comparing the SSIM of the image x generated by the network with that of the real image. Perceptual loss is calculated by comparing the L1 loss of each layer of feature maps of the image x generated by the network and the real image through a VGG19 model pre-trained on ImageNet. The formulas are as follows:

[0087]

[0088]

[0089] The proposed SM-PatchGAN uses the least squares loss for adversarial loss, as shown in Equation (10):

[0090]

[0091] Optimization is performed using the gradient descent algorithm.

[0092] This invention employs a two-stage method that can generate more realistic, high-resolution non-text image predictions and erase selected text. Furthermore, the method is simple and convenient to implement and has strong practicality.

[0093] The above description is merely the preferred embodiment of the present invention and should not be considered as a limitation thereof. Any modifications, substitutions, and improvements within the spirit and principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A two-stage scene text erasure method based on text segmentation, characterized in that, Includes the following steps: Dataset construction: Existing text segmentation datasets and scene text erasure datasets are organized and collected, and precise text masks are manually annotated for the scene text erasure dataset. Using these text masks, natural scene images containing text of different sizes are synthesized through scene text size editing technology. Scene text segmentation: Construct a multi-scale attention text segmentation network, input the preprocessed training set into the multi-scale attention text segmentation network for training, and use the trained multi-scale attention text segmentation network to segment scene text to obtain text images; The multi-scale attention text segmentation network consists of three main modules: a text information enhancement module, a text segmentation module, and a dual-perception decoder. The overall structure is designed based on the classic encoder-decoder structure, and its input is a natural scene text image. The total loss function of the multi-scale attention text segmentation network consists of three loss functions, as shown in formula (1): in, , Represents the weights of each loss function; The images input into the network for training are The predicted segmentation map is obtained by extracting features through a network and applying an activation function to obtain the probability of each pixel corresponding to each category. The predicted segmentation map is derived from the true segmentation map. Supervision is performed to obtain the cross-entropy loss function, as shown in formula (2). Unlike other segmentation tasks, text edge boundaries are crucial to the effectiveness of text segmentation, so here... This indicates that a weighted cross-entropy loss is applied using text edge boundaries to further improve text accuracy; the WCE operation is based on... For weights and Cross-entropy loss between This refers to the binary mapping value of the text boundary region in the image, on the text boundary. =1, otherwise 0; Furthermore, considering the diversity of text styles, multilingual texts often contain many complex strokes, and therefore require additional... The loss is used to suppress background noise, making the text strokes in the segmentation result coherent and smooth, as shown in formula (5): Scene text erasure: Construct a text erasure network based on generative adversarial network. Input the labeled scene text erasure dataset containing text masks into the text erasure network for training. Finally, use the trained text erasure network to perform scene text erasure and generate realistic text-free images.

2. The two-stage scene text erasure method based on text segmentation as described in claim 1, characterized in that, The dataset is constructed using the following method: We collected and organized existing text segmentation datasets and scene text erasure datasets, and used LabelMe software to manually annotate the scene text erasure datasets with accurate text masks. Collect background images with limited textures, utilize existing text masks, and employ OpenCV and image compositing methods to synthesize natural scene text images of various text sizes to increase the diversity of backgrounds and text. The dataset is divided into training, validation and test sets. The training set is preprocessed. Image pairs are randomly selected from the training dataset and transformed, including rotation, scaling and reduction.

3. The two-stage scene text erasure method based on text segmentation as described in claim 1, characterized in that, The text information enhancement module introduces higher-level text semantic information and provides text attention maps as prior knowledge to facilitate text recognition.

4. The two-stage scene text erasure method based on text segmentation as described in claim 1, characterized in that, During the decoding phase, pixel aggregation into a specific category is achieved through an upsampling process.

5. The two-stage scene text erasure method based on text segmentation as described in claim 1, characterized in that, The text erasure network is a model built on generative adversarial network technology, consisting of a generator and a discriminator. The generator uses AOT blocks to aggregate multi-scale contextual features to improve the ability to capture long-range features and rich structural details, while the discriminator uses Soft Mask-Guided PatchGAN.

6. The two-stage scene text erasure method based on text segmentation as described in claim 5, characterized in that, The loss of the text erasure network consists of four parts: reconstruction loss, style loss, perceptual loss, and the adversarial loss of the proposed SM-PatchGAN, as shown in Equation (6): in, , , , This represents the weights of each loss function. The reconstruction loss is the pixel error between the image x generated by the network and the real image, as shown in formula (7): Style loss is calculated by comparing the SSIM of the image x generated by the network with that of the real image. Perceptual loss is calculated by comparing the L1 loss of each layer of feature maps output by the VGG19 model pre-trained on ImageNet with the image x generated by the network and the real image. The formulas are as follows: The proposed SM-PatchGAN uses the least squares loss for adversarial loss, as shown in Equation (10): 。 7. The two-stage scene text erasure method based on text segmentation as described in claim 1, characterized in that, The multi-scale attention text segmentation network and the text erasure network are trained separately. After training, a natural scene text image is first input into the multi-scale attention text segmentation network model, and the output is the segmented text image. Then, this image and the natural scene text image are fed into the generator network of the text erasure network, and the output of the generator network is the final prediction result.

Images