Scene text elimination method free of data annotation and network training

Abstract

The invention provides a scene text elimination method free of data labeling and network training, and the method comprises the steps: carrying out the hierarchical text positioning in a pre-training diffusion denoising model inversion process, obtaining a mask of a first text region from an original image, carrying out the cutting and amplification of the original image corresponding to the first text region, obtaining a plurality of sub-images, and carrying out the segmentation and amplification of the sub-images, obtaining a mask of a second text region from the sub-graph, performing 2-mean clustering on the sub-graph to obtain a sub-graph sub-region, and calculating a mask of a third text region; before the hidden space noise obtained through inversion is reconstructed, a text area of the hidden space noise is destroyed through random Gaussian noise in advance, and a processed hidden space diagram is obtained; and inputting the processed latent space graph into a pre-training diffusion denoising model to automatically recover the background region to obtain an image after text elimination. According to the method, the processes of text positioning, text destruction, background recovery and the like can be completed on the premise of no training and data set labeling, and the image processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of text processing of images, and in particular to a scene text elimination method free of data annotation and network training. Background Art

[0002] The text in the image may contain sensitive information (such as phone numbers, license plate numbers, addresses, etc.), and the leakage and dissemination of this information brings great distress to individuals. Scene text erasure aims to erase the text information in the image and fill the erased text area with a visually credible background. Before the era of deep learning, many technologies have used traditional machine learning and computer vision methods to perform scene text erasure. Later, neural networks, with their powerful learning ability, further enhanced the effect of text erasure. For example, different network structures and processing flows can be designed through technologies such as MTRNet, CTRNet, and DeepEraser to solve this problem. Among them, some technologies are end-to-end trained, which simplifies dataset construction and training; others divide the process into multiple parts. Although it increases the difficulty in data collection and training, it also increases the freedom of adjustment, allowing the model performance to be released to the extreme. In addition, STRDD uses the latest diffusion model for text erasure, which improves the erasure effect. Diffusion models have made remarkable achievements in the field of image editing. The DDIM inversion process provides a path for image reconstruction, so that the image can be restored to the latent space noise with certainty, and then the image editing can be completed by adjusting the reconstruction process. Technologies such as MasaCtrl successfully control the structure and texture of the edited object through the keys and values ​​of the self-attention layer.

[0003] At present, most scene text erasure technologies focus too much on the accuracy of background restoration, and some of them gradually fade the text in an iterative manner, which may leave subtle residues that make the text still readable. In addition, existing methods require training and consume a lot of computing resources. The training datasets require annotated masks and manually drawn real images after erasure. Although STRDD and other works use diffusion models for text removal, they still need retraining and data annotation because they do not fully exploit the pre-trained diffusion model's ability to locate and control objects in the image.

[0004] In addition, many scene text erasure techniques only focus on the process of filling in the image based on the text mask, so users are usually required to provide a text area mask, or use a text detection network to obtain the required mask before text erasure, which increases the difficulty for users.

[0005] In view of this, it is urgent to study a method that can completely eliminate text areas without the need for training, labeling, or user-provided mask processes. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a scene text elimination method that is free of data labeling and network training. The method is improved by using a pre-trained diffusion model, and processes such as text positioning, text destruction, and background restoration can be completed without the need for training and data set labeling.

[0007] The present invention is implemented as follows: a scene text elimination method free of data annotation and network training, the method comprising:

[0008] Step S1: hierarchical text localization is performed in the inversion process of the pre-trained diffusion denoising model by obtaining the mask of the first text area from the original image. The original image corresponding to the first text area is cropped and enlarged to obtain multiple sub-images, and then the mask of the second text area is obtained from the sub-images. Perform 2-means clustering on the subgraph to obtain subgraph subregions, combined with and The background area and sub-image sub-area obtained by subtraction calculate the mask of the refined third text area The third text area is used as the final text area;

[0009] Step S2, before reconstructing the latent space noise obtained by inversion, the text area is pre-destroyed with random Gaussian noise to obtain a processed latent space map;

[0010] Step S3: input the processed latent space map into the pre-trained diffusion denoising model to automatically restore the background area to obtain an image after text removal.

[0011] Furthermore, the step S1 specifically includes:

[0012] Step S11: Perform the first stage of text localization. During inversion, use "text letter character" as the conditional prompt word to extract the attention map. And extract the ending character " <endoftext>The corresponding attention map Aggregate the attention map and perform 3-means clustering on the aggregated attention to obtain the mask of the first text region The aggregation process of the attention map is as follows:

[0013]

[0014] in, represents the aggregated attention, Represents the end character ( <endoftext>) corresponds to the attention map, γ is a constant coefficient;

[0015] Step S12: Perform the second stage of text positioning by cutting and enlarging the original image portion corresponding to each sub-region in the first text region to obtain a sub-image set, and then use the first stage text positioning method for each sub-image to obtain a mask of the second text region.

[0016] Step S13: Perform the third stage of refined text extraction, perform 2-means clustering on each sub-image cut out in the second stage, and obtain the sub-regions of the sub-image. and Combined with and The subtracted background area is used to calculate the text position, and the pixel points of all the text clusters are merged to obtain a fine mask of the third text area. The third text area is used as the final text area, and the mask of the third text area The calculation process is as follows:

[0017]

[0018] Among them, k represents the kth subgraph, Represents both covered by R and The number of pixels masked, Represents both covered by R and The number of pixels to mask.

[0019] Furthermore, the step S11 also includes directly obtaining the mask by using the user-specified area.

[0020] Furthermore, the character area destruction process in step S2 is specifically as follows:

[0021]

[0022] in, represents the original latent space noise obtained by inversion, z' T represents the original latent space noise after the text area is corrupted by Gaussian noise, μ and σ represent The mean and variance of All-1 matrices of the same size.

[0023] Furthermore, the step S3 specifically includes:

[0024] In the process of restoring the background, we input and input z' T Initially, two parallel diffusion denoising processes are performed. The iterative denoising process is as follows:

[0025]

[0026] Where t represents the denoising step, the initial value of t is T, T = 50 to 0, and α is a list of constants related to the time step t. Indicates prompt words After the text encoder τ θ The obtained embedding, z represents the latent space feature, ∈ θ represents the pre-trained denoising network; and in each denoising step, the attention function is as follows:

[0027]

[0028] Among them, Q, K, V are the query, key and value of the attention layer respectively, and CA(Q, K, V) is the final attention calculation result, which is used to input into the next layer of the model;

[0029] The keys and values ​​are extracted from the self-attention layer of the source image denoising process and injected into the reconstruction process according to the text region mask. The keys and values ​​are only injected in one self-attention layer at the front end of the U-Net network, two self-attention layers at the back end, and the time step of t=45→0. The injection process is as follows:

[0030]

[0031] Among them, K′, V′ represent the key and value of self-attention in the denoising process of reconstructing the damaged latent space graph of the text, and K * ,V * They represent the keys and values ​​extracted from the self-attention layer of the source image denoising process, respectively;

[0032] Furthermore, the latent space map is saved at each inversion step and replaced before each denoising step for extracting keys and values ​​to eliminate errors.

[0033] Furthermore, the step S3 further comprises: when t=2 steps, the source latent space map of the non-text area Reintroduced into the background, the specific processing is as follows:

[0034]

[0035] Among them, z′2, They respectively represent the latent space map at t=2 in the denoising process of reconstructing the damaged latent space map of the text, and the latent space map at t=2 in the denoising process from the source image.

[0036] The present invention has the following advantages: by deeply exploring the editing, positioning, and control capabilities of the diffusion model, a process is constructed that does not require training, labeling, or user-provided masks, and ultimately completes the complete elimination of the text area, avoiding readable text residues left by iterative erasure. The present invention realizes automatic elimination of scene text without data labeling and network training, while retaining the interface for users to provide masks, and can simultaneously support the selective erasure function. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0038] Figure 1 This is an execution flow chart of a scene text elimination method without data annotation and network training of the present invention;

[0039] Figure 2 A schematic block diagram of the detailed process of text positioning, text destruction, and background restoration of the present invention;

[0040] Figure 3 This is an example of visualization of attention graphs at different word levels and after aggregation in a specific embodiment of the present invention;

[0041] Figure 4 It is a schematic diagram of the effect of scene text removal under the present invention and other existing processing methods. DETAILED DESCRIPTION

[0042] The embodiment of the present application aims at the problem that the existing operation processes such as training, labeling or user-provided masks are required to completely eliminate text areas, and provides a scene text elimination method that is free of data labeling and network training. The overall idea of ​​the technical solution of the present invention is as follows: an open source pre-trained model stablediffusion v1.5 is used as the basis of the pre-trained diffusion denoising model of the present invention, the data input into the U-Net is pre-processed, and the inversion and reconstruction processes are adjusted (the variational autoencoder and text encoder in the model are not changed), so as to improve the text area destruction capability and the background restoration effect.

[0043] In order to better understand the above technical solution, the technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0044] See also Figures 1 to 4 As shown, the present invention provides a scene text elimination method without data labeling and network training, the method comprising:

[0045] Step S1: hierarchical text localization is performed in the inversion process of the pre-trained diffusion denoising model by obtaining the mask of the first text area from the original image. The original image corresponding to the first text area is cropped and enlarged to obtain multiple sub-images, and then the mask of the second text area is obtained from the sub-images. Perform 2-means clustering on the subgraph to obtain subgraph subregions, combined with and The background area and sub-image sub-area obtained by subtraction calculate the mask of the refined third text area The third text area is used as the final text area;

[0046] Step S2, before reconstructing the latent space noise obtained by inversion, the text area is pre-destroyed with random Gaussian noise to obtain a processed latent space map;

[0047] Step S3, input the processed latent space map into the pre-trained diffusion denoising model to automatically restore the background area to obtain the image after the text is removed. During the restoration process, the keys and values ​​are extracted from the self-attention layer of the original image denoising process, and the keys and values ​​are injected into the reconstruction process according to the third text area mask, and the keys and values ​​are only injected into the front-end self-attention layer and the back-end two self-attention layers of the pre-trained diffusion denoising model and the time step of t=45→0, and the latent space map is saved in each inversion step for replacing the latent space map before each denoising step for extracting keys and values.

[0048] Preferably, the step S1 specifically includes:

[0049] Step S11: Perform the first stage of text localization. During inversion, use "text letter character" as the conditional prompt word to extract the attention map. And extract the ending character " <endoftext>The corresponding attention map Aggregate the attention map and perform 3-means clustering on the aggregated attention to obtain the mask of the first text region The aggregation process of the attention map is as follows:

[0050]

[0051] in, represents the aggregated attention, Represents the end character ( <endoftext>) corresponds to the attention map, γ is a constant coefficient;

[0052] Step S12: Perform the second stage of text positioning by cutting and enlarging the original image portion corresponding to each sub-region in the first text region to obtain a sub-image set. Then, for each sub-image, the first stage text positioning method is used to obtain the mask of the second text area. Specifically, referring to the text localization method in the first stage, the text localization method in the second stage is: by obtaining the corresponding attention map, aggregating the attention map, and performing a 2-means clustering operation after aggregation, and then filling the size of the sub-map back to the size of the original image for subsequent merging;

[0053] Step S13: Perform the third stage of refined text extraction, perform 2-means clustering on each sub-image cut out in the second stage, and obtain the sub-regions of the sub-image. and (that is, the 2-means clustering result of the area corresponding to the mask in the second stage on the sub-image), and combined with and The subtracted background area is used to calculate the text position, and the pixel points of all the text clusters are merged to obtain a fine mask of the third text area. The third text area is used as the final text area, and the mask of the third text area The calculation process is as follows:

[0054]

[0055] Among them, k represents the kth subgraph, Represents both covered by R and The number of pixels masked, Represents both covered by R and The number of pixels to mask. Represents the element-by-element multiplication of two 01 masks, which is equivalent to taking the intersection, that is, taking the value of 1 in R and The pixel value of 1 is also The internal value is 1, and the value is 1, which is covered by the mask. Similarly;

[0056] Preferably, the step S11 further includes directly obtaining the mask by using a user-specified area.

[0057] Preferably, the character area destruction process in step S2 is specifically as follows:

[0058]

[0059] in, represents the original latent space noise obtained by inversion, z' T represents the original latent space noise after the text area is corrupted by Gaussian noise, μ and σ represent The mean and variance of The same size matrix of all 1s. ∈ is pure Gaussian noise, and this expression is an expression that obeys the Gaussian distribution.

[0060] Preferably, the step S3 specifically includes:

[0061] In the process of restoring the background, we input and input z′ T Initially, two parallel diffusion denoising processes are performed. The iterative denoising process is as follows:

[0062]

[0063] Where t represents the denoising step, the initial value of t is T (T is a special value of t, indicating the starting step of denoising, zt is iterative, each time t-1, this iterative process starts from t=T), for example, T=50 to 0, α is a list of constants related to the time step t, Indicates prompt words After the text encoder τ θ The obtained embedding, z represents the latent space feature, ∈ θ represents the pre-trained denoising network, the diffusion step starts from z T (Right now or z′ T )start;

[0064] In each denoising step, the attention function is as follows:

[0065]

[0066] Among them, Q, K, and V are the query, key, and value of the attention layer respectively. CA(Q, K, V) is the final attention calculation result, which is used to input into the next layer of the model. T represents matrix transpose, d is a scalar determined by K;

[0067] The keys and values ​​are extracted from the self-attention layer of the source image denoising process, and injected into the reconstruction process according to the text region mask. The keys and values ​​are only injected into one self-attention layer at the front end of the U-Net network, two self-attention layers at the back end (the entire denoising network is composed of blocks, each block contains a self-attention layer and other layer structures) and the time step t=45→0. The injection process is as follows:

[0068]

[0069] Among them, K′, V′ represent the key and value of self-attention in the denoising process of reconstructing the damaged latent space graph of the text, and K * ,V * denote the keys and values ​​extracted from the self-attention layer of the source image denoising process, respectively, and their superscripts are the same as and z' T Corresponding to; in the above formula The dilated convolution can be used

[0070] In addition, the latent space map is saved at each inversion step and replaced before each denoising step for extracting keys and values ​​(i.e., zt replaces z at the same time step t at each iteration) to eliminate errors.

[0071] Preferably, the step S3 further comprises: when t=2 steps, converting the source latent space map of the non-text area Reintroduce the background, making sure it roughly matches the original image:

[0072]

[0073] Among them, z'2, They respectively represent the latent space map at t=2 in the denoising process of reconstructing the damaged latent space map of the text, and the latent space map at t=2 in the denoising process from the source image.

[0074] The present invention deeply explores the editing, positioning and control capabilities of the diffusion model, and constructs a process that does not require training, labeling, or user-provided masks, ultimately completing the complete elimination of text areas and avoiding readable text residues left behind by iterative erasure. At the same time, the present invention also retains the interface for users to provide masks and the ability to selectively erase.

[0075] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.< / endoftext> < / endoftext> < / endoftext> < / endoftext>

Claims

1. A scene text removal method without data labeling and network training, characterized by: The method comprises: Step S1: hierarchical text localization is performed in the inversion process of the pre-trained diffusion denoising model by obtaining the mask of the first text area from the original image. The original image corresponding to the first text area is cropped and enlarged to obtain multiple sub-images, and then the mask of the second text area is obtained from the sub-images. Perform 2-means clustering on the subgraph to obtain subgraph subregions, combined with and The background area and sub-image sub-area obtained by subtraction calculate the mask of the refined third text area The third text area is used as the final text area; Step S2, before reconstructing the latent space noise obtained by inversion, the text area is pre-destroyed with random Gaussian noise to obtain a processed latent space map; Step S3: input the processed latent space map into the pre-trained diffusion denoising model to automatically restore the background area to obtain an image after text removal.

2. The scene text removal method without data labeling and network training according to claim 1, characterized in that: The step S1 specifically includes: Step S11: Perform the first stage of text localization. During inversion, use "text letter character" as the conditional prompt word to extract the attention map. And extract the ending character " <endoftext>The corresponding attention map Aggregate the attention map and perform 3-means clustering on the aggregated attention to obtain the mask of the first text region The aggregation process of the attention map is as follows:< / endoftext> in, represents the aggregated attention, Represents the end character ( <endoftext> ) corresponds to the attention map, γ is a constant coefficient;< / endoftext> Step S12: Perform the second stage of text positioning by cutting and enlarging the original image portion corresponding to each sub-region in the first text region to obtain a sub-image set, and then use the first stage text positioning method for each sub-image to obtain a mask of the second text region. Step S13: Perform the third stage of refined text extraction, perform 2-means clustering on each sub-image cut out in the second stage, and obtain the sub-regions of the sub-image. and Combined with and The subtracted background area is used to calculate the text position, and the pixel points of all the text clusters are merged to obtain a fine mask of the third text area. The third text area is used as the final text area, and the mask of the third text area The calculation process is as follows: Among them, k represents the kth subgraph, Represents both covered by R and The number of pixels masked, Represents both covered by R and The number of pixels to mask.

3. The scene text removal method without data labeling and network training according to claim 2 is characterized in that: The step S11 also includes directly obtaining a mask by using a user-specified area.

4. The scene text removal method without data labeling and network training according to claim 1, characterized in that: The character area destruction process in step S2 is specifically as follows: in, represents the original latent space noise obtained by inversion, z' T represents the original latent space noise after the text area is corrupted by Gaussian noise, μ and σ represent The mean and variance of All-1 matrices of the same size.

5. The scene text removal method without data labeling and network training according to claim 1, characterized in that: The step S3 specifically includes: In the process of restoring the background, we input and input Z' T Initially, two parallel diffusion denoising processes are performed. The iterative denoising process is as follows: Where t represents the denoising step, the initial value of t is T, and α is a list of constants related to time step t. Indicates prompt words After the text encoder τ θ The obtained embedding, z represents the latent space feature, ∈ θ represents the pre-trained denoising network; and in each denoising step, the attention function is as follows: Among them, Q, K, V are the query, key and value of the attention layer respectively, and CA(Q, K, V) is the final attention calculation result, which is used to input into the next layer of the model; The keys and values ​​are extracted from the self-attention layer of the source image denoising process and injected into the reconstruction process according to the text region mask. The keys and values ​​are only injected in one self-attention layer at the front end of the U-Net network, two self-attention layers at the back end, and the time step of t=45→0. The injection process is as follows: Among them, K′, V′ represent the key and value of self-attention in the denoising process of reconstructing the damaged latent space graph of the text, and K * ,V * They represent the keys and values ​​extracted from the self-attention layer of the source image denoising process, respectively; Furthermore, the latent space map is saved at each inversion step and replaced before each denoising step for extracting keys and values ​​to eliminate errors.

6. The scene text removal method without data labeling and network training according to claim 5, characterized in that: The step S3 further comprises: when t=2 steps, the source latent space map of the non-text area Reintroduced into the background, the specific processing is as follows: Among them, z'2, They respectively represent the latent space map at t=2 in the denoising process of reconstructing the damaged latent space map of the text, and the latent space map at t=2 in the denoising process from the source image.

Images