A method and system for scene text veil removal based on hierarchical context-aware interaction reconstruction

Abstract

This invention discloses a method and system for scene text veil removal based on hierarchical context-aware interactive reconstruction. The method includes: performing preliminary text localization and text region background reconstruction using a lightweight shared encoder-decoder network to predict a coarse text mask and a coarse text erasure image; refining the coarse text mask using a depthwise separable convolutional network, and performing hierarchical refinement of the coarse text mask using a hierarchical mask decoder; downsampling the coarse text erasure image using a U-Net feature extractor, and learning the contextual relationship between the complex background and text region layer by layer based on the hierarchical refined mask to decode the scene text erasure image; and adding the original image and the scene text erasure image to obtain the final predicted image. This invention provides more accurate visual information about text location to guide text region reconstruction, achieving better reconstruction results and can be widely applied in the field of computer technology.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and system for scene text veiling based on hierarchical context-aware interactive reconstruction. Background Technology

[0002] With the rapid development of computer vision technology, scene text erasure has become an important research direction in this field. The core task of scene text erasure is to visually erase text from an image while preserving the integrity of the scene background and the continuity of the erased area as much as possible. This technology has broad application prospects in image editing, data processing, and privacy protection. For example, in image editing applications, removing unwanted text information from photos can make them cleaner and more aesthetically pleasing; in data processing, automated text erasure technology can efficiently clean the training data of neural networks; and in privacy protection, scene text erasure technology can effectively remove sensitive information from images. However, scene text erasure is an extremely challenging task, with its main difficulties concentrated in two aspects: the challenge of text localization and the complexity of background reconstruction. First, the challenge of text localization lies in multiple aspects such as text language, stylistic features, and geometric layout. Text language may include languages ​​from different countries and regions; stylistic features may include different fonts, sizes, and colors; and geometric layout may involve various forms such as long text, slanted text, and perspective text. These diverse styles bring great complexity and difficulty to text localization. Secondly, the complexity of background reconstruction stems primarily from the texture structure and occlusion issues of the scene background. Scene text is often situated on complex backgrounds, which may contain intricate textures and rich color variations, and may even occlude with the text itself. Even subtle text erasure operations, if not handled properly, can lead to visual inconsistencies and detract from the overall aesthetics of the image. Therefore, when performing background reconstruction, accurately reconstructing the background areas covered by text while avoiding excessive modification of unnecessary areas is key to improving erasure effectiveness and efficiency. In conclusion, scene text erasure in natural scenes is an extremely challenging task in the field of computer vision.

[0003] End-to-end scene text erasure models aim to achieve scene text erasure within a unified model. Current methods can be broadly categorized into two types: single-stage and two-stage scene text erasure methods. Single-stage methods implicitly perform text localization and background reconstruction through a single network, a simple process. For example, one method employs an image-to-image translation network structure and uses a local perceptual discriminator to maintain visual consistency of the text erasure region. To further improve the usability of scene text erasure networks, related techniques utilize a coarse-to-fine strategy to refine coarse predictions, reconstructing text regions at the image level. To reduce training data annotation costs, related techniques construct a self-supervised framework with a controllable synthesis module, improving the performance of the scene text erasure network. Furthermore, some researchers have introduced visual Transformer models into scene text erasure tasks. Related techniques propose replacing traditional convolutional structures with visual Transformer structures for long-range modeling, improving the visual effect of text erasure. Two-stage scene text erasure methods explicitly perform text localization and background reconstruction separately. For example, MTRNet directly uses text region mask labels to locate text and reconstructs text regions using a conditional adversarial generative network. Similarly, SSSSTRM proposes using gated attention and region of interest generation techniques to improve text erasure. Some methods do not rely on real text region masks but instead use advanced text detectors to obtain them. For example, CTRNet locates text regions and explores local-global cues to accurately recover the background. Unlike methods that require image-mask pairs for training, related techniques introduce a pre-trained PSENet for text detection and utilize another pre-trained image inpainting model for background recovery, reducing the need for training data. Similarly, related techniques propose training the text detector and background inpainting module separately and then merging them into a single model. To further improve erasure performance, some researchers integrate text mask prediction and background inpainting into an end-to-end framework. For example, MTRNet++ uses two branches to predict the text mask and the inpainted image respectively, and refines the inpainting results through a fine-grained inpainting branch. Similarly, SAEN generates a text mask and a coarsely repaired image in the first stage, and then uses a stroke-aware refinement network to generate a refined image. This refinement strategy has been further developed and applied in subsequent methods. For example, PERT employs multiple progressive erasure stages and a region-based modification strategy to generate better results.

[0004] Generally speaking, these models struggle to reconstruct the background and locate the text when dealing with images with complex backgrounds and text shapes, often resulting in text residue and discontinuous background after erasure. Summary of the Invention

[0005] The main objective of this invention is to propose a scene text veil removal method and system based on hierarchical context-aware interactive reconstruction, which can provide more accurate visual information of text location to guide the reconstruction of text regions and achieve better reconstruction results.

[0006] To achieve the above objectives, one aspect of this invention proposes a scene text veil removal method based on hierarchical context-aware interaction reconstruction, comprising the following steps:

[0007] Preliminary text localization and text region background reconstruction are performed using a lightweight shared encoder-decoder network, which predicts coarse text masks and coarse text erased images.

[0008] The coarse text mask is refined using a depthwise separable convolutional network, and then the coarse text mask is further refined using four-level mask decoders to obtain a refined layered mask.

[0009] The coarse text erasure image is downsampled using the U-Net feature extractor, and the contextual relationship between the complex background and text region is learned layer by layer based on the hierarchical fine mask to decode the scene text erasure image.

[0010] The original image is added to the scene text erased image to obtain the final predicted image.

[0011] In some embodiments, the preliminary text localization and text region background reconstruction using a lightweight shared encoder-decoder network to predict a coarse text mask and a coarse text erasure image includes the following steps:

[0012] Input a scene text image, use the initial text erasure network to coarsely locate and erase the text in the scene text image in parallel, and output a coarse text erasure image and a coarse text mask;

[0013] The scene text image is input into the initial text erasure network, and the expression for generating text image features through the image encoder is as follows: in, Represents text image features; Represents an image encoder; Represents scene text images; Represents an image encoder Learnable parameters;

[0014] The coarse text erasure image and the aforementioned coarse text mask The expression is:

[0015]

[0016] in, and They represent decoders respectively. and The parameters that can be learned.

[0017] In some embodiments, the process of refining the coarse text mask using a depthwise separable convolutional network, and then further refining the coarse text mask using four-level mask decoders to obtain a refined layered mask, includes the following steps:

[0018] Utilizing depthwise separable convolutional networks Extracting rough text erasure images Hierarchical features The expression for this process is: ,in, , Indicates the first Features of the layer Represents network Learnable parameters;

[0019] Coarse text mask and characteristics The data is fed into four different mask decoders. To obtain a fine-grained mask. The expression for this process is: ,in, Indicates mask decoder The parameters that can be learned.

[0020] In some embodiments, the step of downsampling the coarse text erasure image using a U-Net feature extractor and learning the contextual relationship between the complex background and text regions layer by layer based on the hierarchical fine mask to decode the scene text erasure image includes the following steps:

[0021] Using U-Net feature extraction network Generate coarse text erasure image High-dimensional semantic features ;

[0022] In hierarchical fine masking Guided by this, the background of the text region is reconstructed layer by layer; among them, in the first... Layer, through convolutional layers with residual connections Features of the previous layer Upsampling is performed, and a channel-level context-aware reconstruction layer is used. Reconstruct the features;

[0023] Based on the visual continuity between the reconstructed text region and the background region, a channel-level context-aware reconstruction layer is introduced. This is used to capture the contextual relationship between the text area and the background area in order to reconstruct the text area background.

[0024] In some embodiments, the high-dimensional semantic features The expression is: ,in, Represents network Learnable parameters in;

[0025] The channel-level context-aware reconstruction layer The expression for the process of reconstructing features is: in, , and They represent the first Convolutional layers and reconstruction layer Learnable parameters Represents the features of the previous layer. This indicates the feature that will be reconstructed.

[0026] In some embodiments, a channel-level context-aware reconstruction layer is introduced based on the visual continuity between the reconstructed text region and the background region. To capture the contextual relationship between the text region and the background region in order to reconstruct the text region background, the following steps are included:

[0027] Using the first Layer fine mask Foreground features and background features Separation, the expression for this process is:

[0028]

[0029] in, This represents the Hadamard product operation;

[0030] Based on foreground characteristics A set of channel-level foreground features was obtained. Among them, features The Each element can be calculated using the following formula:

[0031]

[0032] in, Representation of features The Channel activation diagram, It represents the expected value of all elements in the matrix;

[0033] Obtain a set of channel-level background features ;

[0034] Learn the contextual relationships between the foreground and background of the text and generate semantic aggregation features. The expression for this process is: ,in, This indicates that two matrices are concatenated. and These represent the learnable weight parameters and bias parameters, respectively.

[0035] The foreground and background fusion module is used to obtain semantic representations of the foreground and background respectively. :

[0036]

[0037] in, Represents the learnable weight parameters; Indicates the bias parameter;

[0038] Using a channel-level weighting mechanism, foreground features are analyzed separately. and background features The features are weighted and then summed to obtain the reconstructed features. :

[0039]

[0040] in, This means broadcasting the feature vectors along the spatial dimension into a shape of... The tensor.

[0041] In some embodiments, adding the original image to the scene text erasure image to obtain the final predicted image includes the following steps:

[0042] After four layers of progressive background reconstruction, through a 1 1 convolutional layer The text-erased image is decoded and added to the original image to obtain the final predicted image. The expression for this process is:

[0043]

[0044] in, Represents the final predicted image; This represents the result after four layers of progressive background reconstruction. Represents the Hadamard product operation; Represents the features of the fourth convolutional layer; Represents scene text images; Represents a convolutional layer The parameters that can be learned.

[0045] Another aspect of this invention provides a scene text veil removal system based on hierarchical context-aware interaction reconstruction, comprising:

[0046] The first module is used to perform preliminary text localization and text region background reconstruction through a lightweight shared encoder-decoder network, predicting a coarse text mask and a coarse text erasure image;

[0047] The second module is used to refine the coarse text mask using a depthwise separable convolutional network, and then to refine the coarse text mask hierarchically through four level mask decoders to obtain a hierarchically refined mask.

[0048] The third module is used to downsample the coarse text erasure image through the U-Net feature extractor, and learn the contextual relationship between the complex background and the text region layer by layer according to the hierarchical fine mask to decode the scene text erasure image;

[0049] The fourth module is used to add the original image to the scene text erasure image to obtain the final predicted image.

[0050] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0051] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0052] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0053] The embodiments of this invention include at least the following beneficial effects: This invention provides a scene text veil removal method and system based on hierarchical context-aware interactive reconstruction. This scheme uses a lightweight shared encoder-decoder network for initial text localization and text region background reconstruction, predicting a coarse text mask and a coarse text erasure image. A depthwise separable convolutional network is used to refine the coarse text mask, and then four hierarchical mask decoders are used to perform hierarchical refinement of the coarse text mask, obtaining a hierarchical refined mask. A U-Net feature extractor is used to downsample the coarse text erasure image, and the contextual relationship between the complex background and text region is learned layer by layer based on the hierarchical refined mask to decode the scene text erasure image. The original image and the scene text erasure image are added together to obtain the final predicted image. The embodiments of this invention can provide more accurate visual information about text location to guide the reconstruction of text regions, achieving better reconstruction results. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0055] Figure 2 This is a flowchart of the overall steps provided in the embodiments of the present invention;

[0056] Figure 3 This is a flowchart illustrating the specific implementation steps provided in the embodiments of the present invention;

[0057] Figure 4 This is a diagram of the channel-level context-aware reconstruction mechanism provided in an embodiment of the present invention;

[0058] Figure 5 This is a visualization comparing the performance of the method of this invention with various text erasure methods, provided by an embodiment of the invention.

[0059] Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0061] It is understood that the terms “first,” “second,” etc., used in this invention may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…” or “when…” or “in response to determination.”

[0062] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0064] The scene text veiling removal method and system based on hierarchical context-aware interactive reconstruction provided in this invention relates to the field of computer technology. The scene text veiling removal method based on hierarchical context-aware interactive reconstruction provided in this invention can be applied to terminals, servers, or software running on terminals or servers. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, etc., but is not limited to these; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the server can also be a node server in a blockchain network; the software can be an application implementing the scene text veiling removal method based on hierarchical context-aware interactive reconstruction, etc., but is not limited to the above forms.

[0065] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0066] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0067] Server 101 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0068] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0069] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc. It can also be a vehicle-mounted terminal of the various device types described above, but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0070] Exemplary based on Figure 1The implementation environment shown in this embodiment of the invention provides a scene text veiling method based on hierarchical context-aware interactive reconstruction. The following description uses the application of this scene text veiling method based on hierarchical context-aware interactive reconstruction in server 101 as an example. It can be understood that this method can also be applied in terminal 102.

[0071] Reference Figure 2 , Figure 2 This is a flowchart illustrating a scene text veiling removal method based on hierarchical context-aware interactive reconstruction applied to a server, provided in an embodiment of the present invention. The execution subject of this method can be any of the aforementioned computer devices (including servers or terminals). (Refer to...) Figure 2 The method may include the following steps:

[0072] Preliminary text localization and text region background reconstruction are performed using a lightweight shared encoder-decoder network, which predicts coarse text masks and coarse text erased images.

[0073] The coarse text mask is refined using a depthwise separable convolutional network, and then the coarse text mask is further refined using four-level mask decoders to obtain a refined layered mask.

[0074] The coarse text erasure image is downsampled using the U-Net feature extractor, and the contextual relationship between the complex background and text region is learned layer by layer based on the hierarchical fine mask to decode the scene text erasure image.

[0075] The original image is added to the scene text erased image to obtain the final predicted image.

[0076] In some embodiments, the preliminary text localization and text region background reconstruction using a lightweight shared encoder-decoder network to predict a coarse text mask and a coarse text erasure image includes the following steps:

[0077] Input a scene text image, use the initial text erasure network to coarsely locate and erase the text in the scene text image in parallel, and output a coarse text erasure image and a coarse text mask;

[0078] The scene text image is input into the initial text erasure network, and the expression for generating text image features through the image encoder is as follows: in, Represents text image features; Represents an image encoder; Represents scene text images; Represents an image encoder Learnable parameters;

[0079] The coarse text erasure image and the aforementioned coarse text mask The expression is:

[0080]

[0081] in, and They represent decoders respectively. and The parameters that can be learned.

[0082] In some embodiments, the process of refining the coarse text mask using a depthwise separable convolutional network, and then further refining the coarse text mask using four-level mask decoders to obtain a refined layered mask, includes the following steps:

[0083] Utilizing depthwise separable convolutional networks Extracting rough text erasure images Hierarchical features The expression for this process is: ,in, , Indicates the first Features of the layer Represents network Learnable parameters;

[0084] Coarse text mask and characteristics The data is fed into four different mask decoders. To obtain a fine-grained mask. The expression for this process is: ,in, Indicates mask decoder The parameters that can be learned.

[0085] In some embodiments, the step of downsampling the coarse text erasure image using a U-Net feature extractor and learning the contextual relationship between the complex background and text regions layer by layer based on the hierarchical fine mask to decode the scene text erasure image includes the following steps:

[0086] Using U-Net feature extraction network Generate coarse text erasure image High-dimensional semantic features ;

[0087] In hierarchical fine masking Guided by this, the background of the text region is reconstructed layer by layer; among them, in the first... Layer, through convolutional layers with residual connections Features of the previous layer Upsampling is performed, and a channel-level context-aware reconstruction layer is used. Reconstruct the features;

[0088] Based on the visual continuity between the reconstructed text region and the background region, a channel-level context-aware reconstruction layer is introduced. This is used to capture the contextual relationship between the text area and the background area in order to reconstruct the text area background.

[0089] In some embodiments, the high-dimensional semantic features The expression is: ,in, Represents network Learnable parameters in;

[0090] The channel-level context-aware reconstruction layer The expression for the process of reconstructing features is: in, , and They represent the first Convolutional layers and reconstruction layer Learnable parameters Represents the features of the previous layer. This indicates the feature that will be reconstructed.

[0091] In some embodiments, a channel-level context-aware reconstruction layer is introduced based on the visual continuity between the reconstructed text region and the background region. To capture the contextual relationship between the text region and the background region in order to reconstruct the text region background, the following steps are included:

[0092] Using the first Layer fine mask Foreground features and background features Separation, the expression for this process is:

[0093]

[0094] in, This represents the Hadamard product operation;

[0095] Based on foreground characteristics A set of channel-level foreground features was obtained. Among them, features The Each element can be calculated using the following formula:

[0096]

[0097] in, Representation of features The Channel activation diagram, It represents the expected value of all elements in the matrix;

[0098] Obtain a set of channel-level background features ;

[0099] Learn the contextual relationships between the foreground and background of the text and generate semantic aggregation features. The expression for this process is: ,in, This indicates that two matrices are concatenated. and These represent the learnable weight parameters and bias parameters, respectively.

[0100] The foreground and background fusion module is used to obtain semantic representations of the foreground and background respectively. :

[0101]

[0102] in, Represents the learnable weight parameters; Indicates the bias parameter;

[0103] Using a channel-level weighting mechanism, foreground features are analyzed separately. and background features The features are weighted and then summed to obtain the reconstructed features. :

[0104]

[0105] in, This means broadcasting the feature vectors along the spatial dimension into a shape of... The tensor.

[0106] In some embodiments, adding the original image to the scene text erasure image to obtain the final predicted image includes the following steps:

[0107] After four layers of progressive background reconstruction, through a 1 1 convolutional layer The text-erased image is decoded and added to the original image to obtain the final predicted image. The expression for this process is:

[0108]

[0109] in, Represents the final predicted image; This represents the result after four layers of progressive background reconstruction. Represents the Hadamard product operation; Represents the features of the fourth convolutional layer; Represents scene text images; Represents a convolutional layer The parameters that can be learned.

[0110] The specific implementation process of the method of the present invention will be described in detail below using a specific application scenario as an example:

[0111] refer to Figure 3 The scene text removal method of the present invention may include the following steps:

[0112] 1) Parallel coarse text localization and erasure. This embodiment uses a lightweight shared encoder-decoder network for initial text localization and text region background reconstruction, thereby predicting a coarse text mask and text erasure image.

[0113] 2) Hierarchical Fine-Grained Mask Generation. After obtaining the coarse text mask and the erased text image, we further refine the coarse text mask. First, a depthwise separable convolution network is used to extract the geometric features of the coarse text erased image. Then, based on the extracted geometric features, the coarse text mask is hierarchically refined using four hierarchical mask decoders to generate four hierarchical fine-grained masks. These hierarchical fine-grained masks will be used to guide subsequent background reconstruction and the stitching of the erased image with the original image.

[0114] 3) Context-aware progressive background reconstruction. First, a U-Net feature extractor downsamples the coarse text erasure image to extract high-dimensional semantic features. Then, using a channel-level context-aware interaction mechanism, guided by hierarchical fine-grained masks, the contextual relationship between complex backgrounds and text regions is learned layer by layer, and the scene text erasure image is decoded. Finally, based on the image segmentation of the first fine-grained mask, the original image and the scene text erasure image are added together to obtain the final predicted image.

[0115] In this embodiment of the invention, the input is an RGB scene text image. The output is a text-erased image. All input images were scaled down to... size.

[0116] The specific steps are as follows:

[0117] 1) The initial text erasure network is used to coarsely locate and erase text in the scene text image in parallel. Its input is the scene text image. The output is a coarse text-erased image. and coarse text mask Specifically, scene text images It is input into the initial text erasure network, and then passed through the image encoder. Generate text image features :

[0118]

[0119] in, Indicates encoder Learnable parameters are then processed by a parallel image decoder. and mask decoder Predict coarse text erasure images separately and coarse text mask :

[0120]

[0121] in, and They represent decoders respectively. and The parameters that can be learned.

[0122] 2) A hierarchical mask generation network is used to further refine the coarse text mask, with the input being the coarse text erasure image. and coarse text mask The output is a fine mask with 4 levels. This embodiment utilizes a depthwise separable convolutional network. Extracting rough text erasure images Hierarchical features :

[0123]

[0124] in, , Indicates the first Features of the layer Represents network Learnable parameters. Next, a coarse text mask is applied. and characteristics The data is fed into four different mask decoders. To obtain a fine-grained mask. :

[0125]

[0126] in, Indicates mask decoder The parameters that can be learned.

[0127] 3) Progressive background reconstruction guided by hierarchical fine-grained masks. First, a U-Net feature extraction network is used... Generate coarse text erasure image High-dimensional semantic features :

[0128]

[0129] in, Represents network Learnable parameters in the [database]. Next, in the hierarchical fine-grained mask... Guided by this principle, this embodiment reconstructs the background of the text region layer by layer. In the first... Layer, through convolutional layers with residual connections Features of the previous layer Upsampling is performed, and a channel-level context-aware reconstruction layer is used. Reconstruct the features:

[0130]

[0131] in, , and They represent the first Convolutional layers and reconstruction layer The parameters that can be learned.

[0132] 4) To achieve visual continuity between the reconstructed text area and the background area, refer to Figure 4 This embodiment introduces a channel-level context-aware reconstruction layer. This is used to capture the contextual relationship between the text region and the background region, in order to better reconstruct the text region's background. For the first... Reconstruction layer Its input is the previous convolutional layer. Output features The output is the reconstructed features. First, this embodiment utilizes the first... Layer fine mask Foreground (text) features and background features Separation:

[0133]

[0134] in, This represents the Hadamard product operation. Next, this embodiment uses foreground features... A set of channel-level foreground features was further obtained. Among them, features The Each element can be calculated using the following formula:

[0135]

[0136] in, Representation of features The Channel activation diagram, This represents the expected value of all elements in the matrix. Similarly, this embodiment can obtain a set of channel-level background features. Then, this embodiment learns the contextual relationship between the foreground and background of the text and generates semantic aggregation features. :

[0137]

[0138] in, This indicates that two matrices are concatenated. and These represent the learnable weight parameters and bias parameters, respectively. Next, this embodiment uses a foreground / background fusion module to obtain the semantic representations of the foreground and background, respectively. :

[0139]

[0140] in, and These represent the learnable weight parameters and bias parameters, respectively. Finally, this embodiment utilizes a channel-level weighting mechanism to separately weight the foreground features. and background features The features are weighted and then summed to obtain the reconstructed features. :

[0141]

[0142] in, This indicates that the feature vectors are broadcast along the spatial dimension into a shape of ( The tensor of ).

[0143] 5) After four layers of progressive background reconstruction, this embodiment uses a 1 1 convolutional layer The text-erased image is decoded and added to the original image to obtain the final predicted image:

[0144]

[0145] in, Represents a convolutional layer The parameters that can be learned.

[0146] The method proposed in this embodiment simultaneously predicts the text mask and the text erasure image, each requiring supervision by corresponding information. Since the text mask comprises four layers, this embodiment supervises each layer. Specifically, for the... Layer text mask This embodiment uses binary cross-entropy loss. Monitor its distribution:

[0147]

[0148] in, Indicates a text mask label. Indicates to Perform adaptive pooling operation to obtain the same as Masks of the same shape. Since binary cross-entropy loss is a pixel-level supervision method, it often produces incorrect predictions when the text layout is complex. To constrain the mask from a global perspective, this embodiment employs soft mask IoU loss. :

[0149]

[0150] in, This represents the sum of all pixel values ​​in the mask. Additionally, this embodiment also employs binarization loss. This forces the mask value closer to 0 or 1 to improve the visual continuity between the text-erased image and the original image:

[0151]

[0152] in, This means taking the mask at each pixel position. and mask The minimum value between. The final loss function for the mask. It is expressed as follows:

[0153]

[0154] in, , These are the weight coefficients for the three loss functions. Meanwhile, for the reconstructed image... The main objective of this embodiment is to reconstruct the foreground (text) region in the image. Therefore, this embodiment needs to ensure that the reconstructed region matches the label image. The corresponding regions are similar, thus introducing a foreground loss:

[0155]

[0156] in, This represents the number of pixels in the three channels of the image. Meanwhile, to achieve better visual continuity, this embodiment introduces a visual perception loss:

[0157]

[0158] in, This indicates that after the image is input into the VGG-16 network, at the [number]th [time]... Feature map of the layer This indicates the number of pixels in the feature map. This indicates the number of feature maps used in this embodiment. However, visual perception loss cannot effectively capture complex text structures and backgrounds. Therefore, this embodiment employs shape similarity constraints on the reconstructed image.

[0159]

[0160] in, and It is a factor in stable weak denominator division. and These represent the predicted images. and label images The mean, and These represent the predicted images. and label images standard deviation Represents the predicted image and label images The covariances of these variables are calculated as follows:

[0161]

[0162]

[0163]

[0164] Ultimately used to supervise the reconstructed image The loss function is expressed as follows:

[0165]

[0166] in, , , These are the weight coefficients of the three loss functions. Similarly, for coarse text erasure images... In this embodiment, the same loss function is used for supervision:

[0167]

[0168] in, Representing an image The loss function used throughout the framework is as follows:

[0169]

[0170] The test environment and experimental results of the hierarchical context-aware interactive reconstruction scene text veiling method proposed in this invention are as follows:

[0171] (1) Test environment:

[0172] System environment: Ubuntu 20.04.

[0173] Hardware configuration: Memory: 64GB, GPU: GeForce RTX 3090, CPU: Intel(R) Xeon(R) Gold6226R, Hard drive: 2TB.

[0174] (2) Experimental data:

[0175] This invention was trained and tested on the real dataset SCUT-EnsText and the synthetic dataset SCUT-Syn, respectively.

[0176] (3) Optimization method:

[0177] In this embodiment, the factor is set as follows: 2. 0.25 , , 0.5 0.25 and This embodiment uses the Adam optimizer with a learning rate of 0.001 and batch sizes of 12 and 60 on the two datasets. The model in this embodiment is implemented using the PyTorch v2.1.2 deep learning framework.

[0178] (4) Experimental results:

[0179] 1) Ablation experiment:

[0180] To verify the effectiveness of the method proposed in this embodiment, the initial text erasure network was set as the baseline model. As shown in Table 1, the peak signal-to-noise ratio (PSNR) of the baseline model was observed to be 34.57. Without mask guidance and channel-level context-aware interaction (CCIM), the PSNR improved by 1.80 compared to the baseline model after applying progressive background reconstruction (PBR). This improvement can be attributed to the fact that adding progressive background reconstruction transforms the network from a single-stage paradigm to a two-stage paradigm, thus allowing for more refined results. When channel-level context-aware interaction is used in progressive background reconstruction, it utilizes a coarse mask to separate text regions from the background, resulting in a 0.22 increase in PSNR while only increasing the parameters by 0.24M. This is because channel-level context-aware interaction can use a coarse mask to guide background reconstruction. When a hierarchical mask generation network (HMGN) is used to provide a fine mask, the PSNR improves by 0.23 while only increasing the parameters by more than 0.66M. This is because the fine mask provides more accurate guidance for background reconstruction.

[0181] Table 1: Validation of the proposed module

[0182]

[0183] 2) Performance comparison:

[0184] refer to Figure 5Since pre-training and data augmentation can cause significant performance fluctuations, none of the experiments reported in this embodiment employed pre-training or data augmentation strategies to ensure fairness in the comparison as much as possible. As shown in Table 2, when this embodiment was tested on the real dataset SCUT-EnsText, compared with other state-of-the-art methods, the proposed method achieved a state-of-the-art PSNR of 36.82 and MSSIM of 97.81% with fewer parameters and faster inference speed. More specifically, the model in this embodiment outperformed the single-stage method EnsNet by 7.28 and 5.07% in PSNR and MSSIM, respectively. Compared with the two-stage method CTRNet, the method in this embodiment also exhibited better performance and faster speed. For example, the method in this embodiment improved PSNR by 1.62 and speed by 40.9 fps. Furthermore, the model in this embodiment achieved a PSNR 0.27 higher than the Transformer-based method ViTEraser and was significantly faster (46 fps vs. 17 fps). Table 3 shows the quantitative comparison results of the proposed method in this embodiment with existing methods on the synthetic dataset SCUT-Syn. This embodiment demonstrates that the model achieves state-of-the-art performance across all metrics. The method in this embodiment comprehensively outperforms the Transformer-based method ViTEraser on this dataset, improving PSNR and MSSIM by 0.44 and 0.88%, respectively. Furthermore, compared to EraseNet, which only employs image thinning without masking, the model in this embodiment achieves significant improvements of 4.57 and 1.64% in PSNR and MSSIM, respectively.

[0185] Table 2: Performance comparison with various text erasure methods on the SCUT-EnsText dataset

[0186]

[0187] Table 3: Performance comparison with various text erasure methods on the SCUT-Syn dataset

[0188]

[0189] In summary, this invention separates text content, text style, and background, and uses the style features of the source text to guide style transfer changes in the content text; then, by replacing the target text with the source text, the minimized reconstruction area is obtained, and this area is reconstructed to complete the scene text editing. This provides more accurate visual information on text location to guide the reconstruction of text areas, achieving better reconstruction results.

[0190] Another aspect of this invention provides a scene text veil removal system based on hierarchical context-aware interaction reconstruction, comprising:

[0191] The first module is used to perform preliminary text localization and text region background reconstruction through a lightweight shared encoder-decoder network, predicting a coarse text mask and a coarse text erasure image;

[0192] The second module is used to refine the coarse text mask using a depthwise separable convolutional network, and then to refine the coarse text mask hierarchically through four level mask decoders to obtain a hierarchically refined mask.

[0193] The third module is used to downsample the coarse text erasure image through the U-Net feature extractor, and learn the contextual relationship between the complex background and the text region layer by layer according to the hierarchical fine mask to decode the scene text erasure image;

[0194] The fourth module is used to add the original image to the scene text erasure image to obtain the final predicted image.

[0195] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0196] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned scene text veiling removal method based on hierarchical context-aware interactive reconstruction. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0197] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0198] Please see Figure 6 , Figure 6 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0199] The processor 601 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0200] The memory 602 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called and executed by the processor 601 to execute the scene text fading method based on hierarchical context-aware interactive reconstruction of the embodiments of this invention.

[0201] The input / output interface 603 is used to implement information input and output;

[0202] The communication interface 604 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0203] Bus 605 transmits information between various components of the device (e.g., processor 601, memory 602, input / output interface 603, and communication interface 604);

[0204] The processor 601, memory 602, input / output interface 603, and communication interface 604 are connected to each other within the device via bus 605.

[0205] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described scene text veiling removal method based on hierarchical context-aware interactive reconstruction.

[0206] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0207] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0208] It should be noted that in various specific embodiments of the present invention, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of the present invention require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of the present invention acquired.

[0209] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0210] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0211] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0212] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0213] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0214] It should be understood that in this invention, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0215] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0216] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0217] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0218] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0219] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. A method for scene text hidden content removal based on hierarchical context-aware interaction reconstruction, characterized in that, Includes the following steps: Preliminary text localization and text region background reconstruction are performed using a lightweight shared encoder-decoder network, which predicts coarse text masks and coarse text erased images. The coarse text mask is refined using a depthwise separable convolutional network, and then the coarse text mask is further refined using four-level mask decoders to obtain a refined layered mask. The coarse text erasure image is downsampled using the U-Net feature extractor, and the contextual relationship between the complex background and text region is learned layer by layer based on the hierarchical fine mask to decode the scene text erasure image. The original image is added to the scene text erased image to obtain the final predicted image; The process involves preliminary text localization and text region background reconstruction using a lightweight shared encoder-decoder network to predict a coarse text mask and a coarse text erasure image, including the following steps: Input a scene text image, use the initial text erasure network to coarsely locate and erase the text in the scene text image in parallel, and output a coarse text erasure image and a coarse text mask; The scene text image is input into the initial text erasure network, and the expression for generating text image features through the image encoder is as follows: in, Represents text image features; Represents an image encoder; Represents scene text images; Represents an image encoder Learnable parameters; The rough text erasure image and the aforementioned coarse text mask The expression is: in, and They represent decoders respectively. and Learnable parameters; The process of refining the coarse text mask using a depthwise separable convolutional network, and then further refining the coarse text mask using four-level mask decoders to obtain a refined layered mask, includes the following steps: Utilizing depthwise separable convolutional networks Extracting rough text erasure images Hierarchical features The expression for this process is: ,in, , Indicates the first Features of the layer Represents network Learnable parameters; Coarse text mask and characteristics The signals are fed into four different mask decoders. To obtain a fine-grained mask. The expression for this process is: ,in, Indicates mask decoder Learnable parameters; The process of downsampling the coarse text erasure image using a U-Net feature extractor and then learning the contextual relationship between the complex background and text regions layer by layer based on the hierarchical fine mask to decode the scene text erasure image includes the following steps: Using U-Net feature extraction network Generate a rough text erasure image High-dimensional semantic features ; In hierarchical fine masking Guided by this, the background of the text region is reconstructed layer by layer; among them, in the first... Layer, through convolutional layers with residual connections Features of the previous layer Upsampling is performed, and a channel-level context-aware reconstruction layer is used. Reconstruct the features; Based on the visual continuity between the reconstructed text region and the background region, a channel-level context-aware reconstruction layer is introduced. To capture the contextual relationship between the text area and the background area, in order to reconstruct the text area background; The high-dimensional semantic features The expression is: ,in, Represents network Learnable parameters in; The channel-level context-aware reconstruction layer The expression for the process of reconstructing features is: in, , and They represent the first Convolutional layers and reconstruction layer Learnable parameters Represents the features of the previous layer. This indicates the feature that will be reconstructed.

2. The scene text hidden surface removal method based on hierarchical context-aware interaction reconstruction according to claim 1, characterized in that, Based on the visual continuity between the reconstructed text region and the background region, a channel-level context-aware reconstruction layer is introduced. To capture the contextual relationship between the text region and the background region in order to reconstruct the text region background, the following steps are included: Using the first Layer fine mask Foreground features and background features Separation; Based on foreground characteristics A set of channel-level foreground features was obtained. Among them, features The Each element can be calculated using the following formula: in, Representation of features The Channel activation diagram, It represents the expected value of all elements in the matrix; Obtain a set of channel-level background features ; Learn the contextual relationships between the foreground and background of the text and generate semantic aggregation features. The expression for this process is: ,in, This indicates concatenating two matrices. and These represent the learnable weight parameters and bias parameters, respectively. The foreground and background fusion module is used to obtain semantic representations of the foreground and background respectively. : in, Represents the learnable weight parameters; Indicates the bias parameter; Using a channel-level weighting mechanism, foreground features are analyzed separately. and background features The features are weighted and then summed to obtain the reconstructed features. : in, This means broadcasting the feature vectors along the spatial dimension into a shape of... The tensor.

3. The scene text hidden surface removal method based on hierarchical context-aware interaction reconstruction according to claim 1, characterized in that, The step of adding the original image to the scene text-erased image to obtain the final predicted image includes the following steps: After four layers of progressive background reconstruction, through a 1 1 convolutional layer The text-erased image is decoded and added to the original image to obtain the final predicted image. The expression for this process is: in, Represents the final predicted image; This represents the result after four layers of progressive background reconstruction. Represents the Hadamard product operation; This represents the features of the fourth convolutional layer; Represents scene text images; Represents a convolutional layer The parameters that can be learned.

4. A system for implementing the scene text veil removal method based on hierarchical context-aware interaction reconstruction as described in any one of claims 1-3, characterized in that, include: The first module is used to perform preliminary text localization and text region background reconstruction through a lightweight shared encoder-decoder network, predicting a coarse text mask and a coarse text erasure image; The second module is used to refine the coarse text mask using a depthwise separable convolutional network, and then to refine the coarse text mask hierarchically through four level mask decoders to obtain a hierarchically refined mask. The third module is used to downsample the coarse text erasure image through the U-Net feature extractor, and learn the contextual relationship between the complex background and the text region layer by layer according to the hierarchical fine mask to decode the scene text erasure image; The fourth module is used to add the original image to the scene text erasure image to obtain the final predicted image.

5. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 3.

Images