Scene text blanking method and system based on hierarchical context-aware interaction reconstruction

Abstract

The invention discloses a scene text blanking method and system based on hierarchical context-aware interaction reconstruction, and the method comprises the steps: carrying out the preliminary text positioning and text region background reconstruction through a lightweight shared coding-separation decoding network, and predicting to obtain a rough text mask and a rough text erasure image; the rough text mask is refined by using a deep separable convolutional network, and the rough text mask is hierarchically refined by using a hierarchical mask decoder; performing down-sampling processing on the rough text erasure image through a U-Net feature extractor, learning a context relationship between a complex background and a text region layer by layer according to a hierarchical fine mask, and decoding a scene text erasure image; and adding the original image and the scene text erasure image to obtain a final prediction image. According to the embodiment of the invention, more accurate text position visual information can be provided to guide the reconstruction of the text region, a better reconstruction effect is realized, and the method can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a scene text elimination method and system based on hierarchical context-aware interactive reconstruction. Background Art

[0002] With the rapid development of computer vision technology, scene text erasure has become an important research direction in this field. The core task of scene text erasure is to visually erase the text in the image while trying to maintain the integrity of the scene background and the coherence of the erased area. Such technology has broad application prospects in image editing, data processing, privacy protection, etc. For example, in image editing applications, removing unnecessary text information from photos can make the photos cleaner and more beautiful; in data processing, automated text erasure technology can efficiently clean the training data of neural networks; in privacy protection, scene text erasure technology can effectively remove sensitive information from images. However, scene text erasure is a very challenging task, and its main difficulties are concentrated in two aspects: the challenge of text positioning and the complexity of background reconstruction. First, the challenge of text positioning is reflected in multiple aspects such as text language, style, and text geometric layout. The text language may include languages ​​of different countries and regions, the style of the text may include different fonts, sizes, and colors, and the geometric layout may involve various forms such as long text, tilted text, and perspective text. These diverse styles bring great complexity and difficulty to text positioning. Secondly, the complexity of background reconstruction mainly stems from the texture structure and occlusion problems of the scene background. Scene text is usually located on a complex background. These backgrounds may contain delicate textures and rich color changes, and even have a certain occlusion relationship with the text itself. Even a subtle text erasure operation, if not handled properly, may cause visual disharmony and destroy the overall beauty of the image. Therefore, when performing background reconstruction, how to accurately reconstruct the background area covered by the text while avoiding excessive modification of non-essential areas is the key to improving the erasure effect and efficiency. In short, scene text erasure in natural scenes is a very challenging task in the field of computer vision.

[0003] The end-to-end scene text erasure model aims to achieve scene text erasure in a unified model. The current methods can be roughly divided into two categories: single-stage scene text erasure methods and two-stage scene text erasure methods. The single-stage method implicitly performs text localization and background reconstruction through a single network, and the process is simple. For example, one method adopts the network structure of image-to-image translation and maintains the visual consistency of the text erasure area through a local perception discriminator. In order to further improve the usability of the scene text erasure network, the related technology uses a coarse-to-fine strategy to refine the rough prediction results and reconstruct the text area at the image level. In order to reduce the cost of training data annotation, the related technology constructs a self-supervised framework with a controllable synthesis module to improve the performance of the scene text erasure network. In addition, some scholars have introduced the visual Transformer model into the scene text erasure task. The related technology proposes to replace the traditional convolutional structure with the visual Transformer structure for long-distance modeling, which improves the visual effect of text erasure. The two-stage scene text erasure method explicitly performs text localization and background reconstruction separately. For example, MTRNet directly uses text region mask labels to locate text and reconstructs text regions through a conditional adversarial generative network. Similarly, SSSTRM proposes to use gated attention and region of interest generation techniques to improve text erasure. Some methods do not rely on real text region masks, but use advanced text detectors to obtain text region masks. For example, CTRNet locates text regions and explores local-global cues to accurately restore the background. Unlike methods that require image-mask pairs for training, related technologies introduce a pre-trained PSENet for text detection and use another pre-trained image restoration model for background restoration, reducing the need for training data. Similarly, related technologies propose to train text detectors and background restoration modules separately and then merge them into one model. To further improve the erasure effect, some researchers integrate text mask prediction and background restoration into an end-to-end framework. For example, MTRNet++ uses two branches to predict text masks and repair images respectively, and refines the repair results through a fine repair branch. Similarly, SAEN generates a text mask and a roughly inpainted image in the first stage, and then uses a stroke-aware refinement network to generate a refined image. This refinement strategy has been further developed and applied in subsequent methods. For example, PERT adopts multiple progressive erasing stages and a region-based modification strategy to generate better results.

[0004] Generally speaking, these models have difficulty in background reconstruction and text localization when dealing with images with complex backgrounds and complex text forms, which often results in residual text and incoherent background after erasure. Summary of the invention

[0005] The main purpose of the embodiments of the present invention is to propose a scene text removal method and system based on hierarchical context-aware interactive reconstruction, which can provide more accurate text position visual information to guide the reconstruction of text areas and achieve better reconstruction effects.

[0006] To achieve the above object, an embodiment of the present invention provides a scene text hiding method based on hierarchical context-aware interactive reconstruction, comprising the following steps:

[0007] A lightweight shared encoding-separation decoding network is used to perform preliminary text localization and text region background reconstruction, and a rough text mask and a rough text erased image are predicted.

[0008] Refining the coarse text mask using a deep separable convolutional network, and then hierarchically refining the coarse text mask through four hierarchical mask decoders to obtain a hierarchical refined mask;

[0009] Downsampling the rough text-erased image through a U-Net feature extractor, and learning the contextual relationship between the complex background and the text area layer by layer according to the hierarchical fine mask to decode the scene text-erased image;

[0010] The original image is added to the scene text erased image to obtain a final predicted image.

[0011] In some embodiments, the method of performing preliminary text localization and text area background reconstruction through a lightweight shared encoding-separation decoding network to predict a rough text mask and a rough text erased image includes the following steps:

[0012] Input a scene text image, use the initial text erasure network to roughly locate and erase the text in the scene text image in parallel, and output a rough text erased image and a rough text mask;

[0013] The scene text image is input into the initial text erasure network, and the expression of the text image feature generated by the image encoder is: en =φ en (I in ; Θ en ), where F en Represents the text image feature; φ en I stands for image encoder; in Represents scene text image; Θ en Represents the image encoder φ en The learnable parameters in ;

[0014] The rough text erased image I coarse and the rough text mask M coarse The expression is:

[0015]

[0016] Among them, Θ ide and θ mde Denote the decoder φ ide and φ mde The learnable parameters in .

[0017] In some embodiments, the using of a deep separable convolutional network to refine the coarse text mask, and then hierarchically refining the coarse text mask through four hierarchical mask decoders to obtain a hierarchical refined mask, comprises the following steps:

[0018] Using the depthwise separable convolutional network φ dsc Extracting Rough Text Erased Image I coarse The hierarchical features The expression of this process is: Among them, l∈{1,2,3,4}, represents the features of the lth layer, Θ dsc Represents the network φ dsc The learnable parameters in ;

[0019] The rough text mask M coarse and Features Send to 4 levels of mask decoder respectively To obtain a hierarchical fine mask The expression of this process is: Among them, Θ dsc Representation Mask Decoder The learnable parameters in .

[0020] In some embodiments, downsampling the rough text-erased image by a U-Net feature extractor, learning the contextual relationship between the complex background and the text area layer by layer according to the hierarchical fine mask, and decoding the scene text-erased image includes the following steps:

[0021] Through U-Net feature extraction network φ fe Generate rough text erased image I cosrse The high-dimensional semantic features F c ;

[0022] Fine masking at the layer level Under the guidance of , the background of the text area is reconstructed layer by layer; in the lth layer, through the convolution layer φ with residual connection uc Front layer features Upsample and use the channel-level context-aware reconstruction layer φ br Reconstruct the features;

[0023] Based on the visual continuity between the reconstructed text area and the background area, a channel-level context-aware reconstruction layer φ is introduced. br To capture the contextual relationship between the text area and the background area to reconstruct the text area background.

[0024] In some embodiments, the high-dimensional semantic feature F c The expression is: F c =φ fe (I coarse ; Θ fe ), where Θ fe Represents the network φ fe The learnable parameters in ;

[0025] The channel-level context-aware reconstruction layer φ br The expression of the process of reconstructing features is: in, and Respectively represent the lth convolutional layer φ uc and the reconstruction layer φ br The learnable parameters in represents the front layer features, Indicates the feature to be refactored.

[0026] In some embodiments, the channel-level context-aware reconstruction layer φ is introduced based on the visual continuity between the reconstructed text area and the background area. br To capture the contextual relationship between the text area and the background area, so as to reconstruct the text area background, the following steps are included:

[0027] Using the lth layer fine mask Foreground features and background characteristics Separation, the expression of this process is:

[0028]

[0029] Among them, ⊙ represents the Hadamard product operation;

[0030] Based on prospect characteristics Get a set of channel-level foreground features Among them, the characteristics The yth element of can be calculated by the following formula:

[0031]

[0032] in, Representation characteristics The activation map of the yth channel of , E(·) represents the mathematical expectation of all elements in the matrix;

[0033] Get a set of channel-level background features

[0034] Learn the contextual relationship between text foreground and background and generate semantically aggregated features The expression of this process is: in, Indicates concatenation of two matrices. and Represent the learnable weight parameters and bias parameters respectively;

[0035] Through the foreground-background fusion module, the semantic representation of the foreground and background is obtained respectively.

[0036]

[0037] in, Represents learnable weight parameters; represents the bias parameter;

[0038] Using the channel-level weighting mechanism, the foreground features are and background characteristics Weighted, and the weighted features are added to obtain the reconstructed features

[0039]

[0040] in, It means broadcasting the feature vector into a tensor of shape h×w×c along the spatial dimensions.

[0041] In some embodiments, the adding of the original image and the scene text erased image to obtain a final predicted image comprises the following steps:

[0042] After 4 layers of progressive background reconstruction, a 1×1 convolutional layer φ im Decode the text erased image and add it to the original image to get the final predicted image. The expression of this process is:

[0043]

[0044] Among them, I out represents the final predicted image; represents the result after 4 layers of progressive background reconstruction; ⊙ represents the Hadamard product operation; Represents the features of the fourth convolutional layer; I in Represents scene text image; Θdsc Represents the convolutional layer φ im The learnable parameters in .

[0045] Another aspect of the embodiment of the present invention further provides a scene text hiding system based on hierarchical context-aware interactive reconstruction, comprising:

[0046] The first module is used to perform preliminary text localization and text region background reconstruction through a lightweight shared encoding-separation decoding network, and predict a rough text mask and a rough text erased image;

[0047] The second module is used to refine the rough text mask using a deep separable convolutional network, and then perform hierarchical refinement on the rough text mask through four hierarchical mask decoders to obtain a hierarchical refined mask;

[0048] The third module is used to downsample the rough text-erased image through a U-Net feature extractor, and learn the contextual relationship between the complex background and the text area layer by layer according to the hierarchical fine mask to decode the scene text-erased image;

[0049] The fourth module is used to add the original image and the scene text erased image to obtain a final predicted image.

[0050] To achieve the above objective, another aspect of an embodiment of the present invention provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0051] To achieve the above objective, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method is implemented.

[0052] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0053] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a scene text hiding method and system based on hierarchical context-aware interactive reconstruction, the scheme performs preliminary text positioning and text area background reconstruction through a lightweight shared encoding-separation decoding network, and predicts a rough text mask and a rough text erasure image; the rough text mask is refined using a deep separable convolutional network, and then the rough text mask is hierarchically refined through four hierarchical mask decoders to obtain a hierarchical fine mask; the rough text erasure image is downsampled by a U-Net feature extractor, and the contextual relationship between the complex background and the text area is learned layer by layer according to the hierarchical fine mask to decode the scene text erasure image; the original image is added to the scene text erasure image to obtain the final predicted image. The embodiments of the present invention can provide more accurate text position visual information to guide the reconstruction of the text area and achieve better reconstruction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0055] Figure 2 is a flow chart of the overall steps provided by an embodiment of the present invention;

[0056] Figure 3 It is a flowchart of specific implementation steps provided by an embodiment of the present invention;

[0057] Figure 4 is a diagram of a channel-level context-aware reconstruction mechanism provided by an embodiment of the present invention;

[0058] Figure 5 is a visualization diagram of the performance comparison between the method of the embodiment of the present invention and various text erasing methods provided by the embodiment of the present invention;

[0059] Figure 6 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0061] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0062] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0064] The scene text hiding method and system based on hierarchical context-aware interactive reconstruction provided by the embodiment of the present invention relate to the field of computer technology. The scene text hiding method based on hierarchical context-aware interactive reconstruction provided by the embodiment of the present invention can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The cloud server of basic cloud computing services, the server can also be a node server in a blockchain network; the software can be an application that implements a scene text hiding method based on hierarchical context-aware interactive reconstruction, etc., but is not limited to the above forms.

[0065] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0066] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0067] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0068] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0069] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0070] Based on the example Figure 1In the implementation environment shown, an embodiment of the present invention provides a scene text hiding method based on hierarchical context-aware interactive reconstruction. The following is an example of applying the scene text hiding method based on hierarchical context-aware interactive reconstruction in the server 101. It can be understood that the method can also be applied to the terminal 102.

[0071] Reference Figure 2 , Figure 2 The flowchart of the scene text hiding method based on hierarchical context-aware interactive reconstruction applied to the server provided by the embodiment of the present invention, the execution subject of the method can be any of the aforementioned computer devices (including servers or terminals). Figure 2 , the method may include the following steps:

[0072] A lightweight shared encoding-separation decoding network is used to perform preliminary text localization and text region background reconstruction, and a rough text mask and a rough text erased image are predicted.

[0073] Refining the coarse text mask using a deep separable convolutional network, and then hierarchically refining the coarse text mask through four hierarchical mask decoders to obtain a hierarchical refined mask;

[0074] Downsampling the rough text-erased image through a U-Net feature extractor, and learning the contextual relationship between the complex background and the text area layer by layer according to the hierarchical fine mask to decode the scene text-erased image;

[0075] The original image is added to the scene text erased image to obtain a final predicted image.

[0076] In some embodiments, the method of performing preliminary text localization and text area background reconstruction through a lightweight shared encoding-separation decoding network to predict a rough text mask and a rough text erased image includes the following steps:

[0077] Input a scene text image, use the initial text erasure network to roughly locate and erase the text in the scene text image in parallel, and output a rough text erased image and a rough text mask;

[0078] The scene text image is input into the initial text erasure network, and the expression of the text image feature generated by the image encoder is: en =φ en (I in ; Θ en ), where F en Represents the text image feature; φ en I stands for image encoder; in Represents scene text image; Θen Represents the image encoder φ en The learnable parameters in ;

[0079] The rough text erased image I coarse and the rough text mask M coarse The expression is:

[0080]

[0081] Among them, Θ ide and θ mde Denote the decoder φ ide and φ mde The learnable parameters in .

[0082] In some embodiments, the using of a deep separable convolutional network to refine the coarse text mask, and then hierarchically refining the coarse text mask through four hierarchical mask decoders to obtain a hierarchical refined mask, comprises the following steps:

[0083] Using the depthwise separable convolutional network φ dsc Extracting Rough Text Erased Image I coarse The hierarchical features The expression of this process is: Among them, l∈{1,2,3,4}, represents the features of the lth layer, Θ dsc Represents the network φ dsc The learnable parameters in ;

[0084] The rough text mask M coarse and Features Send to 4 levels of mask decoder respectively To obtain a hierarchical fine mask The expression of this process is: Among them, Θ dsc Representation Mask Decoder The learnable parameters in .

[0085] In some embodiments, downsampling the rough text-erased image by a U-Net feature extractor, learning the contextual relationship between the complex background and the text area layer by layer according to the hierarchical fine mask, and decoding the scene text-erased image includes the following steps:

[0086] Through U-Net feature extraction network φ fe Generate rough text erased image I coarse The high-dimensional semantic features F c ;

[0087] Fine masking at the layer level Under the guidance of , the background of the text area is reconstructed layer by layer; in the lth layer, through the convolution layer φ with residual connection uc Front layer features Upsample and use the channel-level context-aware reconstruction layer φ br Reconstruct the features;

[0088] Based on the visual continuity between the reconstructed text area and the background area, a channel-level context-aware reconstruction layer φ is introduced. br To capture the contextual relationship between the text area and the background area to reconstruct the text area background.

[0089] In some embodiments, the high-dimensional semantic feature F c The expression is: F c =φ fe (I coarse ; Θ fe ), where Θ fe Represents the network φ fe The learnable parameters in ;

[0090] The channel-level context-aware reconstruction layer φ br The expression of the process of reconstructing features is: in, and Respectively represent the lth convolutional layer φ uc and the reconstruction layer φ br The learnable parameters in represents the front layer features, Indicates the feature to be refactored.

[0091] In some embodiments, the channel-level context-aware reconstruction layer φ is introduced based on the visual continuity between the reconstructed text area and the background area. br To capture the contextual relationship between the text area and the background area, so as to reconstruct the text area background, the following steps are included:

[0092] Using the lth layer fine mask Foreground features and background characteristics Separation, the expression of this process is:

[0093]

[0094] Among them, ⊙ represents the Hadamard product operation;

[0095] Based on prospect characteristics Get a set of channel-level foreground features Among them, the characteristics The yth element of can be calculated by the following formula:

[0096]

[0097] in, Representation characteristics The activation map of the yth channel of , E(·) represents the mathematical expectation of all elements in the matrix;

[0098] Get a set of channel-level background features

[0099] Learn the contextual relationship between text foreground and background and generate semantically aggregated features The expression of this process is: in, Indicates concatenation of two matrices. and Represent the learnable weight parameters and bias parameters respectively;

[0100] Through the foreground-background fusion module, the semantic representation of the foreground and background is obtained respectively.

[0101]

[0102] in, Represents learnable weight parameters; represents the bias parameter;

[0103] Using the channel-level weighting mechanism, the foreground features are and background characteristics Weighted, and the weighted features are added to obtain the reconstructed features

[0104]

[0105] in, It means broadcasting the feature vector into a tensor of shape h×w×c along the spatial dimensions.

[0106] In some embodiments, the adding of the original image and the scene text erased image to obtain a final predicted image comprises the following steps:

[0107] After 4 layers of progressive background reconstruction, a 1×1 convolutional layer φ im Decode the text erased image and add it to the original image to get the final predicted image. The expression of this process is:

[0108]

[0109] Among them, I outrepresents the final predicted image; represents the result after 4 layers of progressive background reconstruction; ⊙ represents the Hadamard product operation; Represents the features of the fourth convolutional layer; I in Represents scene text image; Θ dsc Represents the convolutional layer φ im The learnable parameters in .

[0110] The following describes the specific implementation process of the method of the present invention in detail by taking a specific application scenario as an example:

[0111] refer to Figure 3 The scene text hiding method of the present invention may include the following steps:

[0112] 1) Parallel rough text localization and erasure. This embodiment performs preliminary text localization and text region background reconstruction through a lightweight shared encoding-separation decoding network, thereby predicting a rough text mask and text erasure image.

[0113] 2) Hierarchical fine mask generation. After obtaining the coarse text mask and the text-erased image, we further refine the coarse text mask. First, a depthwise separable convolution network is used to extract the geometric features of the coarse text-erased image. Then, based on the extracted geometric features, the coarse text mask is hierarchically refined through four hierarchical mask decoders to generate four hierarchical fine masks. Among them, the hierarchical fine mask will be used to guide the background reconstruction and the splicing of the erased image with the original image.

[0114] 3) Context-aware progressive background reconstruction. First, the coarse text-erased image is downsampled through a U-Net feature extractor to extract high-dimensional semantic features. Then, the context-aware interaction mechanism at the channel level is used to learn the contextual relationship between the complex background and the text region layer by layer under the guidance of the hierarchical fine mask, and the scene text-erased image is decoded. Finally, based on the image segmentation of the first fine mask, the original image and the scene text-erased image are added to obtain the final predicted image.

[0115] In the embodiment of the present invention, the input is an RGB scene text image I in , the output is the text erased image I out All input images are scaled to 256×256 size.

[0116] The specific steps are as follows:

[0117] 1) Use the initial text erasure network to roughly locate and erase the text in the scene text image in parallel. Its input is the scene text image I in, the output is a rough text erased image I coarse and a rough text mask M coarse Specifically, the scene text image I in ∈R H×W×3 is input into the initial text erasure network and passes through the image encoder φ en Generate text image features F en :

[0118] F en =φ en (I in ; Θ en ),

[0119] Among them, Θ en Indicates encoder φ en The learnable parameters in . Then, the parallel image decoder φ ide and the mask decoder φ mde Predict the rough text erased image I respectively coarse and a rough text mask M coarse :

[0120]

[0121] Among them, Θ ide and θ mde Denote decoder Θ ide and θ mde The learnable parameters in .

[0122] 2) Use the hierarchical mask generation network to further refine the rough text mask, whose input is the rough text erased image I coarse and a rough text mask M coarse , the output is 4 levels of fine mask This embodiment uses a deep separable convolutional network φ dsc Extracting Rough Text Erased Image I coarse The hierarchical features

[0123]

[0124] Among them, l∈{1,2,3,4}, represents the features of the lth layer, Θ dsc Represents the network φ dsc Then the rough text mask M coarse and Features Send to 4 levels of mask decoder respectively To obtain a hierarchical fine mask

[0125]

[0126] Among them, Θ dsc Representation Mask Decoder The learnable parameters in .

[0127] 3) Progressive background reconstruction guided by hierarchical fine mask. First, the U-Net feature extraction network φ fe Generate rough text erased image I coarse The high-dimensional semantic features F c :

[0128] F c =φ fe (I coarse ; Θ fe ),

[0129] Among them, Θ fe Represents the network φ fe Then, we refine the mask at the level Under the guidance of , this embodiment reconstructs the background of the text area layer by layer. In the first layer, through the convolution layer φ with residual connection uc Front layer features Upsample and use the channel-level context-aware reconstruction layer φ br Refactor the features:

[0130]

[0131] in, and Respectively represent the lth convolutional layer φ uc and the reconstruction layer φ br The learnable parameters in .

[0132] 4) In order to achieve visual continuity between the reconstructed text area and the background area, refer to Figure 4 , this embodiment introduces a channel-level context-aware reconstruction layer φ br To capture the contextual relationship between the text area and the background area, so as to help better reconstruct the text area background. br , whose input is the previous convolutional layer φ uc Output characteristics The output is the reconstructed features First, this embodiment uses the first layer of fine mask The foreground (text) feature and background characteristics Separation:

[0133]

[0134] Wherein, ⊙ represents the Hadamard product operation. Next, this embodiment uses the foreground feature Further obtain a set of channel-level foreground features Among them, the characteristics The yth element of can be calculated by the following formula:

[0135]

[0136] in, Representation characteristics The activation map of the yth channel of , E(·) represents the mathematical expectation of all elements in the matrix. In the same way, this embodiment can obtain a set of channel-level background features Then, this embodiment learns the contextual relationship between the text foreground and background and generates semantic aggregation features

[0137]

[0138] in, Indicates concatenation of two matrices. and Respectively represent the learnable weight parameters and bias parameters. Next, this embodiment uses the foreground-background fusion module to obtain the semantic representation of the foreground and background respectively.

[0139]

[0140] in, and denote the learnable weight parameters and bias parameters respectively. Finally, this embodiment uses a channel-level weighting mechanism to separately and background characteristics Weighted, and the weighted features are added to obtain the reconstructed features

[0141]

[0142] in, It means broadcasting the feature vector into a tensor of shape (h×w×c) along the spatial dimensions.

[0143] 5) After 4 layers of progressive background reconstruction, this embodiment uses a 1×1 convolution layer φ im Decode the text-erased image and add it to the original image to get the final predicted image:

[0144]

[0145] Among them, Θdsc Represents the convolutional layer φ im The learnable parameters in .

[0146] The method proposed in this embodiment predicts both the text mask and the text-erased image, which need to be supervised by corresponding information. Since the text mask contains 4 layers of masks, this embodiment supervises each layer of masks. Specifically, for the lth layer of text mask This example uses binary cross entropy loss Monitor its distribution:

[0147]

[0148] Among them, M gt represents the text mask label, Indicates M gt Perform adaptive pooling operation and get Masks of the same shape. Since the binary cross entropy loss is a pixel-level supervision, it tends to produce incorrect predictions when the text layout is complex. In order to constrain the mask from a global perspective, this embodiment uses a soft mask IoU loss

[0149]

[0150] in, Indicates the sum of all pixel values ​​in the mask. At the same time, this embodiment also uses a binarization loss Force the mask values ​​to be close to 0 or 1 to improve the visual continuity of the text-erased image added to the original image:

[0151]

[0152] Among them, min(a,b) means taking the minimum value between mask a and mask b at each pixel position. The final loss function for the mask is It is expressed as follows:

[0153]

[0154] Among them, α 1 ,α 2 ,α 3 are the weight coefficients of the three loss functions respectively. At the same time, for the reconstructed image I out The main purpose of this embodiment is to reconstruct the foreground (text) area in the image, so this embodiment needs to ensure that the reconstructed area is consistent with the label image I gt The corresponding areas in are similar, thus introducing foreground loss:

[0155]

[0156] Wherein, N=3HW represents the number of pixels in the three channels of the image. At the same time, in order to achieve better visual continuity, this embodiment introduces visual perception loss:

[0157]

[0158] Among them, V j (·) represents the feature map of the jth layer after the image is input into the VGG-16 network, N j Represents the number of pixels in the feature map, and M represents the number of feature maps used in this embodiment. However, visual perception loss cannot capture complex text structures and backgrounds well. This embodiment uses shape similarity constraints on the reconstructed image:

[0159]

[0160] Among them, ε 1 and ε 2 is the factor that stabilizes the weak denominator division, U p and U g Represent the predicted image I out and label image I gt The mean value, Q p and Q g Represent the predicted image I p =I out and label image I g =I gt The standard deviation, Q pg Represents the predicted image I out and label image I gt The covariances of are as follows:

[0161] U p =e(I p ),U g =E(I g ),

[0162]

[0163] Q pg =E(I p -U p )(I g -U g ),

[0164] Finally, it is used to supervise the reconstructed image I out The loss function is expressed as follows:

[0165]

[0166] Among them, β 1, β 2 , β 3 are the weight coefficients of the three loss functions respectively. Similarly, for the rough text erased image I coarse , this embodiment applies the same loss function for supervision:

[0167]

[0168] in, Represents image I coarse The loss function used in the entire framework is as follows:

[0169]

[0170] The test environment and experimental results of the scene text elimination method based on hierarchical context-aware interactive reconstruction proposed in this invention are as follows:

[0171] (1) Test environment:

[0172] System environment: Ubuntu 20.04.

[0173] Hardware environment: Memory: 64GB, GPU: GeForce RTX 3090, CPU: Intel(R) Xeon(R) Gold6226R, Hard disk: 2TB.

[0174] (2) Experimental data:

[0175] The present invention is trained and tested on the real dataset SCUT-EnsText and the synthetic dataset SCUT-Syn respectively.

[0176] (3) Optimization method:

[0177] In this embodiment, the factor is set to: α 1 =2, α 2 =0.25, α 3 =10, β 1 =0.25, β 2 =0.5, β 3 = 0.25 and M = 3. This embodiment uses the Adam optimizer, the learning rate is 0.001, and the batch sizes are 12 and 60 on the two data sets, respectively. The model of this embodiment is implemented using the PyTorch v2.1.2 deep learning framework.

[0178] (4) Experimental results:

[0179] 1) Ablation experiment:

[0180] In order to verify the effectiveness of the method proposed in this embodiment, this embodiment sets the initial text erasure network as the baseline model. As shown in Table 1, this embodiment can observe that the peak signal-to-noise ratio (PSNR) of the baseline model is 34.57. Without mask guidance and channel-level context-aware interaction mechanism (CCIM), after applying progressive background reconstruction (PBR), the PSNR is improved by 1.80 compared with the baseline model. This improvement can be attributed to the addition of progressive background reconstruction to transform the network from a single-stage paradigm to a two-stage paradigm, so that the results can be refined. When the channel-level context-aware interaction mechanism is used in progressive background reconstruction, it uses a coarse mask to separate the text area and the background, and the PSNR is improved by 0.22, while the parameters are only increased by 0.24M. The reason is that the channel-level context-aware interaction mechanism can use a coarse mask to guide background reconstruction. When a hierarchical mask generation network (HMGN) is used to provide a fine mask, the PSNR is improved by 0.23, while the parameters are only increased by more than 0.66M. This is because the fine mask provides more accurate guidance for background reconstruction.

[0181] Table 1: Validation of the proposed modules

[0182]

[0183] 2) Performance comparison:

[0184] refer to Figure 5, since pre-training and data augmentation can cause significant fluctuations in performance, all experiments reported in this embodiment do not use pre-training or data augmentation strategies to ensure the fairness of the comparison as much as possible. As shown in Table 2, when this embodiment is experimented on the real dataset SCUT-EnsText, the method proposed in this embodiment achieves the best 36.82PSNR and 97.81% MSSIM with fewer parameters and faster inference speed compared with other state-of-the-art methods. More specifically, the model of this embodiment is 7.28 and 5.07% higher than the single-stage method EnsNet in PSNR and MSSIM, respectively. Compared with the two-stage method CTRNet, the method of this embodiment also shows better performance and faster speed. For example, the method of this embodiment improves PSNR and speed by 1.62 and 40.9fps, respectively. In addition, the PSNR of the model of this embodiment is 0.27 higher than that of the Transformer-based method ViTEraser, and the speed is much faster (46fps vs.17fps). Table 3 shows the quantitative comparison results of the method proposed in this embodiment with existing methods on the synthetic dataset SCUT-Syn. It can be observed that the model of this embodiment achieves state-of-the-art performance in all indicators. The method of this embodiment comprehensively surpasses the Transformer-based method ViTEraser on this dataset, improving PSNR and MSSIM by 0.44 and 0.88% respectively. In addition, compared with EraseNet, which only uses image refinement without mask refinement, the model of this embodiment achieves a significant improvement of 4.57 and 1.64% in PSNR and MSSIM respectively.

[0185] Table 2: Performance comparison with various text erasure methods on the SCUT-EnsText dataset

[0186] Method PSNR MSSIM MSE AGE pE pCEPs #Params Speed(fps) EnsNet 29.54 92.74 0.24 4.16 0.0307 0.0136 12.40M 199 MTRNet++ 29.63 93.71 0.28 3.51 0.0305 0.0168 18.67M 53 EraseNet 32.30 95.42 0.15 3.02 0.0160 0.0090 17.82M 71 CTRNet 35.20 97.36 0.09 2.20 0.0106 0.0068 159.81M 5.1 PERT 33.25 96.95 0.14 2.18 0.0136 0.0088 14.00M 59 ViTEraser-T 36.32 97.48 0.0569 1.81 0.0073 0.0040 65.39M 24 ViTEraser-S 36.55 97.56 0.0497 1.73 0.0072 0.0039 108.15M 17 ViTEraser-L 36.32 97.51 0.0565 1.86 0.0074 0.0041 191.97M 15 HiCIR 36.82 97.81 0.0988 1.36 0.0112 0.0069 32.44M 46

[0187] Table 3: Performance comparison with various text erasure methods on the SCUT-Syn dataset

[0188]

[0189]

[0190] In summary, the present invention separates the text content, text style and background, and uses the style features of the source text to guide the style transfer of the content text; then replaces the target text with the source text to obtain the minimized reconstruction area, and reconstructs this area to complete the scene text editing. It can provide more accurate text position visual information to guide the reconstruction of the text area and achieve better reconstruction effect.

[0191] Another aspect of the embodiment of the present invention further provides a scene text hiding system based on hierarchical context-aware interactive reconstruction, comprising:

[0192] The first module is used to perform preliminary text localization and text region background reconstruction through a lightweight shared encoding-separation decoding network, and predict a rough text mask and a rough text erased image;

[0193] The second module is used to refine the rough text mask using a deep separable convolutional network, and then perform hierarchical refinement on the rough text mask through four hierarchical mask decoders to obtain a hierarchical refined mask;

[0194] The third module is used to downsample the rough text-erased image through a U-Net feature extractor, and learn the contextual relationship between the complex background and the text area layer by layer according to the hierarchical fine mask to decode the scene text-erased image;

[0195] The fourth module is used to add the original image and the scene text erased image to obtain a final predicted image.

[0196] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0197] The embodiment of the present invention further provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned scene text hiding method based on hierarchical context-aware interactive reconstruction when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0198] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0199] See also Figure 6 , Figure 6 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0200] The processor 601 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;

[0201] The memory 602 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 602, and the processor 601 calls and executes the scene text hiding method based on hierarchical context-aware interactive reconstruction of the embodiment of the present invention;

[0202] Input / output interface 603, used to implement information input and output;

[0203] Communication interface 604, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0204] Bus 605 , which transmits information between various components of the device (e.g., processor 601 , memory 602 , input / output interface 603 , and communication interface 604 );

[0205] The processor 601 , the memory 602 , the input / output interface 603 and the communication interface 604 are connected to each other in communication within the device via a bus 605 .

[0206] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned scene text hiding method based on hierarchical context-aware interactive reconstruction.

[0207] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0208] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0209] It should be noted that in various specific embodiments of the present invention, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present invention needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or consent through a pop-up window or jump to a confirmation page, and after clearly obtaining the user's separate permission or consent, it will obtain the necessary user-related data for the normal operation of the embodiment of the present invention.

[0210] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0211] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0212] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0213] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0214] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0215] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0216] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0217] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0218] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0219] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0220] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A scene text removal method based on hierarchical context-aware interactive reconstruction, characterized in that: The following steps are involved: A lightweight shared encoding-separation decoding network is used to perform preliminary text localization and text region background reconstruction, and a rough text mask and a rough text erased image are predicted. Using a deep separable convolutional network to refine the coarse text mask, and then hierarchically refine the coarse text mask through four hierarchical mask decoders to obtain a hierarchical refined mask; Downsampling the rough text-erased image through a U-Net feature extractor, and learning the contextual relationship between the complex background and the text area layer by layer according to the hierarchical fine mask to decode the scene text-erased image; The original image is added to the scene text erased image to obtain a final predicted image.

2. The scene text elimination method based on hierarchical context-aware interactive reconstruction according to claim 1 is characterized in that: The method uses a lightweight shared encoding-separation decoding network to perform preliminary text positioning and text area background reconstruction, and predicts a rough text mask and a rough text erased image, including the following steps: Input a scene text image, use the initial text erasure network to roughly locate and erase the text in the scene text image in parallel, and output a rough text erased image and a rough text mask; The scene text image is input into the initial text erasure network, and the expression of the text image feature generated by the image encoder is: en =φ en (I in ; Θ en ), where F en Represents the text image feature; φ en I stands for image encoder; in Represents scene text image; Θ en Represents the image encoder φ en The learnable parameters in ; The rough text erased image I coarse and the rough text mask M coarse The expression is: Among them, Θ ide and θ mde Denote the decoder φ ide and φ mde The learnable parameters in .

3. The scene text elimination method based on hierarchical context-aware interactive reconstruction according to claim 1 is characterized in that: The method of using a deep separable convolutional network to refine the rough text mask, and then performing hierarchical refinement processing on the rough text mask through four hierarchical mask decoders to obtain a hierarchical refined mask, includes the following steps: Using the depthwise separable convolutional network φ dsc Extracting Rough Text Erased Image I coarse The hierarchical features The expression of this process is: Among them, l∈{1, 2, 3, 4}, represents the features of the lth layer, Θ dsc Represents the network φ dsc The learnable parameters in ; The rough text mask M coarse and Features Send to 4 levels of mask decoder respectively To obtain a hierarchical fine mask The expression of this process is: Among them, Θ dsc Representation Mask Decoder The learnable parameters in .

4. The scene text elimination method based on hierarchical context-aware interactive reconstruction according to claim 1 is characterized in that: The method of downsampling the rough text-erased image by a U-Net feature extractor and learning the contextual relationship between the complex background and the text area layer by layer according to the hierarchical fine mask to decode the scene text-erased image includes the following steps: Through U-Net feature extraction network φ fe Generate rough text erased image I coarse The high-dimensional semantic features F c ; Fine masking at the layer level Under the guidance of , the background of the text area is reconstructed layer by layer; in the lth layer, through the convolution layer φ with residual connection uc Front layer features Upsample and use the channel-level context-aware reconstruction layer φ br Reconstruct the features; Based on the visual continuity between the reconstructed text area and the background area, a channel-level context-aware reconstruction layer φ is introduced. br To capture the contextual relationship between the text area and the background area to reconstruct the text area background.

5. The scene text elimination method based on hierarchical context-aware interactive reconstruction according to claim 4 is characterized in that: The high-dimensional semantic feature F c The expression is: F c =φ fe (I coarse ; Θ fe ), where Θ fe Represents the network φ fe The learnable parameters in ; The channel-level context-aware reconstruction layer φ br The expression of the process of reconstructing features is: in, and Respectively represent the lth convolutional layer φ uc and the reconstruction layer φ br The learnable parameters in represents the front layer features, Indicates the feature to be refactored.

6. The scene text elimination method based on hierarchical context-aware interactive reconstruction according to claim 4 is characterized in that: Based on the visual continuity between the reconstructed text area and the background area, a channel-level context-aware reconstruction layer φ is introduced. br To capture the contextual relationship between the text area and the background area, so as to reconstruct the text area background, the following steps are included: Using the lth layer fine mask Foreground features and background characteristics Separation, the expression of this process is: Among them, ⊙ represents the Hadamard product operation; Based on prospect characteristics Get a set of channel-level foreground features Among them, the characteristics The yth element of can be calculated by the following formula: in, Representation characteristics The activation map of the yth channel of , E(·) represents the mathematical expectation of all elements in the matrix; Get a set of channel-level background features Learn the contextual relationship between text foreground and background and generate semantically aggregated features The expression of this process is: Among them, C(·,·) represents the concatenation of two matrices. and Represent the learnable weight parameters and bias parameters respectively; Through the foreground-background fusion module, the semantic representation of the foreground and background is obtained respectively. in, Represents learnable weight parameters; represents the bias parameter; Using the channel-level weighting mechanism, the foreground features are and background characteristics Weighted, and the weighted features are added to obtain the reconstructed features in, It means broadcasting the feature vector into a tensor of shape h×w×c along the spatial dimensions.

7. The scene text elimination method based on hierarchical context-aware interactive reconstruction according to claim 1 is characterized in that: The process of adding the original image to the scene text erased image to obtain a final predicted image comprises the following steps: After 4 layers of progressive background reconstruction, a 1×1 convolutional layer φ im Decode the text erased image and add it to the original image to get the final predicted image. The expression of this process is: Among them, I out represents the final predicted image; represents the result after 4 layers of progressive background reconstruction; ⊙ represents the Hadamard product operation: Represents the features of the fourth convolutional layer; I in Represents scene text image; Θ dsc Represents the convolutional layer φ im The learnable parameters in .

8. A scene text removal system based on hierarchical context-aware interactive reconstruction, characterized in that: include: The first module is used to perform preliminary text localization and text area background reconstruction through a lightweight shared encoding-separation decoding network, and predict a rough text mask and a rough text erased image; The second module is used to refine the rough text mask using a deep separable convolutional network, and then perform hierarchical refinement on the rough text mask through four hierarchical mask decoders to obtain a hierarchical refined mask; The third module is used to downsample the rough text-erased image through a U-Net feature extractor, and learn the contextual relationship between the complex background and the text area layer by layer according to the hierarchical fine mask to decode the scene text-erased image; The fourth module is used to add the original image and the scene text erased image to obtain a final predicted image.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images