A scene text editing method and system, an electronic device, and a storage medium

Abstract

The application discloses a scene text editing method and system, electronic equipment and a storage medium. The application inputs a scene text image into a text background separation module, predicts a text foreground mask of the scene text image through a lightweight segmentation network, and separates a source text and a background image by using the text foreground mask. A style-guided text migration decoding module is used to align the geometry and style attributes of the content text with the source text, and generate a target text. The generated target text is combined with the background image to determine a minimized background reconstruction area. A context-focused background reconstruction module is used to perform background reconstruction on the minimized background reconstruction area to generate an edited complete scene text image. The embodiment of the application can improve the effect of scene text editing, reduce the influence of the background on text editing, and reduce the complexity of the text editing process, and can be widely applied to the technical field of computer vision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a scene text editing method and system, an electronic device and a storage medium. BACKGROUND

[0002] With the rapid development of computer vision technology, scene text editing has become an important research direction in this field. The core task of scene text editing is to replace the source text in the image while trying to maintain the integrity of the scene background and text style. Such technology has wide application prospects in augmented reality, data generation, privacy protection, etc. For example, in augmented reality applications, real-time translation and text replacement functions are of great significance for cross-language communication; in data generation, synthesized scene text data can be used to train and optimize deep learning models; in privacy protection, automated text editing technology can effectively remove sensitive information from images. However, scene text editing is a highly challenging task, with the main difficulties concentrated in two aspects: the diversity of text style and the complexity of background reconstruction. First, the diversity of text style is reflected in style, size, and color of the text. The style of the text may include different fonts, sizes, and colors, while the geometric layout may involve long text, skewed text, perspective text, and other forms. These diverse styles bring great complexity and difficulty to text migration. Second, the complexity of background reconstruction mainly comes from the texture structure and occlusion of the scene background. Scene text is usually located on a complex background, which may contain detailed textures and rich color changes, and even has some occlusion relationship with the text itself. Even a slight text replacement operation, if not handled properly, can cause visual discordance and damage the overall aesthetics of the image. Therefore, in the background reconstruction, how to accurately reconstruct the background area covered by the text while avoiding excessive modification of unnecessary areas is the key to improving the editing effect and efficiency. In summary, scene text editing in natural scenes is a highly challenging task in the field of computer vision. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide a scene text editing method, system, electronic device and storage medium, which can improve the effect of scene text editing, reduce the influence of background on text editing, and reduce the complexity of the text editing process.

[0004] To achieve the above purpose, one aspect of an embodiment of the present application provides a scene text editing method, comprising the following steps:

[0005] inputting a scene text image into a text-background separation module, predicting a text foreground mask of the scene text image through a lightweight segmentation network, and separating the source text and the background image using the text foreground mask;

[0006] aligning geometric and stylistic properties of the content text with the source text and generating a target text by a style-guided text migration decoding module;

[0007] combining the generated target text with the background image to determine a minimized background reconstruction region;

[0008] reconstructing the background of the minimized background reconstruction region by a context-focused background reconstruction module to generate an edited complete scene text image.

[0009] In some embodiments, the aligning geometric and stylistic properties of the content text with the source text and generating a target text by a style-guided text migration decoding module comprises the following steps:

[0010] extracting geometric features of the source text by a shallow convolutional network and predicting spatial layout anchor points of the source text by a fully connected layer, the spatial layout anchor points being used to represent the geometric structure of the source text;

[0011] based on the spatial layout anchor points, using a thin-plate spline interpolation algorithm to accurately adjust the geometric properties of the content text to match the geometric structure of the source text;

[0012] respectively encoding the source text and the content text by a content encoding network and a style encoding network to obtain text content features and text style features;

[0013] injecting style features into content features by an adaptive instance normalization layer to achieve dynamic adjustment of style representation to content representation, thereby generating a target text consistent with the style of the source text.

[0014] In some embodiments, the combining the generated target text with the background image to determine a minimized background reconstruction region comprises the following steps:

[0015] after obtaining the target text, replacing the target text with the source text in the scene text image to determine the smallest background repair region.

[0016] In some embodiments, the reconstructing the background of the minimized background reconstruction region by a context-focused background reconstruction module to generate an edited complete scene text image comprises the following steps:

[0017] after text replacement, encoding the image using a reconstruction encoder and inputting the encoding result into a residual module to enhance the expression of background details and structures;

[0018] The self-attention module is used to strengthen the network's ability to capture context information, ensuring the accuracy and consistency of background reconstruction.

[0019] The decoder generates a complete scene text editing image based on the enhanced features.

[0020] In some embodiments, the method further comprises a step of training the multi-task model, which comprises:

[0021] The text background segmentation module is used to decouple the source text and background of the scene text image, and the input of the text background segmentation module is the scene text image, and the output is the source text and the background image.

[0022] The text migration module is used to transfer the geometric properties and style properties of the source text to the content text to obtain the target text.

[0023] The target text is embedded in the non-text scene background, and the incomplete image is encoded using the reconstruction encoder, and then the encoding is input into the residual module to effectively enhance the expression of background details and structures.

[0024] According to the overall loss function of the entire multi-task model, the training and learning of each module in the multi-task model are completed.

[0025] In some embodiments, the text background segmentation module is used to decouple the source text and background of the scene text image, and the input of the text background segmentation module is the scene text image, and the output is the source text and the background image, which comprises the following steps:

[0026] The scene text image I s is input into the lightweight segmentation network G s to predict the mask M s of the source text, and then the text background separation is performed through the mask, and the expression of the text background separation is:

[0027]

[0028] Where I sf represents the source text; I sb represents the background image.

[0029] In some embodiments, the text migration module is used to transfer the geometric properties and style properties of the source text to the content text to obtain the target text, which comprises the following steps:

[0030] The shallow CNN network G g is used to extract the features of the source text I sf , and the N position anchors A c that can represent the spatial layout of the source text are predicted through the fully connected layer Fn The expression for this process is: A n =F c (G g (I sf ;Θ g );Θ fc ); where Θ g and Θ fc They represent network G respectively g and network layer F c Learnable parameters;

[0031] After obtaining the target location anchor point, the TPS algorithm is used to process the content text I. t To perform geometric style alignment, the expression for this process is: in This refers to the geometrically aligned text content.

[0032] Through stylistic network G s For source text I sf The stylistic features of the source text are encoded, and the encoded features are then passed through a multilayer perceptron to construct a stylistic representation F of the source text. s The expression for this process is: F s =MLP(G s (I sf ;Θ s );Θ mlp ); where Θ s and Θ mlp They represent network G respectively s and learnable parameters in the MLP layer;

[0033] Content text After content network G c After processing, the content representation F is obtained. t : Where Θ c Content network G c Learnable parameters;

[0034] Introducing an AdaIN-based style decoder and To generate scene text fonts of arbitrary styles, the two decoders share the same network structure, including two residual modules and two upsampling modules, and all normalization layers in the decoder use AdaIN;

[0035] Calculate the average value m of the style representation s and variance and the average value m of content representation t and variance The expression for this process is:

[0036]

[0037] where H represents the height of the input style representation or content representation; W represents the width of the input style representation or content representation; and respectively represent the element at position (s, t) in the style representation and the content representation, and ∈ is a small constant for numerical stability;

[0038] AdaIN is used to perform affine transformation on the font style of the content text to realize font transfer, and the expression of the process is:

[0039]

[0040] wherein, represents the content representation of the kth AdaIN layer;

[0041] the text style representation F s can be gradually injected into the content representation F t in the decoding process through the K AdaIN layers in the decoder, and the expression of the process is:

[0042]

[0043] wherein, O t represents the target text; represents the target text decoder; M t represents the mask corresponding to the target text; represents the target text mask decoder; and respectively represent the learnable parameters of the target text decoder and the mask decoder.

[0044] In some embodiments, the total loss function includes a text background separation loss, a style transfer loss, a mask loss, and a background reconstruction loss.

[0045] wherein, the expression of the text background separation loss is:

[0046]

[0047] wherein, T s is the true value corresponding to M s , and ∩ represents the intersection operation on pixels;

[0048] the expression of the style transfer loss is:

[0049]

[0050] wherein β1 represents a balance factor; T o is a true value corresponding to O t ; φ i represents a feature map from VGG-19;

[0051] Mask loss is expressed as:

[0052]

[0053] wherein T t is a true value corresponding to M t ; D t is a discriminator; I t represents a content text; γ1 and γ2 are balance factors;

[0054] Background reconstruction loss is expressed as:

[0055]

[0056] wherein ξ1 is a balance factor; T f is a true value corresponding to O f .

[0057] Another aspect of the embodiment of the present application further provides a scene text editing system, comprising:

[0058] A first module is configured to input a scene text image into a text background separation module, predict a text foreground mask of the scene text image through a lightweight segmentation network, and separate a source text and a background image from the text foreground mask;

[0059] A second module is configured to align a geometry and a style attribute of a content text with the source text through a style-guided text transfer decoding module, and generate a target text;

[0060] A third module is configured to combine the generated target text with the background image, and determine a minimized background reconstruction region;

[0061] A fourth module is configured to perform background reconstruction on the minimized background reconstruction region through a context-focused background reconstruction module, and generate an edited complete scene text image.

[0062] Another aspect of the embodiment of the present application further provides an electronic device, comprising a processor and a memory;

[0063] The memory is configured to store a program;

[0064] The processor is configured to execute the program to realize the method as described above.

[0065] To achieve the above object, another aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method described above.

[0066] The embodiment of the present application also discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the method described above.

[0067] The embodiment of the present application at least has the following beneficial effects: the present application provides a scene text editing method, system, electronic device and storage medium, the scheme inputs a scene text image into a text background separation module, predicts a text foreground mask of the scene text image through a lightweight segmentation network, separates a source text and a background image by using the text foreground mask, aligns the geometric and style properties of the content text with the source text through a style-guided text transfer decoding module, and generates a target text, combines the generated target text with the background image to determine a minimized background reconstruction area, and performs background reconstruction on the minimized background reconstruction area through a context-focused background reconstruction module to generate an edited complete scene text image. The embodiment of the present application can improve the effect of scene text editing, reduce the influence of the background on text editing, and reduce the complexity of the text editing process. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 is an implementation environment schematic diagram provided by the embodiment of the present application;

[0069] Figure 2 is a flowchart of the overall steps provided by the embodiment of the present application;

[0070] Figure 3 is an overall framework schematic diagram provided by the embodiment of the present application;

[0071] Figure 4 is a hardware structure schematic diagram of an electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0072] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are merely intended to explain the present application and are not intended to limit the present application. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with embodiments of the present application. They are merely examples of apparatuses and methods consistent with some aspects of embodiments of the present application as detailed in the appended claims.

[0073] It can be understood that the terms "first", "second" and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information, without departing from the scope of embodiments of the present application. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0074] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing embodiments of the present application and are not intended to limit the present application.

[0076] Before the embodiments of the present application are described in detail, first, some related technologies involved in the embodiments of the present application are described as follows:

[0077] End-to-end scene text editing models aim to edit scene text in a unified model. Current methods can be roughly divided into two categories: scene text editing methods without mask guidance and scene text editing methods with mask guidance. The former directly edits the entire scene image without text mask guidance, making the process simple. For example, the STEFANN method directly edits individual characters in the scene image, considering the color and font style of the characters. However, this method cannot adaptively adjust the size of the characters or flexibly handle variable-length words or text lines. To eliminate the limitations of single-character editing methods in applications, SwapText achieves word-level editing through multi-character supervision and introduces the thin-plate spline (TPS) algorithm to make scene text adaptable to various geometric layouts. To further improve the usability of scene text editing networks, TextStyleBrush decouples the content and style representations of text in a self-supervised manner, enabling the one-time migration of source styles to new content. To improve the performance of scene text editing networks in complex scenes, SimAN proposes a similarity-aware normalization module to enhance the network's ability to distinguish complex patterns, such as cluttered strokes and cluttered backgrounds. To reduce the high cost of training data synthesis, RewriteNet proposes a self-supervised training scheme that combines synthetic data and real data, improving the performance of scene text editing networks. In addition, some researchers have introduced diffusion models into the scene text editing task. DBEST proposes a diffusion-based scene text operation network that guides scene text synthesis through style adaptation and text recognition. VTNET introduces a conditional diffusion-based scene text editing method that can adapt to text replacement in six popular languages. Scene text editing methods with mask guidance generate a mask of the source text during the scene text editing process, explicitly indicating the text editing area. For example, SRNet achieves scene text editing at the text level, but due to the neglect of the geometric properties of text (arbitrary direction, bending, perspective, etc.), the results lack realism. PSGText adds a pyramid scene parsing (PSP) module to the SRNet framework, enabling efficient and accurate migration of text style and content through stroke guidance. Similarly, to address the text geometry adjustment problem of SRNet, Mask-guided GAN uses a text mask to guide the scene text editing process, making full use of the main body, outline, and shadow of the text to guide the task. MOSTEL improves upon SwapNet by using a text mask to indicate the editable area, helping the network focus on learning the editing rules of the text area. FAST proposes a novel font-independent scene text editing framework to address the issue of poor font style migration in text editing, especially in complex backgrounds.DARLING introduces a learning framework that separates style and content features, supervised by text recognition and alignment loss, and achieves excellent performance in scene text recognition, deletion and editing.

[0078] Generally, these models are trained in an entangled manner, often leading to incoherent and unclear edited text. In contrast, the embodiment of the present application proposes an explicit decoupling scene text editing method, which separates text content, text style and background, and completes scene text editing through a gradual fusion process.

[0079] The scene text editing method, system, electronic device and storage medium provided by the embodiment of the present application relate to the technical field of computer vision. The scene text editing method provided by the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto; the server end can be configured as an independent physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform, and the server can also be a node server in a blockchain network; the software can be an application that implements the scene text editing method, and the like, but is not limited to the above forms.

[0080] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0081] As shown in Figure 1 , it is an implementation environment schematic diagram provided by the embodiment of the present application. Referring to Figure 1The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected by wireless or wired means to complete data transmission and exchange.

[0082] The server 101 can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0083] In addition, the server 101 can also be a node server in a blockchain network. The blockchain is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0084] The terminal 102 can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 can also be a vehicle-mounted terminal of various device types as exemplified above, but is not limited thereto. The terminal 102 and the server 101 can be directly or indirectly connected by wired or wireless communication, and the present embodiment is not limited thereto.

[0085] Exemplarily based on the implementation environment shown in Figure 1 The present embodiment provides a scene text editing method. The following takes the scene text editing method applied to the server 101 as an example for illustration. It can be understood that the method can also be applied to the terminal 102.

[0086] Referring to Figure 2 , Figure 2 The flowchart of the scene text editing method applied to the server provided by the present embodiment can be executed by any one of the computer devices (including servers or terminals) described above. Referring to Figure 2 The method can include the following steps:

[0087] The scene text image is input into a text background separation module, a lightweight segmentation network is used to predict a text foreground mask of the scene text image, and the source text and the background image are separated by using the text foreground mask;

[0088] The style-guided text transfer decoding module is used to align the geometric and stylistic properties of the content text with the source text, and generate a target text;

[0089] combine the generated target text with the background image to determine a minimized background reconstruction region;

[0090] reconstruct the minimized background reconstruction region by a context-focused background reconstruction module to generate an edited complete scene text image.

[0091] In some embodiments, the text migration decoding module guided by style aligns the geometric and stylistic properties of the content text with the source text and generates a target text, including the following steps:

[0092] extract the geometric features of the source text through a shallow convolutional network and predict the spatial layout anchor points of the source text through a fully connected layer, the spatial layout anchor points being used to represent the geometric structure of the source text;

[0093] based on the spatial layout anchor points, accurately adjust the geometric properties of the content text to match the geometric structure of the source text by introducing a thin-plate spline interpolation algorithm;

[0094] respectively encode the source text and the content text through a content encoding network and a style encoding network to obtain text content features and text style features;

[0095] inject the style features into the content features through an adaptive instance normalization layer to achieve dynamic adjustment of the style representation to the content representation, thereby generating a target text consistent with the style of the source text.

[0096] In some embodiments, the combining the generated target text with the background image to determine a minimized background reconstruction region includes the following steps:

[0097] After obtaining the target text, replace the target text with the source text in the scene text image to determine the smallest background repair region.

[0098] In some embodiments, the reconstructing the minimized background reconstruction region by a context-focused background reconstruction module to generate an edited complete scene text image includes the following steps:

[0099] After text replacement, encode the image using a reconstruction encoder and input the encoding result into a residual module to enhance the expression of background details and structures;

[0100] Use a self-attention module to strengthen the network's ability to capture context information to ensure the accuracy and consistency of background reconstruction;

[0101] Generate a complete scene text editing image based on the enhanced features through a decoder.

[0102] In some embodiments, the method further comprises a step of training the multi-task model, which comprises:

[0103] decoupling the source text and the background of the scene text image by using a text background segmentation module, wherein the input of the text background segmentation module is the scene text image, and the output is the source text and the background image;

[0104] transferring the geometric attribute and the style attribute of the source text to the content text by using a text migration module to obtain the target text;

[0105] embedding the target text into the non-text scene background, and encoding the incomplete image by using a reconstruction encoder, and then inputting the encoding into a residual module to effectively enhance the expression of background details and structures;

[0106] completing the training and learning of each module in the multi-task model according to the overall loss function of the entire multi-task model.

[0107] In some embodiments, the decoupling of the source text and the background of the scene text image by using the text background segmentation module, wherein the input of the text background segmentation module is the scene text image, and the output is the source text and the background image, comprises the following steps:

[0108] inputting the scene text image I s into a lightweight segmentation network G s to predict the mask M s of the source text, and then performing text background separation by using the mask, wherein the expression of the text background separation is:

[0109]

[0110] wherein I sf represents the source text, and I sb represents the background image.

[0111] In some embodiments, the transferring of the geometric attribute and the style attribute of the source text to the content text by using the text migration module to obtain the target text comprises the following steps:

[0112] extracting the features of the source text I sf by using a shallow CNN network G g , and predicting N position anchors A n that can represent the spatial layout of the source text by using a fully connected layer F c , wherein the expression of the process is: A n =F c (G g (I sf ;Θ g );Θ fc ); wherein Θg and Θ fc They represent network G respectively g and network layer F c Learnable parameters;

[0113] After obtaining the target location anchor point, the TPS algorithm is used to process the content text I. t To perform geometric style alignment, the expression for this process is: in This refers to the geometrically aligned text content.

[0114] Through stylistic network G s For source text I sf The stylistic features of the source text are encoded, and the encoded features are then passed through a multilayer perceptron to construct a stylistic representation F of the source text. s The expression for this process is: F s =MLP(G s (I sf ;Θ s );Θ mlp ); where Θ s and Θ mlp They represent network G respectively s and learnable parameters in the MLP layer;

[0115] Content text After content network G c After processing, the content representation F is obtained. t : Where Θ c Content network G c Learnable parameters;

[0116] Introducing an AdaIN-based style decoder and To generate scene text fonts of arbitrary styles, the two decoders share the same network structure, including two residual modules and two upsampling modules, and all normalization layers in the decoder use AdaIN;

[0117] Calculate the average value m of the style representation s and variance and the average value m of content representation t and variance The expression for this process is:

[0118]

[0119] Where H represents the height of the input text style or content representation; W represents the width of the input text style or content representation. and respectively denote the element at position (s, t) in style and content representation, ∈ is a small constant for numerical stability;

[0120] AdaIN is used to perform affine transformation on the font style of the content text to realize font transfer, and the expression of the process is:

[0121]

[0122] wherein, represents the content representation of the kth AdaIN layer;

[0123] Text style representation F s can be gradually injected into the content representation F t in the decoding process through K AdaIN layers in the decoder, and the expression of the process is:

[0124]

[0125] wherein, O t represents the target text; represents the target text decoder; M t represents the mask corresponding to the target text; represents the target text mask decoder; and respectively denote the learnable parameters of the target text decoder and the mask decoder.

[0126] In some embodiments, the total loss function includes a text background separation loss, a style transfer loss, a mask loss, and a background reconstruction loss.

[0127] wherein, the expression of the text background separation loss is:

[0128]

[0129] wherein, T s is the true value corresponding to M s , and ∩ represents the intersection operation on pixels;

[0130] The expression of the style transfer loss is:

[0131]

[0132] wherein, β1 represents a balance factor; T o is the true value corresponding to O t ; φ i represents the feature map from VGG-19;

[0133] The expression of the mask loss The expression of is:

[0134]

[0135] Wherein, T t is the true value corresponding to M t , D t is the discriminator; I t represents the content text; γ1 and γ2 are balance factors;

[0136] The expression of background reconstruction loss is:

[0137]

[0138] Wherein, ξ1 is a balance factor; T f is the true value corresponding to O f .

[0139] The implementation process of the scene text editing method in specific scenarios will be described in detail below in conjunction with the drawings in the specification:

[0140] The scene text editing method for implementing the present application, as shown in Figure 3 , includes the following steps:

[0141] 1) Text and background separation. We predict the mask of the source text through a lightweight segmentation network, so as to effectively separate the source text and the background in the scene text image

[0142] 2) Style-guided text transfer. After the text and background are separated, we perform geometric transfer and style transfer on the content text. First, we use a shallow convolutional neural network (CNN) and a fully connected layer to predict the spatial layout of the source text, and use the Thin-Plate Spline (TPS) algorithm to perform geometric alignment on the content text. Then, in order to ensure the consistency of the style, we further adjust the style attribute of the content text after geometric alignment. By using a style and genre extraction network to represent the genre features of the source text, we adjust the adaptive instance normalization (AdaIN) layer in the decoder to realize the injection of the source text style and genre to the content text, so that the content text is consistent with the source text in terms of geometry and genre.

[0143] 3) Minimum region determination. After obtaining the target text, we replace the target text with the source text in the scene text image to determine the minimum background repair region.

[0144] 4) Context-focused background reconstruction. After text replacement, the image is encoded using the reconstruction encoder, and the encoding result is input into the residual module to enhance the expression of background details and structure. Then, the self-attention module is used to strengthen the network's ability to capture context information, ensuring the accuracy and consistency of background reconstruction. Finally, the decoder generates a complete scene text editing image based on the enhanced features.

[0145] In some embodiments of the present application, the input is an RGB scene text image I s and a content text image I t . During training and testing, the image is randomly scaled, and all input images are scaled to 64x256 size.

[0146] The specific steps are as follows:

[0147] 1) Use the text background segmentation module to decouple the source text and background of the scene text image, and input the scene text image I s , output the source text I sf and the background I sb . Specifically, the scene text image I s ∈R H×W×3 is input into the lightweight segmentation network G s to predict the mask N s of the source text, and then the text background is separated by the mask:

[0148]

[0149] 2) Transfer the geometric properties of the source text I sf to the content text I t . The embodiment of the present application uses a shallow CNN network G g to extract the features of the source text I sf , and through a fully connected layer F c , N position anchors A n ∈R N×2 that can represent the spatial layout of the source text are predicted:

[0150] A n =F c (G g (I sf ;Θ g );Θ fc ),

[0151] where Θ g and Θ fc represent the learnable parameters in the network G g and the network layer F c , respectively. After obtaining the target position anchor, the TPS algorithm is used to process the content text It Geometric pattern alignment is performed:

[0152]

[0153] wherein is the content text after geometric alignment.

[0154] 3) The stylistic properties of the source text I sf are transferred to the content text I t . The stylistic style network G s is responsible for encoding the stylistic features of the source text I sf , and these encoded features are then passed through a multi-layer perceptron (MLP) to build the stylistic style representation F s of the source text:

[0155] F s = MLP(G s (I sf ; Θ s ); Θ mlp ),

[0156] wherein Θ s and Θ mlp represent the learnable parameters in the network G s and the MLP layer, respectively. Similarly, the content text is processed by the content network G c to obtain the content representation F t :

[0157]

[0158] wherein Θ c represents the learnable parameters in the content network G c . To adapt to the changes in the stylistic style of the scene text image, the present embodiment introduces a style decoder based on AdaIN and to generate the font of the scene text in any style. The two decoders share the same network structure, including two residual modules and two up-sampling modules, and all normalization layers within the decoder use AdaIN. To use AdaIN to transfer the stylistic style representation to the content representation, the present embodiment calculates the mean m s and variance of the style representation and the mean m t and variance of the content representation:

[0159]

[0160] wherein and respectively represent the element at position (s, t) in style and content representation. ∈ is a small constant for numerical stability. All AdaIN layers in the decoder have no learnable parameters. AdaIN is used to perform affine transformation on the font style of the content text to achieve font transfer:

[0161]

[0162] where, represents the content representation of the k-th AdaIN layer. The text style representation F s can be gradually injected into the content representation F t in the decoding process by K AdaIN layers in the decoder:

[0163]

[0164] where, and represent the learnable parameters of the target text decoder and the mask decoder, respectively.

[0165] 4) After obtaining the target text O t , the embodiment of the present application embeds it in the non-text scene background I sb . The embodiment of the present application uses the target text mask M t to guide the pasting process, minimizing the area that needs to be reconstructed:

[0166]

[0167] where, is an incomplete scene text editing image containing some blank pixels.

[0168] 5) Realistic scene text editing requires changing the text while maintaining the consistency of the background. To achieve this, the embodiment of the present application first encodes the incomplete image using the reconstruction encoder G b . Then the encoding is input into the residual module R e to effectively enhance the expression of background details and structures:

[0169]

[0170] where, b and Θ e represent the learnable parameters of the reconstruction encoder G b and the residual module R e , respectively. Next, these enhanced features F r are input into the self-attention module to strengthen the network's ability to learn contextual information from the image:

[0171]

[0172] where F sa is the image feature obtained by the self-attention module. denotes a convolutional layer. Finally, the decoder decodes these learned features to generate the complete scene text editing image O f :

[0173] O f = D b (F sa ; Θ b ),

[0174] where Θ b denotes the learnable parameters of the reconstruction decoder D b .

[0175] The method proposed by the embodiment of the application is a multi-task model, and different sub-networks are assigned specific tasks during training. Therefore, the output of each sub-network is supervised by corresponding information. The text background separation module aims to predict the mask M s of the input scene text image I s . To achieve this, the embodiment of the application adopts Dice Loss as the text background separation loss The formula is as follows:

[0176]

[0177] where T s is the ground truth corresponding to M s , and represents the intersection operation on pixels. The goal of the style-guided text transfer module is to help the content text learn the geometric and stylistic properties of the source text. The embodiment of the application utilizes the style transfer loss to supervise the learning of geometric and stylistic features of the content text:

[0178]

[0179] where β1 is a balance factor. T o is the ground truth corresponding to O t , and φ i is the feature map from VGG-19. At the same time, the mask loss is utilized to supervise the accurate generation of the target mask:

[0180]

[0181] where γ1 and γ2 are balance factors, T t is the ground truth corresponding to M t , and D tis the discriminator. The context-focused background reconstruction module is responsible for repairing incomplete images and generating the final scene text editing result O f To ensure the high quality of the edited image, the background reconstruction loss is formulated as follows:

[0182]

[0183] wherein, ξ1 is a balance factor. T f is the true value corresponding to O f The loss function of the entire framework is represented as follows:

[0184]

[0185] It is worth noting that in order to let each module focus on learning the rules of its specific task without being disturbed by other modules, the gradient between the modules is blocked in the embodiments of the present application. Among them, the gradient blocking between the text background segmentation module and the domain style guided text transfer decoding model is represented as GB1, and the gradient blocking between the minimization region determination and the context-focused background reconstruction is represented as GB2. Although more training rounds are needed to converge after blocking, the editing performance is greatly improved. In addition, the features of the sub-tasks will not be accumulated with the accumulation of the gradient, avoiding feature redundancy and reducing the decoding burden of each sub-task.

[0186] The scene text editing method for explicitly decoupling text transfer and minimizing background reconstruction proposed by the present application has the following test environment and experimental results:

[0187] (1) Test environment:

[0188] System environment: ubuntu20.04.

[0189] Hardware environment: memory: 24GB, GPU: GeForce RTX 3090Ti, CPU: Intel(R) Xeon(R) Gold 6226R, hard disk: 2TB.

[0190] (2) Experimental data:

[0191] The present application uses the synthetic dataset Tamper-Syn2K for training. In the test phase, the embodiments of the present application respectively evaluate the performance of the model on the real-world dataset Tamper-Scene and the synthetic dataset Tamper-Syn2K.

[0192] (3) Optimization method:

[0193] Embodiments of the present application set the factors as: β1=20, γ1=10, γ2=5, ξ1=10 and K=6. After convergence, embodiments of the present application select the best weight for reasoning, and the batch size is 1. The model of embodiments of the present application is implemented using the PyTorch v1.7 deep learning framework. The experiment is distributed training on a single 3090Ti GPU.

[0194] (4) Experimental results:

[0195] 1) Ablation experiment:

[0196] The ablation experiment results completed on the Tamper-Syn2K dataset are shown in Table 1. Different deformable convolutions, self-attention modules, and gradient clipping strategies were evaluated in the experiment. As shown in the first row of Table 1, without deformable convolution, the evaluation indicators of SSIM and RecAcc decreased significantly. This is because deformable convolution is more effective in simulating complex shapes and details in font transfer, and can generate more accurate and detailed font images when the text changes, making the generated font more artistic and realistic. As shown in the second row, after removing the attention mechanism, the FID increased from 24.67 to 27.68, and the RecAcc decreased from 78.8% to 75.4%. This is because the mechanism enhances the ability to distinguish text and background, which is crucial to ensure that the text structure is not destroyed during background reconstruction. Without it, the network needs to solve the context capture problem and handle local information at the same time, increasing the difficulty of feature learning, and ultimately generating low-quality images. Further, the embodiment of the present application explores the impact of gradient settings between modules on the model. When the gradient blocking between the text-background separation module and the decoupled text transfer module is removed, i.e., GB1 is removed, as shown in Table 1 w / GB1, the RecAcc decreases from 78.8% to 0.3%, and the FID increases from 24.67 to 58.52. This degradation is due to the loss function of decoupled text transfer weakening the optimization of text-background separation. Text-background separation needs to focus on learning source text features to achieve accurate text segmentation, while decoupled text transfer focuses on text style transfer. Gradient blocking between text-background separation and decoupled text transfer helps the text-background separation network find its best optimization direction. When the gradient between the decoupled text transfer module and the background reconstruction module is removed, i.e., GB2 is removed. As shown in Table 1 w / GB2, although the MSE, SSIM, PSNR and FID did not reach the highest value, the RecAcc achieved the highest score of 84.0%. This observation shows that unblocked gradients help text integrity, but also affect image quality. This effect can be attributed to the reconstruction process prioritizing the enhancement of the text area, resulting in more complete text content and clearer strokes. Finally, removing all gradient blocks in the network, i.e., removing GB1 and GB2 at the same time, observed poor results in both visual and quantitative analysis of scene text editing. As shown in Table 1, the MSE, PSNR, SSIM and FID scores are the worst. The main reason for this problem is that the decoupled text transfer and background reconstruction loss function also optimizes the text-background separation. However, this optimization interferes with the specific optimization of text-background separation itself. Therefore, this interference will confuse the optimization direction, resulting in inaccurate text area positioning and ultimately failing to achieve the expected editing results.

[0197] Table 1: Verification of effectiveness of the proposed modules

[0198]

[0199]

[0200] 2) Performance comparison:

[0201] As can be seen from Table 2, the method of the present application achieves the most advanced performance, with MSE of 0.0123, PSNR of 20.83, SSIM of 0.7210, FID of 24.67, and RecAcc of 78.8%. Specifically, compared with Pix2Pix, the model of the embodiment of the present application is significantly improved by 139.57 and 60.4% in FID and RecAcc, respectively. This is because Pix2Pix is a style conversion method designed for the entire image, while the method of the embodiment of the present application only focuses on the text area, which means that the scene text editing task can be completed in a simpler way. In addition, compared with SRNet, the model of the embodiment of the present application achieves a significant improvement of 16.59% and 46.5% in FID and RecAcc, respectively. This is because the text editing of SRNet is difficult to adapt to the background size, often leading to poor image quality and text blur due to background scaling. In addition, compared with SwapNet, the model of the embodiment of the present application achieves an improvement of 0.0686% and 18.17% in SSIM and RecAcc, respectively. This is because the embodiment of the present application avoids the influence of complex background on text editing, enabling the model to generate higher quality text. Compared with DARLING, the model of the embodiment of the present application achieves an improvement of 19.81% in FID and 7.9% in RecAcc. The relatively poor performance of DARLING can be attributed to its insufficient feature learning due to the lack of explicit decoupling between text style and text content features, which easily leads to incomplete generated target text. Finally, compared with MOSTEL, the model of the embodiment of the present application improves by 4.81% and 2.00% in FID and RecAcc, respectively. The adopted scene text element separation strategy enhances the delicacy of the model editing results, making the text content more clear and complete. The qualitative editing results are shown in Fig. \ref{Fig6}, and it can be observed that the method of the embodiment of the present application has the best visual experience.

[0202] Table 2: Performance comparison with various text editing methods

[0203]

[0204] In summary, compared with the prior art, the present application has the following advantages:

[0205] 1. The present application proposes an end-to-end scene text editing method, which ensures effective scene text editing.

[0206] 2. The present application separates the foreground text from the background, reducing the impact of the background on text editing.

[0207] 3. The present application decouples the style attributes of the text into geometric and stylistic attributes, and uses style-guided text transfer to align the target text with the source text in terms of geometric and stylistic attributes.

[0208] 4. The present application uses target text to replace source text to ensure that the reconstructed background area is kept to a minimum, thereby reducing the complexity of the reconstruction process.

[0209] Another aspect of the embodiment of the present application also provides a scene text editing system, comprising:

[0210] A first module for inputting a scene text image into a text background separation module, predicting a text foreground mask of the scene text image through a lightweight segmentation network, and separating a source text and a background image using the text foreground mask;

[0211] A second module for aligning the geometric and stylistic attributes of the content text with the source text through a style-guided text transfer decoding module, and generating a target text;

[0212] A third module for combining the generated target text with the background image to determine a minimized background reconstruction area;

[0213] A fourth module for background reconstruction of the minimized background reconstruction area through a context-focused background reconstruction module to generate an edited complete scene text image.

[0214] It can be understood that the contents in the above method embodiments are applicable to the present system embodiment, the present system embodiment specifically realizes the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0215] The embodiment of the present application also provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor realizes the above scene text editing method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0216] It can be understood that the contents in the above method embodiments are applicable to the present device embodiment, the present device embodiment specifically realizes the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0217] Please refer to Figure 4 , Figure 4 The hardware structure of the electronic device of another embodiment is illustrated, which comprises:

[0218] The processor 401 can be implemented in a manner of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0219] The memory 402 can be implemented in a manner of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), and can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, related program codes are stored in the memory 402 and are called and executed by the processor 401 to implement the scene text editing method of the embodiments of the present application.

[0220] The input / output interface 403 is used to realize information input and output.

[0221] The communication interface 404 is used to realize the communication interaction between the device and other devices, and can realize the communication through a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0222] The bus 405 is used to transmit information between various components (for example, the processor 401, the memory 402, the input / output interface 403, and the communication interface 404) of the device.

[0223] The processor 401, the memory 402, the input / output interface 403, and the communication interface 404 are connected to each other through the bus 405 to realize the communication connection between the device.

[0224] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the scene text editing method.

[0225] It can be understood that the contents in the above method embodiments are all applicable to the present storage medium embodiments, the functions implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved by the present storage medium embodiments are also the same as those of the above method embodiments.

[0226] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0227] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0228] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0229] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0230] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0231] The terms "first", "second", "third", "fourth" and the like used in the specification of the present application and the above-described drawings, if any, are used to distinguish similar objects, and do not necessarily have to be used to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0232] It should be understood that, in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0233] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0234] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0235] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0236] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0237] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for editing scene text, characterized in that, Includes the following steps: The scene text image is input into the text background separation module. A lightweight segmentation network predicts the text foreground mask of the scene text image, and the source text and background image are separated using the text foreground mask. The style-guided text migration decoding module aligns the geometric and stylistic attributes of the content text with the source text and generates the target text. The generated target text is combined with the background image to determine the minimized background reconstruction area; The context-focused background reconstruction module reconstructs the minimized background reconstruction area to generate an edited complete scene text image. The style-guided text migration decoding module aligns the geometric and stylistic attributes of the content text with the source text and generates the target text, including the following steps: Geometric features of the source text are extracted through a shallow convolutional network, and spatial layout anchors of the source text are predicted through a fully connected layer. These spatial layout anchors are used to represent the geometric structure of the source text. Based on the aforementioned spatial layout anchor points, a thin template interpolation algorithm is introduced to precisely adjust the geometric attributes of the content text in order to match the geometric structure of the source text. The adjusted content text is feature-encoded using a content coding network to obtain text content features; the source text is feature-encoded using a style coding network to obtain text style features. By using an adaptive instance normalization layer, style features are injected into content features to achieve dynamic adjustment of style representation to content representation, thereby generating target text with the same style as the source text.

2. The scene text editing method according to claim 1, characterized in that, The step of combining the generated target text with the background image to determine the minimized background reconstruction region includes the following steps: After obtaining the target text, the target text is replaced with the source text in the scene text image to determine the minimum background reconstruction area.

3. The scene text editing method according to claim 2, characterized in that, The background reconstruction module using context focusing reconstructs the minimized background reconstruction region to generate an edited complete scene text image, including the following steps: After text replacement, the background image is encoded using a reconstruction encoder, and the encoded results are sequentially input into the residual module and the self-attention module to obtain enhanced features. The enhanced features are then input into the decoder to obtain the scene text-edited image.

4. The scene text editing method according to claim 1, characterized in that, The method further includes a step of training a multi-task model, which includes: The text background segmentation module is used to decouple the source text and background of the scene text image. The input of the text background segmentation module is the scene text image, and the output is the source text and the background image. The text migration module transfers the geometric and stylistic attributes of the source text to the content text to obtain the target text. The target text is embedded into a textless scene background, and the incomplete image is encoded using a reconstruction encoder. The encoded text is then fed into a residual module to enhance the expression of background details and structure. Based on the overall loss function of the entire multi-task model, the training and learning of each module in the multi-task model is completed.

5. A scene text editing method according to claim 4, characterized in that, The text background segmentation module decouples the source text and background of the scene text image. The input of the text background segmentation module is the scene text image, and the output is the source text and the background image. The steps include: scene text image Input into a lightweight segmentation network Foreground mask for predicted text Next, text background separation is performed using a text foreground mask. The expression for text background separation is: in, Represents the source text; Represents the background image.

6. A scene text editing system, characterized in that, include: The first module is used to input the scene text image into the text background separation module, predict the text foreground mask of the scene text image through a lightweight segmentation network, and separate the source text and background image using the text foreground mask; The second module is used to align the geometric and stylistic attributes of the content text with the source text through a style-guided text migration decoding module, and generate the target text. The third module is used to combine the generated target text with the background image to determine the minimized background reconstruction area; The fourth module is used to reconstruct the minimized background reconstruction area using the context-focused background reconstruction module, and generate an edited complete scene text image; Specifically, the second module is used for: Geometric features of the source text are extracted through a shallow convolutional network, and spatial layout anchors of the source text are predicted through a fully connected layer. These spatial layout anchors are used to represent the geometric structure of the source text. Based on the aforementioned spatial layout anchor points, a thin template interpolation algorithm is introduced to precisely adjust the geometric attributes of the content text in order to match the geometric structure of the source text. The adjusted content text is feature-encoded using a content coding network to obtain text content features; the source text is feature-encoded using a style coding network to obtain text style features. By using an adaptive instance normalization layer, style features are injected into content features to achieve dynamic adjustment of style representation to content representation, thereby generating target text with the same style as the source text.

7. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 5.

Images