Scene text editing method and system, electronic equipment and storage medium

Abstract

The invention discloses a scene text editing method and system, electronic equipment and a storage medium, and the method comprises the steps: inputting a scene text image into a text background separation module, predicting a text foreground mask of the scene text image through a lightweight segmentation network, and separating a source text and a background image through the text foreground mask; through a style-guided text migration decoding module, aligning geometric and literary and physical attributes of the content text with the source text, and generating a target text; combining the generated target text with the background image, and determining a minimized background reconstruction area; and background reconstruction is carried out on the minimized background reconstruction area through a context focusing background reconstruction module, and an edited complete scene text image is generated. According to the embodiment of the invention, the scene text editing effect can be improved, the influence of the background on text editing is reduced, the complexity of the text editing process is reduced, and the method can be widely applied to the technical field of computer vision.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a scene text editing method, system, electronic device and storage medium. Background Art

[0002] With the rapid development of computer vision technology, scene text editing has become an important research direction in this field. The core task of scene text editing is to replace the source text in the image while trying to maintain the integrity of the scene background and text style. Such technologies have broad application prospects in augmented reality, data generation, privacy protection, etc. For example, in augmented reality applications, real-time translation and text replacement functions are of great significance for cross-language communication; in terms of data generation, synthetic scene text data can be used to train and optimize deep learning models; in terms of privacy protection, automated text editing technology can effectively remove sensitive information from images. However, scene text editing is a very challenging task, and its main difficulties are concentrated in two aspects: the diversity of text styles and the complexity of background reconstruction. First, the diversity of text styles is reflected in multiple aspects such as stylistic style and text geometric layout. The stylistic style of the text may include different fonts, sizes, and colors, while the geometric layout may involve various forms such as long text, tilted text, and perspective text. These diverse styles bring great complexity and difficulty to text migration. Secondly, the complexity of background reconstruction mainly stems from the texture structure and occlusion problems of the scene background. Scene text is usually located on complex backgrounds. These backgrounds may contain delicate textures and rich color changes, and may even have a certain occlusion relationship with the text itself. Even a subtle text replacement operation, if not handled properly, may cause visual disharmony and destroy the overall aesthetics of the image. Therefore, when reconstructing the background, how to accurately reconstruct the background area covered by the text while avoiding excessive modification of unnecessary areas is the key to improving editing effects and efficiency. In short, scene text editing in natural scenes is a very challenging task in the field of computer vision. Summary of the invention

[0003] The main purpose of the embodiments of the present invention is to propose a scene text editing method, system, electronic device and storage medium, which can improve the effect of scene text editing, reduce the impact of background on text editing, and reduce the complexity of the text editing process.

[0004] To achieve the above object, an embodiment of the present invention provides a scene text editing method, comprising the following steps:

[0005] Inputting the scene text image into the text-background separation module, predicting the text foreground mask of the scene text image through a lightweight segmentation network, and using the text foreground mask to separate the source text and the background image;

[0006] Aligning geometric and stylistic properties of the content text with the source text and generating a target text through a style-guided text transfer decoding module;

[0007] Combining the generated target text with the background image to determine a minimized background reconstruction area;

[0008] The minimized background reconstruction area is reconstructed by a context-focused background reconstruction module to generate an edited complete scene text image.

[0009] In some embodiments, the style-guided text transfer decoding module aligns the geometric and stylistic attributes of the content text with the source text and generates the target text, including the following steps:

[0010] Extract the geometric features of the source text through a shallow convolutional network, and predict the spatial layout anchor points of the source text through a fully connected layer. The spatial layout anchor points are used to represent the geometric structure of the source text.

[0011] Based on the spatial layout anchor points, a thin template interpolation algorithm is introduced to precisely adjust the geometric properties of the content text to match the geometric structure of the source text;

[0012] The source text and content text are encoded by the content encoding network and the style encoding network respectively to obtain the text content features and text style features;

[0013] Through the adaptive instance normalization layer, the style features are injected into the content features to achieve dynamic adjustment of the style representation to the content representation, thereby generating a target text consistent with the style of the source text.

[0014] In some embodiments, combining the generated target text with the background image to determine a minimized background reconstruction area comprises the following steps:

[0015] After obtaining the target text, the target text is replaced with the source text in the scene text image to determine the smallest background repair area.

[0016] In some embodiments, the background reconstruction module using context focusing performs background reconstruction on the minimized background reconstruction area to generate an edited complete scene text image, comprising the following steps:

[0017] After text replacement, the image is encoded using a reconstruction encoder and the encoding result is input into the residual module to enhance the expression of background details and structures;

[0018] Use the self-attention module to enhance the network’s ability to capture contextual information and ensure the accuracy and consistency of background reconstruction;

[0019] The decoder generates a complete scene text editing image based on the enhanced features.

[0020] In some embodiments, the method further comprises the step of training a multi-task model, the step comprising:

[0021] Decoupling the source text and background of the scene text image by using a text-background segmentation module, wherein the input of the text-background segmentation module is the scene text image, and the output is the source text and the background image;

[0022] The geometric attributes and stylistic attributes of the source text are transferred to the content text through the text migration module to obtain the target text;

[0023] The target text is embedded into the text-free scene background, and the incomplete image is encoded using a reconstruction encoder, which is then input into the residual module to effectively enhance the expression of background details and structures;

[0024] According to the overall loss function of the entire multi-task model, the training and learning of each module in the multi-task model is completed.

[0025] In some embodiments, the method of decoupling the source text and background of the scene text image by using a text background segmentation module, wherein the input of the text background segmentation module is the scene text image and the output is the source text and the background image, comprises the following steps:

[0026] The scene text image I s Input to the lightweight segmentation network G s The mask M of the predicted source text s , then the text and background are separated by mask, and the expression of the text and background separation is:

[0027]

[0028] Among them, I sf Represents the source text; I sb Represents a background image.

[0029] In some embodiments, the step of transferring the geometric attributes and stylistic attributes of the source text to the content text through the text migration module to obtain the target text comprises the following steps:

[0030] Using the shallow CNN network G g Extract source text I sf The features are then passed through the fully connected layer F c Predict N position anchor points A that can represent the spatial layout of the source textn , the expression of this process is: A n =F c (G g (I sf ; Θ g );Θ fc ), where Θ g and θ fc Represents the network G g and network layer F c The learnable parameters in ;

[0031] After obtaining the target position anchor point, the TPS algorithm is used to analyze the content text I t Perform geometric style alignment. The expression of this process is: in It is the content text after geometric alignment;

[0032] Through the style network G s Source Text I sf The stylistic features of the source text are encoded, and the encoded features are then passed through a multi-layer perceptron to construct the stylistic style representation F of the source text. s , the expression of this process is: F s =MLP(G s (I sf ; Θ s );Θ mlp ), where Θ s and θ mlp Represents the network G s and learnable parameters in the MLP layers;

[0033] Content Text Through the content network G c After processing, we get the content representation F t : where Θ c Represents content network G c The learnable parameters in ;

[0034] Introducing AdaIN-based style decoder and To generate scene text fonts of any style, the two decoders share the same network structure, including two residual modules and two upsampling modules, and all normalization layers in the decoder use AdaIN;

[0035] Calculate the mean value m of the style representation s and variance and the mean value m of the content representation t and variance The expression of this process is:

[0036]

[0037] Wherein, H represents the height of the input text representation or content representation; W represents the width of the input text representation or content representation; and denote the element at position (s, t) in the stylistic representation and content representation, respectively, and ∈ is a small constant for numerical stability;

[0038] AdaIN is used to perform affine changes on the font style of the content text to achieve font migration. The expression of this process is:

[0039]

[0040] in, represents the content representation of the kth AdaIN layer;

[0041] Text stylistic representation s can be gradually injected into the content representation F during the decoding process through the K AdaIN layers in the decoder t The expression of this process is:

[0042]

[0043] Among them, O t represents the target text; represents the target text decoder; M t Represents the target text corresponding mask; represents the target text mask decoder; and denote the learnable parameters of the target text decoder and mask decoder respectively.

[0044] In some embodiments, the overall loss function includes text-background separation loss, style transfer loss, mask loss, and background reconstruction loss;

[0045] Among them, the text background separation loss The expression is:

[0046]

[0047] Among them, T s It is with M s The corresponding truth value, ∩, represents the intersection operation on pixels;

[0048] Style transmission loss The expression is:

[0049]

[0050] Among them, β1 represents the balance factor; T o Yes and O t The corresponding truth value; φ i Represents feature maps from VGG-19;

[0051] Mask loss The expression is:

[0052]

[0053] Among them, T t For M t Corresponding to the true value, D t I is the discriminator; t represents the content text; γ1 and γ2 are balance factors;

[0054] Background reconstruction loss The expression is:

[0055]

[0056] Among them, ξ1 is the balance factor; T f Yes and O f The corresponding truth value.

[0057] Another aspect of the embodiment of the present invention further provides a scene text editing system, including:

[0058] The first module is used to input the scene text image into the text-background separation module, predict the text foreground mask of the scene text image through a lightweight segmentation network, and separate the source text and the background image using the text foreground mask;

[0059] A second module is used to align the geometric and stylistic attributes of the content text with the source text through a style-guided text transfer decoding module and generate a target text;

[0060] A third module is used to combine the generated target text with the background image to determine a minimized background reconstruction area;

[0061] The fourth module is used to perform background reconstruction on the minimized background reconstruction area through a context-focused background reconstruction module to generate an edited complete scene text image.

[0062] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0063] The memory is used to store programs;

[0064] The processor executes the program to implement the method described above.

[0065] To achieve the above objective, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method is implemented.

[0066] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0067] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a scene text editing method, system, electronic device and storage medium, the scheme inputs the scene text image into the text background separation module, predicts the text foreground mask of the scene text image through a lightweight segmentation network, and uses the text foreground mask to separate the source text and the background image; aligns the geometric and stylistic attributes of the content text with the source text through a style-guided text migration decoding module, and generates a target text; combines the generated target text with the background image to determine the minimized background reconstruction area; reconstructs the background of the minimized background reconstruction area through a context-focused background reconstruction module, and generates an edited complete scene text image. The embodiments of the present invention can improve the effect of scene text editing, reduce the impact of the background on text editing, and reduce the complexity of the text editing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0069] Figure 2 is a flow chart of overall steps provided by an embodiment of the present invention;

[0070] Figure 3 It is a schematic diagram of the overall framework provided by an embodiment of the present invention;

[0071] Figure 4 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0073] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0074] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0076] Before describing the embodiments of the present invention in detail, some related technologies involved in the embodiments of the present invention are first described as follows:

[0077] End-to-end scene text editing models aim to realize scene text editing in a unified model. Current methods can be roughly divided into two categories: scene text editing methods without mask guidance and scene text editing methods with mask guidance. The mask-free method directly edits the entire scene image without the guidance of text mask, and the process is simple. For example, the STEFANN method directly edits a single character in the scene image, and the editing takes into account the color and font style of the character. However, this method cannot adaptively adjust the size of the character or flexibly handle words or text lines of variable length. To eliminate the application limitations of single character editing methods, SwapText implements word-level editing through multi-character supervision and introduces the thin plate spline (TPS) algorithm to make scene text adaptable to various geometric layouts. To further improve the usability of scene text editing networks, TextStyleBrush decouples the content and style representation of text in a self-supervised manner, thereby achieving a one-time transfer of the source style to the new content. To improve the performance of scene text editing networks in complex scenes, SimAN proposes a similarity-aware normalization module to enhance the network's ability to distinguish complex patterns (such as messy strokes and messy backgrounds). In order to reduce the high cost of training data synthesis, RewriteNet proposed a self-supervised training scheme combining synthetic data and real data to improve the performance of scene text editing networks. In addition, some scholars have introduced diffusion models into scene text editing tasks. DBEST proposed a diffusion-based scene text manipulation network to guide scene text synthesis through style adaptation and text recognition. VTNET introduced a scene text editing method based on conditional diffusion, which can adapt to text replacement in six popular languages. The mask-guided scene text editing method generates a mask of the source text during the scene text editing process, and the mask clearly indicates the text editing area. For example, SRNet implements text-level scene text editing, but the result lacks realism because it ignores the geometric properties of the text (arbitrary direction, curvature, perspective, etc.). PSGText adds a pyramid scene parsing (PSP) module to the SRNet framework to achieve efficient and accurate migration of text style and content through stroke guidance. Similarly, in response to the text geometry adjustment problem of SRNet, Mask-guided GAN uses text masks to guide the scene text editing process, making full use of the main body, outline and shadow of the text to guide the task. MOSTEL improves on SwapNet by indicating editable areas through text masks, helping the network focus on learning the editing rules of text areas. FAST proposes a novel font-independent scene text editing framework to solve the problem of poor font style transfer in text editing, especially in complex backgrounds.DARLING introduces a learning framework for separating style and content features, supervised by text recognition and alignment losses, and achieves excellent performance in scene text recognition, removal, and editing.

[0078] Generally speaking, these models are trained in an entangled manner, which often results in incoherent and unclear text after editing. In contrast, the embodiments of the present invention propose an explicitly decoupled scene text editing method, which separates text content, text style and background, and completes scene text editing through a gradual fusion process.

[0079] The scene text editing method, system, electronic device and storage medium provided by the embodiments of the present invention relate to the field of computer vision technology. The scene text editing method provided by the embodiments of the present invention can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the scene text editing method, etc., but is not limited to the above forms.

[0080] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0081] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0082] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0083] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0084] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0085] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides a scene text editing method. The scene text editing method is applied to the server 101 as an example for explanation below. It can be understood that the method can also be applied to the terminal 102.

[0086] Reference Figure 2 , Figure 2 This is a flowchart of a scene text editing method applied to a server provided in an embodiment of the present invention. The execution subject of the method can be any of the aforementioned computer devices (including a server or a terminal). Figure 2 , the method may include the following steps:

[0087] Inputting the scene text image into the text-background separation module, predicting the text foreground mask of the scene text image through a lightweight segmentation network, and using the text foreground mask to separate the source text and the background image;

[0088] Aligning geometric and stylistic properties of the content text with the source text and generating a target text through a style-guided text transfer decoding module;

[0089] Combining the generated target text with the background image to determine a minimized background reconstruction area;

[0090] The minimized background reconstruction area is reconstructed by a context-focused background reconstruction module to generate an edited complete scene text image.

[0091] In some embodiments, the style-guided text transfer decoding module aligns the geometric and stylistic attributes of the content text with the source text and generates the target text, including the following steps:

[0092] Extract the geometric features of the source text through a shallow convolutional network, and predict the spatial layout anchor points of the source text through a fully connected layer. The spatial layout anchor points are used to represent the geometric structure of the source text.

[0093] Based on the spatial layout anchor points, a thin template interpolation algorithm is introduced to precisely adjust the geometric properties of the content text to match the geometric structure of the source text;

[0094] The source text and content text are encoded by the content encoding network and the style encoding network respectively to obtain the text content features and text style features;

[0095] Through the adaptive instance normalization layer, the style features are injected into the content features to achieve dynamic adjustment of the style representation to the content representation, thereby generating a target text consistent with the style of the source text.

[0096] In some embodiments, combining the generated target text with the background image to determine a minimized background reconstruction area comprises the following steps:

[0097] After obtaining the target text, the target text is replaced with the source text in the scene text image to determine the smallest background repair area.

[0098] In some embodiments, the background reconstruction module using context focusing performs background reconstruction on the minimized background reconstruction area to generate an edited complete scene text image, comprising the following steps:

[0099] After text replacement, the image is encoded using a reconstruction encoder and the encoding result is input into the residual module to enhance the expression of background details and structures;

[0100] Use the self-attention module to enhance the network’s ability to capture contextual information and ensure the accuracy and consistency of background reconstruction;

[0101] The decoder generates a complete scene text editing image based on the enhanced features.

[0102] In some embodiments, the method further comprises the step of training a multi-task model, the step comprising:

[0103] Decoupling the source text and background of the scene text image by using a text-background segmentation module, wherein the input of the text-background segmentation module is the scene text image, and the output is the source text and the background image;

[0104] The geometric attributes and stylistic attributes of the source text are transferred to the content text through the text migration module to obtain the target text;

[0105] The target text is embedded into the text-free scene background, and the incomplete image is encoded using a reconstruction encoder, which is then input into the residual module to effectively enhance the expression of background details and structures;

[0106] According to the overall loss function of the entire multi-task model, the training and learning of each module in the multi-task model is completed.

[0107] In some embodiments, the method of decoupling the source text and background of the scene text image by using a text background segmentation module, wherein the input of the text background segmentation module is the scene text image and the output is the source text and the background image, comprises the following steps:

[0108] The scene text image I s Input to the lightweight segmentation network G s The mask M of the predicted source text s , then the text and background are separated by mask, and the expression of the text and background separation is:

[0109]

[0110] Among them, I sf Represents the source text; I sb Represents a background image.

[0111] In some embodiments, the step of transferring the geometric attributes and stylistic attributes of the source text to the content text through the text migration module to obtain the target text comprises the following steps:

[0112] Using the shallow CNN network G g Extract source text I sf The features are then passed through the fully connected layer F c Predict N position anchor points A that can represent the spatial layout of the source text n , the expression of this process is: A n =F c (G g (I sf ; Θ g );Θ fc ), where Θg and θ fc Represents the network G g and network layer F c The learnable parameters in ;

[0113] After obtaining the target position anchor point, the TPS algorithm is used to analyze the content text I t Perform geometric style alignment. The expression of this process is: in It is the content text after geometric alignment;

[0114] Through the style network G s Source Text I sf The stylistic features of the source text are encoded, and the encoded features are then passed through a multi-layer perceptron to construct the stylistic style representation F of the source text. s , the expression of this process is: F s =MLP(G s (I sf ; Θ s );Θ mlp ), where Θ s and θ mlp Represents the network G s and learnable parameters in the MLP layers;

[0115] Content Text Through the content network G c After processing, we get the content representation F t : where Θ c Represents content network G c The learnable parameters in ;

[0116] Introducing AdaIN-based style decoder and To generate scene text fonts of any style, the two decoders share the same network structure, including two residual modules and two upsampling modules, and all normalization layers in the decoder use AdaIN;

[0117] Calculate the mean value m of the style representation s and variance and the mean value m of the content representation t and variance The expression of this process is:

[0118]

[0119] Wherein, H represents the height of the input text representation or content representation; W represents the width of the input text representation or content representation; and denote the element at position (s, t) in the stylistic representation and content representation, respectively, and ∈ is a small constant for numerical stability;

[0120] AdaIN is used to perform affine changes on the font style of the content text to achieve font migration. The expression of this process is:

[0121]

[0122] in, represents the content representation of the kth AdaIN layer;

[0123] Text stylistic representation s can be gradually injected into the content representation F during the decoding process through the K AdaIN layers in the decoder t The expression of this process is:

[0124]

[0125] Among them, O t represents the target text; represents the target text decoder; M t Represents the target text corresponding mask; represents the target text mask decoder; and denote the learnable parameters of the target text decoder and mask decoder respectively.

[0126] In some embodiments, the overall loss function includes text-background separation loss, style transfer loss, mask loss, and background reconstruction loss;

[0127] Among them, the text background separation loss The expression is:

[0128]

[0129] Among them, T s It is with M s The corresponding truth value, ∩, represents the intersection operation on pixels;

[0130] Style transmission loss The expression is:

[0131]

[0132] Among them, β1 represents the balance factor; T o Yes and O t The corresponding truth value; φ i Represents feature maps from VGG-19;

[0133] Mask loss The expression is:

[0134]

[0135] Among them, T t For M t Corresponding to the true value, D t I is the discriminator; t represents the content text; γ1 and γ2 are balance factors;

[0136] Background reconstruction loss The expression is:

[0137]

[0138] Among them, ξ1 is the balance factor; T f Yes and O f The corresponding truth value.

[0139] The implementation process of the present invention in a specific scenario is described in detail below in conjunction with the accompanying drawings of the specification:

[0140] To implement the scene text editing method of the present invention, Figure 3 As shown, the following steps are included:

[0141] 1) Separation of text and background. We use a lightweight segmentation network to predict the mask of the source text, thereby effectively separating the source text and background in the scene text image.

[0142] 2) Style-guided text transfer. After separating the text from the background, we perform geometric transfer and stylistic transfer on the content text respectively. First, a shallow convolutional neural network (CNN) and a fully connected layer are used to predict the spatial layout of the source text, and the Thin-Plate Spline (TPS) algorithm is used to geometrically align the content text. Then, in order to ensure the consistency of style, we further adjust the stylistic properties of the content text after geometric alignment. The stylistic features of the source text are represented by a stylistic style extraction network, and the adaptive instance normalization (AdaIN) layer in the decoder is adjusted using the stylistic representation to achieve the injection of the source text stylistic style into the content text, so that the content text is consistent with the source text in both geometric and stylistic properties.

[0143] 3) Minimum region determination. After obtaining the target text, we replace the target text with the source text in the scene text image to determine the minimum background repair region.

[0144] 4) Context-focused background reconstruction. After text replacement, the image is encoded using a reconstruction encoder and the encoding result is input into the residual module to enhance the expression of background details and structures. Then, the self-attention module is used to strengthen the network's ability to capture contextual information and ensure the accuracy and consistency of background reconstruction. Finally, the decoder generates a complete scene text-edited image based on the enhanced features.

[0145] In some embodiments of the present invention, the input is an RGB scene text image I s and content text image I t The images are randomly scaled during training and testing, and all input images are scaled to 64×256.

[0146] The specific steps are as follows:

[0147] 1) Use the text-background segmentation module to decouple the source text and background of the scene text image. Its input is the scene text image I s , the output is the source text I sf And Background I sb Specifically, the scene text image I s ∈R H×W×3 is input into the lightweight segmentation network G s The mask N of the predicted source text s , and then separate the text from the background using a mask:

[0148]

[0149] 2) Source text I sf The geometric attributes are transferred to the content text I t In this embodiment of the present invention, a shallow CNN network G is used. g Extract source text I sf The features are then passed through the fully connected layer F c Predict N position anchor points A that can represent the spatial layout of the source text n ∈R N×2 :

[0150] A n =F c (G g (I sf ; Θ g );Θ fc ),

[0151] Among them, Θ g and θ fc Represents the network G g and network layer F c After obtaining the target position anchor point, the TPS algorithm is used to analyze the content text It To perform geometric style alignment:

[0152]

[0153] in The content text after geometric alignment.

[0154] 3) Source Text I sf The stylistic attributes are transferred to the content text I t In. Style Network G s Responsible for the source text I sf The stylistic features of the source text are then encoded, and these encoded features are then passed through a multi-layer perceptron (MLP) to construct the stylistic style representation F of the source text. s :

[0155] F s =MLP(G s (I sf ; Θ s );Θ mlp ),

[0156] Among them, Θ s and θ mlp Represents the network G s and the learnable parameters in the MLP layer. Similarly, the content text Through the content network G c After processing, we get the content representation F t :

[0157]

[0158] Among them, Θ c Represents content network G c In order to adapt to the changes in the style of the text in the scene text image, the embodiment of the present invention introduces a style decoder based on AdaIN and To generate scene text fonts of any style. The two decoders share the same network structure, including two residual modules and two upsampling modules, and all normalization layers in the decoder use AdaIN. In order to use AdaIN to transfer the stylistic style representation to the content representation, the embodiment of the present invention calculates the average value m of the style representation s and variance and the mean value m of the content representation t and variance

[0159]

[0160] in, and denote the element at position (s, t) in the style representation and content representation, respectively. ∈ is a small constant for numerical stability. All AdaIN layers in the decoder have no learnable parameters. AdaIN is used to perform affine changes on the font style of the content text to achieve font migration:

[0161]

[0162] in, Represents the content representation of the kth AdaIN layer. Text style representation F s can be gradually injected into the content representation F during the decoding process through the K AdaIN layers in the decoder t middle:

[0163]

[0164] in, and denote the learnable parameters of the target text decoder and mask decoder respectively.

[0165] 4) Get the target text O t Then, the embodiment of the present invention embeds it into the background of the scene without text. sb The embodiment of the present invention uses the target text mask M t to guide the pasting process and minimize the areas that require background reconstruction:

[0166]

[0167] in, is an incomplete scene text editing image, which contains some blank pixels.

[0168] 5) Realistic scene text editing requires changing the text while maintaining the consistency of the background. To achieve this, the embodiment of the present invention first uses a reconstruction encoder G b For incomplete images Encode. Then input the encoding into the residual module R e In order to effectively enhance the expression of background details and structures:

[0169]

[0170] Among them, Θ b and θ e Represent the reconstruction encoder G b and residual module R e Then, these enhanced features F r Input to the self-attention module to enhance the network’s ability to learn contextual information from the image:

[0171]

[0172] Among them, F sa are the image features obtained through the self-attention module. represents the convolutional layer. Finally, the decoder decodes these learned features to generate the complete scene text editing image O f :

[0173] O f =D b (F sa ; Θ b ),

[0174] Among them, Θ b Denotes the reconstruction decoder D b The learnable parameters of .

[0175] The method proposed in the embodiment of the present invention is a multi-task model. During training, different sub-networks are assigned specific tasks. Therefore, the output of each sub-network is supervised by corresponding information. The goal of the text-background separation module is to predict the input scene text image I s The mask M s To achieve this, the embodiment of the present invention adopts Dice Loss as the text background separation loss The formula is as follows:

[0176]

[0177] Among them, T s It is with M s The corresponding true value, ∩ represents the intersection operation of pixels. The goal of the style-guided text transfer module is to help the content text learn the geometric and stylistic attributes of the source text. Supervised learning of geometric and stylistic features of content text:

[0178]

[0179] Where β1 is the equilibrium factor. o Yes and O t The corresponding truth value. i is the feature map from VGG-19. At the same time, using mask loss Supervised accurate generation of object masks:

[0180]

[0181] Among them, γ1 and γ2 are balance factors, T t For M t Corresponding to the true value, D tis the discriminator. The context-focused background reconstruction module is responsible for repairing the incomplete image and producing the final scene text editing result O f In order to ensure the high quality of the edited image, the background reconstruction loss The formula is as follows:

[0182]

[0183] Where ξ1 is the equilibrium factor. f Yes and O f The corresponding true value. The loss function of the entire framework It is expressed as follows:

[0184]

[0185] It is worth noting that in order to allow each module to focus on learning the rules of its specific task without being disturbed by other modules, the embodiment of the present invention blocks the gradients between modules. Among them, the gradient blocking between the text background segmentation module domain style-guided text transfer decoding model is represented as GB1, and the gradient blocking between the background reconstruction of the minimized region determination domain context focus is represented as GB2. Although more training rounds are required to converge after blocking, the editing performance is greatly improved. In addition, the features of the subtasks will not accumulate with the accumulation of gradients, avoiding feature redundancy and reducing the decoding burden of each subtask.

[0186] The test environment and experimental results of the scene text editing method proposed in this invention that explicitly decouples text migration and minimizes background reconstruction are as follows:

[0187] (1) Test environment:

[0188] System environment: Ubuntu 20.04.

[0189] Hardware environment: Memory: 24GB, GPU: GeForce RTX 3090Ti, CPU: Intel(R)Xeon(R)Gold6226R, Hard disk: 2TB.

[0190] (2) Experimental data:

[0191] The present invention uses the synthetic data set Tamper-Syn2K for training. In the testing phase, the embodiments of the present invention evaluate the performance of the model on the real-world data set Tamper-Scene and the synthetic data set Tamper-Syn2K respectively.

[0192] (3) Optimization method:

[0193] The present invention sets the factors as: β1=20, γ1=10, γ2=5, ξ1=10 and K=6. After convergence, the present invention selects the best weights for inference, and the batch size is 1. The model of the present invention is implemented using the PyTorchv1.7 deep learning framework. The experiment is equipped with a single 3090Ti GPU for distributed training.

[0194] (4) Experimental results:

[0195] 1) Ablation experiment:

[0196] The results of the ablation experiments completed on the Tamper-Syn2K dataset are shown in Table 1. Different deformable convolutions, self-attention modules, and gradient truncation strategies were evaluated in the experiment. As shown in the first row of Table 1, without deformable convolutions, the evaluation indicators of SSIM and RecAcc dropped significantly. This is because deformable convolutions more effectively simulate complex shapes and details in font migration, and can generate more accurate and detailed font images when the text changes, making the generated font more artistic and realistic. As shown in the second row, after removing the attention mechanism, FID increased from 24.67 to 27.68, and RecAcc decreased from 78.8% to 75.4%. This is because the mechanism enhances the ability to distinguish between text and background, which is crucial to ensure that the text structure is not destroyed during background reconstruction. Without it, the network needs to solve the context capture problem and process local information at the same time in the following ways, which increases the difficulty of feature learning and ultimately generates low-quality images. Further, the embodiment of the present invention explores the influence of the gradient setting between modules on the model. When the gradient blocking between the text background separation module and the decoupled text transfer module is cancelled, that is, GB1 is cancelled, as shown in Table 1 w / GB1, RecAcc drops from 78.8% to 0.3%, and FID increases from 24.67 to 58.52. This degradation is due to the loss function of the decoupled text transfer weakening the optimization of text background separation. Text background separation needs to focus on learning the source text features to achieve accurate text segmentation, while decoupled text transfer focuses on the transmission of text style. The gradient blocking between text background separation and decoupled text transfer helps the text background separation network find its optimal optimization direction. When the gradient between the decoupled text transfer module and the background reconstruction module is removed, that is, GB2 is cancelled. As shown in Table 1 w / GB2, although MSE, SSIM, PSNR and FID did not reach the highest value, RecAcc obtained the highest score of 84.0%. This observation shows that the unblocked gradient contributes to text integrity, but also affects image quality. This effect can be attributed to the reconstruction process that prioritizes enhancing text regions, resulting in more complete text content and clearer strokes. Finally, by removing all gradient blockages in the network, i.e., canceling GB1 and GB2 simultaneously, poor results are observed in both the visualization and quantitative analysis of scene text editing. As shown in Table 1, the MSE, PSNR, SSIM, and FID scores are the worst. The main reason for this problem is that the decoupled text transfer and background reconstruction loss function also optimizes text-background separation. However, this optimization interferes with the specific optimization of text-background separation itself. Therefore, this interference confuses the optimization direction, resulting in inaccurate text region positioning, and ultimately failing to achieve the expected editing results.

[0197] Table 1: Validation of the proposed modules

[0198]

[0199]

[0200] 2) Performance comparison:

[0201] Table 2 shows that the method of the present invention achieves state-of-the-art performance, with an MSE of 0.0123, a PSNR of 20.83, an SSIM of 0.7210, a FID of 24.67, and a RecAcc of 78.8%. Specifically, compared with Pix2Pix, the model of the embodiment of the present invention significantly improves by 139.57 and 60.4% in FID and RecAcc, respectively. This is because Pix2Pix is ​​a style transfer method designed for the entire image, while the method of the embodiment of the present invention only focuses on the text area, which means that the scene text editing task can be completed in a simpler way. In addition, compared with SRNet, the model of the embodiment of the present invention achieves a significant improvement of 16.59% and 46.5% in FID and RecAcc, respectively. This is because SRNet's text editing is difficult to adapt to the background size, often resulting in poor image quality and text blur due to background scaling. In addition, compared with SwapNet, the model of the embodiment of the present invention achieves an improvement of 0.0686% and 18.17% in SSIM and RecAcc, respectively. This is because the embodiment of the present invention avoids the influence of complex background on text editing, enabling the model to generate higher quality text. Compared with DARLING, the model of the embodiment of the present invention achieved a 19.81% improvement in FID and a 7.9% improvement in RecAcc. The relatively poor performance of DARLING can be attributed to its insufficient feature learning, which is due to the lack of clear decoupling between text style and text content features, which easily leads to incomplete generation of target text. Finally, compared with MOSTEL, the model of the embodiment of the present invention improved by 4.81 and 2.00% in FID and RecAcc, respectively. The adopted scene text element separation strategy enhances the delicacy of the model editing results and makes the text content clearer and more complete. The qualitative editing results are shown in Fig.\ref{Fig6}, and it can be observed that the method of the embodiment of the present invention has the best visual experience.

[0202] Table 2: Performance comparison with various text editing methods

[0203]

[0204] In summary, compared with the prior art, the present invention has the following advantages:

[0205] 1. The present invention proposes an end-to-end scene text editing method to ensure effective scene text editing.

[0206] 2. The present invention separates the foreground source text from the background, thereby reducing the impact of the background on text editing.

[0207] 3. The present invention decouples the style attributes of the text into geometric and stylistic attributes, and uses style-guided text migration to align the target text with the source text in terms of geometric and stylistic attributes.

[0208] 4. The present invention utilizes the target text to replace the source text to ensure that the reconstructed background area is kept to a minimum, thereby reducing the complexity of the reconstruction process.

[0209] Another aspect of the embodiment of the present invention further provides a scene text editing system, including:

[0210] The first module is used to input the scene text image into the text-background separation module, predict the text foreground mask of the scene text image through a lightweight segmentation network, and separate the source text and the background image using the text foreground mask;

[0211] A second module is used to align the geometric and stylistic attributes of the content text with the source text through a style-guided text transfer decoding module and generate a target text;

[0212] A third module is used to combine the generated target text with the background image to determine a minimized background reconstruction area;

[0213] The fourth module is used to perform background reconstruction on the minimized background reconstruction area through a context-focused background reconstruction module to generate an edited complete scene text image.

[0214] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0215] The embodiment of the present invention further provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above scene text editing method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0216] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0217] See also Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0218] The processor 401 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;

[0219] The memory 402 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 402 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 402, and the processor 401 calls and executes the scene text editing method of the embodiment of the present invention;

[0220] Input / output interface 403, used to implement information input and output;

[0221] Communication interface 404, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0222] Bus 405 , which transmits information between various components of the device (e.g., processor 401 , memory 402 , input / output interface 403 , and communication interface 404 );

[0223] The processor 401 , the memory 402 , the input / output interface 403 and the communication interface 404 are connected to each other in communication within the device via the bus 405 .

[0224] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned scene text editing method is implemented.

[0225] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0226] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0227] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0228] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0229] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0230] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0231] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0232] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0233] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0234] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0235] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0236] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0237] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A scene text editing method, characterized in that: The following steps are involved: Inputting the scene text image into the text-background separation module, predicting the text foreground mask of the scene text image through a lightweight segmentation network, and using the text foreground mask to separate the source text and the background image; Aligning geometric and stylistic properties of the content text with the source text and generating a target text through a style-guided text transfer decoding module; Combining the generated target text with the background image to determine a minimized background reconstruction area; The minimized background reconstruction area is reconstructed by a context-focused background reconstruction module to generate an edited complete scene text image.

2. A scene text editing method according to claim 1, characterized in that: The style-guided text migration decoding module aligns the geometric and stylistic attributes of the content text with the source text and generates a target text, including the following steps: Extract the geometric features of the source text through a shallow convolutional network, and predict the spatial layout anchor points of the source text through a fully connected layer. The spatial layout anchor points are used to represent the geometric structure of the source text. Based on the spatial layout anchor points, a thin template interpolation algorithm is introduced to precisely adjust the geometric properties of the content text to match the geometric structure of the source text; The source text and content text are respectively encoded by the content encoding network and the style encoding network to obtain text content features and text style features; Through the adaptive instance normalization layer, the style features are injected into the content features to achieve dynamic adjustment of the style representation to the content representation, thereby generating a target text consistent with the style of the source text.

3. A scene text editing method according to claim 1, characterized in that: The step of combining the generated target text with the background image to determine a minimized background reconstruction area comprises the following steps: After obtaining the target text, the target text is replaced with the source text in the scene text image to determine the smallest background repair area.

4. A scene text editing method according to claim 1, characterized in that: The background reconstruction module using context focusing performs background reconstruction on the minimized background reconstruction area to generate an edited complete scene text image, comprising the following steps: After text replacement, the image is encoded using a reconstruction encoder and the encoding result is input into the residual module to enhance the expression of background details and structures; Use the self-attention module to enhance the network’s ability to capture contextual information and ensure the accuracy and consistency of background reconstruction; The decoder generates a complete scene text editing image based on the enhanced features.

5. A scene text editing method according to claim 1, characterized in that: The method further comprises the step of training a multi-task model, the step comprising: Decoupling the source text and background of the scene text image by using a text-background segmentation module, wherein the input of the text-background segmentation module is the scene text image, and the output is the source text and the background image; The geometric attributes and stylistic attributes of the source text are transferred to the content text through the text migration module to obtain the target text; The target text is embedded into the text-free scene background, and the incomplete image is encoded using a reconstruction encoder, which is then input into the residual module to effectively enhance the expression of background details and structures. According to the overall loss function of the entire multi-task model, the training and learning of each module in the multi-task model is completed.

6. A scene text editing method according to claim 5, characterized in that: The method of decoupling the source text and background of the scene text image by using a text background segmentation module, wherein the input of the text background segmentation module is the scene text image and the output is the source text and the background image, comprises the following steps: The scene text image I s Input to the lightweight segmentation network G s The mask M of the predicted source text s , then the text and background are separated by mask, and the expression of the text and background separation is: Among them, I sf Represents the source text; I sb Represents a background image.

7. A scene text editing method according to claim 5, characterized in that: The method of transferring the geometric attributes and stylistic attributes of the source text to the content text through the text migration module to obtain the target text includes the following steps: Using the shallow CNN network G g Extract source text I sf The features are then passed through the fully connected layer F c Predict N position anchor points A that can represent the spatial layout of the source text n , the expression of this process is: A n =F c (G g (I sf ; Θ g );Θ fc ), where Θ g and θ fc Respectively represent the network G g and network layer F c The learnable parameters in ; After obtaining the target position anchor point, the TPS algorithm is used to analyze the content text I t Perform geometric style alignment. The expression of this process is: in It is the content text after geometric alignment; Through the style network G s Source Text I sf The stylistic features of the source text are encoded, and the encoded features are then passed through a multi-layer perceptron to construct the stylistic style representation F of the source text. s , the expression of this process is: F s =MLP(G s (I sf ; Θ s );Θ mlp ), where Θ s and θ mlp Respectively represent the network G s and learnable parameters in the MLP layers; Content Text Through the content network G c After processing, we get the content representation F t : where Θ c Represents content network G c The learnable parameters in ; Introducing AdaIN-based style decoder and To generate scene text fonts of any style, the two decoders share the same network structure, including two residual modules and two upsampling modules, and all normalization layers in the decoder use AdaIN; Calculate the mean value m of the style representation s and variance and the mean value m of the content representation t and variance The expression of this process is: Wherein, H represents the height of the input text representation or content representation; W represents the width of the input text representation or content representation; and denote the element at position (s, t) in the stylistic representation and content representation, respectively, and ∈ is a small constant for numerical stability; AdaIN is used to perform affine changes on the font style of the content text to achieve font migration. The expression of this process is: in, represents the content representation of the kth AdaIN layer; Text stylistic representation s can be gradually injected into the content representation F during the decoding process through the K AdaIN layers in the decoder t The expression of this process is: Among them, O t represents the target text; represents the target text decoder; M t Represents the target text corresponding mask; represents the target text mask decoder; and denote the learnable parameters of the target text decoder and mask decoder respectively.

8. A scene text editing method according to claim 5, characterized in that: The overall loss function includes text-background separation loss, style transfer loss, mask loss, and background reconstruction loss; Among them, the text background separation loss The expression is: Among them, T s It is with M s The corresponding truth value, ∩, represents the intersection operation on pixels; Style transmission loss The expression is: Among them, β1 represents the balance factor; T o Yes and O t The corresponding truth value; φ i Represents feature maps from VGG-19; Mask loss The expression is: Among them, T t For M t Corresponding to the true value, D t I is the discriminator; t represents the content text; γ1 and γ2 are balance factors; Background reconstruction loss The expression is: Among them, ξ1 is the balance factor; T f Yes and O f The corresponding truth value.

9. A scene text editing system, characterized in that: include: The first module is used to input the scene text image into the text-background separation module, predict the text foreground mask of the scene text image through a lightweight segmentation network, and separate the source text and the background image using the text foreground mask; A second module is used to align the geometric and stylistic attributes of the content text with the source text through a style-guided text transfer decoding module and generate a target text; A third module is used to combine the generated target text with the background image to determine a minimized background reconstruction area; The fourth module is used to perform background reconstruction on the minimized background reconstruction area through a context-focused background reconstruction module to generate an edited complete scene text image.

10. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 8.

Images