Stroke-level guided diffusion model based method and system for text editing in complex scenes

Abstract

The application provides a complex scene text editing method and system based on stroke level guided diffusion model, and belongs to the technical field of computer vision image generation. The application promotes network learning text conversion rules by using stroke level guidance, and further limits the text area by using enhanced text information, so that significant improvement is achieved in terms of text accuracy and style. The application proposes a background reconstruction module based on global information, reconstructs the background by using whole scene information, and fine tunes the background reconstruction module by using a real world data set. The application improves the accuracy of text generation by using stroke level guided information, completes the background reconstruction task by using the information of the whole scene image, can more accurately generate target text, and improves the effect of reconstructing complex background.

Description

Technical Field

[0001] This invention relates to the field of computer vision image generation technology, and in particular to a method and system for editing text in complex scenes based on a stroke-level guided diffusion model. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Scene text is ubiquitous in daily life, appearing on billboards, road signs, storefronts, product labels, and more. It carries rich semantic information, and the ability to manipulate and edit scene text in images holds significant promise and has wide-ranging practical applications in both the real world and multimedia applications. Text in images often needs editing to correct errors, update information, or adapt it to different contexts. For example, seamlessly replacing text or product labels in existing advertising images. Furthermore, in augmented reality (AR) applications, real-time text translation and replacement are essential for enhancing user experience, making scene text editing tasks applicable across a wide range of scenarios.

[0004] The inventors discovered that traditional scene text editing methods cannot fully address the unique challenges posed by scene text, such as font variations, size, orientation, and complex background textures. Scene text editing tasks mainly face two challenges: 1) accurately generating guide text content while extracting style information from scene images; 2) difficulty in reconstructing the background of complex scenes. In previous methods, the network had to learn how to reconstruct the background while learning the text style. This approach of implementing two tasks within a single module resulted in the network generating unrecognizable fonts and poor background reconstruction effects.

[0005] Furthermore, existing methods only focus on the background around the text and cannot utilize the background information of the entire scene image. When the edited image is merged with the entire scene image after it is generated, there are often boundary issues. At the same time, some traditional scene text editing methods use stroke-level modifications to improve this situation. They use a decoder to obtain the text region of the style image, fill the background, and retain the non-text region. However, for complex background images in real-world data, this method incorrectly identifies the text region as background and retains it, and does not learn the text style correctly, so it cannot generate the target text correctly. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method and system for editing text in complex scenes based on a stroke-level guided diffusion model. By using stroke-level guided information, the accuracy of text generation is improved. By utilizing information from the entire scene image to complete the background reconstruction task, the target text can be generated more accurately, thus improving the effect of reconstructing complex backgrounds.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for editing text in complex scenes based on a stroke-level guided diffusion model.

[0009] A method for text editing in complex scenarios based on a stroke-level guided diffusion model includes the following process:

[0010] Acquire the target text, the scene image to be edited, and the scene mask image of the editing area;

[0011] Obtain the target text mask image based on the target text;

[0012] The target text is encoded into word feature vectors, and the target text mask image is noise-added as stroke-level guiding noise. The text style image is obtained based on the scene image and the scene mask image.

[0013] Based on word feature vectors, stroke-level guided noise, and text style images, a stylized text image and a stylized text mask of the target text are generated after denoising.

[0014] The background reconstruction image is obtained based on the scene image, the scene mask image, and the background reconstruction diffusion model;

[0015] Based on the location information of the scene mask image, the stylized text image is made transparent using a stylized text mask. The reconstructed background image and the stylized text image of the transparent target text are then combined to obtain the edited image.

[0016] As a further limitation of the first aspect of the present invention, the target text is input in the form of text, and the scene image to be edited and the scene mask image of the editing area are the entire scene image.

[0017] As a further limitation of the first aspect of the present invention, encoding the target text into a word feature vector includes: encoding each letter into a number, and encoding a word of length N into an N-dimensional number as a word feature vector, which is then injected into the Unet network of the diffusion model using a cross-attention mechanism.

[0018] As a further limitation of the first aspect of the present invention, obtaining a text style image based on a scene image and a scene mask image includes:

[0019] The scene image is cropped based on the scene mask image, and the part not covered by the mask is cropped out as the text style image.

[0020] As a further limitation of the first aspect of the present invention, word feature vectors, stroke-level guiding noise and text style images are input into a stroke-level guided text conversion model, and after denoising, a stylized text image and a stylized text mask of the target text are generated.

[0021] During the training process of the stroke-level guided text conversion model, the model learns the stylized image of the target text and uses the stylized mask of the target text as stroke-level guiding information.

[0022] As a further limitation of the first aspect of the present invention, the reconstructed scene image is obtained based on the scene image, the scene mask image, and the background reconstruction diffusion model, including:

[0023] The scene image and the mask image are combined into a masked scene image. The scene image, the mask image, and the masked scene image are then processed into low-dimensional vectors using a variational autoencoder.

[0024] The acquired scene image vector, mask image vector, and masked scene image vector are input into the background reconstruction diffusion model to obtain the corresponding background reconstruction image.

[0025] As a further limitation of the first aspect of the present invention, based on the positional information of the mask image reconstructed from the background, a stylized text image is made transparent using a stylized text mask, and the reconstructed background image and the stylized text image of the transparent target text are combined to obtain an edited image, including:

[0026] Obtain the size of the style text image by reconstructing the mask image from the background, adjust the style text image and style text mask to the same size as the area to be edited, and record the coordinates of the area to be edited.

[0027] Based on the coordinates of the area to be edited, the adjusted style text image and style text mask, and the background reconstruction image, the style text image is positioned to the corresponding area to be edited in the background reconstruction image using the coordinates. The text is then made transparent using the style text mask and pasted onto the background reconstruction image to obtain the final edited image.

[0028] Secondly, the present invention provides a complex scene text editing system based on a stroke-level guided diffusion model.

[0029] A complex scene text editing system based on a stroke-level guided diffusion model includes:

[0030] The data acquisition module is configured to acquire the target text, the scene image to be edited, and the scene mask image of the editing area;

[0031] The target text mask image generation module is configured to: obtain a target text mask image based on the target text;

[0032] The text style image generation module is configured to: encode the target text into word feature vectors, add noise to the target text mask image as stroke-level guiding noise, and obtain the text style image based on the scene image and the scene mask image;

[0033] The stroke-level guided text conversion module is configured to generate a stylized text image and a stylized text mask of the target text after denoising based on word feature vectors, stroke-level guided noise, and text style images.

[0034] The global information background reconstruction module is configured to obtain a background reconstruction image based on the scene image, the scene mask image, and the background reconstruction diffusion model.

[0035] The fusion module is configured to: based on the position information of the scene mask image, use a stylized text mask to make the stylized text image transparent, and combine the background reconstructed image and the stylized text image of the transparent target text to obtain the edited image.

[0036] Thirdly, the present invention provides a complex scene text editing system based on a stroke-level guided diffusion model.

[0037] A complex scene text editing system based on a stroke-level guided diffusion model includes: a stroke-level guided text conversion module, a global information background reconstruction module, and a fusion module;

[0038] The stroke-level guided text conversion module includes a first data acquisition unit, a first data preprocessing unit, and a font generation unit;

[0039] The first data acquisition unit is configured to: acquire the target text, the style image, and the mask image of the standard font of the target text;

[0040] The first data preprocessing unit is configured to: add noise to the mask image of the target text standard font by different levels of noise addition, and obtain the text encoding vector of the target text through an encoding strategy;

[0041] The font generation unit is configured to input the acquired text encoding vector, the noise-added mask image, and the style image into the stroke-guided text conversion diffusion model to obtain the corresponding target style text image and style text mask.

[0042] The global information background reconstruction module includes a second data acquisition unit, a second data preprocessing unit, and a background reconstruction unit;

[0043] The second data acquisition unit is configured to acquire the scene image and the mask image of the area to be edited in the scene image;

[0044] The second data preprocessing unit is configured to: combine the scene image and the mask image into a masked scene image, and process the scene image, the mask image, and the masked scene image into a low-dimensional vector through a variational autoencoder;

[0045] The background reconstruction unit is configured to input the acquired scene image vector, mask image vector, and masked scene image vector into the background reconstruction diffusion model to obtain the corresponding background reconstruction image.

[0046] The fusion module includes a third data acquisition unit, a third data preprocessing unit, and a fusion unit;

[0047] The third data acquisition unit is configured to: acquire the style text image and style text mask generated by the text conversion module, and acquire the background reconstruction image output by the global information background reconstruction module;

[0048] The third data preprocessing unit is configured to: obtain the size of the required style text image through the scene mask image, adjust the style text image and style text mask to the same size as the area to be edited, and record the coordinates of the area to be edited.

[0049] The fusion unit is configured to: acquire the coordinates of the area to be edited, the adjusted style text image and style text mask, and the background reconstructed image; locate the style text image to the corresponding area to be edited in the background reconstructed image using the coordinates; and make the text transparent using the style text mask and paste it into the background reconstructed image to obtain the final edited image.

[0050] Fourthly, the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the complex scene text editing method based on a stroke-level guided diffusion model as described in the first aspect of the present invention.

[0051] Fifthly, the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the complex scene text editing method based on the stroke-level guided diffusion model as described in the first aspect of the present invention.

[0052] Compared with the prior art, the beneficial effects of the present invention are:

[0053] 1. This invention innovatively proposes a complex scene text editing strategy based on a stroke-level guided diffusion model. By performing scene text editing based on the diffusion model and decoupling text and background, it can effectively solve the problems of low text accuracy and poor reconstruction quality of complex backgrounds in scene text editing.

[0054] 2. This invention innovatively proposes a stroke-level guided text conversion strategy and a background reconstruction strategy based on global information. It realizes complex background reconstruction based on complete scene image information, solving the problem of poor background reconstruction effect in complex scenes.

[0055] 3. This invention has superior performance in terms of text accuracy, style correctness, and naturalness of integration, especially in the degree of restoration of complex background textures in real datasets. It can generate target text more accurately and improve the effect of reconstructing complex backgrounds.

[0056] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0057] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0058] Figure 1 This is a flowchart of a stroke-level guided diffusion model for text editing tasks in complex scenarios, provided in Embodiment 1 of the present invention.

[0059] Figure 2 A brief comparison of the results of the stroke-level guided diffusion model for complex text editing tasks provided in Embodiment 1 of the present invention;

[0060] Figure 3 This is a diagram illustrating the effect of the method provided in Embodiment 1 of the present invention;

[0061] Figure 4 This is a comparison chart of the qualitative results of the method provided in Embodiment 1 of the present invention and existing methods;

[0062] Figure 5 This is a comparison diagram of the qualitative results of the method provided in Embodiment 1 of the present invention and existing methods for complex scenarios;

[0063] Figure 6 This is a network structure diagram of the stroke-level guided diffusion model for complex text editing tasks provided in Embodiment 3 of the present invention. Detailed Implementation

[0064] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0065] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0066] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0067] Example 1:

[0068] like Figure 1 As shown, Embodiment 1 of the present invention provides a method for editing text in complex scenes based on a stroke-level guided diffusion model, which includes the following process:

[0069] S1: Obtain the target text, the scene image to be edited, and the scene mask image of the editing area;

[0070] S2: Obtain the target text mask image based on the target text;

[0071] S3: Encode the target text into word feature vectors based on the target text, add noise to the target text mask as stroke-level guiding noise, and obtain the text style image based on the scene image and the scene mask image;

[0072] S4: Input the word feature vector, stroke-level guiding noise and text style image into the stroke-based text conversion module to denoise and generate a stylized text image and stylized text mask of the target text;

[0073] S5: Use the scene image and scene mask image to obtain low-dimensional scene image vectors and scene mask image vectors through a variational autoencoder;

[0074] S6: The scene image vector and the scene mask image are denoised and de-denoised by the Unet network of the diffusion model to obtain the reconstructed scene image;

[0075] S7: Based on the position information of the scene mask image, the stylized text is made transparent using a stylized text mask. The reconstructed scene image and the stylized text image of the target text are combined to obtain the edited image.

[0076] Specifically, S1 includes:

[0077] The target text is input in text form. The scene image to be edited and the scene mask image of the editing area are both the entire scene image, not just a partial image around the text.

[0078] Specifically, S2 includes:

[0079] Use the text synthesis model in SRnet to generate a mask image of the target text in the standard font (ARIAL).

[0080] Specifically, in S3, this includes:

[0081] S3.1: In this invention, each letter corresponds to a number, and a word of length N is encoded as an N-dimensional number as a word feature vector. Then, it is injected into the Unet of the diffusion model through a cross-attention mechanism, thereby improving the effectiveness of text guidance.

[0082] S3.2: Randomly add noise to the target text mask image obtained in step 2 using the following formula:

[0083]

[0084] S3.3: Crop the scene image based on the scene mask image, and crop out the part not covered by the mask as the text style image.

[0085] Specifically, in S4, this includes:

[0086] This invention proposes a stroke-level guided text conversion module, which can eliminate background information, allowing the model to focus on text conversion and further improve the accuracy and style of the text. Stroke-level guiding information m_t is incorporated into the model training process to enhance the distinction between text and background, thereby improving the accuracy of text generation. This addition enables the model to prioritize generating precise glyphs for the target text, thus aligning it more accurately.

[0087] Given a set of input data (y, x) text Given x0, m0 and a random integer t from (0, t), where t represents the maximum number of diffusion steps, noise is added to x0 and m0 according to the following formula:

[0088]

[0089] Where, β t Let represent the manually specified hyperparameter sequence, ∈ be random noise, and N(0, I) be a vector sampled from a standard normal distribution. Due to the continuity of the Gaussian distribution, this invention can obtain (x) by adding noise in one step using the following formula. t m t ):

[0090]

[0091] In the reverse process, the model accepts input (y, x). text x t m t ), and output the original (x0, m0) or the noise added to it ∈t The predicted noise is determined by the style of the style image y, and the input text x. text and real image (x t m t The output prediction noise of the model can be obtained using the following formula:

[0092]

[0093] The model's loss function is:

[0094]

[0095] In S5, specifically, this includes:

[0096] The scene image and scene mask image used are 512*512*3 and 512*512*1 in size, respectively. After passing through a variational autoencoder, they are mapped to the latent space to learn the implicit representation of the image. The variational autoencoder is pre-trained with fixed parameters. Given a set of input data (x, mask, text), this invention can obtain the masked image by superimposing the mask onto x. The variational autoencoder ε v Transform (x, mask, masked) into an implicit vector (z). x , z m , z md ).

[0097] In S6, specifically, this includes:

[0098] Compared to previous methods that only reconstruct the background around the text, the model proposed in this invention adopts a holistic perspective for image reconstruction and is trained using real-world data to enhance the naturalness and harmony of the reconstructed real-world scenes. By utilizing information from the entire image, this model is also well-suited for complex background reconstruction tasks. The model uses StableDiffusion2 as its base model, trained on the LION-5B dataset composed of natural images, demonstrating its applicability across various scenarios. However, due to the use of clip as the text processor and the text-object pairing training process for the entire model, it lacks understanding of command-type input text prompts, such as "backgroundreconstruction" or "background inpaint". Therefore, different types of backgrounds require different text prompts: "blue background", "brown wall", and "leaf". The goal of this invention is to achieve unified background reconstruction by using fixed text prompts capable of handling different types and textures. To address this issue, this invention fine-tunes SD2 so that it can perform background reconstruction tasks with standard text input (such as "a photo of nothing").

[0099] In the latent space, to process text cues, StableDiffusion2 uses the FrozenCLIPEmbedder text encoder to encode the text cues into corresponding text vectors c; here, the text cues are uniformly planned as "a photo of nothing" in this invention, and through the attention mechanism (Q, K, V), c is mapped to the intermediate layer of UNet:

[0100]

[0101] The loss function for the entire StableDiffusion2 training process is:

[0102]

[0103] In S7, specifically, this includes:

[0104] S7.1: Read the scene mask image and obtain the coordinates of the top left and bottom right corners of the area to be edited;

[0105] S7.2: Obtain the height and width of the area to be edited based on the coordinates of the upper left corner and the lower right corner, and adjust the height and width of the style text image and style text code generated by the stroke-based text conversion module to be the same as the area to be edited;

[0106] S7.3: Separate the data of the three channels of the style text image, use the three channels of the style text image as the first three channels of the image, and use the corresponding style text mask as the fourth channel, and merge them into an image with transparency;

[0107] S7.4: The background reconstructed image is processed using a scene mask image. For areas where the mask value is 0, the original pixels of the background reconstructed image are used directly. For areas where the mask value is not 0, i.e., the areas to be edited, the original pixels of the background reconstructed image and the pixels of the transparent style text image are weighted and merged. The merging formula is as follows:

[0108] x text [0:3]*x text [3]+x bc *(1-x text [3]) (8)

[0109] The overall fusion formula is as follows:

[0110] x output =x text [0:3]*(m0+x text [3])+(1-(m0+x text [3]))*x bc (9)

[0111] Figure 2 This indicates that the MOSTEL model incorrectly identifies text regions as background and fails to generate accurate correspondences. Although the LETTER model successfully generates text content, the reconstructed solid-color background lacks texture, and the text style is significantly different. While SD2 ensures a natural background, it cannot generate sufficiently accurate text. The model of this invention can best restore background texture while ensuring the accuracy and correct style of the target text.

[0112] Figure 4 The model of the present invention is shown to edit images in different scenes. It can be seen that the model of the present invention has learned the style features of the original very well, and the reconstructed lines and textures of the background can also be handled very well.

[0113] Figure 5The model of this invention is compared with other models on the ICDAR2013, COCOText, TextOCR, and SVT datasets, demonstrating the performance of the model. From the comparison of the ICDAR2013 dataset, it can be seen that STEFANN, MOSTEL, and SD2 did not generate correct text, while SRnet and LETTER correctly generated some text, but there was a clear sense of boundary between the background and the text image of the whole scene. The situation is similar for other datasets. The model of this invention is significantly better than other models in terms of text accuracy and background reconstruction effect.

[0114] exist Figure 6 To better evaluate the effectiveness of the model in editing complex real-world datasets, this invention specifically tested it on text editing data in complex scenes. The experiments covered various real-world scenes, including walls, billboards with grainy textures, and wooden surfaces. However, SRnet and STEFANN models trained solely on the generated datasets struggled to accurately identify text embedded in complex backgrounds; they frequently misinterpreted text as part of the background itself, thus failing to effectively extract crucial text style information, such as... Figure 6 The second and third columns. Therefore, the line details in the background become blurred against the surrounding colors, resulting in an unnatural and discordant outcome when pasted back into the original image, for example... Figure 6 In the third row and fourth column, other models show a significant decrease in text generation accuracy and background reconstruction effect under complex backgrounds, while the model of this invention maintains a relatively good text accuracy and background reconstruction effect.

[0115] Example 2:

[0116] Embodiment 2 of the present invention provides a complex scene text editing system based on a stroke-level guided diffusion model, comprising:

[0117] The data acquisition module is configured to acquire the target text, the scene image to be edited, and the scene mask image of the editing area;

[0118] The target text mask image generation module is configured to: obtain a target text mask image based on the target text;

[0119] The text style image generation module is configured to: encode the target text into word feature vectors, add noise to the target text mask image as stroke-level guiding noise, and obtain the text style image based on the scene image and the scene mask image;

[0120] The stroke-level guided text conversion module is configured to generate a stylized text image and a stylized text mask of the target text after denoising based on word feature vectors, stroke-level guided noise, and text style images.

[0121] The global information background reconstruction module is configured to obtain a background reconstruction image based on the scene image, the scene mask image, and the background reconstruction diffusion model.

[0122] The fusion module is configured to: based on the position information of the scene mask image, use a stylized text mask to make the stylized text image transparent, and combine the background reconstructed image and the stylized text image of the transparent target text to obtain the edited image.

[0123] The specific working methods of each module of the system are described in Embodiment 1, and will not be repeated here.

[0124] Example 3:

[0125] like Figure 3 As shown, Embodiment 3 of the present invention constructs a stroke-level guided diffusion model for text editing tasks in complex scenarios, which is divided into three parts: a stroke-level guided text conversion module, a global information background reconstruction module, and a fusion module.

[0126] The stroke-level guided text conversion module's main task is to accurately extract the style information of the text to be edited using stroke-level guidance information, and generate the target text and its mask image for this style. The global information background reconstruction module's main task is to erase the text in the editing area and reconstruct the background using the background information of the entire scene graphics. The fusion module merges the generated text image, text mask, and reconstructed background image into the final result.

[0127] For stroke-level guided text conversion modules, most studies treat it as a text style transfer task and use generative models like GANs for processing. However, since GANs are a one-step generation method, they cannot fully extract information from style images and text content, resulting in some style differences. Diffusion models can inject guiding information at each step of the denoising process, greatly improving the utilization rate of guiding information.

[0128] Building upon a diffusion model that encodes text as letter vectors, this invention incorporates stroke-level guidance to remove background information, enabling the model to focus on text conversion and further improve text accuracy and style. This enhanced model is termed the stroke-level guided text conversion module. Building upon previous work, this module introduces stroke-level guidance information and stroke-level guidance noise to enhance text generation. During training, stroke-level guidance information is introduced to constrain text regions and separate them from the background, allowing the model to focus on generating high-quality text while enhancing its understanding of font style and texture. Simultaneously, the output stroke-level guidance information is used in the fusion module to easily extract the generated text regions and directly fuse them with the desired background. To further enhance attention to font details during sampling, an additional component, stroke-level guidance noise, is introduced. Instead of starting with random black and white noise from the stroke-level guidance information region, a mask image of the standard font is used with added noise as the denoising starting point. This method effectively improves glyph utilization and text accuracy.

[0129] In the global information background reconstruction module, the style information of the entire scene can be used to fill the background that needs to be reconstructed. This invention uses the pre-trained SD2 model as the baseline of this invention. It is mainly trained on the LION-5B dataset. It pays more attention to natural objects and scenes and can make full use of the entire scene through reasonable pixel filling. This invention uses real-world data for fine-tuning to maximize the background reconstruction of the real scene.

[0130] In the fusion module, no additional network is needed for learning. Instead, the separation of the pure font from the background is achieved directly using a mask with stroke-level guiding information, and the font is seamlessly integrated into the reconstructed background using coordinate information. Since the background reconstruction target of this invention is the entire scene image, the generated local image has no sense of boundary.

[0131] More specifically, the stroke-level guided text conversion module is divided into three parts:

[0132] The data acquisition unit is configured to acquire the target text, style image, and mask image of the target text's standard font.

[0133] The data preprocessing unit is configured to: add noise to the mask image of the target text standard font by different levels of noise addition, and obtain the text encoding vector of the target text through an encoding strategy;

[0134] The font generation unit is configured to input the acquired text encoding vector, the noise-added mask image, and the style image into a stroke-guided text transformation diffusion model to obtain the corresponding target style text image and style text mask.

[0135] The global information background reconstruction module is divided into three parts:

[0136] The data acquisition unit is configured to acquire a scene image and a mask image of the area of ​​the scene image to be edited.

[0137] The data preprocessing unit is configured to: synthesize the scene image and the mask image into a masked scene image, and process the scene image, the mask image, and the masked scene image into low-dimensional vectors through a variational autoencoder;

[0138] The background reconstruction unit is configured to input the acquired scene image vector, mask image vector, and masked scene image vector into the background reconstruction diffusion model to obtain the corresponding background reconstruction image.

[0139] The fusion module is divided into three parts:

[0140] The third data acquisition unit is configured to: acquire the style text image and style text mask generated by the text conversion module, and acquire the background reconstruction image output by the global information background reconstruction module;

[0141] The third data preprocessing unit is configured to: obtain the size of the required style text image through the scene mask image, adjust the style text image and style text mask to the same size as the area to be edited, and record the coordinates of the area to be edited.

[0142] The fusion unit is configured to: acquire the coordinates of the area to be edited, the adjusted style text image and style text mask, and the background reconstructed image; locate the style text image to the corresponding area to be edited in the background reconstructed image using the coordinates; and make the text transparent using the style text mask and paste it into the background reconstructed image to obtain the final edited image.

[0143] Example 4:

[0144] Embodiment 4 of the present invention provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the steps in the complex scene text editing method based on the stroke-level guided diffusion model described in Embodiment 1 of the present invention.

[0145] Example 5:

[0146] Embodiment 5 of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the complex scene text editing method based on the stroke-level guided diffusion model described in Embodiment 1 of the present invention.

[0147] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for editing text in complex scenes based on a stroke-level guided diffusion model, characterized in that, The process includes the following: Acquire the target text, the scene image to be edited, and the scene mask image of the editing area; Obtain the target text mask image based on the target text; The target text is encoded into word feature vectors, and the target text mask image is noise-added as stroke-level guiding noise. The text style image is obtained based on the scene image and the scene mask image. Based on word feature vectors, stroke-level guided noise, and text style images, a stylized text image and a stylized text mask of the target text are generated after denoising. The background reconstruction image is obtained based on the scene image, the scene mask image, and the background reconstruction diffusion model; Based on the positional information of the scene mask image, the stylized text image is made transparent using a stylized text mask. The background reconstructed image and the stylized text image of the transparent target text are then combined to obtain the edited image. The word feature vectors, stroke-level guiding noise, and text style images are input into the stroke-level guided text conversion model. After denoising, the model generates a stylized text image and a stylized text mask of the target text. During the training process of the stroke-level guided text conversion model, the stylized image of the target text is learned, and the stylized mask of the target text is used as the stroke-level guiding information. Given a set of input data And a random integer t from (0, t), where t represents the maximum number of steps of diffusion, according to the following formula. and Add noise: ； in, This indicates a manually specified sequence of hyperparameters. It is random noise, a vector sampled from a standard normal distribution. Due to the continuity of the Gaussian distribution, this invention obtains the following formula through a one-step noise addition. : ； During the reverse process, the model receives input. and output the original Or noise added to it The noise is predicted by the style of the style image y, and the input text is given. and real images The control model's output prediction noise is obtained using the following formula: ； The model's loss function is: 。 2. The complex scene text editing method based on the stroke-level guided diffusion model as described in claim 1, characterized in that, The target text is input in text form, and the scene image to be edited and the scene mask image of the editing area are the entire scene image; The target text is encoded into word feature vectors, including: each letter is assigned a number, and a word of length N is encoded into an N-dimensional number as a word feature vector, which is then injected into the Unet network of the diffusion model using a cross-attention mechanism.

3. The complex scene text editing method based on the stroke-level guided diffusion model as described in claim 1, characterized in that, The text style image is obtained from the scene image and the scene mask image, including: The scene image is cropped based on the scene mask image, and the part not covered by the mask is cropped out as the text style image.

4. The complex scene text editing method based on the stroke-level guided diffusion model as described in claim 1, characterized in that, The reconstructed scene image is obtained based on the scene image, scene mask image, and background reconstruction diffusion model, including: The scene image and the mask image are combined into a masked scene image. The scene image, the mask image, and the masked scene image are then processed into low-dimensional vectors using a variational autoencoder. The acquired scene image vector, mask image vector, and masked scene image vector are input into the background reconstruction diffusion model to obtain the corresponding background reconstruction image.

5. The complex scene text editing method based on the stroke-level guided diffusion model as described in claim 1, characterized in that, Based on the positional information of the reconstructed mask image from the background, a stylized text image is made transparent using a stylized text mask. The reconstructed background image and the stylized text image of the transparent target text are then combined to obtain the edited image, including: Obtain the size of the style text image by reconstructing the mask image from the background, adjust the style text image and style text mask to the same size as the area to be edited, and record the coordinates of the area to be edited. Based on the coordinates of the area to be edited, the adjusted style text image and style text mask, and the background reconstruction image, the style text image is positioned to the corresponding area to be edited in the background reconstruction image using the coordinates. The text is then made transparent using the style text mask and pasted onto the background reconstruction image to obtain the final edited image.

6. A complex scene text editing system based on a stroke-level guided diffusion model, characterized in that, The complex scene text editing method based on the stroke-level guided diffusion model according to any one of claims 1-5 includes: The data acquisition module is configured to acquire the target text, the scene image to be edited, and the scene mask image of the editing area; The target text mask image generation module is configured to: obtain a target text mask image based on the target text; The text style image generation unit is configured to: encode the target text into word feature vectors, add noise to the target text mask image as stroke-level guiding noise, and obtain a text style image based on the scene image and the scene mask image; The stroke-level guided text conversion module is configured to generate a stylized text image and a stylized text mask of the target text after denoising based on word feature vectors, stroke-level guided noise, and text style images. The global information background reconstruction module is configured to obtain a background reconstruction image based on the scene image, the scene mask image, and the background reconstruction diffusion model. The fusion module is configured to: based on the position information of the scene mask image, use a stylized text mask to make the stylized text image transparent, and combine the background reconstructed image and the stylized text image of the transparent target text to obtain the edited image.

7. A complex scene text editing system based on a stroke-level guided diffusion model, characterized in that, The complex scene text editing method based on the stroke-level guided diffusion model according to any one of claims 1-5 includes: a stroke-level guided text conversion module, a global information background reconstruction module, and a fusion module; The stroke-level guided text conversion module includes a first data acquisition unit, a first data preprocessing unit, and a font generation unit; The first data acquisition unit is configured to: acquire the target text, the style image, and the mask image of the standard font of the target text; The first data preprocessing unit is configured to: add noise to the mask image of the target text standard font by different levels of noise addition, and obtain the text encoding vector of the target text through an encoding strategy; The font generation unit is configured to input the acquired text encoding vector, the noise-added mask image, and the style image into the stroke-guided text conversion diffusion model to obtain the corresponding target style text image and style text mask. The global information background reconstruction module includes a second data acquisition unit, a second data preprocessing unit, and a background reconstruction unit; The second data acquisition unit is configured to acquire the scene image and the mask image of the area to be edited in the scene image; The second data preprocessing unit is configured to: combine the scene image and the mask image into a masked scene image, and process the scene image, the mask image, and the masked scene image into a low-dimensional vector through a variational autoencoder; The background reconstruction unit is configured to input the acquired scene image vector, mask image vector, and masked scene image vector into the background reconstruction diffusion model to obtain the corresponding background reconstruction image. The fusion module includes a third data acquisition unit, a third data preprocessing unit, and a fusion unit; The third data acquisition unit is configured to: acquire the style text image and style text mask generated by the text conversion module, and acquire the background reconstruction image output by the global information background reconstruction module; The third data preprocessing unit is configured to: obtain the size of the required style text image through the scene mask image, adjust the style text image and style text mask to the same size as the area to be edited, and record the coordinates of the area to be edited. The fusion unit is configured to: acquire the coordinates of the area to be edited, the adjusted style text image and style text mask, and the background reconstructed image; locate the style text image to the corresponding area to be edited in the background reconstructed image using the coordinates; and make the text transparent using the style text mask and paste it into the background reconstructed image to obtain the final edited image.

8. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by the processor, the program implements the steps in the complex scene text editing method based on the stroke-level guided diffusion model as described in any one of claims 1-5.

9. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the complex scene text editing method based on the stroke-level guided diffusion model as described in any one of claims 1-5.

Images