Method for generating image with scene characters

Abstract

The invention relates to the technical field of computers, in particular to a method for generating an image with scene characters. The method comprises the steps of obtaining text description; obtaining a background image, to-be-generated characters and a text area based on the text description; obtaining a local image based on the background image through an adaptive text block exploration strategy; obtaining a character image through the to-be-generated character; condition embedding is obtained based on the local image and the character image; establishing image guidance based on the text region; inputting the condition embedding and the image guidance into a convolutional neural network to obtain a decoded image; the character generation in the decoded image is determined by the character to be generated and the text area. The problem of small text generation during image generation is solved, the accuracy of text generation is improved, and the small text rendering effect during image generation is good.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method for generating an image with scene text. Background Art

[0002] The generation of diffusion model-based images has been a research hotspot in recent years, among which scene text image generation is a difficult point in image generation. Its task is to generate a scene text image that visually corresponds to the description based on a given text description and specified text. With the development of deep learning, this field has made rapid progress. The current mainstream methods generate text by optimizing the text encoder using the glyph encoder, optimizing the text encoder so that it can generate characters after encoding the text description, and using glyph conditions to strengthen text generation.

[0003] Existing methods do not have precise enough control over the generated text, including accuracy, quantity, and layout. Due to the inherent characteristics of conditional encoders, it is impossible to generate a large amount of text with accurate spelling and reasonable distribution in complex scene layouts. Existing methods do not have flexible enough control over the generated background. Users cannot guarantee that the generated scene does not contain unexpected text elements. Due to the inherent limitations of variational autoencoders (VAEs), they do not work well in detail generation when generating in latent space, which makes it impossible to generate small text. Summary of the invention

[0004] The purpose of the present invention is to design a new generation paradigm, which first generates a background image without text and then performs text rendering to solve the problem of inflexible scene text and background control, ensure that the generated image does not contain unexpected text elements, and enable the generated image to be applied to downstream detection tasks.

[0005] The method for generating an image with scene text comprises:

[0006] Get text description;

[0007] Based on the text description, a background image, text to be generated and a text area are obtained;

[0008] Based on the background image, a local image is obtained by using an adaptive text block exploration strategy; a text image is obtained by using the text to be generated;

[0009] Obtaining conditional embedding based on the local image and the text image; and establishing image guidance based on the text region;

[0010] The conditional embedding and the image guidance are input into a convolutional neural network to obtain a decoded image; the text generation in the decoded image is determined by the text to be generated and the text area.

[0011] Furthermore, after obtaining the background image, text erasure and quality assessment are performed.

[0012] Furthermore, the text area is generated by automatic selection, and the method of automatic selection includes:

[0013] Performing semantic segmentation on the background image;

[0014] Preserve semantically reasonable locations based on set reasonable categories;

[0015] Get the text area in the same way as SynthText.

[0016] Furthermore, the adaptive text block exploration strategy includes:

[0017] Convert the quadrilateral bounding box of the text region into an extended square region.

[0018] Furthermore, the side length of the square is determined by the following formula:

[0019]

[0020] Wherein, s is the side length of the square, and w and h are the width and height of the minimum enclosing rectangle corresponding to the quadrilateral bounding box.

[0021] Furthermore, the image guidance includes:

[0022] Latent vector of an image with masked text regions;

[0023] Word-level masking;

[0024] Character-level segmentation masks;

[0025] Latent vector of the noisy image.

[0026] Furthermore, a loss measurement is performed on the decoded image.

[0027] Furthermore, the loss metric includes:

[0028] Denoising loss, character-aware loss, and text recognition loss.

[0029] The beneficial effects of the present invention are as follows:

[0030] The present invention proposes a new paradigm for visual text generation, which can ensure that no unexpected text appears in the background. By using text images and established image guidance, the problem of small text generation during image generation is solved and the accuracy of text generation is improved. The present invention obtains local images based on an adaptive text block exploration strategy, which not only provides a high-resolution representation of fine-grained glyph features, but also provides sufficient background pixel prior information, avoiding increased computational complexity and visual inconsistency problems, so that small text rendering effects are excellent during image generation. Moreover, the scene text images generated by the present invention can be applied to downstream tasks of text detection, which can improve the performance of text detectors. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the present invention

[0032] Figure 2 This is a comparison of the pictures generated by the present invention and the existing method.

[0033] Figure 3 A comparison of the present invention and the prior art in terms of the ability to avoid irrelevant text DETAILED DESCRIPTION

[0034] This embodiment adopts a two-stage method for generating images with scene text, such as Figure 1 shown.

[0035] Phase 1

[0036] S101

[0037] Obtaining a text description: In this embodiment, the text description is designed by manually designing prompts and automatically designing prompts using ChatGPT.

[0038] For example, when requesting suggestions for scene text image generation, ChatGPT provides many responses, such as “On a typhoon day, there is a store and a store sign.”

[0039] S102

[0040] Synthesize natural images. This embodiment inputs text descriptions into the image model DeepFloyd to synthesize natural images, and better components can be replaced at any time. This embodiment uses ChatGPT and DeepFloyd. Using this method, the present invention can efficiently obtain large-scale natural images in batches.

[0041] S103

[0042] Text erasure. The image generated by the image model DeepFloyd may contain text. The ultimate goal of the first stage of this embodiment is to generate a clean background image without text. Therefore, the image generated by the image model DeepFloyd needs to be erased. In the process of erasing text, PP-OCRv3 is used to check the text box and detect the text area, and then the pre-trained repair model LaMa is used to erase the text.

[0043] S104

[0044] Quality assessment. To ensure high-quality images, this embodiment performs quality assessment on images after text erasure to filter out low-fidelity images. The quality assessment step considers two aspects: (i) using PSNR to assess overall visual quality; (ii) using PP-OCRv3 to assess text residue. In this embodiment, images where "text boxes can still be detected and PSNR is less than 30dB" are considered to be poor quality images. Poor quality images are discarded and regenerated, and images that meet the requirements are retained as background images.

[0045] S105

[0046] Select the text area. The present embodiment provides two methods for position selection, automatic and manual. The area position can be customized by human intervention, or the automatic position selection method of the present invention can be used. In the process of background image generation, the present embodiment proposes more than 1,000 categories that are suitable and unsuitable for generating text areas. The criterion for judging whether it is suitable or unsuitable is whether it violates the common sense of the objective world. For example, "text appears on the signboard" is a reasonable phenomenon, so the signboard is a suitable text area for generation, and "numbers appear in the sky" is an unreasonable phenomenon, so the sky is not suitable for generating text areas. The background image is semantically segmented to leave the semantically reasonable position, and then the specified area is obtained in the same way as SynthText. When generating a single image, it is necessary to additionally specify the area category where text needs to be generated in the Prompt, and then obtain the area in the same automatic way as before and then proceed to the next step of generation.

[0047] Phase II

[0048] S201

[0049] Obtain a partial image. In the first stage of this embodiment, a background image without text is obtained, and an adaptive text block exploration strategy is used to obtain a partial image. In this embodiment, starting from the quadrilateral bounding box representing the text area, the corresponding minimum enclosing rectangle R is first determined, which has a width w and a height h. Using the center point of R as a reference, the present invention converts R into an extended square area SR with a side length s through a heuristic rule. Among them,

[0050]

[0051] The square area SR is the obtained text block, and the background of the text block is the local image c b .

[0052] S202

[0053] Get the text image. In the first stage, the text to be generated is obtained along with the text description, and the text to be generated is converted into a text image c g This step effectively bridges the domain gap and achieves the vision-text fusion goal of the present invention.

[0054] S203

[0055] Conditional embedding. Text image c g and the local image c b Both are encoded by the pre-trained CLIP image encoder to obtain the conditional embedding e g and e b .

[0056] S204

[0057] Image guidance is established through the VAE encoder. A random image is taken as the original image x0. The guidance includes the following: m , the latent vector of the image with the masked text region; b m , word-level mask; c m , character segmentation mask; z t , the potential vector of the image after adding noise. Connect them by channel and record them as z t ′ Among them, l m It is obtained by masking the text area with the local image x0 and passing it through the VAE encoder, b m is a black and white image of the same size as x0, c m It is obtained by x0 through the pre-trained character segmentation model.

[0058] S205

[0059] Model training. g and e b As the condition C is input into the UNet model, the image guidance is input into the UNet model.

[0060] During training, only the parameters of the convolutional layers of the U-Net encoder input, the convolutional layers of the conditional input, and the key and value parameters of the U-Net criss-cross attention layers are updated.

[0061] S206

[0062] Loss metric. The latent features are segmented using a pre-trained character segmentation model to obtain character-level segmentation masks, and the cross entropy loss is used as the character-perceptual loss. In order to improve the accuracy of text rendering, the present invention takes into account text sequence features. To this end, the present invention crops the text area from the decoded image and calculates the text recognition loss through a pre-trained recognition model to supervise the quality of text generation.

[0063] S300

[0064] Effect evaluation. The present invention has conducted extensive experiments to evaluate the effect of the method of the present invention. The present invention has been evaluated from two perspectives, one is the effect experiment of generation, and the other is the experiment applied to downstream detection tasks. Using the test set of the existing real erasure dataset SCUT-EnsText, 500 pictures with a total of 2733 text areas are randomly selected for generation, and the generation accuracy and editing distance are tested. The method of the present invention is used to batch generate scene text detection datasets, and experiments are conducted on DBNet to verify its enhancement effect on detector performance.

[0065] Table 1 shows the accuracy comparison between the present invention and existing text generation methods, including TextDiffuser, DiffSTE and AnyText. In addition, the present invention also provides experimental results on GlyphControl, which is quoted from its original paper. This method cannot perform text fusion and only has generation capabilities. It is worth noting that the dataset used by the present invention is only a subset of the AnyText training dataset, and it shows higher performance in text fusion, which is sufficient to prove the effectiveness of the method of the present invention. Figure 2 ,The visual comparison shows the method of the present invention and the existing methods. Figure 3 The present invention compares the method proposed by the present invention with the currently best method AnyText. The method of the present invention can solve the problem of unexpected irrelevant text appearing in the previous method.

[0066]

[0067] Table 1 Comparative experiments of various modules

[0068] Using the method of the present invention, a text detection dataset SynthGlyph 10K is generated. Table 2 shows the scene text detection results based on the DBNet model pre-trained on each synthetic dataset and tested on the real text dataset (without fine-tuning). It shows the comparison of the effect of the dataset generated by the present invention and the dataset generated by other mainstream methods on the real test dataset after pre-training. The present invention achieves the best performance on all datasets, proving the effectiveness of the present invention. Table 3 shows the scene text detection results based on the DBNet model fine-tuned on IC15, showing that the performance of the detector pre-trained with the dataset generated by the present invention after fine-tuning with real data is better than other methods.

[0069]

[0070] Table 2

[0071] Training Data Accuracy Recall F1 score IC15 83.03 75.64 79.16 IC15+SynthText10K 88.79 80.50 84.44 IC15+VISD10K 90.10 81.08 85.35 IC15+SynthText3D10K 89.45 80.45 84.71 IC15+UnrealText10K 86.67 81.70 84.11 IC15+SynthGlyph10K (the present invention) 88.95 83.29 86.03

[0072] Table 3

Claims

1. A method for generating an image with scene text, comprising: Get text description; Based on the text description, a background image, text to be generated and a text area are obtained; Based on the background image, a local image is obtained by using an adaptive text block exploration strategy; a text image is obtained by using the text to be generated; Obtaining conditional embedding based on the local image and the text image; establishing an image guide based on the text region; Inputting the conditional embedding and the image guidance into a convolutional neural network to obtain a decoded image; The generation of characters in the decoded image is determined by the characters to be generated and the text area.

2. The method according to claim 1, characterized in that After obtaining the background image, text erasure and quality assessment are performed.

3. The method according to claim 1, characterized in that The text area is generated by automatic selection, and the method of automatic selection includes: Performing semantic segmentation on the background image; Preserve semantically reasonable locations based on set reasonable categories; Get the text area in the same way as SynthText.

4. The method according to claim 1, characterized in that: The adaptive text block exploration strategy includes: Convert the quadrilateral bounding box of the text region into an extended square region.

5. The method according to claim 4, characterized in that The side length of the square is determined by the following formula: Wherein, s is the side length of the square, and w and h are the width and height of the minimum enclosing rectangle corresponding to the quadrilateral bounding box.

6. The method according to claim 1, characterized in that The image guidance includes: Latent vector of an image with masked text regions; Word-level masking; Character-level segmentation masks; Latent vector of the noisy image.

7. The method according to claim 1, characterized in that A loss metric is performed on the decoded image.

8. The method according to claim 7, characterized in that The loss metrics include: Denoising loss, character-aware loss, and text recognition loss.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 8.

10. A storage medium storing a computer program, wherein when the computer program is executed by a computer, the method according to any one of claims 1 to 8 is implemented.

Images