A scene text image generation method, system, device and storage medium
By using glyph separation and scene awareness methods, accurate multilingual scene text images are generated, solving the problems of accuracy and realism in text generation in T2I generation models and achieving natural integration of text and background.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PEOPLE CN CO LTD
- Filing Date
- 2025-09-28
- Publication Date
- 2026-06-02
AI Technical Summary
Existing T2I generation models suffer from stroke and structural errors, missing, confused, or irrelevant characters when generating text in images. In particular, the readability of text images in multilingual scenarios is extremely poor, and the relationship between text and scene is not fully considered, resulting in stiff images.
By using glyph separation and scene awareness methods, glyph images and initial background images are generated. The text position is located and a mask image is generated. The glyph images are transformed in combination with the scene image. 3D model reconstruction and texture mapping are used. Finally, the text and background are naturally blended through auxiliary images to generate accurate multilingual scene text images.
It improves the accuracy and reliability of generated text, enhances the realism of text in the scene, avoids additional training, and achieves natural fusion of text and images in multilingual scenes.
Smart Images
Figure CN121304840B_ABST
Abstract
Description
Technical Field
[0001] This disclosure pertains to the field of artificial intelligence, and specifically relates to a method, system, device, and storage medium for generating scene text images. Background Technology
[0002] Significant progress has been made in T2I (Text-to-Image) generation model research in recent years. However, these models are often difficult to publish accurately to generate text in images. The text they generate still has problems such as stroke and structural errors, missing, confused or mixed irrelevant characters, and the readability is extremely poor when generating text in other languages (such as Chinese, Russian and Arabic).
[0003] Scene text image generation has become a new field to address the limitations of T2I generation models. However, existing models do not fully consider the relationship between text and scene when rendering glyph images, and directly use predefined text positions. This results in a stiff and unnatural image after the text and scene are blended together.
[0004] Therefore, a scene text image generation method based on glyph separation and scene awareness is needed to improve the accuracy, reliability and realism of the generated text, and realize the generation of multilingual scene text images. Summary of the Invention
[0005] To address the above problems, this disclosure proposes a method for generating scene text images, the method comprising:
[0006] Obtain the input prompt text for scene text image generation, and generate glyph images and initial background images based on the input prompt text;
[0007] Locate the text position in the initial background image and generate a mask image;
[0008] A scene image is introduced, and the glyph image is transformed based on the scene image, an initial background image, and a mask image to obtain a deformed text image, wherein the scene image is determined based on the initial background image and the mask image;
[0009] An auxiliary image is generated based on the deformed text image and the scene image, and a scene text image is generated based on the auxiliary image.
[0010] Furthermore, generating the glyph image and the initial background image includes:
[0011] Determine the background prompt, target text, and text prompt from the input prompt text;
[0012] A glyph image is generated based on the target text, and an initial background image is generated based on the text prompt. The text prompt is determined by combining the background prompt with the text region, or by using specific characters to replace the target text to generate placeholder text.
[0013] Furthermore, transforming the glyph image to obtain the deformed text image includes generating a background image for erasing the text:
[0014] Generate a background image for erasing text based on background cues and scene images;
[0015] The glyph image is transformed based on the scene image, the background image of the erased text, and the mask image to obtain a deformed text image.
[0016] Furthermore, transforming the glyph image to obtain the deformed text image includes:
[0017] Reconstruct the 3D carrier model and perform UV unwrapping on the reconstructed 3D carrier model;
[0018] Based on the unfolded UV image, the glyph image is used as a texture mapping on the reconstructed 3D carrier model to obtain the texture model;
[0019] Two-dimensional projected image based on background image rendering texture model of erased text;
[0020] The deformed text image is obtained by transforming the two-dimensional projection image based on the mask image.
[0021] Furthermore, the colors of points on the texture model surface correspond one-to-one with the colors of pixels on the character image according to the UV mapping relationship.
[0022] Furthermore, reconstructing the three-dimensional carrier model includes:
[0023] Extract depth normal information, and construct point cloud data of the text carrier region based on the depth normal information and the mask image;
[0024] Reconstruct a 3D carrier model based on point cloud data.
[0025] Furthermore, generating scene text images includes:
[0026] Deformed text images are fused using an edge detection algorithm, and an auxiliary image is generated based on the fused deformed text image and the scene image.
[0027] Generate scene text images based on background cues and auxiliary images.
[0028] This disclosure also provides a scene text image generation system, the system comprising:
[0029] The segmentation unit is configured to acquire input prompt text for scene text image generation, and generate glyph images and initial background images based on the input prompt text;
[0030] A positioning unit is configured to locate the text position in the initial background image and generate a mask image;
[0031] The transformation unit is configured to introduce a scene image and transform the glyph image based on the scene image, an initial background image, and a mask image to obtain a deformed text image, wherein the scene image is determined based on the initial background image and the mask image;
[0032] The synthesis unit is configured to generate an auxiliary image based on the deformed text image and the scene image, and to generate a scene text image based on the auxiliary image.
[0033] This disclosure also provides an electronic device comprising at least one processor and at least one memory, wherein the processor and the memory are data-connected, wherein...
[0034] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described in this disclosure.
[0035] This disclosure also provides a computer storage medium storing one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the methods described in this disclosure.
[0036] This disclosure discloses a method, system, device, and storage medium for generating scene text images, which can effectively generate accurate and realistic multilingual scene text images. It proposes glyph separation, extracting target text from input prompts and generating glyph images according to a specified font, effectively improving the accuracy and reliability of the generated text. It also proposes scene perception, reconstructing a 3D model of the text region based on the depth and normal information of the background image, perceiving the surface characteristics and viewpoint of the carrier, and then deforming the text accordingly based on the scene perception results to better integrate it into the background image, improving the realism of the generated text. Addressing the visual quality degradation caused by directly fusing glyphs and background images, it proposes text inpainting, combining the background image and glyph image to construct an auxiliary image. A pre-trained model is then used with the auxiliary image as an additional condition for full image generation, making the fusion of text and background more natural, avoiding additional training, and realizing the generation of multilingual scene text images. The generation method can generate accurate and realistic multilingual scene text images, meeting practical application needs.
[0037] Other features and advantages of this disclosure will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the disclosure. The objects and other advantages of this disclosure may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description
[0038] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0039] Figure 1 A schematic diagram of the scene text image generation method in an embodiment of this disclosure is shown;
[0040] Figure 2 A schematic diagram of the character separation processing flow in an embodiment of this disclosure is shown;
[0041] Figure 3 A schematic diagram of a masked glyph image is shown in an embodiment of this disclosure;
[0042] Figure 4 A schematic diagram of a scene-aware text deformation processing flow in an embodiment of this disclosure is shown.
[0043] Figure 5 A schematic diagram of scene-aware text deformation in an embodiment of this disclosure is shown;
[0044] Figure 6 A schematic diagram of the glyph and background fusion guide edge in an embodiment of this disclosure is shown;
[0045] Figure 7 A schematic diagram of scene text image generation in an embodiment of this disclosure is shown;
[0046] Figure 8 A schematic diagram of the scene text image generation system structure in an embodiment of this disclosure is shown. Detailed Implementation
[0047] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0048] This disclosure proposes a scene text image generation method based on glyph separation and scene awareness. Glyph separation improves the accuracy and reliability of the generated text; scene awareness enables the generated text to better integrate into the background image, enhancing the realism of the generated text; and text inpainting makes the fusion of text and background more natural, avoiding additional training and realizing multilingual scene text image generation.
[0049] Specifically, in this embodiment of the disclosure, a scene text image is generated through steps such as glyph separation, scene-aware text deformation, and glyph fusion with the background image. Figure 1 A schematic diagram of the scene text image generation method in an embodiment of this disclosure is shown. Figure 1 The method includes: acquiring input prompt text for generating a scene text image, and generating a glyph image and an initial background image based on the input prompt text; locating the text position in the initial background image and generating a mask image; introducing a scene image, and transforming the glyph image based on the scene image, the initial background image, and the mask image to obtain a deformed text image, wherein the scene image is determined according to the initial background image and the mask image; generating an auxiliary image based on the deformed text image and the scene image, and generating a scene text image based on the auxiliary image.
[0050] The scene text image generation method in this embodiment of the present disclosure will be described in detail below through steps S1-S3:
[0051] Step S1: Character Separation
[0052] In this embodiment, the concept of glyph separation is proposed to address the significant shortcomings of existing T2I generation models in expressing text characters, resulting in problems such as stroke and structural errors, missing, confused, or mixed irrelevant characters in the generated text. By extracting the target text from the input prompt, glyph images and initial background images are generated based on the input prompt text using a specified font, and these are used to enhance the T2I generation model's ability to express text characters, thereby improving the accuracy and reliability of the generated text.
[0053] Specifically, generating the glyph image and the initial background image includes: determining the background prompt, the target text, and the text prompt from the input prompt text; generating the glyph image based on the target text, and generating the initial background image based on the text prompt, wherein the text prompt is determined based on the background prompt combined with the text region, or determined by replacing the target text with a specific character to generate placeholder text.
[0054] Optionally, in this embodiment of the disclosure, given input prompt text, the input prompt text is preprocessed using a Large Language Model (LLM) to parse the input prompt text into background prompts, target text, and text prompts. A glyph rendering algorithm is applied to render the target text into glyph images using a user-specified font. The text prompts are used to guide the T2I generation model to generate a background image. Then, text detection or region detection methods are used in the background image to locate the text region and export a mask image.
[0055] It should be noted that, in this embodiment, the text prompt design has two approaches. One is to replace the target text in the input prompt with placeholder text and place it at the beginning of the prompt to emphasize it, thereby guiding the T2I generation model to generate placeholder text. Optionally, the placeholder text can be specific symbols or simple English words, such as ABCDE, 12345, etc. This embodiment does not specifically limit the definition of any particular character or word. The second approach is to add a description of the text region based on the background prompt, thereby guiding the T2I generation model to generate a text region of a specified color and shape. Guided by the text prompt, the generated background image typically has easily identifiable and locatable placeholder text or text regions.
[0056] According to embodiments of this disclosure, glyph separation not only effectively improves the accuracy and reliability of text character generation, but also makes the generation of target text more controllable. During the rendering of glyph images, the font, color, and arrangement of the text can all be specified by the user. Furthermore, glyph separation avoids training different types of text generation models, making the proposed method independent of specific language types and capable of generating images of any language.
[0057] Step S2: Scene-Aware Text Deformation
[0058] In this embodiment of the disclosure, in order to address the problem that although character separation ensures the accurate and controllable generation of the target text, the generated text is severely separated from the background area, resulting in a decrease in realism, the concept of scene perception is introduced. The three-dimensional model of the text area is reconstructed based on the depth and normal information of the background image, the surface characteristics and perspective of the carrier are perceived, and then the character image is deformed accordingly based on the result of scene perception, so that it can be better integrated into the background image, thereby improving the realism of the generated text.
[0059] Specifically, transforming the character image to obtain a deformed text image includes: generating a background image of the erased text based on the background cue and scene image; extracting depth normal information; and constructing point cloud data of the text carrier region based on the depth normal information and the mask image; reconstructing a three-dimensional carrier model based on the point cloud data; performing UV unwrapping on the reconstructed three-dimensional carrier model; mapping the character image as a texture onto the reconstructed three-dimensional carrier model based on the unwrapped UV image to obtain a texture model; rendering a two-dimensional projection image of the texture model based on the background image of the erased text; and transforming the two-dimensional projection image based on the mask image to obtain the deformed text image.
[0060] Optionally, in this embodiment, an edge detection algorithm is applied to the background image, and the text portion is removed based on the mask image to obtain a scene image. To eliminate the interference of text on the reconstruction of the 3D volume model, the background cue and scene image are used as control conditions to guide the T2I generation model to generate a background image with the text erased. Depth maps and normal maps are taken, and point cloud data of the text carrier region is created based on the mask image. The Poisson reconstruction algorithm is applied to the point cloud data to reconstruct the 3D carrier model. The 3D carrier model is UV unwrapped, and the character images are mapped onto the model as textures based on the unwrapped UV image to obtain a texture model. According to the viewing direction of the background image, i.e., facing inwards from the screen, a 2D projection image of the texture model is rendered. Finally, the 2D projection image is transformed based on the mask image to obtain a deformed text image.
[0061] For ease of understanding, UV mapping refers to the 3D modeling process of projecting a 2D image onto the surface of a 3D model to perform texture mapping. U represents the horizontal direction and V represents the vertical direction, which are respectively the axes of the 2D texture.
[0062] To further enhance scene perception, the colors of points on the texture model surface and the colors of pixels on the character image are mapped one-to-one according to the UV mapping relationship.
[0063] Optionally, in this embodiment of the disclosure, when using background prompts and scene images as control conditions to guide the T2I generation model to generate background images for erasing text, the control conditions for background prompts and scene images are introduced through the Controlnet model. It should be noted that the model for introducing these control conditions is only an illustrative example and does not specifically limit the specific model.
[0064] Optionally, in this embodiment of the disclosure, Marigold is used to obtain depth maps and normal maps; when performing UV unwrapping, the three-dimensional carrier model is imported into Blender for UV unwrapping. It should be noted that the methods for obtaining depth maps and normal maps and the related models for performing UV unwrapping are only illustrative examples, and the methods of this disclosure can also use other models or methods to achieve the same purpose.
[0065] This method enables scene awareness to better integrate the generated text into the background image, improving the realism of the generated text.
[0066] Step S3: Blending the font with the background image
[0067] In this embodiment of the disclosure, directly fusing the glyphs and background images may lead to visual quality degradation issues such as blurred text edges and background artifacts. By introducing auxiliary images and using them as additional conditions for full image generation, and applying text repair through the auxiliary images, the fusion of text and background becomes more natural, avoiding additional training and realizing text image generation for multilingual scenarios.
[0068] Specifically, generating scene text images includes: fusing deformed text images using edge detection algorithms, generating auxiliary images based on the fused deformed text images combined with scene images; and generating scene text images based on background cues and auxiliary images.
[0069] Optionally, in this embodiment of the disclosure, an edge detection algorithm is applied to the deformed text image, and then it is combined with the scene image to obtain an auxiliary image used to guide the final generation; the background prompt and the auxiliary image are used as control conditions to guide the T2I generation model to generate the full image and obtain the final scene text image.
[0070] Optionally, in this embodiment of the disclosure, when using background cues and auxiliary images as control conditions to guide the T2I generation model, the control conditions for the background cues and auxiliary images are introduced through the Controlnet model. It should be noted that the introduction model of the control conditions is only an illustrative example and does not specifically limit the specific introduction model.
[0071] In this embodiment, by generating an auxiliary image that takes into account both text and scene, it is more conducive to the natural integration of text and scene. The auxiliary image fully expresses the outline details of the text and the spatial layout of the image, making the integration of text and background more natural. It is worth noting that this process enables the T2I generation model to accurately generate text without relying on the encoding of the target text, thereby avoiding additional training and realizing multilingual scene text image generation.
[0072] This disclosure also provides a detailed description of the specific implementation process of the scene text image generation method through an embodiment and in conjunction with the accompanying drawings. It should be noted that the scope of protection of this disclosure is not limited to this implementation process.
[0073] The specific workflow of this disclosure includes: first, glyph separation is performed to extract the target text from the input prompt and generate glyph images according to the specified font; then, scene-aware text deformation is performed to reconstruct the three-dimensional model of the text region based on the depth and normal information of the background image, perceive the surface characteristics and perspective of the carrier, and then deform the text accordingly based on the results of scene perception; finally, glyph and background fusion is performed to combine the background image and glyph image to form an auxiliary image, and a pre-trained model is used to generate the full image with the auxiliary image as an additional condition to obtain the final scene text image.
[0074] Figure 2 A schematic diagram of the character separation processing flow in an embodiment of this disclosure is shown. Figure 2 In this process, glyph separation is applied, which includes input prompt preprocessing, glyph image rendering, background image generation, and text position positioning.
[0075] Optionally, in this embodiment of the disclosure, the glyph separation process specifically includes:
[0076] Given input prompts First, it is preprocessed using a large language model to parse it into background cues. and target text and design text prompts Text prompt There are two design approaches: one is to replace the target text in the input prompt with placeholder text (such as specific symbols or simple English words). And place it at the beginning of the prompt for emphasis, thereby guiding the T2I generation model to generate placeholder text; secondly, in the background prompt Building upon this foundation, a description of the text region is added to guide the T2I generation model in producing text regions of a specified color and shape. The mathematical expression of this process is as follows:
[0077] (1)
[0078] Then, a character rendering algorithm is applied to render the target text. With user-specified font Rendering is based on glyph images :
[0079] (2)
[0080] Use text prompts Guide the T2I generative model to generate background images In this embodiment of the disclosure, the background image is generated under the guidance of text prompts. They typically have good composition and placeholder text that is easy to identify and position.
[0081] Optionally, in the embodiments of the present disclosure, the process of generating the background image uses a T2I generation model, without limiting the specific model type. For example, the Flux.1-shell model. It should be noted that no specific limitation is made on the model type for generating the background image. The mathematical expression of this process is as follows:
[0082] (3)
[0083] In the background image character detection is used to locate the placeholder text or region detection is used to locate the text region, and a mask image is exported. The mathematical expression of this process is as follows:
[0084] (4)
[0085] Optionally, in the embodiments of the present disclosure, the PaddleOCRv3 model is used to locate the text position in the initial background image and generate a mask image. Specifically, the PaddleOCRv3 model inputs the image to be detected and outputs the text content and the corresponding text position, where the text position is represented by the four vertex coordinates of a rotated rectangle.
[0086] After obtaining the text position, a completely black image with the same resolution as the background image is generated, and the corresponding rectangular area is filled with white pixels to obtain the mask image . Figure 3 shows a schematic diagram of the masked glyph image in the embodiments of the present disclosure; Figure 3 in which, Chinese characters ("All rivers run into sea, cultivate one's morality"), English words ("sunrise, take a break"), Russian and Arabic are used to represent different glyph images respectively, and the area filled with white pixels is shown in the completely black image. The size of the area filled with white pixels in the corresponding rectangular area matches the text position in the initial background image.
[0087] Optionally, in the embodiments of the present disclosure, if the text detection fails, that is, no valid text is detected in the background image , to ensure the integrity of the process, the rectangular area in the center of the image will be directly filled, indicating that the text is located in the center of the image.
[0088] Optionally, in the embodiments of the present disclosure, the processing flow of text deformation based on scene perception specifically includes:
[0089] Scene perception is introduced, and then the text is deformed accordingly based on the result of scene perception. Figure 4 shows a schematic diagram of the processing flow of text deformation based on scene perception in the embodiments of the present disclosure, Figure 4In this process, the steps include text erasure, depth normal information extraction, point cloud data construction, 3D carrier model reconstruction, UV unwrapping and texture bonding, 2D projection image rendering and deformed text image generation.
[0090] Background image Apply edge detection algorithms and based on mask images After removing the text portion, the scene image is obtained. The mathematical expression of this process is as follows:
[0091] (5)
[0092] To eliminate the interference of text on the reconstruction of the 3D carrier model, background cues are used. and scene images As a control condition, the T2I generative model is guided to generate a background image for the erased text. Its mathematical expression is as follows:
[0093] (6)
[0094] Optionally, in this embodiment of the disclosure, the image conditions are introduced through the Controlnet model or the SDXL model. It should be noted that the generation process of the background image of the erased text is not limited to a specific type of T2I generation model. The method of this disclosure can also use other models or methods to achieve the same purpose.
[0095] Use Marigold to obtain a depth map and normal diagram It should be noted that the methods for obtaining depth maps and normal maps in this embodiment are illustrative and not specifically limited to these methods in the actual implementation. and normal diagram Represented as:
[0096] (7)
[0097] And based on the mask image Create point cloud data for text carrier regions :
[0098] (8)
[0099] Point cloud data Reconstructing the 3D carrier model M using the Poisson reconstruction algorithm:
[0100] (9)
[0101] The 3D carrier model M is then imported into Blender for UV unwrapping. It should be noted that Blender is a 3D graphics software. In this embodiment, the unwrapping method for the giant sun is not limited. UV unwrapping is represented as follows:
[0102] (10)
[0103] Based on the unfolded UV image Basic glyph images As a texture mapping onto model M, we obtain the texture model. Texture model Colors and basic glyph images of dots on the surface The colors of the pixels correspond one-to-one according to the UV mapping relationship. The mathematical expression of this process is as follows:
[0104] (11)
[0105] According to the background image The viewing direction, i.e., facing inwards from the screen, is used to render the texture model. Two-dimensional projection image The mathematical expression of this process is as follows:
[0106] (12)
[0107] Based on the mask image Two-dimensional projection image Transform to obtain deformed text image The process first detects the two-dimensional projection image. For all contours, determine the smallest bounding rectangle that can contain all contours, and then calculate the bounding rectangle and the mask image. The perspective transformation matrix of the rectangle is used to map the deformed text to the target location through affine transformation. Figure 5 A schematic diagram of scene-aware text deformation in an embodiment of this disclosure is shown. The mathematical expression of this process is as follows:
[0108] (13)
[0109] Optionally, in this embodiment of the disclosure, the process of blending the glyphs with the background specifically includes auxiliary image synthesis and full image generation:
[0110] To achieve a more natural integration of text and scene, an edge detection algorithm is applied to the deformed text image, which is then compared with the scene image. Combined, an auxiliary image is obtained to guide the final generation. , Figure 6This illustration shows a schematic diagram of the font and background fusion guiding edge in an embodiment of this disclosure, with auxiliary images. This process fully expresses the outline details of the text and the spatial layout of the image, making the blending of text and background more natural. The mathematical expression of this process is as follows:
[0111] (14)
[0112] Use background prompts and auxiliary images The T2I generative model is guided by control conditions to generate the entire image, resulting in the final scene text image. , Figure 7 This illustration shows a scene text image generation diagram in an embodiment of this disclosure, where image conditions are introduced through a ControlNet model. It should be noted that, in this embodiment, the model for introducing image conditions is not limited to SDXL models or other T2I generation models such as ControlNet models. The final scene text image is obtained. The mathematical expression is as follows:
[0113] (15)
[0114] Specifically, in this embodiment of the present disclosure, the process of determining the scene text image enables the T2I generation model to accurately generate text without relying on the encoding of the target text, thereby avoiding additional training.
[0115] Based on the above three steps, the mathematical expression of the scene text image generation process disclosed herein is as follows: (16)
[0116] In equation (16), A represents the synthesized auxiliary image. The entire process.
[0117] This disclosure also provides a scene text image generation system. Figure 8 A schematic diagram of the scene text image generation system structure in an embodiment of this disclosure is shown. Figure 8 The system includes: a segmentation unit configured to acquire input prompt text for generating a scene text image, and generate a glyph image and an initial background image based on the input prompt text; a positioning unit configured to locate the text position in the initial background image and generate a mask image; a transformation unit configured to introduce a scene image and transform the glyph image based on the scene image, the initial background image, and the mask image to obtain a deformed text image, wherein the scene image is determined based on the initial background image and the mask image; and a synthesis unit configured to generate an auxiliary image based on the deformed text image and the scene image, and generate a scene text image based on the auxiliary image.
[0118] Optionally, in this embodiment of the disclosure, the segmentation unit includes a segmentation module and an initialization module. The segmentation module is configured to determine a background prompt, a target text, and a text prompt from the input prompt text. The initialization module is configured to generate a glyph image based on the target text and generate an initial background image based on the text prompt. The text prompt is determined based on the background prompt combined with the text region, or by replacing the target text with a specific character to generate placeholder text.
[0119] Optionally, in this embodiment of the present disclosure, the transformation unit includes an erasing module, an extraction module, a reconstruction module, an unfolding module, a mapping module, a rendering module, and a deformation module. The erasing module is configured to generate a background image of the erased text based on the background cue and the scene image; the extraction module is configured to extract depth normal information and construct point cloud data of the text carrier region based on the depth normal information and the mask image; the reconstruction module is configured to reconstruct a three-dimensional carrier model based on the point cloud data; the unfolding module is configured to perform UV unwrapping on the reconstructed three-dimensional carrier model; the mapping module is configured to map the glyph image as a texture onto the reconstructed three-dimensional carrier model based on the unfolded UV image to obtain a texture model; the rendering module is configured to render a two-dimensional projection image of the texture model based on the background image of the erased text; and the deformation module is configured to transform the two-dimensional projection image based on the mask image to obtain a deformed text image.
[0120] Optionally, in this embodiment of the present disclosure, the synthesis unit includes an auxiliary module and a fusion module. The auxiliary module is configured to fuse the deformed text image using an edge detection algorithm and generate an auxiliary image based on the fused deformed text image and the scene image. The fusion module is configured to generate a scene text image based on the background prompt and the auxiliary image.
[0121] This disclosure also provides an electronic device, which includes at least one processor and at least one memory, wherein the processor and the memory are data-connected.
[0122] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the scene text image generation method described in this embodiment of the disclosure.
[0123] This disclosure also provides a computer storage medium storing one or more instructions, which, when executed by one or more computers, cause the one or more computers to implement the scene text image generation method described in this disclosure.
[0124] This disclosure proposes an accurate, controllable, and highly realistic method for generating scene text images. Addressing the shortcomings of existing T2I generation models in representing text characters, especially non-English characters, leading to errors in strokes and structure, missing, confused, or extraneous characters, this method offers a more accurate and controllable approach. First, the target text is extracted from the input prompt, and glyph images are generated using a specified font. Then, the processed prompt is input into the T2I model to generate a background image, and the embedding position of the target text within it is detected. Finally, the generated glyph images are fused with the background image to generate the scene text image. To address the issue of poor realism in the generated text, a scene-aware method is proposed. This method reconstructs a 3D model of the text region based on the depth and normal information of the background image, perceiving the surface characteristics and viewpoint of the carrier. Based on the scene-aware results, the text is deformed accordingly to better integrate into the background image, thus improving the realism of the generated text. This method, through a separation-then-fusion approach, not only compensates for the shortcomings of existing T2I models in text representation, ensuring accurate text generation, but also enhances the realism of embedded text in natural scene images through scene-aware character-image fusion. Furthermore, this method achieves more natural text-to-background integration through text restoration, avoiding additional training and enabling multilingual scene text-to-image generation.
[0125] Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.
Claims
1. A method for generating scene text images, characterized in that, The method includes: Obtain the input prompt text for scene text image generation, and generate glyph images and initial background images based on the input prompt text; Locate the text position in the initial background image and generate a mask image; A scene image is introduced, and the glyph image is transformed based on the scene image, an initial background image, and a mask image to obtain a deformed text image, wherein the scene image is determined based on the initial background image and the mask image; An auxiliary image is generated based on the deformed text image and the scene image, and a scene text image is generated based on the auxiliary image; The generation of glyph images and initial background images includes: Determine the background prompt, target text, and text prompt from the input prompt text; A glyph image is generated based on the target text, and an initial background image is generated based on the text prompt, wherein the text prompt is determined by combining the background prompt with the text region, or by generating placeholder text by replacing the target text with characters; Transforming the glyph image to obtain a deformed text image includes generating a background image for erasing the text: Generate a background image for erasing text based on background cues and scene images; Based on the scene image, the background image of the erased text, and the mask image, the glyph image is transformed to obtain a deformed text image; Transforming the glyph image to obtain a deformed text image includes: Reconstruct the 3D carrier model and perform UV unwrapping on the reconstructed 3D carrier model; Based on the unfolded UV image, the glyph image is used as a texture mapping on the reconstructed 3D carrier model to obtain the texture model; Two-dimensional projected image based on background image rendering texture model of erased text; The deformed text image is obtained by transforming the two-dimensional projection image based on the mask image.
2. The scene text image generation method according to claim 1, characterized in that, The colors of points on the texture model surface correspond one-to-one with the colors of pixels on the character image according to the UV mapping relationship.
3. The scene text image generation method according to claim 1, characterized in that, The reconstruction of the 3D carrier model includes: Extract depth normal information, and construct point cloud data of the text carrier region based on the depth normal information and the mask image; Reconstruct a 3D carrier model based on point cloud data.
4. The scene text image generation method according to claim 1, characterized in that, The generated scene text image includes: Deformed text images are fused using an edge detection algorithm, and an auxiliary image is generated based on the fused deformed text image and the scene image. Generate scene text images based on background cues and auxiliary images.
5. A scene text image generation system, characterized in that, The system includes: The segmentation unit is configured to acquire input prompt text for scene text image generation, and generate glyph images and initial background images based on the input prompt text; A positioning unit is configured to locate the text position in the initial background image and generate a mask image; The transformation unit is configured to introduce a scene image and transform the glyph image based on the scene image, an initial background image, and a mask image to obtain a deformed text image, wherein the scene image is determined based on the initial background image and the mask image; The synthesis unit is configured to generate an auxiliary image based on the deformed text image and the scene image, and to generate a scene text image based on the auxiliary image; The generation of glyph images and initial background images includes: Determine the background prompt, target text, and text prompt from the input prompt text; A glyph image is generated based on the target text, and an initial background image is generated based on the text prompt, wherein the text prompt is determined by combining the background prompt with the text region, or by generating placeholder text by replacing the target text with characters; Transforming the glyph image to obtain a deformed text image includes generating a background image for erasing the text: Generate a background image for erasing text based on background cues and scene images; Based on the scene image, the background image of the erased text, and the mask image, the glyph image is transformed to obtain a deformed text image; Transforming the glyph image to obtain a deformed text image includes: Reconstruct the 3D carrier model and perform UV unwrapping on the reconstructed 3D carrier model; Based on the unfolded UV image, the glyph image is used as a texture mapping on the reconstructed 3D carrier model to obtain the texture model; Two-dimensional projected image based on background image rendering texture model of erased text; The deformed text image is obtained by transforming the two-dimensional projection image based on the mask image.
6. An electronic device, characterized in that, The electronic device includes at least one processor and at least one memory, wherein the processor and the memory are data connected. The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.
7. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1-4.