Image generation method, electronic device, storage medium and computer program product

By constructing local area data and text content data in image generation, and generating and fusing local images, the problem of poor text image generation effect in the prior art is solved, more natural and efficient image generation is achieved, and text image generation is improved Authenticity and usability.

CN118941675APending Publication Date: 2024-11-12ALIBABA (CHINA) CO LTD
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410921078.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-09
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing text image generation methods are limited in real natural scenes and cannot effectively meet the needs of deep learning models for synthetic data, especially in terms of the authenticity and usability of image generation.

Method used

By obtaining the original image, text content data and text area data, local area data are constructed to represent the area where the text is located, and a local image is generated based on the text content data using the image generation model, and fuse it with the original image to generate the target image, so that the generation The text can be of any scale, which is more in line with the characteristics of natural scenes.

Benefits of technology

It improves the authenticity and usability of image generation, can generate more natural text images, enhances the performance of OCR tasks, and is suitable for text detection and recognition in real natural scenes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941675A_ABST
    Figure CN118941675A_ABST
Patent Text Reader

Abstract

The invention discloses an image generation method, electronic equipment, a storage medium and a computer program product, and relates to the field of large model technology and image generation. The method comprises the steps that an original image, text content data and text area data are obtained, the text content data are used for describing a text needing to be rendered in the original image, and the text area data are used for describing the position, needing to be rendered, of the text in the original image; constructing local region data based on the text region data; using the local area data to control an image generation model to generate a local image based on the text content data and the original image; and based on the local region data, fusing the local image with the original image to generate a target image. The technical problem that the image generation effect is poor in the prior art is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the fields of large model technology and image generation, and more specifically, to an image generation method, electronic device, storage medium, and computer program product. Background Art

[0002] Currently, due to the data-hungry nature of deep learning models and the specific characteristics of specific domains, obtaining sufficient real-world data is extremely expensive and difficult. Numerous studies have demonstrated the importance of synthetic data in various scene text tasks. Consequently, the use of synthetic data has become standard practice, making text image generation a hot research topic. However, existing text image generation methods are primarily divided into two categories: rule-based rendering and diffusion model-based text image generation schemes. Both have limitations in the authenticity or usability of generated images, limiting their effectiveness in real-world scenes. To meet the demand for effective synthetic data in deep learning models, more advanced text image synthesis methods are urgently needed.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide an image generation method, an electronic device, a storage medium, and a computer program product to at least solve the technical problem of poor image generation effect in related technologies.

[0005] According to one aspect of an embodiment of the present application, an image generation method is provided, including: acquiring an original image, text content data, and text area data, wherein the text content data is used to describe the text to be rendered in the original image, and the text area data is used to describe the position in the original image where the text needs to be rendered; constructing local area data based on the text area data, wherein the local area data is used to characterize the area in the original image where the text needs to be rendered; using the local area data, controlling an image generation model to generate a local image based on the text content data and the original image, wherein the local image contains text and the size of the local image is the same as the size of the local area data; and based on the local area data, fusing the local image with the original image to generate a target image.

[0006] According to one aspect of an embodiment of the present application, an image generation method is provided, comprising: capturing a scene image of a real scene, and obtaining text content data and text area data, wherein the text content data is used to describe text that needs to be rendered in the scene image, and the text area data is used to describe a position in the scene image where the text needs to be rendered; constructing local area data based on the text area data, wherein the local area data is used to characterize the position of an area in the scene image where the text needs to be rendered; using the local area data, controlling an image generation model to generate a local image based on the text content data and the scene image, wherein the local image contains text and the size of the local image is the same as the size of the local area data; and fusing the local image with the scene image based on the local area data to generate a target image.

[0007] According to one aspect of an embodiment of the present application, an image generation method is provided, comprising: responding to an input instruction acting on an operation interface, displaying an original image, text content data and text area data on the operation interface, wherein the text content data is used to describe the text to be rendered in the original image, and the text area data is used to describe the position of words in the text to be rendered in the original image; responding to a generation instruction acting on the operation interface, displaying a target image on the operation interface, wherein the target image is an image obtained by fusing a local image with the original image based on the local area data, the local area data is constructed based on the text area data, the local area data is used to characterize the position of the area in the original image where the text is to be rendered, the local image is an image generated based on the text content data and the original image by controlling an image generation model using the local area data, the local image contains text, and the size of the local image is the same as the size of the local area data.

[0008] According to one aspect of an embodiment of the present application, an image generation method is provided, comprising: obtaining an original image, text content data and text area data by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes the original image, text content data and text area data, the text content data is used to describe the text to be rendered in the original image, and the text area data is used to describe the position in the original image where the text needs to be rendered; constructing local area data based on the text area data, wherein the local area data is used to characterize the position of the area in the original image where the text needs to be rendered; using the local area data, controlling an image generation model to generate a local image based on the text content data and the original image, wherein the local image contains text and the size of the local image is the same as the size of the local area data; based on the local area data, fusing the local image with the original image to generate a target image; and outputting the target image by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target image.

[0009] According to another aspect of the embodiments of the present application, an electronic device is provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in the various embodiments of the present application when running.

[0010] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, which includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in various embodiments of the present application.

[0011] According to another aspect of the embodiments of the present application, a computer program product is further provided, including a computer program, which implements the methods in various embodiments of the present application when executed by a processor.

[0012] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0013] According to another aspect of the embodiments of the present application, a computer program is further provided, which implements the methods in various embodiments of the present application when executed by a processor.

[0014] In an embodiment of the present application, an original image, text content data and text area data are obtained, wherein the text content data is used to describe the text to be rendered in the original image, and the text area data is used to describe the position of the text to be rendered in the original image; local area data is constructed based on the text area data, wherein the local area data is used to characterize the position of the area where the text is to be rendered in the original image; the image generation model is controlled by using the local area data to generate a local image based on the text content data and the original image, wherein the local image contains text and the size of the local image is the same as the size of the local area data; based on the local area data, the local image is fused with the original image to generate a target image, thereby improving the image generation effect. It is easy to notice that the image generation process can be decoupled into a method of determining the local image in the original image based on the local area data and the text content data, and the local image can be fused with the original image so that the generated target image can contain text of any scale, which is more in line with the characteristics of the natural scene, and improves the image generation effect, thereby solving the technical problem of poor image generation effect in the related art.

[0015] It is easy to notice that the above general description and the following detailed description are merely for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 is a schematic diagram of an application scenario of image generation according to an embodiment of the present application;

[0018] Figure 2 is a flowchart of the image generation method according to Example 1 of the present application;

[0019] Figure 3 is a schematic diagram of a text image processing structure according to an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of a processing structure of a visualized text image according to an embodiment of the present application;

[0021] Figure 5 is a training flow chart of an image generation model according to an embodiment of the present application;

[0022] Figure 6 is a flowchart of an image generation method according to Example 2 of the present application;

[0023] Figure 7 is a flowchart of an image generation method according to Example 3 of the present application;

[0024] Figure 8 is a flowchart of an image generation method according to Example 4 of the present application;

[0025] Figure 9 is a schematic diagram of an image generating device according to Example 5 of the present application;

[0026] Figure 10 is a schematic diagram of an image generating device according to Example 6 of the present application;

[0027] Figure 11 is a schematic diagram of an image generating device according to Example 7 of the present application;

[0028] Figure 12 is a schematic diagram of an image generating device according to Example 8 of the present application;

[0029] Figure 13 This is a structural block diagram of an electronic device according to Example 9 of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] The technical solution provided in this application is mainly implemented using large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. The large model can also be called a cornerstone model / foundation model. It is pre-trained by using large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as large-scale language models (LLMs) and multi-modal pre-training models.

[0033] It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned with a small number of samples, so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image captioning (IC), and image generation. It can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0034] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0035] Optical Character Recognition (OCR): A technology that converts text in an image into editable text data. OCR technology is widely used in document digitization, intelligent form processing, smart card recognition, and other fields.

[0036] Diffusion Models: It is the current mainstream cultural graph framework.

[0037] Conditional Diffusion Models: Diffusion models based on conditional control. Compared with diffusion models, the images they generate are controlled by input conditions.

[0038] Text image generation: In this application, it refers to the generation of a complete image. The relative concept is sub-image text image generation, which is the process of cutting out single word or text line images.

[0039] Encoding-decoding network (U Net) based on convolutional neural network and skip connection: generally used to generate images of the same size as the input image.

[0040] Authenticity: Generated text pixels should blend naturally with the image background without noticeable artifacts. The text content and area contained in the image should be the same as specified in the given conditions, without incorrect characters or redundant text.

[0041] Usability: The generated images should better enhance the performance of related OCR tasks, such as text detection and recognition.

[0042] Real natural scenes: such as street scenes and indoor shots, as distinguished from synthetic scenes such as e-commerce advertisements and cartoon animations.

[0043] Rule-based rendering text image generation schemes: Given the text content to be generated and the region within the image, these schemes collect a variety of fonts, colors, backgrounds, and other materials, and perform rendering synthesis using manually defined rules such as text layout and attribute selection. The drawbacks of these methods are that randomly selecting text attributes from a limited set of materials can result in noticeable artifacts in text pixels and unnatural fusion. Furthermore, the collected materials are always limited, and different synthesis rules must be manually designed for different scenarios, resulting in low efficiency and limitations. These factors affect the authenticity and usability of the generated images.

[0044] Text image generation scheme based on diffusion model: Given the text content to be generated and the area in the image, this type of scheme generates inputs such as text layout and content based on these priors, adopts the repair mode of the diffusion model, and generates text images in combination with the prompt text. This type of scheme is based on the diffusion model's ability to semantically align the prompt text, and the foreground text and background image are more naturally integrated than schemes based on rule rendering. However, this type of scheme trains the diffusion model based on high-dimensional features, which will lose the texture features of smaller-sized text, and thus performs poorly in small-sized text generation, with a significant gap from real natural scenes. In addition, the text generated by this type of scheme is usually not aligned with the area indicated in the conditions, and may generate redundant text outside the specified area, thus affecting its usability.

[0045] This application decouples text image generation into local text image generation and fusion, so that the generated text can be of any scale, more consistent with the text image characteristics in real natural scenes, greatly improving the authenticity of the generated image. In addition, the generated text has accurate OCR annotation information, which improves the usability of the generated image.

[0046] Example 1

[0047] According to an embodiment of the present application, an image generation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0048] Considering the huge number of model parameters of large models and the limited computing resources of mobile terminals, the above-mentioned image generation method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to these. Figure 1 This is a schematic diagram of an application scenario of image generation according to an embodiment of the present application. Figure 1 In the illustrated application scenario, the large model is deployed on a server 10. The server 10 can be connected to one or more client devices 20 via a local area network, a wide area network, the Internet, or other types of data networks. The client devices 20 herein may include, but are not limited to, smartphones, tablet computers, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. The client devices 20 can interact with users via a graphical user interface to access the large model and thereby implement the methods provided in the embodiments of the present application.

[0049] In an embodiment of the present application, a system composed of a client device and a server can perform the following steps: the client device generates an original image; receives a target image; the server obtains the original image, text content data and text area data, wherein the text content data is used to describe the text that needs to be rendered in the original image, and the text area data is used to describe the position where the text needs to be rendered in the original image; local area data is constructed based on the text area data, wherein the local area data is used to characterize the position of the area where the text needs to be rendered in the original image; using the local area data, the image generation model is controlled to generate a local image based on the text content data and the original image, wherein the local image contains text and the size of the local image is the same as the size of the local area data; based on the local area data, the local image is merged with the original image to generate a target image. It should be noted that when the operating resources of the client device can meet the deployment and operation conditions of the large model, the embodiment of the present application can be carried out in the client device.

[0050] Under the above operating environment, this application provides Figure 2 The image generation method shown. Figure 2 This is a flow chart of the image generation method according to Example 1 of the present application. Figure 2 As shown, the method may include the following steps:

[0051] Step S202: Acquire the original image, text content data, and text area data.

[0052] The text content data is used to describe the text that needs to be rendered in the original image, and the text area data is used to describe the position where the text needs to be rendered in the original image.

[0053] The position of the above text may be the position of an entire text line, and the text line may contain one or more words.

[0054] The original image may be a given background image (Ground truth). The original image may be an image of text to be rendered. The original image may be an unmodified initial image, and may be any image such as a photograph, a painting, or a chart.

[0055] The above-mentioned text content data can be the text to be rendered in the image. The text content data can be any form of text information, such as a title, comment, description, etc. The presentation method of the text content data can also be set according to actual needs and is not limited here.

[0056] The above-mentioned text area data can be the location where the text needs to be rendered in the image. The text area data can be a set of data used to define the specific location of the text content data on the original image. It can be the coordinate information of the text content data, and can also include attributes such as the size, direction, and alignment of the text display. The text area data can include line-level and word-level quadrilateral box coordinates. For each line and each word in the text, a quadrilateral box can be determined to represent the position and size of the text on the page. The text area data is very useful for tasks such as text recognition, text segmentation, and layout analysis, and can help computer systems understand and process text content more accurately.

[0057] The above-mentioned text area data may be word-level and line-level coordinates, and the coordinates may be coordinates of the bounding boxes corresponding to the word-level and line-level.

[0058] In an optional embodiment, after the original image is acquired, some preprocessing can be performed on the original image, for example, adjusting the size, brightness, contrast, etc., to ensure the instructions of the original image used subsequently. The text area data can be used to determine the exact position of the text content data on the original image, and the text content data can be rendered into the original image to ensure that the text content data and the original image are naturally integrated.

[0059] Step S204: construct local region data based on the text region data.

[0060] The local area data is used to represent the location of the area where the text needs to be rendered in the original image.

[0061] The aforementioned local region data can be used as a method for labeling local regions within an image. This method defines a rectangular box within the image to label specific areas, such as words or numbers. Local region data can accurately locate and describe specific content within an image, facilitating subsequent image processing and analysis.

[0062] The above-mentioned local area data can be generated based on the text area data, and the shape and size of the local area data can be pre-set or determined based on the size and shape of the text area data. The construction method of the local area data is not limited here.

[0063] In an optional embodiment, a horizontal bounding rectangle can be generated based on the text region data, and local region data can be generated based on the horizontal bounding rectangle. Optionally, the horizontal bounding rectangle can be determined as the local region data, and the boundary of the horizontal bounding rectangle can be expanded to a certain extent to obtain the local region data. The boundary of the horizontal bounding rectangle can be expanded by 10% to obtain the local region data. The 10% here is only for illustration, and the degree of boundary expansion can be adjusted according to actual needs.

[0064] For example, the horizontal circumscribed rectangle may be a circumscribed quadrilateral of a continuous line of text, where the continuous text is, for example, "Hello world". Generally speaking, coordinate frame labels of these lines of text may be pre-set in the data set.

[0065] In another optional embodiment, a rotated rectangular or polygonal area can be generated based on the text area data, and local area data can be generated based on the rotated rectangular or polygonal area. Optionally, the rotated rectangular or polygonal area can be determined as the local area data, and the boundary of the rotated rectangular or polygonal area can be expanded to a certain extent to obtain the local area data. The boundary of the rotated rectangular or polygonal area can be expanded by 10% to obtain the local area data. The 10% here is only for example, and the degree of boundary expansion can be adjusted according to actual needs.

[0066] It should be noted that the above-mentioned local area data can also be generated based on other external graphics generated from the text area data, and the generation method of the local area data is not limited here.

[0067] Step S206 : Using the local area data, the image generation model is controlled to generate a local image based on the text content data and the original image.

[0068] The local image contains text, and the size of the local image is the same as the size of the local area data.

[0069] The above-mentioned image generation model may be a conditional diffusion model. There is no limitation on the type of the image generation model, and the image generation model may be selected according to actual needs.

[0070] The Conditional Diffusion Model (CDM) is a generative model-based approach for generating images that satisfy specific conditions. This model learns from the data distribution to generate new samples that satisfy the given conditions. The core idea of ​​the CDM is to view the generation process as a step-by-step denoising process, generating images by gradually predicting and removing noise.

[0071] The above-mentioned conditional diffusion model can be an open source generation framework (DALLE2) or a stable diffusion model (Stable-Diffusion). There is no limitation on the type of conditional diffusion model here, and the required conditional diffusion model can be selected according to actual needs.

[0072] In an optional embodiment, local region data can be defined on the original image, marking the target region to be generated. Based on the task requirements, corresponding conditions (i.e., the aforementioned text content data and the original image) can be provided. An image generation model can then be used to generate a local image based on the local region data and conditions. The model will gradually predict and remove noise to generate a local image that meets the conditions. Using local region data, the original image can be locally modified and enhanced to meet specific needs and application scenarios, resulting in an image of the modified local region within the original image.

[0073] Step S208: Based on the local area data, the local image is fused with the original image to generate a target image.

[0074] The aforementioned fusion may refer to the process of combining a partial image with an original image to generate a new target image. Fusion may be a simple superposition or a more complex pixel-level synthesis, and the specific method of fusion is not limited here.

[0075] The above-mentioned local area data is mainly used to determine the fusion area of ​​the local image in the original image. By locating the fusion position of the local image through the local area data, the local image and the original image can be fused more naturally, thereby improving the generation effect of the target image.

[0076] In an optional embodiment, the partial image can be fused with the location corresponding to the local region data in the original image to generate a target image, making the fused target image more natural and realistic. Alternatively, the partial image can be superimposed with the location corresponding to the local region data in the original image to generate a target image.

[0077] Through the above steps, an original image, text content data and text region data are obtained, wherein the text content data is used to describe the text to be rendered in the original image, and the text region data is used to describe the location of the text to be rendered in the original image; local region data is constructed based on the text region data, wherein the local region data is used to characterize the location of the region in the original image where the text is to be rendered; the image generation model is controlled by the local region data to generate a local image based on the text content data and the original image, wherein the local image contains text and the size of the local image is the same as the size of the local region data; based on the local region data, the local image is fused with the original image to generate a target image, thereby improving the image generation effect. It is easy to notice that the image generation process can be decoupled into a method of determining the local image in the original image based on the local region data and the text content data, and the local image can be fused with the original image so that the generated target image can contain text of any scale, which is more consistent with the characteristics of the natural scene, and improves the image generation effect, thereby solving the technical problem of poor image generation effect in the related art.

[0078] In the above embodiment of the present application, local area data is used to control the image generation model to generate a local image based on text content data and the original image, including: generating an input image with the same size as the local area data based on noise; constructing image-level conditions based on the original image and text area data, wherein the image-level conditions are used to characterize the background information in the local image and the position where the text needs to be rendered; constructing feature-level conditions based on the text content data, wherein the feature-level conditions are used to characterize the morphological features and content features of the text; inputting the input image, image-level conditions and feature-level conditions into the image generation model to obtain the target noise output by the image generation model; and denoising the input image based on the target noise to obtain the local image.

[0079] The above-mentioned image generation model can be an image generation model, or can be other models used for image generation. The type of the image generation model is not specifically limited here.

[0080] The target noise may be Gaussian noise, and a suitable noise type may be selected according to the specific application scenario.

[0081] In an optional embodiment, the type and intensity of the target noise can be determined first. Then, an appropriate denoising algorithm can be selected to process the input image to reduce or eliminate the impact of the target noise on the image. Once the target noise is removed, denoised local images can be obtained. These local images may have improved detail, clarity, and quality, making the image easier to analyze and process. Denoising helps improve image quality and accuracy, making subsequent image analysis, recognition, and processing tasks more efficient and reliable.

[0082] If the original image is a full image with a size of 512*512, then the input image can be a local area cropped from a line of text, for example, 32*100. The values ​​here are for illustration only and are not intended to be limiting.

[0083] In an optional embodiment, an input image of the same size as the local area data can be sampled from Gaussian noise. The input image generated by noise can help the subsequent image generation model to better generalize to the noise and changes in the real world. By using the input image of the same size as the local area data, it can be ensured that the model can better capture the details when learning local features, thereby improving the performance and robustness of the model. During the inference process, the input can be a Gaussian noise image of the same size. If the image size is 32*100, then Gaussian sampling is performed at each position of 32*100 as input. This is the basic paradigm of the diffusion model. Gaussian noise is used as input during diffusion model inference.

[0084] Building image-level conditions through original image and text region data can help subsequent image generation models better understand the overall background of the image and text location information.

[0085] By constructing feature-level conditions from text content data, we can characterize the morphological and content features of the text, helping the model better understand the characteristics and content of the text. The input image, image-level conditions, and feature-level conditions can be fed into the image generation model. By learning the relationships between the inputs and the influence between the conditions, we can generate effective local images. The image generation model can combine image and text information to better understand and generate local information containing text within the image. By introducing image-level and feature-level conditions, the image generation model can more accurately capture the relationship between image and text, generating local images with greater contextual information and semantic accuracy.

[0086] In an optional embodiment, the above-mentioned image-level conditions can be used to capture the background features of the local image, such as color, texture, etc., and determine the specific area where the text appears, so that when subsequently generating or modifying text, it can be ensured that the text is coordinated with the background and appears in the appropriate position in the original image. Through the above-mentioned feature-level conditions, attention can be paid to the morphological features of the text, such as font size, style, and color, and the content features of the text, such as the length and subject of the text. Through the feature-level conditions, it can be ensured that the generated or modified text is visually similar to the original text in the original image while maintaining content consistency. The input image, image-level conditions, and feature-level conditions can be provided as input to an image generation model. The image generation model can be a deep learning model that can generate or modify an image based on the input conditions. The local image input to the image generation model contains a new text area, whose background is coordinated with the original image, and the morphological and content features of the text are similar to the original text.

[0087] In the above embodiment of the present application, image-level conditions are constructed based on the original image and text area data, including: cropping the original image based on the local area data to obtain a cropped image, wherein the cropped image is used to represent the background information in the local image; based on the text area data, generating a mask image corresponding to the text area data, wherein the pixel value of the mask image is used to represent the position in the original image where the text needs to be rendered; wherein the image-level conditions include the cropped image and the mask image.

[0088] The mask image mentioned above may be an image obtained by performing row-level masking and word-level masking on the text region data.

[0089] The purpose of the above-mentioned row-level mask is to identify the row-level structure of text in text region data. In the context of text recognition or image segmentation, the row-level mask can help identify and segment rows in text region data. Each pixel can indicate whether the pixel is part of a text line or part of the background. The row-level mask can be used to determine the row boundaries of text in text region data, which helps segment and extract text lines and can highlight text lines in the image.

[0090] The word-level mask described above is similar to the row-level mask. It identifies the location of individual words in the original image. However, this mask is more detailed and can not only segment rows but also further segment rows into individual words. Word-level masks can identify word boundaries in text, extract specific words or phrases, and annotate word-level information in images.

[0091] The aforementioned cropped image may be an area cropped from the background image in the original image by the local area data, and the background image is the aforementioned cropped image.

[0092] In an optional embodiment, the original image may be cropped based on the local area data to obtain a cropped image having the same size as the local area data, thereby determining the area in the original image that needs to be updated based on the cropped image.

[0093] In another optional embodiment, a mask image corresponding to the text area data can be created based on the text area data, and the distribution of the text in the original image can be represented by the pixel values ​​of the mask image. Row-level masks and word-level masks can be created based on the text area data. The mask image corresponding to the above text area data can be obtained by mapping the text area data to corresponding pixels and assigning specific mask values ​​to these pixels. The cropped image and the mask image can be aggregated to obtain an image-level condition. In subsequent specific use, the cropped image and the mask image in the image-level condition can be used together to better understand and manipulate the text information in the image.

[0094] In the above embodiments of the present application, feature-level conditions are constructed based on text content data, including: performing image rendering based on the text content data to generate a rendered image; using a recognizer to recognize the text in the rendered image to obtain morphological features of the text, and using a recognizer to recognize the text content data to obtain content features of the text; wherein the feature-level conditions include: morphological features and content features.

[0095] The above-mentioned rendered image may be a pure text image rendered according to text content data, and the rendered image simply contains text.

[0096] The above-mentioned recognizer can be a program or system that can identify specific targets or features by analyzing images or other input data. In this case, the recognizer is used to process the rendered image to extract the morphological features of the text. This may include recognizing features such as the position, size, font, color of the text, as well as recognizing the content or semantic information of the text. The recognizer can also be a Convolutional Recurrent Neural Network (CRNN), an Attentional Scene Text Recognizer (ASTER), or an Automatic, Bidirectional and Iterative Language Modeling Recognizer (ABINet). There is no specific limitation on the type of recognizer here.

[0097] In the fields of image processing and text recognition, a recognizer is an algorithm or model used to identify and extract specific information from an image. In the context you mentioned, a recognizer could be an optical character recognition (OCR), which reads text from an image and converts it into an editable and searchable text format.

[0098] The morphological features described above refer to the visual characteristics of text in a rendered image. These features can be extracted using image recognition or computer vision algorithms. For example, edge detection, shape recognition, or deep learning models can be used to identify the shape, size, and arrangement of text in an image.

[0099] The aforementioned content features refer to the semantic information of the text content, which can be obtained through natural language processing. Content features can be extracted by analyzing the text's theme, keywords, sentiment, etc.

[0100] In an optional embodiment, after the rendered image is acquired, the rendered image may be pre-processed, for example, by resizing, rotation correction, graying, binarization, etc., to improve the accuracy of recognition. The text may be located in the rendered image by detecting the edges, shapes or other features of the text through a recognizer; the text in the rendered image may be segmented into recognizable units, such as characters, words or lines, thereby helping to decompose the text into more easily processable parts. Morphological features, such as strokes, shapes, sizes, etc., may be extracted from the segmented text units. These features are the basis for the recognizer to classify and recognize, thereby obtaining more accurate morphological features of the text. The content features of the text may be extracted by analyzing the subject, keywords, emotional tendencies, etc. of the text, and the morphological features and content features may be summarized to obtain feature-level conditions. In subsequent specific use, the morphological features and content features in the feature-level conditions may be used together to better understand and manipulate the feature information in the image.

[0101] In real-world applications, the performance of a recognizer is affected by a variety of factors, including image quality, text font, and layout complexity. To improve recognition accuracy, a large amount of training data is typically used to train the recognizer, allowing it to adapt to varying text styles and image conditions. Furthermore, deep learning techniques have made recognizers more efficient and accurate in processing complex images and multiple languages.

[0102] By converting the above text content data into a more intuitive image form and extracting morphological and content features from it, we can provide a basis for subsequent analysis and processing. By combining visual and semantic information, we can understand the text content more comprehensively.

[0103] In the above embodiment of the present application, the input image, image-level conditions and feature-level conditions are input into the image generation model to obtain the target noise output by the image generation model, including: inputting the input image, image-level conditions and feature-level conditions into the image generation model to obtain initial noise; denoising the input image based on the initial noise to obtain an initial image; repeatedly performing the steps of inputting the initial image, image-level conditions and feature-level conditions into the image generation model, and denoising the input image based on the initial noise until the number of iterations reaches a preset number; when the number of iterations reaches the preset number, determining that the noise output by the image generation model during this iteration is the target noise.

[0104] The above preset times can be set according to actual needs and are not specifically limited here.

[0105] The principle of the above image generation model is that it usually takes multiple steps to turn an image into complete noise. Therefore, during the inference process, the noise is gradually predicted and subtracted.

[0106] The above-mentioned image generation model is a deep learning model for image restoration and generation.

[0107] In an optional embodiment, the input image, image-level conditions, and feature-level conditions can be input into the image generation model to obtain initial noise. This initial noise is then used to perform denoising on the input image to obtain an initial image. The steps of inputting the initial image, image-level conditions, and feature-level conditions into the image generation model and performing denoising on the input image again are then repeated until the number of iterations reaches a preset number. During the iterative process, different iterations will continuously optimize the denoising process and the image generation model based on the noise generated last time, so as to gradually approach the target noise. Finally, when the number of iterations reaches a preset number, the noise output by the image generation model during this iteration is determined to be the target noise. This method can effectively remove noise from the image and generate a clear image by repeatedly iteratively optimizing the image generation model and the denoising process. At the same time, by introducing image-level conditions and feature-level conditions, the features and details of the original image can be better preserved.

[0108] Optionally, given an input image, image-level conditions, and feature-level conditions, the image generation model can generate a predicted image. After the predicted image is iteratively generated, a difference image between the input image and the predicted image is calculated. By repeatedly inputting the difference image, image-level conditions, and feature-level conditions into the image generation model, the quality of the predicted image can be gradually improved until a preset number of iterations is reached. In the above iterative process, the difference image is determined to be an improved image because it represents the part that needs further improvement in the iteration. By continuously iteratively generating predicted images and calculating difference images, the image generation model can gradually eliminate defects or noise in the input image, thereby generating a clearer and more accurate predicted image. This iterative process can help the model better understand the features and conditions of the input image, thereby improving the effects of image restoration and generation.

[0109] In the above embodiment of the present application, the method also includes: obtaining a sample image, a sample background image of the sample image, sample text content of the sample text contained in the sample image, and a sample text area of ​​the sample text in the sample text in the sample image; constructing sample local area data based on the sample text area; processing the sample image based on the sample local area data and the sample noise to generate a sample input image; constructing a sample image-level condition based on the sample background image and the sample text area data, wherein the sample image-level condition is used to characterize the background information and the position of the sample text in the predicted image; constructing a sample feature-level condition based on the sample text content, wherein the sample feature-level condition is used to characterize the sample morphological features and sample content features of the sample text; inputting the sample input image, the sample image-level condition and the sample feature-level condition into the initial model to obtain the predicted noise output by the initial model; constructing the total loss function of the initial model based on the sample noise, the predicted noise and the sample input image; adjusting the initial model based on the total loss function to obtain an image generation model.

[0110] The sample image may be a pre-set image for training the image generation model, the sample background image of the sample image may be a pre-cropped background image, and the sample text content may be the content contained in the pre-determined sample image.

[0111] The sample words in the sample text mentioned above may be pre-marked in the sample text area in the sample image.

[0112] In an optional embodiment, sample local area data can be constructed based on the sample text area, sample local area data can be generated based on the horizontal circumscribed rectangle of the sample text area, and the sample image can be processed based on the sample local area data and sample noise to obtain a sample input image.

[0113] Sample image-level conditions can be constructed based on sample background images and sample text region data, which can help subsequent image generation models better understand the overall context and text position information of the sample background image, so as to characterize the background information and the position of sample words in the predicted image.

[0114] Sample feature-level conditions can be constructed based on the sample text content to characterize the sample morphological features and sample content features of the sample text, which can help the subsequent initial model better understand the overall background and text location information of the sample image. A total loss function for the initial model can be constructed based on the sample noise, predicted noise, and sample input image to represent the difference between the predicted noise obtained from the sample input image and the sample noise. The initial model can then be adjusted based on this total loss function to obtain a more accurate image generation model.

[0115] In the above embodiment of the present application, the sample image is processed based on the sample local area data and the sample noise to generate a sample input image, including: cropping the sample image based on the sample local area data to obtain a local sample image; adding sample noise of the same size as the sample local area data to the local sample image to obtain a sample input image.

[0116] In an optional embodiment, the sample image may be cropped based on the sample local area data to obtain a local sample image of the same size as the sample local area data, thereby determining the area in the sample image that needs to be updated based on the local sample image.

[0117] Furthermore, a sample noise of the same size as the sample local area data can be sampled from the Gaussian noise and used as the sample input image. Generating a sample input image from noise can help train the subsequent initial model to better generalize to real-world noise and variations. By using the same sample input image as the sample local area data, the model can better capture detailed information when learning local features, thereby improving the performance and robustness of the model.

[0118] In the above embodiment of the present application, a total loss function of the initial model is constructed based on sample noise, predicted noise, and sample input image, including: generating a noise loss function based on predicted noise and sample noise; denoising the sample input image based on preset noise to obtain a denoised image; generating a text loss based on text information in the sample input image and text information in the denoised image; generating a background loss based on background information in the sample input image and background information in the denoised image; and performing weighted operations on the noise loss function, text loss, and background loss to obtain a total loss function.

[0119] In an optional embodiment, a noise loss function can be generated based on the predicted noise and the sample noise to evaluate the noise processing capability of the initial model, and the denoised image can be obtained by obtaining the difference between the sample input image and the predicted noise. The text information in the sample input image and the text information in the restored image can be used to generate a text loss function to help the model better preserve the text content in the restored image and not lose important information during the denoising process. Similarly, a background loss function can be generated based on the background information in the sample input image and the background information in the restored image to help the model better preserve the background structure and texture of the image and ensure that the image maintains its overall integrity during the denoising process.

[0120] By performing weighted operations on the noise loss function, text loss, and background loss, the total loss function is obtained. By minimizing the total loss function, the model parameters of the initial model can be adjusted so that the model can better remove noise from the image and retain important text and background information, thereby obtaining a clear and complete image, thereby training an image generation model with higher accuracy and better image generation effect.

[0121] In the above embodiment of the present application, after constructing local area data based on text area data, the method also includes: outputting the local area data; upon receiving a first feedback result corresponding to the local area data, using the first feedback result, controlling the image generation model to generate a local image based on the text content data and the original image, wherein the first feedback result is a result obtained by modifying the local area data.

[0122] In an optional embodiment, after constructing the local region data based on the text region data, the local region data can be output so that the user can check whether the local region data is accurate. If the local region data is accurate, the local region data can be adjusted without adjustment, and the image generation model can be directly controlled based on the local region data to generate the local image based on the text content data and the original image. If the local region data is inaccurate, or the user has modified the local region data, a first feedback result corresponding to the local region data can be received, and based on the first feedback result, the image generation model can be controlled to generate the local image based on the text content data and the original image.

[0123] In the above embodiment of the present application, after using local area data to control the image generation model to generate a local image based on text content data and the original image, the method also includes: outputting the local image; upon receiving a second feedback result corresponding to the local image, based on the local area data, fusing the second feedback result with the original image to generate a target image, wherein the second feedback result is a result obtained by modifying the local image.

[0124] In an optional embodiment, after using the local area data to control the image generation model to generate a local image based on the text content data and the original image, or after using the first feedback result to control the image generation model to generate a local image based on the text content data and the original image, the local image can be output so that the user can check whether the local image is accurate. If the local image is accurate, the local image can be directly fused with the original image based on the local area data to generate a target image without adjusting the local image. If the local image is inaccurate, or the user has modified the local image, a second feedback result corresponding to the local image can be received, and the second feedback result can be fused with the original image based on the local area data to generate the target image.

[0125] In the above embodiment of the present application, after the local image is fused with the original image based on the local area data to generate the target image, the method also includes: outputting the target image; receiving a third feedback result corresponding to the target image, wherein the third feedback result is a result obtained by modifying the target image; adjusting the image generation model based on the original image, text content data, text area data and the third feedback result.

[0126] In an optional embodiment, after the local image is fused with the original image based on the local area data to generate a target image, or after the second feedback result is fused with the original image based on the local area data to generate a target image, the target image can be output so that the user can check whether the target image is accurate. If the target image is accurate, the target image can be adjusted without adjustment, and the image generation model can be adjusted directly based on the original image, text content data, text area data and target image. If the target image is inaccurate, or the user has modified the target image, a third feedback result corresponding to the target image can be received, and the image generation model can be adjusted based on the original image, text content data, text area data and the third feedback result.

[0127] Figure 3 is a schematic diagram of a text image processing structure according to an embodiment of the present application, such as Figure 3 As shown, the specific training process is as follows:

[0128] Input: It can be a sample image, a sample background image, a sample text area in the sample image, and sample text content. The text area represented in the sample text area may include line-level or word-level quadrilateral coordinates.

[0129] Constructing local region data: A horizontal bounding rectangle may be generated based on the sample text region, and the rectangle boundary may be expanded by 10% to obtain the sample local region data.

[0130] Constructing the input image: You can crop the sample local area box and the sample image to obtain a local sample image. Add Gaussian noise of the same size to the local sample image to obtain the basic input image for training the image generation model, that is, the sample input image.

[0131] Constructing image-level conditions: The sample background image of the sample local area data can be cropped to obtain the background image area, and the row-level mask and word-level mask of the same size can be generated based on the sample text area. The background image area, row-level mask and word-level mask are new conditions proposed, which can be called image-level conditions. The image-level conditions are used to provide accurate background information, text row position information and word position information.

[0132] Constructing feature-level conditions: Based on the sample text content, you can input the sample text content and a rendered image of the plain text rendered with the sample text content into the recognizer. The recognizer's fixed parameters are not updated, resulting in image features and text features. Image features and text features are new conditions, referred to as feature-level conditions, that provide the morphological and content features of the generated text, respectively.

[0133] Training Model: Sample input images, image-level conditions, and feature-level conditions can be input into the image generation model training process. The training process here follows the general training method of the image generation model. The output is the noise added to the image area, and the noise is supervised by the loss function of the general image generation model. In addition, two auxiliary loss functions are proposed in this application, namely foreground loss and background loss, which are used to improve the generation quality of foreground text content and the restoration system of background images, respectively, to improve the overall quality of the generated text images.

[0134] The above process can be repeated until the model converges to obtain a trained image generation model.

[0135] The process of obtaining the target image based on the reasoning of the original image is as follows:

[0136] Input: It can be the original image, background image, text area data in the original image, and text content data. The text area represented in the text area data may include line-level or word-level quadrilateral box coordinates.

[0137] Constructing local area data: A horizontal bounding rectangle may be generated based on the text area, and the rectangle boundary may be expanded by 10% to obtain the local area data.

[0138] Construct the input image: You can crop the local region frame and the original image to obtain the image region, add Gaussian noise of the same size to the image region, and obtain the basic input image for training the image generation model, that is, the input image.

[0139] Constructing image-level conditions: The background image of the local area data can be cropped to obtain the background image area, and row-level masks and word-level masks of the same size can be generated based on the text area. The background image area, row-level mask, and word-level mask are new conditions proposed, which can be called image-level conditions. The image-level conditions are used to provide accurate background information, text line position information, and word position information.

[0140] Constructing feature-level conditions: Based on the text content data, the text content data and a rendered image of the plain text rendered with the text content data can be input into the recognizer. The fixed recognizer parameters are not updated, resulting in image features and text features. Image features and text features are new conditions, which can be called feature-level conditions. They respectively provide the morphological and content features of the generated text.

[0141] like Figure 3 As shown in Figure 1, the input image, image-level conditions, and feature-level conditions can be fed into the image generation model to obtain a round of noise predictions. The noise predictions obtained by subtracting the input noise are then fed into the image generation model with the image-level conditions and feature-level conditions to obtain a new round of noise predictions. This process is repeated 100 times. The final subtraction results in the generated local image, which can be fused back into the background image according to the local region data.

[0142] Training Model: The noisy input image, image-level conditions, and feature-level conditions can be input into the image generation model training process. The training process here follows the general training method of the image generation model. The output is the noise added to the image area, and the noise is supervised by the loss function of the general image generation model. In addition, two auxiliary loss functions are proposed in this application, namely foreground loss and background loss, which are used to improve the generation quality of foreground text content and the restoration system of background images, respectively, to improve the overall quality of the generated text image.

[0143] The above process can be repeated until the model converges to obtain a trained image generation model.

[0144] Figure 4 is a schematic diagram of a processing structure of a visualized text image according to an embodiment of the present application, such as Figure 4 As shown, the text region in the natural scene image can be cropped according to the text region to obtain the text region in the text image, and the background image in the text image can be cropped according to the text difference to obtain the background region. The line-level mask and word-level mask can be determined according to the text region, and the image-level condition can be constructed according to the background region, the line-level mask and the word-level mask. The image can be rendered according to the text content to obtain the rendered image, and the rendered image can be processed by a fixed parameter recognizer to obtain the morphological features of the text, that is, Figure 4 The image features shown in are used to obtain text features, and feature-level conditions can be constructed based on the image features and text features. The text region, image-level conditions, and feature-level conditions can be input into the image generation model to obtain a predicted image. A foreground loss and background loss can be constructed based on the text region and the predicted image, and the image generation model can be trained based on the foreground loss and background loss. The difference between the text region and the predicted image can be obtained to obtain a difference image, and the steps of inputting the difference image, image-level conditions, and feature-level conditions into the image generation model and obtaining the difference between the difference image and the predicted image are repeated until the number of iterations reaches a preset number; the difference image is determined to be a local image, and the local image can be fused with the original image based on the local region data to generate the target image.

[0145] Figure 5 is a training flow chart of an image generation model according to an embodiment of the present application, such as Figure 5 As shown, the method includes:

[0146] Step S502 : Acquire a sample image, a sample background image of the sample image, sample text content of a sample text contained in the sample image, and a sample text area of ​​sample words in the sample text in the sample image.

[0147] Step S504: construct sample local area data based on the sample text area.

[0148] Step S506 : Process the sample image based on the sample local area data and the sample noise to generate a sample input image.

[0149] Step S508: constructing sample image-level conditions based on the sample background image and the sample text region data.

[0150] Among them, the sample image-level condition is used to characterize the background information and the position of the sample words in the predicted image.

[0151] Step S510: constructing sample feature-level conditions based on the sample text content.

[0152] Among them, the sample feature level conditions are used to characterize the sample morphological features and sample content features of the sample text.

[0153] Step S512: input the sample input image, the sample image-level condition, and the sample feature-level condition into the initial model to obtain the predicted noise output by the initial model.

[0154] Step S514: constructing a total loss function of the initial model based on the sample noise, the prediction noise, and the sample input image.

[0155] Step S516: Adjust the initial model based on the total loss function to obtain an image generation model.

[0156] Compared to rule-based rendering methods for text image generation, this application eliminates the need to collect a wide variety of fonts, colors, backgrounds, and other materials, nor does it require manually set rules for text layout and attribute selection. Furthermore, the multiple conditions constructed contribute to the quality of foreground text and background images, greatly improving the authenticity and usability of the generated images.

[0157] Compared to text image generation solutions based on diffusion models, this application decouples text image generation into local text image generation and fusion, allowing the generated text to be of any scale and more closely resembling the text image characteristics found in real natural scenes. Furthermore, the introduction of background image regions and background loss allows the generated image to blend naturally into the complete background image.

[0158] This application decouples text image generation into local text image generation and fusion, allowing the generated images to contain text of any scale, more accurately reflecting the characteristics of real natural scenes and far exceeding the realism of previous rendering-based and diffusion-based methods. Furthermore, the generated text images can significantly improve the performance of OCR-related models, solving text image detection and recognition problems in real natural scenes, and possessing extremely high usability.

[0159] This application constructs two categories of conditional training: image-level conditions and feature-level conditions. Image-level conditions include background region conditions, line mask conditions, and word mask conditions; feature-level conditions include image feature conditions and content feature conditions. Furthermore, this application proposes two auxiliary loss functions: foreground text loss and background image loss, to improve the quality of generated text images.

[0160] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0161] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0162] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0163] Example 2

[0164] According to an embodiment of the present application, an image generation method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0165] Figure 6 is a flow chart of an image generation method according to embodiment 2 of the present application, such as Figure 6 As shown, the method includes:

[0166] Step S602: Capture a scene image of a real scene, and obtain text content data and text area data.

[0167] The text content data is used to describe the text that needs to be rendered in the scene image, and the text region data is used to describe the position of the words in the text that need to be rendered in the scene image.

[0168] The above-mentioned scene images of real scenes may include but are not limited to remote sensing images, photos, scanned images, etc.

[0169] In an optional embodiment, a scene image of a real scene can be captured by a related device, and the text to be rendered in the scene image and the positions of the words in the text in the scene image can be obtained. The above-mentioned text content data and text area data can be set by the user or according to needs, and are not limited here.

[0170] Step S604: construct local region data based on the text region data.

[0171] The local area data is used to represent the location of the area where the text needs to be rendered in the scene image.

[0172] Step S606: Using the local area data, the image generation model is controlled to generate a local image based on the text content data and the scene image.

[0173] The local image contains text, and the size of the local image is the same as the size of the local area data.

[0174] Step S608: Based on the local area data, the local image is fused with the scene image to generate a target image.

[0175] Through the above steps, a scene image of a real scene is captured, and text content data and text region data are obtained, wherein the text content data is used to describe the text to be rendered in the scene image, and the text region data is used to describe the location of the words in the text to be rendered in the scene image; local region data is constructed based on the text region data, wherein the local region data is used to characterize the location of the region in the original image where the text is to be rendered; using the local region data, an image generation model is controlled to generate a local image based on the text content data and the scene image, wherein the local image contains text and the size of the local image is the same as the size of the local region data; based on the local region data, the local image is fused with the scene image to generate a target image, thereby achieving improved image generation effect. It is easy to notice that the image generation process can be decoupled into a method of determining the local image in the original image based on the local region data and the text content data, and the local image can be fused with the original image so that the generated target image can contain text of any scale, which is more consistent with the characteristics of the natural scene, and improves the image generation effect, thereby solving the technical problem of poor image generation effect in the related art.

[0176] In the above embodiments of the present application, local area data is used to control the image generation model to generate a local image based on text content data and scene images, including: generating an input image of the same size as the local area data based on noise; constructing image-level conditions based on the scene image and text area data, wherein the image-level conditions are used to characterize the background information in the local image and the position where the text needs to be rendered; constructing feature-level conditions based on the text content data, wherein the feature-level conditions are used to characterize the morphological features and content features of the text; inputting the input image, image-level conditions and feature-level conditions into the image generation model to obtain the target noise output by the image generation model; and denoising the input image based on the target noise to obtain the local image.

[0177] Example 3

[0178] According to an embodiment of the present application, an image generation method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0179] Figure 7 is a flow chart of an image generation method according to Example 3 of the present application, such as Figure 7 As shown, the method includes:

[0180] Step S702 : In response to an input instruction on the operation interface, the original image, text content data, and text area data are displayed on the operation interface.

[0181] The text content data is used to describe the text that needs to be rendered in the original image, and the text area data is used to describe the location of the area where the text needs to be rendered in the original image.

[0182] The above-mentioned operation interface can be used to display the original image, wherein the operation interface can include a variety of different controls for the user to operate, thereby displaying the original image on the operation interface.

[0183] Step S704 : responding to the generation instruction on the operation interface, displaying the target image on the operation interface.

[0184] Among them, the target image is an image obtained by fusing the local image with the original image based on the local area data. The local area data is constructed based on the text area data. The local area data is used to represent the area in the original image where text needs to be rendered. The local image is an image generated based on the text content data and the original image by controlling the image generation model using the local area data. The local image contains text, and the size of the local image is the same as the size of the local area data.

[0185] The above-mentioned generation instruction may be an instruction for fusing a local image with an original image. According to the generation instruction, the local area data may be first used to control the image generation model to generate a local image based on the text content data and the original image, and then the local image and the original image may be fused based on the local area data to obtain the target image.

[0186] Through the above steps, in response to the input instruction acting on the operation interface, the original image, text content data and text area data are displayed on the operation interface, wherein the text content data is used to describe the text that needs to be rendered in the original image, and the text area data is used to describe the position of the words in the text that need to be rendered in the original image; in response to the generation instruction acting on the operation interface, the target image is displayed on the operation interface, wherein the target image is an image obtained by fusing the local image with the original image based on the local area data, the local area data is constructed based on the text area data, the local area data is used to characterize the position of the area where the text needs to be rendered in the original image, the local image is an image generated based on the text content data and the original image by controlling the image generation model using the local area data, the local image contains text, and the size of the local image is the same as the size of the local area data, thereby improving the image generation effect. It is easy to notice that the image generation process can be decoupled into a method of determining the local image in the original image based on local area data and text content data, and the local image can be fused with the original image so that the generated target image can contain text of any scale, which is more in line with the characteristics of natural scenes, improves the image generation effect, and thus solves the technical problem of poor image generation effect in related technologies.

[0187] Example 4

[0188] According to an embodiment of the present application, an image generation method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0189] Figure 8 is a flow chart of an image generation method according to Example 4 of the present application, such as Figure 8 As shown, the method includes:

[0190] Step S802: Acquire the original image, text content data, and text area data by calling the first interface.

[0191] Among them, the first interface includes a first parameter, the parameter value of the first parameter includes the original image, text content data and text area data, the text content data is used to describe the text that needs to be rendered in the original image, and the text area data is used to describe the location where the text needs to be rendered in the original image.

[0192] The above-mentioned first interface can be an interface for data interaction between the cloud server and the client. The client can pass the original image into the interface function as the first parameter of the interface function to achieve the purpose of uploading the original image to the cloud server.

[0193] Step S804: construct local region data based on the text region data.

[0194] The local area data is used to represent the location of the area where the text needs to be rendered in the original image.

[0195] Step S806: Using the local area data, the image generation model is controlled to generate a local image based on the text content data and the original image.

[0196] The local image contains text, and the size of the local image is the same as the size of the local area data.

[0197] Step S808: Based on the local area data, the local image is fused with the original image to generate a target image.

[0198] Step S810: Output the target image by calling the second interface.

[0199] The second interface includes a second parameter, and a parameter value of the second parameter includes a target image.

[0200] The above-mentioned second interface can be an interface for data interaction between the cloud server and the client. The cloud server can pass the target image into the interface function as the second parameter of the interface function to achieve the purpose of sending the target image to the client.

[0201] Through the above steps, the original image, text content data and text area data are obtained by calling the first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the original image, text content data and text area data, the text content data is used to describe the text that needs to be rendered in the original image, and the text area data is used to describe the position of the area where the text needs to be rendered in the original image; local area data is constructed based on the text area data, wherein the local area data is used to characterize the area in the original image where the text needs to be rendered; using the local area data, the image generation model is controlled to generate a local image based on the text content data and the original image, wherein the local image contains text and the size of the local image is the same as the size of the local area data; based on the local area data, the local image is fused with the original image to generate a target image; the target image is output by calling the second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter includes the target image, thereby improving the image generation effect. It is easy to notice that the image generation process can be decoupled into a method of determining the local image in the original image based on local area data and text content data, and the local image can be fused with the original image so that the generated target image can contain text of any scale, which is more in line with the characteristics of natural scenes, improves the image generation effect, and thus solves the technical problem of poor image generation effect in related technologies.

[0202] Example 5

[0203] According to an embodiment of the present application, an image generating device for implementing the above-mentioned image generating method is also provided. Figure 9 is a schematic diagram of an image generating device according to embodiment 5 of the present application, such as Figure 9 As shown, the device 900 includes: an acquisition module 902 , a construction module 904 , a generation module 906 , and a fusion module 908 .

[0204] Among them, the acquisition module is used to obtain the original image, text content data and text area data, wherein the text content data is used to describe the text that needs to be rendered in the original image, and the text area data is used to describe the position where the text needs to be rendered in the original image; the construction module is used to construct local area data based on the text area data, wherein the local area data is used to characterize the position of the area where the text needs to be rendered in the original image; the generation module is used to use the local area data to control the image generation model to generate a local image based on the text content data and the original image, wherein the local image contains text and the size of the local image is the same as the size of the local area data; the fusion module is used to fuse the local image with the original image based on the local area data to generate a target image.

[0205] It should be noted that the acquisition module 902, construction module 904, generation module 906, and fusion module 908 correspond to steps S202 to S208 in Example 1. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The above modules can also be run as part of the device in the server 10 provided in Example 1.

[0206] In the above embodiment of the present application, the generation module is used to generate an input image of the same size as the local area data based on noise; based on the original image and text area data, an image-level condition is constructed, wherein the image-level condition is used to characterize the background information in the local image and the location where the text needs to be rendered; based on the text content data, a feature-level condition is constructed, wherein the feature-level condition is used to characterize the morphological features and content features of the text; the input image, the image-level condition and the feature-level condition are input into the image generation model to obtain the target noise output by the image generation model; the input image is denoised based on the target noise to obtain a local image.

[0207] In the above embodiment of the present application, the generation module is used to crop the original image based on the local area data to obtain a cropped image, wherein the cropped image is used to represent the background information in the local image; based on the text area data, a mask image corresponding to the text area data is generated, wherein the pixel value of the mask image is used to represent the position in the original image where the text needs to be rendered; wherein the image-level condition includes the cropped image and the mask image.

[0208] In the above embodiment of the present application, the generation module is used to perform image rendering based on text content data to generate a rendered image; the text in the rendered image is recognized using an identifier to obtain the morphological features of the text, and the text content data is recognized using an identifier to obtain the content features of the text; wherein the feature-level conditions include: morphological features and content features.

[0209] In the above embodiment of the present application, the generation module is used to input the input image, image-level conditions and feature-level conditions into the image generation model to obtain initial noise; denoise the input image based on the initial noise to obtain an initial image; repeatedly perform the steps of inputting the initial image, image-level conditions and feature-level conditions into the image generation model, and denoising the input image based on the initial noise until the number of iterations reaches a preset number; when the number of iterations reaches the preset number, determine that the noise output by the image generation model during this iteration is the target noise.

[0210] In the above embodiments of the present application, the device further includes: an input module and an adjustment module.

[0211] Among them, the acquisition module is used to obtain the sample image, the sample background image of the sample image, the sample text content of the sample text contained in the sample image, and the sample text area of ​​the sample text in the sample text in the sample image; the processing module is used to process the sample image based on the sample local area data and the sample noise to generate a sample input image; the construction module is used to construct the sample image-level conditions based on the sample background image and the sample text area data, wherein the sample image-level conditions are used to characterize the background information and the position of the sample text in the predicted image; the construction module is used to construct the sample feature-level conditions based on the sample text content, wherein the sample feature-level conditions are used to characterize the sample morphological features and sample content features of the sample text; the input module is used to input the sample input image, the sample image-level conditions and the sample feature-level conditions into the initial model to obtain the prediction noise output by the initial model; the construction module is used to construct the total loss function of the initial model based on the sample noise, the prediction noise and the sample input image; the adjustment module is used to adjust the initial model based on the total loss function to obtain the image generation model.

[0212] In the above embodiment of the present application, the generation module is used to crop the sample image based on the sample local area data to obtain a local sample image; and add sample noise of the same size as the sample local area data to the local sample image to obtain a sample input image.

[0213] In the above embodiments of the present application, the construction module is used to generate a noise loss function based on the predicted noise and the sample noise; denoise the sample input image based on the preset noise to obtain a denoised image; generate a text loss based on the text information in the sample input image and the text information in the denoised image; generate a background loss based on the background information in the sample input image and the background information in the denoised image; and perform weighted operations on the noise loss function, the text loss, and the background loss to obtain a total loss function.

[0214] In the above embodiments of the present application, the device further includes: an output module and a receiving module.

[0215] Among them, the output module is used to output local area data; the receiving module is used to control the image generation model to generate a local image based on the text content data and the original image using the first feedback result corresponding to the local area data when receiving the first feedback result, wherein the first feedback result is the result obtained by modifying the local area box.

[0216] In the above embodiment of the present application, the output module is used to output a local image; the receiving module is used to, upon receiving a second feedback result corresponding to the local image, fuse the second feedback result with the original image based on the local area data to generate a target image, wherein the second feedback result is a result obtained by modifying the local image.

[0217] In the above embodiment of the present application, the output module is used to output the target image; the receiving module is used to receive the third feedback result corresponding to the target image, wherein the third feedback result is the result obtained by modifying the target image; the adjustment module is used to adjust the image generation model based on the original image, text content data, text area data and the third feedback result.

[0218] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0219] Example 6

[0220] According to an embodiment of the present application, an image generating device for implementing the above-mentioned image generating method is also provided. Figure 10 is a schematic diagram of an image generating device according to Example 6 of the present application, such as Figure 10As shown, the device 1000 includes: a capture module 1002 , a construction module 1004 , a generation module 1006 , and a fusion module 1008 .

[0221] Among them, the capture module is used to capture the scene image of the real scene and obtain text content data and text area data, wherein the text content data is used to describe the text that needs to be rendered in the scene image, and the text area data is used to describe the position of the words in the text that need to be rendered in the scene image; the construction module is used to construct local area data based on the text area data, wherein the local area data is used to characterize the position of the area where the text needs to be rendered in the scene image; the generation module is used to use the local area data to control the image generation model to generate a local image based on the text content data and the scene image, wherein the local image contains text and the size of the local image is the same as the size of the local area data; the fusion module is used to fuse the local image with the scene image based on the local area data to generate a target image.

[0222] It should be noted that the capture module 1002, construction module 1004, generation module 1006, and fusion module 1008 correspond to steps S302 to S308 in Example 2. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The above modules can also be run as part of the device in the server 10 provided in Example 1.

[0223] In the above embodiment of the present application, the generation module is also used to generate an input image of the same size as the local area data based on noise; construct image-level conditions based on the scene image and text area data, wherein the image-level conditions are used to characterize the background information in the local image and the location where the text needs to be rendered; construct feature-level conditions based on the text content data, wherein the feature-level conditions are used to characterize the morphological features and content features of the text; input the input image, image-level conditions and feature-level conditions into the image generation model to obtain the target noise output by the image generation model; and denoise the input image based on the target noise to obtain a local image.

[0224] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0225] Example 7

[0226] According to an embodiment of the present application, an image generating device for implementing the above-mentioned image generating method is also provided. Figure 11is a schematic diagram of an image generating device according to Example 7 of the present application, such as Figure 11 As shown, the device 1100 includes: a first display module 1102 and a second display module 1104 .

[0227] Among them, the first display module is used to respond to the input instruction acting on the operation interface, and display the original image, text content data and text area data on the operation interface, wherein the text content data is used to describe the text that needs to be rendered in the original image, and the text area data is used to describe the position of the words in the text that need to be rendered in the original image; respond to the generation instruction acting on the operation interface, and display the target image on the operation interface, wherein the target image is based on the local area data, and the image is obtained by fusing the local image with the original image, the local area data is constructed based on the text area data, and the local area data is used to characterize the position of the area where the text needs to be rendered in the original image, and the local image is generated by using the local area data to control the image generation model based on the text content data and the original image. The local image contains text, and the size of the local image is the same as the size of the local area data.

[0228] It should be noted that the first display module 1102 and the second display module 1104 correspond to steps S402 to S404 in Example 3. The examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The above modules can also be run as part of the device in the server 10 provided in Example 1.

[0229] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0230] Example 8

[0231] According to an embodiment of the present application, an image generating device for implementing the above-mentioned image generating method is also provided. Figure 12 is a schematic diagram of an image generating device according to Example 8 of the present application, such as Figure 12 As shown, the device 1200 includes: an acquisition module 1202 , a construction module 1204 , a generation module 1206 , a fusion module 1208 , and an output module 1210 .

[0232] Among them, the acquisition module is used to obtain the original image, text content data and text area data by calling the first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the original image, text content data and text area data, the text content data is used to describe the text that needs to be rendered in the original image, and the text area data is used to describe the position where the text needs to be rendered in the original image; the construction module is used to construct local area data based on the text area data, wherein the local area data is used to characterize the position of the area where the text needs to be rendered in the original image; the generation module is used to use the local area data to control the image generation model to generate a local image based on the text content data and the original image, wherein the local image contains text and the size of the local image is the same as the size of the local area data; the fusion module is used to fuse the local image with the original image based on the local area data to generate a target image; the output module is used to output the target image by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target image.

[0233] It should be noted that the acquisition module 1202, construction module 1204, generation module 1206, fusion module 1208, and output module 1210 described above correspond to steps S502 to S510 in Example 4. The examples and application scenarios implemented by the five modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The above modules can also be run as part of the device in the server 10 provided in Example 1.

[0234] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0235] Example 9

[0236] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.

[0237] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0238] In this embodiment, the electronic device can execute the program code in the method.

[0239] Optionally, Figure 13 This is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 13As shown, the electronic device A may include: one or more (only one is shown in the figure) processors 102, a memory 104, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module and a display.

[0240] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories may be connected to terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0241] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtaining the original image, text content data and text area data, wherein the text content data is used to describe the text that needs to be rendered in the original image, and the text area data is used to describe the position where the text needs to be rendered in the original image; constructing local area data based on the text area data, wherein the local area data is used to characterize the position of the area where the text needs to be rendered in the original image; using the local area data, controlling the image generation model to generate a local image based on the text content data and the original image, wherein the local image contains text and the size of the local image is the same as the size of the local area data; based on the local area data, the local image is merged with the original image to generate a target image.

[0242] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: capture a scene image of a real scene, and obtain text content data and text area data, wherein the text content data is used to describe the position of the area where the text needs to be rendered in the scene image, and the text area data is used to describe the position of the words in the text need to be rendered in the scene image; construct local area data based on the text area data, wherein the local area data is used to characterize the area in the original image where the text needs to be rendered; use the local area data to control the image generation model to generate a local image based on the text content data and the scene image, wherein the local image contains text and the size of the local image is the same as the size of the local area data; based on the local area data, the local image is fused with the scene image to generate a target image.

[0243] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: responding to the input instruction on the operation interface, displaying the original image, text content data and text area data on the operation interface, wherein the text content data is used to describe the text that needs to be rendered in the original image, and the text area data is used to describe the position of the area where the text needs to be rendered in the original image; responding to the generation instruction on the operation interface, displaying the target image on the operation interface, wherein the target image is an image obtained by fusing the local image with the original image based on the local area data, the local area data is constructed based on the text area data, the local area data is used to characterize the area in the original image where the text needs to be rendered, and the local image is an image generated based on the text content data and the original image by controlling the image generation model using the local area data, the local image contains text, and the size of the local image is the same as the size of the local area data.

[0244] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the original image, text content data and text area data by calling the first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes the original image, text content data and text area data, the text content data is used to describe the text that needs to be rendered in the original image, and the text area data is used to describe the position where the text needs to be rendered in the original image; construct local area data based on the text area data, wherein the local area data is used to characterize the position of the area where the text needs to be rendered in the original image; use the local area data to control the image generation model to generate a local image based on the text content data and the original image, wherein the local image contains text and the size of the local image is the same as the size of the local area data; based on the local area data, fuse the local image with the original image to generate a target image; output the target image by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target image.

[0245] According to an embodiment of the present application, an original image, text content data and text area data are obtained, wherein the text content data is used to describe the text to be rendered in the original image, and the text area data is used to describe the position of the text to be rendered in the original image; local area data is constructed based on the text area data, wherein the local area data is used to characterize the position of the area where the text is to be rendered in the original image; the image generation model is controlled by using the local area data to generate a local image based on the text content data and the original image, wherein the local image contains text and the size of the local image is the same as the size of the local area data; based on the local area data, the local image is fused with the original image to generate a target image, thereby improving the image generation effect. It is easy to notice that the image generation process can be decoupled into a method of determining the local image in the original image based on the local area data and the text content data, and the local image can be fused with the original image so that the generated target image can contain text of any scale, which is more in line with the characteristics of the natural scene, and improves the image generation effect, thereby solving the technical problem of poor image generation effect in the related art.

[0246] Those skilled in the art will understand that Figure 13 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Figure 13 It does not limit the structure of the above electronic device. For example, the electronic device A may also include more or fewer components (such as network interfaces, display devices, etc.) than those shown in the figure, or have the same Figure 13 Different configurations shown.

[0247] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0248] Example 10

[0249] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method provided in the above embodiment.

[0250] Optionally, in this embodiment, the storage medium may be located in any electronic device in a group of electronic devices in a computer network, or in any mobile terminal in a group of mobile terminals.

[0251] Optionally, in this embodiment, a computer-readable storage medium is configured to store program code for executing the following steps: obtaining an original image, text content data, and text area data, wherein the text content data is used to describe the text that needs to be rendered in the original image, and the text area data is used to describe the location where the text needs to be rendered in the original image; constructing local area data based on the text area data, wherein the local area data is used to characterize the location of the area where the text needs to be rendered in the original image; using the local area data, controlling the image generation model to generate a local image based on the text content data and the original image, wherein the local image contains text and the size of the local image is the same as the size of the local area data; based on the local area data, fusing the local image with the original image to generate a target image.

[0252] Optionally, in this embodiment, a computer-readable storage medium is configured to store program code for performing the following steps: capturing a scene image of a real scene, and obtaining text content data and text area data, wherein the text content data is used to describe the location of the area in the scene image where the text needs to be rendered, and the text area data is used to describe the location of the words in the text where the text needs to be rendered in the scene image; constructing local area data based on the text area data, wherein the local area data is used to characterize the area in the original image where the text needs to be rendered; using the local area data, controlling the image generation model to generate a local image based on the text content data and the scene image, wherein the local image contains text and the size of the local image is the same as the size of the local area data; based on the local area data, fusing the local image with the scene image to generate a target image.

[0253] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: in response to an input instruction acting on an operation interface, displaying an original image, text content data and text area data on the operation interface, wherein the text content data is used to describe the text to be rendered in the original image, and the text area data is used to describe the position of the area in the original image where the text needs to be rendered; in response to a generation instruction acting on the operation interface, displaying a target image on the operation interface, wherein the target image is an image obtained by fusing a local image with the original image based on the local area data, the local area data is constructed based on the text area data, the local area data is used to characterize the area in the original image where the text needs to be rendered, the local image is an image generated based on the text content data and the original image by controlling the image generation model using the local area data, the local image contains text, and the size of the local image is the same as the size of the local area data.

[0254] Optionally, in this embodiment, a computer-readable storage medium is configured to store program code for executing the following steps: obtaining an original image, text content data, and text area data by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes the original image, text content data, and text area data, the text content data is used to describe the text to be rendered in the original image, and the text area data is used to describe the position in the original image where the text needs to be rendered; constructing local area data based on the text area data, wherein the local area data is used to characterize the position of the area in the original image where the text needs to be rendered; using the local area data, controlling the image generation model to generate a local image based on the text content data and the original image, wherein the local image contains text and the size of the local image is the same as the size of the local area data; based on the local area data, fusing the local image with the original image to generate a target image; outputting the target image by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target image.

[0255] According to an embodiment of the present application, an original image, text content data and text area data are obtained, wherein the text content data is used to describe the text to be rendered in the original image, and the text area data is used to describe the position of the text to be rendered in the original image; local area data is constructed based on the text area data, wherein the local area data is used to characterize the position of the area where the text is to be rendered in the original image; the image generation model is controlled by using the local area data to generate a local image based on the text content data and the original image, wherein the local image contains text and the size of the local image is the same as the size of the local area data; based on the local area data, the local image is fused with the original image to generate a target image, thereby improving the image generation effect. It is easy to notice that the image generation process can be decoupled into a method of determining the local image in the original image based on the local area data and the text content data, and the local image can be fused with the original image so that the generated target image can contain text of any scale, which is more in line with the characteristics of the natural scene, and improves the image generation effect, thereby solving the technical problem of poor image generation effect in the related art.

[0256] Example 11

[0257] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and when the computer program is executed by a processor, the method provided in the embodiment is implemented.

[0258] Example 12

[0259] The embodiments of the present application further provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which may be used to store a computer program that, when executed by a processor, implements the method provided in the embodiments above.

[0260] Example 13

[0261] The embodiment of the present application further provides a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, the method provided in the above embodiment is implemented.

[0262] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0263] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0264] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0265] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0266] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0267] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0268] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating an image, It is characterized in that include: Acquire an original image, text content data, and text region data, wherein the text content data is used to describe the text to be rendered in the original image, and the text region data is used to describe the position of the text to be rendered in the original image; Constructing local area data based on the text area data, wherein the local area data is used to represent the position of the area where the text needs to be rendered in the original image; Using the local area data, controlling the image generation model to generate a local image based on the text content data and the original image, wherein the local image contains the text and the size of the local image is the same as the size of the local area data; Based on the local area data, the local image is fused with the original image to generate a target image.

2. The method according to claim 1, It is characterized in that The method of using the local area data to control the image generation model to generate a local image based on the text content data and the original image includes: generating an input image having the same size as the local area data based on the noise; Based on the original image and the text area data, construct an image-level condition, wherein the image-level condition is used to characterize background information in the partial image and a position where the text needs to be rendered; Based on the text content data, constructing feature-level conditions, wherein the feature-level conditions are used to characterize the morphological features and content features of the text; Inputting the input image, the image level condition and the feature level condition into the image generation model to obtain the target noise output by the image generation model; The input image is denoised based on the target noise to obtain the local image.

3. The method according to claim 2, It is characterized in that The constructing of image-level conditions based on the original image and the text region data comprises: Cropping the original image based on the local area data to obtain a cropped image, wherein the cropped image is used to represent background information in the local image; Based on the text area data, a mask image corresponding to the text area data is generated, wherein the pixel value of the mask image is used to represent the position where the text needs to be rendered in the original image; The image-level condition includes the cropped image and the mask image.

4. The method according to claim 2, It is characterized in that The constructing of feature-level conditions based on the text content data includes: Perform image rendering based on the text content data to generate a rendered image; Using a recognizer to recognize text in the rendered image to obtain morphological features of the text, and using the recognizer to recognize text content data to obtain content features of the text; The feature-level conditions include: the morphological features and the content features.

5. The method according to claim 2, It is characterized in that The step of inputting the input image, the image level condition, and the feature level condition into the image generation model to obtain the target noise output by the image generation model includes: Inputting the input image, the image-level condition and the feature-level condition into the image generation model to obtain initial noise; Performing denoising processing on the input image based on the initial noise to obtain an initial image; Repeating the steps of inputting the initial image, the image-level condition, and the feature-level condition into the image generation model, and performing denoising processing on the input image based on the initial noise until the number of iterations reaches a preset number; When the number of iterations reaches the preset number of times, it is determined that the noise output by the image generation model during this iteration is the target noise.

6. The method according to claim 5, It is characterized in that The method further comprises: Acquire a sample image, a sample background image of the sample image, sample text content of a sample text contained in the sample image, and a sample text area of ​​the sample text in the sample image; Constructing sample local area data based on the sample text area; Processing the sample image based on the sample local area data and the sample noise to generate a sample input image; Based on the sample background image and the sample text area data, constructing a sample image level condition, wherein the sample image level condition is used to characterize the background information in the predicted image and the position of the sample text; Based on the sample text content, constructing a sample feature level condition, wherein the sample feature level condition is used to characterize the sample morphological features and sample content features of the sample text; Inputting the sample input image, the sample image level condition and the sample feature level condition into an initial model to obtain a predicted noise output by the initial model; Constructing a total loss function of the initial model based on the sample noise, the prediction noise, and the sample input image; The initial model is adjusted based on the total loss function to obtain the image generation model.

7. The method according to claim 6, It is characterized in that The processing of the sample image based on the sample local area data and the sample noise to generate a sample input image includes: Cutting the sample image based on the sample local area data to obtain a local sample image; A sample noise having the same size as the sample local area data is added to the local sample image to obtain the sample input image.

8. The method according to claim 7, It is characterized in that The constructing the total loss function of the initial model based on the sample noise, the prediction noise, and the sample input image comprises: generating a noise loss function based on the prediction noise and the sample noise; Performing denoising on the sample input image based on preset noise to obtain a denoised image; Generate a text loss based on the text information in the sample input image and the text information in the denoised image; Generate background loss based on background information in the sample input image and background information in the denoised image; The noise loss function, the text loss and the background loss are weighted to obtain the total loss function.

9. The method according to any one of claims 1 to 8, It is characterized in that After constructing the local area data based on the text area data, the method further includes: Outputting the local area data; When a first feedback result corresponding to the local area data is received, the image generation model is controlled to generate the local image based on the text content data and the original image using the first feedback result, wherein the first feedback result is a result obtained by modifying the local area box.

10. The method according to any one of claims 1 to 8, It is characterized in that After using the local area data to control the image generation model to generate a local image based on the text content data and the original image, the method further includes: outputting the partial image; When a second feedback result corresponding to the local image is received, the second feedback result is fused with the original image based on the local area data to generate the target image, wherein the second feedback result is a result obtained by modifying the local image.

11. The method according to any one of claims 1 to 8, It is characterized in that After fusing the local image with the original image based on the local area data to generate a target image, the method further includes: Outputting the target image; receiving a third feedback result corresponding to the target image, wherein the third feedback result is a result obtained by modifying the target image; The image generation model is adjusted based on the original image, the text content data, the text area data and the third feedback result.

12. A method for generating an image, It is characterized in that include: Capturing a scene image of a real scene, and acquiring text content data and text area data, wherein the text content data is used to describe the text to be rendered in the scene image, and the text area data is used to describe the position where the text needs to be rendered in the scene image; Constructing local area data based on the text area data, wherein the local area data is used to represent the position of the area where the text needs to be rendered in the scene image; Using the local area data, controlling the image generation model to generate a local image based on the text content data and the scene image, wherein the local image contains the text and the size of the local image is the same as the size of the local area data; Based on the local area data, the local image is fused with the scene image to generate a target image.

13. The method according to claim 12, It is characterized in that The using the local area data to control the image generation model to generate a local image based on the text content data and the scene image includes: generating an input image having the same size as the local area data based on the noise; Based on the scene image and the text region data, construct an image-level condition, wherein the image-level condition is used to characterize background information in the local image and a position where the text needs to be rendered; Based on the text content data, constructing feature-level conditions, wherein the feature-level conditions are used to characterize the morphological features and content features of the text; Inputting the input image, the image level condition and the feature level condition into the image generation model to obtain the target noise output by the image generation model; The input image is denoised based on the target noise to obtain the local image.

14. A method for generating an image, It is characterized in that include: Acquire an original image, text content data, and text area data by calling a first interface, wherein the first interface includes a first parameter, a parameter value of the first parameter includes the original image, the text content data, and the text area data, the text content data is used to describe the text to be rendered in the original image, and the text area data is used to describe the position of the text to be rendered in the original image; Constructing local area data based on the text area data, wherein the local area data is used to represent the position of the area where the text needs to be rendered in the original image; Using the local area data, controlling the image generation model to generate a local image based on the text content data and the original image, wherein the local image contains the text and the size of the local image is the same as the size of the local area data; Based on the local area data, the local image is fused with the original image to generate a target image; The target image is output by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter includes the target image.

15. An electronic device, It is characterized in that include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 14 when running.

16. A computer-readable storage medium, It is characterized in that The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 14.

17. A computer program product, It is characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 14.

Citation Information

Cited By

  • Image generation method and system based on object semantics and medium

    CN119379842A

  • Local modification redrawing method and system for generating planning intention graph

    CN120912724A

  • Automatic rendering optimization method based on visual UI

    CN121280560A