Training-free text sharpness enhancement in diffusion image based generation guided by ocr

Abstract

Embodiments of the present disclosure relate to training-free text sharpness enhancement in diffusion image based generation guided by OCR. A method, apparatus, non-transitory computer readable medium, and system for image processing includes obtaining an input prompt word indicative of a target visual text, generating an intermediate image based on the input prompt word, computing a character recognition loss based on the target visual text and the intermediate image, and generating a synthetic image based on the character recognition loss, wherein the synthetic image depicts the target visual text.

Description

Cross-reference to related applications

[0001] This application claims priority to U.S. Provisional Application No. 63 / 793,086, filed April 23, 2025, with the U.S. Patent and Trademark Office under 35 USC § 119, and to U.S. Application No. 19 / 375,918, filed October 31, 2025, the disclosure of which is incorporated herein by reference in its entirety. Background Technology

[0002] The following typically refers to image processing, and more specifically, image generation using machine learning models. Image processing refers to using computers to edit or generate images using algorithms or processing networks. In some cases, image processing software can be used for a variety of image processing tasks, such as image restoration, image detection, image editing, image compositing, and image generation.

[0003] In some cases, image generation involves tuning a generative machine learning model based on input text prompts to produce an output image that includes one or more visual features described in the input text prompts. For example, the model can be tuned to generate images that reflect the described scene and include text such as titles, labels, or quotation marks. However, while generative models are able to capture high-level semantic context, they often struggle to generate clear and structurally consistent text within the generated images. Summary of the Invention

[0004] Embodiments of this disclosure provide a method and system for text-to-image generation, and more specifically, for text enhancement in image generation. In one aspect, the system generates a synthetic image depicting the target visual text based on an input text cue describing the target visual text. In one aspect, the system includes an optical character recognition (OCR) model configured to perform character recognition on the input text cue to generate a character recognition loss. For example, the character recognition loss represents the “legibility” of the target visual text in an intermediate image. In some aspects, the system generates a character recognition gradient (in a latent space) based on the character recognition loss (e.g., from pixel space). In some embodiments, the character recognition loss is generated based on an intermediate image generated in pixel space, but in other embodiments, the character recognition loss may be generated based on character recognition performed in the latent space.

[0005] During the image generation process (e.g., a backdiffusion process or a denoising process), the system includes an image generation model configured to generate image features based on text embeddings of input text prompts. In an embodiment, a character recognition loss is used to guide the image generation process, enabling the image generation model to generate a synthetic image that incorporates content from the input text prompts and depicts the clear target visual text described by the text prompts.

[0006] A method, apparatus, non-transitory computer-readable medium, and system for image processing includes acquiring an input cue word indicating target visual text, generating an intermediate image based on the input cue word, calculating a character recognition loss based on the target visual text and the intermediate image, and generating a synthetic image based on the character recognition loss, wherein the synthetic image depicts the target visual text.

[0007] A method, apparatus, non-transitory computer-readable medium, and system for image processing includes acquiring an input cue word indicating target visual text, generating an intermediate image based on the input cue word, performing character recognition on the intermediate image to obtain predicted visual text, and generating a synthetic image based on the target visual text and the predicted visual text, wherein the synthetic image depicts the target visual text.

[0008] An apparatus and system for image processing includes a memory component and a processing device coupled to the memory component, the processing device being configured to perform operations including acquiring an input cue word indicating target visual text, generating an intermediate image based on the input cue word, calculating a character recognition loss based on the target visual text and the intermediate image, and generating a synthetic image based on the character recognition loss, wherein the synthetic image depicts the target visual text. Attached Figure Description

[0009] Figure 1 Examples of image processing systems according to various aspects of this disclosure are shown.

[0010] Figure 2 Examples of methods for text-adjusted image generation according to various aspects of this disclosure are shown.

[0011] Figure 3 Examples of methods for image generation using character recognition loss according to various aspects of this disclosure are shown.

[0012] Figure 4 Examples of methods for generating images by performing optical character recognition, according to various aspects of this disclosure, are shown.

[0013] Figure 5 Examples of text-to-image generation according to various aspects of this disclosure are shown.

[0014] Figure 6 Examples of machine learning models based on various aspects of this disclosure are shown.

[0015] Figure 7 Examples of image generation models based on various aspects of this disclosure are shown.

[0016] Figure 8 An example of the U-Net architecture according to various aspects of this disclosure is shown.

[0017] Figure 9 An example of a diffusion converter model according to various aspects of this disclosure is shown.

[0018] Figure 10 Examples of diffusion processes according to various aspects of this disclosure are shown.

[0019] Figure 11 An example of pseudocode for OCR-guided diffusion is shown according to various aspects of this disclosure.

[0020] Figure 12 An example flowchart is shown in an example implementation of an algorithm that describes a step-by-step process for training a machine learning model, according to aspects of this disclosure.

[0021] Figure 13 Examples of methods for training diffusion models according to various aspects of this disclosure are shown.

[0022] Figure 14 Examples of image processing apparatuses according to various aspects of this disclosure are shown.

[0023] Figure 15 Examples of computing devices according to various aspects of this disclosure are shown. Detailed Implementation

[0024] The following relates to text-to-image generation using machine learning models. Embodiments of this disclosure relate to an image generation system that accurately generates synthetic images depicting displayed text described by input text prompts. The system generates synthetic images that enhance the readability of the text generated within the synthetic images without retraining the image generation model or changing its architecture. This is achieved by integrating a character recognition loss or feedback into the noise prediction process (or diffusion process) of the image generation model, which refines the generation of text elements while maintaining image quality and fidelity.

[0025] According to some embodiments, the system includes an optical character recognition (OCR) model configured to generate OCR guidance (e.g., character recognition loss or character recognition gradient) during the inference phase of the image generation process. The character recognition loss provides an accuracy level representing the text generated in an intermediate stage of the diffusion process. During intermediate image processing time steps, the OCR model computes gradients (e.g., latent vectors) based on the OCR loss and applies normalized guidance to adjust the latent representation generated by the image generation model. This guidance enhances the semantic accuracy and structural clarity of the rendered text while maintaining image quality and fidelity.

[0026] In some embodiments, during the guided image generation process, the model applies OCR-based updates iteratively or in a single pass. Guidance is applied selectively within a predetermined time step to maximize its impact. By using semantic feedback from the OCR model, the system enhances the practical utility of the diffusion-based image generation model in applications that generate images including those displaying text.

[0027] Traditional image generation systems may include diffusion-based generative models trained to synthesize images from input text cues. These systems operate through a denoising process that transforms random noise into sharp images aligned with the input cues. During inference, these systems apply guidance mechanisms, such as classifier-free guidance (CFG), to improve the alignment between the generated image and the text conditions. While CFG enhances visual fidelity and cue relevance, it lacks the ability to explicitly control or improve the sharpness of the embedded text within the generated image.

[0028] Some traditional systems attempt to improve text fidelity by modifying the model architecture or retraining on specialized datasets with annotated text regions. These systems may inject structural priors or use segmentation masks to locate text regions, which are then refined during training. In some cases, this technique can enhance the structural arrangement of the text. However, these systems can be computationally expensive and cannot be generalized across different domains. Furthermore, architectural changes may introduce trade-offs that negatively impact overall image quality or model flexibility.

[0029] Some traditional systems use external control signals (e.g., edge maps or pose estimation) to influence the layout and appearance of the generated image. These systems can force alignment with desired structures, including text shapes or arrangements. However, these systems do not provide semantic-level feedback and are limited to visual similarity, which does not guarantee that the generated text is actually readable or semantically accurate.

[0030] Some traditional systems modify the conditioning embeddings used during the generation process to implicitly guide the output content. These systems can use lightweight enhancer networks to amplify cue word alignment and improve global visual consistency. However, these systems do not explicitly address the text rendering problem and therefore cannot generate consistently clear or accurate text within the image.

[0031] Gradient-free classifier-guided (GFCG) and other classifier-guided sampling systems provide mechanisms to improve class-specific image accuracy by using pre-trained classifiers. These systems guide the sampling process to the image using an external classifier with higher prediction confidence for a given class. However, these systems are trained for class-based guidance and do not provide functionality for evaluating or optimizing text readability. Therefore, these systems are unsuitable for applications requiring accurate text generation.

[0032] Therefore, embodiments of this disclosure improve upon conventional image generation systems by using OCR guidance to accurately generate clear text within synthesized images. This is achieved by integrating an OCR model into the inference process of the image generation model to generate an OCR-based guiding signal. In one aspect, the system computes gradient information based on an OCR loss, which represents the accuracy of the generated text (in intermediate images) relative to the target text described in the input text prompt. By selectively applying OCR gradients during the denoising step, the system refines the latent representation that enhances the semantic and visual quality of the rendered text. In some cases, the system can effectively apply OCR guidance by applying OCR gradients within a predetermined denoising time step. Thus, the image generation model can generate images containing accurate content and clear text aligned with the input text prompt without retraining the image generation model.

[0033] refer to Figure 1 and reference Figure 15 An example system for image processing is provided in this disclosure. References Figure 2 and reference Figure 5 Example applications of this disclosure in image processing are provided. References Figure 6 For reference Figure 9 and reference Figure 14 Provides details about the architecture of the image processing device. References Figure 3 For reference Figure 4 and reference Figure 10 For reference Figure 11 Examples of procedures for image processing are provided. (Reference) Figure 13 For reference Figure 14 A description of an example training process is provided.

[0034] Therefore, embodiments of this disclosure provide a system and method that improve upon conventional image generation systems by enhancing the clarity of text generated within a synthesized image. By directly integrating OCR-based guidance into the diffusion sampling process, the system can control image generation to produce semantically accurate and visually coherent text. In some aspects, by applying guidance over a predetermined range of diffusion time steps, the system reduces computational costs while maintaining image quality. The guidance mechanism is plug-and-play and can be extended to pre-trained diffusion models. In some aspects, the system provides normalized gradient application techniques to ensure stability and prevent distortion of non-textual elements in the synthesized image.

[0035] As used herein, input prompts, text prompts, or input text prompts refer to inputs, such as descriptions (or natural language descriptions), provided to a machine learning model to guide the composition and layout of a synthesized image. Input prompts may include natural language instructions, keywords, or phrases describing target visual elements, topics, or text content to appear in the synthesized image. In some cases, input prompts may include non-textual modalities such as image or layout information.

[0036] Target visual text, visual text, or intermediate visual text refers to a specific text string that is to be clearly generated within the synthesized image. This can include phrases, words, or characters that are semantically meaningful and visually coherent at the time of generation.

[0037] Character recognition loss is a loss function used to quantify the difference between, for example, the text recognized by an OCR model from a generated image and the target visual text. This loss represents how accurately the text content of the synthesized image is aligned with the target visual text. For example, a lower character recognition loss indicates better clarity and semantic alignment of the presented text. In some cases, character recognition loss can be calculated in pixel space.

[0038] Synthetic images are images generated by a machine learning model, such as a diffusion model, in response to input prompts. Synthetic images can include visual elements such as the synthesized object, background, and text content. They are generated by iteratively denoising the noisy input. In some cases, the pixels of the synthetic image are generated by the machine learning model itself.

[0039] An intermediate image is a partially denoised image generated at a specific time step during the backdiffusion process. Intermediate images represent a transitional stage in the generation of the final synthesized image and are used to guide the progression of visual content, including text rendering. Intermediate images can include noisy input, pixels, image features, or combinations thereof.

[0040] Image features are abstract representations of visual characteristics extracted from images using convolutional or transformer-based encoders. These features include spatial, textural, and semantic information and are used by image generation models to generate modified images. In some cases, image features can be represented in a latent space or embedding space.

[0041] A latent space is a continuous, multi-dimensional space representing embeddings. It encodes semantic or visual features in a compact form, allowing for efficient manipulation and processing. In image generation, both text and image embeddings reside in the latent space (or multimodal space) to guide output generation.

[0042] Character recognition refers to the process of using OCR models to detect and interpret text content in images. This includes the recognition of individual characters, words, and the semantic structure of text. The output of character recognition can be used to evaluate the accuracy and clarity of text within synthetic images.

[0043] The character recognition gradient is the derivative of the character recognition loss with respect to the latent image representation. This gradient indicates how variations in the image affect OCR performance and is used to guide the image generation process towards improved text sharpness. The gradient is applied during diffusion to refine the visual text features in the image. In some cases, the character recognition gradient is generated in the latent space.

[0044] The diffusion time step or diffusion time step range refers to a specified set of time steps in which character recognition guidance is applied during the back-diffusion process. By limiting the guidance to a predetermined range (e.g., during the intermediate phase of sampling), the system maximizes the impact of the OCR gradient while maintaining overall image fidelity.

[0045] Classifier-free guidance (CFG) is a method used in diffusion models to improve alignment with moderating information (e.g., input prompts) by combining conditional and unconditional noise predictions. This method enhances control over the generation process without requiring an external classifier. CFG operates during inference and the moderating strength can be adjusted via a guidance scaling parameter.

[0046] The character recognition intensity parameter is a scalar value that determines the magnitude of the influence of the character recognition gradient during image generation. This parameter controls the degree to which OCR guides changes in the latent representation at each time step. For example, a high intensity parameter leads to a high degree of attention to text clarity, while a low value preserves more of the original generated distribution. OCR-guided image generation

[0047] exist Figures 1 to 5 and Figures 10 to 11Among them, methods, apparatus, non-transitory computer-readable media and systems for image processing include acquiring an input cue word indicating target visual text, generating an intermediate image based on the input cue word, calculating a character recognition loss based on the target visual text and the intermediate image, and generating a synthetic image based on the character recognition loss, wherein the synthetic image depicts the target visual text.

[0048] Examples of methods, apparatuses, non-transitory computer-readable media, and systems also include performing character recognition on an intermediate image to obtain predicted visual text. Some examples also include comparing the predicted visual text with target visual text.

[0049] Examples of methods, apparatuses, non-transitory computer-readable media, and systems also include acquiring a noisy input. Some examples also include denoising the noisy input based on a character recognition loss. Examples of methods, apparatuses, non-transitory computer-readable media, and systems also include computing a character recognition gradient based on a character recognition loss, wherein the noisy input is denoised based on the character recognition gradient.

[0050] Examples of the method, apparatus, non-transitory computer-readable medium, and system also include identifying a diffusion time step range within which a character recognition loss is applied. Examples of the method, apparatus, non-transitory computer-readable medium, and system also include computing classifier-free guidance based on the character recognition loss.

[0051] Examples of the method, apparatus, non-transitory computer-readable medium, and system further include obtaining a character recognition strength parameter that indicates the target strength of the character recognition loss, wherein the character recognition loss is applied based on the character recognition strength parameter. Examples of the method, apparatus, non-transitory computer-readable medium, and system further include normalizing the character recognition loss to obtain a normalized character recognition loss, wherein a synthesized image is generated based on the normalized character recognition loss.

[0052] A method, apparatus, non-transitory computer-readable medium, and system for image processing includes acquiring an input cue word indicating target visual text, generating an intermediate image based on the input cue word, performing character recognition on the intermediate image to obtain predicted visual text, and generating a synthetic image based on the target visual text and the predicted visual text, wherein the synthetic image depicts the target visual text.

[0053] Examples of the method, apparatus, non-transitory computer-readable medium, and system include calculating a character recognition loss based on character recognition, wherein the synthesized image is generated based on the character recognition loss. Examples of the method, apparatus, non-transitory computer-readable medium, and system also include acquiring a noisy input. Some examples also include denoising the noisy input based on the character recognition loss.

[0054] Examples of methods, apparatuses, nontransitory computer-readable media, and systems also include computing character recognition gradients based on character recognition loss, wherein noisy inputs are denoised based on character recognition gradients.

[0055] Examples of the method, apparatus, non-transitory computer-readable medium, and system also include identifying a diffusion time step range within which a character recognition loss is applied. Examples of the method, apparatus, non-transitory computer-readable medium, and system also include computing classifier-free guidance based on the character recognition loss.

[0056] Examples of methods, apparatuses, nontransitory computer-readable media, and systems also include obtaining a character recognition strength parameter that indicates the target strength of the character recognition loss, wherein the character recognition loss is applied based on the character recognition strength parameter.

[0057] Examples of methods, apparatuses, non-transitory computer-readable media, and systems also include normalizing the character recognition loss to obtain a normalized character recognition loss, wherein the synthesized image is generated based on the normalized character recognition loss.

[0058] Figure 1 An example of an image processing system according to various aspects of this disclosure is shown. The example shown includes a user 100, a user device 105, an image processing apparatus 110, a cloud 115, and a database 120. The image processing apparatus 110 is a reference. Figure 13 The description includes examples of the corresponding elements or aspects thereof.

[0059] refer to Figure 1 User 100 provides input prompts to image processing device 110 via user device 105 and cloud 115. The input prompts include descriptive text describing semantic content and target visual text to be generated in the synthesized image. For example, the prompt might state: “Text: You are a gem. A perfume bottle.” Image processing device 110 receives the input prompts and generates a text embedding based on them. The text embedding is used to guide the image generation process to produce a synthesized image. The synthesized image depicts the visual elements described by the input prompts and the target visual text.

[0060] In some aspects, the image processing apparatus 110 includes a text encoder configured to encode input prompts into text embeddings, wherein the text embeddings capture a combination of context and expected text content. The text embeddings are used to guide an image generation process (e.g., a diffusion process) that operates in a latent space to synthesize image features across a predetermined diffusion time step range. The image processing apparatus 110 includes an OCR model configured to generate a character recognition loss that evaluates intermediate image outputs and provides character recognition gradients during sampling to enhance the clarity of target visual text (e.g., “YOU'RE A GEM” and “Victoria”) in the synthesized image. A machine learning model can generate the synthesized image based on the text embeddings and character recognition gradients. The synthesized image is transmitted to a user device 105 via a cloud 115, where it is displayed to a user 100 with enhanced text clarity and visual coherence.

[0061] User device 105 may be a personal computer, laptop computer, mainframe computer, handheld computer, personal assistant, mobile device, or any other suitable processing device. In some examples, user device 105 includes software incorporated into an image processing application. In some examples, the image processing application on user device 105 may include the functionality of image processing device 110.

[0062] The user interface enables user 100 to interact with user device 105. In some embodiments, the user interface may include an audio device, such as an external speaker system, an external display device, such as a display screen, or an input device (e.g., a remote control device that interfaces directly with the user interface or via an I / O controller module). In some cases, the user interface may be a graphical user interface (GUI). In some examples, the user interface may be represented by code, which is sent to user device 105 and rendered locally by a browser. Reference Figure 2 The process of using the image processing device 110 is also described.

[0063] According to several aspects, the image processing apparatus 110 includes a computer-implemented network comprising a machine learning model, a text encoder, an image encoder, and an image generation model. The image processing apparatus 110 also includes a processor unit, a memory unit, an I / O module, and a training component. In some embodiments, the image processing apparatus 110 further includes, as referenced... Figure 15 The described communication interface, user interface components, and bus. Additionally or alternatively, the image processing apparatus 110 communicates with the user equipment 105 and the database 120 via a cloud 115. The image processing apparatus 110 is a reference. Figure 14 The description includes examples of the corresponding elements or aspects thereof. References Figure 2More details are described regarding the operation of the image processing device 110.

[0064] In some cases, the image processing apparatus 110 is implemented on a server. The server provides one or more functions to users linked through one or more networks in various networks. In some cases, the server includes a single microprocessor board comprising a microprocessor responsible for controlling various aspects of the server. In some cases, the server uses the microprocessor and protocols to exchange data with other devices / users on one or more networks via Hypertext Transfer Protocol (HTTP) and Simple Mail Transfer Protocol (SMTP), although other protocols such as File Transfer Protocol (FTP) and Simple Network Management Protocol (SNMP) may also be used. In some cases, the server is configured to send and receive files in Hypertext Markup Language (HTML) format (e.g., for displaying web pages). In various embodiments, the server includes a general-purpose computing device, a personal computer, a laptop computer, a mainframe computer, a supercomputer, or any other suitable processing device.

[0065] Cloud 115 is a computer network configured to provide on-demand availability of computer system resources such as data storage and computing power. In some examples, Cloud 115 provides resources that do not require active management by users (e.g., user 100). The term "cloud" is sometimes used to describe a data center available to many users via the Internet. Some large cloud networks have the capability to distribute resources from a central server across multiple locations. If a server has a direct or close connection to a user, that server is designated as an edge server. In some cases, Cloud 115 is limited to a single organization. In other examples, Cloud 115 can be used by many organizations. In one example, Cloud 115 includes a multi-layered communication network comprising multiple edge routers and a core router. In another example, Cloud 115 is based on a local collection of switches in a single physical location.

[0066] Depending on several factors, database 120 stores training data (or training set), input images, text prompts, etc. Database 120 is an organized collection of data. For example, database 120 stores data in a specified format known as a schema. Database 120 can be constructed as a single database, a distributed database, multiple distributed databases, or an emergency backup database. In some cases, a database controller can manage the storage and processing of data in database 120. In some cases, a user (e.g., user 100) interacts with the database controller. In other cases, the database controller can operate automatically without user interaction.

[0067] Figure 2Examples of a method 200 for text-adjusting image generation according to aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of a device. Additionally or alternatively, dedicated hardware is used to perform certain processes. Typically, these operations are performed using methods and processes described according to aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0068] At step 205, the user provides a text prompt to the system. In some cases, this step involves or may be performed by, as referenced... Figure 1 The user-defined input prompts include text descriptions instructing the generation of target visual content and target visual text, such as "Text: You are a gem. Perfume bottle." The text prompts are sent to an image processing device for further processing.

[0069] At operation 210, the system generates a character recognition loss. In some cases, this step involves or can be derived from a reference. Figure 1 and reference Figure 14 The image processing apparatus described herein shall perform this step. In some cases, the operation of this step involves or may be performed by reference. Figure 6 and reference Figure 14 The optical character recognition model described is used for execution. In some cases, the text encoder encodes input prompts into text embeddings. Text embeddings capture semantic and textual information from the text prompts and are used to guide the image generation process. In some cases, the guided embedding includes a character recognition loss. For example, the optical character recognition model generates a character recognition loss based on the target visual text and an intermediate image generated by the image generation model. In some cases, text embeddings and character recognition losses can be combined to obtain the guided embedding. The guided embedding can be applied via cross-attention mechanisms at various layers of the image generation model to modulate the sampling process toward the scene and the target visual text.

[0070] At step 215, the system initializes the noise input. In some cases, this step involves or can be referenced. Figure 1 and reference Figure 4 The image processing apparatus described herein shall perform this step. In some cases, the operation of this step involves or may be performed by reference. Figure 6 and reference Figure 14The image generation model described is used for execution. In some cases, the noisy input is initialized with random noise. The noisy input can be in the latent space. By initializing the image generation model with random noise, different variations of the synthetic image can be generated. In some cases, cross-attention blocks within the image generation model can be used to combine conditional embeddings such as text embeddings with noisy features to guide the image generation process. In other cases, cross-attention blocks within the image generation model can be used to combine conditional embeddings or features such as character recognition loss or character recognition gradients with noisy features to guide the image generation process.

[0071] At step 220, the system generates media content. In some cases, this step involves or can be referenced. Figure 1 and reference Figure 14 The image processing apparatus described herein shall perform this step. In some cases, the operation of this step involves or may be performed by reference. Figure 6 and reference Figure 14 The system uses a described image generation model to perform this process. For example, it generates media content, such as synthetic images, by progressively improving a noisy input. During the denoising process, character recognition gradients computed from an OCR model can be applied within a predetermined diffusion time step to enhance the clarity of target text in the image (e.g., "You are a gem" and "Victoria"). The system generates a synthetic image that depicts visual features and includes readable text elements aligned with the input text prompts. The synthetic image is then returned to the user for viewing or to downstream applications.

[0072] Figure 3 Examples of a method 300 for image generation using character recognition loss according to aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of a device. Additionally or alternatively, dedicated hardware is used to perform certain processes. Typically, these operations are performed using methods and processes described according to aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0073] At step 305, the system retrieves the input prompt word indicating the target visual text. In some cases, this step involves or can be performed using a reference. Figure 6 and reference Figure 14 The system uses a described image generation model to perform this process. In some cases, the system acquires input prompts that include or describe the target visual text. These input prompts can be provided by the user and include the semantics and textual content that the synthetic image is intended to depict.

[0074] Input prompts, text prompts, or input text prompts refer to text descriptions (or natural language descriptions) provided to machine learning models to guide the content and composition of synthesized images. Input prompts can include natural language instructions, keywords, or phrases describing target visual elements, themes, or text content to appear in the synthesized image.

[0075] Target visual text, visual text, or intermediate visual text refers to a specific text string that is to be clearly generated within the synthesized image. This can include phrases, words, or characters that are semantically meaningful and visually coherent at the time of generation.

[0076] At step 310, the system generates an intermediate image based on the input prompt. In some cases, this step involves or can be performed using a reference. Figure 6 and reference Figure 14 The image generation model described is used to perform this process. For example, an input cue word is provided to a text encoder to generate a text embedding. The image generation model then guides the generation of latent image features based on the text embedding. For example, the image generation model is initialized with random noise, and the text embedding is used to guide the backdiffusion process. In some cases, a decoder is used to decode intermediate image features to generate an intermediate image that depicts what is described by the text cue word (or input cue word) (e.g., target visual text or predicted visual text).

[0077] At step 315, the system calculates the character recognition loss based on the target visual text and the intermediate image. In some cases, this step involves or can be performed using a reference. Figure 6 and reference Figure 14 The optical character recognition (OCR) model is described for execution. In some cases, the system computes the character recognition loss based on the target visual text. In others, the system (or image generation model) generates intermediate images (e.g., noisy images), and the system generates the character recognition loss by evaluating how accurately the generated text within the intermediate image matches the target visual text. The OCR loss quantifies the mismatch between the recognized text within the intermediate image and the target visual text. In some cases, the character recognition loss is computed in pixel space.

[0078] Character recognition loss is a loss function used to quantify the difference between the text recognized by the OCR model from the generated image and the target visual text. This loss represents how accurately the text content of the synthesized image is aligned with the target visual text. For example, a lower character recognition loss indicates better clarity and semantic alignment of the presented text. In some cases, character recognition loss can be calculated in pixel space.

[0079] An intermediate image is a partially denoised image generated at a specific time step during the backdiffusion process. Intermediate images represent a transitional stage in the generation of the final synthesized image and are used to guide the progression of visual content, including text rendering. Intermediate images can include noisy input, pixels, image features, or combinations thereof.

[0080] Image features are abstract representations of visual characteristics extracted from images using convolutional or transformer-based encoders. These features include spatial, textural, and semantic information and are used by image generation models to generate modified images. In some cases, image features can be represented in a latent space or embedding space.

[0081] A latent space is a continuous, multi-dimensional space representing embeddings. It encodes semantic or visual features in a compact form, allowing for efficient manipulation and processing. In image generation, both text and image embeddings reside in the latent space (or multimodal space) to guide output generation.

[0082] At operation 320, the system generates a synthetic image based on character recognition loss, wherein the synthetic image depicts the target visual text. In some cases, this step involves or can be performed by reference. Figure 6 and reference Figure 14 The system employs a described image generation model. In some cases, it uses a diffusion-based image generation model to generate synthetic images, where the generation process is guided by input prompts and a computed character recognition loss. During backdiffusion, the system applies character recognition gradients derived from the OCR loss to selectively modify latent image features at predetermined diffusion time steps. By doing so, the system enhances the sharpness and semantic correctness of the rendered text, allowing the synthetic image to more accurately and clearly depict the target visual text.

[0083] The character recognition gradient is the derivative of the character recognition loss with respect to the latent image representation. This gradient indicates how variations in the image affect OCR performance and is used to guide the image generation process towards improved text sharpness. The gradient is applied during diffusion to refine the visual text features in the image. In some cases, the character recognition gradient is generated in the latent space.

[0084] The diffusion time step or diffusion time step range refers to a specified set of time steps in which character recognition guidance is applied during the back-diffusion process. By limiting the guidance to a predetermined range (e.g., during the intermediate phase of sampling), the system maximizes the impact of the OCR gradient while maintaining overall image fidelity.

[0085] Synthetic images are images generated by a machine learning model, such as a diffusion model, in response to input prompts. Synthetic images can include visual elements such as the synthesized object, background, and text content. They are generated by iteratively denoising the noisy input. In some cases, the pixels of the synthetic image are generated by the machine learning model itself.

[0086] Figure 4 Examples of methods for generating images by performing optical character recognition according to aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of a device. Additionally or alternatively, dedicated hardware is used to perform certain processes. Generally, these operations are performed using the methods and processes described according to aspects of this disclosure. In some cases, the operations described herein are performed by various sub-steps or in combination with other operations.

[0087] At step 405, the system retrieves the input prompt word indicating the target visual text. In some cases, this step involves or can be performed using a reference. Figure 6 and reference Figure 14 The image generation model described is used to perform this process. Input prompts can be provided by the user and include both a semantic descriptor and the target visual text to be depicted in the final synthesized image. For example, the prompt might state, “Text ‘You are a gem.’ Dreamy pastel rainbow-colored starlight energy women’s perfume bottle, name ‘Victoria,’ 3D rendering.”

[0088] At step 410, the system generates an intermediate image based on the input prompt. In some cases, this step involves or can be derived from a reference. Figure 6 and reference Figure 14 The image generation model described above is used to perform this. For example, a text encoder generates text embeddings based on input prompts, and the image generation model synthesizes latent image features based on the text embeddings. A decoder can then be used to decode the latent features into an intermediate image that at least partially depicts the semantic content described by the text prompts and the target visual text.

[0089] At step 415, the system performs character recognition on the intermediate image to obtain the predicted visual text. In some cases, this step involves or can be performed by a reference. Figure 6 and reference Figure 14The described optical character recognition (OCR) model is used to perform this operation. The OCR model extracts text content from an intermediate image and outputs a predicted text string. The predicted visual text may match, partially match, or deviate from the target visual text indicated in the input cue. In some cases, the predicted visual text is used to compare with the displayed text from the text cue (or input cue) to generate a character recognition loss. During inference time, this loss is used to update and guide image generation to produce more accurate latent image features aligned with the content described by the input cue.

[0090] At operation 420, the system generates a synthetic image based on the target visual text and the predicted visual text, wherein the synthetic image depicts the target visual text. In some cases, this step involves or can be performed by a reference. Figure 6 and reference Figure 14 The system employs a described image generation model. For example, during the backdiffusion process, the system applies gradients derived from a comparison of the predicted visual text and the target visual text to refine latent features, thereby improving text clarity and semantic accuracy. By iteratively applying this feedback at one or more time steps in the diffusion process (e.g., the backdiffusion process), the system enhances the alignment between the synthesized image and the target visual text, ensuring that the final output depicts text that is both readable and consistent with the input prompts.

[0091] Figure 5 Examples of text-to-image generation according to various aspects of this disclosure are shown. The examples include an image generation system 500, input prompts 505, a machine learning model 510, and a synthesized image 515. In some embodiments, the image generation system 500 is implemented in a user interface.

[0092] refer to Figure 5 The image generation system 500 receives an input prompt 505 and uses a machine learning model 510 to generate a synthetic image 515. In some embodiments, the input prompt 505 includes one or more instructions describing the visual content and target visual text to be depicted in the synthetic image 515. For example, the input prompt 505 may state, “Text ‘You are a gem.’ Dreamy pastel rainbow star energy women’s perfume bottle, name ‘Victoria,’ 3D rendering.” Based on the input prompt 505, the machine learning model 510 initiates a guided image generation process.

[0093] According to some embodiments, machine learning model 510 includes a text encoder configured to encode input prompt 505 to obtain a text embedding. Machine learning model 510 also includes an image generation model that combines the text embedding with noisy input to generate a synthetic image 515. In some aspects, during the denoising process, machine learning model 510 applies OCR-guided feedback (e.g., character recognition loss) to guide latent features to enhance the accuracy and clarity of the target visual text (e.g., “You are a gem” and “Victoria”). The character recognition loss is computed by comparing the recognized text in an intermediate image generated by the image generation model with the target visual text, and the image is refined using the character recognition gradient within a predetermined diffusion time step.

[0094] In some embodiments, a machine learning model 510 is used to generate a synthetic image 515 based on text embedding and character recognition loss. In some cases, an OCR-guided mechanism is used during inference, and visual text is enhanced based on feedback from the OCR model. Embodiments of this disclosure provide enhanced visual quality and semantic correctness of the generated text in the synthetic image.

[0095] Traditional systems rely on classifier-less guidance or architecture-based constraints and cannot render visually accurate and clear text in the output image. Traditional systems can generate output images containing artifacts or useless text lacking semantic meaning. In contrast, image generation system 500 applies OCR-based character recognition gradients to generate a synthetic image 515 with readable and coherent text aligned with the input prompt 505.

[0096] Input suggestion 505 is for reference. Figure 6 Examples of the corresponding elements described, or aspects thereof. Composite image 515 is a reference. Figure 6 The description includes examples of the corresponding elements or aspects thereof. System Architecture

[0097] exist Figures 6 to 9 and Figures 14 to 15 Among them, an apparatus and system for image processing includes a memory component and a processing device coupled to the memory component, the processing device being configured to perform operations including acquiring an input cue word indicating target visual text, generating an intermediate image based on the input cue word, calculating a character recognition loss based on the target visual text and the intermediate image, and generating a synthetic image based on the character recognition loss, wherein the synthetic image depicts the target visual text.

[0098] Some examples of devices and systems also include optical character recognition (OCR) models configured to perform character recognition. Some examples of devices and systems also include text encoders configured to encode input prompts to obtain text embeddings, wherein synthetic images are generated based on these text embeddings. In some aspects, image generation models include diffusion models.

[0099] Figure 6 Examples of machine learning models according to various aspects of this disclosure are shown. The examples shown include a machine learning system 600, an input prompt word 605, a text encoder 610, a text embedding 615, a noisy input 620, an image generation model 625, image features 630, a decoder 635, an intermediate image 640, displayed text 645, an optical character recognition model (or OCR model 650), a character recognition loss 655, and a synthetic image 660. In some aspects, the machine learning system 600 includes a text encoder 610, an image generation model 625, a decoder 635, and an OCR model 650. In some aspects, the image generation model 625 includes a text encoder 610 and a decoder 635.

[0100] refer to Figure 6 Machine learning system 600 generates synthetic image 660 based on input cue words 605. For example, text encoder 610 generates text embedding 615 based on input cue words. In some cases, input cue words might state “a pink glass brain hovers above a frame, lightning comes from the frame, there is a sign saying ‘think outside the frame,’ 3D rendering.” Text embedding 615 encodes semantic attributes (e.g., object type, color, lighting, composition) and target visual text (“think outside the frame”) designed to be clearly generated in synthetic image 660.

[0101] Image generation model 625 is initialized on noisy input 620. In some cases, noisy input 620 includes random noise or a noisy image. During the denoising process, image generation model 625 generates intermediate image 640 based on text embedding 615. For example, text embedding 615 can be combined with intermediate image features via a cross-attention layer, thereby allowing the model to modulate image synthesis on the visual scene and the target visual text. In some embodiments, decoder 635 is used to decode image features 630 to obtain intermediate image 640. In some cases, decoder 635 transforms image features 630 from latent space to intermediate image 640 in pixel space.

[0102] According to some embodiments, the image generation model 625 includes a diffusion model. In some aspects, a diffusion model is a class of generative models that transforms a simple noise distribution into a complex data distribution through a progressive denoising process. These models operate through two stages, including a forward diffusion process and a backward denoising process. For example, the forward diffusion process gradually adds noise to the data samples, transforming the initial data points... x 0 Transform into a sequence of latent variables with increasing noise x 1 , x 2 ,…, x T In one respect, the noise follows variance scheduling. β 1 ,β 2 ,…,β T The forward diffusion process can be represented as: (1)

[0103] In some respects, the aforementioned forward diffusion process can be used for sampling at any time step t. This eliminates the need to sequentially execute the entire forward diffusion process. (2) in and .

[0104] In the back diffusion process, the diffusion model learns to gradually remove noise through a Markov chain, from... Let's begin. In one respect, the forward diffusion process can be represented as: (3) in Prediction is made using a neural network. This prediction can be represented as an estimate. The noise component in the data, where the training objective can be acquired and represented as: (4) in This represents the noise introduced during the forward diffusion process, and This represents the noise predicted by the neural network.

[0105] In some aspects, the system uses classifier-less guidance, which is configured to enhance regulatory control without an additional classifier. The prediction noise of classifier-less guidance can be expressed as: (5) in This represents guided scaling, balancing adherence to conditioning information with sample diversity. (Reference) Figure 7 For reference Figure 10 More details about the image generation model 625 are described below.

[0106] In some embodiments, during the denoising process, OCR feedback can be provided to the image generation model 625 during a predetermined diffusion time step. For example, display text 645 can be obtained from input prompt 605, wherein display text 645 describes the target visual text to be generated in the synthetic image 660. For example, display text 645 may state “think outside the box”. Display text 645 and intermediate image 640 are then provided to OCR model 650 to generate character recognition loss 655. In some aspects, character recognition loss 655 quantifies the difference between the recognized text in intermediate image 640 and the display text 645 described in input prompt 605. In some aspects, character recognition loss 655 is backpropagated to image generation model 625 in the form of a gradient, which serves as a correction signal for improving text sharpness in subsequent time steps. In some embodiments, the gradient (or OCR gradient) is transformed from character recognition loss 655 in pixel space to latent space.

[0107] In one aspect, OCR feedback (or OCR guidance) is selectively applied over a diffusion time step range (e.g., the middle stage of a backdiffusion process) where the structural features of the text exist but remain scalable. This OCR guidance ensures computational efficiency while maximizing the impact of OCR feedback on text refinement. In some cases, character recognition gradients can be normalized to stabilize training dynamics and prevent excessive distortion of non-textual elements within the image. By applying this feedback loop, the system incrementally adjusts the latent representation (e.g., image features 630) to resemble readable, semantically accurate text.

[0108] According to some embodiments, the machine learning system 600 uses an OCR-guided diffusion process to generate a synthetic image 660. An OCR model 650 generates an OCR loss (e.g., character recognition loss 655) based on the intermediate image 640 and the input prompt word 605. The OCR loss can be expressed as: (6) in This indicates a pre-trained model (e.g., OCR model 650). y represents the generated image (e.g., intermediate image 640 generated at an intermediate diffusion time step), and y represents the display text 645 from the input prompt 605 (e.g., text prompt).

[0109] Each time step during the back diffusion process The system modifies the noise by adding gradient terms (e.g., character recognition gradients in the latent space) based on the OCR loss (from the pixel space). In some cases, the character recognition gradient can be represented as: (7) in This represents a noisy prediction from a standard classifier-free system. This indicates OCR-guided scaling at time step t. It is a decoded image from the latent representation ,and Represents relative to the latent vector The gradient. According to some embodiments, noise prediction can be used to initialize subsequent denoising processes. In some cases, noise prediction includes image features 630 or intermediate images 640.

[0110] In some aspects, OCR guides scaling. It can be represented as: (8) in These are hyperparameters (e.g., character recognition strength parameters) that control the intensity of OCR guidance and This indicates the time step range for applying OCR guidance. In some cases, OCR guidance ensures that guidance is applied during specified time step intervals in the diffusion process, striking a balance between controlled and adaptive text generation. In other cases, OCR guidance can be applied throughout the entire inverse denoising process.

[0111] According to some embodiments, the system normalizes the character recognition loss 655 to stabilize the bootstrapping process. For example, the system normalizes the character recognition gradient using the following: (9) in It is a small constant that prevents the normalization equation from being divisible by zero. Therefore, the noise prediction can be expressed as: (10)

[0112] According to some embodiments, the system effectively integrates OCR-guided generation into the diffusion process without retraining the image generation model 625 (e.g., the diffusion model). Therefore, the system can be used to enhance existing text-to-image diffusion models. Furthermore, the system improves text rendering in diffusion-based text-to-image generation models.

[0113] According to some embodiments, latent OCR guidance can be used to guide image generation model 625. For example, OCR model 650 can be configured to generate a latent character recognition loss (in the latent space) based on text embedding 615 and image features 630 at an intermediate diffusion time step. The latent character recognition loss is used to steer the image generation process (e.g., as represented by Equation 6) toward generating clear target visual text image features (e.g., image features 630) as described in input cue 605. In some cases, by computing the latent character recognition loss in the latent space, machine learning system 600 can improve runtime efficiency by eliminating the decoding process over a predetermined diffusion time step range.

[0114] In some embodiments, image generation model 625 generates image features 630 based on text embedding 615 (e.g., image features without OCR guidance). In some cases, image features 630 are latent representations within image generation model 625 that encode the visual and structural properties of the synthesized image 660, including object layout, texture, color, and embedded text elements. In some embodiments, image features 630 are generated as intermediate output after applying character recognition gradients derived from OCR model 650.

[0115] Decoder 635 is configured to decode image features 630 (e.g., image features 630 at the final diffusion time step) to generate a synthetic image 660. In some embodiments, decoder 635 may include a variational autoencoder (VAE) decoder or a similar neural network component trained to transform latent representations from a latent space back to a pixel space. Decoder 635 reconstructs visual elements, including objects, textures, and text, based on refined latent features encoded within image features 630. In some cases, image features 630 may include high-level information influenced by both input cue word 605 and character recognition gradients applied during the diffusion process. In some cases, synthetic image 660 depicts a visually rich scene and clearly defined target visual text as described by input cue word 605.

[0116] According to some embodiments, the system includes a decoder configured to decode image features to generate a synthetic image. For example, the decoder may use a decoding function configured to convert latent image features (or representations) into a synthetic image. To perform the operation, as follows: (11) in and Let represent the standard deviation and mean of the latent space, respectively. This indicates element-wise multiplication.

[0117] According to some embodiments, the impact of OCR guidance on the synthesized image can be quantified. For example, the method described below can be used to analyze how the OCR gradient propagates through the backdiffusion process. In some cases, at a time step... OCR gradient at point A affects the synthesized image The effect can be expressed as: (12) in This refers to the total changes in the final image (e.g., composite image 660 or intermediate image 640) guided by OCR. This term... It is the gradient change of the time step. The Jacobian matrix of changes mapped to the final image. (This term...) In time step OCR-guided scaling. This item This represents the normalized OCR gradient at time step t.

[0118] In some respects, the scheduler in the diffusion model controls the denoising steps during image generation. The scheduler determines how noise is gradually removed over time. The scheduler defines the pacing and approach of this reverse process, guiding the quality, style, and speed of the final image. For example, the scheduler in the sampling process can be represented as: (13) in This represents the sampling variance (e.g., DDIM sampling variance), and This represents random noise. For deterministic sampling, where... The system can calculate: (14)

[0119] In some cases, the Jacobian matrix It can be calculated as from Down to The product of the stepwise derivatives, which will take the time step as an example. Changes at a point are mapped to the final image. The changes within. Therefore, the Jacobian transform. It can be expanded to: (15) in This represents the noise scheduling parameters.

[0120] In some cases, the perceived impact of OCR guidance can be represented by measuring the increase in OCR confidence between guided and non-guided generations, which can be expressed as: (16) in It is an improvement in OCR confidence. This indicates that the OCR model is based on images. Assigned to target text The probability of. This indicates a synthesized image generated under OCR guidance, and This indicates a composite image generated without OCR guidance.

[0121] In some respects, the effects of OCR-guided diffusion occur at intermediate time steps in the diffusion process. The time steps are effective. During this period, image structure forms, but some details remain extensible. Early time steps have a high Jacobian effect but lack sufficient structure for OCR feedback. During later time steps, the image generation model generates clear text features, but there is not enough diffusion time step retention for additional changes.

[0122] According to some embodiments, the system can enhance OCR guidance by applying multiple iterations of gradient descent to the OCR loss at each diffusion time step. For example, at each time step in the backdiffusion process, the system performs... Iteration of OCR gradient update: .Then, The latent representation (e.g., intermediate image features) can be represented as: (17) (18) (19) in This represents the number of OCR optimization iterations at time step t. Indicates after OCR-guided iteration The potential representation, This indicates the step size for OCR gradient updates. In some cases, It can be different . Iteration OCR gradient at time, and It is iteration The normalized gradient at time. In some embodiments, the system after iteration Applying standard noise prediction, the predicted noise can be expressed as: .

[0123] In some respects, the number of iterations can be dynamically calculated based on OCR loss enhancement, which is expressed as: (20) in This represents the maximum number of iterations, and It is used to determine the number of times the enhancement threshold is applied.

[0124] Input suggestion 605 is for reference. Figure 5 The description includes examples of the corresponding elements or aspects thereof. Text encoder 610 is a reference. Figure 7 and reference Figure 14 Examples of the corresponding elements described, or aspects thereof. Image generation model 625 is a reference. Figure 14 The description includes examples of the corresponding elements or aspects thereof.

[0125] OCR model 650 is a reference. Figure 14 Examples of the corresponding elements described, or aspects thereof. Composite image 660 is a reference. Figure 5 The description includes examples of the corresponding elements or aspects thereof.

[0126] Figure 7 Examples of image generation models according to various aspects of this disclosure are shown. The examples shown include a diffusion model 700, an original image 705, a pixel space 710, an image encoder 715, original image features 720, a latent space 725, a forward diffusion process 730, noise features 735, a backward diffusion process 740, denoised image features 745, an image decoder 750, an output image 755, text prompts 760, a text encoder 765, guiding features 770, and a guiding space 775.

[0127] Diffusion models are a class of generative neural networks that can be trained to generate new data with features similar to those found in the training data. In particular, diffusion models can be used to generate novel images. Diffusion models can be used for a variety of image generation tasks, including image super-resolution, image generation with perceptual metrics, conditional generation (e.g., text-guided, color-guided, style-guided, and image-guided generation), image inpainting, and image manipulation.

[0128] Diffusion models can be categorized into Denoising Diffusion Probabilistic Models (DDPM) and Denoising Diffusion Implicit Models (DDIM). In DDPM, the generation process involves an inverted stochastic Markov diffusion process. On the other hand, DDIM uses a deterministic process, so the same input yields the same output. Diffusion models can also be characterized by whether noise is added to the image or by image features generated by the encoder (e.g., latent diffusion).

[0129] The diffusion model works by iteratively adding noise to the data during the forward process and then learning to recover the data from the denoised data during the backward process. For example, during training, the diffusion model 700 can take the original image 705 in the pixel space 710 as input and apply an image encoder 715 to transform the original image 705 into original image features 720 in the latent space 725. Then, the forward diffusion process 730 gradually adds noise to the original image features 720 to obtain noisy features 735 (also in the latent space 725) at various noise levels.

[0130] Next, the backdiffusion process 740 (e.g., U-Net ANN) progressively removes noise from the noise features 735 at various noise levels to obtain denoised image features 745 in the latent space 725. In some examples, at each of the various noise levels, the denoised image features 745 are compared with the original image features 720, and the parameters of the backdiffusion process 740 of the diffusion model are updated based on this comparison. Finally, the image decoder 750 decodes the denoised image features 745 to obtain an output image 755 in the pixel space 710. In some cases, the output image 755 is created at each of the various noise levels. The output image 755 can be compared with the original image 705 to train the backdiffusion process 740. In some cases, the output image 755 refers to a synthetic image (e.g., a reference image). Figure 5 For reference Figure 6 (Described).

[0131] In some cases, the image encoder 715 and image decoder 750 are pre-trained before the backdiffusion process 740 is trained. In some examples, the image encoder 715 and image decoder 750 are trained jointly, or the image encoder 715 and image decoder 750 are fine-tuned jointly with the backdiffusion process 740.

[0132] The backdiffusion process 740 can also be guided based on a text cue 760 or another guiding cue such as an image, layout, style, color, segmentation map, etc. The text cue 760 can be encoded using a text encoder 765 (e.g., a multi-mode encoder) to obtain guiding features 770 in the guiding space 775. The guiding features 770 can be combined with noise features 735 at one or more layers of the backdiffusion process 740 to ensure that the output image 755 includes what is described by the text cue 760. For example, the guiding features 770 can be combined with noise features 735 using cross-attention blocks within the backdiffusion process 740.

[0133] Cross-attention, also known as multi-head attention, is an extension of the attention mechanism used in some artificial neural networks, such as for NLP tasks. In some cases, cross-attention simultaneously attends to multiple parts of an input sequence, capturing the interactions and correlations between different elements. In cross-attention, there are two input sequences: a query sequence and a key-value sequence. The query sequence represents the elements that need attention, while the key-value sequence contains the elements that need attention. In some cases, to compute cross-attention, the cross-attention block transforms each element in the query sequence (e.g., using linear projection) into a "query" representation, and transforms the elements in the key-value sequence into "key" and "value" representations.

[0134] Cross-attention blocks calculate attention scores by measuring the similarity between each query representation and a key representation, where higher similarity indicates more attention is given to the key element. The attention score indicates the importance or relevance of each key element and its corresponding query element.

[0135] The cross-attention block then normalizes the attention scores to obtain attention weights (e.g., using a softmax function), where the attention weights determine how much information from each value element is incorporated into the final attention representation. By simultaneously attending to different parts of the key-value sequence, the cross-attention block captures relationships and relevance across the input sequence, enabling the machine learning model to understand the context and generate more accurate and context-sensitive output.

[0136] In some examples, the diffusion model is based on a neural network architecture called U-Net. U-Net takes input features with an initial resolution and an initial number of channels, and processes the input features using initial neural network layers (e.g., convolutional network layers) to generate intermediate features. The intermediate features are then downsampled using downsampling layers, such that the downsampled features have a resolution smaller than the initial resolution and a number of channels greater than the initial number of channels.

[0137] This process is repeated multiple times, and then reversed. For example, an upsampling process is used to upsample downsampled features to obtain upsampled features. The upsampled features can be combined with intermediate features having the same resolution and number of channels via skip connections. These inputs are processed using a final neural network layer to produce output features. In some cases, the output features have the same resolution and the same number of channels as the initial resolution.

[0138] In some cases, U-Net employs additional input features to generate conditionally generated outputs. For example, additional input features may include vector representations of input prompts. These additional input features can be combined with intermediate features within the neural network at one or more layers. For instance, cross-attention modules can be used to combine additional input features and intermediate features. (Reference) Figure 8 and reference Figure 10 Describe more details about U-Net.

[0139] The diffusion process can also be modified based on conditional guidance. In some cases, the user provides text prompts describing what to include in the generated image (e.g., text prompt 760). In some examples, guidance can be provided in forms other than text, such as via images, sketches, colors, styles, or layouts. The system converts text prompt 760 (or other guidance) into conditional guidance vectors or other multidimensional representations. For example, a transformer model or multimodal encoder can be used to convert the text into vectors or series vectors. In some cases, the encoder for conditional guidance is trained independently of the diffusion model.

[0140] Initialization involves a noise map containing random noise. The noise map can be in pixel space or latent space. By initializing the image with random noise, different variations of the image, including content described by conditional guidance, can be generated. Then, the diffusion model 700 generates an image based on the noise map and the conditional guidance vectors.

[0141] The diffusion process may include both a forward diffusion process 730 and a backward diffusion process 740. The forward diffusion process 730 adds noise to the image (e.g., the original image 705) or features (e.g., original image features 720) in the latent space 725. The backward diffusion process 740 denoises the image (or features) to obtain a denoised image (e.g., the output image 755). The forward diffusion process 730 can be represented as... And the reverse diffusion process 740 can be represented as . refer to Figure 12 More details describing the diffusion process.

[0142] The diffusion model 700 can be trained using both forward diffusion process 730 and backward diffusion process 740. In one example, the user initializes the untrained model. Initialization may include defining the model architecture and initial values ​​for the model parameters. In some cases, initialization may include defining hyperparameters such as the number of layers, the resolution and channels per layer block, and the location of skip connections.

[0143] Then, the system sends training image segments... The stage uses a forward diffusion process 730 to add noise. In some cases, the forward diffusion process 730 is a fixed process in which Gaussian noise is continuously added to the image. In the latent diffusion model, Gaussian noise can be continuously added to features in the latent space 725 (e.g., original image features 720).

[0144] In each stage place, from At the start of the phase, the backdiffusion process 740 is used to predict the phase. The image or image features at that location. For example, the backdiffusion process 740 can predict the noise added by the forward diffusion process 730, and can remove the predicted noise from the image to obtain the predicted image. In some cases, the original image 705 is predicted at each stage of the training process.

[0145] The training components will be phased The predicted image (or image features) is compared with the actual image (or image features), such as the stage. The image at the location or the original input image. For example, given the observed data. It is possible to train a diffusion model 700 to minimize the negative logarithm of the training data. The upper bound of the likelihood variation. Then, the training components update the parameters of the diffusion model 700 based on comparisons. For example, the parameters of U-Net can be updated using gradient descent. Time-dependent parameters of the Gaussian transform can also be learned. (Reference) Figure 13 Describe more details about training the diffusion model.

[0146] Original image 705 is a reference. Figure 9 Examples of the corresponding elements described, or aspects thereof. Forward diffusion process 730 is a reference. Figure 9 Examples of the corresponding elements described, or aspects thereof. The back diffusion process 740 is a reference. Figure 10 The description includes examples of the corresponding elements or aspects thereof.

[0147] The text prompt word 760 is for reference. Figure 5 For reference Figure 6 Examples of input prompts or their aspects are described. The text encoder 765 is a reference. Figure 6 and reference Figure 14 The description includes examples of the corresponding elements or aspects thereof.

[0148] Figure 8 An example of a U-Net 800 architecture according to various aspects of this disclosure is shown. The example shown includes U-Net 800, input features 805, an initial neural network layer 810, intermediate features 815, a downsampling layer 820, downsampling features 825, an upsampling process 830, upsampling features 835, skip connections 840, a final neural network layer 845, and output features 850.

[0149] In some examples, U-Net 800 is the execution reference. Figure 7 Examples of components of the reverse diffusion process 740 of the described diffusion model 700, and including references. Figure 14The architectural elements of the image generation model 1430 are described. Figure 8 The U-Net 800 depicted in the reference is Figure 7 Examples of the architectures used in the described reverse diffusion process, or aspects thereof.

[0150] In some examples, the diffusion model is based on a neural network architecture called U-Net. U-Net 800 takes input features 805 with an initial resolution and an initial number of channels, and processes the input features 805 using an initial neural network layer 810 (e.g., a convolutional network layer) to produce intermediate features 815. The intermediate features 815 are then downsampled using a downsampling layer 820, such that the downsampled features 825 have a resolution smaller than the initial resolution and a number of channels greater than the initial number of channels.

[0151] This process is repeated multiple times, and then reversed. For example, upsampling process 830 is used to upsample downsampled feature 825 to obtain upsampled feature 835. Upsampled feature 835 can be combined with intermediate feature 815 having the same resolution and number of channels via skip connection 840. These inputs are processed using final neural network layer 845 to produce output feature 850. In some cases, output feature 850 has the same resolution and the same number of channels as the initial resolution.

[0152] In some cases, U-Net 800 employs additional input features to produce conditionally generated outputs. For example, additional input features may include vector representations of input cue words. These additional input features may be combined with intermediate features 815 within the neural network at one or more layers. For instance, a cross-attention module may be used to combine additional input features and intermediate features 815.

[0153] Figure 9An example of a diffusion converter model according to various aspects of this disclosure is shown. The example shown includes predicted noise 905, predicted covariance 910, linear and reshaping layer 915, normalization layer 920, (multiple) DiT blocks 925, block processing operation 930, embedding 935, noise potential 940, time step information 945, tag information 950, and DiT block 996, which is an implementation of one of the (multiple) DiT blocks 925. In one aspect, the DiT block 996 includes a second residual connection 960, a second scaling operation 962, a feedforward network 964, a normalized second scaling and shifting operation 966, a second normalization operation 968, a first residual connection 970, a first scaling operation 972, self-attention 974, a normalized first scaling and shifting operation 976, a first normalization operation 978, an input flag 980, a conditioning flag 982, a multilayer perceptron (or MLP 984), a normalized first scaling and shifting parameter 986, a first scaling parameter 988, a normalized second scaling and shifting parameter 990, and a second scaling parameter 992. In some embodiments, the architecture uses a latent diffusion transformer 994. In some embodiments, the DiT block 996 uses the "adaLN-Zero" technique.

[0154] The Diffusion Transformer (DiT) is a popular architecture for diffusion models and is designed to be structurally faithful to standard transformer architectures. DiT incorporates the scaling properties of transformer structures. To train a denoising diffusion probabilistic model (DDPM) for images (e.g., spatial representations of images), DiT is based on the Visual Transformer (ViT) architecture, which operates on sequences of tiles. DiT processes images by dividing them into tiles, converting these tiles into labels, and applying an attention mechanism to model the relationships between different regions of the image. This approach enables the model to capture both local and long-range correlations in the image generation process.

[0155] In some cases, the input to DiT is a spatial representation. .for image, Shape The first layer of DiT performs a patching operation, where DiT divides the input image into multiple patches and transforms the patches (in the form of the spatial input) into each dimension by linearly embedding each patch into the input. of Tag sequence. After the block processing, position embeddings based on ViT frequencies are applied to all input tags. In some cases, the number of tags created by the block processing... By tile size hyperparameter And thus determined. In some cases, ,in It's another shape parameter, so it's halved. Four times In some cases, this is at least four times the total number of giga-float operations (GFLOPs) of the converter. In some examples, the change It has no effect on downstream parameter counts; for example, parameter counts in downstream layers of DiT are independent of... In some examples, and Various tile sizes, transformer block architectures, and model sizes have been implemented.

[0156] Following the block-based processing, an attention mechanism is applied to model the relationships between different regions of the image within one or more DiT blocks. In addition to the noisy image input, the diffusion model sometimes processes additional conditional information, such as the noise time step. Class tags Natural language information, etc. The following describes four variations of a transformer block used to process conditional inputs that include both input information and conditional information.

[0157] In some cases, the Adaptive Layer Canonicalization (adaLN) block is used to implement the DiT block in the DiT network. Following the adaptive normalization layer in a Generative Adversarial Network (GAN) and a traditional diffusion model with a U-Net backbone, in some examples, the standard normalization layer in the transformer block is replaced by the Adaptive Layer Canonicalization (adaLN). In adaLN, the system does not directly learn dimensionality scaling. and shift parameters Instead, it starts from the noise time step. and category labels The sum of the embedding vectors Perform regression adaLN adds a relatively small number of GFLOPs and is more efficient. Furthermore, adaLN is a modulation mechanism that applies the same functionality to all tags.

[0158] In some cases, adaLN-Zero blocks are used to implement DiT blocks in DiT networks, which utilize zero-initialization techniques. In Residual Networks (ResNet), each residual block is initialized with an identity function. This is beneficial. In some examples, the final batch specification scaling factor in each block is... Zero initialization accelerates large-scale training in supervised learning settings. U-Net-based diffusion models use a similar initialization strategy, performing zero initialization on the final convolutional layers in each block before residual connections. A similar zero initialization technique is used to modify adaLN-Zero blocks from adaLN blocks. This is in addition to regression dimensionality scaling. and shift parameters In addition, the system also adjusts the dimensional scaling parameters. Perform regression, scaling parameters for this dimension This is applied immediately before the residual connections within the DiT block. The network initializes the multilayer perceptron (MLP) to output all... The zero vector of s; this initializes the entire DiT block to the identity function. Similar to the adaLN block, adaLNZero adds a negligible Gflop to the model.

[0159] In some cases, DiT blocks in DiT networks are implemented using in-context conditioning, where and The vector embeddings are appended as two additional tags to the input sequence, and after the final block, the network removes the two adjustment tags from the sequence.

[0160] In some cases, the DiT block in a DiT network includes a cross-attention block. The DiT network concatenates the embeddings of t and c into a sequence of length 2, which is separated from the image label sequence. The transformer block is modified to include a multi-head cross-attention layer appended after the multi-head self-attention block.

[0161] In some cases, DiT networks consist of sequential NDiT blocks, each operating on a hidden dimension of size d. Following ViT, DiT networks use a standard transformer configuration that scales N, d, and the attention head simultaneously. In some examples, small (S), basic (B), large (L), and XLarge (XL) variants of the model size are implemented. The small or basic model size has N=12 DiT blocks, the large model size has 24 DiT blocks, and the XLarge model size has 28 DiT blocks.

[0162] Following the final DiT block, the DiT network decodes the image label sequence into output noise prediction and output diagonal covariance prediction. Both outputs have the same shape as the original spatial input. A standard linear decoder is used for decoding, with a final normalization layer (or an adaptive normalization layer if the DiT block is an adaLN block) and linearly decoding each label into... p ×p×2C Tensor, where C is the number of channels in the spatial input of the DiT network, and p is the tile size hyperparameter. Finally, the decoded labels are rearranged into the original spatial layout to obtain the predicted noise and covariance.

[0163] In some cases, the diffusion transformer model employs a latent diffusion transformer 994. The diffusion transformer model processes a noisy latent 940, which can be a noisy version of the input image encoded in the latent space. A patchify processing operation 930 divides the noisy latent into a sequence of patches that are processed as labels. A label is a vector representation of each patch in the latent space and is adjusted through an attention process. Each label also receives time step information 945 and label information 950, and an embedding 935 encodes the current denoising time step and class label as conditional information. In some cases, the embedding 935 is referred to as a conditional embedding or conditional information embedding. In some cases, at the patch processing operation 930, a positional embedding encoding the spatial location of each label in the image is applied to the patch-processed input labels. In some examples, the positional embedding is a frequency-based positional embedding. The input labels 980 generated by the patch processing operation 930 and the adjusted labels 982 generated by the embedding 935 are processed through N (or more) DiT blocks 925, where N can be 92, 24, or 28. Other values ​​for N can be used. In some cases, the adjustment tag refers to a tag generated based on the embedding 935, the encoded time step information 945, and the tag information 950.

[0164] Each of the (multiple) DiT blocks 925 includes multiple processing stages. DiT block 996 illustrates an embodiment of one of the (multiple) DiT blocks 925. In some embodiments, DiT block 996 is an example of an adaLN-Zero block or includes aspects of an adaLN-Zero block. In some cases, input marker 980 interacts with condition marker 982 through multiple attention mechanisms. Specifically, after a first normalization 978 is applied to the input marker and an MLP 984 is applied to the condition marker, the MLP 984 generates or updates a post-normalized first scaling and shift parameter 986, denoted as... This is used for post-normalization of the first scaling and shift 976 to scale and shift the output of the first normalization 978 accordingly. Since the normalized input marker obtained from the first normalization 978 is used at least at post-normalization of the first scaling and shift 976... The carried conditional information is scaled and shifted, allowing the input and conditional information to interact. Self-attention 974 enables the scaled and shifted normalized input labels (i.e., the output from the post-normalized first scale and shift 976) to pay attention to each other. MLP 984 also generates or updates the representation as... The first scaling parameter 988 used for the first scaling operation 972 scales the output of the self-attention 974 (e.g., multi-head self-attention), and also interacts with input and conditional information. Then, at the first residual connection 970, the input marker 980 is summed with the output of the first scaling operation 972. In some examples, It has an initial value of 0, and DiT block 996 is initialized to the identity function.

[0165] A similar process is performed in the second half of DiT block 996. MLP 984 generates or updates the normalized second scaling and shift parameters 990, represented as... This is used for post-normalization of the second scaling and shift 966, to scale and shift the output of the second normalization 968 accordingly. Since the output from the second normalization 968 uses at least... The conditional information carried within is scaled and shifted, allowing further interaction between the input and conditional information. The feedforward network 964 then processes the scaled and shifted output from the post-normalized second scaling and shifting 966. The MLP 984 also generates or updates the representation as... The first scaling parameter 992, used for the second scaling operation 962, scales the output of the feedforward network 964 and also interacts with input and conditional information. In some cases, the feedforward network 964 is a pointwise feedforward network. The output from the first residual connection 970 is then added to the output of the second scaling operation 962 at the second residual connection 960, and the result is the final output of the DiT block 996. In some examples, It has an initial value of 0, and DiT block 996 is initialized to the identity function. This process is repeated for each DiT block in the sequence.

[0166] After processing through all (multiple) DiT blocks 925, the output undergoes a normalization layer 920, followed by a linearization layer and a reshaping layer 915. The final output is predicted noise 905 and predicted covariance 910. Predicted noise 905 represents the model's prediction of noise, which is added to initially create a noise latent 940. Predicted covariance 910 represents the model's prediction of covariance. Predicted noise 905 is removed from the noise latent 940 at each diffusion time step, and predicted covariance can influence how noise is removed or resampled during inversion or denoising. At the end of the denoising schedule, the latent samples are decoded to generate a synthetic image in pixel space. diffusion process

[0167] Figure 10Examples of diffusion processes 1000 according to various aspects of this disclosure are shown. The examples shown include diffusion process 1000, forward diffusion process 1005, reverse diffusion process 1010, noisy image 1015, first intermediate image 1020, second intermediate image 1025 and original image 1030.

[0168] The diffusion process 1000 may include methods for spreading the original image 1030 (e.g., reference image) into the potential space. Figure 7 The original image 705 described or features (e.g., reference) Figure 7 The original image features 720 described are subjected to a forward diffusion process 1005 that adds noise. In some aspects, the diffusion process 1000 includes a backward diffusion process 1010 for denoising the noisy image 1015 (or image features) to obtain a denoised image (or the original image 1030). The forward diffusion process 1005 can be represented as follows: And the reverse diffusion process 1010 can be represented as In some cases, a forward diffusion process 1005 is used during training to generate images with continuously large noise, and the neural network is trained to perform a backward diffusion process 1010 (e.g., to continuously remove noise).

[0169] In the context of potential diffusion models (e.g., reference models) Figure 7 In the example forward diffusion process 1005 of the diffusion model 700 described, the diffusion model uses a Markov chain to map the observed variables. (In pixel space or latent space) to obtain intermediate variables … The Markov chain gradually adds Gaussian noise to the data to obtain an approximate posterior. Because the latent variables are processed through neural networks such as U-Net, where ,… Having with Same dimension.

[0170] The neural network can be trained to perform the backdiffusion process 1010. During the backdiffusion process 1010, the diffusion model uses noisy data... (For example, noisy image 1015) starts, and the data is denoised to obtain In each step The reverse diffusion process 1010 will Such as the first intermediate image 1020, and As input. Here, Representing the steps in the transition sequence associated with different noise levels, the backdiffusion process 1010 iteratively outputs... Such as the second intermediate image 1025, until Restored to The original image is 1030. The back diffusion process 1010 can be represented as: (twenty one)

[0171] The joint probability of a sequence of samples in a Markov chain can be written as the product of the conditional probability and the marginal probability: : , (twenty two) in It is a pure noise distribution because the back diffusion process 1010 takes the result of the forward diffusion process 1005, the pure noise sample, as input, and This represents the Gaussian transformed sequence corresponding to the sequence to which Gaussian noise has been added to the sample.

[0172] During the interference time, the observed data in pixel space can be... Mapped into the latent space as input, and the generated data The output is mapped from the latent space back to the pixel space. In some examples, Represents the original input image with low image quality; latent variables , ..., Represents a noisy image, and This indicates a generated image with high image quality.

[0173] Forward diffusion process 1005 is a reference. Figure 7 Examples of the corresponding elements described, or aspects thereof. The back diffusion process 1010 is a reference. Figure 7 Examples of the corresponding elements described, or aspects thereof. Original image 1030 is for reference. Figure 7 The description includes examples of the corresponding elements or aspects thereof.

[0174] Figure 11 Examples of pseudocode for OCR-guided diffusion according to various aspects of this disclosure are shown. The examples shown include pseudocode 1100. In some cases, the pseudocode represents a simplified version of an algorithm using common languages ​​and basic programming concepts. Pseudocode 1100 is used for planning and communication logic prior to actual encoding. In some cases, embodiments of this disclosure can be implemented using various types of code or programming languages ​​without departing from the spirit of the basic principles.

[0175] refer to Figure 11 Pseudocode 1100 outlines the integration of OCR guidance into diffusion models (e.g., reference...). Figure 6This describes an algorithm for enhancing text clarity in image generation during the denoising process of an image generation model. The model receives input including a latent noise input at a predetermined time step, target text described by input prompts (e.g., target visual text), and OCR-guided scaling (e.g., character recognition strength parameters). The algorithm uses classifier-free guidance and OCR-based feedback to guide the model in generating synthetic images that match the target visual text.

[0176] In the first step, the algorithm computes a classifier-free guided (CFG) prediction. For example, the model combines the target-conditional output with the unconditional input of the denoising network scaled by guided weights (e.g., w). This step provides a baseline prediction that enhances the alignment with the target visual text without external feedback.

[0177] The second step involves applying OCR guidance within a predetermined time step. If the current time step falls within the predetermined range, the algorithm decodes the latent variables into an image, calculates the OCR loss between the recognized text in the image and the target text described by the input prompt, and then calculates the gradient of this loss relative to the latent input. The gradient is normalized to control scaling and then used to adjust the CFG prediction. This adjustment helps control the generation to produce text that better matches the input prompt.

[0178] If the time step is outside the specified range, the algorithm skips the OCR feedback and simply uses CFG prediction. This allows for flexibility in how much influence the OCR guidance has throughout the sampling process (or denoising process).

[0179] Then, the latent variables are updated using a rule-based algorithm. Thus, pseudocode 1100 enables the diffusion model to incorporate text accuracy into the image generation process, allowing the image generation model to generate synthetic images that accurately depict the target visual text described by the input prompt words. Training and Assessment

[0180] Figure 12 An example flowchart illustrating a step-by-step process in an example implementation of an algorithm according to aspects of this disclosure, depicting it as an operation for training a machine learning model, is shown. In some embodiments, process 1200 describes the operation of a training component described for training an image generation model. Process 1200 provides one or more examples of generating training data, using the training data to train a machine learning model, and using the trained machine learning model to perform a task.

[0181] In this example, the machine learning system begins by collecting training data (box 1202) to serve as the basis for training a machine learning model that defines what is being modeled. Training data can be collected by the machine learning system from a variety of sources. Examples of training data sources include public datasets, service provider system platforms that demonstrate application programming interfaces (APIs) (e.g., social media platforms), user data collection systems (e.g., digital surveys and online crowdsourcing systems), and so on. Training data collection may also include data augmentation and synthetic data generation techniques to expand and diversify the available training data, balancing techniques for balancing multiple positive and negative examples, and so on.

[0182] The machine learning system can also be configured to identify features relevant to the type of task the machine learning model is to be trained on (box 1204). Examples of tasks include classification, natural language processing, generative artificial intelligence, recommendation engines, reinforcement learning, clustering, etc. To this end, the machine learning system collects training data based on the identified features and / or filters the training data based on the identified features after collection. The training data is then used to train the machine learning model.

[0183] To train the machine learning model in the example shown, the machine learning model is first initialized (box 1206). Initialization of the machine learning model includes selecting the model architecture to be trained (box 1208). Examples of model architectures include neural networks, convolutional neural networks (CNNs), long short-term memory (LSTM) neural networks, generative adversarial networks (GANs), decision trees, support vector machines, linear regression, logistic regression, Bayesian networks, random forest learning, dimensionality reduction algorithms, boosting algorithms, deep learning neural networks, U-Net architecture, etc.

[0184] Also select a loss function (box 1210). The loss function is used to measure the difference between the output of the machine learning model (e.g., model prediction) and the target value (e.g., as expressed by the training data) that will be used to train the machine learning model. Additionally, select an optimization algorithm (box 1212) to use in conjunction with the loss function to optimize the parameters of the machine learning model during training; examples include gradient descent, stochastic gradient descent (SGD), etc.

[0185] Initializing the machine learning model also includes setting initial values ​​for the model (block 1216), examples of which include initializing node weights and biases to increase training efficiency and reduce computational resource consumption as part of training. Hyperparameters (box 1214) are also set to control the training of the machine learning model, examples of which include regularization parameters, model parameters (e.g., multiple layers in a neural network), learning rate, batch size selected from training data, etc. Various techniques are used to set hyperparameters, including using randomization techniques, using a trial-and-error approach learned from other training scenarios, etc.

[0186] The machine learning system then uses the training data to train the machine learning model (Box 1218). A machine learning model is a computer representation that can be fine-tuned (e.g., trained and retrained) based on inputs of training data to approximate an unknown function. In particular, the term machine learning model can include models that learn from known data and make predictions about known data by using algorithms (e.g., using the model architecture described above) to learn and relearn by analyzing training data to generate outputs that reflect the patterns and properties expressed by the training data.

[0187] Examples of training types include supervised learning using labeled data, unsupervised learning involving finding underlying structures or patterns within the training data, reinforcement learning based on optimization functions (e.g., rewards and / or penalties), using nodes as part of "deep learning," and so on. For example, a machine learning model can be configured to include multiple nodes that together form multiple layers. For example, a layer can be configured to include an input layer, an output layer, and one or more hidden layers. Computation is performed by the nodes within a layer through the hidden states, via a weighted connection system that is "learned" during training, for example, by optimizing the performance of the machine learning model to perform the associated task using a chosen loss function and backpropagation.

[0188] As part of training the machine learning model, a stopping criterion (decision block 1220) is determined to validate the machine learning model. The stopping criterion aims to reduce overfitting, decrease computational resource consumption, and improve the machine learning model's ability to handle unseen data not included as examples in the training data. Examples of stopping criteria include, but are not limited to, a predetermined number of periods, validation loss stability, achievement of performance improvement thresholds, whether a threshold level of accuracy has been met, or performance metrics such as precision and recall. If the stopping criterion is not met (“No” from decision block 1220), process 1200 continues training the machine learning model using the training data in this example (block 1218).

[0189] If the stopping criterion ("Yes" from decision box 1220) is met, the trained machine learning model is used to generate an output based on subsequent data (box 1222). For example, the trained machine learning model is trained to perform the task as described above, and therefore, once trained, it is configured to perform the task based on subsequent data received as input and processed by the machine learning model.

[0190] Figure 13Examples of methods 1300 for training a diffusion model according to aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of a device. Additionally or alternatively, dedicated hardware is used to perform certain processes. Typically, these operations are performed using methods and processes described according to aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0191] In some embodiments, method 1300 describes the operation of the training components described above for training an image generation model. Method 1300 represents the operation for training the training components described above. Figure 10 Examples of the described reverse diffusion process. In some examples, these operations are performed by a system including a processor that executes a set of code to control the functional elements of the device, such as... Figure 14 The image generation model described in [the document].

[0192] At operation 1305, the system initializes the untrained model. In some cases, this step involves or can be performed by, as referenced... Figure 14 The training components described are used to perform this process. Initialization may include defining the model architecture and establishing initial values ​​for the model parameters. In some cases, initialization may include defining hyperparameters such as the number of layers, the resolution and channels per layer block, the location of skip connections, etc.

[0193] At operation 1310, the system uses a forward diffusion process to add noise to the medium term in N stages. In some cases, this step involves or can be performed by, as referenced... Figure 14 The training components described are used to perform this process. In some cases, for example, the media item is the training image. In other cases, the forward diffusion process is a fixed process in which Gaussian noise is continuously added to the media item (such as the original image). In the latent diffusion model, Gaussian noise can be continuously added to features in the latent space.

[0194] At operation 1315, the system predicts media items for stage n-1 at each stage n starting from stage N. In some cases, this step involves or can be performed by, as referenced... Figure 14 The training components described are used to perform this process. In some cases, the media item is a synthetic image generated using an image generation model. For example, a backdiffusion process can predict noise added by a forward diffusion process, and the predicted noise can be removed from the noise input to obtain the predicted output. In some cases, the original media item is predicted at each stage of the training process.

[0195] At operation 1320, the system compares the predicted media item (or feature) at stage n-1 with the media at stage n-1. In some cases, for example, the system compares the synthesized image (or predicted image features) at state n-1 with the ground truth image (or basic ground truth features) at state n-1. In some cases, this step involves or can be performed by, as referenced... Figure 14 The described training components are used to execute this. For example, given the observed data... A diffusion model can be trained to minimize the negative logarithm of the training data. The upper limit of the variation of likelihood.

[0196] At operation 1325, the system updates the model parameters based on comparisons. In some cases, this step involves or can be performed by, for example, a reference. Figure 14 The training components described are used to perform this. For example, the parameters of U-Net can be updated using gradient descent. The time-dependent parameters of the Gaussian transformation can also be learned. System device

[0197] Figure 14 An example of an image processing apparatus 1400 according to various aspects of the present disclosure is shown. The example shown includes an image processing device 1400, a processor unit 1405, an I / O module 1410, and a memory unit 1415. In one aspect, the memory unit 1415 includes a text encoder 1420, an optical character recognition model 1425, and an image generation model 1430.

[0198] According to some embodiments of this disclosure, image processing apparatus 1400 includes a computer-implemented artificial neural network (ANN). An ANN is a hardware or software component comprising multiple connected nodes (e.g., artificial neurons) that loosely correspond to neurons in the human brain. Each connection or edge transmits a signal from one node to another (much like a physical synapse in the brain). When a node receives a signal, it processes the signal and then sends the processed signal to other connected nodes. In some cases, the signals between nodes include real numbers, and the output of each node is computed as a function of the sum of its inputs. In some examples, nodes may use other mathematical algorithms (e.g., selecting the maximum value from the inputs as the output) or any other suitable algorithm for activating nodes to determine the output. Each node and edge is associated with one or more node weights that determine how to process and send signals. Image processing apparatus 1400 is a reference. Figure 1 The description includes examples of the corresponding elements or aspects thereof.

[0199] Processor unit 1405 is an intelligent hardware device (e.g., a general-purpose processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof). In some cases, processor unit 1405 is configured to use a memory controller to operate a memory array. In other cases, the memory controller is integrated into the processor. In some cases, processor unit 1405 is configured to execute computer-readable instructions stored in memory to perform various functions. In some embodiments, processor unit 1405 includes dedicated components for modem processing, baseband processing, digital signal processing, or transmission processing. Processor unit 1405 is a reference Figure 15 The description includes examples of processors or aspects thereof.

[0200] I / O module 1410 (e.g., input / output interface) may include an I / O controller. The I / O controller can manage the device's input and output signals. The I / O controller can also manage peripheral devices not integrated into the device. In some cases, the I / O controller may represent a physical connection or port to an external peripheral device. In some cases, the I / O controller may utilize an operating system such as iOS®, Android®, MS-DOS®, MS-WINDOWS®, OS / 2®, UNIX®, LINUX®, or another known operating system. In other cases, the I / O controller may represent or interact with a modem, keyboard, mouse, touchscreen, or similar device. In some cases, the I / O controller may be implemented as part of a processor. In some cases, a user can interact with the device via the I / O controller or hardware components controlled by the I / O controller.

[0201] In some examples, I / O module 1410 includes a user interface. The user interface enables a user to interact with the device. In some embodiments, the user interface may include an audio device, such as an external speaker system, an external display device, such as a display screen, or an input device (e.g., a remote control device that interfaces directly with the user interface or via an I / O controller module). In some cases, the user interface may be a graphical user interface (GUI). In some examples, the communication interface operates at the boundary between the communication entity and the channel, and may also record and process communications. This document provides a communication interface to enable a processing system coupled to a transceiver (e.g., a transmitter and / or receiver). In some examples, the transceiver is configured to transmit (or emit) and receive signals for a communication device via an antenna. I / O module 1410 is a reference. Figure 15The description includes examples of I / O interfaces or aspects thereof.

[0202] Examples of memory unit 1415 include random access memory (RAM), read-only memory (ROM), or a hard disk. Examples of memory unit 1415 include solid-state memory and hard disk drives. In some examples, memory unit 1415 is used to store computer-readable, computer-executable software including instructions that, when executed, cause the processor to perform the various functions described herein.

[0203] In some cases, memory cell 1415 includes a basic input / output system (BIOS) that controls basic hardware or software operations such as interaction with peripheral components or devices. In some cases, a memory controller operates the memory cell. For example, the memory controller may include a row decoder, a column decoder, or both. In some cases, the memory cells within memory cell 1415 store information in the form of logical states.

[0204] In one aspect, memory unit 1415 includes a machine learning model. The machine learning model includes a text encoder 1420, an optical character recognition model 1425, and an image generation model 1430. Memory unit 1415 is a reference. Figure 15 The description includes examples of memory subsystems or aspects thereof.

[0205] In some cases, a machine learning model is a computational algorithm, model, or system designed to recognize patterns, make predictions, or perform a specific task (e.g., image processing) without explicit programming. According to some aspects, the machine learning model is implemented as software, firmware, one or more hardware circuits, or a combination thereof, stored in memory unit 1415 and executable by processor unit 1405.

[0206] According to some embodiments of this disclosure, machine learning models include ANNs, which are hardware or software components comprising multiple connected nodes (e.g., artificial neurons) loosely corresponding to neurons in the human brain. Each connection or edge transmits a signal from one node to another (much like a physical synapse in the brain). When a node receives a signal, it processes the signal and then sends the processed signal to other connected nodes. In some cases, the signals between nodes consist of real numbers, and the output of each node is computed as a function of the sum of its inputs. In some examples, nodes may use other mathematical algorithms (e.g., selecting the maximum value from the inputs as the output) or any other suitable algorithm for activating nodes to determine the output. Each node and edge is associated with one or more node weights that determine how to process and send signals.

[0207] During training, the weights of one or more nodes are adjusted to increase the accuracy of the results (e.g., by minimizing a loss function that corresponds in some way to the difference between the current result and the target result). Edge weights increase or decrease the strength of the signal transmitted between nodes. In some cases, nodes have a threshold below which they do not transmit a signal at all. In some examples, nodes are clustered into layers. Different layers perform different transformations on their corresponding inputs. The initial layer is called the input layer, and the last layer is called the output layer. In some cases, the signal passes through certain layers multiple times.

[0208] According to some embodiments, machine learning models include computer-implemented convolutional neural networks (CNNs). CNNs are a commonly used class of neural networks in computer vision or image classification systems. In some cases, CNNs enable the processing of digital images with minimal preprocessing. CNNs can be characterized by using convolutional (or cross-correlation) hidden layers. These layers apply convolution operations to the input before signaling the results to the next layer. Each convolutional node can process data from a finite field (e.g., the receptive field) of the input. During the forward pass of the CNN, the filters of each layer can be convolved over the input volume, calculating the dot product between the filter and the input. During the training process, the filters can be modified such that they activate when they detect specific features in the input.

[0209] In one aspect, a machine learning model includes machine learning parameters. Also known as model parameters or weights, machine learning parameters are variables that provide the behavior and characteristics of a machine learning model. Machine learning parameters can be learned or estimated from training data and are used to make predictions or perform tasks based on patterns learned from the data and relationships within the data.

[0210] During training, machine learning parameters are adjusted to minimize the loss function or maximize the performance metric. The goal of training is to find the optimal values ​​of the parameters that allow the machine learning model to make accurate predictions or perform well on a given task.

[0211] For example, during training, the algorithm adjusts machine learning parameters using optimization techniques such as gradient descent, stochastic gradient descent, or other optimization algorithms to minimize the error or loss between the predicted output and the actual target. Once the machine learning parameters have been learned from the training data, they are used to make predictions on new, unseen data.

[0212] According to some embodiments, machine learning models include computer-implemented recurrent neural networks (RNNs). RNNs are a class of ANNs where connections between nodes form a directed graph along an ordered (e.g., temporal) sequence. This allows RNNs to model temporal dynamics, such as predicting what element should appear next in a sequence. Therefore, RNNs are suitable for tasks involving ordered sequences, such as text recognition (where words are ordered within a sentence). In some cases, RNNs include one or more finite spurious recurrent networks (characterized by nodes forming a directed acyclic graph), one or more infinite spurious recurrent networks (characterized by nodes forming a directed cyclic graph), or combinations thereof.

[0213] According to some embodiments, machine learning models include transformers (or transformer models or transformer networks), where a transformer is a type of neural network model used for natural language processing tasks. A transformer network uses an encoder and a decoder to transform one sequence into another. The encoder and decoder include modules that can be stacked on top of each other multiple times. These modules include multi-head attention and feedforward layers. The input and output (target sentence) are first embedded into an n-dimensional space. Positional encodings of different words (e.g., giving each word / part of the sequence a relative position, since the sequence depends on the order of its elements) are added to the embedding representation (n-dimensional vector) of each word. In some examples, the transformer network includes an attention mechanism, where attention looks at the input sequence and determines at each step which other parts of the sequence are important.

[0214] The attention mechanism involves queries, keywords, and values, represented by Q, K, and V, respectively. Q is a matrix containing the query (a vector representation of a word in the sequence), K is the keyword (a vector representation of a word in the sequence), and V is the value, which is also a vector representation of a word in the sequence. For the encoder and decoder, in a multi-head attention module, V consists of the same sequence of words as Q. However, for an attention module that considers both the encoder and decoder sequences, V differs from the sequence represented by Q. In some cases, the values ​​in V are multiplied by some attention weights and summed.

[0215] In machine learning, attention mechanisms (e.g., implemented in one or more ANNs) are methods for assigning different levels of importance to different elements of the input. Computing attention likely involves three basic steps. First, the similarity between the query and the key vectors obtained from the input is calculated to generate attention weights. Similarity functions used for this process can include dot products, concatenation, detectors, etc. Next, the attention weights are normalized using a softmax function. Finally, the attention weights are weighted together with their corresponding values. In the context of attention networks, keywords and values ​​are vectors or matrices used to represent the input data. Keywords are used to determine which parts of the input the attention mechanism should focus on, while values ​​are used to represent the actual data being processed.

[0216] Attention mechanisms are a key component in some ANN architectures, particularly those used in Natural Language Processing (NLP) and sequence-to-sequence tasks. This allows ANNs to focus on different parts of the input sequence when making predictions or generating output. Some sequence models (such as RNNs) process the input sequence sequentially, preserving internal hidden states that capture information from previous steps. However, in some cases, this sequential processing makes it difficult to capture long-range dependencies or pay attention to specific parts of the input sequence.

[0217] Attention mechanisms address these difficulties by enabling ANNs to selectively focus on different parts of the input sequence, assigning different levels of importance or attention to each part. Attention mechanisms achieve selective focusing by considering the relevance of each input element to the current state of the ANN.

[0218] The term "self-attention" refers to a machine learning model in which representations of the inputs interact with each other to determine attention weights on the inputs. Self-attention can be distinguished from other attention models because the attention weights are determined at least in part by the inputs themselves.

[0219] According to some aspects, the text encoder 1420 is implemented as software, firmware, one or more hardware circuits, or a combination thereof, stored in memory unit 1415 and executable by processor unit 1405. According to some aspects, the text encoder 1420 is configured to encode input prompts to obtain text embeddings, wherein a synthetic image is generated based on the text embeddings. The text encoder 1420 is a reference. Figure 5 and reference Figure 6 The description includes examples of the corresponding elements or aspects thereof.

[0220] According to some aspects, the optical character recognition model 1425 is implemented as software, firmware, one or more hardware circuits, or a combination thereof, stored in memory unit 1415 and executable by processor unit 1405. According to some aspects, the optical character recognition model 1425 calculates character recognition loss based on the target visual text. In some examples, the optical character recognition model 1425 performs character recognition on an intermediate image to obtain intermediate visual text. In some examples, the optical character recognition model 1425 compares the intermediate visual text with the target visual text.

[0221] In some examples, the optical character recognition model 1425 calculates the character recognition gradient based on the character recognition loss, where noisy input is denoised based on the character recognition gradient. In some examples, the optical character recognition model 1425 obtains a character recognition intensity parameter, which indicates the target intensity of the character recognition loss, where the character recognition loss is applied based on the character recognition intensity parameter. In some examples, the optical character recognition model 1425 normalizes the character recognition loss 555 to obtain a normalized character recognition loss, where a synthesized image is generated based on the normalized character recognition loss. The optical character recognition model 1425 is a reference. Figure 6 The description includes examples of the corresponding elements or aspects thereof.

[0222] According to some aspects, the image generation model 1430 is implemented as software, firmware, one or more hardware circuits, or a combination thereof, stored in memory unit 1415 and executable by processor unit 1405. According to some aspects, the image generation model 1430 acquires input cue words indicating target visual text. In some examples, the image generation model 1430 generates a synthetic image based on the input cue words and a character recognition loss, wherein the synthetic image depicts the target visual text.

[0223] In some examples, image generation model 1430 generates an intermediate image based on input prompts; in some examples, image generation model 1430 receives noisy input. In some examples, image generation model 1430 denoises the noisy input based on character recognition loss. In some examples, image generation model 1430 identifies a diffusion time step range, within which character recognition loss is applied. In some examples, image generation model 1430 computes classifier-free guidance based on character recognition loss 555.

[0224] According to some aspects, the image generation model 1430 includes acquiring input cue words indicating target visual text. In some examples, the image generation model generates a synthetic image based on the input cue words and a character recognition loss, wherein the synthetic image depicts the target visual text. In some aspects, the image generation model 1430 includes a diffusion model. In some cases, the diffusion model includes a U-Net architecture. In some aspects, the image generation model 1430 includes a diffusion transformer model. The image generation model 1430 is a reference. Figure 6 The description includes examples of the corresponding elements or aspects thereof.

[0225] According to some aspects, the image processing apparatus 1400 includes a training component. The training component is implemented as software, firmware, one or more hardware circuits, or a combination thereof, stored in a memory unit 1415 and executable by a processor unit 1405. According to some embodiments, the training component is implemented as software, firmware in a separate computing device, one or more hardware circuits, or a combination thereof, stored in a memory unit and executable by a processor in a separate computing device's processor unit. In some examples, the training component is part of another device besides the image processing apparatus 1400 and communicates with the image processing apparatus 1400. In some examples, the training component is part of the image processing apparatus 1400. computing devices

[0226] Figure 15 An example of a computing device 1500 according to various aspects of this disclosure is shown. The example shown includes a computing device 1500, a processor 1505, a memory subsystem 1510, a communication interface 1515, an I / O interface 1520, a user interface component 1525, and a channel 1530.

[0227] In some embodiments, computing device 1500 is a reference Figure 1 and reference Figure 14 Examples of the image processing apparatus described herein, or aspects thereof, are included. In some embodiments, computing device 1500 includes processor 1505, which can execute instructions stored in memory subsystem 1510 to obtain input prompts indicating target visual text, generate intermediate images based on the input prompts, calculate character recognition loss based on the target visual text and the intermediate images, and generate synthetic images based on the character recognition loss.

[0228] According to some embodiments, processor 1505 includes one or more processors. In some cases, processor 1505 is a smart hardware device (e.g., a general-purpose processing component, digital signal processor (DSP), central processing unit (CPU), graphics processing unit (GPU), microcontroller, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), programmable logic device, discrete gate or transistor logic component, discrete hardware component, or combination thereof). In some cases, processor 1505 is configured to use a memory controller to operate a memory array. In other cases, the memory controller is integrated into processor 1505. In some cases, processor 1505 is configured to execute computer-readable instructions stored in memory to perform various functions. In some embodiments, processor 1505 includes dedicated components for modem processing, baseband processing, digital signal processing, or transmission processing. Processor 1505 is a reference... Figure 14 The description includes examples of processor units or aspects thereof.

[0229] According to some embodiments, memory subsystem 1510 includes one or more memory devices. Examples of memory devices include random access memory (RAM), read-only memory (ROM), or a hard disk. Examples of memory devices include solid-state memory and hard disk drives. In some examples, the memory is used to store computer-readable, computer-executable software including instructions that, when executed, cause a processor to perform the various functions described herein. In some cases, the memory particularly includes a basic input / output system (BIOS) that controls basic hardware or software operations such as interaction with peripheral components or devices. In some cases, a memory controller operates memory cells. For example, a memory controller may include a row decoder, a column decoder, or both. In some cases, memory cells within the memory store information in the form of logical states. Memory subsystem 1510 is a reference. Figure 14 Examples of memory cells described or including aspects thereof.

[0230] According to some embodiments, communication interface 1515 operates at the boundary between a communication entity (such as computing device 1500, one or more user devices, a cloud, and one or more databases) and channel 1530, and can record and process communications. In some cases, communication interface 1515 is provided to enable a processing system coupled to a transceiver (e.g., a transmitter and / or receiver). In some examples, the transceiver is configured to transmit (or emit) and receive signals for the communication device via an antenna. In some cases, a bus is used in communication interface 1515.

[0231] According to some embodiments, I / O interface 1520 is controlled by an I / O controller to manage input and output signals for computing device 1500. In some cases, I / O interface 1520 manages peripheral devices not integrated into computing device 1500. In some cases, I / O interface 1520 represents a physical connection or port to an external peripheral device. In some cases, the I / O controller may use an operating system such as iOS®, ANDROID®, MS-DOS®, MS-WINDOWS®, OS / 2®, UNIX®, LINUX®, or other known operating systems. In some cases, the I / O controller represents or interacts with a modem, keyboard, mouse, touchscreen, or similar device. In some cases, the I / O controller is implemented as a component of the processor. In some cases, a user interacts with the device via I / O interface 1520 or via hardware components controlled by the I / O controller. I / O interface 1520 is a reference. Figure 14 The description includes examples of I / O modules or aspects thereof.

[0232] According to some embodiments, user interface component 1525 enables a user to interact with computing device 1500. In some cases, user interface component 1525 includes audio devices, such as an external speaker system, external display devices, such as a display screen, input devices (e.g., remote control devices that interface directly with the user interface or via an I / O controller), or combinations thereof.

[0233] The performance of the apparatus, system, and method of this disclosure has been evaluated, and the results indicate that embodiments of this disclosure have achieved improved performance compared to conventional techniques (e.g., conventional image generation models). Example experiments demonstrate that the image processing apparatus based on this disclosure outperforms conventional image generation models. References Figure 5 Details of example use cases based on embodiments of this disclosure are described.

[0234] The descriptions and figures in this document represent exemplary configurations and do not represent all implementations in all aspects. For example, operations and steps may be rearranged, combined, or otherwise modified. Furthermore, structures and devices may be represented in block diagram form to show relationships between components and avoid obscuring the described concepts. Similar components or features may have the same name but may have different reference numerals corresponding to different figures.

[0235] Some modifications to this disclosure will be apparent to those skilled in the art, and the principles defined herein can be applied to other variations without departing from the scope of this disclosure. Therefore, this disclosure is not limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0236] The described methods can be implemented or performed by devices including general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor can be a microprocessor, a conventional processor, a controller, a microcontroller, or a state machine. The processor can also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration). Therefore, the functions described herein can be implemented in hardware or software and can be performed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions can be stored on a computer-readable medium in the form of instructions or code.

[0237] Computer-readable media include non-transitory computer storage media and communication media, with communication media including any medium that facilitates the transfer of code or data. Non-transitory storage media can be any available medium that can be accessed by a computer. For example, non-transitory computer-readable media can include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), optical disc (CD) or other optical disc storage devices, magnetic disk storage devices, or any other non-transitory medium used to carry or store data or code.

[0238] Furthermore, the connecting components may be appropriately referred to as computer-readable media. For example, if code or data is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology such as infrared, radio, or microwave signals, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology is included in the definition of media. Combinations of media are also included within the scope of computer-readable media.

[0239] In this disclosure and the appended claims, the word "or" indicates a list of inclusion properties, such that a list of, for example, X, Y, or Z means X or Y or Z or XY or XZ or YZ or XYZ. Furthermore, the phrase "based on" is not used to indicate a closed set of conditions. For example, a step described as "based on condition A" could be based on both condition A and condition B. In other words, the phrase "based on" should be interpreted as meaning "at least partially based on." Additionally, the words "a" or "an" mean "at least one."

Claims

1. A method comprising: Obtain input prompts for the visual text indicating the target; An intermediate image is generated based on the input prompt using an image generation model; The character recognition loss is calculated based on the target visual text and the intermediate image; as well as Using the image generation model, a synthetic image is generated based on the character recognition loss, wherein the synthetic image depicts the target visual text.

2. The method according to claim 1, wherein calculating the character recognition loss comprises: Perform character recognition on the intermediate image to obtain the predicted visual text; as well as The predicted visual text is compared with the target visual text.

3. The method of claim 1, wherein generating the synthesized image comprises: Acquire noise input; as well as The noise input is denoised based on the character recognition loss.

4. The method according to claim 3, further comprising: The character recognition gradient is calculated based on the character recognition loss, wherein the noise input is denoised based on the character recognition gradient.

5. The method according to claim 3, further comprising: The diffusion time step range is defined, wherein the character recognition loss is applied within the diffusion time step range.

6. The method according to claim 3, further comprising: The classifier-free guidance is calculated based on the character recognition loss.

7. The method according to claim 1, further comprising: Obtain a character recognition strength parameter, which indicates the target strength of the character recognition loss, wherein the character recognition loss is applied based on the character recognition strength parameter.

8. The method according to claim 1, further comprising: The character recognition loss is normalized to obtain a normalized character recognition loss, wherein the synthesized image is generated based on the normalized character recognition loss.

9. A non-transitory computer-readable medium storing code for image processing, the code including instructions that, when executed by at least one processor, cause the at least one processor to perform operations, including: Obtain input prompts for the visual text indicating the target; An intermediate image is generated based on the input prompt using an image generation model; Perform character recognition on the intermediate image to obtain the predicted visual text; as well as Using the image generation model, a synthetic image is generated based on the target visual text and the predicted visual text, wherein the synthetic image depicts the target visual text.

10. The non-transitory computer-readable medium of claim 9, wherein the code further comprises instructions, which, when executed by at least one processor, cause the at least one processor to perform an operation, including: The character recognition loss is calculated based on the character recognition, wherein the synthesized image is generated based on the character recognition loss.

11. The non-transitory computer-readable medium of claim 10, wherein generating the synthetic image comprises: Acquire noise input; as well as The noise input is denoised based on the character recognition loss.

12. The non-transitory computer-readable medium of claim 11, wherein the code further comprises instructions, which, when executed by at least one processor, cause the at least one processor to perform an operation, including: The character recognition gradient is calculated based on the character recognition loss, wherein the noise input is denoised based on the character recognition gradient.

13. The non-transitory computer-readable medium of claim 11, further comprising: The diffusion time step range is defined, wherein the character recognition loss is applied within the diffusion time step range.

14. The non-transitory computer-readable medium of claim 11, further comprising: The classifier-free guidance is calculated based on the character recognition loss.

15. The non-transitory computer-readable medium of claim 10, further comprising: Obtain a character recognition strength parameter, which indicates the target strength of the character recognition loss, wherein the character recognition loss is applied based on the character recognition strength parameter.

16. The non-transitory computer-readable medium of claim 10, further comprising: The character recognition loss is normalized to obtain a normalized character recognition loss, wherein the synthesized image is generated based on the normalized character recognition loss.

17. A system comprising: Memory components; as well as A processing device, coupled to the memory component, the processing device being configured to perform operations including: Obtain input prompts for the visual text indicating the target; An intermediate image is generated based on the input prompt using an image generation model; Calculate character recognition loss based on the target visual text and the intermediate image; and Using the image generation model, a synthetic image is generated based on the character recognition loss, wherein the synthetic image depicts the target visual text.

18. The system of claim 17, further comprising: An optical character recognition (OCR) model is configured to perform character recognition.

19. The system of claim 17, further comprising: A text encoder is configured to encode the input prompt to obtain a text embedding, wherein the synthetic image is generated based on the text embedding.

20. The system according to claim 17, wherein: The image generation model includes a diffusion model.

Images