High-fidelity visual text image generation method, system, device and storage medium

Abstract

The application discloses a high-fidelity visual character image generation method, system, device and storage medium, which are corresponding solutions, in which, based on a dual-domain character shape coding mechanism, spatial domain and spectral domain feature mutual learning is realized, and a character structure disorder problem is solved; based on a frequency domain perception refinement mechanism, a frequency domain signal is dynamically modulated to improve stroke detail precision, and through an instance-level scaling coefficient, character details and background quality are cooperatively optimized, thereby breaking through the bottleneck of character distortion and background instability in the prior art. In addition, the application supports multi-language accurate generation, can be widely applied to commercial scenes such as advertisement design, product packaging, film and television poster and teaching material, and can greatly improve creation efficiency and reduce manual correction cost; meanwhile, the feasibility of cross-domain feature cooperative improvement of multi-modal generation efficiency is verified, high-quality weak supervision training samples in a scarce scene can be provided for a character recognition model, related data blank can be effectively filled, and the application has advanced technology and market landing value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and computer vision, and particularly relates to a high-fidelity visual-text image generation method, system, device and storage medium. BACKGROUND

[0002] In recent years, diffusion models have become the mainstream technical solution in the field of natural scene image generation due to their stable training process and high-fidelity generation quality. These models can synthesize natural scene images with reasonable structures and complex textures according to user instructions through a gradual denoising generation mechanism.

[0003] With the deepening of commercial demand, visual-text image generation is becoming a new frontier for the landing of generative artificial intelligence technology. Unlike ordinary scene generation, this task requires the model to accurately render the user's required text content while synthesizing the visual background, achieving advanced creation of "text-image fusion". Specifically, the input is a text prompt containing scene description and text content, and an optional position image indicating the layout of the text, such as "a sunny beach billboard with the text 'Summer Promotion' written on it", and a user-input position layout condition image. The output image needs to meet the scene elements consistent with the user's semantics, ensure the correct and beautiful rendering of the text content, and seamlessly integrate with the background.

[0004] Although the latest diffusion models such as Stable Diffusion 3 and Flux1.1 claim to improve text rendering capability through improved architecture and training strategy, they still face difficulties in accurate text rendering when dealing with longer texts or more complex characters in practical applications.

[0005] A class of data-dependent schemes (such as GlyphByT5) trains a text encoder by large-scale pairwise labeled data to solve the above problems, but such schemes have extremely high requirements for data resources, and the generated text style is stiff, similar to "copy-paste" standard fonts into background images, lacking a sense of fusion with the scene. Another class of glyph image assisted schemes, such as GlyphControl (a visual text generation method based on conditional control network and stable diffusion model), AnyText (a glyph image assisted visual text generation scheme), inject the glyph image of visual text as a condition into the generation model to improve the distortion problem of visual text. However, such schemes are still limited to spatial domain operations, bringing two important problems: first, spatial domain operations focus on extracting local information, and the model relies too much on local area features, making it difficult to accurately model the overall structural relationship of the text, and prone to problems such as disordered structure and meaningless stroke combination, leading to the generated text being unrecognizable. Second, compared with natural images, visual text images usually contain a large number of small strokes and sharp edges, and these information is mainly carried in high-frequency components. However, the standard U-Net (U-shaped network) architecture of the diffusion model has limited perception and recovery ability for high-frequency components, and the stroke detail capture ability is insufficient, resulting in blurred edges and missing details in the generated text. These problems are particularly prominent in complex writing systems such as Chinese.

[0006] In addition, when existing schemes attempt to enhance text details, they often damage the stability of the background image, causing background texture distortion, overall picture quality degradation, and making the text and background unable to naturally blend in.

[0007] The above problems seriously restrict the commercial application development of such technology in advertising design, product packaging, film posters and other commercial applications that require precise text embedding in scenes, making designers and other content creators still unable to completely rely on AI tools to complete the end-to-end creation process. Text errors in images require high-cost manual correction, and may even cause serious cultural misunderstandings and legal disputes.

[0008] Therefore, there is an urgent need for a visual text image generation technology scheme that can enhance the stroke generation ability of the text, stabilize the quality of the background, and seamlessly integrate both.

[0009] In view of this, the present application is proposed. SUMMARY

[0010] The purpose of the present application is to provide a high-fidelity visual text image generation method, system, device and storage medium, which can generate visual text with high readability in complex text rendering scenarios, and can be seamlessly integrated with natural background images.

[0011] The purpose of the present application is achieved by the following technical solutions:

[0012] A high-fidelity visual text image generation method, comprising:

[0013] respectively encoding the input position image and the text description, and correspondingly extracting the spatial layout feature of the target text and the text description feature; encoding the glyph image of the target text through a dual-domain glyph encoding mechanism; completing bidirectional optimization through a dual-path alternating information flow in the spatial domain and the spectral domain, realizing mutual learning of the spatial local detail feature and the frequency domain global structure feature of the glyph, and obtaining dual-domain glyph conditional features;

[0014] combining the latent representation of the input noise image, the spatial layout feature, and the dual-domain glyph conditional feature to generate multi-scale conditional features;

[0015] extracting multi-scale skip connection features based on the latent representation of the noise image, the time step, and the text description feature from the down-sampling layer of the diffusion model, and obtaining initial backbone features from the bottleneck layer of the diffusion model; introducing a frequency domain perception refinement mechanism in the up-sampling layer of the diffusion model, which respectively performs instance-level scaling enhancement on the input backbone feature and the corresponding scale conditional feature, and performs adaptive dynamic modulation in the spectral domain on the corresponding scale skip connection feature and the enhanced conditional feature, and then concatenates the enhanced backbone feature with the enhanced conditional feature in the channel dimension, and processes them step by step through the up-sampling layer to obtain the denoised image latent representation, and finally decodes the visual text generation image; wherein the input backbone feature is obtained based on the initial backbone feature or is the output of the previous up-sampling layer.

[0016] A high-fidelity visual text image generation system for implementing the foregoing method, comprising:

[0017] a position image encoder for encoding the input position image and extracting the spatial layout feature of the target text;

[0018] a text encoder for encoding the input text description and extracting the text description feature;

[0019] a dual-domain glyph encoder for encoding the glyph image of the target text through a dual-domain glyph encoding mechanism, i.e., through bidirectional optimization in the spatial domain and the spectral domain, realizing mutual learning of the spatial local detail feature and the frequency domain global structure feature of the glyph, and obtaining dual-domain glyph conditional features;

[0020] a condition control network for combining the latent representation of the input noise image, the spatial layout feature, and the dual-domain glyph conditional feature to generate multi-scale conditional features;

[0021] The diffusion model includes a down-sampling layer, a bottleneck layer and an up-sampling layer; the down-sampling layer extracts multi-scale skip-connection features based on a latent representation of a noisy image, a time step and a text description feature, and obtains an initial backbone feature from the bottleneck layer; a frequency domain perception refinement mechanism is introduced in the up-sampling layer, which respectively performs instance-level scaling enhancement on the input backbone feature and the corresponding scale condition feature, performs adaptive dynamic modulation on the corresponding scale skip-connection feature and the enhanced condition feature in the frequency domain, and then splices the enhanced backbone feature and the enhanced condition feature in the channel dimension, and processes them step by step through the up-sampling layer to obtain a denoised image latent representation; wherein the input backbone feature is obtained based on the initial backbone feature or is the output of the previous up-sampling layer;

[0022] The variational autoencoder is used for outputting the latent representation of the noisy image and inputting into the diffusion model, and is also used for decoding the denoised image latent representation into a visual text generation image.

[0023] A processing device includes one or more processors; a memory for storing one or more programs;

[0024] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0025] A readable storage medium stores a computer program, which, when executed by a processor, implements the aforementioned method.

[0026] As can be seen from the technical solutions provided by the above-mentioned application, the spatial domain and frequency domain feature mutual learning is realized based on the dual-domain character coding mechanism, the problem of text structure disorder is solved, the frequency domain signal is dynamically modulated based on the frequency domain perception refinement mechanism, the stroke detail precision is improved, and the text details and background quality are cooperatively optimized through the instance-level scaling coefficient, thereby breaking through the bottleneck of text distortion and background instability in the prior art. The application supports accurate generation of multiple languages (including Chinese complex characters and the like), and can be widely applied to commercial scenarios such as advertisement design, product packaging, film poster, teaching material and the like, and can greatly improve the creation efficiency and reduce the manual correction cost; at the same time, the feasibility of cross-domain feature cooperative improvement of multi-modal generation efficiency is verified, and high-quality samples in a scarce scene can be provided for a text recognition model such as OCR (optical character recognition), thereby effectively filling the related data gap, and having advanced technology and market landing value. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creating laborious work.

[0028] Figure 1 A flow chart of a high-fidelity visual text image generation method provided by an embodiment of the present application.

[0029] Figure 2 A schematic diagram of the overall framework of a high-fidelity visual text image generation method provided by an embodiment of the present application.

[0030] Figure 3 A schematic diagram of a dual-domain glyph encoder provided by an embodiment of the present application.

[0031] Figure 4 A core flow schematic diagram of a frequency-aware refinement module provided by an embodiment of the present application.

[0032] Figure 5 An effect implementation comparison chart in English provided by an embodiment of the present application.

[0033] Figure 6 An effect implementation comparison chart in Chinese provided by an embodiment of the present application.

[0034] Figure 7 An effect implementation chart of the present application provided by an embodiment of the present application.

[0035] Figure 8 A schematic diagram of a high-fidelity visual text image generation system provided by an embodiment of the present application.

[0036] Figure 9 A schematic diagram of a processing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0038] First, the terms possibly used in the present text are explained as follows:

[0039] The terms “include”, “contain”, “have”, “possess” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example: including a technical feature element (such as raw materials, components, ingredients, carriers, dosage forms, materials, sizes, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or articles, etc.), should be interpreted as not only including the explicitly listed technical feature element, but also including other technical feature elements not explicitly listed but known in the art.

[0040] A high-fidelity visual text image generation method, system, device and storage medium are described in detail below. The content not described in detail in the embodiments of the present application belongs to the prior art known to those skilled in the art. If no specific conditions are specified in the embodiments of the present application, the conventional conditions or the conditions recommended by the manufacturer are used. If no manufacturer is specified for the instruments used in the embodiments of the present application, they are all conventional products that can be obtained by market purchase.

[0041] Embodiment one

[0042] The embodiments of the present application provide a high-fidelity visual text image generation method, as shown in the figure, mainly comprising the following steps: Figure 1

[0043] Step 1, information encoding.

[0044] In the embodiments of the present application, this part mainly relates to three types of information encoding: (1) encoding the input position image to extract the spatial layout features of the target text; (2) encoding the text description input by the user to extract the text description features; (3) encoding the glyph image of the target text through a dual-domain glyph encoding mechanism, that is, through the bidirectional optimization of the spatial domain and the frequency domain, the spatial local detail features and the frequency domain global structure features of the glyph are mutually learned to obtain the dual-domain glyph conditional features.

[0045] In the embodiments of the present application, the bidirectional optimization of the spatial domain and the frequency domain includes: optimizing the input information in the spatial domain to obtain output features with enhanced local detail perception; mapping the output features with enhanced local detail perception to the frequency domain to obtain frequency domain features, optimizing the frequency domain features, and converting the optimized frequency domain features back to the spatial domain, and fusing the output features with enhanced local detail perception through a residual mechanism to complete bidirectional optimization through dual-path alternating information flow; wherein the bidirectional optimization is executed multiple times, if it is the first execution, the input information is the glyph image of the target text, if it is not the first execution, the input information is the output of the last bidirectional optimization.

[0046] In the embodiments of the present application, the optimization processing of the frequency domain features includes: decomposing the frequency domain features into amplitude components and phase components, and based on the conjugate symmetry principle of Fourier transform, retaining half of the amplitude components and phase components, and splicing along the channel dimension to obtain complete frequency domain features; sequentially performing convolution processing, normalization processing and activation function processing on the complete frequency domain features to obtain the optimized frequency domain features.

[0047] Step 2, generate conditional information.

[0048] ​In the embodiment of the present application, based on the conditional control network, the latent representation of the input noise image, the spatial layout feature and the dual-domain glyph conditional feature are combined to generate multi-scale conditional features.

[0049] In the embodiment of the present application, first, the spatial layout feature and the dual-domain glyph conditional feature are spliced in the channel dimension, and the obtained spliced feature is subjected to convolution processing. Here, 3x3 convolution (padding=1) is preferably adopted to keep the feature map size unchanged and map the number of channels to the target dimension of the model (for example, 320). Among them, the convolution processing adopts a zero initialization (Zero Initialization) strategy, and the output thereof is used as a guided hint, which is injected into the feature extraction layer of the conditional control network in a residual addition manner with the latent representation of the noise image of the corresponding resolution. This design makes the value of the guided hint all zero in the initial stage of training, ensuring that the initial behavior of the conditional control network remains consistent when no conditions are added, thereby protecting the feature extraction capability of the pre-trained model from abrupt interference, and subsequently learning effective guide weights step by step through back propagation in the training process. Finally, the conditional control network processes the noise image based on the guided hint to generate multi-scale conditional features.

[0050] Step 3, output visual text generation image.

[0051] In the embodiment of the present application, the latent representation of the input noise image, the time step, the text description feature and the multi-scale conditional feature are input into the diffusion model, multi-scale skip connection features are extracted from each down-sampling layer of the diffusion model, and initial backbone features are obtained through the bottleneck layer of the diffusion model. At the same time, a frequency domain perception refinement mechanism is introduced in each up-sampling layer of the diffusion model, which respectively performs instance-level scaling enhancement on the input backbone feature and the conditional feature of the corresponding scale in the spatial domain, and performs adaptive dynamic modulation on the skip connection feature and the enhanced conditional feature in the frequency domain. After the backbone feature is spliced in the channel dimension, it is further processed through the up-sampling layer; the denoised image latent feature representation is obtained based on the step-by-step processing of each up-sampling layer, and the visual text generation image is decoded therefrom; wherein the input backbone feature of the up-sampling layer is the output of the previous up-sampling layer or is obtained based on the initial backbone feature.

[0052] In the embodiment of the present application, the diffusion model comprises a plurality of down-sampling layers and up-sampling layers, and a bottleneck layer; the latent representation of the input noise image, the time step and the text description feature are input into the down-sampling layer, and the output feature of each down-sampling layer is a single-scale skip connection feature, a plurality of multi-scale skip connection features are extracted layer by layer through the plurality of down-sampling layers, and hierarchical features from local details to global semantics are constructed; the bottleneck layer obtains an initial backbone feature using the output feature of the last down-sampling layer; after the initial backbone feature is fused with the conditional feature of the corresponding scale, the initial backbone feature is used as the backbone feature input into the first up-sampling layer, and the output feature of the last down-sampling layer is used as the skip connection feature input into the first up-sampling layer; the input backbone feature of each up-sampling layer is the output of the previous up-sampling layer, and the skip connection feature is the output feature of the corresponding down-sampling layer with the same spatial resolution; wherein, the multi-scale conditional feature is one-to-one aligned with the multi-scale skip connection feature and the initial backbone feature in resolution.

[0053] For a single up-sampling layer, the corresponding frequency domain perception refinement mechanism processing process is as follows: the input backbone feature is subjected to instance-level scaling enhancement in the spatial domain, an instance-level scaling coefficient is calculated based on the backbone feature, which is used to dynamically adjust the scaling amplitude of the backbone feature, and an enhanced backbone feature is output; the corresponding scale conditional feature is subjected to instance-level scaling enhancement in the same way, and an enhanced conditional feature is output; the skip connection feature and the enhanced conditional feature are adaptively dynamically modulated in the frequency domain to obtain modulated skip connection features and conditional features; the modulated skip connection features and conditional features are added element by element, and then concatenated with the enhanced backbone feature in the channel dimension to obtain frequency perception refinement features; the frequency perception refinement features are subjected to subsequent processing by the up-sampling layer to obtain the output feature of the up-sampling layer.

[0054] In the embodiment of the present application, the instance-level scaling enhancement in the spatial domain for the backbone feature and the output of the enhanced backbone feature comprise: for the pth up-sampling layer, the input backbone feature is up-sampled to obtain an up-sampled backbone feature , an adaptive adjustment factor is generated by a complexity perception network according to the up-sampled backbone feature : a global average pooling operation is used to compress the up-sampled backbone feature into a channel descriptor, and a global distribution statistic of the up-sampled backbone feature in the channel dimension is obtained, the global distribution statistic is input into a multi-layer perception structure, and a hyperbolic tangent activation function is used to output an adaptive adjustment factor with a numerical range of (-1, 1); the channel average feature map of the backbone feature is calculated, and a normalized feature map is obtained through normalization processing ; Combining normalized feature maps With adaptive adjustment factor Build instance-level scaling factor For the upsampled backbone features Channel-level refinement is represented as follows:

[0055] ;

[0056] ;

[0057] in, For enhanced skeletal features, This is a channel-level multiplication operation.

[0058] In this embodiment of the invention, the step of adaptively and dynamically modulating the skip connection features and the enhanced conditional features in the spectral domain to obtain the modulated skip connection features and conditional features includes: for the p-th upsampling layer, the corresponding skip connection feature is denoted as... The corresponding enhanced conditional features are denoted as ; The skip connection feature Enhanced conditional features Each frequency domain feature is mapped to the spectral domain to obtain its corresponding frequency domain features. For each frequency domain feature, it is divided into low-frequency and high-frequency components based on the radial distance of each frequency point from the spectral center (DC component) in the frequency domain, and the corresponding dynamic frequency weighting mask is determined. The corresponding frequency domain features are modulated using the dynamic frequency weighting mask, and then converted back to the spatial domain to obtain the modulated skip connection features. With conditional features .

[0059] In this embodiment of the invention, the step of dividing each frequency domain feature into low-frequency components and high-frequency components based on the radial distance of each frequency point from the center of the spectrum in the frequency domain, and determining the corresponding dynamic frequency weighting mask, includes:

[0060] For a single frequency domain feature, the radial distance of each frequency point from the center of the spectrum in the frequency domain is defined as follows: The boundaries between high and low frequency components are set as follows: It is a parameter adjusted according to the resolution; combined with and The frequency domain features are divided into low-frequency and high-frequency components, and corresponding modulation parameters are assigned to obtain the corresponding dynamic frequency weighting mask. ; Dynamic frequency weighted mask Defined as:

[0061] ;

[0062] wherein, and are learnable parameters, is used to modulate the low-frequency signal, is used to modulate the high-frequency signal;

[0063] for the skip connection feature corresponding to the frequency domain feature, the corresponding dynamic frequency weighting mask denoted as ; for the enhanced conditional feature corresponding to the frequency domain feature, the calculated dynamic frequency weighting mask denoted as .

[0064] The above scheme provided by the embodiments of the present application realizes mutual learning of spatial domain and frequency domain features based on a dual-domain character shape coding mechanism, solves the problem of disordered character structure, dynamically modulates frequency domain signals based on a frequency domain perception refinement mechanism to improve stroke detail accuracy, and realizes collaborative optimization of character details and background quality through instance-level scaling coefficients, thereby breaking through the bottleneck of character distortion and unstable background in the prior art. The present application supports accurate generation of multiple languages (including Chinese complex characters and the like), and can be widely applied to commercial scenarios such as advertisement design, product packaging, film and television posters, and teaching materials, thereby greatly improving creation efficiency and reducing manual correction costs. At the same time, the feasibility of collaborative improvement of cross-domain features to improve the efficiency of multi-modal generation is verified, which can provide high-quality samples in a scarce scenario for character recognition models such as OCR (Optical Character Recognition), effectively fill the data gap, and have advanced technology and market landing value.

[0065] In order to more clearly show the technical solutions provided by the present application and the technical effects produced, the method provided by the embodiments of the present application is described in detail below with specific embodiments.

[0066] I. Overall summary of the scheme.

[0067] In order to solve the technical problems existing in the prior art, the embodiments of the present application provide a high-fidelity visual character image generation method, mainly including:

[0068] 1. Design a dual-domain character shape coding mechanism: complete bidirectional optimization through dual-channel alternating information flow of spatial domain and frequency domain, realize mutual learning of spatial local detail features and frequency domain global structure features of character shapes, retain local details of character strokes, strengthen overall structure coherence, effectively solve problems such as disordered character structure and meaningless stroke combination, enhance the expression ability of character shape embedding, and realize fine control of the diffusion process.

[0069] 2. Propose a frequency domain perception refinement mechanism: introduce a frequency domain perception mechanism in the upsampling layer of the diffusion model, adaptively and dynamically modulate the decoupled high and low frequency signal components, and on the basis of stabilizing the overall structure of the low frequency text, focus on enhancing the capture ability of high frequency details such as small strokes and sharp edges, significantly improve the generation accuracy of text details, and solve the problem of blurred text edges and missing details.

[0070] 3. To solve the problem of unstable background caused by enhancing text details, on the basis of frequency domain perception refinement, combined with instance-level scaling coefficients, the intermediate features in the denoising process of the diffusion model are weighted and adjusted at the channel level, realizing the collaborative optimization of text detail enhancement and background quality stability, ensuring the natural integration of text and background in visual style and structure, and balancing the generation effect and overall picture coordination.

[0071] Thanks to the above improvements, the application can generate visual text with high readability in complex text rendering scenarios, and seamlessly integrate with natural background images.

[0072] II. Details of the scheme.

[0073] 1. Overall framework.

[0074] As shown in Figure 2 , to realize the overall framework of the method of the application, it mainly includes: position image encoder, text encoder (Caption Encoder), dual-domain glyph encoder (Dual-Domain Glyph Encoder, abbreviated as DoGE), conditional control network (ControlNet) and diffusion model, and the upsampling layer of the diffusion model is provided with a frequency perception refinement module (Frequency-Aware Refinement Module, abbreviated as FARM).

[0075] 2. Overall process.

[0076] (1) Auxiliary condition coding.

[0077] (1.1) Position image coding: The position image is used to accurately indicate the target area of the target text in the generated image, which is obtained from the real image by OCR detection technology during training, and can be specified by the user during inference. The position encoder extracts the spatial layout features of the target text in the image (position image features), providing accurate guidance for target text positioning.

[0078] (1.2) Text description coding: The text encoder encodes the user's demand into a semantic embedding (text description feature), wherein the target text (marked with double quotes) that needs to be accurately generated is replaced by an OCR model, realizing the cross-modal connection between text semantics and visual presentation.

[0079] In the embodiment of the present application, the user demand is a text description input by the user, including overall image requirements and specific target text to be rendered. For example: "Help me generate a promotional poster, the content is XX, and the text 'XX' is written on it." That is, it includes a description of the overall content and also includes specific text to be rendered.

[0080] As shown in Figure 2 , an example of the input text description is provided as "NEVER GIVE UP" Canvas print, in which the target text is marked with quotation marks, i.e., "NEVER GIVE UP" (i.e., never give up) is the target text, and the subsequent Canvas print is the related requirement. Of course, the target text and language here are only examples, and in actual applications, they can be replaced as needed.

[0081] (2) Dual-domain glyph feature extraction.

[0082] The dual-domain glyph encoder encodes the glyph image through a dual-domain glyph encoding mechanism to extract and fuse dual-domain glyph conditional features in the spatial and frequency domains, providing comprehensive visual condition guidance for text generation.

[0083] In the embodiment of the present application, the glyph image (for example, the printed glyph image) of the target text can be generated by a standard font rendering tool. The font involved can be set according to actual conditions, for example, it can be set as Arial font (which is a sans-serif font).

[0084] (3) Conditional feature generation.

[0085] In the embodiment of the present application, based on the conditional control network, the latent representation of the input noise image, the spatial layout feature, and the dual-domain glyph conditional feature are combined to generate multi-scale conditional features. In the embodiment of the present application, the spatial layout feature and the dual-domain glyph conditional feature are concatenated in the channel dimension, and the obtained concatenated feature is subjected to convolution processing. Here, a 3x3 convolution (padding=1) is preferably used to keep the feature map size unchanged and map the number of channels to the target dimension of the model (for example, 320). The convolution processing adopts a zero initialization (Zero Initialization) strategy, and the output thereof is used as a guided hint, which is injected into the feature extraction layer of the conditional control network in a residual addition manner with the latent representation of the noise image of the corresponding resolution. This design ensures that the value of the guided hint is zero in the initial stage of training, ensuring that the initial behavior of the conditional control network remains consistent with that without the condition, thereby protecting the feature extraction capability of the pre-trained model from abrupt interference, and subsequently learning effective guide weights through backpropagation during the training process. Finally, the conditional control network processes the noise image based on the guided hint to generate multi-scale conditional features.

[0086] After that, the multi-scale conditional features are injected into the bottleneck layer and the up-sampling layer of the diffusion model. The specific injection mechanism will be introduced in the frequency-aware refinement module section.

[0087] In the embodiment of the present application, the conditional control network replicates the encoder (down-sampling layer) and the bottleneck layer of the diffusion model in structure. The guided features fused through the above-mentioned method are input into the conditional control network, and the features are extracted layer by layer through the down-sampling layer and the bottleneck layer. Finally, the features extracted by each layer are output after being processed by the zero convolution layer, constituting multi-scale conditional features. Among them, the conditional features corresponding to the bottleneck layer are directly injected into the bottleneck layer of the diffusion model in a residual manner; and the conditional features corresponding to each level of down-sampling layer are input into the frequency-aware refinement module for fine processing before being fused with the features of the up-sampling layer of the diffusion model.

[0088] (4) Frequency-aware refinement-based image denoising generation.

[0089] The text description features and conditional features obtained through the above steps are input into the diffusion model, which also contains the latent representation of the time step t and the noise image.

[0090] In the embodiment of the present application, the diffusion model can be based on the U-Net symmetric architecture, including an encoding path, a decoding path and a bottleneck layer. The encoding path includes multiple down-sampling layers, which extract multi-scale features step by step to construct a hierarchical feature representation from local details to global semantics. The decoding path includes multiple up-sampling layers, and each up-sampling layer constructs features in the following way: (A1) performing up-sampling operation on the backbone features output by the previous layer and enhancing them to obtain strongly enhanced backbone features; (A2) obtaining the skip connection features of the corresponding scale in the encoding path, and fusing them with the enhanced conditional features of the same scale through the frequency-aware refinement mechanism; (A3) concatenating the enhanced backbone features and the fusion features output by step (A2) by channel to obtain frequency-aware refinement features; (A4) performing convolution on the frequency-aware refinement features to generate the output of the current layer. The specific technical details such as enhancement and fusion will be described in detail later. Specifically, the output features of the bottleneck layer of the diffusion model are added to the output conditional features of the bottleneck layer of the conditional control network, which are used as the input backbone features of the first up-sampling layer. The output of the last down-sampling layer is used as the skip connection feature. The input backbone features of the remaining up-sampling layers are the output of the previous up-sampling layer, and the skip connection features come from the corresponding down-sampling layer output in the encoding path with the same spatial resolution.

[0091] In the embodiment of the present application, the diffusion model realizes the interaction between the text description features and the latent representation of the noise image through the cross-attention mechanism. Specifically, the text description features are injected into the spatial transformer modules (Spatial Transformer Blocks) configured in the down-sampling path, the bottleneck layer and the up-sampling path of the diffusion model, thereby guiding the denoising process of the noise image at different feature scales.

[0092] Those skilled in the art can understand that (1) the above time steps are encoded as time embeddings, which act on each layer of the diffusion model, so that the diffusion model can dynamically adjust the denoising behavior according to the current noise level; (2) during training, the noise image is obtained by adding noise to the real image of the training set (the forward process of the diffusion model); during inference, Gaussian noise is directly initialized in the latent space as the latent representation input to the diffusion model.

[0093] In the embodiment of the present application, the conditional control network and the diffusion model are collectively referred to as a conditional diffusion model. The conditional diffusion model is an enhanced form of the diffusion model, which allows the generation process to be controlled by additional inputs (conditions). In the classical diffusion model, the image is generated by gradually denoising from random noise, and the generation result has randomness. The conditional diffusion model guides the model to generate images that meet specific requirements by injecting condition information into the denoising process at each step.

[0094] The present application is based on the latent diffusion model (Latent Diffusion Model) architecture. In the inference generation stage, Gaussian noise is directly initialized in the latent space as the latent representation of the noise image, and is input to the diffusion model. The diffusion model gradually denoises the latent representation in combination with the text description features and the condition features, and finally outputs the denoised latent representation, which is decoded back to the pixel space by the decoder of the VAE (Variational Autoencoder) to obtain the visual text generation image. In the training stage, the encoder of the VAE is used to map the training image to the latent representation and add noise. Considering that this part of the process belongs to the general knowledge in the art, it will not be described in detail.

[0095] A frequency-aware refinement module is configured in the up-sampling layer of the conditional diffusion model, which is used to optimize various features of the up-sampling process using a frequency domain-aware refinement mechanism, dynamically adjust high and low frequency components and stabilize background features, and finally output high-fidelity visual text generation images from the conditional diffusion model.

[0096] In addition to the need to input the above features, the conditional diffusion model also needs to input the time step t. Considering that the specific process involved in the conditional diffusion model can be implemented by conventional technology, it will not be described in detail.

[0097] 3. Module principle introduction.

[0098] (1) Dual-Domain Glyph Encoder.

[0099] Dual-Domain Glyph Encoder (DoGE) is one of the core modules to solve the problem of character distortion. Its core logic is to continuously learn and optimize each other through the dual-channel alternating information flow and residual mechanism of spatial domain and spectral domain features, extract the integrated glyph features with "local stroke details" and "global structure continuity", and avoid problems such as character stroke disorder and meaningless combination from the root.

[0100] As shown in Figure 3 , the dual-domain glyph encoder is composed of a series of dual-domain blocks connected in series, each dual-domain block internally includes a local spatial network and a global spectral network, and the information flow is alternately transmitted between the two networks to complete feature optimization, wherein the local spatial network enhances the local detail perception of the character, and the global spectral network enhances the global structure stability.

[0101] Taking the i-th dual-domain block as an example, the core process is as follows:

[0102] The dimension of the input information is , wherein H and W are the height and width of the glyph image respectively, r i is the down-sampling rate of the i-th dual-domain block, C i is the number of feature channels of the i-th dual-domain block, is the real set symbol. The local spatial network performs convolution operation on the input information through a 3x3 convolution layer (Conv 3x3), and at the same time completes the dimension mapping of the feature channel number to enrich the glyph information. After activation by SiLU activation function (a continuous, smooth and derivable activation function), the output feature enhanced in local detail perception is obtained, and the specific calculation process is as follows:

[0103] .

[0104] It should be noted that the 3x3 convolution layer and the SiLU activation function layer in the local spatial network are basic structures, and in actual application, the above basic structures can be stacked and designed. The basic structures are processed in sequence, and the output of the end basic structure is taken as the output feature of the local spatial network.

[0105] In the embodiment of the application, two basic structures can be stacked as a convolution processing group, and a plurality of convolution processing groups can be connected in series to form a local spatial network; for example,Figure 3 As shown, an example of two convolution processing groups in series is provided, under which each convolution processing group contains two 3x3 convolution layers and corresponding SiLU activation function layers. Specifically, the first convolution layer of each convolution processing group is used to spatially downsample the input features while increasing the number of feature channels, and the second convolution layer is used to further extract features while maintaining the current spatial resolution and number of feature channels; a SiLU activation function layer is connected after each convolution layer, which is a continuous, smooth, and derivable nonlinear function. By combining the downsampling operation with the increase in the number of channels, each convolution processing group can expand the receptive field while compensating for the loss of details caused by spatial information compression by increasing the number of feature channels, thereby retaining rich glyph information at a higher semantic level. The subsequent second convolution layer focuses on local feature refinement at the new spatial scale and channel dimension.

[0106] Subsequently, the feature F is mapped to the frequency domain using a fast Fourier transform F i , and the complex-valued spectral feature F i is obtained, with the conversion formula being:

[0107] ;

[0108] wherein, , correspond to the height and width of the converted spectral feature, respectively, (u, v) is the coordinate in the frequency domain, (h, w) is the coordinate of the current feature map in the spatial domain, is the value of the spectral feature F i at the coordinate (u, v), j is the imaginary unit, e is the natural constant, and π is the circular constant.

[0109] The global spectral network performs deep processing on the spectral feature F i . First, F i is decomposed into an amplitude component A i and a phase component P i , with the decomposition formula being:

[0110] ;

[0111] wherein R i and I i are the real and imaginary parts of F i , respectively, both having a dimension of , and arctan is the inverse tangent function.

[0112] Based on the conjugate symmetry principle of Fourier transform, only half of the spectrum is retained to fully characterize the original information, thereby reducing the computational cost. The amplitude component A i and the phase component Pi By concatenating along the channel dimension, we obtain a dimension of Complete spectral characteristics Specifically, it is expressed as:

[0113] .

[0114] Subsequently, complete spectral features After learning through a 1×1 convolutional layer (Conv 1×1) and performing batch normalization (BN), the optimized spectral features are obtained by activation using the ReLU (Modified Linear Unit) activation function. The calculation process is as follows:

[0115] .

[0116] Similarly, the structure of convolutional layers, batch normalization layers, and ReLU activation function layers in the global spectrum network can be set in multiple groups (e.g., two groups). When multiple groups are set, the groups are concatenated sequentially, with the output features of the previous group used as input for each subsequent group, and the output features of the last group used as the optimized spectrum features output by the global spectrum network. .

[0117] The above process utilizes the inherent global receptive field of the spectral domain to enhance the perception of the overall structure of characters, aiding in the learning of stroke combination rules. Finally, through the inverse fast Fourier transform F... -1 The optimized spectral domain features Transform back to the spatial domain, and then connect the output features of the local spatial network through the residual connection mechanism. The two domains are merged to form the input for the next two-domain block. The specific formula is as follows:

[0118] .

[0119] (2) Frequency perception refinement module.

[0120] The Frequency-Aware Refinement Module (FARM) is embedded in the upsampling layer of the diffusion model. It comprises two core components: instance-level scaling enhancement and adaptive dynamic modulation. These components are responsible for spatial feature stabilization and spectral feature modulation, respectively, ultimately constructing a complete frequency-aware refinement path to achieve synergistic improvement in visual text detail quality and background quality. Its core process is as follows: Figure 4 As shown, Figure 4 Different features are identified by using cube frames filled with different symbols.

[0121] (2.1) Instance-level scaling enhancement.

[0122] In the embodiment of the present application, the instance-level scaling coefficient is an adaptive scaling parameter dynamically generated based on the exclusive features of a single input instance, which is different from the traditional global fixed scaling coefficient (all input instances share the same set of parameters). The core feature is that the final value of the coefficient is not directly determined by the fixed parameters predefined by the model, but is combined with the feature representation (normalized feature distribution) of the current input instance to realize differentiated scaling. In generative models such as visual text image generation, this coefficient can dynamically adjust the scaling amplitude of the features according to the feature characteristics of different input instances (such as different text content, background complexity, etc.), so that the model has the ability to adapt to the feature distribution of a single instance, thereby enhancing the robustness of the model to complex and diverse input instances and improving the quality of the generated image background and text.

[0123] The core of the design of the instance-level scaling coefficient is to stabilize the latent features and enhance the denoising ability of the backbone model to ensure the generation quality of the text and background image. Specifically, for the p-th upsampling layer, the input backbone feature is upsampled to obtain the upsampled backbone feature ; a lightweight network (complexity-aware network) is used to extract the global context information of the instance, and a global average pooling (Global Average Pooling) operation is used to compress the spatial dimension of the upsampled backbone feature into a channel descriptor, thereby obtaining the global distribution statistics of the input feature in the channel dimension; then, the global distribution statistics are sent to a multi-layer perceptron (MLP) structure, which includes a dimension reduction layer and a dimension increase layer, and the feature expression ability is increased through a nonlinear activation function (ReLU) in the middle, and finally an adaptive adjustment factor between (-1, 1) is output through a Tanh (hyperbolic tangent) activation function; this process enables the model to automatically determine the degree of enhancement or suppression for each channel according to the texture richness and structural complexity of the input feature, and the generated vector is the adaptive adjustment factor for the current input feature (i.e., the upsampled backbone feature ).

[0124] In order to preserve the fine-grained information of the spatial structure, the present application further combines spatial dimension information to construct the final instance-level scaling coefficient. The channel average feature map of the upsampled backbone feature is calculated, that is, the mean value of all channels is calculated, and the formula is as follows:

[0125] ;

[0126] wherein, is the number of channels of the upsampled backbone feature , For the up-sampled backbone feature The feature map of the qth channel.

[0127] The channel average feature map is normalized to map its value range to the interval [0, 1] to obtain a normalized feature map , which is as follows:

[0128] ;

[0129] wherein, , is the maximum value and the minimum value in the channel average feature map .

[0130] This normalized feature map reflects the relative intensity distribution of the current instance in the spatial dimension. Combined with the adaptive adjustment factor and the normalized feature map , an instance-level scaling coefficient is constructed to refine the up-sampled backbone feature in the channel level, which is expressed as:

[0131] ;

[0132] ;

[0133] wherein, is the enhanced backbone feature, is a multiplication operation in the channel dimension.

[0134] In the embodiment of the present application, the instance-level scaling coefficient converts the spatial and channel statistical characteristics of each input instance into a modulation weight, so that the scaling coefficients of different instances in the same layer can change with their own sample feature distribution. In the training process, the complexity-aware lightweight network is optimized together with the main model, and the mapping relationship between feature complexity and scaling strength is adaptively learned. For example, when the background texture of the input feature is complex or the text details are more, the network will output a larger scaling factor to enhance the noise reduction and feature extraction capability; on the contrary, for a smooth background, a smaller factor is output to maintain the stability of the feature. In the inference stage, the complexity-aware network can directly infer the optimal adaptive adjustment factor in real time according to the input feature distribution. Based on the above enhancement scheme for the backbone feature, the same complexity-aware process is also used to process the conditional feature and the skip connection feature.

[0135] The above is an introduction to the instance-level scaling enhancement scheme for the backbone feature. For the conditional feature, a similar process is also used for instance-level scaling enhancement to consolidate the text conditional feature, which will not be described here.

[0136] By dynamically scaling both, the abnormal fluctuations in the potential space feature distribution caused by frequency modulation are effectively suppressed, improving the overall denoising ability of the conditional diffusion model and avoiding problems such as text or background image blurring and distortion.

[0137] (2.2) Adaptive dynamic modulation.

[0138] This part is dynamically modulated by adaptive frequency domain weighting mask, which focuses on precise modulation of high and low frequency signals of intermediate frequency domain features, while stabilizing low frequency structure and gradually enhancing the model's perception of high frequency signals, improving text rendering quality.

[0139] Similarly, take the pth up-sampling layer as an example, map the skip connection feature and the enhanced conditional feature to the frequency domain respectively to obtain their corresponding frequency domain features; for each frequency domain feature, according to the radial distance of each frequency point from the frequency domain center (direct current component), it is divided into low frequency component and high frequency component, and the corresponding dynamic frequency weighting mask is determined; after modulating the corresponding frequency domain feature using the dynamic frequency weighting mask, it is converted back to the spatial domain through inverse Fourier transform to obtain the modulated skip connection feature and the conditional feature .

[0140] Considering the different needs for frequency signals at different stages of the denoising process (generally, deep networks focus on restoring global low frequency structure, and shallow networks focus on restoring local high frequency details), the dynamic frequency weighting mask is defined as:

[0141] ;

[0142] wherein, and are learnable parameters, modulate low frequency signals to ensure the coherence of the overall text structure, modulate high frequency signals to accurately capture and restore key high frequency details such as small strokes and sharp edges; the boundary between high and low frequency components can be adjusted according to actual conditions, for example, the value is dynamically adjusted to 1 / 8 of the up-sampling resolution to adapt to the frequency needs of features of different scales.

[0143] Based on the above method, the corresponding dynamic frequency weighting mask of the skip connection feature and the enhanced conditional feature is calculated. .

[0144] Based on the above introduction, the frequency-aware refinement module constructs a comprehensive dynamic frequency-aware path, optimizes the backbone feature, the conditional feature and the skip connection feature in coordination to improve the generation effect, which is formalized as:

[0145] ;

[0146] ;

[0147] ;

[0148] ;

[0149] wherein, , is the instance-level scaling coefficient of the conditional feature , is the element multiplication according to the spatial position, + is the element-wise addition, is the channel dimension splicing, is the frequency-aware refinement feature.

[0150] Based on the above formula, the main process of the frequency-aware refinement module can be described as: the backbone feature is enhanced by instance-level scaling; after the conditional feature is enhanced by instance-level scaling and adaptive dynamic modulation, it is mapped back to the spatial domain by inverse fast Fourier transform; after the skip connection feature is adaptively dynamically modulated and mapped back to the spatial domain, it is added to the enhanced conditional feature element by element, and finally the optimized backbone feature is obtained by feature splicing operation. The feature The subsequent processing process is performed through the pth upsampling layer, which is the steps (A1)~(A4) introduced in the foregoing, and the backbone feature is input to the (p+1)th upsampling layer, thereby providing an input feature with both detail precision and structural stability for the next round of upsampling, realizing seamless integration of text and background.

[0151] The above is the principle introduction of the core module of the present application, and the parts not described in detail, such as the specific principles of the position image encoder, the text encoder, the conditional control network and the diffusion model, can refer to the conventional technology, and the present application will not be repeated.

[0152] III. Example Introduction

[0153] 1. Overall architecture example

[0154] The method of the present application takes Stable Diffusion v1.5 and ControlNet as the basic conditional diffusion model, and modifies it to support the joint generation of images and text. The U-Net in Stable Diffusion v1.5 is used as the core generation model, and the FARM (Frequency-Aware Refinement Module) is embedded to predict the denoised latent representation from the noisy latent representation. ControlNet is used to fuse various conditional encodings, including spatial layout features and dual-domain glyph conditional features. In addition, VAE (Variational Autoencoder) is used to compress the input image into a latent space representation and decode it back to a pixel space image after denoising. Considering that this part of the process belongs to general knowledge in the field, it is not described in detail.

[0155] 2. Training process.

[0156] The above overall architecture is trained on a large-scale multilingual public dataset AnyWord-3M excluding a subset of watermarks, which contains about 2.48 million images. The dataset covers real commercial scene images including advertisements, posters, packaging, etc.

[0157] The training is performed on a server equipped with 32 NVIDIA V100 GPUs. All input image sizes are adjusted to 512x512 pixels. The training strategy uses a two-stage training process. In the first stage, the standard denoising loss is used for 10 rounds, and in the second stage, the OCR perceptual loss is added for another 5 rounds. The learning rate is 1e−5, the AdamW optimizer is used, and the global batch size is 128.

[0158] During the training process, the parameters of the position encoder, dual-domain glyph encoder, conditional control network, and frequency-aware refinement module are mainly optimized.

[0159] (3) Implementation details.

[0160] The standard font rendering tool uses the Pillow library (a powerful image processing library) of Python (an interpreted scripting language). The text encoder uses the text encoder in the CLIP-VIT-base-patch32 model, CLIP is a contrastive language-image pre-training model, VIT is a visual transformation neural network, base represents the basic scale, and patch32 represents a patch size of 32; the OCR encoding and OCR detection model of the target text both use PP-OCRv3, which is the third version of the practical super-lightweight OCR system. The position encoder uses 6 stacked convolutional layers, normalization and ReLU activation functions.

[0161] During the training process, the training parameters of the text encoder and the original pre-training network parameters of the U-Net in the denoising network are frozen, so as to fully utilize the knowledge of the pre-training model, ensure that the model retains the original background generation capability, and reduce the training cost while ensuring the effect.

[0162] IV. Performance description

[0163] In order to illustrate the performance of the above-mentioned scheme of the present application, the effect is verified through experiments.

[0164] In order to ensure the fairness of comparison, all methods are evaluated under consistent settings. In order to comprehensively measure the performance of the model, the following evaluation indicators are used: the OCR accuracy is calculated using PP-OCRv3 to measure the recognizability of the generated text; the normalized edit distance (Normalized Edit Distance, NED) is used to quantify the character-level text similarity; the CLIPScore (CLIP score) is introduced to evaluate the semantic alignment degree of text-image; the FID (it compares the distribution difference of generated images and real images in the Inception network feature space) is used to evaluate the overall image generation quality, and the Inception network is a convolutional neural network architecture.

[0165] The existing methods involved in the experiment are introduced as follows: (1) ControlNet: an existing conditional control network that introduces multi-modal conditional signals such as edge maps and depth maps to constrain the picture structure and layout of the generated content, injects precise control capability into the diffusion model, and realizes fine guidance of the generated results. (2) GlyphControl: an early visual text generation method based on conditional control network and stable diffusion model, which introduces standard font glyph images as conditions to improve the accuracy of visual text generation. (3) Textdiffuser: an early visual text generation model based on diffusion model, which improves the distortion problem of visual text generation by introducing text position prediction conditions and OCR perception loss function. (4) Textdiffuser-2: an upgraded version of Textdiffuser, which further optimizes the performance of long text generation and improves the rationality of text layout planning through two-stage training and hybrid coding mechanism. (5) UdiffText: a diffusion model scheme focusing on visual text editing, which only supports English scenarios and lacks adaptation ability for complex languages such as Chinese. (6) AnyText: a glyph image assisted visual text generation scheme that uses glyph images of visual text as conditions to inject diffusion models, but relies only on spatial domain operations, and has limited modeling ability for the overall structure of the text. (7) GlyphByT5: a visual text generation model that trains a text encoder with large-scale paired annotation data to optimize character generation accuracy. (8) ARTIST: a pre-trained special text diffusion model that captures text structure and integrates a pre-trained large language model to understand user intent, improving the accuracy of text rendering for rich text images. (9) SA-OCR-Paint: a benchmark dataset specifically designed to test the ability of long and complex visual text generation, and a method to enhance text generation without training framework.

[0166] On the widely recognized AnyText benchmark and MARIO-Eval benchmark, the method of the present application achieved the highest OCR accuracy and NED score on both Chinese and English datasets, highlighting its excellent ability to generate accurate visual text; the competitive CLIPScore indicates that the model has achieved strong semantic alignment; and the FID result shows that the method effectively guarantees the quality of background generation, achieving a key balance between text recognizability and overall image coherence.

[0167] The experimental results show that the present application can generate accurate and background style consistent visual text images, especially in scenes where other methods cannot guarantee recognizability. For English text, the present application can render complex details more clearly and accurately, such as difficult-to-distinguish letters, repeated letters in long sequences, and connected parts; for Chinese characters, especially for characters with complex stroke structure and high visual density, the accuracy of the results generated by the present application is significantly higher. Specifically as follows:

[0168] (1) The text generation accuracy is greatly improved. Visual text in different languages can be accurately generated, effectively avoiding common problems such as character repetition, spelling errors, and stroke distortion in existing technologies, especially in the generation accuracy of complex Chinese characters. As shown in Figure 5 is an effect comparison chart implemented on English, the left first column (Prompt) is an example of prompt text (i.e. input text description), and the remaining columns are images generated by corresponding methods, as shown in Figure 6 is an effect comparison chart implemented on Chinese.

[0169] (2) The core performance is better than existing technologies: through comprehensive comparison experiments with existing methods, the present application (DualDiff) in two public benchmark data sets, word-level text accuracy (ACC), NED, CLIPScore, FID evaluation indicators all show excellent performance, overall better than other models, and requires less training data. As shown in Tables 1-2.

[0170] Table 1: Quantitative comparison results of the present application and existing methods in AnyText-Benchmark

[0171]

[0172] Table 2: Quantitative comparison results of the present application and existing methods in MARIO-Eval-Benchmark

[0173]

[0174] (3) The text style is diverse and consistent with the background. As shown in Figure 7 is an implementation effect chart of the present application, it can be seen that the present application has the ability to generate diverse fonts, colors and other attributes, and can stably output high-fidelity visual text consistent with the background style in different scenes, adapting to real application needs.

[0175] In summary, the present application not only outperforms existing technologies in quantitative indicators, but also shows significant advantages in qualitative effects, fully demonstrating its innovation and practicality in the field of visual text generation.

[0176] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software and necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application.

[0177] Embodiment two

[0178] The present application also provides a high-fidelity visual text image generation system, which corresponds to the overall framework provided by the above-mentioned embodiments, and is mainly used to implement the method provided by the above-mentioned embodiments, as shown in the figure, which mainly includes: Figure 8

[0179] A position image encoder is used to encode the input position image and extract the spatial layout features of the target text.

[0180] A text encoder is used to encode the input text description and extract the text description features.

[0181] A dual-domain glyph encoder is used to encode the glyph image of the target text through a dual-domain glyph encoding mechanism, that is, to complete bidirectional optimization through a dual-channel alternating information flow of the spatial domain and the frequency domain, to realize mutual learning of the spatial local detail features and the frequency domain global structure features of the glyph, and to obtain dual-domain glyph conditional features.

[0182] A condition control network is used to generate multi-scale conditional features in combination with the latent representation of the input noise image, the spatial layout features and the dual-domain glyph conditional features.

[0183] A diffusion model includes a down-sampling layer, a bottleneck layer and an up-sampling layer; the down-sampling layer extracts multi-scale skip connection features based on the latent representation of the noise image, the time step and the text description features, and obtains an initial backbone feature from the bottleneck layer; in the up-sampling layer, a frequency domain perception refinement mechanism is introduced, which respectively performs instance-level scaling enhancement on the input backbone feature and the corresponding scale conditional feature, performs adaptive dynamic modulation in the frequency domain on the corresponding scale skip connection feature and the enhanced conditional feature, and after the enhanced backbone feature is spliced with the enhanced conditional feature in the channel dimension, the up-sampling layer is gradually processed to obtain the denoised image latent representation; wherein the input backbone feature is obtained based on the initial backbone feature or is the output of the previous up-sampling layer.

[0184] A variational autoencoder is used to output the latent representation of the noise image and input it into the diffusion model, and is also used to decode the denoised image latent representation into a visual text generation image.​

[0185] In view of the main technical details involved in the system have been described in detail in the previous embodiments, no longer tedious.

[0186] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional modules is taken as an example, and in actual application, the above-mentioned functions can be completed by different functional modules according to needs, that is, the internal structure of the system is divided into different functional modules to complete all or part of the functions described above.

[0187] Example three

[0188] The application also provides a processing device, as shown in the figure, mainly comprising: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors realize the method provided by the preceding embodiments. Figure 9

[0189] Further, the processing device further comprises at least one input device and at least one output device; in the processing device, the processor, the memory, the input device, the output device are connected through the bus.

[0190] In the embodiments of the application, the specific types of the memory, the input device and the output device are not limited; for example:

[0191] The input device can be a touch screen, an image acquisition device, a physical button or a mouse, etc.

[0192] The output device can be a display terminal.

[0193] The memory can be a random access memory (RAM), and can also be a non-volatile memory, such as a disk memory.

[0194] Example four

[0195] The application also provides a readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the method provided by the preceding embodiments is realized.

[0196] In the embodiments of the application, the readable storage medium as a computer readable storage medium can be arranged in the preceding processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, etc. Various media that can store program codes. ​

[0197] The above description is only the preferred embodiment of the present application, but the protection scope of the present application is not limited to this. Any change or replacement within the technical range disclosed by the present application can be easily thought by those skilled in the art, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims. The information disclosed in the background section of the present application is only intended to deepen the understanding of the general background of the present application, and should not be regarded as acknowledging or implying in any form that the information constitutes the prior art known by those skilled in the art.

Claims

1. A high-fidelity visual text image generation method, characterized by, The method comprises: respectively encoding the input position image and the text description, and correspondingly extracting spatial layout features and text description features of the target text; encoding the glyph image of the target text through a dual-domain glyph encoding mechanism: completing bidirectional optimization through dual-path alternating information flow in the spatial domain and the spectral domain, realizing mutual learning of spatial local detail features and frequency domain global structure features of the glyph, and obtaining dual-domain glyph conditional features; combining the latent representation of the input noise image, the spatial layout features, and the dual-domain glyph conditional features to generate multi-scale conditional features; extracting multi-scale skip connection features based on the latent representation of the noise image, the time step, and the text description features from the down-sampling layer of the diffusion model, and obtaining initial backbone features from the bottleneck layer of the diffusion model; introducing a frequency domain perception refinement mechanism in the up-sampling layer of the diffusion model, which respectively performs instance-level scaling enhancement on the input backbone features and the corresponding scale conditional features, performs adaptive dynamic modulation in the spectral domain on the corresponding scale skip connection features and the enhanced conditional features, and concatenates the enhanced backbone features with the enhanced conditional features in the channel dimension, and then processes them step by step through the up-sampling layer to obtain the denoised image latent representation, and finally decodes the visual text generation image; wherein the input backbone features are obtained based on the initial backbone features or are the output of the previous up-sampling layer.

2. The high-fidelity visual text image generation method of claim 1, wherein, The bidirectional optimization through dual-path alternating information flow in the spatial domain and the spectral domain comprises: optimizing the input information in the spatial domain to obtain output features with enhanced local detail perception; mapping the output features with enhanced local detail perception to the spectral domain to obtain spectral features, optimizing the spectral features, and then converting the optimized spectral features back to the spatial domain to fuse them with the output features with enhanced local detail perception through a residual mechanism to complete one bidirectional optimization; wherein the bidirectional optimization is executed multiple times, if it is the first execution, the input information is the glyph image of the target text, and if it is not the first execution, the input information is the output of the last bidirectional optimization.

3. The high-fidelity visual text image generation method of claim 2, wherein, The optimization processing of the spectral features comprises: decomposing the spectral features into amplitude components and phase components, based on the conjugate symmetry principle of Fourier transform, retaining half of the amplitude components and the phase components, and concatenating them along the channel dimension to obtain complete spectral features; and sequentially performing convolution processing, batch normalization processing, and activation function processing on the complete spectral features to obtain optimized spectral features.

4. The high-fidelity visual text image generation method of claim 1, wherein, The diffusion model comprises multiple down-sampling layers and up-sampling layers, and a bottleneck layer; Each down-sampling layer outputs a single-scale skip-connection feature, and multiple down-sampling layers are used to extract multi-scale skip-connection features layer by layer; the bottleneck layer uses the output feature of the last down-sampling layer to obtain an initial backbone feature; after the initial backbone feature is fused with the conditional feature of the corresponding scale, the initial backbone feature is used as the input backbone feature of the first up-sampling layer, and the output feature of the last down-sampling layer is used as the skip-connection feature of the first up-sampling layer; the input backbone feature of each up-sampling layer is the output of the previous up-sampling layer, and the skip-connection feature is the output feature of the corresponding down-sampling layer with the same spatial resolution; wherein, the multi-scale conditional features, the multi-scale skip-connection features and the initial backbone feature are one-to-one aligned in resolution; For a single up-sampling layer, the corresponding frequency domain-aware refinement mechanism processing process is as follows: the backbone feature is scaled and enhanced at the instance level in the spatial domain, the instance-level scaling coefficient is calculated based on the backbone feature, which is used to dynamically adjust the scaling amplitude of the backbone feature, and the enhanced backbone feature is output; the corresponding scale conditional feature is scaled and enhanced at the instance level in the same way, and the enhanced conditional feature is output; the modulated skip-connection feature and the conditional feature are adaptively and dynamically modulated in the frequency domain, and the modulated skip-connection feature and the conditional feature are obtained; the modulated skip-connection feature and the conditional feature are added element by element, and then the enhanced backbone feature is concatenated in the channel dimension, and the frequency-aware refined feature is obtained; the frequency-aware refined feature is processed by the up-sampling layer, and the output feature of the up-sampling layer is obtained.

5. The high-fidelity visual text image generation method of claim 4, wherein, The instance-level scaling and enhancement of the backbone feature in the spatial domain includes: For the pth upsampling layer, the input backbone feature is upsampled to obtain an upsampled backbone feature ; an adaptive adjustment factor is generated from the upsampled backbone feature by a complexity-aware network : the upsampled backbone feature is compressed into a channel descriptor by a global average pooling operation , to obtain an upsampled backbone feature The global distribution statistics in the channel dimension are fed into a multi-layer perceptron structure, and an adaptive adjustment factor with a value range of (-1, 1) is output through a hyperbolic tangent activation function ;​ Calculate the backbone features after upsampling Channel average feature map Then, through normalization processing, a normalized feature map is obtained. Combined with adaptive adjustment factor With normalized feature map Build instance-level scaling factor And the upsampled backbone features Channel-level refinement is represented as follows: ； ； wherein, are the enhanced backbone features, is a multiplication operation in the channel dimension.

6. The high-fidelity visual text image generation method of claim 4 or 5, wherein, The adaptive and dynamic modulation of the skip-connection feature and the enhanced conditional feature in the frequency domain includes: For the p-th up-sampling layer, the corresponding skip connection feature is denoted as The corresponding enhanced conditional feature is denoted as ; Jump connection features with enhanced conditional features are respectively mapped to a frequency domain to obtain respective corresponding frequency domain features; for each frequency domain feature, according to a radial distance of each frequency point from a frequency spectrum center in the frequency domain, the frequency domain feature is divided into a low frequency component and a high frequency component, and a corresponding dynamic frequency weighting mask is determined; the corresponding frequency domain feature is modulated using the dynamic frequency weighting mask, and then converted back to a spatial domain to obtain a modulated jump connection feature with conditional features .

7. The high-fidelity visual text image generation method of claim 6, wherein, The division of each frequency domain feature into a low-frequency component and a high-frequency component according to the radial distance of each frequency point from the spectral center in the frequency domain, and the determination of the corresponding dynamic frequency weighting mask include: For a single frequency domain feature, set the radial distance of each frequency point from the center of the frequency spectrum in the frequency domain as , and set the boundary of the high and low frequency components as , which is a resolution-adjusted parameter; in combination with and , the frequency domain feature is divided into low and high frequency components, and the corresponding modulation parameters are allocated to obtain the corresponding dynamic frequency weighting mask ; the dynamic frequency weighting mask is defined as: ； wherein, and are learnable parameters, for modulating the low frequency signal, for modulating the high frequency signal; For the skip connection feature The corresponding frequency domain feature, the corresponding dynamic frequency weighting mask Denoted as ; for the enhanced conditional feature The corresponding frequency domain feature, the corresponding dynamic frequency weighting mask Denoted as .

8. A high-fidelity visual text image generation system, characterized by, A method for implementing any one of claims 1-7, comprising: a position image encoder configured to encode an input position image to extract a spatial layout feature of a target text; a text encoder configured to encode an input text description to extract a text description feature; a dual-domain glyph encoder configured to encode a glyph image of the target text through a dual-domain glyph encoding mechanism, i.e., through a dual-path alternating information flow in the spatial domain and the frequency domain to complete bidirectional optimization, realize mutual learning of spatial local detail features and frequency domain global structure features of the glyph, and obtain dual-domain glyph conditional features; a conditional control network configured to generate multi-scale conditional features in combination with a latent representation of an input noise image, the spatial layout feature, and the dual-domain glyph conditional features; and a text generator configured to generate a text image in combination with the multi-scale conditional features. The diffusion model comprises a down-sampling layer, a bottleneck layer and an up-sampling layer; the down-sampling layer extracts multi-scale skip-connection features based on a latent representation of a noisy image, a time step and a text description feature, and obtains an initial backbone feature from the bottleneck layer; a frequency domain perception refinement mechanism is introduced in the up-sampling layer, which respectively performs instance-level scaling enhancement on input backbone features and corresponding scale condition features, performs adaptive dynamic modulation on corresponding scale skip-connection features and enhanced condition features in a frequency domain, and after being spliced with enhanced backbone features in a channel dimension, gradually processes through the up-sampling layer to obtain a denoised image latent representation; wherein the input backbone feature is obtained based on the initial backbone feature or is an output of a previous up-sampling layer; a variational autoencoder, configured to output a latent representation of a noisy image and input to the diffusion model, and also configured to decode the denoised image latent representation into a visual text generation image.

9. A processing device, characterized by comprise: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-7.

10. A readable storage medium, storing a computer program, characterized in that, The computer program, when executed by the processor, implements the method of any one of claims 1-7.

Images