Style image generation method, apparatus, device, medium, and product

Abstract

The application provides a style image generation method, device, equipment, medium and product. The method comprises: obtaining input data comprising at least a description text; inputting the description text into a text encoder to obtain text encoding features comprising at least word-level encoding features; inputting the text encoding features into a text structure projection head and a text style projection head respectively to obtain word-level text structure features and word-level text style features; inputting the word-level text structure features and the word-level text style features into a style-aware gating module to obtain word-level style gating weights; determining word-level modulation features based on the word-level style gating weights; inputting the text encoding features into an image generation model to obtain a style image output by the image generation model, wherein word-level intermediate features obtained by processing the text encoding features in the image generation model are modulated by using the word-level modulation features. The application can improve the generalization and automation of style image generation.

Description

Technical Field

[0001] This application relates to the field of image generation technology, and in particular to style image generation methods, apparatus, devices, media and products. Background Technology

[0002] Existing style image generation techniques typically rely on manual selection of style words from prompts. Different style words are then manually assigned weights, which are then weighted and reinforced in the model's processing mechanism to guide the model in generating outputs of a specific style. This approach is highly subjective and lacks scalability. Especially in complex or multi-style prompt scenarios, the word selection and weighting processes struggle to maintain consistency, resulting in low generalization and automation levels in style image generation. Summary of the Invention

[0003] This application provides a method, apparatus, device, medium, and product for generating style images, in order to solve the defects of the prior art that relies on manual selection of style words and setting weights, resulting in low generalization and automation of style image generation, and to improve the generalization and automation of style image generation.

[0004] This application provides a style image generation method, including: Obtain input data, which includes at least descriptive text; The descriptive text is input into a text encoder to obtain the text encoding features output by the text encoder, wherein the text encoding features include at least word-level encoding features; The text encoding features are input into the text structure projection head and the text style projection head respectively to obtain word-level text structure features and word-level text style features; The word-level text structure features and word-level text style features are input into the style-aware gating module to obtain the word-level style gating weights output by the style-aware gating module. Based on the word-level style gating weights, word-level modulation features are determined; The text encoding features are input into an image generation model to obtain a style image output by the image generation model. The word-level intermediate features obtained by processing the text encoding features in the image generation model are modulated using the word-level modulation features.

[0005] According to the style image generation method provided in this application, the image generation model includes a noise prediction model and a denoising module. The text encoding features are input into the image generation model to obtain the style image output by the image generation model, including: Initial noise is generated by inputting the text encoding features and the noise at the current time step into the noise prediction model to obtain the predicted noise at the next time step output by the noise prediction model. Based on the predicted noise at the next time step, the noise at the next time step is denoised. The denoising result at the last time step is used as the style image, and the noise at the first time step is the initial noise. In each time step, the intermediate word-level features obtained by processing the text encoding features in the noise prediction model are modulated based on the word-level modulation features of the current time step. The step of determining word-level modulation features based on the word-level style gating weights includes: Obtain the control ratio at each time step; The word-level style gating weights at the current time step are processed based on the control ratios at each time step to obtain the word-level modulation features at each time step.

[0006] According to the style image generation method provided in this application, the noise prediction model includes a cross-attention layer and a classification-free guided model; The word-level intermediate features obtained by processing the text encoding features are modulated using the word-level modulation features, including: The attention vectors of each word in the cross-attention layer and the conditional difference terms of each word in the classification-guided layer are modulated based on the word-level modulation features.

[0007] According to the style image generation method provided in this application, the text encoding features further include sentence-level text encoding features; the acquisition of the control ratio at each time step includes: The control ratio is determined based on a preset time function or user input data; or, The sentence-level text encoding features are input to the proportional controller to obtain the control ratio output by the proportional controller; or, The control ratio is generated based on the generation process signal of the style image.

[0008] According to the style image generation method provided in this application, the step of inputting the word-level text structure features and the word-level text style features into a style-aware gating module to obtain the word-level style gating weights output by the style-aware gating module includes: Multiple initial word-level style weights are determined using at least one weight determination method, the word-level text structure features, and the word-level text style features. The multiple initial word-level style weights are fused to obtain the word-level style gating weights; The word-level style gating weights are sparsed.

[0009] According to the style image generation method provided in this application, the training process of the text encoder, the text structure projection head, the text style projection head, and the style-aware gating module includes: Acquire sample data, which includes sample text and sample images; The sample text is input into the text encoder to obtain the sample text encoding features output by the text encoder. The sample image is input into the image encoder to obtain the sample image encoding features output by the image encoder. The sample image encoding features include local image encoding features and global image encoding features. The sample text encoding features are input into the text structure projection head and the text style projection head, respectively, and the sample image encoding features are input into the image structure projection head and the image style projection head, respectively, to obtain text structure features, text style features, image structure features and image style features; Based on the word-level style gating weights, the text structure features and the text style features are aggregated to obtain aggregated text structure features and aggregated text style features. Based on the image structure features and the image style features, the aggregated text structure features and the aggregated text style features are aligned with the image modality to obtain aligned text structure features and aligned text style features. Based on the control ratio, the aligned text structural features and the aligned text style features are fused to obtain a unified target; The local image coding features and the global image coding features are aggregated to obtain aggregated image features; The training loss is determined based on the difference between the unified objective and the aggregated image features; The text encoder, the image encoder, the text structure projector, the text style projector, the image structure projector, the image style projector, and the style-aware gating module are updated based on the training loss.

[0010] This application also provides a style image generation apparatus, including: An input data acquisition module is used to acquire input data, which includes at least descriptive text; A text encoding module is used to input the description text into a text encoder to obtain text encoding features output by the text encoder, wherein the text encoding features include at least word-level encoding features; The dual projection module is used to input the text encoding features into the text structure projection head and the text style projection head respectively to obtain word-level text structure features and word-level text style features; The style gating weight determination module is used to input the word-level text structure features and the word-level text style features into the style-aware gating module to obtain the word-level style gating weights output by the style-aware gating module. The modulation feature determination module is used to determine word-level modulation features based on the word-level style gating weights; An image generation module is used to input the text encoding features into an image generation model and obtain a style image output by the image generation model, wherein the word-level intermediate features obtained by processing the text encoding features in the image generation model are modulated using the word-level modulation features.

[0011] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the style image generation method as described above.

[0012] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the style image generation method as described above.

[0013] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the style image generation method as described above.

[0014] The style image generation method, apparatus, device, medium, and product provided in this application acquire input data including descriptive text, encode the descriptive text by inputting it into a text encoder, and then project the text encoding features based on a text structure projection head and a text style projection head to obtain word-level text structure features and text style features in the descriptive text. These word-level text structure features and text style features are then input into a style-aware gating module to obtain word-level style gating weights output by the style-aware gating module. Based on these word-level style gating weights, word-level modulation features are determined. During the process of inputting the text encoding features into an image generation model for image generation, the intermediate features of the model are modulated based on the word-level modulation features. This eliminates the need for manual word selection or a style prototype library; instead, the model automatically identifies potential style factors in the descriptive text, achieving adaptive style guidance and weight allocation, thus improving the generalization and automation of style image generation. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating the style image generation method provided in this application.

[0017] Figure 2 This is a schematic diagram of the data processing process in the style image generation method provided in this application.

[0018] Figure 3 This is a schematic diagram of the style image generation device provided in this application.

[0019] Figure 4 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0022] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0023] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0025] The following is combined with Figure 1 Describe the style image generation method provided in this application. For example... Figure 1 As shown, this style image generation method includes the following steps: S110. Obtain input data, which includes at least descriptive text; S120. Input the descriptive text into the text encoder to obtain the text encoding features output by the text encoder. The text encoding features include at least word-level encoding features. S130. Input the text encoding features into the text structure projection head and the text style projection head respectively to obtain word-level text structure features and word-level text style features; S140. Input the word-level text structure features and word-level text style features into the style-aware gating module to obtain the word-level style gating weights output by the style-aware gating model. S150. Determine word-level modulation features based on word-level style gating weights; S160. Input the text encoding features into the image generation model and obtain the style image output by the image generation model. The word-level intermediate features obtained by the image generation model from the text encoding features are modulated using word-level modulation features.

[0026] The style image generation method provided in this application acquires input data including descriptive text, encodes the descriptive text by inputting it into a text encoder, projects the text encoding features based on a text structure projection head and a text style projection head to obtain word-level text structure features and text style features in the descriptive text, and further inputs these word-level text structure features and text style features into a style-aware gating module to obtain word-level style gating weights output by the style-aware gating module. Based on these word-level style gating weights, word-level modulation features are determined. During the process of inputting the text encoding features into the image generation model for image generation, the intermediate features of the model are modulated based on the word-level modulation features. In this way, there is no need to rely on manual word selection or style prototype libraries. Instead, the model automatically identifies potential style factors in the descriptive text, achieving adaptive style guidance and weight allocation, thereby improving the generalization and automation of style image generation.

[0027] In the method provided in this application, the input data is data used to guide the generation of style images. The input data includes descriptive text, which describes the image the user wants to generate. For example, the descriptive text could be: "Give me an image of students learning in a classroom." The descriptive text is first input into a text encoder for semantic encoding to obtain text encoding features. In the method provided in this application, the text encoding features output by the text encoder include at least word-level encoding features, i.e., the encoding features corresponding to each word or phrase. In one possible implementation, the text encoder is a multi-granular encoder; that is, the text encoding features output by the text encoder include sentence-level encoding features in addition to word-level encoding features. In this implementation, sentence-level encoding features can be optionally utilized in subsequent steps, as will be explained later.

[0028] Furthermore, since the descriptive text is used to generate style images, the space of text-encoded features obtained by encoding the descriptive text should ideally be cross-modal aligned with the space of image features. In the method provided in this application, the text encoding part of a multimodal alignment encoder is used as a text encoder for encoding the descriptive text. The multimodal alignment encoder (Vision-Language Model for Representation Alignment, VLM2Vec) can encode cross-modal data (image-text data) into a consistent representation space.

[0029] The method provided in this application introduces structural / style dual projection decoupling within the encoder's representation space, enabling the separate extraction of structural and stylistic features from the descriptive text. This eliminates the need to construct a style prototype library, allowing for the decoupled extraction of structural and stylistic features directly based on the projection head. The text structure projection head and text style projection head can be viewed as lightweight feature extractors, essentially learnable subspace decomposers. The structure and style projection heads typically employ an MLP (Multilayer Perceptron) / linear layer + LN (Layer Normalization) structure. However, it is understood that other subspace decomposer structures can also be used.

[0030] Based on the extracted word-level text structure features (describing the structural features of each word in the text) and word-level text style features (describing the style features of each word in the text), this application sets up a style gating perception module. This style gating perception module outputs word-level style gating weights. Compared with manually determining style words and applying certain weights to style words, this application can adaptively determine the gating weights of each word in the text, applying stronger guidance to keywords and weaker guidance to non-style words, thereby improving the generalization and automation of style image generation.

[0031] In one possible implementation, the style gating perception module can employ multiple weight determination methods to generate style gating weights. Specifically, word-level text structure features and word-level text style features are input into the style-aware gating module to obtain the word-level style gating weights output by the style-aware gating module, including: Multiple initial word-level style weights are determined by using at least one weight determination method, word-level text structure features, and word-level text style features. Multiple initial word-level style weights are fused to obtain word-level style gating weights.

[0032] In this implementation, multiple weighting methods are used to determine the importance of each word in style guidance for its text structure and style features. For example, weighting methods may include: weighting based on occlusion difference, weighting based on cross-attention saliency / gradient sensitivity, and weighting based on syntactic priors. The initial word-level style weights obtained from multiple weighting methods are then fused using a weighted summation method to obtain the word-level style gating weights.

[0033] In some embodiments, the style gating awareness module may further include a sparse layer. After fusing multiple initial word-level style weights to obtain word-level style gating weights, the word-level style gating weights can be further sparsified. Sparsity processing can employ strategies such as Top-K sparsity or hard-concrete sparsity. Through sparsity processing, only the terms that truly carry style semantics can be amplified, suppressing style-irrelevant word noise while maintaining the sparsity, stability, and interpretability of the signal.

[0034] After obtaining the word-level style gating weights, the word-level modulation features are determined based on the word-level style gating weights, and these word-level modulation features are applied to the image generation model in the process of processing text encoding features.

[0035] Image generation models can adopt the architecture of existing "text-to-image" models. In one possible implementation, the image generation model can be a direct text-to-image model, such as a GAN (Generative Adversarial Network) model. In another possible implementation, the image generation model includes a noise prediction model and a denoising module. Text-encoded features are input into the image generation model to obtain the style image output by the model, including: Initial noise is generated by inputting the text encoding features and the noise at the current time step into the noise prediction model to obtain the predicted noise at the next time step output by the noise prediction model. Based on the predicted noise at the next time step, the noise at the next time step is denoised. The denoising result of the last time step is used as the style image, and the noise at the first time step is the initial noise. In each time step, the intermediate word-level features obtained by processing the text encoding features in the noise prediction model are modulated based on the word-level modulation features of the current time step.

[0036] In this implementation, noise prediction is performed at each time step, and denoising is further performed based on the predicted noise, generating the final image after multiple time steps. The noise prediction model can be the U-Net model.

[0037] In some embodiments of this implementation, word-level modulation features are determined based on word-level style gating weights, including: Obtain the control ratio at each time step; The word-level style gating weights at the current time step are processed based on the control ratio at each time step to obtain the word-level modulation features at each time step.

[0038] The control ratio is used to control the importance of style gating weights. In the method provided in this application, the control ratio gradually increases the importance of style gating weights over time. That is, in the technical route of image generation based on denoising processing at multiple time steps, a control ratio can be set for each time step, and the word-level style gating weights can be processed based on the control ratio of each time step to obtain the word-level modulation features of each time step. In this way, during the image generation process of the image generation model, the intermediate signal of the model can be scaled by "word level × time step", thereby strengthening the guidance on keywords and weakening the guidance on non-style words, and smoothly transitioning from "structure priority" to "style refinement" over time, to obtain a fine-grained and numerically stable generation control effect.

[0039] Furthermore, in some embodiments of this implementation, the style gating awareness module outputs word-level style gating weights that include multiple time steps, i.e., time-related word-by-word gating weights. ,so, It adapts and evolves with the sampling time step to achieve dynamic allocation of "early stage with structure bias and later stage with style enhancement".

[0040] The word-level style gating weights at each time step are processed based on the control ratios at each time step to obtain the word-level modulation features at each time step. This is achieved by weighting the word-level style gating weights according to the control ratios, thus increasing the importance of the style gating weights over time, smoothly transitioning from "structure-first" to "style-refined" over time. In some embodiments, the calculation formula for the word-level modulation features can be expressed as follows: Among them, This represents the modulation feature of the i-th word at time step t. This indicates the control ratio at time step t. This represents the style gating weight of the i-th word at time step t.

[0041] In one possible implementation, the generated word-level modulation features can be further optimized. For structural words, a lower upper limit for word-level modulation features is set, while for style words, a higher upper limit is set. The difference between structural words and style words can be determined by the magnitude of their corresponding style gating weights.

[0042] In text-based graph denoising mechanisms based on time-step denoising, noise prediction models often include cross-attention layers and classifier-free guidance (CFG). In one implementation of the method provided in this application, word-level intermediate features obtained by processing text encoding features are modulated using word-level modulation features, including: The attention vectors of each word in the cross-attention layer and the conditional difference terms of each word in the classification-guided layer are modulated based on word-level modulation features.

[0043] In this implementation, modulation based on word-level modulation features involves scaling the K / V vector and CFG guidance intensity of the cross-attention of the noise prediction model by "word-level × time step" based on the word-level modulation features. This scaling is then directly applied to the cross-attention and conditional branches of each layer and each step of the diffusion U-Net, thereby strengthening guidance on keywords and weakening guidance on non-style words. Over time, the approach smoothly transitions from "structure priority" to "style refinement," achieving a fine-grained and numerically stable generation control effect.

[0044] In this implementation, noise prediction of the noise prediction model is performed at each time step and applied synchronously. The conditional intensity of attention and CFG is modulated until the target image is output as a style image. In the whole process, the structure is enhanced in the early stage and the style texture and brushstroke are enhanced in the later stage, so as to achieve stable, controllable and consistent generation results on the continuous adjustable trajectory of "preserving structure ↔ transferring style".

[0045] At each time step Obtain control ratio The style-aware gating module outputs word-level gating weights. The scaling factor for each term at that time step was calculated based on both methods. .Will The cross-attention layer of the diffusion U-Net scales the word-level key / value vectors corresponding to the text conditions word by word and simultaneously applies them to the conditional difference term of the CFG to achieve conditional intensity modulation at the word level × time step, thereby enhancing style presentation while maintaining structure.

[0046] When text encoding features also include sentence-level text encoding features, sentence-level text encoding can also be used as the output of the noise prediction model. During the processing of the noise prediction model, sentence-level text encoding features remain stable as a global condition, while word-level conditions are only adjusted at each time step. Perform scaling modulation.

[0047] The control ratio at each time step can be obtained in several ways, including: The control ratio is determined based on a preset time function or user input data; or... Input the sentence-level text encoding features into the proportional controller to obtain the control ratio output by the proportional controller; or... Based on the signal generated during the style image generation process, a generation control ratio is generated.

[0048] In some embodiments, the preset time function can be a cosine ramp function, that is, the formula for calculating the control ratio at each time step is: ,default .

[0049] In some embodiments, the generation control ratio is based on the signal of the style image generation process, and is dynamically adjusted based on the confidence level, attention entropy or structure preservation index in the style image generation process.

[0050] In some embodiments, a proportional controller can be set up, which takes sentence-level text encoding features as input and outputs a control ratio. In this implementation, the proportional controller is trained as a trainable module together with other trainable modules. The specific training mechanism will be described later.

[0051] It is understood that the implementation of this application is not limited to the various methods of obtaining the control ratios mentioned above, as long as the generated control ratios can gradually increase the importance of the style gating weights as the time step changes.

[0052] In one possible implementation, the method provided in this application may include a reference image in addition to descriptive text as input data. When the input data only includes descriptive text, the initial noise is random noise. However, when the input data includes a reference image, the initial noise is noise obtained by inverting the reference image. Inverting the reference image can be implemented using DDIM (Denoising Diffusion Implicit Models).

[0053] In one possible implementation of the method provided in this application, the image generation model can be a pre-trained image generation model, meaning the image generation model does not participate in the training of other modules (such as encoders, style-aware gating modules, etc.), or only the image generation model is fine-tuned during training. For example, when the image generation model includes a noise prediction model, the cross-attention mechanism of the noise prediction model can be fine-tuned using LoRA. This can effectively reduce the training cost of the method provided in this application and can be quickly deployed under mainstream computing power.

[0054] As can be seen from the foregoing description, in order to achieve higher quality style images generated by the method provided in this application, the obtained text structure features and text style features should have higher decoupling and higher alignment with the image representation space. To achieve this, the training process of the text encoder, text structure projection head, text style projection head, and style-aware gating module in the method provided in this application includes: Acquire sample data, which includes sample text and sample images; The sample text is input into the text encoder to obtain the sample text encoding features output by the text encoder. The sample image is input into the image encoder to obtain the sample image encoding features output by the image encoder. The sample image encoding features include local image encoding features and global image encoding features. The sample text encoding features are input into the text structure projection head and the text style projection head, respectively, and the sample image encoding features are input into the image structure projection head and the image style projection head, respectively, to obtain text structure features, text style features, image structure features and image style features; Based on word-level style gating weights, text structure features and text style features are aggregated to obtain aggregated text structure features and aggregated text style features. Based on image structure features and image style features, the aggregated text structure features and aggregated text style features are aligned with image modalities to obtain aligned text structure features and aligned text style features. Based on the control ratio, the structural features and style features of aligned text are fused to obtain a unified objective; The local image coding features and the global image coding features are aggregated to obtain the aggregated image features; The training loss is determined based on the differences between the unified target and the aggregated image features; The text encoder, image encoder, text structure projector, text style projector, image structure projector, image style projector, and style-aware gating module are updated based on the training loss.

[0055] It is worth noting that when the proportional controller is a trainable module, that is, when the control ratio is based on word-level text encoding features input to the proportional controller and output by the proportional controller, the proportional controller is also updated and trained based on the training loss. That is, the text encoder, image encoder, text structure projector, text style projector, image structure projector, image style projector, style-aware gating module and proportional controller are updated based on the training loss.

[0056] like Figure 2 As shown, in the method provided in this application, during training, the training sample data includes text-image pairs, that is, sample text and sample images matching the sample text. For the text in the training sample data, it is processed according to the processing method for descriptive text described above, while for the sample images, an image encoder is used for encoding to obtain global features. With local features During training, under the condition of no style prototype library, this proposal sets up structured projection heads for text and images separately. With style projector head This yields text structural features and text style features at the word level. Image structure features and image style features on the image side For text structure features and text style features, aggregation is first performed within the modality. This involves aggregating word-level text structure features to obtain aggregated text structure features, and aggregating word-level text style features to obtain aggregated text style features. When aggregating structure and style features, word-level style gating weights are used as aggregation weights. The aggregation process within the text modality can be expressed by the following formula: ; in, To aggregate text structural features, This is for aggregating text style features.

[0057] After obtaining the aggregated text structure features and aggregated text style features, the text features are aligned with the image features across modalities, that is, the text features are aligned with the image features across modalities. , After aligning the spaces closely, then use a controlled ratio to... and To achieve a unified goal through integration It can be expressed by the formula: .

[0058] On the image side, global and local features of the image are aggregated to obtain aggregated image features. The training loss is determined based on the difference between the unified target and the aggregated image features. This single alignment loss can avoid gradient conflicts caused by "spatial / semantic double loss" and can introduce PCGrad or uncertainty weighting at the optimizer end to further improve stability, so that the "structure-style" tradeoff can be continuously and smoothly learned in one objective.

[0059] In this implementation, a style-structure unity is adopted as the guiding principle, with a single target vector. The method replaces the traditional "spatial / semantic" dual loss with a unified alignment loss, which leads to gradient conflicts and frequent manual tuning. In the past, parallel dual loss methods were prone to conflicts. The method provided in this application uses "one goal" to unify the trade-offs, avoiding gradient conflicts and repeated parameter tuning, significantly improving training stability and convergence efficiency, while maintaining an interpretable trade-off between structure and style.

[0060] In the method provided in this application, the single alignment loss is the main loss, serving as the primary supervision signal for training. Furthermore, in one possible implementation, the training loss includes, in addition to the single alignment loss... In addition, it can also include decoupling constraint loss, which is responsible for ensuring that the structure and style projection heads learn two different subspaces, preventing the structure and style projection heads from outputting the same subspace features, thus failing to achieve decoupling between structure features and style features.

[0061] In this implementation, for sample images Image style features and image structural features Decoupling constraint loss is obtained by applying decorrelation and / or orthogonality constraints to both. Under orthogonality constraints, the decoupling constraint loss... During the decorrelation constraint process, for the image style feature matrix composed of image style features from multiple sample images within the training batch and the image structure feature matrix composed of image structure features from multiple sample images within the training batch, the off-diagonal term of the cross covariance is minimized to decouple the constraint loss. .

[0062] Furthermore, in another implementation, the training loss can include consistency constraint loss in addition to alignment loss and decoupling constraint loss. This consistency constraint loss is used to constrain the consistency between structural features and style features. Specifically, for sample images... Constructing a geometry-enhanced view: And enhanced appearance view: Geometric enhancement views can be constructed using radial / deformation / cropping / inversion techniques, while appearance enhancement views can be constructed using color dithering / contrast / texture perturbation / light noise reduction techniques. Sample images are processed by an image encoder and projected by a projection head to obtain image structural features: and image style features: Consistency constraint loss includes structural consistency loss, which stipulates that even if the appearance of an image changes, the structure should remain consistent. The consistency constraint also includes style consistency loss, which means that even if the image geometry changes, the style should remain consistent. Combining style consistency loss with style consistency loss yields consistency constraint loss. .

[0063] For different training loss terms, with alignment loss as the primary constraint, the different training loss terms are weighted and fused to obtain the final training loss, which can be expressed by the formula: ,in, and These are the weighting coefficients.

[0064] Furthermore, when the image generation model is also involved in the training (as mentioned above, the cross-attention mechanism in the noise prediction model is fine-tuned using LoRA), the training loss can also include a quality evaluation term for the sample style images generated by the image generation model during training.

[0065] In one possible implementation, the training process can be divided into two phases, a first phase and a second phase, with the second phase being optional. In the first phase, the backbone of the text / image encoder is frozen, and only the projection head, style gating module, controller (optional), and cross-attention mechanism part of the noise prediction model are trained. In the second phase, the last Transformer block of the text tower and vision tower of the text / image encoder is unfrozen, fine-tuned with a small learning rate, and the decorrelation metric is monitored to avoid decoupling leakage.

[0066] In this implementation approach, a low-cost configuration of "freezing the backbone + LoRA fine-tuning" is adopted at the implementation level, which can be quickly deployed under mainstream computing power. When it is necessary to further improve image quality and style consistency, it supports small-scale unfreezing of upper-layer parameters for further refinement, balancing performance and cost.

[0067] The style image generation apparatus provided in this application is described below. The style image generation apparatus described below can be referred to in correspondence with the style image generation method described above. For example... Figure 3 As shown, the style image generation apparatus provided in this application includes: The input data acquisition module 310 is used to acquire input data, which includes at least descriptive text. The text encoding module 320 is used to input descriptive text into the text encoder and obtain the text encoding features output by the text encoder. The text encoding features include at least word-level encoding features. The dual projection module 330 is used to input text encoding features into the text structure projection head and the text style projection head respectively to obtain word-level text structure features and word-level text style features; The style gating weight determination module 340 is used to input word-level text structure features and word-level text style features into the style-aware gating module to obtain the word-level style gating weights output by the style-aware gating module. The modulation feature determination module 350 is used to determine word-level modulation features based on word-level style gating weights; The image generation module 360 ​​is used to input text encoding features into the image generation model and obtain the style image output by the image generation model. The word-level intermediate features obtained by processing the text encoding features in the image generation model are modulated using word-level modulation features.

[0068] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communications bus 440. The processor 410 can call logical instructions in the memory 430 to execute a style image generation method, which includes: acquiring input data, the input data including at least descriptive text; inputting the descriptive text to a text encoder to obtain text encoding features output by the text encoder, the text encoding features including at least word-level encoding features; inputting the text encoding features to a text structure projection head and a text style projection head respectively to obtain word-level text structure features and word-level text style features; inputting the word-level text structure features and word-level text style features to a style-aware gating module to obtain word-level style gating weights output by the style-aware gating module; determining word-level modulation features based on the word-level style gating weights; inputting the text encoding features to an image generation model to obtain a style image output by the image generation model, wherein the word-level intermediate features obtained by processing the text encoding features in the image generation model are modulated using word-level modulation features.

[0069] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0070] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the style image generation method provided by the above methods. The method includes: acquiring input data, the input data including at least descriptive text; inputting the descriptive text to a text encoder to obtain text encoding features output by the text encoder, the text encoding features including at least word-level encoding features; inputting the text encoding features to a text structure projection head and a text style projection head respectively to obtain word-level text structure features and word-level text style features; inputting the word-level text structure features and word-level text style features to a style-aware gating module to obtain word-level style gating weights output by the style-aware gating module; determining word-level modulation features based on the word-level style gating weights; inputting the text encoding features to an image generation model to obtain a style image output by the image generation model, wherein the word-level intermediate features obtained by processing the text encoding features in the image generation model are modulated using word-level modulation features.

[0071] Furthermore, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the style image generation method provided by the above methods. The method includes: acquiring input data, the input data including at least descriptive text; inputting the descriptive text to a text encoder to obtain text encoding features output by the text encoder, the text encoding features including at least word-level encoding features; inputting the text encoding features to a text structure projection head and a text style projection head respectively to obtain word-level text structure features and word-level text style features; inputting the word-level text structure features and word-level text style features to a style-aware gating module to obtain word-level style gating weights output by the style-aware gating module; determining word-level modulation features based on the word-level style gating weights; inputting the text encoding features to an image generation model to obtain a style image output by the image generation model, wherein the word-level intermediate features obtained by processing the text encoding features in the image generation model are modulated using word-level modulation features.

[0072] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0073] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating style images, characterized in that, include: Obtain input data, which includes at least descriptive text; The descriptive text is input into a text encoder to obtain the text encoding features output by the text encoder, wherein the text encoding features include at least word-level encoding features; The text encoding features are input into the text structure projection head and the text style projection head respectively to obtain word-level text structure features and word-level text style features; The word-level text structure features and word-level text style features are input into the style-aware gating module to obtain the word-level style gating weights output by the style-aware gating module. Based on the word-level style gating weights, word-level modulation features are determined; The text encoding features are input into an image generation model to obtain a style image output by the image generation model. The word-level intermediate features obtained by processing the text encoding features in the image generation model are modulated using the word-level modulation features.

2. The style image generation method according to claim 1, characterized in that, The image generation model includes a noise prediction model and a denoising module. The text encoding features are input into the image generation model to obtain the style image output by the image generation model, including: Initial noise is generated by inputting the text encoding features and the noise at the current time step into the noise prediction model to obtain the predicted noise at the next time step output by the noise prediction model. Based on the predicted noise at the next time step, the noise at the next time step is denoised. The denoising result at the last time step is used as the style image, and the noise at the first time step is the initial noise. In each time step, the intermediate word-level features obtained by processing the text encoding features in the noise prediction model are modulated based on the word-level modulation features of the current time step. The step of determining word-level modulation features based on the word-level style gating weights includes: Obtain the control ratio at each time step; The word-level style gating weights at the current time step are processed based on the control ratios at each time step to obtain the word-level modulation features at each time step.

3. The style image generation method according to claim 2, characterized in that, The noise prediction model includes a cross-attention layer and classification-free guidance; The word-level intermediate features obtained by processing the text encoding features are modulated using the word-level modulation features, including: The attention vectors of each word in the cross-attention layer and the conditional difference terms of each word in the classification-guided layer are modulated based on the word-level modulation features.

4. The style image generation method according to claim 2, characterized in that, The text encoding features also include sentence-level text encoding features; the acquisition of the control ratio for each time step includes: The control ratio is determined based on a preset time function or user input data; or, The sentence-level text encoding features are input to the proportional controller to obtain the control ratio output by the proportional controller; or, The control ratio is generated based on the generation process signal of the style image.

5. The style image generation method according to claim 1, characterized in that, The step of inputting the word-level text structure features and the word-level text style features into the style-aware gating module to obtain the word-level style gating weights output by the style-aware gating module includes: Multiple initial word-level style weights are determined using at least one weight determination method, the word-level text structure features, and the word-level text style features. The multiple initial word-level style weights are fused to obtain the word-level style gating weights; The word-level style gating weights are sparsed.

6. The style image generation method according to claim 1, characterized in that, The training process for the text encoder, the text structure projector, the text style projector, and the style-aware gating module includes: Acquire sample data, which includes sample text and sample images; The sample text is input into the text encoder to obtain the sample text encoding features output by the text encoder. The sample image is input into the image encoder to obtain the sample image encoding features output by the image encoder. The sample image encoding features include local image encoding features and global image encoding features. The sample text encoding features are input into the text structure projection head and the text style projection head, respectively, and the sample image encoding features are input into the image structure projection head and the image style projection head, respectively, to obtain text structure features, text style features, image structure features and image style features; Based on the word-level style gating weights, the text structure features and the text style features are aggregated to obtain aggregated text structure features and aggregated text style features. Based on the image structure features and the image style features, the aggregated text structure features and the aggregated text style features are aligned with the image modality to obtain aligned text structure features and aligned text style features. Based on the control ratio, the aligned text structural features and the aligned text style features are fused to obtain a unified target; The local image coding features and the global image coding features are aggregated to obtain aggregated image features; The training loss is determined based on the difference between the unified objective and the aggregated image features; The text encoder, the image encoder, the text structure projector, the text style projector, the image structure projector, the image style projector, and the style-aware gating module are updated based on the training loss.

7. A style image generation apparatus, characterized in that, include: An input data acquisition module is used to acquire input data, which includes at least descriptive text; A text encoding module is used to input the description text into a text encoder to obtain text encoding features output by the text encoder, wherein the text encoding features include at least word-level encoding features; The dual projection module is used to input the text encoding features into the text structure projection head and the text style projection head respectively to obtain word-level text structure features and word-level text style features; The style gating weight determination module is used to input the word-level text structure features and the word-level text style features into the style-aware gating module to obtain the word-level style gating weights output by the style-aware gating module. The modulation feature determination module is used to determine word-level modulation features based on the word-level style gating weights; An image generation module is used to input the text encoding features into an image generation model and obtain a style image output by the image generation model, wherein the word-level intermediate features obtained by processing the text encoding features in the image generation model are modulated using the word-level modulation features.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the style image generation method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the style image generation method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the style image generation method as described in any one of claims 1 to 6.

Images