A method, system, and product for controllable human image generation based on multimodal control.

Abstract

This invention discloses a controllable person image generation method, system, and product based on multimodal control. First, multimodal input data is acquired, including text data, pose image data, and appearance image data. Then, the modal input data is converted into feature embeddings. Through a cross-modal attention mechanism, the conditional information of these different modalities is combined to generate unified pose conditional embeddings and unified appearance conditional embeddings. Next, the pose and appearance conditional embeddings are input into a transformer-based latent diffusion model for inference to obtain spatial features. Finally, the obtained spatial features are used to generate the final image through a VAE decoder. This invention combines multimodal input and a latent diffusion model, achieving precise control over the generated image under complex conditions. It can generate high-quality person images that meet user needs, greatly expanding the applicability of person image generation technology in fields such as virtual try-on, digital content creation, and entertainment.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and computer graphics, and to a method, system and product for generating controllable human images, specifically a method, system and product for generating controllable human images based on multimodal control. Background Technology

[0002] With the development of generative artificial intelligence technology, significant progress has been made in human image generation algorithms. The goal of human image generation is to generate high-quality human images that conform to given pose or appearance information.

[0003] Patents such as CN202311405005.8, "Digital Character Generation Method, Apparatus, Device, and Storage Medium with Multimodal Input," CN202410601732.X, "Multimodal Character Image Generation Method Based on Shared Attention," and CN202311368128.9, "Image Generation Method, Apparatus, Device, and Storage Medium," all relate to multimodal image generation technology. However, existing technologies mostly rely on fixed input-output paradigms when generating multimodal character images, such as limiting inputs to pose maps and reference images. Therefore, they lack flexible control over the combination of multimodal inputs (e.g., text, poses, appearance images, etc.) (i.e., supporting arbitrary combinations of inputs under arbitrary modal conditions). Existing methods exhibit certain limitations when facing diverse generation tasks and are difficult to adapt to the complex requirements of practical applications. Summary of the Invention

[0004] This invention aims to address the problem in existing methods for generating high-quality portrait images using multiple modal inputs. It proposes a controllable portrait image generation method, system, and product based on multimodal control.

[0005] The technical solution adopted by the method of the present invention is: a controllable human image generation method based on multimodal control, comprising the following steps:

[0006] Step 1: Acquire multimodal input data, including text data, pose image data, and appearance image data; where the text data includes pose text and appearance text;

[0007] Step 2: Use a text encoder to convert pose text data into pose text features, and divide pose image data into blocks to directly use pose features. Pose image features and pose text features interact through a multimodal input module to generate a unified pose conditional embedding. Use an image encoder to convert appearance image data into feature embeddings and then divide it into blocks to use appearance conditional embeddings.

[0008] Step 3: Input the pose condition embedding and appearance condition embedding into the transformer-based latent diffusion model for inference to predict the noise added in the forward step;

[0009] Step 4: Iterate through Step 3, predict the noise at each step, perform gradual denoising, and generate the final image.

[0010] Preferably, the text encoder in step 2 includes a text tokenization module, an embedding and position encoding module, several Transformer stack encoding modules arranged in series, and an EOS feature extraction module.

[0011] The text tokenization module includes segmenting the input text using lowercase Byte Pair Encoding;

[0012] The embedding and position encoding module includes a Token Embedding layer and a learnable position embedding layer. The Token Embedding layer is used to map each TokenID to an N-dimensional real-valued vector. The learnable position embedding layer adds a corresponding N-dimensional position vector to each position in the sequence, where N is a preset value.

[0013] The Transformer stack encoding module includes a layer normalization layer, a multi-head self-attention layer, a residual connection layer, and a feedforward neural network layer. The layer normalization layer and the multi-head self-attention layer are connected sequentially, and the output of the layer normalization layer is input to the feedforward neural network layer after being normalized by the residual connection layer. The input and output of the feedforward neural network layer are then output after being normalized by the residual connection layer.

[0014] The EOS feature extraction module is used to select the hidden vector corresponding to the EOS position at the end of the sequence in the output of the last layer of the Transformer, and output the vector after performing layer normalization processing.

[0015] Preferably, the feedforward neural network consists of a linear transformation layer, a quick_gelu activation layer, and a re-linear transformation layer.

[0016] Preferably, the image encoder in step 2 includes an input preprocessing module, an initial convolution and pooling module, a residual stacking module, a downsampling module, and a global average pooling module connected in sequence.

[0017] The input preprocessing module is used to normalize the original RGB image and perform random cropping and flipping operations.

[0018] The initial convolution and pooling module includes a 7×7 convolutional layer and a 3×3 max pooling layer, which downsamples the feature map to 1 / 4 of its original size and initially extracts low-level texture and edge information.

[0019] The residual stacking module is composed of several residual blocks connected in series. Its three convolutional layers are 1×1 dimensionality reduction, 3×3 main convolution, and 1×1 dimensionality increase, respectively. BatchNorm and ReLU are set after each intermediate layer. Cross-layer direct connection also uses 1×1 convolution to align channels and add them together.

[0020] The downsampling module halves the spatial size through the first layer of convolution and simultaneously doubles the number of channels to enhance deep semantic expression.

[0021] The global average pooling module is used to compress the spatial dimension of each channel of the feature map into a scalar using global average pooling, so as to obtain a global feature vector with a length equal to the number of channels.

[0022] Preferably, in step 2, the multimodal input module connects the pose text features and pose image features, then passes them through a linear projection layer, adds position encoding, and passes them through several transformer base blocks to obtain a unified pose condition embedding.

[0023] The transformer base block includes a layer normalization layer, a multi-head self-attention layer, a residual connection layer, and a feedforward neural network layer. The layer normalization layer and the multi-head self-attention layer are connected sequentially, and the output of the normalization layer is input to the feedforward neural network layer after being normalized by the residual connection layer. The input and output of the feedforward neural network layer are output after being normalized by the residual connection layer.

[0024] Preferably, the latent diffusion model based on transformer in step 3 includes a transformer base block, an appearance control block, and a pose control block, with the control block receiving appearance condition embedding and pose condition embedding as inputs, respectively.

[0025] Preferably, in step 3, a cross-attention mechanism is used to inject appearance and pose control signals during the generation process:

[0026]

[0027]

[0028] in, , and These are the weight matrices for the query, key, and value; key Sum Embedded by unified conditions It comes from changes, and the query From the basic blocks Dimensional embedding Derived from;

[0029] During the inference phase, a technique that does not require an accumulation classifier is used to predict each time step. noise :

[0030]

[0031] in It is a noise predictor. and It is a unified conditional embedding. and These are the corresponding guidance standards; For the potential features at step t, This refers to the conditional embedding features when the modality is set to empty.

[0032] Preferably, the latent diffusion model based on transformer is a pre-trained model; the training process specifically includes the following steps:

[0033] (1) Collect pose images, pose text, appearance images and appearance text data in real-world scenarios, construct a multimodal dataset, and divide it into training set and validation set according to a certain ratio;

[0034] (2) Text encoder, pose estimation network and image encoder are used to convert text data, pose image data and appearance image data into feature embeddings. Through cross-modal attention mechanism, the conditional information of these different modalities is combined to generate unified pose conditional embeddings and unified appearance conditional embeddings respectively.

[0035] (3) Insert the uniform pose embedding and uniform appearance embedding into the latent diffusion model for inference;

[0036] (4) The obtained spatial features are used to generate the final image through the VAE decoder. The texture enhancement decoding strategy is used to calculate the edge-aware L2 reconstruction loss, backpropagation is performed and the network parameters are optimized.

[0037] The loss function used is:

[0038]

[0039] in and These are the width and height of the image, respectively. and These are real images and generated images. Pixel value at the location; Is it a real image in position? The Canny edge map detected at the location, It is a hyperparameter used to balance edge and global information;

[0040] (5) After training for a preset number of rounds, a well-trained potential diffusion model is obtained.

[0041] The technical solution adopted by the system of this invention is: a controllable human image generation system based on multimodal control, comprising:

[0042] One or more processors;

[0043] A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the controllable human image generation method based on multimodal control.

[0044] The technical solution adopted by the product of the present invention is: a controllable human image generation product based on multimodal control, including computer program instructions, which, when the computer program instructions are run on a computer, cause the computer to execute the controllable human image generation method based on multimodal control.

[0045] Compared with existing technologies, it has the following beneficial effects:

[0046] (1) The present invention proposes to use information from multiple modalities such as text description, pose image and appearance image as conditional input, which provides richer information and makes the generated image more consistent with the input information in terms of pose, appearance and details.

[0047] (2) This invention introduces a transformer-based controllable latent diffusion model, which, combined with the advantages of diffusion models, enables fine-grained control of the generation process. During the generation process, pose control and appearance control respectively adjust the influence of pose and appearance conditions on image generation. Through a cross-modal attention mechanism, the generation process can simultaneously handle multiple input conditions to generate images that meet the specified pose and appearance requirements. This reduces computational complexity, improves generation efficiency, and reduces the consumption of computational resources.

[0048] (3) This invention proposes a texture enhancement decoding strategy, which optimizes the VAE decoder by using edge-aware L2 reconstruction loss, thereby improving image details and texture quality and avoiding image distortion caused by decoder uncertainty. Attached Figure Description

[0049] The technical solutions of the present invention will be further illustrated below using embodiments and specific implementation methods. In addition, some accompanying drawings are used in the description of the technical solutions. Those skilled in the art can obtain other drawings and the intent of the present invention from these drawings without any creative effort.

[0050] Figure 1 This is a schematic diagram illustrating the overall framework of the method according to an embodiment of the present invention.

[0051] Figure 2 This is a diagram of the text encoder network structure according to an embodiment of the present invention;

[0052] Figure 3 This is a diagram of the image encoder network structure according to an embodiment of the present invention;

[0053] Figure 4 This is a network structure diagram of the multimodal input module according to an embodiment of the present invention;

[0054] Figure 5 This is a diagram of the latent diffusion model network structure based on transformer, according to an embodiment of the present invention.

[0055] Figure 6 This is a flowchart illustrating the training process of a potential diffusion model based on transformers according to an embodiment of the present invention. Detailed Implementation

[0056] To help those skilled in the art better understand and implement the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described are only for illustrating the principles and applications of the present invention and are not intended to limit the present invention.

[0057] In fields such as virtual try-on, digital content creation, and entertainment, diverse and uncertain input modalities often arise. To address these technical challenges, please see... Figure 1 This embodiment provides a controllable human image generation method based on multimodal control, including the following steps:

[0058] Step 1: Acquire multimodal input data, including text data, pose image data, and appearance image data; where the text data includes pose text and appearance text;

[0059] Step 2: Use a text encoder, pose estimation network, and image encoder to convert text data, pose image data, and appearance image data into feature embeddings. Through a cross-modal attention mechanism, combine the conditional information of these different modalities to generate unified pose conditional embeddings and unified appearance conditional embeddings respectively.

[0060] In one implementation, please see Figure 2The text encoder includes a text tokenization module, an embedding and position encoding module, a Transformer stack encoding module, and an EOS feature extraction module;

[0061] The text tokenization module includes segmenting the input text using lowercase Byte Pair Encoding (BPE), with a vocabulary size of 49408 and a maximum sequence length of 77 tokens;

[0062] The embedding and position encoding module includes: Token Embedding: mapping each token ID to a 768-dimensional real-valued vector; Learnable position embedding: adding a corresponding 768-dimensional position vector to each position in the sequence, with a maximum of 77 positions;

[0063] The Transformer stack encoding module comprises 12 identical Transformer sublayers, each consisting of: layer normalization; multi-head self-attention (12 heads, each with 64 dimensions, using causal masking); residual connections; layer normalization; a feedforward network consisting of a linear transformation 768→3072, quick_gelu activation, and another linear transformation 3072→768; and residual connections; totaling six parts.

[0064] The EOS feature extraction module is used to select the hidden vector corresponding to the EOS position at the end of the sequence in the output of the last layer of the Transformer, and output the vector after performing layer normalization processing.

[0065] In one implementation, please see Figure 3 The image encoder includes an input preprocessing module, an initial convolution and pooling module, a residual stacking module, a downsampling module, and a global average pooling module;

[0066] The input preprocessing module is used to normalize the original RGB image (according to ImageNet mean and standard deviation) and perform data augmentation operations such as random cropping and flipping to improve the robustness and generalization ability of the network.

[0067] The initial convolution and pooling module includes a 7×7 convolutional layer with a stride of 2 and 64 output channels; followed by a 3×3 max pooling layer with a stride of 2, which downsamples the feature map to 1 / 4 of its original size and initially extracts low-level texture and edge information.

[0068] The residual stacking module is composed of several residual blocks connected in series. Its three convolutional layers are 1×1 dimensionality reduction, 3×3 main convolution, and 1×1 dimensionality increase, respectively. Each intermediate layer is followed by BatchNorm and ReLU. Cross-layer direct connections also use 1×1 convolution to align channels and add them together.

[0069] The downsampling module, after several residual blocks, halves the spatial size by changing the stride of the first convolution to 2 or the stride of the 1×1 convolution within the block to 2, and simultaneously doubles the number of channels to enhance deep semantic expression.

[0070] The global average pooling module is used to compress the spatial dimension of each channel of the feature map into a scalar after the last set of residual blocks are output, so as to obtain a global feature vector with a length equal to the number of channels.

[0071] In one implementation, please see Figure 4 The multimodal input module concatenates the pose text embedding and the segmented pose image as its input. After the pose image and pose text features form a sequence input, it passes through a linear projection layer, adds positional encoding, and then passes through three transformer base blocks to obtain a unified pose conditional embedding. The appearance conditional embedding is obtained similarly to the pose embedding, except that the image modal input is a segmented latent code obtained after encoding by the appearance encoder, rather than the original segmented image.

[0072] Another specific implementation is provided here, using DI-NOv2 as the appearance image encoder and CLIP as the text encoder. The 2D pose image is drawn based on the keypoint coordinates of OpenPose. Experiments were conducted at a target resolution of 512 × 512. The appearance image was resized to 256 × 256, and the pose image drawn based on the keypoint coordinates of OpenPose was resized to 128 × 128. The latent image patch size was set to 2 × 2, and the pose image patch size was 8 × 8. During training, we discarded appearance conditions, pose conditions, and all conditions with probabilities of 0.1, 0.1, and 0.1, respectively.

[0073] Step 3: Input the pose condition embedding and appearance condition embedding into the transformer-based latent diffusion model for inference to predict the noise added in the forward step;

[0074] In one implementation, please see Figure 5 The transformer-based latent diffusion model consists of 28 blocks: a base block, an appearance control block, and a pose control block. The appearance and pose control blocks are located in layers [4, 8, 12] and [2, 6, 10], respectively. The difference between the control blocks and the base blocks is that the key and value in the attention mechanism are replaced by conditional features. The control blocks receive unified appearance and pose conditional embeddings processed by the multimodal input module as input.

[0075] In order to inject appearance and attitude control signals during the generation process, a cross-attention mechanism is adopted to combine the conditional embeddings of these different modalities;

[0076] The cross-modal attention mechanism includes feature alignment, attention computation, and information fusion;

[0077] The feature alignment projectes features from different modalities onto the same semantic space through a linear layer to generate a query Q, key K, and value V triplet.

[0078] The attention calculation involves calculating a similarity matrix between Q of mode A and K of mode B, which is then normalized to weights using Softmax.

[0079] The information fusion uses weighted summation of V values ​​for mode B to generate enhanced features for mode A.

[0080] The conditional embedding information of different modalities is used as the key K and value V to perform cross-modal attention calculation.

[0081] The cross-attention mechanism:

[0082]

[0083]

[0084] in, , and These are the weight matrices for the query, key, and value; key Sum Embedded by unified conditions It comes from changes, and the query From the basic blocks Dimensional embedding Derived from;

[0085] During the inference phase, a technique that does not require an accumulation classifier is used to predict each time step. noise :

[0086]

[0087] in It is a noise predictor. and It is a unified conditional embedding. and These are the corresponding guidance standards; For the potential features at step t, This is the conditional embedding feature when the modality is set to empty (text is empty, image is all zero pixels).

[0088] Step 4: Generate the final image from the obtained spatial features using a VAE decoder.

[0089] Please see Figure 6 In one implementation, the transformer-based latent diffusion model is a pre-trained model; the training process specifically includes the following steps:

[0090] (1) Collect pose images, pose text, appearance images and appearance text data in real-world scenarios, construct a multimodal dataset, and divide it into training set and validation set according to a certain ratio;

[0091] (2) Text encoder, pose estimation network and image encoder are used to convert text data, pose image data and appearance image data into feature embeddings. Through cross-modal attention mechanism, the conditional information of these different modalities is combined to generate unified pose conditional embeddings and unified appearance conditional embeddings respectively.

[0092] To reduce the number of control condition terms and facilitate model training and inference, multimodal inputs representing the same control signal are encoded into a unified conditional embedding, which can be represented as:

[0093]

[0094] in This indicates a multimodal input module. and These are conditional embeddings of different modalities and the final unified conditional embedding. The input power set represents all possible modal combinations. We create the input power set by randomly discarding inputs; empty modal inputs are set to zero, such as empty text or images with all-black pixels. Finally, the pose text embedding and the block-processed pose image are concatenated as input to the multimodal input module. Unified multimodal conditional embedding is achieved after self-attention information interaction within the multimodal input module, which consists of multiple transformer base blocks. The appearance conditional embedding is obtained similarly to pose embedding, except that the image modal input is a block-based latent code encoded by the appearance encoder, rather than the original block-based image.

[0095] (3) Insert the uniform pose embedding and uniform appearance embedding into the latent diffusion model for inference;

[0096] (4) The obtained spatial features are used to generate the final image through the VAE decoder. The texture enhancement decoding strategy is used to calculate the edge-aware L2 reconstruction loss, backpropagation is performed and the network parameters are optimized.

[0097] After multiple denoising processes, the predicted spatial features are obtained. The VAE decoder then obtains the final generated image. Due to uncertainties in the VAE decoding and reconstruction process, texture distortion may occur. To mitigate this problem, this embodiment employs a texture enhancement decoding strategy. Specifically, the VAE encoder is fixed, while only the decoder is optimized using an improved edge-aware L2 reconstruction loss function.

[0098] The loss function used is:

[0099]

[0100] in and These are the width and height of the image, respectively. and These are real images and generated images. Pixel value at the location; Is it a real image in position? The Canny edge map detected at the location, It is a hyperparameter used to balance edge and global information;

[0101] (5) After training for 100,000 steps, a well-trained potential diffusion model is obtained.

[0102] In this embodiment, the test set is further input into the model for model testing, and the corresponding evaluation metrics are calculated to obtain the test results, which reflect the overall performance of the model. The performance evaluation of the model relies on three commonly used metrics: FID, LPIPS, and SSIM.

[0103] The invention will be further illustrated by specific experiments below.

[0104] In this experiment, the latent diffusion model based on the transformer consists of 28 Transformer blocks. The appearance and pose control blocks are located at layers [4, 8, 12] and [2, 6, 10], respectively. The appearance multimodal input module contains 5 Transformer base blocks, while the pose multimodal input module contains 3 base blocks. A multi-head attention mechanism is applied to the appearance and pose control blocks, with 16 heads. The experiment was conducted at a target resolution of 512×512. The appearance image was resized to 256×256, and the pose image drawn from keypoint coordinates was resized to 128×128. The latent image block size was set to 2×2, and the pose image block size was 8×8. During training, appearance, pose, and all conditions were discarded with probabilities of 0.1, 0.1, 0.1, and 0.1, respectively. Furthermore, data from different modalities were randomly discarded in the appearance and pose conditions. The Adam optimizer was used with a learning rate of 2×10⁻⁶.-4 The weight decays to 3×10 -2 The model was trained on eight NVIDIA A100 GPUs for 100,000 iterations with a batch size of 616. The decoder underwent 10 epochs of hyperparameter fine-tuning. Set to 0.5. For sampling, the appearance guide scale. and posture guidance scale Set them to 4.0 and 3.0 respectively.

[0105] This experiment compares the proposed unified framework with existing methods for pose-guided human image generation. The performance of different methods is evaluated using the following metrics: FID, LPIPS, and SSIM. FID measures the similarity between two distributions by calculating the Wasserstein-2 distance between feature representations extracted from a pre-trained Inception network. LPIPS calculates the perceptual distance between image pairs by measuring the difference in deep feature activations using a pre-trained neural network. SSIM evaluates image quality by analyzing the degradation of structural information between two images, focusing on brightness, contrast, and structure. Evaluations are performed at a resolution of 256×176. The dataset used is the DeepFashion in-store clothing retrieval benchmark dataset. This dataset is split into training and test sets, containing 101,966 and 8,570 pairs, respectively. Pose and appearance texts are then supplemented from DF-PASS and DeepFashion-MultiModal. The results for FID, LPIPS, and SSIM on this dataset are 7.718, 0.1926, and 0.6928, respectively.

[0106] Compared to traditional portrait image generation techniques, this invention, based on a latent diffusion model and a transformer architecture, can flexibly combine different input conditions such as text, pose, and appearance images to generate high-quality images. This method employs a unified multimodal conditional embedding, mapping different input conditions to a unified conditional space, and uses the transformer architecture to achieve fine-grained control over the generation process. This invention differs from ordinary portrait image generation primarily in the following ways:

[0107] (1) Traditional human image generation algorithms mainly rely on a single modality input, such as text description or pose image. However, in practical applications, multiple input modalities often coexist. In this case, relying solely on one modality for human image generation will greatly reduce the effectiveness, making it difficult for the generated image to perfectly match the input information. To solve this problem, this invention proposes to utilize information from multiple modalities, such as text description, pose image, and appearance image, as conditional input, providing richer information and making the generated image more consistent with the input information in terms of pose, appearance, and details.

[0108] (2) Traditional human image generation algorithms have limited control over the generation process, making it difficult to precisely control various aspects of the generated image. This invention achieves precise control over the generated image through conditional control and texture enhancement decoding strategies, making the generated image more in line with expectations and improving the image quality and realism.

[0109] (3) Traditional human image generation algorithms may face the problem of low computational efficiency during the generation process, and are mainly suitable for simple human image generation tasks. This invention proposes to use a latent diffusion model to generate images in the latent space, thereby reducing computational complexity, improving generation efficiency, and reducing the consumption of computing resources.

[0110] The flexible multimodal input controllable generation method for human images proposed in this invention has significant technical and social implications, and plays an important role in fields such as virtual try-on, digital content creation, and entertainment.

[0111] It should be understood that the embodiments described above are only some, not all, of the embodiments of the present invention. Furthermore, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form feasible technical solutions. Such combinations are not constrained by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0112] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A controllable character image generation method based on multi-modal control, characterized in that, The method comprises the following steps: Step 1: obtaining multi-modal input data, including text data, pose image data and appearance image data; Wherein the text data includes pose text and appearance text; Step 2: converting the pose text data into pose text features using a text encoder, and directly taking the pose image data as pose features after blocking, and generating unified pose condition embedding through self-attention information interaction of the pose image features and the pose text features through a multi-modal input module; Converting the appearance image data into feature embedding using an image encoder and then blocking as appearance condition embedding; The multi-modal input module connects the pose text features and the pose image features, then passes through a linear projection layer, adds position encoding, and obtains unified pose condition embedding through a plurality of transformer basic blocks; The transformer basic block comprises a layer normalization layer, a multi-head self-attention layer, a residual connection layer and a feedforward neural network layer; the layer normalization layer and the multi-head self-attention layer are sequentially connected, and the output is input into the feedforward neural network layer after being normalized through the residual connection layer; The input and output of the feedforward neural network layer are output after being normalized through the residual connection layer; Step 3: inputting the pose condition embedding and the appearance condition embedding into a transformer-based latent diffusion model to infer and predict the noise added in the forward step; The transformer-based latent diffusion model comprises a transformer basic block, an appearance control block and a pose control block, and the appearance control block and the pose control block respectively receive the appearance condition embedding and the pose condition embedding as input; In order to inject appearance and pose control signals in the generation process, a cross-attention mechanism is adopted: wherein, , and are weight matrices for queries, keys and values, respectively; keys and values are derived from uniform condition embeddings , while queries are derived from dimensional intermediate embeddings of base blocks ; In the inference phase, a technique that does not require cumulative classifiers is employed to predict each time of noise : wherein is a noise predictor, and is a unified conditional embedding, and are respective guiding scales; is a latent feature for the t-th step, is a conditional embedding feature when the modality is empty. Step 4: iteratively performing step 3 to predict the noise of each step and gradually denoise to generate a final image.

2. The controllable character image generation method based on multi-modal control according to claim 1, characterized in that: The text encoder in step 2 comprises a text tokenization module, an embedding and position encoding module, a plurality of serially connected transformer stack encoding modules and an EOS feature extraction module; The text tokenization module comprises using Byte Pair Encoding to tokenize the input text; The embedding and position encoding module comprises a Token Embedding layer and a learnable position embedding layer, the Token Embedding layer is used to map each Token ID into an N-dimensional real value vector, and the learnable position embedding layer is used to add a corresponding N-dimensional position vector to each position in the sequence, wherein N is a preset value; The transformer stack encoding module comprises a layer normalization layer, a multi-head self-attention layer, a residual connection layer and a feedforward neural network layer; the layer normalization layer and the multi-head self-attention layer are sequentially connected, and the output is input into the feedforward neural network layer after being normalized through the residual connection layer; the input and output of the feedforward neural network layer are output after being normalized through the residual connection layer; The EOS feature extraction module is configured to select a hidden vector corresponding to an EOS position at the end of a sequence from an output of a last layer of a Transformer, and output the vector after performing layer normalization processing.

3. The controllable character image generation method based on multi-modal control according to claim 2, characterized in that: The feedforward neural network is composed of a linear transformation layer, a quick_gelu activation layer, and a re-linear transformation layer.

4. The controllable character image generation method based on multi-modal control according to claim 1, characterized in that: The image encoder in step 2 comprises sequentially connected input preprocessing modules, an initial convolution and pooling module, a residual stacking module, a down-sampling module, and a global average pooling module. The input preprocessing module is configured to perform normalization, random cropping, and flipping operations on an original RGB image. The initial convolution and pooling module comprises a 7*7 convolution layer and a 3*3 max-pooling layer, which down-samples a feature map to 1 / 4 of the original size, and preliminarily extracts low-level texture and edge information. The residual stacking module is composed of a plurality of residual blocks connected in series, and the three layers of convolution are 1*1 dimension reduction, 3*3 main convolution, and 1*1 dimension increase, respectively. BatchNorm and ReLU are arranged after the middle layer. A cross-layer direct connection also uses a 1*1 convolution to align the channels and add them. The down-sampling module realizes a halving of the spatial size through a first layer of convolution, and simultaneously expands the channel number to twice the original number, so as to enhance the deep semantic expression. The global average pooling module is configured to use global average pooling to compress the spatial dimension of each channel of the feature map into a scalar, to obtain a global feature vector with a length equal to the number of channels.

5. The controllable character image generation method based on multi-modal control according to any one of claims 1-4, characterized in that: The transformer-based latent diffusion model is a trained model; the training process specifically includes the following steps: (1) Collect pose images, pose texts, appearance images, and appearance text data in real scenes, construct a multi-modal data set, and divide it into a training set and a validation set according to a certain proportion; (2) Use a text encoder, a pose estimation network, and an image encoder to convert text data, pose image data, and appearance image data into feature embeddings, combine the conditional information of these different modalities through a cross-modal attention mechanism, and generate unified pose condition embeddings and unified appearance condition embeddings, respectively; (3) Insert the unified pose embedding and the unified appearance embedding into the latent diffusion model for inference; (4) Generate a final image from the obtained spatial features through a VAE decoder, use a texture enhancement decoding strategy, calculate a perceptual L2 reconstruction loss, and back-propagate and optimize network parameters; The loss function used is: in and These are the width and height of the image, respectively. and These are real images and generated images. Pixel value at the location; Is it a real image in position? The Canny edge map detected at the location, It is a hyperparameter used to balance edge and global information; (5) After training for a preset number of rounds, a trained latent diffusion model is obtained.

6. A controllable character image generation system based on multi-modal control, characterized by, It includes: One or more processors; A storage device for storing one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the controllable character image generation method based on multi-modal control as claimed in any one of claims 1 to 5.

7. A controllable character image generation product based on multi-modal control, comprising computer program instructions, characterized in that: When the computer program instructions run on the computer, the computer executes the controllable character image generation method based on multi-modal control as claimed in any one of claims 1 to 5.

Images