Training method of image generation model, image generation method and device, equipment, medium and product

Abstract

The invention provides a training method of an image generation model, an image generation method and device, equipment, a medium and a product, the training method comprises performing at least one round of training on the image generation model, each round of training comprising: obtaining training sample data, the training sample data comprising a plurality of sample images; generating a prediction image corresponding to each sample image by using the image generation model; and training the image generation model based on each sample image and the corresponding prediction image, and in the prediction image generation process, performing routing decision on each sample image in the plurality of sample images by using a router of the expert hybrid model, selecting a target expert model corresponding to each sample image from a plurality of expert models of the expert hybrid model; and utilizing the target expert model corresponding to each sample image to generate a prediction image corresponding to the sample image.

Description

Technical Field

[0001] This disclosure relates to the field of image generation technology. More specifically, this disclosure relates to a training method for an image generation model, an image generation method and apparatus, device, medium, and product. Background Technology

[0002] Mixture of Experts (MoE) is a modeling technique that employs a multi-expert collaborative architecture. It can select a small number of experts to participate in computation through a router, significantly improving the ratio of model capacity to computational efficiency. It has been widely used in large model architectures.

[0003] Taking the Large Language Model (LLM) domain as an example, MoE can achieve dynamic computation allocation through expert-choice routing strategies, such as using batch level choice in LLM to globally optimize resource allocation.

[0004] However, visual tasks differ fundamentally from language tasks. Visual data has a stronger spatial structure dependency and higher requirements for sample independence. Therefore, when directly transferring existing MoE routing strategies to visual tasks, cross-sample interference can easily lead to model instability and affect the reliability of generated images. Summary of the Invention

[0005] The embodiments of this disclosure provide a training method for an image generation model, an image generation method and apparatus, device, medium and product, to at least solve or alleviate the above-mentioned problems.

[0006] According to a first aspect of the present disclosure, a method for training an image generation model is provided. The image generation model includes an expert fusion model. The training method includes: acquiring training sample data, wherein the training sample data includes multiple sample images; using the router of the expert fusion model, performing routing decisions for each of the multiple sample images, and selecting a target expert model corresponding to each sample image from the multiple expert models of the expert fusion model; using the target expert model corresponding to each sample image, generating a predicted image corresponding to that sample image; and training the image generation model based on each sample image and the corresponding predicted image.

[0007] In embodiments of this disclosure, for each sample image, the router makes routing decisions in the following manner: based on the image features of the sample image, the routing weight of each of the plurality of expert models for the sample image is determined, wherein the routing weight represents the probability that the expert model corresponds to the sample image; based on the routing weight, the target expert model corresponding to the sample image is determined.

[0008] In embodiments of this disclosure, determining the target expert model corresponding to the sample image based on the routing weights includes: modulating the routing weights of the plurality of expert models for the sample image according to the current training time step to obtain modulated weights; and determining the target expert model corresponding to the sample image based on the modulated weights.

[0009] In embodiments of this disclosure, the image generation model further includes a first conditional modulation module, which is used to generate a time-aware vector based on the current training time step. The step of modulating the routing weights of the plurality of expert models for the sample image based on the current training time step to obtain modulated weights includes: obtaining the time-aware vector from the first conditional modulation module; and modulating the routing weights of the plurality of expert models for the sample image based on the time-aware vector to obtain the modulated weights.

[0010] In embodiments of this disclosure, the training sample data further includes sample text corresponding to each sample image, each sample text describing the content of the corresponding sample image. The first conditional modulation module is further configured to generate a modulation vector based on the current training time step and the sample text. The step of modulating the routing weights of the multiple expert models for the sample image based on the time-aware vector to obtain the modulated weights includes: obtaining the modulation vector from the first conditional modulation module; fusing the time-aware vector and the modulation vector to obtain a fused vector; and modulating the routing weights of the multiple expert models for the sample image based on the fused vector to obtain the modulated weights.

[0011] In embodiments of this disclosure, the image generation model further includes a second conditional modulation module and an attention layer, wherein the output of the second conditional modulation module is input to the attention layer, and the second conditional modulation module is the same as the first conditional modulation module.

[0012] In embodiments of this disclosure, the training method includes multiple training rounds, wherein the multiple expert models are obtained by dividing a preset expert network according to the current training round to obtain the multiple expert models.

[0013] In embodiments of this disclosure, the step of dividing a preset expert network according to the current training epoch to obtain the plurality of expert models includes: determining a division ratio for the expert network according to the current training epoch, wherein the division ratio is negatively correlated with the number of expert models to be divided; and dividing the output dimension of the previous layer and the input dimension of the next layer in adjacent network layers of the expert network based on the division ratio and the number of expert models to be divided to obtain the plurality of expert models.

[0014] In embodiments of this disclosure, determining the partitioning ratio for the expert network based on the current training epoch includes: determining a first ratio as the partitioning ratio in response to the current training epoch being in a first training phase; and determining a second ratio as the partitioning ratio in response to the current training epoch being in a second training phase, wherein the first training phase precedes the second training phase, the first ratio is greater than the second ratio, and the number of expert models partitioned in the first training phase is less than the number of expert models partitioned in the second training phase.

[0015] According to a second aspect of the present disclosure, an image generation method is provided, the image generation method comprising: receiving an image generation instruction; and generating an image based on the image generation instruction using an image generation model, wherein the image generation model is trained according to the image generation model training method described in the embodiments of the present disclosure.

[0016] According to a third aspect of the present disclosure, a training apparatus for an image generation model is provided. The image generation model includes an expert fusion model. The training apparatus includes: an acquisition unit configured to acquire training sample data, wherein the training sample data includes a plurality of sample images; a routing decision unit configured to use a router of the expert fusion model to perform routing decisions for each of the plurality of sample images, and select a target expert model corresponding to each sample image from the plurality of expert models of the expert fusion model; a prediction unit configured to use the target expert model corresponding to each sample image to generate a prediction image corresponding to that sample image; and a training unit configured to train the image generation model based on each sample image and the corresponding prediction image.

[0017] In embodiments of this disclosure, for each sample image, the routing decision unit is configured to make routing decisions using the router in the following manner: based on the image features of the sample image, determine the routing weight of each of the plurality of expert models for the sample image, wherein the routing weight represents the probability that the expert model corresponds to the sample image; and based on the routing weight, determine the target expert model corresponding to the sample image.

[0018] In embodiments of this disclosure, the routing decision unit is further configured to use the router to make routing decisions in the following manner: modulate the routing weights of the plurality of expert models for the sample image according to the current training time step to obtain modulated weights; and determine the target expert model corresponding to the sample image based on the modulated weights.

[0019] In embodiments of this disclosure, the image generation model further includes a first conditional modulation module, which is used to generate a time-aware vector based on the current training time step. The routing decision unit is further configured to use the router to make routing decisions in the following manner: obtaining the time-aware vector from the first conditional modulation module; and modulating the routing weights of the plurality of expert models for the sample image based on the time-aware vector to obtain the modulated weights.

[0020] In embodiments of this disclosure, the training sample data further includes sample text corresponding to each sample image, each sample text describing the content of the corresponding sample image. The first conditional modulation module is further configured to generate a modulation vector based on the current training time step and the sample text. The routing decision unit is further configured to use the router to make routing decisions in the following manner: obtaining the modulation vector from the first conditional modulation module; fusing the time-aware vector and the modulation vector to obtain a fused vector; and modulating the routing weights of the multiple expert models for the sample image based on the fused vector to obtain the modulated weights.

[0021] In embodiments of this disclosure, the image generation model further includes a second conditional modulation module and an attention layer, wherein the output of the second conditional modulation module is input to the attention layer, and the second conditional modulation module is the same as the first conditional modulation module.

[0022] In embodiments of this disclosure, the training device performs multiple training rounds, wherein the training device further includes an expert partitioning unit, the expert partitioning unit being configured to obtain the multiple expert models by partitioning a preset expert network according to the current training round.

[0023] In embodiments of this disclosure, the expert partitioning unit is configured to: determine a partitioning ratio for the expert network based on the current training epoch, wherein the partitioning ratio is negatively correlated with the number of expert models to be partitioned; and partition the output dimension of the previous layer and the input dimension of the next layer in adjacent network layers of the expert network based on the partitioning ratio and the number of expert models to be partitioned, thereby obtaining the plurality of expert models.

[0024] In embodiments of this disclosure, the expert segmentation unit is configured to: determine a first ratio as the segmentation ratio in response to the current training epoch being in a first training phase; and determine a second ratio as the segmentation ratio in response to the current training epoch being in a second training phase, wherein the first training phase precedes the second training phase, the first ratio is greater than the second ratio, and the number of expert models segmented in the first training phase is less than the number of expert models segmented in the second training phase.

[0025] According to a fourth aspect of the present disclosure, an image generation apparatus is provided, the image generation apparatus comprising: a receiving unit configured to receive an image generation instruction; and an image generation unit configured to generate an image based on the image generation instruction and using an image generation model, wherein the image generation model is trained according to a training method for an image generation model according to an exemplary embodiment of the present disclosure.

[0026] According to a fifth aspect of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; and a memory for storing processor-executable instructions, wherein, when executed by the processor, the processor causes the processor to perform a training method for an image generation model or an image generation method according to exemplary embodiments of the present disclosure.

[0027] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform a training method for an image generation model or an image generation method according to exemplary embodiments of the present disclosure.

[0028] According to a sixth aspect of the present disclosure, a computer program product is provided, including computer-executable instructions that, when executed by at least one processor, implement a training method for an image generation model or an image generation method according to exemplary embodiments of the present disclosure.

[0029] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects: According to the training and image generation scheme of the image generation model in the embodiments of this disclosure, an expert hybrid model can be set in the image generation model, and the router of the expert hybrid model can make separate routing decisions for each sample image and select the target expert model corresponding to each sample image. This makes the routing decision specific to a single image, rather than a unified decision for the entire batch. This avoids the problem of cross-sample interference that is easily caused when migrating the existing unified routing selection strategy for the entire batch to visual tasks, ensuring the stability of the model-generated images, avoiding excessive differences in image quality generated in different image generation tasks, and improving the reliability of image generation.

[0030] Furthermore, the training and image generation scheme of the image generation model according to the embodiments of this disclosure can enhance the dynamic perception capability of the expert hybrid model of the diffusion process time step by introducing the routing weight of the time-aware modulation expert model, optimize the expert allocation, and make the expert model more professional. When facing different image generation tasks, similar tasks can be more likely to be assigned to the same expert model, thereby further improving the stability of the image quality of the generated images.

[0031] Furthermore, the training and image generation scheme of the image generation model according to the embodiments of this disclosure can achieve fine-grained division of the expert structure by adopting different expert partitioning strategies at different training stages, achieving optimal performance balance under a fixed total number of parameters, avoiding diminishing returns caused by blindly expanding the number of experts, and matching the number or scale of experts with the requirements of the image generation task. This allows for the use of multiple experts to improve the image generation quality while also taking into account the impact of increasing the number of experts on the image generation speed and efficiency.

[0032] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0034] Figure 1 This is a schematic flowchart illustrating a training method for an image generation model according to an exemplary embodiment of the present disclosure.

[0035] Figure 2 This is a schematic structural diagram illustrating an image generation model according to an exemplary embodiment of the present disclosure.

[0036] Figure 3 This is a schematic structural diagram illustrating each block in an image generation model according to an exemplary embodiment of the present disclosure.

[0037] Figure 4 This is a schematic flowchart illustrating the determination of a target expert model in a training method for an image generation model according to an exemplary embodiment of the present disclosure.

[0038] Figure 5 This is a schematic structural diagram illustrating a conditional modulation module in an image generation model according to an exemplary embodiment of the present disclosure.

[0039] Figure 6 This is a schematic flowchart illustrating an image generation method according to an exemplary embodiment of the present disclosure.

[0040] Figure 7 This is a schematic flowchart illustrating a training apparatus for an image generation model according to an exemplary embodiment of the present disclosure.

[0041] Figure 8 This is a schematic flowchart illustrating an image generation apparatus according to an exemplary embodiment of the present disclosure.

[0042] Figure 9 This is a schematic flowchart illustrating an electronic device according to exemplary embodiments of the present disclosure. Detailed Implementation

[0043] Referring now to the detailed description, examples of which are illustrated in the accompanying drawings. Numerous non-limiting details are set forth in the following detailed description to aid in understanding the subject matter presented herein. However, various alternatives may be used without departing from the scope of the claims, and the subject matter may be practiced without these specific details. For example, the subject matter presented herein can be implemented on many types of electronic devices with digital video capabilities.

[0044] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this disclosure are used to distinguish objects and not to describe any specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in sequences other than those shown in the drawings or described in this disclosure.

[0045] As described above, MoE has been extensively studied in order to improve parameter efficiency while maintaining performance, and because MoE can scale the model size without significantly increasing computational cost, it has been widely used in large model architectures.

[0046] However, most existing MoE mechanisms come from the fields of Natural Language Processing (NLP) / LLM. When directly transferred to image generation models such as Diffusion Model, problems such as unstable training, severe cross-sample interference, and unbalanced expert load will occur.

[0047] Specifically, in LLM inference using MoE, expert routing is uniformly performed for the entire input batch, meaning that all sequences share the same set of activation experts. This scheme has been deployed at scale in LLM, significantly improving parameter efficiency.

[0048] Image generation models typically generate high-quality images through progressive denoising, and their performance is highly dependent on model size and structure design. With breakthroughs achieved by image generation models such as diffusion models in image generation tasks, model size continues to grow, leading to a rapid increase in forward computation and training costs.

[0049] In visual tasks, the spatial structure of image sequences needs to remain independent. However, in a strategy of uniform routing selection across the entire batch, all samples are forced to share routing decisions, which can lead to cross-sample interference problems. This causes features of different images to interfere with each other during noise propagation (e.g., noise patterns of different images are incorrectly associated), reducing the quality of generation.

[0050] Furthermore, the diffusion model needs to handle dynamic noise distribution at different time steps, but the global routing strategy cannot perceive local changes in time steps, leading to inaccurate routing decisions in high-noise stages (such as early diffusion steps), model training oscillations, and instability under diffusion noise.

[0051] Furthermore, in visual tasks, the pixel sequence of a single image needs to be modeled independently, but the uniform routing selection of the entire batch destroys the independence of the sequence structure, resulting in distortion of spatial detail generation and thus destroying the sequence structure.

[0052] In addition to the unified routing strategy for the entire batch mentioned above, in some cases, a token-level choice routing strategy can also be adopted. This approach can be migrated to the diffusion model, where each input token is independently selected by experts to avoid cross-token interference.

[0053] However, in such a scheme, token-level selection over-fines the routing granularity, leading to a decrease in the ability of tokens to cooperate within a sequence, redundancy in the sequence structure, and disruption of the global consistency of the image.

[0054] Furthermore, in such a scheme, routing decisions rely solely on the characteristics of the current token, without integrating time-step information of the diffusion process (such as noise intensity and iteration stages), resulting in inefficient expert allocation at some time steps (such as low-noise stages).

[0055] Furthermore, this approach lacks a fine-grained optimization mechanism for the expert structure, and the number and scale of experts do not match the task requirements, resulting in resource waste and insufficient parameter efficiency. For example, the FID score is 8.73 with a 16-expert configuration, which is inferior to the performance of an 8-expert configuration.

[0056] To address or at least mitigate some of the aforementioned problems, exemplary embodiments of this disclosure provide a method for training an image generation model, an image generation method, a training apparatus for an image generation model, an image generation apparatus, an electronic device, a computer-readable storage medium, and a computer program product. These will now be described in detail with reference to the accompanying drawings.

[0057] According to a first aspect of the embodiments of this disclosure, a method for training an image generation model is provided. This method can perform individual routing decisions for each sample image, selecting a target expert model corresponding to each sample image. This allows routing decisions to be made at the sequence level, rather than a unified decision for the entire batch. This avoids the problem of cross-sample interference that can easily occur when transferring existing batch-wide unified routing selection strategies to visual tasks, ensuring the stability of the images generated by the model, avoiding excessive differences in image quality across different image generation tasks, and improving the reliability of image generation.

[0058] Figure 1 A schematic flow of a training method for an image generation model according to an exemplary embodiment of the present disclosure is shown, wherein the image generation model may include an expert hybrid model (MoE), and the training method includes performing at least one round of training on the image generation model.

[0059] As an example, an image generation model may include multiple blocks, such as... Figure 2 As shown, an image generation model can include blocks 1 to N, which can be identical, and each block can include learnable parameters. As an example, the image generation model can be a diffusion model based on the Transformer framework, where each block can be a Transformer layer.

[0060] Furthermore, as an example, the input data for an image generation model can include input text that guides image generation. This input text can indicate the requirements for generating the image. Users can describe the content or generation needs of the target image through the input text. For example, the input text could be: "A singer is performing energetically on stage, singing into a microphone, wearing a dazzling blue and silver sequined dress against a backdrop of vibrant and abstract pink and purple patterns, adding a dynamic atmosphere to the concert." The image generation model can then generate the corresponding image based on the input text.

[0061] Furthermore, as another example, the input data of an image generation model may not include any information guiding image generation (such as the input text mentioned above). The input data may only include image generation instructions. In this example, the model may randomly generate images, such that the randomly generated images can be used as input data, training data, etc. for other image processing models.

[0062] In embodiments of this disclosure, such as Figure 3 As shown, each block may include an expert hybrid model (MoE), which may include a router and multiple expert models. The router can generate an image based on the current image generation task by selecting a suitable expert model from multiple expert models through an expert selection routing strategy.

[0063] In at least one round of training performed on the image generation model, end-to-end training can be employed, such as... Figure 1 As shown, each round of training may include the following steps: In step S110, training sample data can be obtained.

[0064] In one example, the training sample data may include multiple sample images. In this example, the sample images can train the model's unconditional image generation ability, allowing the model to learn the structure and distribution patterns of image features.

[0065] In another example, in addition to multiple sample images, the training sample data may also include sample text corresponding to each sample image, with each sample text describing the content of the corresponding sample image. In this example, the sample images and sample text can train the model's conditional image generation ability, allowing the model to learn to generate images that conform to the text description under the guidance of textual conditions.

[0066] In step S120, the router of the expert hybrid model can be used to make routing decisions for each of the multiple sample images, and select the target expert model corresponding to each sample image from the multiple expert models of the expert hybrid model.

[0067] In this step, the router can employ a sequence-level choice (SLC) routing mechanism. Specifically, the pixel block sequence (e.g., a 16×16 pixel block sequence) of each sample image in the training input can be divided into an independent batch, and routing can be performed independently for each sample image's pixel block sequence. In other words, for each sample image, a target expert model is determined to generate the prediction image for that sample image. Specific examples of routing decisions will be described in detail below.

[0068] In step S130, a predicted image corresponding to each sample image can be generated using the target expert model corresponding to that sample image.

[0069] In this step, a corresponding predicted image can be generated using the target expert model corresponding to each sample image, which can then be used for subsequent loss calculation.

[0070] In one example, an image generation model (or target expert model) can be instructed to randomly generate a predicted image without input data. The predicted image can be used to calculate the loss with the corresponding sample image, so that the trained model can generate an image that corresponds to the style of the sample image.

[0071] In another example, sample text can be input into an image generation model (or target expert model), which can output a predicted image corresponding to the sample text. The predicted image can be used to calculate the loss with the corresponding sample image, so that the trained model can generate an image that matches the sample text.

[0072] In this way, routing decisions can be made at the sequence level for individual images, rather than uniformly across the entire batch. This avoids cross-sequence interference during model training, which can lead to unstable performance when using the model to generate images for different generation tasks. Because the routing decisions are more targeted, the expert model will gradually specialize during model training, forming roles such as "texture expert" and "edge expert." The quality of images generated using such a hybrid expert model can be more stable. For example, in image generation tasks, the same expert can be stably selected for specific visual features or regions of different images (such as the texture of a face or the edges of a building), improving the reliability of image generation.

[0073] Furthermore, the single-image routing mechanism not only adapts to the sequence independence requirements of the diffusion model compared to the batch-based routing mechanism, but also maintains structural stability during noisy iterations, avoids excessive fragmentation, and reduces the extraction of information from within the image compared to the token-based routing mechanism.

[0074] In step S140, the image generation model can be trained based on each sample image and the corresponding predicted image.

[0075] In this step, the learnable parameters in the model can be updated by backpropagation through the loss function based on each sample image and the corresponding predicted image, until the preset training completion conditions are met.

[0076] The following will describe in detail a specific example of the routing decision-making process using a router in step S120.

[0077] As an example, such as Figure 4 As shown, for each sample image, the router makes routing decisions in the following way: In step S410, the routing weights of each of the multiple expert models for the sample image can be determined based on the image features of the sample image.

[0078] Here, the routing weight can represent the probability that the expert model corresponds to the sample image.

[0079] As an example, the routing weight of each expert model in a plurality of expert models can be determined, for example, by the following equation (1): (1) in, Indicates the first Expert weight vectors for each sequence; Indicates the first The input features of a sequence, such as features extracted from pixel blocks of a single sample image; The parameter matrix of the router can be a learnable weight matrix with dimensions such as d × N, where d represents the dimension of the input vector (e.g., the token embedding vector from the previous layer) and N represents the total number of expert models. This matrix maps the input vector to an N-dimensional log-probability space (logits), with each dimension corresponding to the score of an expert model. The Softmax function ensures that the sum of the weights is 1.

[0080] As an example, such as Figure 3 As shown, the image generation model may further include a first normalization (Root Mean Square Normalization, RMSNorm) layer 301, an attention layer (e.g., a self-attention layer) 303, and a second normalization layer 305. The input features of the router can be the features of the input data of the block where the router is located after passing through the normalization layer and the attention layer. Furthermore, the image generation model may also include multiple conditional modulation modules (e.g., Figure 3 As shown in 302, 304, 306, and 308, the conditional modulation module can introduce external information (such as the sample text and time steps mentioned above) into the image generation process, such as... Figure 3 As shown, conditional modulation modules can be set at each normalization layer, attention layer, and MoE to guide the processing of image features. Here, each conditional modulation module may, for example, include an Mbase network and / or an Mtime network (which will be described in detail below).

[0081] In step S420, the target expert model corresponding to the sample image can be determined based on the routing weight.

[0082] In this step, the target expert model can be selected according to preset rules based on the routing weights of each expert model for the current sample image determined in step S410 above.

[0083] As an example, in step S420, a preset number of expert models with the highest routing weight among multiple expert models can be determined as the target expert model corresponding to the sample image.

[0084] As an example, a Top-K expert model can be selected based on the routing weight. Here, the Top-K expert model can represent a preset number of K expert models before the routing weight, where K can be, for example, but not limited to, 2. According to this rule, only the selected K experts can be activated to process the sequence of the current image, and the weights of the unselected expert models can be set to zero. Furthermore, as an example, the weights of the selected K experts can be normalized so that the sum of the weights of the K experts is 1, resulting in the final fusion weight.

[0085] As another example, the routing weights of multiple expert models can be modulated, and the expert model with the highest modulated routing weight can be determined as the target expert model corresponding to the sample image.

[0086] Specifically, in step S420, the step of determining the target expert model corresponding to the sample image based on the routing weights may include: modulating the routing weights of multiple expert models for the sample image according to the current training time step to obtain modulated weights; and determining the target expert model corresponding to the sample image based on the modulated weights.

[0087] By introducing a time step to modulate the routing weights used to select expert models, MoE's ability to dynamically perceive the time step of the diffusion process can be enhanced, thus optimizing expert allocation during the diffusion phase.

[0088] As an example, routing weights can be modulated by introducing time-aware vectors. Specifically, the image generation model can also include a first conditional modulation module (e.g., Figure 3 The conditional modulation module 306 shown can be used to generate a time-aware vector based on the current training time step.

[0089] In this example, the step of modulating the routing weights of multiple expert models for the sample image according to the current training time step to obtain the modulated weights may include: obtaining a time-aware vector from the first conditional modulation module; and modulating the routing weights of multiple expert models for the sample image based on the time-aware vector to obtain the modulated weights.

[0090] Specifically, the first conditional modulation module can implement time-step-specific modulation, which may include, for example, a newly added independent time modulation network Mtime, which can generate a time-aware vector vt=Mtime(t) with only time step t as input. As an example, the Mtime network can be a multilayer perceptron (MLP).

[0091] After determining the time-aware vector vt, the time-aware vector vt can be injected into the expert selection weights of the MoE. For example, the modulated weights can be represented by the following equation (2): = Sigmoid(vt) (2) in, This represents the modulated weights.

[0092] In the above way, the sensitivity of MoE to the diffusion stage can be enhanced by time awareness. Furthermore, by adopting a time awareness introduction scheme with dedicated time step modulation, the dedicated time step modulation network Mtime only adds a small number of parameters (taking Mtime network as MLP as an example, only 0.05B ​​of parameters are added). Therefore, while improving the time dynamic awareness in the diffusion stage, the total computational cost is also kept controllable.

[0093] Similar to the example above, this example can also select Top-K expert models based on the modulated weights, activating only the selected K experts to process the current image sequence, while the weights of the unselected expert models can be set to zero. As an example, the weights of the selected K experts can also be normalized so that the sum of their weights is 1, resulting in the final fused weights.

[0094] Furthermore, in the above example, during the selection of the Top-K expert model based on routing weights or modulated weights, a fixed K value can be used, or the K value can be dynamically adjusted to achieve sequence-adaptive routing. As an example, different K values ​​can be selected based on different noise stages of the diffusion process. For instance, different numbers of expert models can be selected during the noisy and low-noise stages; in the noisy stage, K could be, for example, 1; and in the low-noise stage, K could be, for example, 3. This approach further adapts the diffusion process, making the selection of expert models sensitive to the diffusion stage, such as more accurately selecting experts suitable for detail generation during the low-noise stage.

[0095] Having identified the target expert model through the examples above, the outputs of the target expert model can be aggregated. For instance, each target expert model can be used to generate the current image separately, and the outputs of each target expert model can be weighted and fused to generate a denoised prediction image corresponding to the current sample image. As an example, weighted fusion can refer to using normalized weights to perform a weighted average of the outputs of the selected expert models to generate the final denoised prediction result.

[0096] Furthermore, as mentioned above, in some embodiments, in addition to multiple sample images, the training sample data may also include sample text corresponding to each sample image, whereby each sample text can be used to describe the content of the corresponding sample image. In such embodiments, the first conditional modulation module may also be used to generate a modulation vector based on the current training time step and the sample text.

[0097] As an example, the steps of modulating the routing weights of multiple expert models for the sample image based on the time-aware vector to obtain the modulated weights may include: obtaining the modulation vector from the first conditional modulation module; fusing the time-aware vector and the modulation vector to obtain the fused vector; and using the fused vector to modulate the routing weights of multiple expert models for the sample image to obtain the modulated weights.

[0098] Specifically, the first conditional modulation module can also implement basic modulation, which may include, for example, a basic modulation network Mbase. This network can generate a modulation vector v = Mbase(t, c) with a time step t and a text condition c as input. Here, the text condition c is the external condition that the model needs to input, such as the text condition corresponding to the currently processed sample image. As an example, the Mbase network can be a multilayer perceptron (MLP).

[0099] After determining the modulation vector v and the time-aware vector vt, the modulation vector v and the time-aware vector vt can be injected into the expert selection weights of the MoE. For example, the modulated weights can be expressed as follows: = Sigmoid(vt, v).

[0100] As an example, the image generation model may also include a second conditional modulation module and an attention layer. The output of the second conditional modulation module can be input to the attention layer, and the second conditional modulation module can be the same as the first conditional modulation module. Figure 3 For example, the second conditional modulation module may include, for instance, [the following modules may be included]. Figure 3 The conditional modulation modules 302 and 304 are shown.

[0101] As an example, Figure 5 This illustrates an example process of the conditional modulation module performing conditional modulation on input features, such as... Figure 5 As shown, for the input features input to a conditional modulation module such as a second conditional modulation module We can first apply the modulation vector v output by the basic modulation network Mbase to it to obtain the features. Then you can analyze the features The features are obtained by applying the time-aware vector vt output by the time-step-specific modulation network Mtime. This feature The features can be modulated by the conditional modulation module. Here, applying the modulation vector v and the time-aware vector vt to the features can employ existing conditional modulation schemes, and the embodiments of this disclosure do not impose any particular limitations on this.

[0102] In this way, the first conditional modulation module for the MoE layer and the second conditional modulation module for the attention layer can share the same conditional modulation network, such as the basic modulation network Mbase. In this way, it is possible to achieve differentiated enhancement of MoE based on sharing the basic modulation with the attention layer, and achieve a performance leap in the diffusion stage with extremely low parameter increments (not exceeding 0.05B).

[0103] However, the embodiments disclosed herein are not limited to this. The first conditional modulation module for the MoE layer and the second conditional modulation module for the Attention layer can also be completely independent modulation networks. They do not need to share the same conditional modulation network. When they are independent, the total number of model parameters remains unchanged, but feature decoupling can be improved. Some experiments show that in the example where the MoE layer and the Attention layer share the conditional modulation network, the quantization metric FID (Fréchet Inception Distance) used to evaluate the performance of the image generation model can be 7.41 in the example with 300k training steps; in the example where the MoE layer and the Attention layer do not share the conditional modulation network, the quantization metric FID can be 8.17 in the example with 300k training steps.

[0104] Furthermore, in exemplary embodiments of this disclosure, the aforementioned routing strategy can be executed for the entire image region of a single image, or for a portion of the image region within a single image. For example, a hybrid level choice of multiple routing strategies can be executed for a single image. For a first preset region, such as the image center region, a sequence-level routing decision as described in the embodiments of this disclosure can be adopted; for a second preset region, such as the image edge region, an existing routing decision can be adopted, such as an existing token-level routing strategy. In this way, a balance between efficiency and accuracy can be achieved.

[0105] The foregoing describes in detail an example process of selecting a target expert model from multiple expert models using a router according to an exemplary embodiment of the present disclosure. In one example, the multiple expert models may be obtained by employing an expert partitioning strategy in an existing MoE. In another example, an exemplary embodiment of the present disclosure also provides an expert partitioning strategy (Expert Granularity) to define the composition of the expert pool adopted by the model.

[0106] In an exemplary embodiment of this disclosure, multiple expert models can be obtained by dividing a preset expert network according to the current training round to obtain multiple expert models.

[0107] Here, the pre-defined expert network may include, but is not limited to, a feed-forward network (FFN). Different expert partitioning strategies can be used to partition the pre-defined expert network in different training epochs of the model, resulting in the composition of multiple expert models used in the current training epoch. For example, the number of expert models partitioned in different training epochs may be different.

[0108] The above methods allow for flexible switching or alteration of the expert model's partitioning method to adapt to the characteristics of different training stages, making the trained expert model suitable for capturing more diverse feature patterns.

[0109] As an example, the steps of dividing a pre-defined expert network into multiple expert models based on the current training epoch may include: determining the division ratio for the expert network based on the current training epoch; and dividing the output dimension of the previous layer and the input dimension of the next layer in the adjacent network layers of the expert network based on the division ratio and the number of expert models to be divided, thereby obtaining multiple expert models.

[0110] Here, the partitioning ratio can be negatively correlated with the number of expert models to be partitioned. For example, the product of the partitioning ratio and the number of expert models to be partitioned can be a fixed value, such as 8 or 16.

[0111] Specifically, MoE can be viewed as a single feed-forward network (FFN), and multiple expert models of MoE can be obtained by partitioning the FFN. The output dimension of the first linear layer and the input dimension of the second linear layer of the FFN can be divided into blocks according to the partition ratio. Here, partitioning according to the partition ratio can refer to splitting the dimensions of the two linear transformation layers of the FFN.

[0112] For example, assuming the output dimension of the first linear layer of the original FFN is D, the input dimension of the second linear layer is also D, and the partition ratio is r, the FFN can be divided into blocks by splitting the output dimension of the first linear layer into D / r and the input dimension of the second linear layer into D / r.

[0113] In this example, the output dimension of the previous layer in the adjacent network layers of each expert model can be determined based on the output dimension of the previous layer in the adjacent network layers of the expert network, the partition ratio, and the number of expert models to be partitioned; the input dimension of the next layer in the adjacent network layers of each expert model can be determined based on the input dimension of the next layer in the adjacent network layers of the expert network and the partition ratio.

[0114] Through the above asymmetric partitioning, the expert structure can be changed while keeping the total number of parameters unchanged, so that the sum of all expert parameters obtained by partitioning is always equal to the number of parameters of the original FFN.

[0115] For example, assuming the original FFN layer dimension D=2048, and the product of the partition ratio and the number of expert models is fixed at 8, if the partition ratio r=1.0, the number of expert models to be partitioned n is 8, and the output dimension of the first linear layer of each expert can be (D / r) / n = 2048 / 8 = 256; the input dimension of the second linear layer of each expert can be (D / r) / n = 2048 / 8 = 256; if the partition ratio r=2.0, the number of expert models to be partitioned n is 4, and the output dimension of the first linear layer of each expert can be (D / r) / n = 2048 / (8×2) = 128; the input dimension of the second linear layer of each expert can be (D / r) / n = 2048 / (8×2) = 128.

[0116] In the example above, the total number of parameters in the expert model is always equal to the number of parameters in the original FFN. This allows the divided expert models to capture more diverse feature patterns. For example, some experts naturally tend to process high-frequency details (texture), while other experts focus on low-frequency structures (edges). This block-based strategy enables the model to optimize performance by adjusting the internal structure of the experts at a fixed computational cost.

[0117] Regarding the aforementioned expert partitioning strategy, in the embodiments of this disclosure, experiments were also conducted using existing Lightning-XL-Dense models, Lightning-XXL-Dense models, and the 8E2A2S, 4E1A1S, and 16E4A4S architectures according to embodiments of this disclosure (where E represents the number of expert models, A represents the number of attention layer configurations, and S represents shared expert partitioning). The FID values ​​were compared at training steps of 100k, 200k, 300k, 400k, 600k, 800k, and 1M (as shown in Table 1 below). The experimental results demonstrate that, when using the expert partitioning strategy of the embodiments of this disclosure, with an 8-expert configuration and a partitioning ratio r=1.0, the model performance inflection point can be obtained under the condition that the total parameter scale reaches 1.1B and the activation parameter is only 0.6B, achieving the optimal configuration with the lowest FID score. For example, at 1M training steps, the FID score can be 5.01. Furthermore, the above experiments also demonstrated that the number and scale of experts need to be designed in a coordinated manner, rather than simply increasing the number of experts. It is not true that the more experts there are, the higher the quality of the generated image, as is commonly believed in traditional methods.

[0118] Table 1

[0119] In an exemplary embodiment of this disclosure, the block ratio can also be dynamically adjusted according to the training phase to achieve dynamic expert model partitioning for the training time sequence.

[0120] As an example, the steps described above for determining the partition ratio for the expert network based on the current training epoch may include: determining the first ratio as the partition ratio in response to the current training epoch being in the first training phase; and determining the second ratio as the partition ratio in response to the current training epoch being in the second training phase.

[0121] Here, the first training phase can precede the second training phase, the first ratio can be smaller than the second ratio, and the number of expert models divided in the first training phase can be smaller than the number of expert models divided in the second training phase.

[0122] As an example, the first training phase (or "early training phase") can be a first predetermined proportion or a first predetermined number of training steps in the total training process, and the second training phase (or "late training phase") can be a second predetermined proportion or a second predetermined number of training steps in the total training process. For example, a relatively high split ratio can be used in the early training phase, resulting in a relatively small number of expert models, to accelerate convergence; a relatively low split ratio can be used in the late training phase, resulting in a relatively large number of expert models, to optimize details. As an example, the first ratio could be 2.0, and the second ratio could be 1.0.

[0123] For example, based on the number of training steps, the first 40% of the total training steps can be used as the first training phase, and the last 60% of the training steps can be used as the second training phase. Taking a total of 300k training steps as an example, the first 120k steps use a division ratio r=2.0, and the last 180k steps switch to a division ratio r=1.0.

[0124] In exemplary embodiments of this disclosure, alternatively or additionally, the block ratio can be dynamically adjusted according to image features to achieve dynamic expert model partitioning for image spatial distribution.

[0125] As an example, in the step above of determining the partition ratio for the expert network based on the current training round, different partition ratios can be used for different image regions of the sample image, or in other words, the number of expert models partitioned can be different.

[0126] Specifically, a third ratio can be used for pixel blocks in the first region (e.g., high-frequency detail region) of the sample image; and a fourth ratio can be used for pixel blocks in the second region (e.g., low-frequency contour region) of the sample image. The third ratio can be greater than the fourth ratio, and the number of expert models for the first region is greater than the number of expert models for the second region.

[0127] In this way, a non-uniform block strategy can be used to allocate expert dimensions according to the importance of image features. For example, a larger expert capacity can be allocated to high-frequency detail regions, which can improve resource utilization while ensuring image generation quality and avoid excessive expert capacity.

[0128] Furthermore, it should be noted that the aforementioned dynamic expert model partitioning for training time sequence and dynamic expert model partitioning for image spatial distribution can be applied individually or in combination.

[0129] The training method for the image generation module according to the exemplary embodiments of the present disclosure can provide a MoE design and routing mechanism for an image generation diffusion model to adapt to the characteristics of visual sequence structure and noise diffusion law, thereby obtaining a more stable and better performing model structure.

[0130] Specifically, the training method of the image generation module according to the exemplary embodiments of this disclosure can eliminate cross-sample interference caused by existing batch-level selection routing mechanisms in visual tasks, and ensure training stability under diffuse noise.

[0131] Furthermore, the training method of the image generation module according to the exemplary embodiments of this disclosure can enhance MoE's dynamic perception of the diffusion process time step and optimize expert allocation in key stages.

[0132] Furthermore, the training method of the image generation module according to the exemplary embodiments of this disclosure can achieve optimal performance balance under a fixed total number of parameters by fine-grained partitioning of the expert structure, avoiding diminishing returns caused by blindly expanding the number of experts.

[0133] In exemplary embodiments of this disclosure, comparative experiments were also conducted with other existing methods based on the methods of the embodiments of this disclosure. Here, a model training example is given, in which the number of parameters is set to 1B; the batch size is set to 1024; dropout is set to 0.1; the learning rate is set to 0.0001; the image size is set to 512×512; the MLP_ratio is set to 1.0 / 0.5; the number of experts is set to 16 experts / 8 experts; the number of shared experts is set to 2; the number of activated experts is set to 4 / 2; the β2 parameter of the AdamW optimizer is set to 0.999; the classifier guidance strength is set to 1.0, i.e., no guidance; and the absolute error tolerance of the ODE solver is set to 10. -6 The number of channels in the input latent representation is set to 16; data loading uses 12 subprocesses to read in parallel.

[0134] In the above comparative experiments, the comparison methods included the Expert Race method, the DiffMoE method, and the DiT-MoE method. The experimental results are shown in Table 2 below. Table 2 shows the FID scores of each method at 100k, 200k, and 300k training steps.

[0135] Table 2

[0136] As can be seen from Table 2 above, the training method according to the embodiments of this disclosure can achieve a low FID at different training steps, which proves that the image generation model trained by the above training method generates images of better quality.

[0137] The training method according to embodiments of this disclosure can eliminate cross-sample interference and enhance time step awareness. For example, by employing conditional modulation according to embodiments of this disclosure, the expert activation accuracy of MoE in a low-noise phase (e.g., t=100) can be improved by 27%. Furthermore, this method can enhance the specialization of expert models by encouraging different experts to focus on specific visual features (such as texture and edges) through fine-grained segmentation, resulting in a significant improvement in the clarity of generated image details.

[0138] According to a second aspect of the embodiments of this disclosure, an image generation method is provided, such as... Figure 6 As shown, the image generation method may include: step S610, receiving an image generation instruction; step S620, generating an image based on the image generation instruction and using an image generation model, wherein the image generation model is trained according to the image generation model training method described in the embodiments of this disclosure.

[0139] As an example, the image generation instruction can be an instruction to instruct the image generation model to generate a random image, or it can include text describing the image to be generated, instructing the image generation model to generate an image that matches the text.

[0140] In this image generation method, the process of generating images using the image generation model is similar to the process of generating predicted images described in the training method above, and can achieve the corresponding technical effects, so it will not be repeated here.

[0141] The image generation method according to the embodiments of this disclosure can improve the stability of generated images, avoid excessive differences in image quality generated in different image generation tasks, and improve the reliability of image generation.

[0142] According to a third aspect of the embodiments of this disclosure, a training apparatus for an image generation model is provided, the image generation model including an expert hybrid model, such as... Figure 7As shown, the training device 700 includes an acquisition unit 710, a routing decision unit 720, a prediction unit 730, and a training unit 740.

[0143] The acquisition unit 710 is configured to acquire training sample data, wherein the training sample data includes multiple sample images.

[0144] The routing decision unit 720 is configured to use a hybrid expert model as a router to make routing decisions for each of a plurality of sample images, selecting the target expert model corresponding to each sample image from a plurality of expert models in the hybrid expert model.

[0145] The prediction unit 730 is configured to generate a prediction image corresponding to each sample image using a target expert model corresponding to that sample image.

[0146] Training unit 740 is configured to train the image generation model based on each sample image and the corresponding prediction image.

[0147] In embodiments of this disclosure, for each sample image, the routing decision unit 720 is configured to make routing decisions using a router in the following manner: based on the image features of the sample image, determine the routing weight of each of a plurality of expert models for the sample image, wherein the routing weight characterizes the probability that the expert model corresponds to the sample image; and based on the routing weight, determine the target expert model corresponding to the sample image.

[0148] In embodiments of this disclosure, the routing decision unit 720 is further configured to make routing decisions using a router in the following manner: modulating the routing weights of multiple expert models for the sample image according to the current training time step to obtain modulated weights; and determining the target expert model corresponding to the sample image based on the modulated weights.

[0149] In embodiments of this disclosure, the image generation model further includes a first conditional modulation module, which is used to generate a time-aware vector based on the current training time step. The routing decision unit 720 is further configured to use a router to make routing decisions in the following manner: obtain the time-aware vector from the first conditional modulation module; and modulate the routing weights of multiple expert models for the sample image based on the time-aware vector to obtain modulated weights.

[0150] In embodiments of this disclosure, the training sample data further includes sample text corresponding to each sample image, each sample text describing the content of the corresponding sample image. The first conditional modulation module is further configured to generate a modulation vector based on the current training time step and the sample text. The routing decision unit 720 is further configured to make routing decisions using a router in the following manner: obtaining the modulation vector from the first conditional modulation module; fusing the time-aware vector and the modulation vector to obtain a fused vector; and modulating the routing weights of multiple expert models for the sample image based on the fused vector to obtain modulated weights.

[0151] In embodiments of this disclosure, the image generation model further includes a second conditional modulation module and an attention layer. The output of the second conditional modulation module is input to the attention layer. The second conditional modulation module is the same as the first conditional modulation module.

[0152] In embodiments of this disclosure, the training device performs multiple training rounds, wherein the training device further includes an expert partitioning unit, which is configured to obtain multiple expert models by partitioning a preset expert network according to the current training round to obtain multiple expert models.

[0153] In embodiments of this disclosure, the expert partitioning unit is configured to: determine a partitioning ratio for the expert network based on the current training epoch, wherein the partitioning ratio is negatively correlated with the number of expert models to be partitioned; and partition the output dimension of the previous layer and the input dimension of the next layer in the adjacent network layers of the expert network based on the partitioning ratio and the number of expert models to be partitioned, thereby obtaining multiple expert models.

[0154] In embodiments of this disclosure, the expert partitioning unit is configured to: determine a first ratio as a partitioning ratio in response to the current training epoch being in a first training phase; and determine a second ratio as a partitioning ratio in response to the current training epoch being in a second training phase, wherein the first ratio is greater than the second ratio before the second training phase, and the number of expert models partitioned in the first training phase is less than the number of expert models partitioned in the second training phase.

[0155] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0156] According to a fourth aspect of the embodiments of this disclosure, an image generation apparatus is provided, such as... Figure 8 As shown, the image generating apparatus 800 includes a receiving unit 810 and an image generating unit 820.

[0157] The receiving unit 810 is configured to receive an image generation instruction. The image generation unit 820 is configured to generate an image based on the image generation instruction and using an image generation model, wherein the image generation model is trained according to a training method for an image generation model according to an exemplary embodiment of the present disclosure.

[0158] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0159] According to a fifth aspect of the embodiments of this disclosure, an electronic device is provided, such as... Figure 9 As shown, the electronic device includes: a processor 910; and a memory 920 for storing processor-executable instructions, wherein, when executed by the processor 910, the processor 910 causes the processor 910 to perform a training method or an image generation method for an image generation model according to an exemplary embodiment of the present disclosure.

[0160] As an example, an electronic device does not necessarily have to be a single device; it can also be a collection of any means or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. An electronic device can also be part of an integrated control system or system manager, or can be configured to interface with a server, either locally or remotely (e.g., via wireless transmission).

[0161] In electronic devices, processors may include central processing units (CPUs), graphics processing units (GPUs), programmable logic devices, dedicated processor systems, microcontrollers, or microprocessors. By way of example and not limitation, processors may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0162] The processor can execute instructions or code stored in memory, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transport protocol.

[0163] Memory can be integrated with the processor; for example, RAM or flash memory can be housed within an integrated circuit microprocessor. Alternatively, memory can comprise a separate device, such as an external disk drive, storage array, or other storage device that can be used by any database system. Memory and processor can be operatively coupled, or can communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor to read files stored in the memory.

[0164] In addition, electronic devices may include video displays (such as liquid crystal displays) and user interaction interfaces (such as keyboards, mice, touch input devices, etc.). All components of the electronic device can be interconnected via buses and / or networks.

[0165] In an exemplary embodiment, a computer-readable storage medium may also be provided, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform a training method or an image generation method for an image generation model according to an exemplary embodiment of the present disclosure.

[0166] Computer-readable storage media may be, for example, a memory that includes instructions. Optionally, computer-readable storage media may be: read-only memory (ROM), random access memory (RAM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0167] In an exemplary embodiment, a computer program product is also provided, including computer-executable instructions that, when executed by at least one processor, implement a training method or an image generation method for an image generation model according to embodiments of the present disclosure.

[0168] The description in this disclosure has been presented for illustrative purposes and is not intended to be exhaustive or limited to this disclosure. Many modifications, variations, and alternative embodiments will be apparent to those skilled in the art from the teachings presented in the foregoing description and the associated drawings.

[0169] Unless otherwise specifically stated, the order of steps in the method according to this disclosure is intended to be illustrative only, and the steps of the method according to this disclosure are not limited to the specific order described above, but may be changed according to actual circumstances. Furthermore, at least one step in the method according to this disclosure may be adjusted, combined, or omitted as needed.

[0170] The examples chosen and described are intended to explain the principles of this disclosure and to enable others skilled in the art to understand the various embodiments of this disclosure, and preferably to utilize the basic principles and various embodiments with various modifications suitable for the intended particular purpose. Therefore, it will be understood that the scope of this disclosure is not limited to the specific examples of the disclosed embodiments, and that modifications and other embodiments are intended to be included within the scope of this disclosure.

[0171] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0172] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for an image generation model, characterized in that, The image generation model includes an expert hybrid model, and the training method includes: Acquire training sample data, wherein the training sample data includes multiple sample images; The router using the expert fusion model performs routing decisions for each of the multiple sample images, selecting the target expert model corresponding to each sample image from the multiple expert models of the expert fusion model; Using the target expert model corresponding to each sample image, a predicted image corresponding to that sample image is generated; The image generation model is trained based on each sample image and the corresponding predicted image.

2. The training method according to claim 1, characterized in that, For each sample image, the router makes routing decisions in the following manner: Based on the image features of the sample image, the routing weight of each expert model among the plurality of expert models for the sample image is determined, wherein the routing weight represents the probability that the expert model corresponds to the sample image; Based on the routing weights, the target expert model corresponding to the sample image is determined.

3. The training method according to claim 2, characterized in that, The step of determining the target expert model corresponding to the sample image based on the routing weights includes: Based on the current training time step, the routing weights of the multiple expert models for the sample image are modulated to obtain the modulated weights; Based on the modulated weights, the target expert model corresponding to the sample image is determined.

4. The training method according to claim 3, characterized in that, The image generation model further includes a first conditional modulation module, which is used to generate a time-aware vector based on the current training time step. The step of modulating the routing weights of the multiple expert models for the sample image according to the current training time step to obtain the modulated weights includes: The time-aware vector is obtained from the first conditional modulation module; Based on the time-aware vector, the routing weights of the multiple expert models for the sample image are modulated to obtain the modulated weights.

5. The training method according to claim 4, characterized in that, The training sample data also includes sample text corresponding to each sample image, whereby each sample text describes the content of the corresponding sample image. The first conditional modulation module is further configured to generate a modulation vector based on the current training time step and the sample text. The step of modulating the routing weights of the multiple expert models for the sample image based on the time-aware vector to obtain the modulated weights includes: The modulation vector is obtained from the first conditional modulation module; The time-aware vector and the modulation vector are fused to obtain a fused vector; Based on the fusion vector, the routing weights of the multiple expert models for the sample image are modulated to obtain the modulated weights.

6. The training method according to claim 5, characterized in that, The image generation model further includes a second conditional modulation module and an attention layer. The output of the second conditional modulation module is input to the attention layer. The second conditional modulation module is the same as the first conditional modulation module.

7. The training method according to claim 1, characterized in that, The training method includes multiple training rounds, wherein the multiple expert models are obtained through the following method: Based on the current training round, the preset expert network is divided to obtain the multiple expert models.

8. The training method according to claim 7, characterized in that, The step of dividing the preset expert network according to the current training round to obtain the multiple expert models includes: Based on the current training epoch, determine the partitioning ratio for the expert network, wherein the partitioning ratio is negatively correlated with the number of expert models to be partitioned; Based on the partitioning ratio and the number of expert models to be partitioned, the output dimension of the previous layer and the input dimension of the next layer in the adjacent network layers of the expert network are partitioned to obtain the multiple expert models.

9. The training method according to claim 8, characterized in that, Determining the partitioning ratio for the expert network based on the current training epoch includes: In response to the fact that the current training round is in the first training phase, the first ratio is determined as the division ratio; Since the current training round is in the second training phase, the second ratio is determined as the aforementioned division ratio. Wherein, the first training phase precedes the second training phase, the first ratio is greater than the second ratio, and the number of expert models divided in the first training phase is less than the number of expert models divided in the second training phase.

10. An image generation method, characterized in that, The image generation method includes: Receive image generation instructions; Based on the image generation instructions, an image is generated using an image generation model, wherein the image generation model is trained using the training method for the image generation model according to any one of claims 1 to 9.

11. A training device for an image generation model, characterized in that, The image generation model includes an expert hybrid model, and the training device includes: The acquisition unit is configured to acquire training sample data, wherein the training sample data includes multiple sample images; The routing decision unit is configured to utilize the expert fusion model of the router to make routing decisions for each of the plurality of sample images, and to select the target expert model corresponding to each sample image from the plurality of expert models of the expert fusion model. The prediction unit is configured to generate a prediction image corresponding to each sample image using a target expert model corresponding to that sample image. The training unit is configured to train the image generation model based on each sample image and the corresponding prediction image.

12. An image generation apparatus, characterized in that, The image generation device includes: The receiving unit is configured to receive image generation instructions; An image generation unit is configured to generate an image based on the image generation instructions and using an image generation model, wherein the image generation model is trained using the training method for the image generation model according to any one of claims 1 to 9.

13. An electronic device, characterized in that, The electronic device includes: Processor; and Memory used to store processor-executable instructions. Wherein, when the processor executes the processor, the processor causes the processor to perform the training method of the image generation model according to any one of claims 1 to 9 or the image generation method according to claim 10.

14. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the training method of the image generation model according to any one of claims 1 to 9 or the image generation method according to claim 10.

15. A computer program product comprising computer-executable instructions, characterized in that, When the computer-executable instructions are executed by at least one processor, they implement the training method of the image generation model according to any one of claims 1 to 9 or the image generation method according to claim 10.

Images