Image generation method and device

Abstract

The invention provides an image generation method and device, and is applied to the technical field of artificial intelligence. The method comprises the steps that control features are obtained, the control features are obtained by performing different image task processing by a dynamic hybrid expert module according to a control image, and the control image is a structure description image corresponding to text features; and inputting the first data and the control features into an image generation model to obtain a target image, the first data including text features, a plurality of time steps and a noise sample, and the control features being used for guiding the noise sample to be restored to an image corresponding to a structure described by the control image.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to an image generation method and apparatus. Background Technology

[0002] Currently, image generation models are unable to fully understand users' increasingly complex and personalized image generation needs, thus reducing the user experience. Summary of the Invention

[0003] In view of this, the present disclosure provides an image generation method and apparatus.

[0004] According to a first aspect of this disclosure, an image generation method is provided, comprising: obtaining control features, wherein the control features are obtained by a dynamic hybrid expert module performing different image task processing on a control image, the control image being a structural description image corresponding to text features; inputting first data and control features into an image generation model to obtain a target image, wherein the first data includes text features, multiple time steps and noise samples, and the control features are used to guide the noise samples to be restored to an image corresponding to the structure described by the control image.

[0005] A second aspect of this disclosure provides an image generation apparatus, comprising: an acquisition module for acquiring control features, wherein the control features are obtained by a dynamic hybrid expert module performing different image task processing on a control image, and the control image is a structural description image corresponding to text features; and an image generation module capable of independently executing or calling an image generation model to perform the following operations: inputting first data and control features into the image generation model to obtain a target image, wherein the first data includes text features, multiple time steps, and noise samples, and the control features are used to guide the noise samples to be restored to an image corresponding to the structure described by the control image.

[0006] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0007] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0008] Figure 1 This illustration schematically depicts an application scenario of the image generation method and apparatus according to embodiments of the present disclosure;

[0009] Figure 2 A flowchart illustrating an image generation method according to an embodiment of the present disclosure is shown schematically;

[0010] Figure 3A schematic diagram of a dynamic hybrid expert module according to an embodiment of the present disclosure is shown.

[0011] Figure 4 A schematic diagram illustrating the principle of an image generation method according to an embodiment of the present disclosure is shown.

[0012] Figure 5 This illustration schematically shows the effect of an image generation method according to an embodiment of the present disclosure;

[0013] Figure 6 A schematic block diagram of an image generation apparatus according to an embodiment of the present disclosure is shown. Detailed Implementation

[0014] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0015] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0016] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0017] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0018] This disclosure provides an image generation method. Before introducing the technical solutions provided by this disclosure, the relevant technologies involved in this disclosure will be described first.

[0019] For example, with the rapid development of generative AI technology, image generation models can generate high-quality and creative images based on text prompts. However, in real-world applications, users' control requirements for image generation are highly dynamic, diverse, and continuously evolving. For instance, users often not only require "generating an image that matches the description," but also want to impose various control conditions on the generation process (such as style, composition, pose, edge / depth / semantic segmentation constraints, specific themes or hotspot elements, etc.) to improve the usability and controllability of the image.

[0020] The need for continuous learning is particularly prominent in application scenarios. On the one hand, trending events or popular trends are often time-sensitive, requiring models to quickly learn and implement new control capabilities within a short period. On the other hand, users may frequently switch between different types of control condition inputs within the same system, expecting models to have the ability to uniformly handle multiple control conditions, rather than relying on switching between multiple dedicated models. Therefore, building an image generation system that can continuously absorb new knowledge and uniformly handle multi-condition control has become a key challenge in promoting the practical application of image generation technology.

[0021] Therefore, a novel control architecture is urgently needed that can efficiently adapt to new tasks and support multi-condition unified modeling and dynamic scheduling to achieve a controllable image generation system for a continuous stream of new tasks in an open set. The Mixture of Experts (MoE) architecture, with its dynamic routing and sparse activation characteristics, naturally aligns with the continuous learning requirements of multi-condition unified control models. However, traditional MoE architectures still have key drawbacks when applied to this task: their structure is typically statically designed, with the number of experts fixed during the initial training phase, making it unable to support dynamic growth of the model after deployment to adapt to a continuous stream of open set tasks.

[0022] Before providing a further detailed description of the embodiments of this disclosure, the nouns and terms involved in the embodiments of this disclosure will be explained, and the nouns and terms involved in the embodiments of this disclosure shall be interpreted as follows.

[0023] StableDiffusion (SD) model: a stable diffusion model.

[0024] LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method designed to fine-tune large pre-trained models (such as Stable Diffusion) with minimal computational cost and parameter count, adapting them to new, specific tasks or data.

[0025] Figure 1 The illustration shows an application scenario of the image generation method according to an embodiment of the present disclosure.

[0026] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a terminal 110 and a server 120, wherein the terminal 110 and the server 120 can communicate through a wired or wireless network connection.

[0027] Terminal 110 (running a client capable of image generation) can be used to receive image generation requests. For example, a user sends an image generation request to server 120 through terminal 110. Server 120 obtains first data and control features based on the received image generation request. Server 120 invokes an image generation model and inputs the first data and control features into the image generation model to obtain the target image. Server 120 returns the target image to terminal 110.

[0028] Terminal 110 includes, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, and aircraft. Terminal 110 is equipped with client software with image processing capabilities, such as an application (App). This application can be a standalone application or a subroutine within an application. The image processing capabilities may include image generation functions, such as generating images based on prompt text.

[0029] Server 120 can provide background services for applications in terminal 110. These background services may include image processing services, such as image generation services. Server 120 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0030] The following will be based on Figure 1 The described scene, through Figures 2-5 The image generation method according to the embodiments of this disclosure will be described in detail.

[0031] Figure 2 A flowchart illustrating an image generation method according to an embodiment of the present disclosure is shown schematically.

[0032] like Figure 2 As shown, the image generation method of this embodiment includes operations S210 to S220.

[0033] In operation S210, control features are obtained. The control features are obtained by the dynamic hybrid expert module through different image task processing based on the control image. The control image is a structural description image corresponding to the text features.

[0034] In operation S220, the first data and control features are input into the image generation model to obtain the target image. The first data includes text features, multiple time steps, and noise samples. The control features are used to guide the noise samples to be restored to an image corresponding to the structure described by the control image.

[0035] For example, text features are a series of high-dimensional numerical vectors obtained by transforming a user's natural language description of the image generation request through a text encoder. For instance, text features could be text features corresponding to prompts entered by the user for image generation; for example, a user might enter the following text in an image generation application: "Generate an image of a snow-capped mountain bathed in the glow of a sunset, with a tranquil lake at its foot, and a film-like texture." This text information is then transformed into text features through a text encoder.

[0036] Control images can be visual sketches or professional images used to define the structure of the target image. Control images focus on the spatial structure, contours, and geometric layout of the generated image content, supplementing the visual structure of text features. For example, a control image can be a contour map defining the outer boundaries of objects and scenes, such as a sketch of a cat's outline; it can also be a pose skeleton map, such as an image of a specific dance pose. A control image can also be a depth map, such as a depth map of a landscape, ensuring the generated image has the correct foreground, midground, and background relationships. A control image can also be a semantic segmentation map, such as a segmentation map labeled "sky," "mountain," "tree," and "road," ensuring the generated elements appear in the correct positions. A control image can also be a color block map, such as a color block map dominated by red and yellow tones, guiding the generation of a warm-toned sunset scene.

[0037] For example, the text feature is "a modern-style wooden house covered in snow, nestled at the foot of a mountain." The corresponding control image could be a sketch of the wooden house's architectural outline, or a simple region segmentation map indicating the location of the mountain, flat land, and house. The sketch / segmentation map provides structural information that is either implicit or not explicitly described in the text.

[0038] Control features can be the result of deep processing of information extracted from the control image by a dynamic hybrid expert module. During iterative denoising, control features act as constraints, causing each prediction step (such as noise prediction / residual prediction / velocity prediction) to tend to generate results that conform to the structure of the control image, thus ensuring that the final output is consistent with or highly consistent with the control image in terms of contour, pose, depth, or layout. For example, a user inputs a human pose skeleton as the control image. This image is then fed into the dynamic hybrid expert module. Within this module, a specialist expert handling pose tasks may be activated, parsing and encoding the skeleton into a set of feature vectors that describe the human joint angles, limb lengths, and spatial positions. These feature vectors are the control features.

[0039] A dynamic hybrid expert module can be a scalable neural network architecture composed of multiple expert subnetworks, each designed to handle different image tasks. For example, when given a control image as input, the dynamic hybrid expert module can dynamically select one or more expert subnetworks to process the control image based on its type, thereby obtaining more accurate and task-specific control features.

[0040] Different image tasks can control the different types or processing targets of the images, such as edge detection (corresponding to line drawings), depth estimation (corresponding to depth maps), human pose estimation (corresponding to pose maps), and semantic segmentation (corresponding to segmentation mask maps). For example, when the input is a sketch of an indoor scene, the dynamic hybrid expert module may activate two expert sub-networks that excel at "scene layout understanding" and "edge semantic extraction." When the input is a semantic segmentation map of a human face (e.g., distinguishing hair, skin, and clothing regions), the dynamic hybrid expert module activates an expert sub-network that excels at "semantic region encoding." The control features generated in each case have internal representations optimized for the current task.

[0041] An image generation model can be a neural network system based on a diffusion model, capable of accepting multimodal conditional inputs and generating high-quality images. For example, an image generation model can be an image generation model, such as the StableDiffusion series of models.

[0042] Multiple time steps can be discrete index sequences representing different noise levels during the iterative denoising process of the diffusion model.

[0043] The noisy sample can be a random starting point for the generation process, which is a random tensor sampled from a standard normal distribution. The entire generation process starts with this noise, gradually subtracts the predicted noise, and finally obtains a clear image.

[0044] In one example, text features, control features, multiple time steps, and noise samples are input into the StableDiffusion model. This model performs a denoising process from noise to a clear image under strong constraints, ultimately outputting a target image that meets all conditions. The text features are: "a woman in an evening gown dancing in the center of a ballroom"; control features are: joint coordinates and orientation features (extracted from the pose map); multiple time steps are: [1000, 850, 700, ..., 1] (representing the denoising progress); and the noise samples are: a randomly sampled Gaussian noise tensor (e.g., 64x64x4). The control features provide additional, more precise geometric and spatial constraints. They work in conjunction with the text features at each step of denoising, forcing the model to align the structure of the generated image with the structure defined by the control images when removing noise.

[0045] Understandably, by introducing control features generated by the dynamic hybrid expert module and inputting them together with the first data containing text features, multiple time steps, and noise samples into the image generation model, this scheme can adaptively route and perform specialized processing on different types of structural control images within the same generation framework. This enables structural guidance for the noise iterative restoration process, thereby significantly improving structural controllability and cross-control task uniformity while ensuring text semantic consistency. It also reduces the need for frequent switching or maintenance of multiple dedicated models for different control conditions and improves the scalability when new control tasks are added.

[0046] As described above, in operation S210, control features are obtained. In one possible implementation, this operation may further include the following steps: inputting second data into a multi-condition control module to obtain a first feature, the second data including latent space features, text features, and time steps, the latent space features being encoded features of the structure described by the control image, and the first feature being a fused representation of the latent space features and text features of the control image; inputting the first feature into a self-attention module to obtain a second feature; inputting the first feature into a dynamic hybrid expert module to obtain a third feature; and determining the control features corresponding to each time step based on the second and third features.

[0047] For example, latent space features can be a low-dimensional feature representation of the control image after compression by an encoder. For instance, when the control image is a 512×512 resolution human pose skeleton image, after being processed by a variational autoencoder (VAE), the image can be compressed into a 64×64×4 latent space feature tensor, where 4 represents the number of feature channels.

[0048] The multi-conditional control module can be a neural network processing unit capable of simultaneously receiving and fusing inputs from multiple modalities. Internally, this module performs feature alignment and cross-modal fusion operations, uniformly encoding conditional signals from different information sources into an intermediate representation adapted to the image generation model. For example, the multi-conditional control module can employ a modulation mechanism to generate dynamic modulation parameters based on text features and time steps, scaling and shifting latent space features channel-by-channel, thereby injecting textual semantics and temporal progression information into the structural features.

[0049] The first feature can be a comprehensive feature representation that integrates control image structural information, textual semantic information, and generation progress information. This feature preserves the spatial layout constraints of the control image, embeds semantic guidance from the text description, and carries the progress encoding of the current denoising time step, providing comprehensive conditional guidance for subsequent image generation. For example, in generating the scene "a woman in an evening gown dancing in the center of a ballroom," the first feature simultaneously includes limb angle and position information extracted from the pose skeleton map, style and scene semantics extracted from words such as "evening gown" and "ballroom," and the denoising intensity encoding corresponding to the current time step (e.g., t=850).

[0050] A self-attention module can be a feature enhancement unit based on an attention mechanism, used to capture long-range dependencies between different spatial locations within a feature. This module aggregates and propagates global contextual information by calculating the similarity weights of each location in a feature with all other locations. For example, when a location in the first feature corresponds to the structural information of a "person's head," the self-attention module can associate it with related structural information such as "shoulder" or "arm" in other locations, thereby enhancing the overall consistency of the feature representation. The self-attention module can also employ a multi-head attention mechanism, computing attention weights in parallel across different representation subspaces, enabling the model to simultaneously focus on multiple types of spatial relationships. For instance, one attention head might focus on left-right symmetry, while another focuses on vertical continuity.

[0051] The second feature can be a feature representation with stronger global consistency after self-attention enhancement. Each position in this feature no longer contains only its local information, but integrates the contextual information of all positions related to it in the entire feature map. For example, when dealing with human posture control tasks, the representation of the "left hand" position in the second feature will incorporate information from the "right hand" and "torso" positions, making the generated image more reasonable in terms of limb coordination and movement continuity.

[0052] The third feature can be a specialized feature representation processed by a task-specific expert network. This feature is optimized for the specific needs of the current control task, and has stronger task relevance and discriminative ability. For example, when dealing with posture control tasks, the third feature will strengthen the expression of key posture information such as joint position and limb orientation; when dealing with depth control tasks, the third feature will highlight the encoding of key depth information such as foreground and background layers and object occlusion relationships.

[0053] In one example, the second and third features can be fused element-wise to obtain the final control feature. In another example, the second and third features can be projected separately through a learnable linear layer and then concatenated, followed by outputting the control feature through a zero-convolutional layer. The zero-convolutional layer can be a convolutional layer with weights and biases initialized to zero. It is used to set the output of the control module to zero in the early stages of training to ensure that it does not interfere with the original performance of the pre-trained image generation model. As training progresses, the parameters of the zero-convolutional layer gradually learn effective fusion weights.

[0054] In one example, continuing the previous task of generating the image of "a woman dancing in the center of a ballroom in an evening gown," the complete process of obtaining the control features is as follows. First, the user provides a human posture skeleton image as the control image. This image is processed by a VAE encoder to obtain latent space features of size 64×64×4. This latent space feature, along with text features and the current time step t=850, is input into the multi-condition control module. Internally, the multi-condition control module generates modulation coefficients based on the text features and the time step, and modulates the latent space feature channel by channel to obtain a first feature that integrates text semantics and temporal information, still with a size of 64×64×4. Subsequently, the first feature is input into the self-attention module, which calculates the similarity matrix between various positions within the feature map and aggregates global context information accordingly, outputting an enhanced second feature. Simultaneously, the first feature is sent to the dynamic hybrid expert module. The router in this module recognizes that the current task is posture control, activates two posture expert networks, and after the output features from both are calculated, they are weighted and summed with weights of 0.6 and 0.4 respectively to obtain the third feature. Finally, the second and third features are fused element-wise and passed through a 1×1 convolutional layer with zero weights to output the final control feature. This control feature, along with the noise samples, is input into the image generation model during subsequent diffusion model iterations for denoising. At each time step, it guides the model to gradually approximate the target image that both conforms to the semantic requirements of the text description and strictly follows the human motion structure defined by the pose skeleton diagram.

[0055] Understandably, the ability to achieve deep fusion of cross-modal information, effective modeling of global spatial relationships, and precise injection of task-specific knowledge within a unified processing flow enables control features to simultaneously possess structural accuracy, semantic consistency, and task specificity. This significantly improves the ability of generated images to strictly adhere to control image structural constraints while satisfying textual descriptions, and supports flexible handling of multiple types of control conditions within a single framework, reducing the development costs of building independent processing flows for different control tasks.

[0056] As described above, the dynamic hybrid expert module includes multiple initial image expert models. The initial image expert models are used to enhance different types of features of the control image. The image generation method of this embodiment may further include the following operations: creating routing information corresponding to the first feature based on the first feature; using the routing information to determine at least one target image expert model corresponding to the first feature; and using at least one target image expert model to process the first feature to obtain a second feature.

[0057] For example, routing information can be a decision vector or weight distribution indicating which expert models should be activated. This routing information calculates the suitability of each initial image expert model for the current task by analyzing the content characteristics, task attributes, and semantic features of the first feature, thereby providing a quantitative basis for subsequent expert selection.

[0058] The target image expert model can be one or more expert subnetworks selected from multiple initial image expert models that are best suited to handle the task corresponding to the current first feature. This selection process is based on the weights or probability distribution in the routing information, selecting expert models whose activation scores exceed a preset threshold or whose ranking is in the top K as the target image expert model.

[0059] Initial image expert models can be pre-trained or initialized neural network modules, each focused on processing specific types of control image features. Different initial expert models differ in network parameters and feature extraction emphases, making them adept at handling different types of structural control tasks such as pose, depth, edge, semantic segmentation, and color. For example, initial image expert models could include pose enhancement experts, which enhance the feature representation of human joint positions and limb connections through a dedicated skeleton keypoint encoding layer; depth enhancement experts, which use a multi-scale spatial pyramid structure to capture foreground and background hierarchical relationships; and edge enhancement experts, which use a high-pass filtering mechanism to highlight contour and boundary information. Initial image expert models can also be implemented using a low-rank adaptation (LoRA) structure, which inserts a low-rank decomposition matrix into the pre-trained backbone network, achieving task-specific feature enhancement with a small number of additional parameters, thereby giving each expert differentiated processing capabilities while maintaining overall parameter efficiency.

[0060] Initial image expert models are used to enhance different types of features in the control image. Specifically, different initial expert models, through their respective parameter configurations and network structures, selectively amplify and refine the information dimensions in the input features that are relevant to their specialized task. For example, when the first feature contains latent spatial features encoded from the pose skeleton map, the pose enhancement expert will focus on enhancing the channels or spatial locations in the features that represent pose-related aspects such as joint angles, limb length ratios, and motion coherence, while suppressing irrelevant information such as background texture and color distribution. This targeted enhancement enables each expert model to provide high-quality feature transformations in its respective domain, providing more accurate structural guidance for subsequent image generation.

[0061] When the determined target image expert model is a single expert, the first feature is directly input into the expert model for forward propagation, and the feature output by the expert is the second feature. When the determined target image expert model consists of multiple experts, the first feature is input into each target expert model for parallel processing. Each expert outputs an intermediate feature, and then these intermediate features are weighted and summed according to the weight values ​​of each expert in the routing information to obtain the final second feature.

[0062] In one example, continuing the previous task of generating "a woman in an evening gown dancing in the center of a ballroom," the complete dynamic hybrid expert processing procedure is as follows. In the preliminary steps, the multi-condition control module has output a first feature F1, which integrates latent space features of the pose skeleton map, text features, and information from time step t=850, with a size of 64×64×4. This first feature is fed into the dynamic hybrid expert module, which currently contains five initial image expert models: edge enhancement expert E1, pose enhancement expert E2, depth enhancement expert E3, semantic segmentation enhancement expert E4, and color enhancement expert E5. First, the first feature F1 is input into the router network, which processes it to obtain routing information [0.05, 0.68, 0.03, 0.21, 0.03]. According to the Top-2 selection strategy, the two experts with the highest weights are selected from the routing information: pose enhancement expert E2 (weight 0.68) and semantic segmentation enhancement expert E4 (weight 0.21), and these two are determined as the target image expert models. It's important to note that although the current control image is a pose skeleton map, the router simultaneously activates the semantic segmentation expert E4. This is because the text features contain semantic elements such as "evening dress" and "dance hall," implying that the generated image needs to present the corresponding content in the correct semantic regions (e.g., the person is wearing an evening dress, and the background represents a dance hall environment). Therefore, the semantic segmentation expert is partially activated to assist in the structural alignment of the semantic regions. Subsequently, the first feature F1 is input into experts E2 and E4 for parallel processing. Expert E2 enhances the feature channels encoding the human joint positions in F1, outputting the intermediate feature F2', which remains 64×64×4 but with an average increase in the response amplitude of the joint position channels of approximately 2.3 times. Expert E4 strengthens the feature representation of the boundary between the main body region and the background region in F1, outputting the intermediate feature F4'. Finally, the two intermediate features are weighted and fused according to the routing weights: F2 = 0.68×F2' + 0.21×F4', resulting in the second feature F2. This second feature retains the precise joint position information enhanced by the pose expert E2 (dominant, weight 0.68) and incorporates the separation cues between the person and the background region provided by the semantic expert E4 (auxiliary role, weight 0.21). This enables the subsequently generated image to strictly follow the skeleton pose while correctly rendering the details of the evening gown in the person region and accurately presenting the ballroom environment in the background region, thus achieving synergy between pose structure control and semantic region control.

[0063] It is understandable that routing information enables adaptive and specialized processing of different types of control images within a unified dynamic hybrid expert framework.

[0064] As described above, the operation involves using routing information to determine at least one target image expert model corresponding to the first feature. In one possible implementation, this operation may further include: using routing information to determine a candidate image task corresponding to the first feature; and determining at least one target image expert model from multiple initial image expert models based on the candidate image task, the task attribute information of the initial image expert model, and the state information of the initial image expert model.

[0065] For example, the candidate image task can be one or more image processing task types that are most likely involved in the current first feature, inferred from the activation weights or score distributions of each expert in the routing information.

[0066] Task attribute information can be pre-configured metadata for each initial image expert model, used to identify static attributes such as the type of task the expert excels at, the types of control images applicable, and optimization objectives. For example, the task attribute information for pose enhancement expert E2 could be labeled as {task type: human pose estimation, applicable input: skeleton keypoint map, optimization objective: joint position accuracy and limb coherence}; the task attribute information for semantic segmentation enhancement expert E4 could be labeled as {task type: semantic region segmentation, applicable input: multi-class labeled map, optimization objective: region boundary clarity and category consistency}. Task attribute information can also include expert capability labels, for example, one depth expert might be labeled as "excellent at indoor scene depth estimation," and another depth expert might be labeled as "excellent at outdoor distant scene depth estimation," enabling the system to match experts based on more granular task requirements.

[0067] State information can be dynamic data reflecting the current running status, load, or performance metrics of the initial image expert model. This information allows the expert selection process to rely not only on task matching but also on the feasibility and efficiency of actual execution. For example, state information can include metrics such as whether an expert is currently active, the average inference time over the past N calls, memory usage, and the success rate of recent tasks. State information can also include the training maturity of the expert model; for example, the state information of a newly added expert E6 might be labeled as "Training rounds: 5000, Validation loss: 0.032," while the mature expert E2 might be labeled as "Training rounds: 50000, Validation loss: 0.008." State information can also record the expert's call history; for example, an expert might have been successfully activated and returned valid results in 92 out of the last 100 calls.

[0068] In one example, continuing the previous task of generating "a woman in an evening gown dancing in the center of a ballroom," the complete target expert determination process is as follows. In the preceding steps, the router network has calculated routing information [0.05, 0.68, 0.03, 0.21, 0.03] based on the first feature F1. The five dimensions of this routing information correspond to five task categories: edge, pose, depth, semantics, and color. First, the system uses this routing information to determine candidate image tasks. According to the preset Top-2 strategy, the two dimensions with the highest weights in the routing information are selected: the second dimension (weight 0.68) and the fourth dimension (weight 0.21). The second and fourth dimensions are designated as "human pose estimation task" and "semantic region segmentation task," respectively, and these two are determined as candidate image tasks. Second, the system performs expert screening based on the candidate image tasks, expert task attribute information, and state information. The current dynamic hybrid expert module contains seven initial image expert models: edge expert E1, pose expert E2, depth expert E3, semantic expert E4, color expert E5, newly added pose expert E9, and outdoor depth expert E7. The system first iterates through the task attribute information of each expert and finds that E2's task type is labeled "human pose estimation," E9's task type is also labeled "human pose estimation," and E4's task type is labeled "semantic region segmentation." These three match the candidate image task and are included in the initial selection set. The remaining experts E1, E3, E5, and E7 are excluded because their task types do not match. Subsequently, the system checks the status information of the three experts in the initial selection set. Expert E2's status information shows: {Active: Available, Average inference time: 48 milliseconds, Memory usage: 1.2GB, Recent success rate: 98%, Training epochs: 50000}; Expert E9's status information shows: {Active: Available, Average inference time: 115 milliseconds, Memory usage: 1.5GB, Recent success rate: 86%, Training epochs: 8000}; Expert E4's status information shows: {Active: Available, Average inference time: 52 milliseconds, Memory usage: 1.1GB, Recent success rate: 96%, Training epochs: 45000}. For the pose task, although both E2 and E9 are available, E2 has a better state, so E2 is selected. For the semantic task, E4 is directly selected. After state filtering, expert E2 and expert E4 are finally determined as the target image expert models. The first feature F1 is input into E2 and E4 respectively for enhancement. E2 enhances joint position features through a skeleton keypoint convolutional layer, and E4 enhances the semantic boundary features between the person and the background through a region-aware attention mechanism. The intermediate features output by the two are fused into the second feature F2 with weights of 0.68 and 0.21 respectively.

[0069] It is understandable that by combining the task attribute information and state information of the initial expert model to select the most matching and usable target image expert model from multiple initial experts, task-oriented intelligent expert selection and state-aware load optimization can be achieved within a dynamic hybrid expert framework, avoiding the waste of computing resources and the degradation of output quality caused by blindly activating experts that are not task-matched or have poor performance.

[0070] Figure 3 A schematic diagram of a dynamic hybrid expert module according to an embodiment of the present disclosure is shown.

[0071] As described above, the operation involves determining at least one target image expert model from multiple initial image expert models based on the candidate image task, the task attribute information of the initial image expert model, and the state information of the initial image expert model. In some embodiments, this operation may further include at least one of the following operations: if the candidate image task has the same task attribute information as the first initial image expert model, and the learning state of the first initial image expert model satisfies a first condition, the first initial image expert model is selected as the first target image expert model, where the first condition indicates that the initial image expert model is in a non-converged state.

[0072] If the candidate image task has the same task attribute information as the first initial image expert model, the learning state of the first initial image expert model satisfies the second condition, and the first number of initial image expert models is less than the target threshold, then the second candidate image expert model is added as the target image expert model. The second condition indicates that the initial image expert model is in a convergent state, and the target threshold is the upper limit of the number of image expert models that can be accommodated in the dynamic hybrid expert module.

[0073] For example, state information can be a set of dynamic metrics reflecting the current learning progress and parameter stability of the initial image expert model. State information includes at least the learning state, which is used to determine whether the expert model has sufficiently learned the current task and achieved performance stability. For instance, the learning state can be evaluated by monitoring the average gradient magnitude of the expert model in recent training batches. When the average gradient magnitude consistently falls below a preset threshold, the expert is considered to be in a convergent state, corresponding to the second condition; conversely, when the average gradient magnitude remains at a high level and continues to fluctuate, the expert is considered to be in a non-converged state, corresponding to the first condition. The learning state can also be determined by evaluating the rate of change of the expert model's performance on the validation set. For example, if the decrease in validation loss is less than 0.1% over several consecutive training epochs, the expert is considered to have converged.

[0074] The first quantity can be the total number of initial image expert models already existing in the current dynamic hybrid expert module. The first quantity is a dynamically changing integer value that is updated in real time as new tasks are introduced and old experts are phased out.

[0075] The target threshold can be a pre-defined upper limit on the number of expert models based on computing resource constraints, inference efficiency requirements, or model design specifications. For example, the target threshold can be set to 8, indicating that the dynamic hybrid expert module can accommodate a maximum of 8 expert models.

[0076] In a comprehensive example, refer to Figure 3 Currently, there are four initial image expert models, with task attributes of "human pose estimation," "depth estimation," "edge detection," and "semantic segmentation," respectively, and a target threshold set to 8. The user inputs a new human pose skeleton image as the control image, with corresponding text features of "a ballet dancer, graceful posture, stage lighting." The system executes the following processing flow:

[0077] First, the pose skeleton diagram is input into the multi-condition control module, and the first feature is obtained after processing. Then, corresponding routing information is created based on this first feature. The router analyzes the first feature and identifies obvious human joint coordinate features and limb connection patterns, thus determining the candidate image task as "human pose estimation". Next, the task attribute information of all initial image expert models is retrieved. A first initial image expert model with the task attribute information of "human pose estimation" is found. Further querying of the state information of this first initial image expert model is performed. By statistically analyzing the average gradient magnitude of its most recent 50 training steps, the detection value is 0.015, which is significantly higher than the convergence threshold of 0.005, and the parameter update frequency reaches 78%. Based on this, it is determined that the expert's learning state meets the first condition, i.e., it is in a non-converged state and is still actively learning the detailed features of the pose task.

[0078] Since the candidate image task "human pose estimation" perfectly matches the task attribute information of the first initial image expert model, and the learning state of the expert model satisfies the first condition, the first initial image expert model is selected as the first target image expert model. This expert model is activated, and its internal low-rank adaptation matrix performs deep processing on the first feature, extracting joint angles, limb proportions, and spatial topological relationships from the pose skeleton map to generate targeted control features.

[0079] In another scenario, assuming that after continuous training, the average gradient magnitude of the first initial image expert model drops to 0.003 and the parameter update frequency drops to 4% after 500 training steps, and the maximum change in the validation loss over the last 10 epochs is only 0.08%, the system determines that its learning state has transitioned to satisfy the second condition, i.e., it has entered the convergence state. At this point, if a new human pose skeleton image is input again, and this image contains complex dance movements not fully covered in the previous training data, the routing information still identifies it as a "human pose estimation" task.

[0080] The system detects that the candidate image task has the same task attribute information as the first initial image expert model, but the expert's learning state has met the second condition, indicating that its learning of the existing pose patterns has become saturated. The system further checks the first quantity, which is currently 4, less than the target threshold of 8. Therefore, a second candidate image expert model is dynamically created and added as the new target image expert model. The parameters of this new expert are generated through random initialization or transfer initialization based on the first initial image expert model, and its task attribute information is also labeled as "human pose estimation", but its learning state is initialized to non-converged.

[0081] In this process, the first feature is processed by the self-attention module to generate the second feature, and then by the selected target image expert model in the dynamic hybrid expert module to generate the third feature. Based on the second and third features, the system determines the control features corresponding to each time step. These control features are fused with the Multi-Modal Diffusion Transformer (MMDIT) module in the image generation model through zero convolutional layers. At each time step of denoising, the noisy samples are gradually restored into clear images that conform to the structural constraints of the pose skeleton diagram. Finally, a high-quality target image of a dancer gracefully dancing in a specified pose is generated. The limb positions and joint angles are highly consistent with the input pose skeleton diagram, while the semantic requirements of "elegance" and "stage lighting" in the text features are fully reflected in terms of clothing details and lighting effects.

[0082] As described above, the operation involves determining at least one target image expert model from multiple initial image expert models based on the candidate image task, the task attribute information of the initial image expert model, and the state information of the initial image expert model. In some embodiments, this operation may further include the following steps: if the candidate image task has the same task attribute information as the first initial image expert model, the learning state of the first initial image expert model satisfies a second condition, and the first number of initial image expert models is greater than or equal to a target threshold, at least one of the following is performed: deleting the third initial image expert model from the dynamic hybrid expert module, adding the third candidate image expert model as the target image expert model, the target threshold being the upper limit of the number of image expert models that can be accommodated in the dynamic hybrid expert module, and the usage frequency of the third initial image expert model being less than the first threshold; resetting the parameters of the fourth initial image expert model to obtain the target image expert model, and the usage frequency of the fourth initial image expert model being less than the first threshold.

[0083] For example, usage frequency can be a measure of the proportion of times an initial image expert model has been selected and activated by the router to perform inference or training over a period of time. For instance, usage frequency can be calculated by counting the number of times an expert has been activated in the most recent 1000 training batches; if an expert is activated only 15 times, its usage frequency is 1.5%.

[0084] The first threshold can be a critical usage frequency value used to determine whether an expert model is a "low-activity" or "unpopular" expert. The first threshold is used to identify expert models that contribute very little to the current task distribution and have not been effectively utilized for a long time. For example, the first threshold can be set to 3%. The first threshold can also be dynamically adjusted according to the total number of experts. For example, when there are 8 experts, the first threshold is set to 5%; when there are 12 experts, to ensure the sensitivity of the elimination mechanism, the first threshold can be reduced to 2%.

[0085] The third initial image expert model can be the expert model whose usage frequency in the current expert pool is lower than the first threshold and whose overall score is the lowest. Deleting the third initial image expert model means completely removing it from the expert list of the dynamic hybrid expert module, releasing the parameter storage space and computing resources it occupies. For example, suppose the current expert pool contains 8 experts, and an expert with the task attribute information "line sketch processing" has only been activated 25 times in the last 2000 batches, with a usage frequency of 1.25%, far below the first threshold of 3%, and the task type corresponding to this expert has hardly appeared in recent user inputs. The system marks it as the third initial image expert model and performs the deletion operation.

[0086] The third candidate image expert model can be a newly created expert model adapted to the current candidate image task, used to replace the deleted third initial image expert model. The parameters of the third candidate image expert model are generated through random initialization or transfer initialization based on existing experts similar to the candidate task, and its task attribute information is set to the category label corresponding to the candidate image task.

[0087] The fourth initial image expert model can be an expert model whose usage frequency is below the first threshold, but whose structural position is retained by the system. Resetting the parameters of the fourth initial image expert model means restoring its internal low-rank adaptation matrix or other trainable parameters to their initial state, while retaining the expert's index position and routing connection relationship in the module.

[0088] In a comprehensive example, the system has currently reached the target threshold. The expert pool contains eight initial image expert models with the following task attributes and usage frequencies: "Human Pose Estimation" (usage frequency 45%, learning state converged), "Depth Estimation" (usage frequency 28%, learning state converged), "Edge Detection" (usage frequency 12%, learning state converged), "Semantic Segmentation" (usage frequency 8%, learning state converged), "Line Sketching" (usage frequency 1.5%, learning state converged), "Color Style Transfer" (usage frequency 2.2%, learning state converged), "Texture Detail Enhancement" (usage frequency 2.8%, learning state converged), and "Light and Shadow Distribution Estimation" (usage frequency 0.5%, learning state converged). The first threshold is set to 3%, and the target threshold is 8.

[0089] The user inputs a new, complex human skeletal image as the control image. This image depicts a challenging yoga pose, incorporating limb twisting angles and balance postures rarely seen in previous training data. The corresponding text feature for this input is "A yoga master performs a one-handed handstand on a cliff edge, with magnificent mountains and a sunrise in the background."

[0090] The system executes the following processing flow:

[0091] First, the pose skeleton diagram is input into the multi-condition control module, and the first feature is obtained after processing.

[0092] Subsequently, corresponding routing information is created based on this first feature. The router analyzes the human skeleton topology pattern and joint angle distribution in the first feature, identifies obvious posture control signals, and determines the candidate image task as "human pose estimation".

[0093] Next, the task attribute information of all initial image expert models was retrieved, and the first initial image expert model with the task attribute information of "human pose estimation" was found. Its usage frequency was 45%, and it had processed a large number of common pose samples in the past 1500 training steps. The average gradient magnitude had dropped to 0.002, the parameter update frequency was only 3%, and the verification loss had hardly changed in the last 15 cycles. The system determined that its learning state met the second condition, that is, it had fully converged to the existing pose pattern.

[0094] The system further checks the first quantity, which is currently 8, equal to the target threshold of 8, indicating that the expert pool is full and new experts cannot be added directly. At this point, the system needs to make room for new, complex pose learning requirements.

[0095] The system scanned the usage frequency of all initial image expert models and identified the following low-frequency experts:

[0096] "Line drawing processing" expert: Usage frequency is 1.5%, below the first threshold of 3%.

[0097] A color style transfer expert stated that the usage frequency is 2.2%, below the first threshold of 3%.

[0098] "Texture detail enhancement" expert: Usage frequency 2.8%, below the first threshold of 3%.

[0099] "Light and shadow distribution estimation" expert: Usage frequency 0.5%, below the first threshold of 3%.

[0100] The system further analyzed the task relevance of these low-frequency experts. It was found that the task attributes of the two experts, "line sketching" and "light and shadow distribution estimation," differed significantly from the current candidate image task, "human pose estimation," and they were not selected by the router at all in the most recent 500 batches.

[0101] Under the first processing strategy, the system marks the "light and shadow distribution estimation" expert with the lowest usage frequency as the third initial image expert model and performs a deletion operation. Subsequently, the system creates a third candidate image expert model. The task attribute information of this new expert is set to "human pose estimation". The parameters are initialized by copying the low-rank matrix of the first initial image expert model and adding a small random perturbation. The learning state is initialized to non-converged, and the usage frequency is initialized to 0. After this new expert is added to the expert pool, it is immediately selected by the router as one of the target image expert models, and works with the first initial image expert model to process the complex pose features of the current input.

[0102] Under the second processing strategy, the system chooses to preserve the stability of the expert pool's index structure and marks the "texture detail enhancement" expert, with a usage frequency of 2.8%, as the fourth initial image expert model. The system then resets the parameters of this expert: the weights of its internal low-rank adaptation matrices A and B are randomly re-initialized to values ​​following a normal distribution N(0, 0.01), the bias term is cleared, the task attribute information is updated from "texture detail enhancement" to "human pose estimation," the learning state is reset to non-converged, and the usage frequency is returned to zero. This reset fourth initial image expert model is selected by the router as the target image expert model, along with the first initial image expert model.

[0103] In practice, the router calculates the activation scores of all experts based on the first feature, generating an 8-dimensional vector R(c). The first initial image expert model (index 1) has a score of 0.72, the reset fourth initial image expert model (index 7) has a score of 0.68, and the scores of the remaining experts are all below 0.3. The TopK function selects the two experts with the highest scores (K=2), corresponding to indices 1 and 7. After Softmax normalization, the fusion weights of the two experts are 0.51 and 0.49, respectively.

[0104] The first initial image expert model performs stable feature encoding on the basic skeletal structure in the first feature, outputting intermediate features reflecting standard human proportions and common joint angle ranges. The reset fourth initial image expert model outputs intermediate features containing complex torsional angles and unconventional balance posture information from the first feature. The output features of the two experts are weighted and fused with weights of 0.51 and 0.49 to obtain the third feature. This third feature integrates stable historical posture knowledge with incremental learning results for novel postures, ensuring the accuracy of the basic skeletal structure while flexibly adapting to the special joint configurations of challenging yoga poses.

[0105] The third feature is further combined with the second feature output by the self-attention module to determine the control features corresponding to each time step. These control features are injected into the multimodal diffusion Transformer module of the image generation model through zero convolutional layers, guiding the gradual reconstruction of noisy samples at each denoising step. Ultimately, the system generates a high-quality target image.

[0106] As described above, the operation involves determining at least one target image expert model from multiple initial image expert models based on the candidate image task, the task attribute information of the initial image expert models, and the state information of the initial image expert models. In some embodiments, this operation may further include at least one of the following operations: if the candidate image task is different from the task attribute information of the multiple initial image expert models, the learning state of the multiple initial image expert models satisfies a second condition, and the first number of initial image expert models is less than a target threshold, a fourth candidate image expert model is added as the target image expert model; if the candidate image task is different from the task attribute information of the multiple initial image expert models, the learning state of the multiple initial image expert models satisfies a first condition, and the first number of initial image expert models is less than a target threshold, a fourth candidate image expert model is added as the target image expert model.

[0107] In one example, the system's current expert pool contains 5 initial image expert models, with the following task attributes and states: "Human Pose Estimation" (usage frequency 42%, learning state converged, average gradient 0.003), "Depth Estimation" (usage frequency 31%, learning state converged, average gradient 0.002), "Edge Detection" (usage frequency 15%, learning state converged, average gradient 0.004), "Semantic Segmentation" (usage frequency 9%, learning state converged, average gradient 0.003), and "Color Style Transfer" (usage frequency 3%, learning state converged, average gradient 0.002). The initial quantity is 5, the target threshold is set to 8, and the first threshold is set to 3%. The system has undergone approximately 5000 training batches, and the performance of all experts on their respective tasks has stabilized. The overall learning states of the multiple initial image expert models satisfy the second condition.

[0108] In a generation request, the user inputs a surface normal map of a building's exterior wall as a control image. This normal map uses RGB color encoding to represent the direction of the normal vector at each pixel on the wall, and is used to precisely control the lighting and reflection characteristics and three-dimensionality of the building's surface in the generated image. The corresponding text feature for this input is "a modern-style concrete building with a rough textured surface, casting deep shadows in the afternoon sunlight."

[0109] The system executes the following processing flow:

[0110] First, the normal map is input into the multi-condition control module. After processing, the first feature is obtained.

[0111] Subsequently, corresponding routing information is created based on this first feature. The router analyzes the channel distribution pattern of the first feature and detects that it contains a typical three-component correlation structure of the normal vector, that is, the RGB values ​​of adjacent pixels show a smooth gradient and the values ​​conform to the unit vector constraint, which is significantly different from the feature patterns of common pose skeleton maps, depth maps, or line drawings. The router further extracts the frequency domain characteristics of this feature and finds that its high-frequency components are mainly concentrated in the surface detail change area, which is significantly different from the smooth gradient characteristics of the depth map. Based on these analyses, the router determines the candidate image task as "surface normal estimation".

[0112] Next, the system retrieved the task attribute information of all initial image expert models and found that the candidate image task was different from the task attribute information of multiple initial image expert models.

[0113] The system further examines the learning status of multiple initial image expert models. Statistics show that the learning status of the five initial image expert models meets the second condition: all existing experts have fully converged on their respective tasks. The system detects a first quantity of 5, which is less than the target threshold of 8, indicating that the expert pool still has 3 vacant positions, sufficient to add new experts.

[0114] Since the candidate image task is different from the task attribute information of multiple initial image expert models, the learning state of multiple initial image expert models satisfies the second condition, and the first number is less than the target threshold, the system satisfies the first addition condition and executes the operation of adding a fourth candidate image expert model.

[0115] The system creates a fourth candidate image expert model, and the specific process is as follows: First, the system analyzes the feature similarity between the "surface normal estimation" task and existing tasks, and finds that although the "depth estimation" task has different objectives, it also involves the encoding of three-dimensional spatial geometric information, and the two have certain commonalities in the underlying visual features. Therefore, the system chooses to adopt a transfer initialization strategy, copying the weights of the low-rank adaptation matrix A of the "depth estimation" expert as the initial value of the A matrix of the fourth candidate image expert model. This matrix has already learned to extract the basic features of spatial structure from the input image. At the same time, the B matrix is ​​initialized by randomly sampling a normal distribution N(0, 0.015) to reserve parameter space for the specific learning of the new task. The task attribute information of the fourth candidate image expert model is set to "surface normal estimation", the learning state is marked as non-converged, the usage frequency is initialized to 0, and the index position in the expert pool is set to 6.

[0116] The router's weight projection matrix is ​​expanded from 5-dimensional to 6-dimensional accordingly, with the newly added 6th dimension weight generated through random initialization. The system recalculates the routing information and generates a 6-dimensional activation score vector R(c) based on the first feature. The fourth candidate image expert model (index 6) has an initial score of 0.35 because its A matrix, which is transferred from the "depth estimation" expert, has a certain responsiveness to spatial geometric features. Although this is lower than the score of 0.58 if the "depth estimation" expert itself is forcibly activated, it is significantly higher than other experts that are completely unrelated to the normal task (all scores are below 0.15).

[0117] The TopK function selects the K=2 experts with the highest scores: the "depth estimation" expert (index 2, score 0.58) and the fourth candidate image expert model (index 6, score 0.35). The fusion weights for the two are 0.64 and 0.36, respectively.

[0118] When processing the first feature, although the "depth estimation" expert's task attributes are not perfectly matched, its converged parameters can extract spatial depth cues implicit in the normal map. For example, changes in normal orientation often correspond to surface concavity and convexity, which are related to the depth gradient. This expert outputs a set of intermediate features containing prior spatial structure. The fourth candidate image expert model is in an active learning state that has not yet converged, with a learning rate set to 0.0001. Its randomly initialized B matrix explores and encodes specific information about the normals in the first feature (such as the correlation between XYZ components in RGB encoding). It quickly adjusts its parameters through gradient backpropagation and gradually learns to map the normal vector direction into a high-dimensional representation suitable for controlling the image generation process.

[0119] The outputs of the two experts are fused with weights of 0.64 and 0.36 to obtain the third feature. In the early training phase, this third feature mainly relies on the spatial structure foundation provided by the "depth estimation" expert. As the fourth candidate image expert model continues to learn, its contribution weight gradually increases, ultimately achieving specialized encoding of the normal task.

[0120] The third feature is combined with the second feature output by the self-attention module to determine the control features corresponding to each time step. These control features are injected into the image generation model through zero-convolutional layers to precisely control the lighting and reflection effects on the building surface during the denoising process. The final generated target image shows a modern concrete building whose rough texture on the exterior wall strictly follows the geometric details defined by the input normal map. The light and shadow changes of every bump and every texture groove are perfectly matched with the normal direction. Under the afternoon sunlight, the wall surface presents rich tonal gradations and a sense of three-dimensionality, and the viewer can clearly perceive the rough texture of the surface material and the solid volume of the building.

[0121] In another scenario, the system encountered new task expansion requirements based on the aforementioned five experts. At this point, although the "human pose estimation" expert was used frequently, the average gradient magnitude of this expert rebounded to 0.019 due to the recently introduced pose data containing a large number of novel dance movements and martial arts postures. The parameter update frequency reached 28%, and the learning state re-entered the non-convergent phase, thus satisfying the first condition. The other four experts remained in the convergent state. The system as a whole determined that the learning states of the multiple initial image expert models satisfied the first condition because there was at least one actively learning expert.

[0122] In this scenario, the user inputs a flat, unfolded image of a garment as the control image. This unfolded image uses lines and color blocks to mark the garment's cut lines, seam locations, and fabric sections, controlling the garment's style structure in the generated image. After analysis, the router determines the candidate image task as "garment style structure estimation," a task type that differs from the task attribute information of the five existing experts.

[0123] The system detects a first number of 5, which is less than the target threshold of 8. Since the candidate image task is different from the task attribute information of the multiple initial image expert models, the learning states of the multiple initial image expert models meet the first condition, and the first number is less than the target threshold. Therefore, the system meets the second addition condition and performs the operation of adding a fourth candidate image expert model.

[0124] The system creates a fourth candidate image expert model for the "clothing style and structure estimation" task. The expert's parameters are generated through random initialization, and the task attribute information is set to "clothing style and structure estimation," and it is added to expert pool index position 7. Although the "human pose estimation" expert among the existing experts is still actively learning, its learning objective focuses on the pose encoding of the human skeleton, which is unrelated to the style analysis of clothing structure. Therefore, the addition of a new expert will not interfere with the learning process of existing experts.

[0125] The router selects a new expert and an "edge detection" expert to jointly process the garment unfolding diagram because the style structure diagram contains a large amount of line boundary information. The two experts work together to extract the geometric layout of the cutting lines and the semantic information of the fabric partitions, generating control features to guide the image generation model to output a garment image that conforms to the specified style structure.

[0126] As described above, the operation involves determining at least one target image expert model from multiple initial image expert models based on the candidate image task, the task attribute information of the initial image expert model, and the state information of the initial image expert model. In some embodiments, this operation may further include at least one of the following operations: if the candidate image task is different from the task attribute information of the multiple initial image expert models, and the first number of initial image expert models is greater than a target threshold, the third initial image expert model is deleted from the dynamic hybrid expert module, and a fifth candidate image expert model is added as the target image expert model, wherein the usage frequency of the fifth initial image expert model is less than the first threshold; if the candidate image task is different from the task attribute information of the multiple initial image expert models, and the first number of initial image expert models is greater than the target threshold, the parameters of the third initial image expert model are reset to obtain the target image expert model, wherein the usage frequency of the third initial image expert model is less than the first threshold.

[0127] In one example, the current expert pool has reached the target threshold limit and currently contains 8 initial image expert models. Their task attributes and usage frequencies are as follows: "Human pose estimation" (usage frequency 38%), "Depth estimation" (usage frequency 29%), "Edge detection" (usage frequency 14%), "Semantic segmentation" (usage frequency 10%), "Surface normal estimation" (usage frequency 6%, this expert was newly added in the previous embodiment), "Clothing style structure estimation" (usage frequency 2.5%), "Color style transfer" (usage frequency 0.8%), and "Light and shadow distribution estimation" (usage frequency 0.3%). The first quantity is 8, which is equal to the target threshold of 8, and the first threshold is set to 3%.

[0128] The user inputs a 3D CAD wireframe of an industrial part as the control image. The corresponding text feature for this input is "a precision-machined metal gear with a frosted surface, reflecting a cool metallic luster under the spotlight on the workbench."

[0129] The system executes the following processing flow:

[0130] First, the CAD wireframe is input into the multi-condition control module. After processing, the first feature is obtained. Then, corresponding routing information is created based on this first feature. The task attribute information of the eight initial image expert models is different from that of the CAD wireframe.

[0131] Obtain the usage frequency of all initial image expert models and identify the following three low-frequency experts:

[0132] "Clothing style structure estimation" expert: Usage frequency is 2.5%, below the first threshold of 3%.

[0133] Color style transfer expert: Usage frequency is 0.8%, below the first threshold of 3%.

[0134] "Light and shadow distribution estimation" expert: Usage frequency 0.3%, below the first threshold of 3%.

[0135] The system further analyzed the historical contributions and task evolution trends of these low-frequency experts. It was found that the "Light and Shadow Distribution Estimation" expert was activated only 6 times in the most recent 2000 training batches, with a usage frequency of 0.3%, the lowest among all experts.

[0136] Under the first processing strategy, the system performs the deletion operation. The specific process is as follows:

[0137] First, the "Light and Shadow Distribution Estimation" expert is removed. A fifth candidate image expert model is created, which is used as the target image expert model for the "Industrial CAD Geometry Analysis" task.

[0138] Under the second processing strategy, the system chooses to perform a parameter reset operation instead of deletion. The specific scenario is as follows:

[0139] Instead of deleting the "Light and Shadow Distribution Estimation" expert, reset its parameters. Use the reset "Light and Shadow Distribution Estimation" expert as the expert model for the target image.

[0140] It should be noted that, for the above multiple execution strategies, if multiple initial image expert models do not meet the conditions, different execution strategies can be adopted for multiple initial image expert models.

[0141] As described above, in some embodiments, the image generation method of this embodiment may further include at least one of the following operations: if the usage frequency of the third initial image expert model is greater than the second threshold, and the third initial image expert model is used to process the target image task, the parameters of the second initial image expert model are fixed so that the parameters of the second initial image expert model are not trainable; if the usage frequency of the fourth initial image expert model is greater than the second threshold, and the fourth initial image expert model is used to process the target image task, the learning rate of the second initial image expert model is always less than the third threshold.

[0142] For example, the second threshold can be a critical value used to determine whether an image expert model is a frequently used expert. For instance, the second threshold can be set to 60%, meaning that when the usage frequency of an image expert model exceeds 60%, it is determined to be a frequently used expert. This disclosure does not limit this approach.

[0143] Fixing parameters can be done by setting the weight matrix of the image expert model to an untrainable state, so that it no longer accepts gradient updates during subsequent training.

[0144] The learning rate can be a hyperparameter that controls the step size for updating the parameters of an image expert model.

[0145] The third threshold can be an upper limit on the learning rate used to limit the update speed of image expert model parameters. For example, the third threshold can be set to 0.00005, so that even if the original learning rate of an image expert model is 0.0001, once the protection mechanism is triggered, its learning rate will be forcibly reduced to no more than 0.00005.

[0146] For example, the dynamic hybrid expert module includes an image expert model specifically for processing pose skeleton maps, denoted as expert E1. In the 5000 training iterations of this phase, expert E1 was selected by the router 4200 times, a usage frequency of 84%, exceeding the preset second threshold of 60%, and expert E1 was explicitly used for the target image task of processing human pose. At this point, the system recognized expert E1 as the core expert for this task and initiated a parameter protection mechanism. The system fixed all parameters of expert E1, disabling the gradient calculation switches for its low-rank matrices A and B, ensuring that its weights remained unchanged when training new tasks. Subsequently, the system entered the second phase, beginning to learn the indoor scene depth control task. Although the new task introduced new training data and gradient signals, the parameters of expert E1 remained at the state at the end of the first phase of training, thus ensuring that the model maintained its previous high-precision performance when generating human pose-related images.

[0147] In some embodiments, inputting the second data into the multi-condition control module to obtain the first feature includes: normalizing the latent space features to obtain normalized features; determining the modulation parameters corresponding to each time step based on the text features and time steps; using the modulation parameters to perform channel-by-channel modulation on the normalized features to obtain modulation features; and linearly fusing the modulation features to obtain the first feature.

[0148] For example, the inputs to the multi-condition control module include latent space features, text features, and time step information. The latent space features are first normalized, and the normalized features then enter the linear transformation module for feature space mapping using a learnable weight matrix. Next, the module generates modulation parameters based on the text features and time step information, performing channel-by-channel modulation on the normalized features. The text features and time step information are input from external sources to the modulation parameter generation unit. The modulated features undergo a final linear fusion step, integrating multi-channel information to obtain a unified first feature representation.

[0149] Figure 4 A schematic diagram illustrating the principle of an image generation method according to an embodiment of the present disclosure is shown. Figure 5 The illustration shows a schematic diagram of the effect of the image generation method according to an embodiment of the present disclosure.

[0150] Reference Figure 4 and Figure 5 The system comprises three core functional modules: a multi-condition control module, a dynamic hybrid expert module, and a frozen image generation model.

[0151] A lightweight multi-condition control module is used in parallel with the MMDIT module of the target processing model, and conditional control information is injected. First, all parameters Θ in the pre-trained MMDIT module are frozen. Then, the multi-condition control module is initialized by copying a portion of the module parameters from the image denoising path in the MMDIT module. The parameters of this module are denoted as follows. This module is denoted as The multi-condition control module's processing flow is as follows: it accepts latent space features, text features, and time step information as input. Following the processing approach of SD3, this module adopts a basic strategy of dual-path feature processing and attention fusion: the text and time step fusion feature path is activated using the SiLU function and then passes through a linear layer to obtain two modulation coefficients; the control image feature path first undergoes feature normalization and is then linearly modulated with the modulation coefficients of the other path to incorporate text and time information. Subsequently, the fused features pass through a self-attention module to obtain the final external conditional vector output. After a certain scale of initial pre-training of this module, the model possesses preliminary multi-condition control capabilities.

[0152] The dynamic hybrid expert module consists of several experts and routers. Based on task requirements, the experts and routers in the dynamic hybrid expert module are adaptively expanded: for a new task Tt, the model dynamically creates a task-specific router Rt and uses router Rt when processing this task later; simultaneously, the model can detect the weight status of the current expert set, and if most expert weights are frozen or the learning rate is too low, the model will dynamically add new experts to adapt to subsequent tasks. This design allows the model to have fewer parameters initially and can adapt to an unknown number of new tasks without manually specifying the model size, which is more in line with the requirements of real-world applications. A low-rank matrix LoRA adapter module is used to act as an expert in the dynamic hybrid expert module setting, improving the adaptation speed during training. In each control block, the linearly fused control image features c are used as input, and the model selects a task-specific router. Give the selection scores of all experts. The results are further processed into weights W that combine the expert calculations. Specifically, the gating weights are calculated as follows: where Rt projects c onto a 1-NE vector, representing the selection score for each expert activation. The function selects the k most relevant experts, and sets the rest to... . The function normalizes these weights to emphasize the contributions of the selected experts.

[0153]

[0154] Subsequently, based on the expert weights W, the model activates the selected Top K experts to process this batch of tasks. We use LoRA as the expert in the dynamic hybrid expert module, introducing two low-rank matrices A and B, and efficiently fine-tuning the model through low-rank decomposition. Finally, the combined output of this module... Calculated as:

[0155]

[0156] Finally, the multi-condition control module is connected to the frozen pre-trained basic modules through a zero convolutional layer, denoted as... It is a 1x1 convolutional layer with its weights and biases initialized to zero, and its parameters are denoted as... .

[0157] In the initial stages of continuous learning, the dynamic hybrid expert module contains only a small number of LoRA experts. When a new control task arrives, the system first evaluates the current learning status of each expert, including their average gradient magnitude, parameter update frequency, and route activation frequency. If it detects that the learning progress of most experts is stagnating, it determines that the current expert capacity is close to saturation and triggers the expert addition mechanism. At this time, the system dynamically adds several LoRA experts. The parameters of the new experts are initialized randomly or based on migration from similar tasks, and their corresponding route weights are initially set to a uniform distribution to ensure that the new task has sufficient learning opportunities.

[0158] During subsequent continuous training, the system tracks the usage frequency and conditional affiliation of each expert online. For experts that are frequently activated and stably serve specific historical tasks, parameter freezing or learning rate penalty strategies are employed to protect their learned knowledge and prevent catastrophic forgetting due to interference from new tasks. During continuous learning, experts with frozen or reduced learning rates will not only be used for the current task but will also remain accessible in subsequent new task learning. When faced with a new task, the router can select frozen or low-learning-rate experts to utilize knowledge from historical tasks and optimize unfrozen experts to acquire task-specific information. In this way, the model allows experts to collaboratively consolidate their knowledge, achieving knowledge sharing between continuous learning tasks, similar to the human brain's mechanism of reinforcing and connecting new information with existing memories.

[0159] Meanwhile, to prevent the model from expanding indefinitely, an upper limit is set on the total number of experts. When the upper limit is reached and new experts need to be introduced, the system will prioritize eliminating "unpopular" experts with the lowest usage frequency, the weakest gradient response, or those not selected in multiple tasks, resetting their parameters and reactivating them for learning new tasks, thereby achieving a dynamic balance between the recycling of expert resources and model capacity.

[0160] In general, the fusion process between the multi-condition control module and the base model MMDIT is as follows:

[0161] First, the parameter-frozen MMDIT module has input noise features x and text features y. After MMDIT processing, new noise features x1 and text features y1 are formed. Let... This represents a pre-trained MMDIT module with the following parameters: The processing procedure can be formalized as follows:

[0162]

[0163] In parallel processing, the control condition feature c, noise feature x, and text feature y are processed by the multi-condition control module DyCE-block to obtain the control network output. Module parameters are denoted as .

[0164]

[0165] Finally, the multi-condition control module outputs... By using a zero-convolutional layer initialized to zero, the noise feature x1 in the image generation primitive is fused to ensure that the original generation performance is not affected during the initial training phase. The formula is as follows:

[0166]

[0167] Understandably, a unified control architecture based on dynamically scalable, lightweight hybrid expert modules enables efficient modeling and sparse activation for various control conditions. This framework allows for the flexible insertion of new experts to adapt to new tasks without retraining the base model, thus enabling continuous evolution of model capabilities. Compared to traditional "one condition, one model" or LoRA fine-tuning approaches, this method significantly reduces storage and inference overhead while maintaining high control accuracy. During continuous learning, this method identifies key expert modules belonging to historical tasks by online statistical analysis of their usage frequency and task affiliation. These modules are then subjected to parameter freezing or learning rate decay during subsequent training, effectively protecting existing knowledge and significantly mitigating catastrophic forgetting.

[0168] The specific advantages are as follows:

[0169] 1. It supports a dynamically expandable hybrid expert control architecture, breaking through the limitations of the traditional static structure of MoE. It designs a modular control network that can incrementally add experts, enabling plug-and-play adaptation to new tasks, effectively supporting continuous learning under open sets, and improving the long-term evolution capability of the model while avoiding catastrophic forgetting.

[0170] 2. Through a dynamic adjustment mechanism, it effectively responds to the ever-changing and diverse task requirements of users, achieving unified control over multiple control conditions. Compared to the traditional single-condition ControlNet combined with LoRA fine-tuning method, it significantly reduces the number of additional parameters by approximately 40%.

[0171] This method features a low number of control module activation parameters, enabling personal terminal devices with limited GPU memory to fully utilize the latest image generation models for refined image generation and control. This design significantly improves the device compatibility and deployment flexibility of advanced generative AI technologies, providing a feasible solution for achieving high-quality, controllable image generation in resource-constrained environments.

[0172] Furthermore, this method, building upon the general multi-condition control capabilities acquired through pre-training, further achieves efficient incremental learning for new tasks through a dynamic hybrid expert module. This approach not only preserves the original model's stable response capabilities to existing control types but also possesses scalability for open environments. The sparse activation characteristics of the dynamic hybrid expert module ensure inference efficiency, while the dynamic addition, removal, and protection mechanisms of experts effectively balance model capacity, knowledge retention, and zero-shot transfer capability, making it particularly suitable for real-world application scenarios where user needs continuously evolve.

[0173] Based on the above image generation method, this disclosure also provides an image generation apparatus. The following will be combined with... Figure 6 The device is described in detail.

[0174] Figure 6 A schematic block diagram of an image generation apparatus according to an embodiment of the present disclosure is shown.

[0175] like Figure 6 As shown, the image generation apparatus 300 of this embodiment includes an acquisition module 310 and an image generation module 320.

[0176] The acquisition module 310 is used to acquire control features, which are obtained by the dynamic hybrid expert module through different image task processing based on the control image. The control image is a structural description image corresponding to the text features. In one embodiment, the acquisition module 310 can be used to perform the operation S210 described above, which will not be repeated here.

[0177] The image generation module 320 is configured to independently execute or invoke the image generation model to perform the following operations: inputting first data and control features into the image generation model to obtain a target image. The first data includes text features, multiple time steps, and noise samples. The control features are used to guide the noise samples to be restored to an image corresponding to the structure described by the control image. In one embodiment, the image generation module 320 can be used to perform the operation S220 described above, which will not be repeated here.

[0178] According to embodiments of this disclosure, any plurality of modules in the acquisition module 310 and the image generation module 320 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the acquisition module 310 and the image generation module 320 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the acquisition module 310 and the image generation module 320 may be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

Claims

1. An image generation method, comprising: The control features are obtained by the dynamic hybrid expert module performing different image task processing on the control image, which is a structural description image corresponding to the text features. The first data and the control features are input into the image generation model to obtain the target image. The first data includes the text features, multiple time steps, and noise samples. The control features are used to guide the noise samples to be restored to an image corresponding to the structure described by the control image.

2. The method according to claim 1, wherein obtaining the control feature comprises: The second data is input into the multi-condition control module to obtain the first feature. The second data includes latent space features, text features and time steps. The latent space features are the encoding features of the structure described by the control image. The first feature is a fusion representation of the latent space features and text features of the control image. The first feature is input into the self-attention module to obtain the second feature; The first feature is input into the dynamic hybrid expert module to obtain the third feature. Based on the second feature and the third feature, the control features corresponding to each time step are determined.

3. The method according to claim 1 or 2, wherein the dynamic hybrid expert module comprises a plurality of initial image expert models, the initial image expert models being used to enhance different types of features of the control image, and the method further comprises: Based on the first feature, create routing information corresponding to the first feature; Using the routing information, at least one target image expert model corresponding to the first feature is determined; The first feature is processed using the at least one target image expert model to obtain the second feature.

4. The method according to claim 3, wherein the routing information is used to determine at least one target image expert model corresponding to the first feature, comprising: Using the routing information, candidate image tasks corresponding to the first feature are determined; Based on the candidate image task, the task attribute information of the initial image expert model, and the state information of the initial image expert model, at least one target image expert model is determined from the plurality of initial image expert models.

5. The method according to claim 4, wherein at least one target image expert model is determined from the plurality of initial image expert models based on the candidate image task, the task attribute information of the initial image expert model, and the state information of the initial image expert model, comprising at least one of the following: If the candidate image task has the same task attribute information as the first initial image expert model, and the learning state of the first initial image expert model satisfies the first condition, the first initial image expert model is used as the first target image expert model. The first condition indicates that the initial image expert model is in a non-converged state. If the candidate image task has the same task attribute information as the first initial image expert model, the learning state of the first initial image expert model satisfies the second condition, and the first number of the initial image expert models is less than the target threshold, then a second candidate image expert model is added as the target image expert model. The second condition indicates that the initial image expert model is in a convergent state, and the target threshold is the upper limit of the number of image expert models that can be accommodated in the dynamic hybrid expert module.

6. The method according to claim 4, wherein at least one target image expert model is determined from the plurality of initial image expert models based on the candidate image task, the task attribute information of the initial image expert model, and the state information of the initial image expert model, comprising: If the candidate image task has the same task attribute information as the first initial image expert model, the learning state of the first initial image expert model satisfies the second condition, and the first number of the initial image expert model is greater than or equal to the target threshold, at least one of the following shall be executed: The third initial image expert model is removed from the dynamic hybrid expert module, and a third candidate image expert model is added as the target image expert model. The target threshold is the upper limit of the number of image expert models that can be accommodated in the dynamic hybrid expert module. The usage frequency of the third initial image expert model is less than the first threshold. The parameters of the fourth initial image expert model are reset to obtain the target image expert model, wherein the usage frequency of the fourth initial image expert model is less than the first threshold.

7. The method according to claim 4, wherein at least one target image expert model is determined from the plurality of initial image expert models based on the candidate image task, the task attribute information of the initial image expert model, and the state information of the initial image expert model, comprising at least one of the following: If the candidate image task is different from the task attribute information of the multiple initial image expert models, the learning state of the multiple initial image expert models satisfies the second condition, and the first number of the initial image expert models is less than the target threshold, then a fourth candidate image expert model is added as the target image expert model. If the candidate image task is different from the task attribute information of the multiple initial image expert models, the learning state of the multiple initial image expert models satisfies the first condition, and the first number of the initial image expert models is less than the target threshold, then a fourth candidate image expert model is added as the target image expert model.

8. The method according to claim 4, wherein at least one target image expert model is determined from the plurality of initial image expert models based on the candidate image task, the task attribute information of the initial image expert model, and the state information of the initial image expert model, comprising at least one of the following: If the candidate image task is different from the task attribute information of the multiple initial image expert models, and the first number of the initial image expert models is greater than the target threshold, the third initial image expert model is deleted from the dynamic hybrid expert module, and a fifth candidate image expert model is added as the target image expert model. The usage frequency of the fifth initial image expert model is less than the first threshold. If the candidate image task is different from the task attribute information of the multiple initial image expert models, and the first number of the initial image expert models is greater than the target threshold, the parameters of the third initial image expert model are reset to obtain the target image expert model, wherein the usage frequency of the third initial image expert model is less than the first threshold.

9. The method of claim 5, further comprising at least one of the following: If the frequency of use of the third initial image expert model is greater than the second threshold, and the third initial image expert model is used to process the target image task, the parameters of the second initial image expert model are fixed so that the parameters of the second initial image expert model are not trainable. If the frequency of use of the fourth initial image expert model is greater than the second threshold, and the fourth initial image expert model is used to process the target image task, the learning rate of the second initial image expert model is always less than the third threshold.

10. An image generation apparatus, comprising: The acquisition module is used to acquire control features, which are obtained by the dynamic hybrid expert module through different image task processing based on the control image. The control image is a structural description image corresponding to the text features. The image generation module is capable of independently executing or calling the image generation model to perform the following operations: inputting the first data and the control features into the image generation model to obtain the target image. The first data includes the text features, multiple time steps, and noise samples. The control features are used to guide the noise samples to be restored to an image corresponding to the structure described by the control image.

Images