Image generation method, device and storage medium

Abstract

The application provides an image generation method, device and storage medium, which separates the optimization of image structure extraction and style rendering by embedding a low-rank adaptive module and a mixed expert adaptive module, so as to improve the image generation quality. In the method, a to-be-processed image and style description information are input into a trained image generation model, the model including a first layer embedding a low-rank adaptive module and a second layer embedding a mixed expert adaptive module; feature recognition is performed on the to-be-processed image and the style description information to obtain initial features of the image and text; feature correction is performed on the initial features of the image and text based on the low-rank adaptive module to obtain structure features of the image and text; feature extraction is performed on the structure features of the image and text based on the mixed expert adaptive module to obtain style rendering features; and feature fusion is performed on the style rendering features and the structure features of the image and text to generate a target image meeting the style description information.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image generation method, device and storage medium. Background Technology

[0002] With the rapid development of generative artificial intelligence technology, image generation models based on the Transformer architecture have become the core support for image synthesis and customized artistic creation. In multi-style image generation tasks, it is usually necessary to introduce efficient parameter fine-tuning techniques on the basis of pre-trained models so that a single model can adapt to different artistic styles and generate target images that meet the description of a specific style.

[0003] In related technologies, low-rank adaptation (LoRA) modules are often used to fine-tune each image style independently. This is mainly achieved by training a set of dedicated low-rank weight matrices for each style and attaching the low-rank adaptation module to the pre-trained model in a plug-and-play manner, so as to realize the rapid transfer of image rendering styles.

[0004] However, in multi-style application scenarios, to achieve switching or fusion between different styles, it is necessary to repeatedly perform dynamic loading and unloading of the LoRA modules corresponding to each style. The resulting I / O scheduling latency and memory bandwidth overhead are difficult to meet real-time requirements. Moreover, since each LoRA module is independently trained in the same layer of the pre-trained model, there is a lack of collaborative optimization and effective decoupling between the adaptation parameters of different styles. This leads to mutual interference between the general preservation of structural features and the differentiated expression of style features, thus affecting the quality of image generation. Summary of the Invention

[0005] This application provides an image generation method, device, and storage medium. By embedding a low-rank adaptation module and a hybrid expert adaptation module in a layered manner, the separation and optimization of image structure extraction and style rendering are achieved to improve the image generation quality.

[0006] In a first aspect, embodiments of this application provide an image generation method, comprising: inputting an image to be processed and style description information into a trained image generation model; wherein the image generation model includes a first layer embedding a low-rank adaptation module and a second layer embedding a hybrid expert adaptation module; performing feature recognition on the image to be processed and the style description information to obtain initial image-text features; performing feature correction on the initial image-text features based on the low-rank adaptation module to obtain image-text structural features; performing feature extraction on the image-text structural features based on the hybrid expert adaptation module to obtain style rendering features; and performing feature fusion on the style rendering features and the image-text structural features to generate a target image that satisfies the style description information.

[0007] In some embodiments, the hybrid expert adaptation module includes multiple expert branches and a gate-controlled routing network; the step of extracting features from the graphic structure features based on the hybrid expert adaptation module to obtain style rendering features includes: performing feature recognition on the graphic structure features based on the gate-controlled routing network to obtain the style adaptation degree corresponding to each expert branch; obtaining a target expert branch that meets the style adaptation conditions from the expert branches based on the style adaptation degree; and extracting features from the graphic structure features based on the target expert branch to obtain the style rendering features.

[0008] In some embodiments, obtaining target expert branches that meet the style fit conditions from the expert branches based on the style fit degree includes: sorting the expert branches sequentially in descending order of style fit degree, determining the first-order difference result of the style fit degree corresponding to each pair of adjacent expert branches; if all first-order difference results are lower than the difference threshold, then all expert branches are taken as target expert branches; if at least one first-order difference result is not lower than the difference threshold, then a specified number of expert branches with the highest style fit degree are selected from the expert branches as target expert branches.

[0009] In some embodiments, the image generation model is obtained by two-stage fine-tuning of the basic image generation model based on style image samples; the style image samples include a first image sample and a second image sample; in the first stage of fine-tuning, the first image sample is input into the basic image generation model to obtain a first reference image, and a first fine-tuning loss is determined based on the first reference image and the sample labels corresponding to the first image sample, and the matrix parameters of the low-rank adaptation module are adjusted using the first fine-tuning loss; in the second stage of fine-tuning, the second image sample is input into the basic image generation model to obtain a second reference image, and a second fine-tuning loss is determined based on the second reference image and the sample labels corresponding to the second image sample, and the matrix parameters of the hybrid expert adaptation module are adjusted using the second fine-tuning loss.

[0010] In some embodiments, determining the first fine-tuning loss based on the first reference image and the sample labels corresponding to the first image samples includes: extracting features from the first image samples to obtain initial sample features; performing feature correction on the initial sample features based on the low-rank adaptation module to determine sample structural features; determining a basis correction loss based on the initial sample features and the sample structural features; determining a first flow matching loss based on the image difference between the first reference image and the sample labels; and determining the first fine-tuning loss based on the first flow matching loss and the basis correction loss.

[0011] In some embodiments, determining the second fine-tuning loss based on the sample labels corresponding to the second reference image and the second image samples includes: constructing an orthogonal projection matrix of each expert branch in the feature space based on the low-rank dimensionality reduction matrix associated with each expert branch in the hybrid expert adaptation module; performing regularization processing on the orthogonal projection matrix of each expert branch to obtain an orthogonal regularization loss; determining a second flow matching loss based on the image difference between the second reference image and the sample labels; and determining the second fine-tuning loss based on the second flow matching loss and the orthogonal regularization loss.

[0012] In some embodiments, the regularization process of the orthogonal projection matrices of each expert branch to obtain the orthogonal regularization loss includes: for every two expert branches, determining the matrix product of the orthogonal projection matrices of the two expert branches; and, based on the cyclic shift invariance of the matrix trace and the symmetric idempotency of the orthogonal projection matrix, transforming the Frobenius norm calculation of the matrix product into a trace function constructed from the low-rank reduced-dimensional matrices corresponding to the two expert branches; performing trace operation on the trace functions corresponding to each two expert branches to obtain the orthogonal loss corresponding to each two expert branches; and accumulating the orthogonal losses corresponding to each two expert branches to obtain the orthogonal regularization loss.

[0013] In some embodiments, the low-rank adaptation module and the hybrid expert adaptation module are embedded in the base image generation model before the two-stage fine-tuning is performed; during the first-stage fine-tuning, the matrix parameters of the hybrid expert adaptation module are frozen; during the second-stage fine-tuning, the matrix parameters of the low-rank adaptation module are frozen.

[0014] Secondly, embodiments of this application provide an image generation apparatus, comprising: an information acquisition unit configured to: input an image to be processed and style description information into a trained image generation model; the image generation model includes a first layer embedded with a low-rank adaptation module and a second layer embedded with a hybrid expert adaptation module; a feature recognition unit configured to: perform feature recognition on the image to be processed and the style description information to obtain initial image-text features; a feature correction unit configured to: perform feature correction on the initial image-text features based on the low-rank adaptation module to obtain image-text structural features; a style rendering unit configured to: extract features from the image-text structural features based on the hybrid expert adaptation module to obtain style rendering features; and an image generation unit configured to: perform feature fusion on the style rendering features and the image-text structural features to generate a target image that satisfies the style description information.

[0015] In some embodiments, the hybrid expert adaptation module includes multiple expert branches and a gate-controlled routing network; the hybrid expert adaptation module is used to extract features from the text-image structure features to obtain style rendering features. Specifically, the style rendering unit is configured to: perform feature recognition on the text-image structure features based on the gate-controlled routing network to obtain the style adaptation degree corresponding to each expert branch; based on the style adaptation degree, obtain a target expert branch that meets the style adaptation conditions from the expert branches; and extract features from the text-image structure features based on the target expert branch to obtain the style rendering features.

[0016] In some embodiments, the process of obtaining target expert branches that meet the style adaptation conditions from the expert branches based on the style adaptation degree is performed. Specifically, the style rendering unit is configured to: sort the expert branches in descending order of style adaptation degree, determine the first-order difference result of the style adaptation degree corresponding to each pair of adjacent expert branches; if all first-order difference results are lower than the difference threshold, then all expert branches are taken as target expert branches; if at least one first-order difference result is not lower than the difference threshold, then a specified number of expert branches with the highest style adaptation degree are selected from the expert branches as target expert branches.

[0017] In some embodiments, the image generation model is obtained by two-stage fine-tuning of the basic image generation model based on style image samples; the style image samples include a first image sample and a second image sample; in the first stage of fine-tuning, the first image sample is input into the basic image generation model to obtain a first reference image, and a first fine-tuning loss is determined based on the first reference image and the sample labels corresponding to the first image sample, and the matrix parameters of the low-rank adaptation module are adjusted using the first fine-tuning loss; in the second stage of fine-tuning, the second image sample is input into the basic image generation model to obtain a second reference image, and a second fine-tuning loss is determined based on the second reference image and the sample labels corresponding to the second image sample, and the matrix parameters of the hybrid expert adaptation module are adjusted using the second fine-tuning loss.

[0018] In some embodiments, the first fine-tuning loss is determined by: extracting features from the first image sample to obtain initial sample features; performing feature correction on the initial sample features based on the low-rank adaptation module to determine sample structural features; determining a basis correction loss based on the initial sample features and the sample structural features; determining a first flow matching loss based on the image difference between the first reference image and the sample label; and determining the first fine-tuning loss based on the first flow matching loss and the basis correction loss.

[0019] In some embodiments, the second fine-tuning loss is determined by: constructing an orthogonal projection matrix of each expert branch in the feature space based on the low-rank dimensionality reduction matrix associated with each expert branch in the hybrid expert adaptation module; performing regularization processing on the orthogonal projection matrix of each expert branch to obtain an orthogonal regularization loss; determining a second flow matching loss based on the image difference between the second reference image and the sample label; and determining the second fine-tuning loss based on the second flow matching loss and the orthogonal regularization loss.

[0020] In some embodiments, the orthogonal regularization loss is determined by: for every two expert branches, determining the matrix product of the orthogonal projection matrices of the two expert branches; and, based on the cyclic shift invariance of the matrix trace and the symmetric idempotency of the orthogonal projection matrix, transforming the Frobenius norm calculation of the matrix product into a trace function constructed from the low-rank reduced-dimensional matrices corresponding to the two expert branches; performing trace operation on the trace functions corresponding to each two expert branches to obtain the orthogonal loss corresponding to each two expert branches; and summing the orthogonal losses corresponding to each two expert branches to obtain the orthogonal regularization loss.

[0021] In some embodiments, the low-rank adaptation module and the hybrid expert adaptation module are embedded in the base image generation model before the two-stage fine-tuning is performed; during the first-stage fine-tuning, the matrix parameters of the hybrid expert adaptation module are frozen; during the second-stage fine-tuning, the matrix parameters of the low-rank adaptation module are frozen.

[0022] Thirdly, embodiments of this application provide an electronic device, including: a memory for storing program instructions; and a processor for calling the program instructions stored in the memory and executing the steps included in the method described in any one of the first aspects according to the obtained program instructions.

[0023] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method described in any one of the first aspects.

[0024] Fifthly, embodiments of this application provide a computer program product comprising: computer program code, which, when executed on a computer, causes the computer to perform the method described in any of the first aspects.

[0025] In this embodiment, the image to be processed and style description information are input into a trained image generation model. The model includes a first layer with an embedded low-rank adaptation module and a second layer with an embedded hybrid expert adaptation module. Feature recognition is performed on the image to be processed and the style description information to obtain initial image-text features. Based on the low-rank adaptation module, feature correction is performed on the initial image-text features to obtain image-text structural features. Based on the hybrid expert adaptation module, feature extraction is performed on the image-text structural features to obtain style rendering features. Feature fusion is performed on the style rendering features and the image-text structural features to generate a target image that meets the style description information. The above process configures the image generation model as a heterogeneous architecture containing a first layer with an embedded low-rank adaptation module and a second layer with an embedded hybrid expert adaptation module. This allows the first layer and the low-rank adaptation module to collaboratively extract and correct cross-style shared image-text structural features. The hybrid expert adaptation module in the second layer then performs style-specific feature extraction based on these image-text structural features. This physically separates the propagation path of general structural features from the multi-style rendering path, constraining gradient regression of different styles to their respective independent adaptation paths. This avoids feature crosstalk and gradient pulling, thereby achieving separation and optimization of image structure extraction and style rendering, and ultimately improving the quality of generated images. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. The drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A block diagram of the model structure of the image generation model provided in the embodiments of this application.

[0028] Figure 2 This is a structural block diagram of the hybrid expert adaptation module provided in the embodiments of this application.

[0029] Figure 3 This is an overall flowchart of an image generation method provided in an embodiment of this application.

[0030] Figure 4 This is a structural block diagram of an image generation apparatus provided in an embodiment of this application.

[0031] Figure 5 A structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] The technical solutions in the embodiments of this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " will mean "or", for example, A / B can mean A or B; "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0033] In the description of the embodiments of this application, unless otherwise stated, the term "multiple" refers to two or more, and other quantifiers are similarly understood. The preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0034] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on conventional or non-inventive effort. For steps that do not logically have a necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application. In actual processing or when the control device executes the method, it may be executed sequentially or in parallel according to the method shown in the embodiments or drawings.

[0035] As mentioned earlier, many related technologies use the LoRA module to fine-tune each image style independently. This is mainly achieved by training a set of dedicated low-rank weight matrices for each style and attaching the low-rank adaptation module to the pre-trained model in a plug-and-play manner, so as to realize the rapid transfer of image rendering styles.

[0036] However, in multi-style application scenarios, to achieve switching or fusion between different styles, it is necessary to repeatedly perform dynamic loading and unloading of the LoRA modules corresponding to each style. The resulting I / O scheduling latency and memory bandwidth overhead are difficult to meet real-time requirements. Moreover, since each LoRA module is independently trained in the same layer of the pre-trained model, there is a lack of collaborative optimization and effective decoupling between the adaptation parameters of different styles. This leads to mutual interference between the general preservation of structural features and the differentiated expression of style features, thus affecting the quality of image generation.

[0037] While some studies have proposed using a Mixture of Experts architecture (MoE-Lora) to eliminate I / O latency for parameter swapping by embedding multiple homogeneous MoE-Lora modules across all network layers, this indiscriminate deployment ignores the hierarchical feature distribution patterns during model forward propagation. Specifically, lower and middle layers are primarily responsible for extracting general geometric lines and macroscopic compositions, while top layers handle higher-order semantics and artistic style rendering. Blindly stacking MoE-Lora modules across all layers significantly inflates the number of fine-tuning parameters and optimization freedom.

[0038] However, when faced with small sample data, this overly redundant architecture can lead to a lost gradient optimization path, resulting in severe overfitting or underfitting, manifesting as blurred styles, texture corruption, and color distortion in the generated images. Simultaneously, due to the lack of functional separation between structural and style features, the low-level general texture gradient and the high-level specific style gradient overlap within the same physical channel, causing interference between features of different styles, leading to catastrophic forgetting and style crosstalk, ultimately severely impacting the stability and image quality of multi-style image generation. Even more serious is the fact that traditional hybrid expert architectures commonly employ a single-stage unconstrained optimization paradigm during fine-tuning, failing to perform functional separation between structural and style features at the physical topology level. When directly training on multi-style mixed datasets, the velocity field prediction gradients generated by different style samples will create disordered mutual pulls in the lower layers of the model, causing severe drift in latent space features, making it extremely difficult for the global loss function to converge smoothly, thus affecting the quality of generated images.

[0039] In view of this, the inventive concept of this application is as follows: The image to be processed and style description information are input into a trained image generation model, which includes a first layer embedding a low-rank adaptation module and a second layer embedding a hybrid expert adaptation module; feature recognition is performed on the image to be processed and the style description information to obtain initial image-text features; feature correction is performed on the initial image-text features based on the low-rank adaptation module to obtain image-text structural features; feature extraction is performed on the image-text structural features based on the hybrid expert adaptation module to obtain style rendering features; feature fusion is performed on the style rendering features and the image-text structural features to generate a target image that satisfies the style description information.

[0040] In the above process, by configuring the image generation model as a heterogeneous architecture containing a first layer with an embedded low-rank adaptation module and a second layer with an embedded hybrid expert adaptation module, the first layer and the low-rank adaptation module collaboratively extract and correct cross-style shared image-text structural features. The hybrid expert adaptation module in the second layer then performs style-specific feature extraction based on these image-text structural features. This physically separates the propagation path of general structural features from the multi-style rendering path, constraining gradient regression of different styles to their respective independent adaptation paths, avoiding feature crosstalk and gradient pulling. This achieves separation and optimization of image structure extraction and style rendering, thereby improving the quality of generated images.

[0041] To facilitate understanding, the model structure of the image generation model in the embodiments of this application will be described below: The image generation model in this application is obtained by fine-tuning a basic image generation model in two stages based on style image samples. The basic image generation model can be a pre-trained model with general image generation capabilities, or it can be an initialized model that has not been pre-trained.

[0042] In this application, the preferred embodiment is to use a pre-trained model with general image generation capabilities as the basic image generation model. The basic image generation model can be constructed based on the flow matching paradigm, or based on a denoising diffusion probability model, or based on other pre-trained deep learning networks with hierarchical feature extraction capabilities. This application does not limit the specific pre-training method of the basic image generation model.

[0043] The base image generation model is used as a frozen backbone network during fine-tuning, meaning its original weights remain frozen throughout the process. The low-rank adaptation module and the hybrid expert adaptation module are embedded into the base image generation model before the two-stage fine-tuning. Specifically, the low-rank adaptation module is first embedded into the layer responsible for extracting general structural features within the base image generation model, and the hybrid expert adaptation module is embedded into the layer responsible for higher-order semantics and style rendering, before the two-stage fine-tuning begins. As described above... Figure 1 For example, remove Figure 1 The LoRA and Moe-Lora modules shown represent the basic image generation model structure before fine-tuning. This method of embedding first and then fine-tuning in stages ensures that the entire model structure remains unchanged throughout the fine-tuning process. The model only selectively updates the matrix parameters of different modules at different stages, without repeatedly adding or deleting modules during the fine-tuning process.

[0044] It should be noted that the "first layer" and "second layer" in this application embodiment do not specifically refer to a single network layer in the model, but should be understood as a set of layers corresponding to different functional levels. In actual deployment, the image generation model includes multiple first layers and multiple second layers cascaded sequentially along the data flow direction. Each first layer embeds a corresponding low-rank adaptation module, and each second layer embeds a corresponding hybrid expert adaptation module. In the data flow direction, the output of the previous layer serves as the input of the next layer, forming a progressively advancing processing chain.

[0045] For multiple first layers, the first first layer receives the initial image-text features obtained by mapping the image to be processed and style description information as input. Its embedded low-rank adaptation module performs feature correction on the input and outputs the corrected image-text structural features. Subsequent first layers take the corrected image-text structural features obtained from the adjacent first layer as their own input. Their embedded low-rank adaptation module performs feature correction on the corrected image-text structural features of the previous first layer again and outputs the corrected image-text structural features to the next layer. Through the step-by-step correction of multiple first layers, style interference in the initial image-text features is gradually suppressed, while the universal structural representation shared across styles is strengthened and solidified layer by layer.

[0046] Correspondingly, for multiple second layers, the first second layer takes the image-text structural features obtained from the correction of the last first layer as input. Its embedded hybrid expert adaptation module extracts and fuses style features based on these image-text structural features, outputting high-order features that have initially fused style information. Subsequent second layers take the output of the adjacent previous second layer as their input, and their embedded hybrid expert adaptation module extracts and fuses style features again. By layering and refining the style rendering effect, the final style rendering features are generated. The last second layer generates a target image that satisfies the style description information based on the final fused features.

[0047] This deep cascaded architecture, consisting of multiple first and second layers, allows structural correction and style rendering to proceed progressively at different depths. Without significantly increasing the computational burden of a single layer, it achieves nonlinear modeling of complex feature transformations through inter-layer combination, thereby improving the model's adaptability to diverse content and the quality of its generation.

[0048] Figure 1 A schematic diagram of the image generation model structure in an embodiment of this application is shown. For example... Figure 1 As shown, the image generation model is illustrated using a two-stream Transformer architecture as an example. However, it is understood that the heterogeneous adaptation architecture of this application embodiment can also be applied to a single-stream Transformer structure or other deep learning networks with hierarchical feature extraction capabilities.

[0049] To facilitate understanding, we will use a style image sample as an example to illustrate the overall processing flow of the model. This style image sample contains a content image (e.g., a picture of a kitten) and style description text (e.g., converting the image into a realistic style).

[0050] First, the content image is fed into the image feature stream (e.g. Figure 1 The style description text shown ("img") is fed into the text feature stream (e.g., ...). Figure 1 As shown in “txt”, auxiliary information such as time step embedding and conditional control vectors are also injected into the model as vector condition inputs.

[0051] In this embodiment, the first layer of the image generation model includes a dual-stream feature encoder capable of processing image and text information in parallel. Features from the image and text streams are respectively normalized, scaled, and translated to generate query / key / value pairs (e.g., ...). Figure 1 After the shown "LayerNorm, ScaleAndShift, Self-attention-qkv"), a concatenation operation (such as...) is performed. Figure 1 The "cat" shown is fused, and joint attention is calculated (e.g., Figure 1 (The "attention" shown). After completing the joint attention interaction, the features are processed through a splitting operation (such as... Figure 1 The “split” shown is re-split into image-side attention output branches (such as...). Figure 1 The “img_attn” shown and the attention output branch on the text side (such as) Figure 1 The "txt_attn" shown is passed through a linear gate (e.g., Figure 1 The “Linear_gate” shown controls the activation intensity of subsequent multilayer perceptron branches. The aforementioned modules, including layer normalization, scaling and translation modulation, self-attention mechanism, splicing, joint attention computation, splitting, and linear gating, are all components of the first layer.

[0052] Subsequently, the features are fed into the top layer of the model, namely the high-order rendering layer (MLP), which is responsible for high-order semantics and style rendering, for deep feature transformation, and finally output a realistic cat image.

[0053] This application embeds two types of key modules at specific locations in the model. The modules labeled "LoRA" are low-rank adaptation modules embedded in the first layer of the model. These low-rank adaptation modules can be distributed in the lower, middle, and top layers of the model, excluding MLP dual-stream Transformer linear layers, including but not limited to linear mapping layers in self-attention and cross-attention mechanisms. Their main responsibility is to comprehensively capture and solidify universal visual features shared across styles, such as the outline of a cat and its spatial geometry. The modules labeled "MoE-Lora" are hybrid expert adaptation modules embedded in the second layer of the model. These hybrid expert adaptation modules are deployed in the top-level high-order rendering layer of the model.

[0054] In some embodiments, the hybrid expert adaptation module incorporates a gated routing network and multiple sets of parameter-independent expert branches deployed in parallel. This gated routing network and the multiple expert branches are connected in parallel at the input of the current layer, sharing identical input features. When the structural features representing the cat's content structure are passed to this layer, the gated routing network dynamically selects the target expert branch best suited to handle this style based on the style description text required by the image sample. Each expert branch extracts the corresponding style offset from the image and text structural features and fuses these style rendering features with the original structural features.

[0055] After the above layer-by-layer processing, the model decodes and reconstructs the comprehensive representation after fusing style rendering features and sample structural features at the output end, directly generating a cat image that meets the "realistic style". This hierarchical embedding heterogeneous design deploys the low-rank adaptation module for maintaining content structure and the hybrid expert adaptation module for realizing style rendering at different functional levels of the model, respectively. This achieves deep decoupling of general structural features and high-order semantic style from a physical topology perspective, enabling the model to accurately render style rendering features onto the image without destroying the content structure of the original image.

[0056] Figure 2 The module structure of the hybrid expert adaptation module in this embodiment is shown. This module may include a gate-controlled routing network (Router) and multiple expert branches. The gate-controlled routing network is used to output the style adaptation degree corresponding to each expert branch based on the input features. Figure 2 The “A1, A2, A3, A4” shown represent the dimension-up matrices of different expert branches, while “B1, B2, B3, B4” represent the low-rank dimension-reduction matrices of different expert branches. Each set of dimension-up matrices and low-rank dimension-reduction matrices together constitutes the low-rank transformation path of an expert branch. Figure 2The circled multiplication sign indicates that the gating weights are applied to the outputs of each expert branch, while the circled plus sign indicates that the outputs of the target expert branch are fused and the fused result is added to the original outputs (Pre-trained Weights) of the second layer as residuals.

[0057] It should be further explained that in the baseline mapping of the linear layer of the model, the input features of the current layer can be defined as follows: N is the number of tokens, d in Let be the dimension of the input features. The pre-trained weight matrix of the aforementioned frozen backbone network is defined as follows: If dout is the dimension of the output feature, then its linear mapping result is... (That is, the output) can be shown as in the following formula (1): (1) To implicitly capture general visual representations across styles, embodiments of this application introduce a low-rank adaptation module in the first layer of the model for incremental correction mapping. The low-rank dimensionality reduction matrix is ​​defined as follows: The upgraded matrix is Then the final output of the first layer is the sample correction feature. It can be shown in the following formula (2): (2) in, For constant hyperparameters, The specified rank for the low-rank adapter module, its value can be set to... .

[0058] In the first stage of fine-tuning, a first image sample can be input into the base image generation model to obtain a first reference image. A first fine-tuning loss is then determined based on the first reference image and sample labels, and the matrix parameters of the low-rank adaptation module are adjusted using this first fine-tuning loss. Specifically, during this first stage of fine-tuning, all parameters of the hybrid expert adaptation module are always set to an untrainable state (i.e., their matrix parameters are frozen), thus isolating the hybrid expert adaptation module from interference with sample structure learning. The first stage of fine-tuning in this embodiment focuses on training the low-rank adaptation module, enabling it to learn to extract and correct sample structural features, laying a solid content foundation for subsequent stylization.

[0059] Regarding the specific composition of the first fine-tuning loss, this application provides a composite loss function. It not only includes the standard flow matching loss to ensure image generation quality, but also innovatively introduces a basis correction loss.

[0060] In some embodiments, feature extraction can be performed on the first image sample to obtain initial sample features; feature correction can be performed on the initial sample features based on the low-rank adaptation module to determine the sample structural features; basis correction loss can be determined based on the initial sample features and the sample structural features; first flow matching loss can be determined based on the image difference between the first reference image and the sample labels corresponding to the first image sample; and first fine-tuning loss can be determined based on the first flow matching loss and the basis correction loss.

[0061] Specifically, this application pre-specifies the first layer set in the basic image generation model. Through the forward hook mechanism, the baseline feature matrix of the aforementioned frozen backbone network in the first layer, i.e., the initial features of the samples, is extracted in real time. And the structural feature matrix at the same position after the external low-rank adaptation module is connected, i.e., the sample structural features. .

[0062] In some embodiments, basis correction loss The Frobenius norm can be calculated by taking the sum of squares and mean of the matrix differences between the initial features and structural features of the sample, as shown in the following formula (3): (3) Among them, the Frobenius norm The mathematical essence of this is represented by the mean of the sum of squares of the matrix elements, used to accurately characterize the absolute geometric distance between the fine-tuned path and the original path in the latent space. This basis correction loss When extracting multi-style shared features, the forced low-rank adaptation module maintains a minimal Frobenius distance between the output sample structure features and the initial sample features in geometric space. This ensures that the low-rank adaptation module learns only style-independent geometric structure information during fine-tuning without introducing any style-specific high-frequency perturbations.

[0063] In some embodiments, the target vector (i.e., sample label) of the noisy velocity field in the packed latent space can be defined as ;in, To introduce Gaussian noise, Features of a clean target image in the packed space. First-flow matching loss. It can be explicitly defined as the weighted mean square error in the packed space, as shown in the following formula (4): (4) in, This is the model's predicted output vector in the forward flow field, which is the final image output by the model. This is the step-size sampling loss weight matrix, which is composed of the corresponding step size. The density of the log-normal distribution is derived from this. D is the total dimension of the samples after the packed latent space is flattened. This indicates that the arithmetic mean of all samples within the Batch axis is calculated.

[0064] In some embodiments, the first fine-tuning loss The first-order matching loss can be used. and substrate correction loss The weighted summation is shown in the following formula (5): = +λ1 (5) Where λ1 is a weighted hyperparameter for the quality of the balanced flow matching generation and the generalization power of the latent space general structure. By using the geometric distance topological constraint of the intermediate layer, the low-rank adaptation module is forced to actively discard the high-frequency style-specific noise brought by multi-style samples in the backpropagation error stream, forcibly locking in and inheriting the powerful macroscopic visual understanding ability of the basic image generation model, thus solving the problem of disordered gradient pulling on the latent space caused by multi-style mixed datasets during direct mixed training.

[0065] In the second stage of fine-tuning, the second image sample is input into the base image generation model after the first stage of fine-tuning to obtain the second reference image. The second fine-tuning loss is determined based on the sample labels corresponding to the second reference image and the second image sample, and the matrix parameters of the hybrid expert adaptation module are adjusted using the second fine-tuning loss.

[0066] In this stage, all parameters of the low-rank adaptation module are always set to an untrainable state, i.e., the matrix parameters of the low-rank adaptation module are frozen to preserve the structure fidelity learned in the first stage. In the second layer of the model (i.e., the aforementioned MLP), let its input features be... (i.e., sample structure features). For the multiple sets of parameter-independent expert branches deployed in parallel within the hybrid expert adaptation module, the low-rank dimensionality reduction matrix of the i-th expert branch is defined as... The upgraded matrix is The style increment output by the i-th expert branch It can be shown in the following formula (6): (6) in, The rank of a single expert branch is specified, and its value is 32; Hyperparameters scaled for experts.

[0067] In some embodiments, the weight parameter matrix of the gate-controlled routing network can be defined as follows: After fully connected linear mapping and Softmax normalization, the output shows the style fit for the four expert branches. Specifically, it can be shown in the following formula (7): = (7) In some embodiments, the style fit can be derived from the respective expert branches. In the process, the top two are selected as the target expert branches. Then, the style increments output by each target expert branch are summed to obtain the style rendering features.

[0068] Since the first-stage fine-tuning has completely anchored the feature sources entering the second layer and eliminated high-frequency jitter, the gated routing network can perform forward linear mapping and Softmax normalization on a feature domain with extremely stable spatial distribution. This makes the gradient optimization path of the routing weight parameters extremely smooth, enabling it to converge quickly and robustly to the global optimum. This fundamentally eliminates the model non-convergence and degradation caused by multi-objective pulling, as well as the fatal gated routing network decision drift problem in traditional hybrid expert fine-tuning.

[0069] Regarding the specific composition of the second fine-tuning loss, this application provides a composite loss function that not only includes the second flow matching loss but also introduces orthogonal regularization loss.

[0070] In some embodiments, an orthogonal projection matrix of each expert branch in the feature space can be constructed based on the low-rank dimensionality reduction matrix of each expert branch; the orthogonal projection matrix of each expert branch is regularized to obtain an orthogonal regularization loss; a second flow matching loss is determined based on the image difference between the second reference image and the sample label; and a second fine-tuning loss is determined based on the second flow matching loss and the orthogonal regularization loss.

[0071] Specifically, in the architecture of style rendering feature extraction based on target expert branches, due to the lack of explicit physical constraints on the expert parameter space, multiple expert branches are prone to falling into the dilemma of parameter homogenization and high overlap of representation spaces during joint optimization. When the gate-controlled routing network simultaneously activates and merges the outputs of multiple expert branches, if the latent space representations of these expert branches have serious overlap, it will inevitably lead to feature aliasing and style crosstalk in forward propagation, resulting in blurred details or color structure conflicts at the intersection of different styles in the final generated target image, and significantly squeezing the limited parameter capacity. In order to completely eliminate the spatial overlap and redundancy between experts while ensuring the collaborative expressiveness of multi-experts and achieving smooth style transition, this application constructs a subspace orthogonal projection regularization mechanism based on low-rank matrices.

[0072] First, based on the low-rank dimensionality reduction matrices of each expert branch, construct the orthogonal projection matrices of each expert branch in the feature space. According to the definition of formula (6) above, the dimensionality reduction matrix of the i-th expert branch in the second layer is derived as follows: .matrix The column vector space is explicitly spanned to form the exclusive low-rank subspace of this expert branch. The symmetric idempotent orthogonal projection matrix of this subspace is defined as follows: Its mathematical closed-loop mapping equation can be expressed as shown in the following formula (8): (8) In formula (8), the built-in low-rank order is much smaller than the dimension of the input feature (i.e., Therefore, inner product terms are set for small-scale matrices. Can guarantee the inverse matrix The absolute invertibility of the orthogonal projection matrix. Therefore, by taking advantage of the scale independence inherent in the orthogonal projection matrix, the interference of the absolute value of the parameter on the gradient flow can be removed and canceled. This forces the weights of each expert branch to physically and geometrically rotate and separate the low-rank subspace spanned by their respective column vector spaces in the high-dimensional space during backpropagation update, so that the product of the orthogonal projection matrices of any two expert subspaces converges to the zero matrix globally.

[0073] To achieve pairwise orthogonality of low-rank subspaces for multiple expert groups and minimize dimensionality interleaving and parameter redundancy, this application designs a pairwise orthogonal regularization loss function. For each pair of expert branches, the matrix product of the orthogonal projection matrices of the two expert branches is determined. Based on the cyclic shift invariance of the matrix trace and the symmetric idempotency of the orthogonal projection matrix, the calculation of the Frobenius norm of the matrix product is transformed into a trace function constructed from the low-rank dimensionality reduction matrices corresponding to the two expert branches. The trace operation is performed on the trace functions corresponding to each pair of expert branches to obtain the orthogonal loss corresponding to each pair of expert branches. The orthogonal losses corresponding to each pair of expert branches are accumulated to obtain the orthogonal regularization loss.

[0074] In some embodiments, orthogonal regularization loss The mathematical modeling form can be shown in the following formula (9): (9) in, This indicates finding the trace of the matrix. Formula (9) will take the trace that originally belonged to... The calculation of overlap in high-dimensional latent spaces is ingeniously condensed into a single space. Within a very small dimension matrix of 32×32, the computational complexity and memory usage of the algorithm are significantly reduced. With ultra-low computing power and memory overhead at the microsecond level, the redundancy of representations between experts is forcibly removed from the model.

[0075] When orthogonal regularization loss When the convergence reaches a minimum, the overlap dimension of any two expert subspaces in the latent space approaches zero, and the parameter response spaces of each expert branch achieve a high degree of orthogonality and independence at the mathematical level. This makes the dual expert output features that are finally dynamically activated by the gate-controlled routing network highly mathematically complementary, thus completely eliminating the problems of feature crosstalk, noise superposition, and texture blurring when multiple expert branches are weighted and fused.

[0076] Second-stream matching loss Definition and first-order matching loss Similarly, the specific calculation can be based on the image difference between the second reference image and the sample label, which will not be elaborated here.

[0077] In some embodiments, the second fine-tuning loss The second-flow matching loss can be used. and orthogonal regularization loss The weighted summation is shown in formula (10): = +λ2× (10) In the formula, λ2 is the orthogonal hyperparameter weighting coefficient, used to accurately balance the image quality accuracy of multi-style image stream matching and the spatial decoupling independence of multi-expert branches in the backpropagation gradient flow. By introducing this mechanism, this application, while leveraging the advantages of multi-expert collaborative representation and smooth style transition, mathematically removes redundant components between experts, fundamentally overcoming the core pain points of parameter crowding, feature crosstalk, and fusion blurring in existing multi-style image generation technologies.

[0078] This decoupled two-stage fine-tuning strategy injects structure preservation and style representation capabilities into the base image generation model in an orderly manner, avoiding the problem of conflict and compromise between the two capabilities during joint learning. The first-stage basis correction loss, through strong intermediate layer geometric distance topological constraints, forces the low-rank adaptation module to actively discard high-frequency style-specific noise from multi-style samples, constructing an absolutely stable feature pre-propagation basis without any fluctuations. The second-stage orthogonal regularization loss fine-tunes the hybrid expert adaptation module and the gated routing network on a fully anchored stable latent space feature source, making the termination condition of the entire fine-tuning process clear, and ensuring the step-by-step stable convergence of sample structural features and style rendering features. This significantly improves the robustness and reproducibility of model fine-tuning. By using flow matching loss in both stages, it ensures that the newly introduced low-rank module, while learning new functions, does not deviate from the original image generation quality and distribution of the model.

[0079] It should be noted that the first image sample used in the first stage of fine-tuning and the second image sample used in the second stage of fine-tuning can be the same or different, and this application does not limit this.

[0080] For example, the same batch of style image samples can be used in both the first and second stage fine-tuning processes. This batch of samples contains content images with clear outlines and stable structures. In the first stage, this batch of image samples is used to focus on training the low-rank adaptation module, enabling it to learn and solidify the sample structural features shared across styles. In the second stage, this batch of samples is used to train the hybrid expert adaptation module and the gated routing network based on the solidified sample structural features, enabling it to learn the differentiated style rendering features corresponding to different styles.

[0081] For example, the first image sample used in the first stage of fine-tuning can be a style image sample with rich structural features, clear subject edges and spatial layout, and small style differences. This allows the low-rank adaptation module to focus on learning style-independent general structural features. The second image sample used in the second stage of fine-tuning can be a style image sample with significant style differences, covering multiple styles. This is used to train the hybrid expert adaptation module and the gated routing network to learn style-independent style rendering features based on the fixed sample structural features. By using samples that emphasize image structure in the first stage and samples that emphasize image style in the second stage, the learning interference between structure and style can be reduced.

[0082] The above describes the specific process by which the image generation model in this embodiment is obtained through two-stage fine-tuning. It can be understood that after the above two-stage fine-tuning, the image generation model possesses stable structure preservation capabilities and multi-style-specific style rendering capabilities.

[0083] Regarding model convergence performance, this application employs a two-stage fine-tuning strategy to strongly isolate structural correction and style rendering across the temporal and parameter update dimensions. The first stage uses basis correction loss to forcibly latch and align the general representation of the base image generation model, eliminating the disordered pull of multi-style gradients and constructing a stable, unwavering feature base. The second stage fine-tunes the hybrid expert adaptation module and the gated routing network on a fully anchored, stable latent space feature source, enabling the sample structural features and style rendering features to converge stably step-by-step, thereby improving the model's robustness and reproducibility.

[0084] Regarding few-sample adaptation capability, this application employs a heterogeneous physical topology by deploying the low-rank adaptation module in the first layer and the hybrid expert adaptation module only in the second layer. This strongly decouples the structure extraction and style rendering functions, significantly reducing the ineffective learning degrees of freedom during model fine-tuning. Combined with orthogonal regularization loss constraints that keep the expert branches at geometrical angles in the latent space and focus on capturing independent style semantic features, this significantly reduces cross-dimensional color and texture noise interference. This allows the model to accurately learn exclusive style features even with a small number of samples, greatly improving style reproduction accuracy and generation fidelity in few-sample scenarios.

[0085] Regarding the stability of multi-style generation, this application employs a dual mechanism of hierarchical structure decoupling and orthogonal regularization loss to ensure the physical isolation of textual and image structural features from style rendering features, as well as the absolute independence of different expert subspace dimensions. This completely blocks the pathways of parameter crowding and gradient cross-contamination at both the physical topology and feature algebraic space levels, effectively avoiding the catastrophic forgetting problem during concurrent iterative training of multiple styles. When the orthogonal regularization loss tends to converge to its minimum value, the overlap dimension of any two expert subspaces in the latent space approaches zero, and the parameter response spaces of each expert branch achieve a high degree of orthogonal independence at the mathematical level, significantly enhancing the global consistency and coexistence stability of multi-style generation.

[0086] In terms of generating visual effects, this application adopts a fixed-target expert branch activation strategy with training and inference having the same origin. Throughout the entire forward propagation lifecycle, the learning of the nonlinear fusion mapping law of the gate-controlled routing network on the multi-experts is forcibly solidified. Based on orthogonal and independent low-dimensional subspaces, the activated dual-expert output features have extreme geometric complementarity. This not only eliminates dimensional redundancy and accurately restores the main tone and color features of various styles in a single-layer local area, but also makes full use of the huge deep cascaded topology structure constructed by dozens of layers. Through the dynamic alternation of routing nodes, the orthogonal semantics are fully integrated in the high-dimensional combination space, giving the picture a more delicate and rich sense of layering and light and shadow texture, achieving an extremely natural and smooth style transition.

[0087] In terms of engineering deployment performance, this application removes invalid expert branches in the middle and lower layers through layered simplification, embedding a hybrid expert adaptation module only in the second layer, significantly reducing the overall number of parameters and inference computation. Orthogonal regularization loss achieves dimensionality distillation of high-dimensional matrices through the cyclic shift invariance of the matrix trace and the symmetry idempotency of the projection matrix, successfully reducing the dimensionality of giant orthogonal projection multiplication operations to micro-matrix computation, resulting in extremely fast computational response and lightweight engineering deployment. A single model integrates multiple styles and semantic capabilities, requiring no weight file swapping during online runtime, completely eliminating the hardware scheduling latency of traditional multi-LoRA deployments, significantly improving resource utilization and online service concurrency throughput, and perfectly adapting to high-concurrency cloud services and mobile real-time beautification and other industrial applications.

[0088] After introducing the two-stage fine-tuning process of the image generation model in this application, the use of the model inference stage will be explained next. Figure 3 A flowchart illustrating the image generation method in an embodiment of this application is shown. Figure 3 As shown, the method includes the following steps: S301. Input the image to be processed and style description information into the trained image generation model; wherein, the image generation model includes a first layer with an embedded low-rank adaptation module and a second layer with an embedded hybrid expert adaptation module.

[0089] The image to be processed in this embodiment is the source image from which style transfer is desired. For example, it could be a user-provided photograph, illustration, sketch, or an intermediate image that has already been generated but requires further stylization. Style description information is guiding data used to define the target style; it could be natural language text prompts, reference style images, style tags, style vectors, or any combination thereof.

[0090] As mentioned earlier, the image generation model in this embodiment includes multiple cascaded first layers and multiple cascaded second layers. Each first layer embeds a low-rank adaptation module, and each second layer embeds a hybrid expert adaptation module. By inputting the image to be processed along with style description information, the dual conditions of content source and target style can be explicitly passed to the model, thereby providing a complete input foundation for solving the collaborative problem of subsequent content structure preservation and accurate style transfer.

[0091] It should be noted that the initial features and structural features of the samples used in the aforementioned two-stage fine-tuning process are essentially the same type of features as the initial features and structural features of the images and text used in the subsequent inference generation process. Specifically, the model input in the two-stage fine-tuning stage is style image samples; therefore, the initial features extracted from the first layer are called initial features of the samples, and the features corrected by the low-rank adaptation module are called structural features of the samples.

[0092] During the inference and generation phase, the model's input consists of the image to be processed and style description information. Therefore, the initial features extracted from the first layer are called initial image-text features, and the features corrected by the low-rank adaptation module are called image-text structural features. Essentially, both are the original features output from the first layer and the structural features corrected by the low-rank adaptation module; they are distinguished only by their naming conventions based on the stage they are in and the input data, for ease of understanding and description.

[0093] S302. Perform feature recognition on the image to be processed and the style description information to obtain the initial image and text features.

[0094] In this embodiment, the first layer is a feature extraction network in the image generation model used for preliminary processing of multimodal inputs. During the inference generation stage, the image to be processed is fed into an image feature stream, the style description text is fed into a text feature stream, and auxiliary information is injected into the model as a vector conditional input. The feature encoder in the first layer performs preliminary feature extraction on the pixel information of the image to be processed and the semantic information of the style description information to obtain initial image-text features. These initial image-text features refer to the original multimodal representation obtained after the first layer performs preliminary feature extraction on the image and text inputs; for example, it can be a set of intermediate feature map tensors that are not decoupled from image structure information and style information.

[0095] In a cascaded architecture containing multiple first layers, the first first layer receives the original multimodal representation obtained by mapping the image to be processed and style description information as input. After feature recognition, it outputs preliminary initial image and text features. Subsequent first layers use the features output by the adjacent previous first layer as their own input and continue to perform feature recognition and transmission layer by layer.

[0096] S303. Based on the low-rank adaptation module, feature correction is performed on the initial features of the image and text to obtain the image and text structural features.

[0097] In this embodiment, the low-rank adaptation module is a computational unit embedded within the model that achieves efficient parameter fine-tuning through low-rank matrix factorization. For example, this module can be a set of low-rank residual matrices, whose dimension is much smaller than the weight matrix of the model layer being adapted. The low-rank adaptation module does not have expert branches, does not introduce routing mechanisms, and does not perform sparse activation; instead, it performs full-scale intensive computation on all input feature streams, thereby solidifying the basic visual representation shared across styles.

[0098] During the inference generation stage, the low-rank adaptation module performs feature correction on the initial image and text features output from the first layer to obtain image and text structural features. These image and text structural features refer to the feature representations obtained after correcting the initial image and text features, which place greater emphasis on the fidelity of content structure. For example, compared to the initial image and text features, the structural information represented by the image and text structural features, such as edges, contours, and object layouts, is enhanced or preserved, while style-related interferences such as textures and color distributions are effectively suppressed.

[0099] In the cascaded architecture containing multiple first layers, each first layer embeds a corresponding low-rank adaptation module. The low-rank adaptation module of the first first layer performs feature correction on its input initial image-text features and outputs the corrected image-text structural features to the next first layer. Subsequent low-rank adaptation modules of each first layer use the corrected image-text structural features of the adjacent previous first layer as their own input, perform feature correction again, and output the corrected image-text structural features to the next layer. Through the step-by-step correction of multiple first layers, style interference in the initial image-text features is gradually suppressed, while the general structural representation shared across styles is strengthened and solidified layer by layer, finally outputting stable image-text structural features by the last first layer. The specific processing logic can be referred to the introduction of the two-stage fine-tuning process mentioned above, and will not be repeated here.

[0100] By obtaining image and text structure features through the low-rank adaptation module, a "content anchor point" can be provided for subsequent style rendering operations, thereby solving problems such as image distortion and content distortion that occur during style rendering and effectively ensuring the content integrity of the target image.

[0101] Step S304: Based on the hybrid expert adaptation module, feature extraction is performed on the text and image structure features to obtain style rendering features; As explained in the previous section on the two-stage fine-tuning phase, the hybrid expert adaptation module in the second layer consists of multiple parallel expert branches and a gated routing network. The gated routing network dynamically calculates the style fit degree corresponding to each expert branch based on the input features of the current layer, and selects the target expert branch accordingly.

[0102] During the inference generation phase, the first second layer receives the text-image structure features output from the last first layer as input. Its embedded hybrid expert adaptation module extracts and fuses style features based on these text-image structure features, outputting high-order features that have initially incorporated style information. Each subsequent second layer takes the output of the adjacent previous second layer as input, and its own embedded hybrid expert adaptation module extracts and fuses style features again. Through the progressive stacking and refinement of multiple second layers, the final second layer outputs complete style rendering features.

[0103] More specifically, in each second layer, the gate-controlled routing network dynamically calculates the style fit degree corresponding to each expert branch based on the input features of the current layer, and selects the target expert branch that meets the style fit condition accordingly. Each target expert branch performs a low-rank transformation on the input features and outputs its corresponding style increment. These style increments are accumulated according to the corresponding style fit degree to obtain the style rendering features output by the second layer.

[0104] In some embodiments, when obtaining target expert branches that meet the style adaptation conditions from each expert branch based on the style adaptation degree of each expert branch, a fixed target expert branch selection strategy can be adopted, that is, selecting a specified number of expert branches with the highest style adaptation degree as target expert branches.

[0105] For example, throughout the entire lifecycle of forward propagation (including the second-stage fine-tuning training phase and the final inference generation phase), each second layer employs a fixed target expert branch activation strategy where training and inference are from the same source; that is, it consistently uses a Top-2 dual-expert sparse activation and weighted fusion mechanism with a fixed number of 2. During the inference phase, the gate-controlled routing network directly receives the input features (i.e., the aforementioned X). top The mapping is completed according to the aforementioned formula (7), and the style fit for the four expert branches of the current layer is output. Subsequently, the model executes the Top-2 operator internally to retrieve the two expert index numbers j with the highest current activation weight values ​​and construct the target expert branch set Top-2(j) of the current layer.

[0106] Subsequently, based on the retrieved target expert branch index, the corresponding dedicated low-rank matrix parameters are called, and the style increments of the two expert branches are calculated in parallel according to the aforementioned formula (6). The gate-controlled routing network performs sparse weighted product fusion on the outputs of these two branches to obtain the style rendering features for the current layer's higher-order semantic rendering. The specific calculation process is shown in the following formula (11): (11) Furthermore, considering the above-mentioned strategy of selecting a fixed number of target expert branches, when faced with complex and ever-changing style requirements, if the current input features match multiple expert branches to a similar degree, but only a few expert branches are fixedly activated, it may result in the loss of useful style details held by other expert branches, thus limiting the completeness and richness of style rendering.

[0107] To improve the accuracy and flexibility of target expert branch selection and thus enhance the quality of target image generation, this application also provides another adaptive target expert branch number selection mechanism. Each second layer can dynamically decide whether to use a few top experts or integrate all experts based on the steepness of the style fit distribution.

[0108] In practice, the expert branches can be sorted sequentially from high to low style fit, and the sorted style fit sequence can be expressed as w(1)≥w(2)≥…w(i). Subsequently, the first-order difference result Δk=w(k)-w(k+1) of the style fit corresponding to each two adjacent expert branches is determined, where k=1,2,…,i-1; i is the number of expert branches in the hybrid expert fit module.

[0109] If all first-order difference results are below the preset difference threshold (Δk < τ), it indicates that there is no expert branch with a high degree of style fit or overwhelming advantage, and the adaptability of each expert branch to the current task is relatively similar. In this case, all expert branches are taken as the target expert branch. When the style requirements and the capabilities of multiple expert branches are well matched, fusing the outputs of all expert branches can obtain richer style rendering features, thus solving the problem of losing the useful style details held by other expert branches when only one or a few experts are selected.

[0110] Correspondingly, if at least one first-order difference result is not lower than the difference threshold, it indicates that there are one or more expert branches with particularly outstanding style fit, forming a significant capability gap with other branches. In this case, only a specified number of expert branches with the highest style fit (such as the aforementioned Top 2) can be selected as the target experts. This prevents unsuitable expert branches from introducing irrelevant style noise or drift, thereby solving the problem of redundant experts interfering with the accuracy of style rendering, while minimizing unnecessary computational overhead.

[0111] Step S305: Perform feature fusion on the style rendering features and the image structure features to generate a target image that satisfies the style description information.

[0112] In this embodiment, the second layer is a feature decoding or image reconstruction network located after the hybrid expert adaptation module in the image generation model. For example, the second layer could be a feature decoder that decodes and upsamples the fused feature map to the target image resolution. The second layer simultaneously receives and fuses features from both content and style paths, giving the final image reconstruction process both a clear structural blueprint and refined style feature guidance.

[0113] Specifically, the style rendering features calculated above can be represented as shown in the following formula (12). The original output Y, which is finally mapped to the frozen backbone network. base By summing the residuals, we obtain the final forward output of the model, which is the target image Y. final : Y final =Y base + (12) Through the above process, this embodiment constructs a deep cascaded architecture consisting of a first layer and a second layer in the image generation model, forming a phased, multi-level processing path of layer-by-layer structure correction, layer-by-layer style extraction, and feature fusion, completely decoupling style expression from structural content. Specifically, in the first layer of the model, a low-rank adaptation module is embedded to perform full-volume intensive computation on all input feature streams without introducing additional routing mechanisms and sparse activation, thereby suppressing style-related interferences such as texture and color distribution layer by layer and solidifying cross-style shared and universal structural representations. In the second layer of the model, a hybrid expert adaptation module is embedded, which dynamically calculates the style adaptation degree corresponding to each expert branch through a gate-controlled routing network, and selects the target expert branch accordingly. Each target expert branch outputs its corresponding style increment, which is then weighted and accumulated to obtain the style rendering feature. The style rendering feature and the image-text structure feature are rendered onto the image layer by layer, thereby improving the style accuracy and content fidelity of the target image.

[0114] Based on the same inventive concept, this application also provides an image generation apparatus 400, the structure of which is as follows: Figure 4 As shown, it includes: Information acquisition unit 401 is configured to: input the image to be processed and style description information into a trained image generation model; the image generation model includes a first layer embedding a low-rank adaptation module and a second layer embedding a hybrid expert adaptation module; feature recognition unit 402 is configured to: perform feature recognition on the image to be processed and the style description information to obtain initial image-text features; feature correction unit 403 is configured to: perform feature correction on the initial image-text features based on the low-rank adaptation module to obtain image-text structural features; style rendering unit 404 is configured to: extract features from the image-text structural features based on the hybrid expert adaptation module to obtain style rendering features; image generation unit 405 is configured to: perform feature fusion on the style rendering features and the image-text structural features to generate a target image that satisfies the style description information.

[0115] In some embodiments, the hybrid expert adaptation module includes multiple expert branches and a gate-controlled routing network; the hybrid expert adaptation module is used to extract features from the graphic structure features to obtain style rendering features. Specifically, the style rendering unit 404 is configured to: perform feature recognition on the graphic structure features based on the gate-controlled routing network to obtain the style adaptation degree corresponding to each expert branch; based on the style adaptation degree, obtain a target expert branch that meets the style adaptation conditions from the expert branches; and extract features from the graphic structure features based on the target expert branch to obtain the style rendering features.

[0116] In some embodiments, the process of obtaining target expert branches that meet the style adaptation conditions from the expert branches based on the style adaptation degree is performed. Specifically, the style rendering unit 404 is configured to: sort the expert branches in descending order of style adaptation degree, determine the first-order difference result of the style adaptation degree corresponding to each pair of adjacent expert branches; if all first-order difference results are lower than the difference threshold, then all expert branches are taken as target expert branches; if at least one first-order difference result is not lower than the difference threshold, then a specified number of expert branches with the highest style adaptation degree are selected from the expert branches as target expert branches.

[0117] In some embodiments, the image generation model is obtained by two-stage fine-tuning of the basic image generation model based on style image samples; the style image samples include a first image sample and a second image sample; in the first stage of fine-tuning, the first image sample is input into the basic image generation model to obtain a first reference image, and a first fine-tuning loss is determined based on the first reference image and the sample labels corresponding to the first image sample, and the matrix parameters of the low-rank adaptation module are adjusted using the first fine-tuning loss; in the second stage of fine-tuning, the second image sample is input into the basic image generation model to obtain a second reference image, and a second fine-tuning loss is determined based on the second reference image and the sample labels corresponding to the second image sample, and the matrix parameters of the hybrid expert adaptation module are adjusted using the second fine-tuning loss.

[0118] In some embodiments, the first fine-tuning loss is determined by: extracting features from the first image sample to obtain initial sample features; performing feature correction on the initial sample features based on the low-rank adaptation module to determine sample structural features; determining a basis correction loss based on the initial sample features and the sample structural features; determining a first flow matching loss based on the image difference between the first reference image and the sample label; and determining the first fine-tuning loss based on the first flow matching loss and the basis correction loss.

[0119] In some embodiments, the second fine-tuning loss is determined by: constructing an orthogonal projection matrix of each expert branch in the feature space based on the low-rank dimensionality reduction matrix associated with each expert branch in the hybrid expert adaptation module; performing regularization processing on the orthogonal projection matrix of each expert branch to obtain an orthogonal regularization loss; determining a second flow matching loss based on the image difference between the second reference image and the sample label; and determining the second fine-tuning loss based on the second flow matching loss and the orthogonal regularization loss.

[0120] In some embodiments, the orthogonal regularization loss is determined by: for every two expert branches, determining the matrix product of the orthogonal projection matrices of the two expert branches; and, based on the cyclic shift invariance of the matrix trace and the symmetric idempotency of the orthogonal projection matrix, transforming the Frobenius norm calculation of the matrix product into a trace function constructed from the low-rank reduced-dimensional matrices corresponding to the two expert branches; performing trace operation on the trace functions corresponding to each two expert branches to obtain the orthogonal loss corresponding to each two expert branches; and summing the orthogonal losses corresponding to each two expert branches to obtain the orthogonal regularization loss.

[0121] In some embodiments, the low-rank adaptation module and the hybrid expert adaptation module are embedded in the base image generation model before the two-stage fine-tuning is performed; during the first-stage fine-tuning, the matrix parameters of the hybrid expert adaptation module are frozen; during the second-stage fine-tuning, the matrix parameters of the low-rank adaptation module are frozen.

[0122] The image generation apparatus provided in this embodiment belongs to the same concept as the image generation method provided in the above embodiments of this application. It can execute the image generation method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects of executing the method. Technical details not described in detail in this embodiment can be found in the corresponding method embodiments of this application, and will not be repeated here. It should be understood that the functions implemented by each unit in the above apparatus can be implemented by the same or different processors, and this embodiment of the application does not limit this.

[0123] Based on the same inventive concept, embodiments of this application also provide an electronic device. See also Figure 5 As shown, the electronic device may include a memory 1000 and a processor 1010. The memory 1000 is connected to the processor 1010 and is used to store programs; the processor 1010 is used to implement an image generation method disclosed in any of the above embodiments by running the programs stored in the memory 1000.

[0124] Specifically, the aforementioned electronic device may further include: a bus, a communication interface 1020, an input device 1030, and an output device 1040. The processor 1010, memory 1000, communication interface 1020, input device 1030, and output device 1040 are interconnected via the bus. The bus may include a pathway for transmitting information between various components of the computer system.

[0125] Processor 1010 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 1010 may include a main processor, and may also include a baseband chip, a modem, etc.

[0126] The memory 1000 stores a program for executing the technical solution of this application, and may also store an operating system and other critical business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 1000 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0127] Input device 1030 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0128] Output device 1040 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0129] The communication interface 1020 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0130] The processor 1010 executes the program stored in the memory 1000 and calls other devices, and can be used to implement each step of any of the image generation methods provided in the above embodiments of this application.

[0131] Based on the same inventive concept, embodiments of this application also provide a computer program product, which includes computer program instructions that, when run by a processor, cause the processor to perform the steps of an image generation method according to various embodiments of this application as described in any of the above embodiments of this specification.

[0132] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server; this application does not limit this.

[0133] Based on the same inventive concept, embodiments of this application also provide a storage medium storing a computer program thereon, the computer program being executed by a processor of steps in an image generation method according to various embodiments of this application as described in any of the foregoing embodiments of this specification.

[0134] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0135] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0136] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0137] The functional modules or sub-modules in the various embodiments of this application can be integrated into a single processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into a single module. The integrated modules or sub-modules described above can be implemented in hardware or as software functional modules or sub-modules. Those skilled in the art will further recognize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0138] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0139] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0140] The above description of the embodiments disclosed in this application enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image generation method, characterized in that, include: The image to be processed and style description information are input into a trained image generation model; wherein, the image generation model includes a first layer with an embedded low-rank adaptation module and a second layer with an embedded hybrid expert adaptation module; Feature recognition is performed on the image to be processed and the style description information to obtain initial image and text features; Based on the low-rank adaptation module, feature correction is performed on the initial image and text features to obtain image and text structural features; Based on the hybrid expert adaptation module, feature extraction is performed on the graphic structure features to obtain style rendering features; The style rendering features and the graphic structure features are fused to generate a target image that satisfies the style description information.

2. The method according to claim 1, characterized in that, The hybrid expert adaptation module includes multiple expert branches and a gate-controlled routing network; the feature extraction of the text-image structure features based on the hybrid expert adaptation module to obtain style rendering features includes: Based on the gate control routing network, feature recognition is performed on the graphic structure features to obtain the style fit degree corresponding to each expert branch; Based on the style fit, a target expert branch that meets the style fit condition is obtained from each expert branch; Based on the target expert branch, feature extraction is performed on the text and image structure features to obtain the style rendering features.

3. The method according to claim 2, characterized in that, The step of obtaining the target expert branch that meets the style adaptation condition from each expert branch based on the style adaptation degree includes: The expert branches are sorted sequentially from high to low style fit, and the first-order difference result of style fit corresponding to each pair of adjacent expert branches is determined. If all first-order difference results are below the difference threshold, then all expert branches will be used as the target expert branch. If at least one first-order difference result is not lower than the difference threshold, then a specified number of expert branches with the highest style fit are selected from the expert branches as the target expert branches.

4. The method according to claim 1, characterized in that, The image generation model is obtained by fine-tuning the basic image generation model in two stages based on style image samples; the style image samples include a first image sample and a second image sample. In the first stage of fine-tuning, the first image sample is input into the basic image generation model to obtain the first reference image, and the first fine-tuning loss is determined based on the first reference image and the sample labels corresponding to the first image sample, and the matrix parameters of the low-rank adaptation module are adjusted using the first fine-tuning loss. In the second stage of fine-tuning, the second image sample is input into the basic image generation model to obtain the second reference image, and the second fine-tuning loss is determined based on the sample labels corresponding to the second reference image and the second image sample. The matrix parameters of the hybrid expert adaptation module are then adjusted using the second fine-tuning loss.

5. The method according to claim 4, characterized in that, The step of determining the first fine-tuning loss based on the first reference image and the sample labels corresponding to the first image samples includes: Feature extraction is performed on the first image sample to obtain the initial features of the sample; Based on the low-rank adaptation module, feature correction is performed on the initial features of the sample to determine the sample structural features; Based on the initial features and structural features of the samples, the basis correction loss is determined; The first-stream matching loss is determined based on the image difference between the first reference image and the sample label; The first fine-tuning loss is determined based on the first flow matching loss and the basis correction loss.

6. The method according to claim 4, characterized in that, The step of determining the second fine-tuning loss based on the second reference image and the sample labels corresponding to the second image samples includes: Based on the low-rank dimensionality reduction matrix associated with each expert branch in the hybrid expert adaptation module, construct the orthogonal projection matrix of each expert branch in the feature space. The orthogonal projection matrix of each expert branch is regularized to obtain the orthogonal regularization loss; The second flow matching loss is determined based on the image difference between the second reference image and the sample label; The second fine-tuning loss is determined based on the second flow matching loss and the orthogonal regularization loss.

7. The method according to claim 6, characterized in that, The orthogonal projection matrix of each expert branch is regularized to obtain the orthogonal regularization loss, including: For every two expert branches, determine the matrix product of the orthogonal projection matrices of the two expert branches; and, based on the cyclic shift invariance of the matrix trace and the symmetric idempotency of the orthogonal projection matrix, transform the Frobenius norm calculation of the matrix product into a trace function constructed from the low-rank reduced-dimensional matrices corresponding to the two expert branches. Perform trace operation on the trace function corresponding to each pair of expert branches to obtain the orthogonal loss corresponding to each pair of expert branches; The orthogonal regularization loss is obtained by summing the orthogonal losses corresponding to every two expert branches.

8. The method according to any one of claims 4-7, characterized in that, The low-rank adaptation module and the hybrid expert adaptation module are embedded in the basic image generation model before the two-stage fine-tuning is performed; during the first stage fine-tuning, the matrix parameters of the hybrid expert adaptation module are frozen; during the second stage fine-tuning, the matrix parameters of the low-rank adaptation module are frozen.

9. An electronic device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the steps of the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-8.

Images