A method, system and apparatus for image and video customization using multi-diffusion model fusion

Abstract

The application discloses a kind of multi-diffusion model fusion's image and video customization method and system, based on pre-training module, fusion module and generation module, the weight of multiple pre-trained diffusion model is fused by selecting fusion strategy, output target content is generated based on fusion weight, compared with prior art, the method supports the diffusion model with different architecture, training process and sampling scheduler to be fused, so as to realize the diffusion model without training or architecture modification to be fused;And based on the complementary advantages of each diffusion model, the generated image and video have higher quality in visual realism, identity preservation and style consistency, and can provide customized solutions for different scenarios and user needs, suitable for complex real-world scenarios based on diffusion theory Application;The application also provides a kind of multi-diffusion model fusion's image and video customization device, realizes the multi-diffusion model fusion's image and video customization method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and pattern recognition, and particularly relates to an image and video customization method, system and device based on multi-diffusion model fusion. BACKGROUND

[0002] Modeling generation has become a cornerstone of the development of the field of artificial intelligence, especially in the field of image and video synthesis. Diffusion models have attracted much attention due to their ability to generate high-quality, high-fidelity images and videos. Diffusion models gradually denoise random noise,

[0003] Generate coherent data samples and effectively capture complex data distributions.

[0004] In recent years, specialized diffusion models have been developed for specific tasks or to generate content with specific attributes. For example:

[0005] Style customization models: In digital art creation and style transfer applications, these style customization models are used to generate images with specific artistic styles, enabling ordinary photos to be converted into works of art with the style of famous artists or specific artistic schools.

[0006] Identity customization models: In applications such as personalized avatar creation, virtual reality, and character animation, it is crucial to maintain the unique identity features of individuals. These identity customization models ensure that the generated content accurately reflects the facial features and expressions of a specific individual.

[0007] Motion dynamics models: For video generation and animation production, capturing realistic motion dynamics is key. These motion dynamics models generate sequences that exhibit smooth and coherent motion, making them ideal for applications such as film production, gaming, and simulation.

[0008] Although these specialized models have made significant progress in their respective fields, they still face major challenges when it comes to integrating multiple specialized capabilities into a single generation process. Real-world applications often require such multi-faceted generation capabilities. For example, creating a personalized animated video requires not only capturing realistic motion, but also maintaining individual identity while applying a specific artistic style.

[0009] Diffusion models, as a powerful generation tool, have made great progress, but existing diffusion models mainly focus on single-modal control, i.e., the diffusion process is driven by only one modality condition. To further adapt to various personalized needs, existing model fusion strategies have been developed, but they have the following limitations:

[0010] 1. Homogeneous model requirement: Traditional fusion methods usually require the models to be fused to have similar architectures and training paradigms. This homogeneity is a prerequisite for techniques such as weight averaging to function properly. For example, reference 1 (Collaborative Diffusion for Multi-Modal Face Generation and Editing, 2023, Computer Vision and Pattern Recognition, DOI: arxiv-2304.10530) proposes collaborative diffusion for isomorphic models, which is based on multiple pre-trained single-modal diffusion models to collaboratively realize multi-modal face generation capabilities, and integrates various single-modal operations to perform multi-modal editing. However, this multi-modal collaborative diffusion model fusion is for isomorphic models, and has limitations for heterogeneous model fusion, and the model used for generation needs to be trained, which increases the computational overhead and reduces the efficiency of computational task processing.

[0011] 2. Weight-based fusion: Traditional fusion methods average or combine the weights of models to achieve fusion. For example, reference 2 (ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs, 2023, Computer Vision and Pattern Recognition, DOI: rxiv-2311.13600) proposes ZipLoRA, which combines independently trained style and topic LoRAs to generate any user-provided style and topic. However, this method assumes that the models have the same architecture and compatible parameterization. If the models differ significantly, weight-based fusion may result in performance degradation or convergence problems.

[0012] 3. Switch-based fusion: Different models are used at different stages of the generation process, such as reference 3 (Multi-LoRA Composition for Image Generation, 2024, Computer Vision and Pattern Recognition, DOI: arxiv-2402.16843) which proposes two training-free methods centered on decoding: LoRA Switch for alternating different LoRAs at each denoising step and LoRA Composite for combining all LoRAs to guide more coherent image synthesis. Based on these two training-free methods, multiple LoRA combinations are studied to achieve more complex image generation. However, due to differences in model training objectives and data distribution, this method may result in inconsistencies and artifacts.

[0013] 4. Architecture compatibility constraints: Many fusion methods rely on the structural compatibility of the models. Models with different network architectures, layer configurations, or activation functions are a major challenge for these methods, often leading to ineffective fusion.

[0014] These limitations hinder the ability to fully exploit the complementary advantages of diverse models, and existing fusion strategies often fail or produce suboptimal results when trying to fuse specialized models developed independently by different architectures and training settings, which also hinders the progress of complex generation tasks. SUMMARY

[0015] The purpose of the present application is to provide a multi-diffusion model fusion image and video customization method, system and device, which fuses the weights of multiple pre-trained diffusion models by selecting a fusion strategy, generates an output image or video based on the fusion weights, and is used to fuse multiple diffusion models with different architectures, training settings and sampling schedules. Take full advantage of their complementary advantages to improve the quality of generated images and videos.

[0016] To achieve the above-mentioned purpose of the application, an embodiment provides a multi-diffusion model fusion image and video customization method, which comprises the following steps:

[0017] Step 1: Provide multiple pre-trained diffusion models and construct a dataset based on the needs of the diffusion model fusion target;

[0018] Step 2: Based on the multiple pre-trained diffusion models, calculate the noise residuals of each diffusion model, and dynamically calculate the fusion weights through the fusion strategy;

[0019] The fusion strategy is one or more of the following: fixed denoising time step switching denoising U-Net network, combining the noise features of multiple pre-trained diffusion models through equal proportion fusion, dynamically calculating based on the noise features of multiple pre-trained diffusion models, and manually adjusting based on the output quality requirements of multiple pre-trained diffusion models;

[0020] Step 3: Based on the fusion weights, perform denoising prediction fusion on the multiple pre-trained diffusion models, and iteratively train to complete the target generation task.

[0021] In one embodiment, the multiple pre-trained diffusion models in step 1 include a denoising U-Net network and a corresponding noise scheduler, wherein the multiple pre-trained diffusion models have different architectures, training processes and noise schedulers.

[0022] In one embodiment, the dataset in step 1 includes text prompts, images, pose sequences or other data related to the model target.

[0023] In one embodiment, the denoising U-Net network switching at fixed denoising time steps comprises: using a denoising U-Net-1 network at fixed denoising time steps (1, t), using a denoising U-Net-2 network at fixed denoising time steps (t+1, t+n), and using a denoising U-Net-3 network at fixed denoising time steps (t+n, T), wherein the fixed denoising time step t and the fixed denoising time step n are both within the denoising time steps (1, T).

[0024] In one embodiment, the noise feature of the plurality of pre-trained diffusion models is combined by equal proportion fusion, comprising: calculating the noise residual of each pre-trained diffusion model respectively, and fusing the weights of the plurality of pre-trained diffusion models based on an equal proportion fusion manner, expressed as:

[0025]

[0026] Wherein, L t represents the noise residual of the fused diffusion model, represents the noise residual of the first diffusion model, represents the noise residual of another diffusion model.

[0027] In one embodiment, the noise feature of the plurality of pre-trained diffusion models is combined by equal proportion fusion, comprising: calculating the noise residual of each pre-trained diffusion model respectively, and fusing the weights of the plurality of pre-trained diffusion models based on an equal proportion fusion manner, expressed as:

[0028] The noise hidden variable is expressed as ∑|latent i，t |, the noise difference value is expressed as ∑|latent i，t -latent i，t-1 |, and the noise ratio value is expressed as Wherein, latent i，t represents the noise hidden variable of the i-th diffusion model at the current time step t, and latent i，t-1 represents the noise hidden variable of the i-th diffusion model at the previous time step t-1.

[0029] The application also provides a multi-diffusion model fusion image and video customization system, comprising: a pre-training module, a fusion module and a generation module.

[0030] The pre-training module is used to construct a data set based on the needs of diffusion model fusion targets, and calculate the noise residuals of a plurality of pre-trained diffusion models.

[0031] The fusion module is configured to select a fusion strategy to fuse the plurality of pre-trained diffusion models based on noise residuals of the plurality of pre-trained diffusion models to obtain fusion weights.

[0032] The generation module is configured to perform denoising prediction fusion on the plurality of pre-trained diffusion models based on the fusion weights, and iteratively train the diffusion model to generate target content.

[0033] The application further provides an image and video customization device based on multi-diffusion model fusion, comprising a memory and a processor, the memory is configured to store a computer program, and the processor is configured to implement the image and video customization method based on multi-diffusion model fusion when executing the computer program.

[0034] Compared with the prior art, the application has at least the following beneficial effects:

[0035] The image and video customization method and system based on multi-diffusion model fusion provided by the application are based on a pre-training module, a fusion module and a generation module, the weights of a plurality of pre-trained diffusion models are fused by selecting a fusion strategy, and output target content is generated based on the fusion weights. Compared with the prior art, the method supports the fusion of diffusion models with different architectures, training processes and sampling schedulers, so that the diffusion models can be fused without training or architecture modification. Based on the complementary advantages of each diffusion model, the generated images and videos have higher quality in terms of visual realism, identity preservation and style consistency, and can provide customized solutions for different scenes and user needs, and are suitable for complex real-world scenarios based on diffusion theory. The application further provides an image and video customization device based on multi-diffusion model fusion, which implements the image and video customization method based on multi-diffusion model fusion. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description.

[0037] Figure 1 Flowchart of the image and video customization method based on multi-diffusion model fusion;

[0038] Figure 2 Structure diagram of the image and video customization system based on multi-diffusion model fusion;

[0039] Figure 3 Application diagram of the image and video customization based on multi-diffusion model fusion in dance video generation;

[0040] Figure 4 Application diagram of the image and video customization based on multi-diffusion model fusion in role image stylization. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely intended to explain the present application and do not limit the protection scope of the present application.

[0042] In order to effectively fuse heterogeneous models and provide personalized and high-quality generated content without training, the embodiments provide a multi-diffusion model fusion image and video customization method, which fuses the weights of multiple pre-trained diffusion models by selecting a fusion strategy, generates an output image or video based on the fused weights, and is used to fuse multiple diffusion models with different architectures, training settings and sampling schedules, fully utilizes their complementary advantages to improve the quality of generated images and videos. As shown in Figure 1 The multi-diffusion model fusion image and video customization method provided by the embodiments includes the following steps:

[0043] S1, providing multiple pre-trained diffusion models and constructing a dataset based on the needs of diffusion model fusion targets.

[0044] In the embodiments, first, the pre-trained diffusion models that need to be fused are loaded, denoted as model A and model B, but not limited to these two models. Each model can have different architectures, training processes and sampling schedulers. Then, the required conditional inputs for each model are prepared to construct a dataset, which can include text prompts, images, pose sequences or other data related to the model target, to ensure that the model can perform the denoising task according to the specific conditions and generate the target content.

[0045] S2, based on the multiple pre-trained diffusion models, calculating the noise residuals of each diffusion model, and dynamically calculating the fusion weights through a fusion strategy;

[0046] The fusion strategy is one or more of fixed denoising time step switching denoising U-Net network, equal proportion fusion combining noise features of multiple pre-trained diffusion models, dynamic calculation based on noise features of multiple pre-trained diffusion models, and manual adjustment based on requirements of output quality of multiple pre-trained diffusion models.

[0047] In the embodiments, the multiple pre-trained diffusion models provided all include a denoising U-Net network and a corresponding sampling noise scheduler. Based on each denoising time step in the diffusion process of the multiple pre-trained diffusion models, the noise residuals of the current time step are calculated for each model. The calculation of the noise residuals depends on the denoising U-Net network, the noise latent variable and the conditional input, and the calculation formula is

[0048] L i，t =U i (zt ,t,c i )

[0049] where z is the noise residual of the i-th diffusion model at the current time step t, t is the noise latent variable of the i-th diffusion model at the current time step t, c i is the conditional input of the i-th diffusion model; in this embodiment, the noise residuals of model A and model B are calculated and recorded as and

[0050] After obtaining the noise residual, the fusion weight is dynamically calculated through the fusion strategy; the fusion weight can be set using multiple strategies: (1) Switch the denoising U-Net network at a fixed denoising time step, for example, use U-Net-1 for time steps 1 to 5 and use U-Net-2 for time steps 6 to 10.

[0051] (2) The noise characteristics of multiple pre-trained diffusion models are combined by proportional fusion. The proportional fusion formula is:

[0052]

[0053] Among them, L t represents the noise residual of the fused diffusion model, represents the noise residual of the diffusion model A, represents the noise residual of diffusion model B.

[0054] (3) Dynamic calculation of noise features based on multiple pre-trained diffusion models: Save the noise residual of each diffusion model at each denoising time step, and calculate the fusion weight of each denoising time step based on the noise latent variable, noise difference or noise ratio obtained in each denoising process; the noise latent variable is expressed as ∑|latent i，t |, the noise difference is expressed as ∑|latent i，t -latent i，t-1 |, the noise ratio is expressed as Among them, latent i，t Represented as the noise latent variable of the i-th diffusion model at the current time step t, latent i，t-1 is represented as the noise latent variable of the i-th diffusion model at the previous time step t-1.

[0055] (4) Manual adjustment based on the output quality requirements of multiple pre-trained diffusion models.

[0056] S3. Based on the fusion weights, multiple pre-trained diffusion models are fused for denoising and prediction, and the target generation task is completed through iterative training.

[0057] In an embodiment, after determining the fusion weight, the noise residuals of model A and model B are adaptively fused to generate the fusion noise residual at the current time step, and the fusion formula is:

[0058]

[0059] wherein, represents the noise residual of diffusion model A, represents the noise residual of diffusion model B, and t represents the fusion weight determined based on the fusion strategy.

[0060] The fused noise residual L t is then used to update the latent variable of the diffusion model, and this process is iterated in all denoising time steps, with the time step t decreasing from T to 1, gradually optimizing the state of the latent variable until the diffusion process ends.

[0061] Finally, after all denoising time steps (1, T) are completed, the final latent variable is decoded using the decoder to generate the output image or video. The decoder can be the decoder provided with the basic diffusion model, or other decoders compatible with the model.

[0062] This method fully utilizes the complementary advantages of model A and model B by dynamically adjusting the fusion parameters, achieving high-quality denoising effect and flexible adaptation to different task requirements.

[0063] As Figure 2 shown is an image and video customization system for multiple diffusion model fusion, which includes multiple core components. First is the pre-training module, which includes a set of diffusion models, each composed of a denoising U-Net network and a corresponding sampling noise scheduler. These diffusion models may have different architectures, such as variations in denoising U-Net network structure, or training on different datasets and target tasks.

[0064] Second is the fusion module for selecting a fusion strategy based on the noise residuals of multiple pre-trained diffusion models to obtain a fusion weight.

[0065] Finally, the generation module is used to perform denoising prediction fusion based on the fusion weight of the multiple pre-trained diffusion models to iteratively train the diffusion model to generate the target content.

[0066] In an embodiment, an image and video customization device for multiple diffusion model fusion is also provided, including a memory and a processor, the memory is used to store a computer program, and the processor is used to implement the image and video customization method for multiple diffusion model fusion when executing the computer program.

[0067] To better illustrate the technical effects of the present application, the above method and system are applied to the generation of dance videos and the stylization of character images.

[0068] As shown in Figure 3 The application of image and video customization based on multi-diffusion model fusion in the generation of dance videos is shown in the schematic diagram.

[0069] First of all, AnimateAnyone-Net is a pre-trained video diffusion model specifically designed to capture motion dynamics, capable of generating the overall motion and appearance of a character. The reference image extracts appearance features through a VAE encoder and a CLIP text encoder, and the pose guide encodes the input pose sequence. This embodiment uses the pre-trained model AnimateAnyone provided by Moore Threads, which supports learning transferable visual models from natural language supervision.

[0070] Secondly, Face-Net is a pre-trained video diffusion model specifically optimized for facial features and expressions, capable of effectively preserving the identity features of a character. The facial image extracts identity features through InsightFace, and its underlying diffusion model is derived from a latent diffusion model for high-resolution video generation. This embodiment also uses ControlNet to encode the full-body pose for more accurate pose control.

[0071] In addition, the system uses a mask to identify the facial region in the video frame. The mask is extracted by the Facer model and combined with the video generated by AnimateAnyone for data preprocessing. The mask ensures that the facial region can benefit from the optimization of Face-Net during the fusion process, while the other parts of the video are controlled by AnimateAnyone-Net.

[0072] In the specific implementation steps, first, the AnimateAnyone-Net and Face-Net models need to be loaded, each containing its denoising U-Net network and corresponding sampling noise scheduler. Next, the input data needs to be prepared, including the character reference image used to extract identity information and the pose sequence representing the target dance action. Among them, the conditional input of AnimateAnyone-Net is the pose sequence and the reference image, and the conditional input of Face-Net is the facial identity information extracted from the reference image and the pose sequence.

[0073] In each denoising time step of the diffusion process, first, calculate the noise residuals of AnimateAnyone-Net and Face-Net, denoted as and Subsequently, according to the selected fusion strategy, calculate the fusion weight αt For example, manually adjust the weights to prioritize the optimization of facial details. Then, apply the fusion operation in the face region using the face mask mask, with the fusion formula as follows:

[0074]

[0075] wherein, is the noise residual in the dance video generation application, is the fusion weight in the dance video generation application, and M is the fusion operation of the mask mask.

[0076] Based on the fusion formula, the face region can benefit from the dedicated optimization of Face-Net, while the rest of the video frame is generated by AnimateAnyone-Net, maintaining the overall motion dynamics.

[0077] After completing the operation of each time step, the system updates the latent variable and performs the above process iteratively for all denoising time steps. Finally, the final latent variable is decoded using the decoder to generate the dance video. The output video not only has smooth and coherent dance movements, but also retains the identity features of the character and optimizes the facial details. This method can generate high-quality video content, showing smooth and realistic dance dynamics.

[0078] As Figure 4 shows the application of multi-diffusion model fusion in image and video customization in character image stylization.

[0079] First, the system includes a ControlNet-based basic diffusion model that can provide structured generation guidance based on pose information. The basic diffusion model is derived from a latent diffusion model for high-resolution video generation, while ControlNet focuses on accurate estimation of full-body pose, providing strong conditional control capabilities. In addition, the system loads two low-rank adaptive LoR models, including: a character LoRA model that focuses on maintaining the identity features of the character, and a style LoRA model that generates images with a specific artistic style. These pre-trained models are derived from the Multi-LoRA combined model, supporting efficient and diverse image generation.

[0080] In the specific implementation steps, first, provide the ControlNet-based basic diffusion model, and load the character LoRA and style LoRA models. These models are associated with the basic diffusion model to form a complete generation framework. Next, prepare the input data, including the pose information of the character, which is used to provide structured guidance in the image; the identity input of the character, usually in the form of a reference image or an embedding, to represent the unique features of the character; and the style description, which can be a text prompt or an embedding that defines the desired artistic style.

[0081] In each denoising time step of the diffusion process, first, the noise residuals of the role LoRA model and the style LoRA model are calculated respectively, denoted as and Subsequently, the fusion weight is dynamically set according to the needs of different time steps. For example, a higher weight can be allocated to the style model in the early time steps to highlight the artistic style, and the weight of the role model is gradually increased in the later time steps to better maintain the role identity features. The noise residuals of the two models are fused through the following formula:

[0082]

[0083] wherein, represents the noise residual in the role image stylization application, represents the fusion weight in the role image stylization application.

[0084] The fusion formula ensures a reasonable balance between the style and the role features in the image generation process.

[0085] After completing the fusion of each time step, the latent variable is updated, and the above process is repeated for all denoising time steps. After the diffusion process ends, the final latent variable is decoded using the decoder to generate the role image. The output image not only accurately presents the unique features of the role, but also integrates the specified artistic style while maintaining the accuracy of the role's pose structure. The finally generated image exhibits a perfect combination of the role's identity features and the artistic style, providing an efficient and flexible solution for high-quality role image stylization.

[0086] The specific embodiments described above have detailed the technical solutions and beneficial effects of the present application. It should be understood that the above description is only the most preferred embodiment of the present application and is not intended to limit the present application. Any modifications, supplements, and equivalent replacements made within the principle range of the present application should be included in the protection scope of the present application.

Claims

1. A method for customizing images and videos by fusion of multiple diffusion models, characterized in that: The following steps are involved: Step 1: Providing multiple pre-trained diffusion models and constructing a dataset based on the needs of the diffusion model fusion target, wherein the multiple pre-trained diffusion models include a denoising U-Net network and a corresponding noise scheduler, wherein the multiple pre-trained diffusion models have different architectures, training processes, and noise schedulers; Step 2: Based on multiple pre-trained diffusion models, calculate the noise residual of each diffusion model and dynamically calculate the fusion weight through the fusion strategy; The fusion strategy is one or more of switching the denoising U-Net network at a fixed denoising time step, combining the noise features of multiple pre-trained diffusion models through proportional fusion, dynamically calculating the noise features based on multiple pre-trained diffusion models, and manually adjusting the output quality requirements of multiple pre-trained diffusion models; wherein, based on the noise features of multiple pre-trained diffusion models, the fusion weight is dynamically calculated, including: saving the noise residual of each diffusion model at each denoising time step, and calculating the fusion weight of each denoising time step based on the noise latent variable, noise difference or noise ratio obtained in each denoising process; The noise latent variable is expressed as , the noise difference is expressed as , the noise ratio is expressed as ,in, Represents the current time step The next The noise latent variables of the diffusion model, Represents the previous time step The next Noise latent variables of the diffusion model; Step 3: Based on the fusion weights, multiple pre-trained diffusion models are fused for denoising prediction and iterative training to complete the target generation task, including: the noise residual after fusion This is then used to update the latent variables of the diffusion model and this process is iterated over all denoising time steps. from It gradually decreases to 1, and gradually optimizes the state of the hidden variable until the diffusion process ends; Finally, at all denoising time steps Once completed, the final latent variables are decoded using a decoder to generate the output image or video.

2. The image and video customization method according to claim 1, characterized in that: The dataset described in step 1 includes textual prompts, images, pose sequences, or other data related to the model's goals.

3. The image and video customization method according to claim 1, characterized in that: The switching of the denoising U-Net network at a fixed denoising time step includes: Using the denoising U-Net-1 network, at a fixed denoising time step Using the denoising U-Net-2 network, at a fixed denoising time step Use a denoising U-Net-3 network with a fixed denoising time step and a fixed denoising time step All in the denoising time step within the range.

4. The image and video customization method according to claim 1, wherein: The noise characteristics of multiple pre-trained diffusion models are combined by proportional fusion, including: calculating the noise residual of each pre-trained diffusion model respectively, and fusing the weights of multiple pre-trained diffusion models based on the proportional fusion method, which is expressed as: ， in, represents the noise residual of the fused diffusion model, represents the noise residual of the first diffusion model, represents the noise residual of another diffusion model.

5. A system for customizing images and videos by integrating multiple diffusion models, characterized in that: The image and video customization method using the multi-diffusion model fusion according to any one of claims 1 to 4 comprises: a pre-training module, a fusion module and a generation module; The pre-training module is used to construct a data set based on the needs of the diffusion model fusion target and calculate the noise residuals of multiple pre-trained diffusion models; The fusion module is used to select a fusion strategy to fuse multiple pre-trained diffusion models based on the noise residuals of multiple pre-trained diffusion models to obtain a fusion weight; The generation module is used to perform denoising prediction fusion on multiple pre-trained diffusion models based on fusion weights, and iteratively train the diffusion models to generate target content.

6. A device for customizing images and videos using multiple diffusion models, comprising a memory and a processor, wherein the memory is used to store a computer program, characterized in that: The processor is configured to implement the image and video customization method for multi-diffusion model fusion according to any one of claims 1 to 4 when executing the computer program.

Images