Multi-diffusion model fused image and video customization method, system and device
By dynamically calculating the fusion weights and generating the output content based on the weights, the problem of difficult to effectively fusion diffusion models with different architectures and training settings in the prior art is solved, high-quality image and video generation is achieved, and computing efficiency is improved.
Patent Information
- Application Number
- CN202411822988.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-10
- Publication Date
- 2025-03-21
- Estimated Expiration
- 2044-12-10
AI Technical Summary
The existing diffusion model fusion strategy has limitations, and it is difficult to effectively fuse multiple diffusion models with different architectures, training settings and sampling scheduling, resulting in low quality of generated images and videos and low computational efficiency.
The fusion of diffusion models with different architectures and training settings is achieved by selecting the appropriate fusion strategy to fuse the weights of multiple pre-trained diffusion models, dynamically compute the fusion weights, and generate output images or videos based on the fusion weights.
This method can fully utilize the complementary advantages of multiple diffusion models, and the generated images and videos have higher quality in visual realism, identity maintenance and style consistency, and are suitable for applications in complex real-life scenarios, and can achieve model fusion without training or architectural modification.
Smart Images

Figure CN119676532A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision and pattern recognition, and in particular to a method, system and device for customizing images and videos by fusing multiple diffusion models. Background Art
[0002] Generative modeling has become a cornerstone of the development of artificial intelligence, especially in the field of image and video synthesis. Diffusion models have attracted much attention for their ability to generate high-quality, high-fidelity images and videos. Diffusion models gradually denoise random noise.
[0003] Generate coherent data samples and effectively capture complex data distributions.
[0004] In recent years, specialized diffusion models have been developed to perform specific tasks or to generate content with specific properties. For example:
[0005] Style-customized models: In digital art creation and style transfer applications, these style-customized models are used to generate images with a specific artistic style, which can transform ordinary photos into works of art in the style of famous artists or specific art genres.
[0006] Identity-customized models: In applications such as personalized avatar creation, virtual reality, and character animation, it is critical to maintain the unique identity of an individual. These identity-customized models ensure that the generated content accurately reflects the facial features and expressions of a specific individual.
[0007] Motion dynamics models: Capturing realistic motion dynamics is key for video generation and animation. These motion dynamics models generate sequences that exhibit smooth and coherent motion, making them ideal for applications such as filmmaking, gaming, and simulation.
[0008] Although these specialized models have made significant progress in their respective fields, they still face significant challenges when it comes to fusing multiple specialized capabilities into a single generation process. Real-world applications often require such multifaceted generation capabilities. For example, creating a personalized animated video requires not only capturing realistic motion but also maintaining individual identity while applying a specific artistic style.
[0009] As a powerful generative tool, the diffusion model has made great progress. However, the existing diffusion models mainly focus on single-mode control, that is, the diffusion process is driven by only one modal condition. In order to further adapt to various personalized needs, the existing model fusion strategy has been developed, but it has the following limitations:
[0010] 1. Homogeneous model requirements: Traditional fusion methods usually require that the fused models have similar architectures and training paradigms. This homogeneity is a prerequisite for the normal operation of techniques such as weighted averaging. For example, reference 1 (Collaborative Diffusion for Multi-Modal Face Generation and Editing, 2023, Computer Vision and Pattern Recognition, DOI: arxiv-2304.10530) proposed the collaborative diffusion of homogeneous models, which collaboratively realizes multimodal face generation capabilities based on multiple pre-trained unimodal diffusion models, and integrates multiple unimodal operations to perform multimodal editing. However, this multimodal collaborative diffusion model fuses homogeneous models, which has limitations for the fusion of heterogeneous models, and the models used for generation need to be trained, which increases the computational overhead and reduces the efficiency of computing task processing.
[0011] 2. Weight-based fusion: The traditional fusion method is to average or combine the weights of the model to achieve fusion. For example, reference 2 (ZipLoRA: Any Subject inAny Style by Effectively Merging LoRAs, 2023, Computer Vision and Pattern Recognition, DOI: rxiv-2311.13600) proposed ZipLoRA, which combines independently trained style and theme LoRA methods to achieve any user-provided style to generate any user-provided theme. However, this method assumes that the models have the same architecture and compatible parameterization. If the model differences are large, weight-based fusion may lead to performance degradation or convergence problems.
[0012] 3. Switching-based fusion: Use different models at different stages of the generation process. For example, reference 3 (Multi-LoRA Composition for Image Generation, 2024, Computer Vision and Pattem Recognition, DOI: arxiv-2402.16843) proposes two untrained methods centered on decoding: LoRA Switch is used to alternately use different LoRAs in each denoising step and LoRA Composite is used to combine all LoRAs to guide more coherent image synthesis. Based on these two untrained methods, multiple LoRA combinations are studied to achieve more complex image generation. However, due to the different training objectives and data distribution of the model, this approach may lead to inconsistencies and artifacts.
[0013] 4. Architecture compatibility constraints: Many fusion methods rely on the structural compatibility of models. Models with different network architectures, layer configurations, or activation functions are significant challenges for these methods and often lead to ineffective fusion.
[0014] These limitations hinder the ability to fully exploit the complementary strengths of diverse models. Existing fusion strategies often fail or produce suboptimal results when attempting to fuse specialized models independently developed with different architectures and training settings, which also hinders progress on complex generation tasks. Summary of the invention
[0015] The purpose of the present invention is to provide a method, system and device for customizing images and videos by fusing multiple diffusion models. The method selects a fusion strategy to fuse the weights of multiple pre-trained diffusion models, and generates an output image or video based on the fusion weights. The method is used to fuse multiple diffusion models with different architectures, training settings and sampling schedules, and make full use of their complementary advantages to improve the quality of generated images and videos.
[0016] To achieve the above-mentioned purpose of the invention, an embodiment provides a method for customizing images and videos by fusion of multiple diffusion models, comprising the following steps:
[0017] Step 1: Provide multiple pre-trained diffusion models and construct a dataset based on the needs of diffusion model fusion objectives;
[0018] Step 2: Based on multiple pre-trained diffusion models, calculate the noise residual of each diffusion model, and dynamically calculate the fusion weight through the fusion strategy;
[0019] The fusion strategy is one or more of switching the denoising U-Net network with a fixed denoising time step, combining the noise features of multiple pre-trained diffusion models by equal-proportional fusion, dynamically calculating the noise features based on multiple pre-trained diffusion models, and manually adjusting the output quality requirements based on multiple pre-trained diffusion models;
[0020] Step 3: Based on the fusion weights, multiple pre-trained diffusion models are fused for denoising prediction and iterative training is performed to complete the target generation task.
[0021] In one embodiment, the multiple pre-trained diffusion models described in step 1 include a denoising U-Net network and a corresponding noise scheduler, wherein the multiple pre-trained diffusion models have different architectures, training processes, and noise schedulers.
[0022] In one embodiment, the dataset described in step 1 includes text prompts, images, pose sequences, or other data related to the model target.
[0023] In one embodiment, the switching of the denoising U-Net network at the fixed denoising time step includes: using the denoising U-Net-1 network at the fixed denoising time step (1, t), using the denoising U-Net-2 network at the fixed denoising time step (t+1, t+n), and using the denoising U-Net-3 network at the fixed denoising time step (t+n, T), wherein the fixed denoising time step t and the fixed denoising time step n are both within the range of the denoising time step (1, T).
[0024] In one embodiment, the noise features of multiple pre-trained diffusion models are combined by equal-proportional fusion, including: respectively calculating the noise residual of each pre-trained diffusion model, and fusing the weights of multiple pre-trained diffusion models based on the equal-proportional fusion method, which is expressed as:
[0025]
[0026] Among them, L t represents the noise residual of the fused diffusion model, represents the noise residual of the first diffusion model, represents the noise residual of another diffusion model.
[0027] In one embodiment, the noise characteristics based on multiple pre-trained diffusion models and the dynamic calculation of fusion weights include: saving the noise residual of each diffusion model at each denoising time step, and calculating the fusion weight of each denoising time step based on the noise latent variable, noise difference or noise ratio obtained in each denoising process;
[0028] The noise latent variable is represented by ∑|latent i,t |, the noise difference is expressed as ∑|latent i,t -latent i,t-1 |, the noise ratio is expressed as Among them, latent i,t Represented as the noise latent variable of the i-th diffusion model at the current time step t, latent i,t-1 is represented as the noise latent variable of the i-th diffusion model at the previous time step t-1.
[0029] The present invention also provides a system for customizing images and videos by fusion of multiple diffusion models, including: a pre-training module, a fusion module and a generation module;
[0030] The pre-training module is used to construct a data set based on the needs of the diffusion model fusion target and calculate the noise residuals of multiple pre-trained diffusion models;
[0031] The fusion module is used to select a fusion strategy to fuse multiple pre-trained diffusion models based on the noise residuals of multiple pre-trained diffusion models to obtain a fusion weight;
[0032] The generation module is used to perform denoising prediction fusion on multiple pre-trained diffusion models based on fusion weights, and iteratively train the diffusion models to generate target content.
[0033] The present invention also provides an image and video customization device for multi-diffusion model fusion, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to implement the image and video customization method for multi-diffusion model fusion when executing the computer program.
[0034] Compared with the prior art, the present invention has the following beneficial effects:
[0035] The image and video customization method and system for multi-diffusion model fusion provided by the present invention are based on a pre-training module, a fusion module and a generation module, and the weights of multiple pre-trained diffusion models are fused by selecting a fusion strategy, and the output target content is generated based on the fusion weights. Compared with the existing methods, the method supports the fusion of diffusion models with different architectures, training processes and sampling schedulers, so that the diffusion models can be fused without training or architecture modification; and based on the complementary advantages of each diffusion model, the generated images and videos have higher quality in visual realism, identity retention and style consistency, and can provide customized solutions for different scenes and user needs, which is suitable for applications in complex real-life scenes based on diffusion theory; the present invention also provides an image and video customization device for multi-diffusion model fusion, which realizes the image and video customization method for multi-diffusion model fusion. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.
[0037] Figure 1 Flowchart of the method for customizing images and videos for multi-diffusion model fusion;
[0038] Figure 2 Schematic diagram of the structure of the image and video customization system for multi-diffusion model fusion;
[0039] Figure 3 Schematic diagram of the application of image and video customization of multi-diffusion model fusion in dance video generation;
[0040] Figure 4 Schematic diagram of the application of image and video customization fused by multi-diffusion model in character image stylization. DETAILED DESCRIPTION
[0041] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.
[0042] In order to effectively fuse heterogeneous models and provide personalized and high-quality generated content without training, the embodiment provides a method for customizing images and videos by fusing multiple diffusion models. The method fuses the weights of multiple pre-trained diffusion models by selecting a fusion strategy, and generates an output image or video based on the fusion weights. The method is used to fuse multiple diffusion models with different architectures, training settings, and sampling schedules, and fully utilize their complementary advantages to improve the quality of generated images and videos. Figure 1 As shown, the method for customizing images and videos by fusing multiple diffusion models provided in the embodiment includes the following steps:
[0043] S1. Provide multiple pre-trained diffusion models and construct datasets based on the needs of diffusion model fusion objectives.
[0044] In the embodiment, the pre-trained diffusion models to be fused are first loaded, which are referred to as model A and model B in the present invention, but are not limited to these two models. Each model may have a different architecture, training process, and sampling scheduler. Next, the required conditional inputs are prepared for each model to build a data set. These inputs may include text prompts, images, posture sequences, or other data related to the model's objectives to ensure that the model can perform denoising tasks according to specific conditions and generate target content.
[0045] S2, based on multiple pre-trained diffusion models, calculate the noise residual of each diffusion model, and dynamically calculate the fusion weight through the fusion strategy;
[0046] The fusion strategy is one or more of switching the denoising U-Net network at a fixed denoising time step, combining the noise features of multiple pre-trained diffusion models by equal-proportional fusion, dynamically calculating the noise features based on multiple pre-trained diffusion models, and manually adjusting the output quality requirements based on multiple pre-trained diffusion models.
[0047] In the embodiment, the multiple pre-trained diffusion models provided all include a denoising U-Net network and a corresponding sampling noise scheduler. Based on each denoising time step of the multiple pre-trained diffusion models in the diffusion process, the noise residual of the current time step is calculated for each model. The calculation of the noise residual depends on the denoising U-Net network, the noise latent variable and the conditional input. The calculation formula is:
[0048] L i,t =U i (zt ,t,c i )
[0049] Where, represents the noise residual of the i-th diffusion model at the current time step t, z t is the noise latent variable of the i-th diffusion model at the current time step t, c i is the conditional input of the i-th diffusion model; in this embodiment, the noise residuals of model A and model B are calculated and recorded as and
[0050] After obtaining the noise residual, the fusion weight is dynamically calculated through the fusion strategy. There are many strategies for setting the fusion weight: (1) Switch the denoising U-Net network at a fixed denoising time step, for example, use U-Net-1 for time steps 1 to 5 and use U-Net-2 for time steps 6 to 10.
[0051] (2) The noise features of multiple pre-trained diffusion models are combined by proportional fusion. The proportional fusion formula is:
[0052]
[0053] Among them, L t represents the noise residual of the fused diffusion model, represents the noise residual of the diffusion model A, represents the noise residual of diffusion model B.
[0054] (3) Dynamic calculation of noise features based on multiple pre-trained diffusion models: Save the noise residual of each diffusion model at each denoising time step, and calculate the fusion weight of each denoising time step based on the noise latent variable, noise difference or noise ratio obtained in each denoising process; the noise latent variable is expressed as ∑|latent i,t |, the noise difference is expressed as ∑|latent i,t -latent i,t-1 |, the noise ratio is expressed as Among them, latent i,t Represented as the noise latent variable of the i-th diffusion model at the current time step t, latent i,t-1 is represented as the noise latent variable of the i-th diffusion model at the previous time step t-1.
[0055] (4) Manual adjustment based on the output quality requirements of multiple pre-trained diffusion models.
[0056] S3. Based on the fusion weights, multiple pre-trained diffusion models are fused for denoising prediction and iterative training is performed to complete the target generation task.
[0057] In the embodiment, after the fusion weight is determined, the noise residuals of model A and model B are adaptively fused to generate the fused noise residual of the current time step, and the fusion formula is:
[0058]
[0059] in, represents the noise residual of the diffusion model A, represents the noise residual of diffusion model B, α t Represents the fusion weight determined based on the above fusion strategy.
[0060] The noise residual L after fusion t It is then used to update the latent variables of the diffusion model, and this process is iterated in all denoising time steps, with the time step t gradually decreasing from T to 1, gradually optimizing the state of the latent variables until the diffusion process ends.
[0061] Finally, after all denoising time steps (1, T) are completed, the final latent variable is decoded using a decoder to generate an output image or video. The decoder can be the decoder that comes with the basic diffusion model or other decoders that are compatible with the model.
[0062] This method fully utilizes the complementary advantages of model A and model B by dynamically adjusting the fusion parameters. It can not only achieve high-quality denoising effects, but also flexibly adapt to the needs of different tasks.
[0063] like Figure 2 The image and video customization system of multi-diffusion model fusion is shown, including several core components. The first is the pre-training module, which contains a set of diffusion models, each of which consists of a denoising U-Net network and a corresponding sampling noise scheduler. These diffusion models may have different architectures, such as changes in the denoising U-Net network structure, or be trained on different data sets and target tasks.
[0064] Secondly, the fusion module is used to select a fusion strategy based on the noise residuals of multiple pre-trained diffusion models to fuse multiple pre-trained diffusion models to obtain the fusion weight.
[0065] Finally, the generation module is used to perform denoising prediction fusion on multiple pre-trained diffusion models based on fusion weights, and iteratively train the diffusion model to generate target content.
[0066] An embodiment also provides a device for customizing images and videos by fusion of multiple diffusion models, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to implement a method for customizing images and videos by fusion of multiple diffusion models when executing the computer program.
[0067] In order to better illustrate the technical effects of the present invention, the specific application of the above method and system in dance video generation and character image stylization is demonstrated.
[0068] like Figure 3 Shown is a schematic diagram of the application of image and video customization based on multi-diffusion model fusion in dance video generation.
[0069] First, AnimateAnyone-Net is a pre-trained video diffusion model specifically designed to capture motion dynamics, which can generate the overall motion and appearance of the character. The reference image is extracted from the appearance features through the VAE encoder and the CLIP text encoder, and the posture guide encodes the input posture sequence. This embodiment uses the pre-trained model AnimateAnyone provided by Moore Threads, which supports learning transferable visual models from natural language supervision.
[0070] Secondly, Face-Net is a pre-trained video diffusion model that specifically optimizes facial features and expressions, and can effectively maintain the identity characteristics of the character. The facial image extracts identity features through InsightFace, and its basic diffusion model is derived from the potential diffusion model generated by high-resolution video. This embodiment also uses ControlNet to encode the whole body posture to achieve more precise posture control.
[0071] In addition, the system uses a mask to identify the facial area in the video frame. The mask is extracted by the Facer model and combined with the video generated by AnimateAnyone for data preprocessing. The mask ensures that during the fusion process, the facial area can benefit from the optimization of Face-Net, while the rest of the video is controlled by AnimateAnyone-Net.
[0072] In the specific implementation steps, you first need to load the AnimateAnyone-Net and Face-Net models, each of which contains its denoising U-Net network and the corresponding sampling noise scheduler. Next, you need to prepare the input data, including the character reference image used to extract identity information and the pose sequence representing the target dance movement. Among them, the conditional input of AnimateAnyone-Net is the pose sequence and the reference image, and the conditional input of Face-Net is the facial identity information extracted from the reference image and the pose sequence.
[0073] In each denoising time step of the diffusion process, the noise residuals of AnimateAnyone-Net and Face-Net are first calculated respectively, denoted as and Then, the fusion weight α is calculated according to the selected fusion strategyt , for example, manually adjusting the weights to prioritize the optimization of facial details. Next, the fusion operation is applied to the facial area using the facial mask. The fusion formula is as follows:
[0074]
[0075] in, Denoted as the noise residual in the dance video generation application, It is represented as the fusion weight in the dance video generation application, and M is represented as the fusion operation of the mask.
[0076] This fusion formula ensures that the facial area benefits from the dedicated optimization of Face-Net, while the rest of the video frame is generated by AnimateAnyone-Net, maintaining the overall motion dynamics.
[0077] After completing the operation at each time step, the system updates the latent variables and iterates the above process for all denoising time steps. Finally, the final latent variables are decoded using a decoder to generate a dance video. The output video not only has smooth and coherent dance movements, but also preserves the identity characteristics of the character and optimizes facial details. This method can generate high-quality video content that shows smooth and realistic dance dynamics.
[0078] like Figure 4 Shown is a schematic diagram of the application of image and video customization based on multi-diffusion model fusion in character image stylization.
[0079] First, the system includes a basic diffusion model based on ControlNet, which can provide structured generation guidance based on posture information. The basic diffusion model is derived from the potential diffusion model generated by high-resolution videos, while ControlNet focuses on the accurate estimation of full-body posture and provides powerful conditional control capabilities. In addition, the system loads two low-rank adaptive LoR models, including: the character LoRA model focuses on maintaining the identity characteristics of the character, and the style LoRA model is used to generate images with a specific artistic style. These pre-trained models are derived from the Multi-LoRA combination model and support efficient and diverse image generation.
[0080] In the specific implementation steps, we first need to provide a basic diffusion model based on ControlNet and load the character LoRA and style LoRA models. These models are associated with the basic diffusion model to form a complete generation framework. Next, prepare the input data, including the character's pose information, which is used to provide structured guidance in the image; the character's identity input, which is usually provided in the form of a reference image or embedding to represent the character's unique features; and the style description, which can be a text prompt or embedding that defines the desired artistic style.
[0081] In each denoising time step of the diffusion process, the noise residuals of the character LoRA model and the style LoRA model are first calculated respectively, denoted as and Then, the fusion weights are dynamically set according to the needs of different time steps. For example, in the early time steps, a higher weight can be assigned to the style model to highlight the artistic style; while in the later time steps, the weight of the character model is gradually increased to better maintain the character identity characteristics. The noise residuals of the two models are fused by the following formula:
[0082]
[0083] in, Denoted as the noise residual in character image stylization application, Denoted as fusion weights in character image stylization applications.
[0084] Based on this fusion formula, it is ensured that the style and character characteristics are reasonably balanced during the image generation process.
[0085] After the fusion of each time step is completed, the latent variables are updated and the above process is repeated for all denoised time steps. After the diffusion process is completed, the final latent variables are decoded using a decoder to generate a character image. The output image not only accurately presents the unique characteristics of the character, but also incorporates the specified artistic style while maintaining the accuracy of the character's posture structure. The final generated image shows a perfect combination of the character's identity characteristics and artistic style, providing an efficient and flexible solution for high-quality character image stylization.
[0086] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for customizing images and videos by fusion of multiple diffusion models, characterized in that: The following steps are involved: Step 1: Provide multiple pre-trained diffusion models and construct a dataset based on the needs of diffusion model fusion objectives; Step 2: Based on multiple pre-trained diffusion models, calculate the noise residual of each diffusion model, and dynamically calculate the fusion weight through the fusion strategy; The fusion strategy is one or more of switching the denoising U-Net network with a fixed denoising time step, combining the noise features of multiple pre-trained diffusion models by equal-proportional fusion, dynamically calculating the noise features based on multiple pre-trained diffusion models, and manually adjusting the output quality requirements based on multiple pre-trained diffusion models; Step 3: Based on the fusion weights, multiple pre-trained diffusion models are fused for denoising prediction and iterative training is performed to complete the target generation task.
2. The image and video customization method according to claim 1, characterized in that: The multiple pre-trained diffusion models described in step 1 include a denoising U-Net network and a corresponding noise scheduler, wherein the multiple pre-trained diffusion models have different architectures, training processes and noise schedulers.
3. The image and video customization method according to claim 1, characterized in that: The dataset described in step 1 includes textual prompts, images, pose sequences, or other data related to the model's goals.
4. The image and video customization method according to claim 1, characterized in that: The switching of the denoising U-Net network at the fixed denoising time step includes: using the denoising U-Net-1 network at the fixed denoising time step (1, t), using the denoising U-Net-2 network at the fixed denoising time step (t+1, t+n), and using the denoising U-Net-3 network at the fixed denoising time step (t+n, T), wherein the fixed denoising time step t and the fixed denoising time step n are both within the range of the denoising time step (1, T).
5. The image and video customization method according to claim 1, characterized in that: The noise features of multiple pre-trained diffusion models are combined by equal-proportional fusion, including: respectively calculating the noise residual of each pre-trained diffusion model, and fusing the weights of multiple pre-trained diffusion models based on the equal-proportional fusion method, which is expressed as: Among them, L t represents the noise residual of the fused diffusion model, represents the noise residual of the first diffusion model, represents the noise residual of another diffusion model.
6. The image and video customization method according to claim 1, characterized in that: The noise characteristics based on multiple pre-trained diffusion models and the dynamic calculation of fusion weights include: saving the noise residual of each diffusion model at each denoising time step, and calculating the fusion weight of each denoising time step based on the noise latent variables, noise differences or noise ratios obtained in each denoising process; The noise latent variable is represented by ∑|latent i,t |, the noise difference is expressed as ∑|latent i,t -latent i,t-1 |, the noise ratio is expressed as Among them, latent i,t Represented as the noise latent variable of the i-th diffusion model at the current time step t, latent i,t-1 is represented as the noise latent variable of the i-th diffusion model at the previous time step t-1.
7. A system for customizing images and videos by integrating multiple diffusion models, comprising: Pre-training module, fusion module and generation module; The pre-training module is used to construct a data set based on the needs of the diffusion model fusion target and calculate the noise residuals of multiple pre-trained diffusion models; The fusion module is used to select a fusion strategy to fuse multiple pre-trained diffusion models based on the noise residuals of multiple pre-trained diffusion models to obtain a fusion weight; The generation module is used to perform denoising prediction fusion on multiple pre-trained diffusion models based on fusion weights, and iteratively train the diffusion models to generate target content.
8. An image and video customization device for multi-diffusion model fusion, comprising a memory and a processor, wherein the memory is used to store a computer program, characterized in that: The processor is used to implement the image and video customization method of multi-diffusion model fusion as described in any one of claims 1 to 7 when executing the computer program.
Citation Information
Patent Citations
Image data set expansion method based on diffusion model, medium and equipment
CN116883545A
Construction method and device of diffusion model
CN117522675A
Image super-resolution reconstruction method and system based on diffusion model
CN117522694A
Short-term power load prediction method based on ICEEMDAN and integrated LSTM-TCN model
CN117592615A
Video generation method, device and equipment
CN117788656A
Cited By
Motion video generation method and device, electronic equipment and storage medium
CN120264038A
Digital content generation method and system based on adaptive frequency domain fusion
CN121037646A
A digital content generation method and system based on adaptive frequency domain fusion
CN121037646B