Diffusion model fusion method, device and electronic equipment

Abstract

The present application discloses a diffusion model fusion method, device and electronic device. Relating to the field of artificial intelligence, the method comprises: obtaining text information, wherein the text information includes description information of the target image to be generated; determining a target style hint vector that matches the text information, wherein the target style hint vector is used to instruct a target student model to output an image in the target image style, and the target student model is a student model obtained by performing knowledge distillation based on a variety of teacher models with different image styles under a distributed training framework; inputting the text information and the target style hint vector into the target student model, and the target student model performs a diffusion generation process and outputs the target image. Through this application, the problem of low image quality generated by the fusion model obtained by the existing model fusion method in the related art when generating images of different styles is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to a diffusion model fusion method, device, and electronic device. Background Art

[0002] In the field of artificial intelligence, and particularly in generative models for deep learning, model diversity and specialization are core characteristics. As research deepens and technology matures, different models for generating text from images are often optimized for specific styles or tasks. For example, some models excel at generating highly realistic real-world images, while others excel at generating images in cartoon or artistic styles or within specific domains.

[0003] When it is necessary to generate multiple images of different styles, the server usually needs to deploy multiple models to control the generation of different styles, which greatly increases the overhead. Client users usually need to switch between multiple models, which is very inflexible and time-consuming to load on the hard disk and memory, affecting the user experience. To solve this problem, it is necessary to fuse multiple different style models into a target model that covers the knowledge of various expert fields, but there is currently no good solution to achieve this goal. Existing model fusion methods face many challenges when dealing with needs that span different fields or styles. For example, directly fusing a model that specializes in generating artistic style paintings with a cartoon image generation model often fails to properly handle the two distinct style differences, resulting in the generated image containing both style features and unable to clearly display either style, causing style confusion and affecting image quality and user experience.

[0004] There is currently no effective solution to the problem that the fusion model obtained by the existing model fusion method in the related art generates low image quality when generating images of different styles. Summary of the Invention

[0005] The main purpose of this application is to provide a diffusion model fusion method, device and electronic device to solve the technical problem in the related art that the fusion effect is poor when using the existing model fusion method to fuse models of different styles.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a diffusion model fusion method is provided. The method comprises: obtaining text information, wherein the text information includes description information of a target image to be generated; determining a target style hint vector that matches the text information, wherein the target style hint vector is used to instruct a target student model to output an image of a target image style, wherein the target student model is a student model obtained by performing knowledge distillation based on a plurality of teacher models with different image styles in a distributed training framework, wherein the distributed training framework deploys teacher models with different image styles on a plurality of image processing units, and the target student model is trained in parallel on the plurality of image processing units while sharing parameters; inputting the text information and the target style hint vector into the target student model, wherein the target student model performs a diffusion generation process and outputs a target image.

[0007] Furthermore, the text information and the target style cue vector are input into the target student model, and the target image is output through the diffusion generation process of the target student model, including: inputting the text information into the text embedding layer in the target student model to convert the text information into a text vector; based on the text vector and the target style cue vector, the target image is generated by modulating the generation mechanism of the target student model.

[0008] Furthermore, the target student model is obtained through the following steps: deploying teacher models of different image styles on multiple image processing units of a distributed training framework; determining the style hint vector corresponding to the teacher model for each image style, wherein the style hint vector is used to instruct the student model to simulate the generation mechanism of the image style of the corresponding teacher model; based on the style hint vector corresponding to the teacher model for each image style, the knowledge of teacher models of multiple image styles is integrated into the student model with shared parameters through knowledge distillation to obtain the target student model.

[0009] Furthermore, based on the style cue vector corresponding to the teacher model of each image style, the knowledge of the teacher models of multiple image styles is integrated into the student model with shared parameters through knowledge distillation to obtain the target student model, including: selecting sample images from the training sample set; adding noise to the sample images according to a predetermined noise scheduling strategy to obtain a noise image; and obtaining the target student model by using the style cue vector and noise image corresponding to the teacher model of each image style and utilizing the target loss function to constrain the knowledge distillation process between the teacher model and the student model.

[0010] Furthermore, when the target loss function is fractional distillation, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and obtaining the target student model includes: inputting the noise image into the teacher model corresponding to the style hint vector, predicting the noise of the noise image through the teacher model, and obtaining a first score of the noise image; inputting the noise image and the style hint vector into the student model, predicting the noise of the noise image through the student model, and obtaining a second score of the noise image; calculating the difference between the first score of the noise image and the second score of the noise image to obtain the fractional distillation loss; updating the parameters of the student model based on the fractional distillation loss until the fractional distillation loss meets the first preset condition.

[0011] Furthermore, when the target loss function is feature distillation, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and obtaining the target student model includes: inputting the noise image into the teacher model corresponding to the style hint vector, extracting the intermediate feature map of the noise image through the teacher model, and obtaining the first intermediate feature map of the noise image; inputting the noise image into the student model, extracting the intermediate feature map of the noise image through the student model, and obtaining the second intermediate feature map of the noise image; calculating the similarity between the first intermediate feature map of the noise image and the second intermediate feature map of the noise image to obtain the feature distillation loss; updating the parameters of the student model based on the feature distillation loss until the feature distillation loss meets the second preset condition.

[0012] Furthermore, when the target loss function is a multi-category adversarial loss, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and obtaining the target student model includes: inputting the noise image into the teacher model corresponding to the style hint vector, and outputting the real image; inputting the noise image and the style hint vector into the student model, and outputting the predicted image; inputting the real image and the predicted image into the discriminator to obtain the prediction result; based on the prediction result, calculating the multi-category adversarial loss; updating the parameters of the student model based on the multi-category adversarial loss until the multi-category adversarial loss meets the third preset condition.

[0013] Furthermore, when the target loss function is score distillation, feature distillation and multi-category adversarial loss, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and obtaining the target student model includes: weighted summing the loss values ​​of score distillation loss, feature distillation loss and multi-category adversarial loss to obtain the total training loss; updating the parameters of the student model based on the total training loss; repeatedly executing the process of inputting noise images and style hint vectors, calculating the total training loss, and updating the parameters until the preset training stop condition is reached to obtain the target student model.

[0014] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a diffusion model fusion method is provided, including: obtaining text information uploaded by a client, wherein the text information includes description information of a target image to be generated; in a cloud server, determining a target style hint vector that matches the text information, wherein the target style hint vector is used to instruct a target student model to output an image of a target image style, and the target student model is a student model obtained by performing knowledge distillation based on teacher models of multiple different image styles under a distributed training framework. The distributed training framework deploys teacher models of different image styles on multiple image processing units, and the target student model is trained in parallel on multiple image processing units while sharing parameters; inputting the text information and the target style hint vector into the target student model, and the target student model executes the diffusion generation process and outputs the target image; and returning the target image to the client.

[0015] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a diffusion model fusion device is provided. The device includes: an acquisition unit for acquiring text information, wherein the text information includes description information of the target image to be generated; a first confirmation unit for determining a target style hint vector that matches the text information, wherein the target style hint vector is used to instruct a target student model to output an image of the target image style, and the target student model is a student model obtained by performing knowledge distillation based on a plurality of teacher models with different image styles under a distributed training framework. The distributed training framework deploys teacher models with different image styles on a plurality of image processing units, and the target student model is trained in parallel on the plurality of image processing units while sharing parameters; an output unit for inputting the text information and the target style hint vector into the target student model, and the target student model performs a diffusion generation process and outputs the target image.

[0016] Furthermore, the output unit includes: a conversion module for inputting text information into a text embedding layer in a target student model to convert the text information into a text vector; a production module for generating a target image by modulating a generation mechanism of the target student model based on the text vector and the target style hint vector.

[0017] Furthermore, the device includes: a deployment unit for deploying teacher models of different image styles on multiple image processing units of a distributed training framework; a confirmation unit for determining a style hint vector corresponding to the teacher model of each image style, wherein the style hint vector is used to instruct the student model to simulate the generation mechanism of the image style of the corresponding teacher model; an integration unit for integrating the knowledge of teacher models of multiple image styles into a student model with shared parameters through knowledge distillation based on the style hint vector corresponding to the teacher model of each image style, so as to obtain a target student model.

[0018] Furthermore, the integration unit includes: a selection module for selecting sample images from a training sample set; an adding module for adding noise to the sample image according to a predetermined noise scheduling strategy to obtain a noise image; a generation module for obtaining a target student model by using the style hint vector and noise image corresponding to the teacher model of each image style and utilizing the target loss function to constrain the knowledge distillation process between the teacher model and the student model.

[0019] Furthermore, the generation module includes: a first input submodule, used to input the noise image into the teacher model corresponding to the style hint vector, predict the noise of the noise image through the teacher model, and obtain a first score of the noise image; a second submodule, used to input the noise image and the style hint vector into the student model, predict the noise of the noise image through the student model, and obtain a second score of the noise image; a first calculation submodule, used to calculate the difference between the first score of the noise image and the second score of the noise image to obtain the fractional distillation loss; a first update submodule, used to update the parameters of the student model based on the fractional distillation loss until the fractional distillation loss meets the first preset condition.

[0020] Furthermore, the generation module includes: a third input sub-module, used to input the noise image into the teacher model corresponding to the style hint vector, extract the intermediate feature map of the noise image through the teacher model, and obtain the first intermediate feature map of the noise image; a fourth input sub-module, used to input the noise image into the student model, extract the intermediate feature map of the noise image through the student model, and obtain the second intermediate feature map of the noise image; a second calculation sub-module, used to calculate the similarity between the first intermediate feature map of the noise image and the second intermediate feature map of the noise image, and obtain the feature distillation loss; a second update sub-module, used to update the parameters of the student model based on the feature distillation loss until the feature distillation loss meets the second preset condition.

[0021] Furthermore, the generation module includes: a fifth input submodule, used to input the noise image into the teacher model corresponding to the style hint vector, and output the real image; a sixth input submodule, used to input the noise image and the style hint vector into the student model, and output the predicted image; a seventh input submodule, used to input the real image and the predicted image into the discriminator to obtain the prediction result; a third calculation submodule, used to calculate the multi-category adversarial loss based on the prediction result; and a third update submodule, used to update the parameters of the student model based on the multi-category adversarial loss until the multi-category adversarial loss meets the third preset condition.

[0022] Furthermore, the generation module includes: a weighted summation submodule, which is used to weightedly sum the loss values ​​of the score distillation loss, feature distillation loss, and multi-category adversarial loss to obtain the total training loss; a fourth update submodule, which is used to update the parameters of the student model based on the total training loss; and a generation submodule, which is used to repeatedly execute the process of inputting noise images and style hint vectors, calculating the total training loss, and updating the parameters until the preset training stop condition is reached to obtain the target student model.

[0023] According to another aspect of the present application, a computer-readable storage medium is provided. The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the diffusion model fusion methods.

[0024] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the diffusion model fusion methods.

[0025] According to another aspect of the present application, a computer program product is provided, comprising computer instructions, which implement the steps of any one of the above-mentioned diffusion model fusion methods when executed by a processor.

[0026] In an embodiment of the present application, by obtaining text information, wherein the text information includes descriptive information of the target image to be generated; determining a target style hint vector that matches the text information, wherein the target style hint vector is used to instruct the target student model to output an image in the target image style, the target student model is a student model obtained by performing knowledge distillation based on a plurality of teacher models with different image styles under a distributed training framework, the distributed training framework deploys teacher models with different image styles on a plurality of image processing units respectively, and the target student model is trained in parallel on the plurality of image processing units while sharing parameters; the text information and the target style hint vector are input into the target student model, the target student model performs a diffusion generation process, and outputs the target image, thereby solving the problem that the fusion model obtained by the existing model fusion method in the related art generates images of different styles, and the generated image quality is low. In the present application, a method of generating a target image based on a target student model is adopted, and a target image with a specified style that is consistent with the description of the text information is output through the diffusion model generation process of the target student model, thereby achieving the purpose of accurately generating images of different styles based on the target student model, thereby achieving the technical effect of improving the accuracy of generating images of different styles. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0028] Figure 1 This is a schematic diagram of an application scenario of a multi-style diffusion model fusion method provided according to an embodiment of the present application;

[0029] Figure 2 This is a flowchart of a multi-style diffusion model fusion method provided in an embodiment of the present application;

[0030] Figure 3 This is a flowchart of constructing a target student model according to an embodiment of the present application;

[0031] Figure 4 is a schematic diagram of implementing model fusion on a distributed training framework according to an embodiment of the present application;

[0032] Figure 5 is a flowchart of integrating a teacher model into a student model through knowledge distillation according to an embodiment of the present application;

[0033] Figure 6 is a flowchart of integrating a teacher model into a student model through score distillation according to an embodiment of the present application;

[0034] Figure 7 is a flowchart of integrating a teacher model into a student model through feature distillation according to an embodiment of the present application;

[0035] Figure 8 is a flowchart of integrating a teacher model into a student model through multi-class adversarial loss according to an embodiment of the present application;

[0036] Figure 9 This is a flowchart of integrating a teacher model into a student model through score distillation, feature distillation, and multi-class adversarial loss according to an embodiment of the present application;

[0037] Figure 10 is a schematic diagram of integrating a teacher model into a student model through score distillation, feature distillation, and multi-category adversarial loss according to an embodiment of the present application;

[0038] Figure 11 is a schematic diagram of a multi-style diffusion model fusion device provided according to an embodiment of the present application;

[0039] Figure 12 This is a flowchart of another multi-style diffusion model fusion method provided according to an embodiment of the present application;

[0040] Figure 13This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0042] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0043] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0044] Example 1

[0045] The technical solution provided in this application is primarily implemented using large-scale model technology. Large models here refer to deep learning models with large-scale model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. Large models, also known as cornerstone models / foundation models, are pre-trained on large-scale unlabeled corpora to produce pre-trained models with parameters exceeding 100 million. Such models are adaptable to a wide range of downstream tasks and exhibit good generalization capabilities.

[0046] It should be noted that when the large model is actually used, the pre-trained model can be fine-tuned through a small number of samples, so that the large model can be applied to different tasks. For example, the large model can be widely used in fields such as natural language processing, computer vision, and speech processing. Specifically, it can be applied to computer vision tasks such as visual question answering, image description, and image generation. It can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiment of the present application, data processing through the target student model in the Wenshengtu scenario is used as an example for explanation.

[0047] According to an embodiment of the present application, a diffusion model fusion method is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system, such as a set of computer-executable instructions. Moreover, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown.

[0048] Considering the huge number of model parameters of large models and the limited computing resources of mobile terminals, the above method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to this. Figure 1 In the illustrated application scenario, the large model is deployed on a server 10. The server 10 can be connected to one or more client devices 20 via a local area network, a wide area network, the Internet, or other types of data networks. The client devices 20 herein may include, but are not limited to, smartphones, tablet computers, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. The client devices 20 can interact with users via a graphical user interface to access the large model and thereby implement the methods provided in the embodiments of the present application.

[0049] It should be noted that, with the rapid development of high-performance computing units, in other application scenarios, the above method provided in the embodiment of the present application can also be applied to the model all-in-one machine. In an optional embodiment, the model all-in-one machine has multiple models built in, and the user can choose to adjust with a model as needed to obtain the user's own model, so that the high-performance computing unit built in the model all-in-one machine can directly call the adjusted model to execute the above method provided in the embodiment of the present application. In another optional embodiment, a trained model is built in the large model all-in-one machine, so that the high-performance computing unit built in the model all-in-one machine can directly call the model to execute the above method provided in the embodiment of the present application.

[0050] Furthermore, when users need to train their own models, they can upload their own datasets through the client. This dataset is then sent to the server, which then adjusts the pre-trained model using the dataset to create the user's own model, which can then be deployed in production. To facilitate user model adjustment needs, the server provides a complete set of adjustment tools, development frameworks, and processes, supporting a variety of adjustment strategies, making the adjusted model more adaptable to different application fields and highly customized.

[0051] Under the above operating environment, this application provides Figure 2 The multi-style diffusion model fusion method shown. Figure 2 This is a flowchart of the multi-style diffusion model fusion method according to Example 1 of the present application.

[0052] S201: Acquire text information, where the text information includes description information of a target image to be generated.

[0053] Optionally, the above text information may be descriptive information input by the user indicating the generation of a target image of one style or the generation of target images of multiple styles. For example, the text information may be “sunset on a summer beach”.

[0054] S202, determining a target style hint vector that matches the text information, wherein the target style hint vector is used to instruct the target student model to output an image in a target image style, and the target student model is a student model obtained by performing knowledge distillation based on a plurality of teacher models with different image styles under a distributed training framework. The distributed training framework deploys teacher models with different image styles on a plurality of image processing units, and the target student model is trained in parallel on the plurality of image processing units while sharing parameters.

[0055] The teacher model referred to in this application can be a pre-trained, high-performance text graph model, which serves as a "teacher" to guide the training of a simpler, smaller model (the student model). The teacher model typically has high accuracy and a complex structure, capable of handling complex tasks and generating relatively refined outputs.

[0056] The student model used in this application learns by imitating the output of the teacher model, thereby achieving high performance while maintaining a small model size. Therefore, the student model is a relatively small and simple model designed to learn and imitate the behavior or prediction results of another more complex and higher-performing model (i.e., the teacher model). The design goal of the student model is to reduce computational complexity and resource consumption while maintaining high performance, making it more suitable for deployment in resource-constrained environments.

[0057] For example, consider a teacher model with the following image styles: Renaissance, Street Art, Japanese Ukiyo-e, and Modern Photography. Now, we want to create a target student model that, when given text, generates an image that matches the text. The user enters the text message: "Depicts a quiet night, with a full moon high in the sky and waves gently lapping on the beach." First, we need to identify the implicit image style information from the text. In this example, the scene described in the text is tranquil and poetic, likely closer to the Renaissance or Modern Photography styles. However, since the text doesn't directly mention a specific artistic style, semantic understanding and style mapping mechanisms can be used to determine the style that best matches the text. For example, the keywords "night," "full moon," and "beach" may be associated with a specific vector that encodes stylistic features of tranquility, nature, and rich detail, similar to the Renaissance style. Through this mapping mechanism, we can determine a target style hint vector that matches the text. This vector contains sufficient information for the target student model to understand the requested style and, when generating images, leverage the teacher model's style knowledge to generate images that match the text description.

[0058] S203: Input the text information and the target style cue vector into the target student model, and the target student model performs a diffusion generation process to output a target image.

[0059] In an optional embodiment, the text information and the target style cue vector are input into the target student model, and the target image is output through the diffusion generation process of the target student model, including:

[0060] Input the text information into the text embedding layer of the target student model to convert the text information into a text vector;

[0061] According to the text vector and the target style cue vector, the target image is generated by modulating the generation mechanism of the target student model.

[0062] Optionally, the text embedding layer that feeds the text information into the target student model can first convert each word in the text into a vector. A sequence encoder then aggregates these vectors into a context-aware text vector, which incorporates the semantic information of the text and guides the details of image generation. A target style cue vector is fed into the target student model. This target style cue vector is learned during the target image generation process and is used to trigger the target student model's ability to mimic the specified style. Different style cue vectors correspond to different artistic styles, such as realism, cartoons, comics, and impressionism. The text vector and style cue vector are used together to modulate the target student model's generation mechanism, jointly influencing its internal parameters and guiding how the target student model generates images from noise. This modulation process ensures that the generated image not only conforms to the text description but also exhibits the specified style. The diffusion generation process starts with a completely noisy image. The target student model removes the noise through a stepwise back-diffusion process, generating an image that matches the text information. At each step, the target student model predicts and removes some noise based on the current image state (i.e., the current noise level and image details), the text vector, and the style cue vector. This process is repeated until a clear target image is recovered from the noise. Finally, after a series of denoising steps, the target student model will output a clear image that conforms to the text description and the specified style (corresponding to the target image mentioned above).

[0063] For example, a user is using a target student model to generate an image. The user inputs textual information such as "summer beach sunset" and the target style is "Van Gogh style." The target student model's text embedding layer converts this information into a text vector. This vector contains semantic information about summer, beach, and sunset. A "Van Gogh style" hint vector is then input. This hint vector, learned during training, modulates the target student model's generative process to mimic Van Gogh's distinctive brushstrokes and color application. The textual hint vector and style hint vector influence the target student model's generative mechanism, guiding how the target student model gradually recovers a "summer beach sunset" image that matches the description from noise, imbued with Van Gogh's stylistic characteristics. The target student model starts with a noisy image and gradually denoises it, guided at each step by the textual hint vector and style hint vector. During the denoising process, the model gradually recovers the summer beach scene while mimicking Van Gogh's stylistic brushstrokes and color application. Ultimately, the target model generates a clear, summer beach sunset image that exhibits Van Gogh-style painting characteristics, such as intense colors and distinct brushstrokes.

[0064] By combining the above technical solutions with text information and style hint vectors, high-quality images that meet user needs can be generated through the diffusion generation process, thereby improving the accuracy of generating images of different styles.

[0065] In summary, the diffusion model fusion method provided by the embodiment of the present application obtains text information, wherein the text information includes descriptive information of the target image to be generated; determines a target style hint vector that matches the text information, wherein the target style hint vector is used to instruct the target student model to output an image of the target image style, and the target student model is a student model obtained by knowledge distillation based on a plurality of teacher models with different image styles under a distributed training framework. The distributed training framework deploys teacher models with different image styles on multiple image processing units respectively, and the target student model is trained in parallel on multiple image processing units while sharing parameters; the text information and the target style hint vector are input into the target student model, and the target student model executes the diffusion generation process and outputs the target image, which solves the problem that the fusion model obtained by the existing model fusion method in the related art generates images of different styles, and the generated image quality is low. In this application, a method of generating a target image based on a target student model is adopted, and a target image with a specified style that is consistent with the description of the text information is output through the diffusion model generation process of the target student model, thereby achieving the purpose of accurately generating images of different styles based on the target student model, thereby achieving the technical effect of improving the accuracy of generating images of different styles.

[0066] In an optional embodiment, as Figure 3 As shown, the target student model can be obtained through the following steps:

[0067] S301, deploy teacher models with different image styles on multiple image processing units of the distributed training framework.

[0068] For example, consider three teacher models, each excelling at generating images in three styles: Van Gogh (Post-Impressionism), Ukiyo-e (Japanese Ukiyo-e), and Surrealism. The goal is to use a single student model with shared parameters to learn and fuse these three styles, enabling it to generate images in corresponding styles based on different style cue vectors. Suppose that in a distributed training framework, there are three GPUs (the image processing units mentioned above), with a teacher model deployed on each GPU: one excelling in Van Gogh, one excelling in Ukiyo-e, and one excelling in Surrealism. Deploying teacher models of different image styles in a distributed training framework is a technical strategy for efficiently utilizing computing resources and accelerating model training and knowledge distillation.

[0069] S302: Determine a style hint vector corresponding to the teacher model for each image style, wherein the style hint vector is used to instruct the student model to simulate a generation mechanism of the image style of the corresponding teacher model.

[0070] Optionally, before training begins, a specific style hint vector can be assigned to each teacher model. A vector encoding table can be initialized containing three style hint vectors, one for each of the three styles of the teacher models. These style hint vectors effectively instruct the student model to emulate the corresponding style of the teacher model during training.

[0071] S303 , based on the style hint vector corresponding to the teacher model of each image style, the knowledge of the teacher models of multiple image styles is integrated into the student model with shared parameters through knowledge distillation to obtain the target student model.

[0072]

[0073] For example, after training is complete, the resulting target student model is capable of generating images in not only the style of Van Gogh, but also Ukiyo-e and Surrealist styles. It can also flexibly switch between these styles based on the input style cue vector, achieving hybrid and innovative image styles. If a Van Gogh-style vector is input, the student model will generate images with Van Gogh-like characteristics; if a vector is input that blends Ukiyo-e and Surrealist styles, the student model can generate an entirely new style that combines the flatness of Ukiyo-e with the unconventional elements of Surrealism.

[0074] In summary, the target student model obtained through the above technical solution achieves high efficiency and flexibility in multi-style generation. It can not only independently replicate various artistic styles, but also innovatively mix and integrate these styles, bringing a more general and practical image generation solution to the field of literary graphics.

[0075] Optionally, the embodiment of the present application also provides a framework for implementing model fusion knowledge distillation on a distributed training framework. Figure 4As shown. In the GPU cluster, a teacher model is configured on each GPU, and the student model shares parameters on each GPU, thereby efficiently utilizing the GPU. The network structure of the student model is basically the same as the basic model, except that a small number of parameters are introduced to process the input style index i. The prior distribution of different models is maintained as an encoding table of N learnable vectors, and these learnable vectors are called the above-mentioned style cue vectors. These style cue vectors are used to specify the style of the image and modulate the student model to imitate the corresponding teacher model. Based on the style cue vector corresponding to the teacher model of each image style, the knowledge of teacher models of multiple image styles is integrated into the student model with shared parameters through knowledge distillation to obtain the target student model.

[0076] The steps of the process can be as follows Figure 5 shown.

[0077] S501: Select a sample image from a training sample set.

[0078] For example, we need to fuse three teacher models—Van Gogh, Japanese Ukiyo-e, and Surrealist—to obtain a target student model. Sample images can be randomly selected from a large dataset containing a wide range of images. These sample images can include natural landscapes, portraits, and artworks, covering a wide range of styles and content, ensuring that the target student model learns rich stylistic features and generative capabilities. For example, we can select image A of Van Gogh, a typical Japanese Ukiyo-e painting, and image B of the surrealist painter Salvador Dalí from a dataset of artworks.

[0079] S502: Add noise to the sample image according to a predetermined noise scheduling strategy to obtain a noisy image.

[0080] Optionally, the above-mentioned predetermined noise scheduling strategy can be linear noise scheduling, exponential noise scheduling, adaptive noise scheduling, staged noise scheduling, etc. In linear noise scheduling, the noise level increases or decreases linearly over time. The linear scheduling strategy is easy to understand and implement. In exponential noise scheduling, the noise intensity changes according to an exponential function over time. Exponential scheduling can accelerate model training and convergence, especially in the early generation stage of the image. Adaptive noise scheduling dynamically adjusts the noise level according to the current state of the image, which can more intelligently manage the addition and removal of noise and avoid introducing too much or too little noise in certain parts of the image. Staged noise scheduling divides the denoising process into several stages, each stage using different noise intensities and types, which helps the model focus on different features of the image at different stages. A predetermined noise scheduling strategy is used to artificially add noise to the sample image to obtain a noisy image. The degree of noise addition can be adjusted according to the training stage and needs, with the aim of enabling the model to learn the ability to restore images under different noise levels.

[0081] S503, by using the style cue vector and noise image corresponding to the teacher model of each image style, and utilizing the target loss function to constrain the knowledge distillation process between the teacher model and the student model, a target student model is obtained.

[0082] Optionally, the objective loss function described above can be a combination of one or more of score distillation, feature distillation, and multi-class adversarial loss. Score distillation ensures that the student model can learn the score prediction ability of the teacher model, feature distillation promotes the learning of style and features, and multi-class adversarial loss further improves the realism and style differentiation of generated images. By combining these loss functions, the student model can learn more comprehensive and in-depth knowledge in a shorter time, improving the efficiency of knowledge distillation.

[0083] In an optional embodiment, Figure 6 This is a flowchart of integrating the teacher model into the student model through fractional distillation according to an embodiment of the present application. When the target loss function is fractional distillation, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including:

[0084] S601: Input the noise image into the teacher model corresponding to the style hint vector, and predict the noise of the noise image through the teacher model to obtain a first score of the noise image.

[0085] For example, to train a target student model to generate images in the style of Van Gogh, we first select an image from the training dataset, such as Van Gogh's image A. This image is then noised according to a predefined noise scheduling strategy, resulting in a noisy version of image A, referred to as the noise image above. Next, the noisy image is fed into a Van Gogh-style teacher model, which is a pre-trained model that excels at generating images in the style of Van Gogh. The teacher model predicts the noise distribution of this noisy image and outputs a score, referred to as the first score for the noise image above, representing its prediction of the noise distribution.

[0086] S602: Input the noise image and the style cue vector into the student model, and predict the noise of the noise image by the student model to obtain a second score of the noise image.

[0087] For example, a noisy image A and a specified style hint vector (representing the style of Van Gogh) are input into the student model. The student model attempts to predict a noise distribution similar to that of the teacher model based on this style hint vector and the noisy image, and obtains the second score of the noisy image mentioned above.

[0088] S603: Calculate the difference between the first score of the noisy image and the second score of the noisy image to obtain a fractional distillation loss.

[0089] For example, after the student model predicts a score, the difference between its predicted score (the second score) and the teacher model's score (the first score) can be calculated. This difference is expressed as the score distillation loss, which is usually quantified using the mean squared error (MSE) or a more complex loss function. The goal is to make this loss as small as possible, which means that the student model's predicted score should be as close as possible to the teacher model's score.

[0090] Alternatively, the fractional distillation loss function can be:

[0091]

[0092] Where N is the number of teacher models, i is the style index, is the training sample picture, t is the time step, are the student model parameters, are the teacher model parameters, is the noise score predicted by the model for noisy images.

[0093] S604 , updating parameters of the student model based on the fractional distillation loss until the fractional distillation loss satisfies a first preset condition.

[0094] Optionally, the student model parameters are updated based on the fractional distillation loss. This can be done through backpropagation. During training, the student model continuously learns and adjusts to reduce the fractional distillation loss until it meets a first pre-determined condition. This pre-determined condition can be when the fractional distillation loss falls below a certain threshold or when training reaches a limited number of iterations (e.g., 100,000). Once this condition is met, the student model has achieved the desired level of imitation of the teacher model's style and can be considered to have completed knowledge distillation in the current style.

[0095] For example, during the training process, the fractional distillation loss value is continuously monitored. If after a certain iteration, the fractional distillation loss is lower than the threshold (for example, 0.01), it can be considered that the student model has imitated the current style accurately enough, and it can be considered to stop learning the current style or switch to learning the next style. At the same time, it will also check whether the maximum number of iterations (for example, 100,000) has been reached. If it has been reached, training can be stopped even if the loss value has not yet met the threshold. Once the student model reaches the predetermined fractional distillation loss threshold for all styles, or the overall number of training iterations reaches the set upper limit, the knowledge distillation process can be considered complete, and the student model has the ability to integrate multiple styles.

[0096] In summary, through the above technical solutions, the knowledge distillation process based on fractional distillation can efficiently extract and transfer style knowledge from the teacher model, so that the generated student model has the ability to generate multiple image styles.

[0097] In an optional embodiment, Figure 7 This is a flowchart of integrating the teacher model into the student model through feature distillation according to an embodiment of the present application. When the target loss function is feature distillation, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including:

[0098] S701: Input the noise image into the teacher model corresponding to the style hint vector, extract the intermediate feature map of the noise image through the teacher model, and obtain a first intermediate feature map of the noise image.

[0099] Optionally, the intermediate feature maps of the noise image come from the internal layers of the teacher model, which can be the output of the convolutional layer or the Transformer layer.

[0100] For example, we first select an image A of a Van Gogh painting from the training dataset and add noise according to the noise scheduling strategy to obtain a noisy version of Image A, which is the noisy image mentioned above. The noisy image is then processed using the Van Gogh-style teacher model. In this step, we focus on the intermediate layers of the teacher model, namely the intermediate feature maps generated by the teacher model when processing the image. These feature maps contain information such as the image's structure, texture, and color, as well as features specific to Van Gogh's style. After the teacher model processes the noisy image, it obtains the feature maps of its intermediate layers, namely the first intermediate feature map.

[0101] S702: Input the noise image into the student model, extract the intermediate feature map of the noise image through the student model, and obtain a second intermediate feature map of the noise image.

[0102] For example, if the noisy image A is input into the student model, which has a similar target architecture to the teacher model, the student model generates its own intermediate feature maps, namely the second intermediate feature maps, which reflect the student model's understanding and processing of the noisy image.

[0103] S703: Calculate the similarity between the first intermediate feature map of the noise image and the second intermediate feature map of the noise image to obtain a feature distillation loss.

[0104] Optionally, after obtaining the intermediate feature maps of the teacher and student models, the similarity between them is calculated to quantify whether the student model's feature learning performance is close to that of the teacher model. Similarity can be calculated using methods such as mean squared error (MSE), structural similarity index (SSIM), cosine similarity, feature matching loss, and so on. For example, MSE can be used to measure the difference between two feature maps, and the calculated value is the feature distillation loss. The goal is to minimize this loss to ensure that the student model's feature representation in the intermediate layers is consistent with or very close to that of the teacher model, thereby learning the style and feature expression capabilities of the teacher model.

[0105] Optionally, the feature distillation loss function can be:

[0106]

[0107] Where N is the number of teacher models, is the training sample picture, t is the time step, are the student model parameters, are the teacher model parameters, is the feature map of the student model, is the feature map of the teacher model, i is the feature map of the student model at layer i, and j is the feature map of the teacher model at layer j.

[0108] S704: Update the parameters of the student model based on the feature distillation loss until the feature distillation loss meets the second preset condition.

[0109] Optionally, a backpropagation algorithm can be used to adjust the parameters of the student model based on the feature distillation loss to reduce feature differences. Training continues until the feature distillation loss reaches a second pre-determined condition, which can be either below a certain threshold, such as 0.001, or reaching a maximum number of iterations, such as 50,000. Once this condition is met, the student model is considered to have mastered the style knowledge of the teacher model at the intermediate feature level, completing feature distillation of that style.

[0110] Through the above technical solution, the use of feature distillation loss ensures that the target student model not only matches the style of the teacher model in the final generated image, but also that the feature representation during the generation process is consistent with the teacher model, enhancing the quality of knowledge transfer and style learning during the knowledge distillation process.

[0111] In an optional embodiment, Figure 8This is a flowchart of integrating the teacher model into the student model through multi-class adversarial loss according to an embodiment of the present application. When the target loss function is a multi-class adversarial loss, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including:

[0112] S801, input the noise image into the teacher model corresponding to the style hint vector, and output the real image.

[0113] For example, you can take a real image with added noise (a Van Gogh-style image) and feed it into the teacher model along with a style cue vector representing Van Gogh's style. The teacher model is a pre-trained generative model that can output a real image with a highly similar style to the original based on the input noise and style cue vector. If the input is a Van Gogh-style style cue vector, the teacher model will generate an image in Van Gogh's style.

[0114] S802: Input the noise image and style cue vector into the student model and output a predicted image.

[0115] For example, the same noisy image and style cue vector are used as input and passed to a student model. The student model's goal is to learn and emulate the teacher model's behavior, so it also attempts to generate an image that matches the input style cue vector. Initially, the student model's predicted images may differ significantly from the real images generated by the teacher model. However, as training progresses, the student model will gradually improve the quality of its generated images.

[0116] S803: Input the real image and the predicted image into the discriminator to obtain the prediction result.

[0117] Optionally, in order to evaluate the quality of the images generated by the student model, an additional model - the discriminator is used. The discriminator is a classifier that can determine whether the input image is a real image and which style it belongs to. Suppose there are N teacher models, each corresponding to a specific style. The dimension of the classification head of the discriminator is set to 2N, which means that it can recognize 2N different categories, where the first N categories correspond to real images of N target styles, and the last N categories correspond to predicted images of these N styles (that is, images generated by the student model). The cleverness of this design is that it not only allows the discriminator to distinguish between real images and predicted images, but also allows it to further distinguish between real images of different styles, as well as the difference between real and predicted images of the same style. In this way, the discriminator can learn more detailed discrimination capabilities, while also providing richer feedback to the student model, guiding it to generate images that are more in line with the specified style.

[0118] Optionally, the discriminator's tasks can include distinguishing between real and fake categories and distinguishing between different styles. Distinguishing between real and fake categories means learning to distinguish between real images and fake images. For real images of each style, the discriminator's goal is to classify them into the correct style category in the top N categories; and for images generated by the student model in the same style, the discriminator's goal is to classify them into the fake category of the corresponding style in the bottom N categories. In addition to real and fake classification, the discriminator also needs to learn to distinguish between images of different styles. This means that even if two images are real, if their styles are different, the discriminator can correctly classify them into different style categories.

[0119] For example, in this stage, the real image generated by the teacher model and the predicted image generated by the student model are simultaneously input into the discriminator, which outputs the classification results and the probability of real / fake for the real and predicted images. A multi-class adversarial loss is then calculated based on the discriminator's predictions. This calculation involves fine-tuning the loss function for each category (real style) and each model (student model, teacher model) to ensure that the images generated by the student model are close to the output of the teacher model in terms of style attributes and realism.

[0120] S804: Calculate the multi-category adversarial loss based on the prediction results.

[0121] Optionally, the multi-class adversarial loss function can be:

[0122]

[0123] Where N is the number of teacher models, i is the style index, is the training sample picture, t is the time step, are the student model parameters, are the teacher model parameters.

[0124] S805 , updating the parameters of the student model based on the multi-class adversarial loss until the multi-class adversarial loss satisfies a third preset condition.

[0125] Optionally, based on the calculated multi-class adversarial loss, a backpropagation algorithm can be used to update the parameters of the student model. The goal is to reduce the difference between images generated by the student model and real images, while ensuring that images generated by the student model can be correctly classified. The training process continues until the multi-class adversarial loss meets a third preset condition, which can be that the multi-class adversarial loss is below a certain threshold (such as 0.005) or the training reaches a predetermined number of iterations (such as 30,000). When the student model performs well on the multi-class adversarial loss, it can be considered to be able to generate real images that are similar to the teacher model and can be identified as the specified style, completing the knowledge distillation process.

[0126] By adding supervision from the discriminator, the above technical solution enables the student model to more efficiently learn and integrate style knowledge from different teacher models, ultimately achieving the goal of multi-style generation in the target student model.

[0127] In an optional embodiment, Figure 9 This is a flowchart of integrating a teacher model into a student model through score distillation, feature distillation, and multi-class adversarial loss according to an embodiment of the present application. When the target loss function is score distillation, feature distillation, and multi-class adversarial loss, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including:

[0128] S901: Perform weighted summation on the loss values ​​of score distillation loss, feature distillation loss, and multi-category adversarial loss to obtain the total training loss.

[0129] Optionally, the three loss functions mentioned above are used together, and the final optimization goal is a weighted combination of the three:

[0130]

[0131] in, is the fractional distillation loss function, is the feature distillation loss function, is a multi-class adversarial loss function, is the weight corresponding to the feature distillation loss function, The corresponding weights of the loss function for multi-category adversarial loss distillation.

[0132] Optionally, through weighted combination, the student model can be guided to learn both accurate score predictions to replicate the teacher model's generation process, as well as rich intermediate-layer features to grasp the deep representation of style. Furthermore, it must generate images that can pass the discriminator's multi-category adversarial training to improve the realism and style consistency of the generated images. This combined training strategy ensures that the student model achieves optimal performance in terms of style fusion, image quality, and generation efficiency.

[0133] S902: Update the parameters of the student model based on the total training loss.

[0134] Optionally, after obtaining the total training loss, the student model parameters can be updated using the backpropagation algorithm. The backpropagation algorithm calculates the gradients of the student model parameters based on the total training loss and then uses the optimizer to adjust the parameters to reduce the loss. This process is repeated until the student model parameters converge or a predetermined training stopping condition is reached.

[0135] Optionally, during training, the weight coefficients can be dynamically adjusted based on the student model's learning progress and performance. For example, in the early stages of training, score distillation and feature distillation can be given greater weight to ensure that the student model can accurately learn score predictions and style features from the teacher model; in the later stages of training, the weight of the multi-class adversarial loss can be increased to improve the visual quality and style discrimination of the images generated by the student model.

[0136] Optionally, to find a better combination of weights, you can perform hyperparameter tuning, such as using grid search, random search, or Bayesian optimization. The goal is to find a balance between learning multiple capabilities so that the student model can achieve a good state in terms of style fusion, image quality, and generation efficiency.

[0137] Optionally, during the evaluation phase of the training process, you can check the performance metrics of the student model under different styles, such as FID score, PSNR (peak signal-to-noise ratio), SSIM (structural similarity index), etc., to determine whether the weights need to be adjusted.

[0138] S903, repeatedly executing the process of inputting the noise image and the style hint vector, calculating the total training loss, and updating the parameters until the preset training stop condition is reached, and obtaining the target student model.

[0139] Optionally, training is an iterative process. Noise images and style cue vectors are continuously fed into the student model, the three losses described above are calculated, and a weighted sum is taken to obtain the total training loss. The student model parameters are then updated. This process is repeated, with each iteration attempting to reduce the total training loss, allowing the student model to better learn and incorporate the knowledge of the teacher model.

[0140] Optionally, preset training stopping conditions can include reaching a maximum number of iterations (e.g., 50,000 iterations), the total training loss falling below a threshold (e.g., 0.005), or the student model's performance on the validation set meeting a certain standard. When any of these stopping conditions is met, training ends, and the resulting target student model is able to integrate the capabilities of multiple teacher models to efficiently generate images in a variety of styles while ensuring image quality, style accuracy, and realism.

[0141] In summary, the target student model constructed through the above scheme can not only learn and integrate knowledge from multiple teacher models, but also maintain a high degree of style consistency and image authenticity when generating images, greatly improving the generalization ability and application value of the target student model.

[0142] According to the process of integrating the teacher model into the student model through score distillation, feature distillation and multi-category adversarial loss provided in the embodiment of the present application, the following can be done: Figure 10 As shown in Figure 2, a noisy image is fed into the student model and the teacher model, respectively, and a style cue vector is fed into the student model. The fractional distillation loss is used to learn the teacher model's score prediction ability, ensuring that the student model can reproduce the data distribution modeled by each teacher model, specifically its score prediction ability for images of different styles. The feature distillation loss is used to compare the feature maps of the student and teacher models at certain intermediate layers to promote style and feature learning, enhancing knowledge transfer. The multi-class adversarial loss introduces the concept of generative adversarial networks. It requires not only that the generated images can fool the discriminator into being considered real but also that the discriminator can correctly identify the image's style, further improving the style consistency and realism of the student model's generated images. Finally, by combining these three loss functions, the student model's parameters are updated based on the total training loss. Therefore, by combining fractional distillation, feature distillation, and the multi-class adversarial loss, a more powerful and flexible target student model can be constructed. This model can not only learn and integrate knowledge from multiple teacher models, but also maintain a high degree of style consistency and realism when generating images, significantly improving the generalization ability and application value of the target student model.

[0143] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0144] Example 2

[0145] The embodiment of the present application further provides a diffusion model fusion device. It should be noted that the diffusion model fusion device of the embodiment of the present application can be used to execute the diffusion model fusion method provided in the embodiment of the present application. The diffusion model fusion device provided in the embodiment of the present application is introduced below.

[0146] According to an embodiment of the present application, a device for implementing the above diffusion model fusion method is also provided, such as Figure 11 As shown, the device includes: an acquisition unit 1101, a first confirmation unit 1102 and an output unit 1103.

[0147] Specifically, the acquiring unit 1101 is configured to acquire text information, wherein the text information includes description information of the target image to be generated;

[0148] A first confirmation unit 1102 is configured to determine a target style cue vector that matches the text information, wherein the target style cue vector is used to instruct a target student model to output an image of a target image style. The target student model is a student model obtained by performing knowledge distillation based on multiple teacher models with different image styles in a distributed training framework. The distributed training framework deploys teacher models with different image styles on multiple image processing units, and the target student model is trained in parallel on the multiple image processing units while sharing parameters.

[0149] The output unit 1103 is used to input the text information and the target style cue vector into the target student model, and the target student model performs a diffusion generation process to output the target image.

[0150] In summary, the diffusion model fusion device provided in the embodiment of the present application obtains text information through the acquisition unit 1101, wherein the text information includes descriptive information of the target image to be generated; the first confirmation unit 1102 determines the target style prompt vector that matches the text information, wherein the target style prompt vector is used to instruct the target student model to output an image of the target image style, and the target student model is a student model obtained by performing knowledge distillation based on teacher models of multiple different image styles under a distributed training framework. The distributed training framework deploys teacher models of different image styles on multiple image processing units, and the target student model is trained in parallel on multiple image processing units while sharing parameters; the output unit 1103 inputs the text information and the target style prompt vector into the target student model, and the target student model executes the diffusion generation process and outputs the target image, which solves the problem that the fusion model obtained by the existing model fusion method in the related art generates images of different styles, and the generated image quality is low. In this application, a method of generating a target image based on a target student model is adopted. Through the target student model diffusion model generation process, a target image that is consistent with the text information description and has a specified style is output, thereby achieving the purpose of accurately generating images of different styles based on the target student model, thereby achieving the technical effect of improving the accuracy of generating images of different styles.

[0151] Optionally, in the diffusion model fusion device provided in an embodiment of the present application, the output unit 1103 includes: a conversion module, used to input text information into the text embedding layer in the target student model, and convert the text information into a text vector; a production module, used to generate a target image by modulating the generation mechanism of the target student model based on the text vector and the target style prompt vector.

[0152] Optionally, the diffusion model fusion device provided in an embodiment of the present application includes: a deployment unit for deploying teacher models of different image styles on multiple image processing units of a distributed training framework; a confirmation unit for determining the style hint vector corresponding to the teacher model of each image style, wherein the style hint vector is used to instruct the student model to simulate the image style generation mechanism of the corresponding teacher model; an integration unit for integrating the knowledge of teacher models of multiple image styles into a student model with shared parameters through knowledge distillation based on the style hint vector corresponding to the teacher model of each image style, to obtain a target student model.

[0153] Optionally, in the diffusion model fusion device provided in an embodiment of the present application, the integration unit includes: a selection module for selecting sample images from a training sample set; an adding module for adding noise to the sample image according to a predetermined noise scheduling strategy to obtain a noise image; a generation module for obtaining a target student model by using the style hint vector and noise image corresponding to the teacher model of each image style, and using the target loss function to constrain the knowledge distillation process between the teacher model and the student model.

[0154] Optionally, in the diffusion model fusion device provided in an embodiment of the present application, the generation module includes: a first input submodule, used to input the noise image into the teacher model corresponding to the style hint vector, predict the noise of the noise image through the teacher model, and obtain a first score of the noise image; a second submodule, used to input the noise image and the style hint vector into the student model, predict the noise of the noise image through the student model, and obtain a second score of the noise image; a first calculation submodule, used to calculate the difference between the first score of the noise image and the second score of the noise image to obtain a fractional distillation loss; a first update submodule, used to update the parameters of the student model based on the fractional distillation loss until the fractional distillation loss meets a first preset condition.

[0155] Optionally, in the diffusion model fusion device provided in an embodiment of the present application, the generation module includes: a third input submodule, used to input the noise image into the teacher model corresponding to the style hint vector, extract the intermediate feature map of the noise image through the teacher model, and obtain the first intermediate feature map of the noise image; a fourth input submodule, used to input the noise image into the student model, extract the intermediate feature map of the noise image through the student model, and obtain the second intermediate feature map of the noise image; a second calculation submodule, used to calculate the similarity between the first intermediate feature map of the noise image and the second intermediate feature map of the noise image to obtain the feature distillation loss; and a second update submodule, used to update the parameters of the student model based on the feature distillation loss until the feature distillation loss meets the second preset condition.

[0156] Optionally, in the diffusion model fusion device provided in an embodiment of the present application, the generation module includes: a fifth input submodule, used to input the noise image into the teacher model corresponding to the style hint vector, and output the real image; a sixth input submodule, used to input the noise image and the style hint vector into the student model, and output the predicted image; a seventh input submodule, used to input the real image and the predicted image into the discriminator to obtain a prediction result; a third calculation submodule, used to calculate the multi-category adversarial loss based on the prediction result; and a third update submodule, used to update the parameters of the student model based on the multi-category adversarial loss until the multi-category adversarial loss meets the third preset condition.

[0157] Optionally, in the diffusion model fusion device provided in the embodiment of the present application, the generation module includes: a weighted summation submodule, which is used to weightedly sum the loss values ​​of the score distillation loss, the feature distillation loss, and the multi-category adversarial loss to obtain the total training loss; a fourth update submodule, which is used to update the parameters of the student model based on the total training loss; and a generation submodule, which is used to repeatedly execute the process of inputting a noise image and a style hint vector, calculating the total training loss, and updating the parameters until a preset training stop condition is reached to obtain a target student model.

[0158] It should be noted that the aforementioned acquisition unit 1101, first confirmation unit 1102, and output unit 1103 correspond to steps S201 to S203 in Example 1. The examples and application scenarios implemented by the three units and the corresponding steps are the same, but are not limited to the contents disclosed in the aforementioned Example 1. It should be noted that the aforementioned modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The aforementioned modules may also be executed as part of a device in the computer terminal 10 provided in Example 1.

[0159] Example 3

[0160] According to an embodiment of the present application, another diffusion model fusion method is also provided, such as Figure 12 As shown, the method includes:

[0161] S1201, obtaining text information uploaded by the client, wherein the text information includes description information of the target image to be generated;

[0162] S1202: In the cloud server, a target style cue vector is determined that matches the text information. The target style cue vector is used to instruct a target student model to output an image of a target image style. The target student model is a student model obtained by performing knowledge distillation based on multiple teacher models with different image styles in a distributed training framework. The distributed training framework deploys teacher models with different image styles on multiple image processing units, and the target student model is trained in parallel on the multiple image processing units while sharing parameters.

[0163] S1203, inputting the text information and the target style cue vector into the target student model, and the target student model performs a diffusion generation process to output a target image;

[0164] S1204: Return the target image to the client.

[0165] In the cloud server, the diffusion model fusion method is the same as that in the first embodiment and will not be described again here.

[0166] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0167] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0168] Example 4

[0169] The embodiment of the present application may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal or an electronic device.

[0170] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0171] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the diffusion model fusion method: obtaining text information, wherein the text information includes descriptive information of the target image to be generated; determining a target style prompt vector that matches the text information, wherein the target style prompt vector is used to instruct the target student model to output an image of the target image style, and the target student model is a student model obtained by performing knowledge distillation based on teacher models of multiple different image styles under a distributed training framework. The distributed training framework deploys teacher models of different image styles on multiple image processing units, and the target student model is trained in parallel on multiple image processing units while sharing parameters; inputting the text information and the target style prompt vector into the target student model, and the target student model executes the diffusion generation process to output the target image.

[0172] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the diffusion model fusion method: inputting text information and target style prompt vectors into the target student model, and outputting the target image through the diffusion generation process of the target student model, including: inputting text information into the text embedding layer in the target student model, and converting the text information into a text vector; based on the text vector and the target style prompt vector, generating the target image by modulating the generation mechanism of the target student model.

[0173] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the diffusion model fusion method: the target student model is obtained by the following steps: deploying teacher models of different image styles on multiple image processing units of a distributed training framework; determining the style hint vector corresponding to the teacher model of each image style, wherein the style hint vector is used to instruct the student model to simulate the generation mechanism of the image style of the corresponding teacher model; based on the style hint vector corresponding to the teacher model of each image style, the knowledge of teacher models of multiple image styles is integrated into the student model with shared parameters through knowledge distillation to obtain the target student model.

[0174] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the diffusion model fusion method: based on the style hint vector corresponding to the teacher model of each image style, the knowledge of the teacher models of multiple image styles is integrated into the student model of shared parameters through knowledge distillation to obtain the target student model, including: selecting a sample image from the training sample set; adding noise to the sample image according to a predetermined noise scheduling strategy to obtain a noise image; by using the style hint vector and noise image corresponding to the teacher model of each image style, and using the target loss function to constrain the knowledge distillation process between the teacher model and the student model to obtain the target student model.

[0175] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the diffusion model fusion method: when the target loss function is fractional distillation, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including: inputting the noise image into the teacher model corresponding to the style hint vector, predicting the noise of the noise image through the teacher model, and obtaining a first score of the noise image; inputting the noise image and the style hint vector into the student model, predicting the noise of the noise image through the student model, and obtaining a second score of the noise image; calculating the difference between the first score of the noise image and the second score of the noise image to obtain the fractional distillation loss; updating the parameters of the student model based on the fractional distillation loss until the fractional distillation loss meets the first preset condition.

[0176] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the diffusion model fusion method: when the target loss function is feature distillation, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including: inputting the noise image into the teacher model corresponding to the style hint vector, extracting the intermediate feature map of the noise image through the teacher model, and obtaining the first intermediate feature map of the noise image; inputting the noise image into the student model, extracting the intermediate feature map of the noise image through the student model, and obtaining the second intermediate feature map of the noise image; calculating the similarity between the first intermediate feature map of the noise image and the second intermediate feature map of the noise image to obtain the feature distillation loss; updating the parameters of the student model based on the feature distillation loss until the feature distillation loss meets the second preset condition.

[0177] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the diffusion model fusion method: when the target loss function is a multi-category adversarial loss, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including: inputting the noise image into the teacher model corresponding to the style hint vector, and outputting the real image; inputting the noise image and the style hint vector into the student model, and outputting the predicted image; inputting the real image and the predicted image into the discriminator to obtain the prediction result; based on the prediction result, calculating the multi-category adversarial loss; updating the parameters of the student model based on the multi-category adversarial loss until the multi-category adversarial loss meets the third preset condition.

[0178] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the diffusion model fusion method: when the target loss function is score distillation, feature distillation and multi-category adversarial loss, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including: weighted summing the loss values ​​of the score distillation loss, feature distillation loss, and multi-category adversarial loss to obtain the total training loss; based on the total training loss, updating the parameters of the student model; repeatedly executing the process of inputting noise images and style hint vectors, calculating the total training loss, and updating the parameters until the preset training stop condition is reached to obtain the target student model.

[0179] In an embodiment of the present application, a solution for diffusion model fusion is provided. By obtaining text information, wherein the text information includes description information of the target image to be generated; determining a target style hint vector that matches the text information, wherein the target style hint vector is used to instruct the target student model to output an image of the target image style, the target student model is a student model obtained by performing knowledge distillation based on a plurality of teacher models with different image styles under a distributed training framework, the distributed training framework deploys teacher models with different image styles on a plurality of image processing units, and the target student model is trained in parallel on the plurality of image processing units while sharing parameters; the text information and the target style hint vector are input into the target student model, the target student model executes the diffusion generation process, and outputs the target image, thereby solving the technical problem in the related art of using the existing model fusion method to fuse models of different styles with poor fusion effect. In this application, a method for generating a target image based on a target student model is adopted. Text information is input into the target student model, and a target style prompt vector matching the text information is determined. The target style vector can instruct the target student model to output an image in the target image style. The target student model outputs the target image based on the text information and the target style prompt vector, thereby achieving the purpose of accurately generating images of different styles based on the target student model, thereby realizing the technical effect of improving the accuracy of generating images of different styles.

[0180] Optionally, Figure 13 This is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 13 As shown, the electronic device may include: one or more ( Figure 13 Only one is shown) processor 1302, memory 1304, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0181] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the diffusion model fusion method and apparatus in the embodiments of the present application. The processor executes the software programs and modules stored in the memory to perform various functional applications and data processing, thereby implementing the diffusion model fusion method described above. The memory can include high-speed random access memory (RAM) and can also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some embodiments, the memory can further include memory remotely located from the processor, and such remote memory can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0182] The processor may call the information and application programs stored in the memory through the transmission device to execute the above steps in the above diffusion model fusion method.

[0183] Using the embodiment of the present application, a solution for diffusion model fusion is provided. By obtaining text information, wherein the text information includes descriptive information of the target image to be generated; determining a target style hint vector that matches the text information, wherein the target style hint vector is used to instruct the target student model to output an image of the target image style, the target student model is a student model obtained by performing knowledge distillation based on a plurality of teacher models with different image styles under a distributed training framework, the distributed training framework deploys teacher models with different image styles on a plurality of image processing units respectively, and the target student model is trained in parallel on the plurality of image processing units while sharing parameters; the text information and the target style hint vector are input into the target student model, the target student model executes the diffusion generation process, and outputs the target image, thereby solving the technical problem in the related art of using the existing model fusion method to fuse models of different styles with poor fusion effect. In this application, a method for generating a target image based on a target student model is adopted. Text information is input into the target student model, and a target style prompt vector matching the text information is determined. The target style vector can instruct the target student model to output an image in the target image style. The target student model outputs the target image based on the text information and the target style prompt vector, thereby achieving the purpose of accurately generating images of different styles based on the target student model, thereby realizing the technical effect of improving the accuracy of generating images of different styles.

[0184] It can be understood by those skilled in the art that Figure 13 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 13It does not limit the structure of the above electronic device. For example, the electronic device may also include Figure 13 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 13 Different configurations shown.

[0185] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0186] Example 5

[0187] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the diffusion model fusion method provided in the first embodiment.

[0188] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0189] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the steps of the diffusion model fusion method.

[0190] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of the diffusion model fusion method.

[0191] Example 6

[0192] According to another aspect of an embodiment of the present application, a computer program product is further provided, which, when executed on a data processing device, is suitable for executing the steps of the diffusion model fusion method.

[0193] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0194] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0195] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0196] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0197] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0198] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program code.

[0199] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A diffusion model fusion method, characterized in that: include: Acquiring text information, wherein the text information includes description information of the target image to be generated; Determining a target style cue vector that matches the text information, wherein the target style cue vector is used to instruct a target student model to output an image of a target image style, the target student model being a student model obtained by performing knowledge distillation based on multiple teacher models of different image styles in a distributed training framework, wherein the distributed training framework deploys teacher models of different image styles on multiple image processing units, and the target student model is trained in parallel on the multiple image processing units while sharing parameters; Inputting the text information and the target style cue vector into the target student model, and having the target student model perform a diffusion generation process to output a target image; wherein the text information is processed through a semantic understanding and style mapping mechanism to determine a target style hint vector that matches the text information; Integrate the knowledge of teacher models of multiple image styles into a student model with shared parameters through knowledge distillation, and use a target loss function to constrain the knowledge distillation process between the teacher model and the student model to obtain the target student model, wherein the target loss function can be a combination of one or more of score distillation, feature distillation, and multi-class adversarial loss; Among them, a target student model is created. When the target student model receives text information, it generates an image that matches the text information. The target student model has the ability to generate images of different styles; When the target loss function is feature distillation, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including: Inputting the noise image into the teacher model corresponding to the style hint vector, extracting an intermediate feature map of the noise image through the teacher model to obtain a first intermediate feature map of the noise image, wherein the noise image is obtained by adding noise to the sample image; Inputting the noise image into the student model, extracting an intermediate feature map of the noise image through the student model, and obtaining a second intermediate feature map of the noise image; Calculating a similarity between a first intermediate feature map of the noise image and a second intermediate feature map of the noise image to obtain a feature distillation loss; Parameters of the student model are updated based on the feature distillation loss until the feature distillation loss satisfies a second preset condition.

2. The method according to claim 1, characterized in that Inputting the text information and the target style cue vector into the target student model, and outputting a target image through a diffusion generation process of the target student model includes: Inputting the text information into a text embedding layer in the target student model to convert the text information into a text vector; The target image is generated by modulating a generation mechanism of the target student model according to the text vector and the target style hint vector.

3. The method according to claim 1, characterized in that The target student model is obtained by the following steps: Deploying teacher models of different image styles on multiple image processing units of the distributed training framework; Determining a style hint vector corresponding to the teacher model for each image style, wherein the style hint vector is used to instruct the student model to simulate the generation mechanism of the image style of the corresponding teacher model; According to the style hint vector corresponding to the teacher model of each image style, the knowledge of the teacher models of multiple image styles is integrated into the student model with shared parameters through knowledge distillation to obtain the target student model.

4. The method according to claim 3, characterized in that Based on the style cue vector corresponding to the teacher model of each image style, the knowledge of the teacher models of multiple image styles is integrated into the student model with shared parameters through knowledge distillation. The target student model includes: Select sample images from the training sample set; adding noise to the sample image according to a predetermined noise scheduling strategy to obtain a noise image; The target student model is obtained by using the style cue vector corresponding to the teacher model of each image style and the noise image, and utilizing the target loss function to constrain the knowledge distillation process between the teacher model and the student model.

5. The method according to claim 4, characterized in that When the target loss function is score distillation, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained including: Inputting the noise image into a teacher model corresponding to the style hint vector, predicting the noise of the noise image by the teacher model, and obtaining a first score for the noise image; Inputting the noise image and the style cue vector into the student model, and predicting the noise of the noise image by the student model to obtain a second score for the noise image; calculating a difference between a first score of the noisy image and a second score of the noisy image to obtain a fractional distillation loss; Parameters of the student model are updated based on the fractional distillation loss until the fractional distillation loss satisfies a first preset condition.

6. The method according to claim 5, characterized in that When the target loss function is a multi-class adversarial loss, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including: Input the noise image into the teacher model corresponding to the style hint vector and output a real image; Inputting the noise image and the style hint vector into the student model and outputting a predicted image; Inputting the real image and the predicted image into a discriminator to obtain a prediction result; Based on the prediction results, calculating a multi-class adversarial loss; Parameters of the student model are updated based on the multi-class adversarial loss until the multi-class adversarial loss satisfies a third preset condition.

7. The method according to claim 6, characterized in that When the target loss function is score distillation, feature distillation, and multi-category adversarial loss, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including: Performing a weighted summation on the loss values ​​of the score distillation loss, the feature distillation loss, and the multi-class adversarial loss to obtain a total training loss; Based on the total training loss, updating the parameters of the student model; The process of inputting the noise image and the style hint vector, calculating the total training loss, and updating the parameters is repeatedly performed until a preset training stop condition is reached to obtain the target student model.

8. A diffusion model fusion method, characterized in that: include: Acquire text information uploaded by the client, wherein the text information includes description information of the target image to be generated; In a cloud server, a target style cue vector matching the text information is determined, wherein the target style cue vector is used to instruct a target student model to output an image of a target image style, the target student model being a student model obtained by performing knowledge distillation based on a plurality of teacher models with different image styles in a distributed training framework. The distributed training framework deploys teacher models with different image styles on a plurality of image processing units, and the target student model is trained in parallel on the plurality of image processing units while sharing parameters. Inputting the text information and the target style cue vector into the target student model, and having the target student model perform a diffusion generation process to output a target image; Returning the target image to the client; wherein the text information is processed through a semantic understanding and style mapping mechanism to determine a target style hint vector that matches the text information; Integrate the knowledge of teacher models of multiple image styles into a student model with shared parameters through knowledge distillation, and use a target loss function to constrain the knowledge distillation process between the teacher model and the student model to obtain the target student model, wherein the target loss function can be a combination of one or more of score distillation, feature distillation, and multi-class adversarial loss; Among them, a target student model is created. When the target student model receives text information, it generates an image that matches the text information. The target student model has the ability to generate images of different styles; When the target loss function is feature distillation, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including: Inputting the noise image into the teacher model corresponding to the style hint vector, extracting an intermediate feature map of the noise image through the teacher model to obtain a first intermediate feature map of the noise image, wherein the noise image is obtained by adding noise to the sample image; Inputting the noise image into the student model, extracting an intermediate feature map of the noise image through the student model, and obtaining a second intermediate feature map of the noise image; Calculating a similarity between a first intermediate feature map of the noise image and a second intermediate feature map of the noise image to obtain a feature distillation loss; Parameters of the student model are updated based on the feature distillation loss until the feature distillation loss satisfies a second preset condition.

9. A diffusion model fusion device, characterized in that: include: an acquiring unit, configured to acquire text information, wherein the text information includes description information of a target image to be generated; a first confirmation unit, configured to determine a target style cue vector that matches the text information, wherein the target style cue vector is used to instruct a target student model to output an image of a target image style, the target student model being a student model obtained by performing knowledge distillation based on a plurality of teacher models with different image styles in a distributed training framework, the distributed training framework deploying teacher models with different image styles on a plurality of image processing units, and the target student model being trained in parallel on the plurality of image processing units while sharing parameters; an output unit, configured to input the text information and the target style cue vector into the target student model, and have the target student model perform a diffusion generation process to output a target image; wherein the text information is processed through a semantic understanding and style mapping mechanism to determine a target style hint vector that matches the text information; Integrate the knowledge of teacher models of multiple image styles into a student model with shared parameters through knowledge distillation, and use a target loss function to constrain the knowledge distillation process between the teacher model and the student model to obtain the target student model, wherein the target loss function can be a combination of one or more of score distillation, feature distillation, and multi-class adversarial loss; The device is further configured to: create a target student model, wherein the target student model generates an image matching the text information when receiving the text information, and the target student model has the ability to generate images of different styles; The device is also used to, when the target loss function is feature distillation, input the noise image into the teacher model corresponding to the style hint vector, extract the intermediate feature map of the noise image through the teacher model, and obtain a first intermediate feature map of the noise image, wherein the noise image is obtained by adding noise to the sample image; input the noise image into the student model, extract the intermediate feature map of the noise image through the student model, and obtain a second intermediate feature map of the noise image; calculate the similarity between the first intermediate feature map of the noise image and the second intermediate feature map of the noise image to obtain a feature distillation loss; and update the parameters of the student model based on the feature distillation loss until the feature distillation loss meets a second preset condition.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored executable program, wherein when the executable program is run, the device where the computer-readable storage medium is located is controlled to execute the diffusion model fusion method according to any one of claims 1 to 8.

11. An electronic device, characterized in that: include: a memory storing an executable program; A processor is configured to run the program, wherein the program, when running, executes the diffusion model fusion method according to any one of claims 1 to 8.

12. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the diffusion model fusion method according to any one of claims 1 to 8 are implemented.

Images