Diffusion model fusion method and device and electronic equipment

Abstract

The invention discloses a diffusion model fusion method and device and electronic equipment. Relates to the field of artificial intelligence, and the method comprises the steps: obtaining text information which comprises description information of a to-be-generated target image; a target style prompt vector matched with the text information is determined, the target style prompt vector is used for indicating a target student model to output an image of a target image style, and the target student model is a student model obtained by performing knowledge distillation based on teacher models of various different image styles under a distributed training framework; and inputting the text information and the target style prompt vector into a target student model, executing a diffusion generation process by the target student model, and outputting a target image. Through the method and the device, the problem that the quality of generated images is relatively low when images of different styles are generated by a fusion model obtained by an existing model fusion method in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to a diffusion model fusion method, device and electronic equipment. Background Art

[0002] In the field of artificial intelligence, especially in generative models of deep learning, model diversity and specialization are its core features. As research deepens and technology matures, different models are often optimized for specific styles or tasks in text-to-image tasks. For example, some models are good at generating high-fidelity real-world images, while others excel in generating images in cartoon style, artistic style, or in specific fields.

[0003] When it is necessary to generate multiple images of different styles, the server usually needs to deploy multiple models to control the generation of different styles, which greatly increases the overhead. Client users usually need to switch between multiple models, which is very inflexible and time-consuming to load on the hard disk and memory, affecting the user experience. In order to solve this problem, it is necessary to fuse multiple different style models into a target model that covers the knowledge of various expert fields, but there is currently no good solution to achieve this goal. Existing model fusion methods face many challenges when dealing with requirements that span different fields or styles. For example, directly fusing a model that specializes in generating artistic style paintings with a cartoon image generation model often fails to properly handle the differences between the two distinct styles, resulting in the generated image containing both style features and failing to clearly display either style, causing style confusion and affecting image quality and user experience.

[0004] There is currently no effective solution to the problem that when a fusion model obtained by an existing model fusion method in the related art generates images of different styles, the generated images have low quality. Summary of the invention

[0005] The main purpose of the present application is to provide a diffusion model fusion method, device and electronic device to solve the technical problem in the related art that the fusion effect is poor when using the existing model fusion method to fuse models of different styles.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a diffusion model fusion method is provided. The method comprises: obtaining text information, wherein the text information includes description information of the target image to be generated; determining a target style hint vector matching the text information, wherein the target style hint vector is used to instruct the target student model to output an image of the target image style, and the target student model is a student model obtained by knowledge distillation based on a plurality of teacher models with different image styles under a distributed training framework, and the distributed training framework deploys teacher models with different image styles on a plurality of image processing units, and the target student model is trained in parallel on the plurality of image processing units while sharing parameters; inputting the text information and the target style hint vector into the target student model, and the target student model performs the diffusion generation process and outputs the target image.

[0007] Furthermore, the text information and the target style cue vector are input into the target student model, and the target image is output through the diffusion generation process of the target student model, including: inputting the text information into the text embedding layer in the target student model, converting the text information into a text vector; and generating the target image by modulating the generation mechanism of the target student model based on the text vector and the target style cue vector.

[0008] Furthermore, the target student model is obtained through the following steps: deploying teacher models of different image styles on multiple image processing units of a distributed training framework; determining a style hint vector corresponding to the teacher model for each image style, wherein the style hint vector is used to instruct the student model to simulate the generation mechanism of the image style of the corresponding teacher model; and integrating the knowledge of teacher models of multiple image styles into a student model with shared parameters through knowledge distillation based on the style hint vector corresponding to the teacher model for each image style, thereby obtaining the target student model.

[0009] Furthermore, based on the style cue vector corresponding to the teacher model of each image style, the knowledge of teacher models of multiple image styles is integrated into a student model with shared parameters through knowledge distillation to obtain a target student model, including: selecting a sample image from a training sample set; adding noise to the sample image according to a predetermined noise scheduling strategy to obtain a noisy image; and obtaining a target student model by using the style cue vector and the noise image corresponding to the teacher model of each image style and utilizing the target loss function to constrain the knowledge distillation process between the teacher model and the student model.

[0010] Furthermore, when the target loss function is fractional distillation, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and obtaining the target student model includes: inputting the noise image into the teacher model corresponding to the style hint vector, predicting the noise of the noise image through the teacher model, and obtaining a first score of the noise image; inputting the noise image and the style hint vector into the student model, predicting the noise of the noise image through the student model, and obtaining a second score of the noise image; calculating the difference between the first score of the noise image and the second score of the noise image to obtain the fractional distillation loss; updating the parameters of the student model based on the fractional distillation loss until the fractional distillation loss meets the first preset condition.

[0011] Furthermore, when the target loss function is feature distillation, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and obtaining the target student model includes: inputting the noise image into the teacher model corresponding to the style hint vector, extracting the intermediate feature map of the noise image through the teacher model, and obtaining the first intermediate feature map of the noise image; inputting the noise image into the student model, extracting the intermediate feature map of the noise image through the student model, and obtaining the second intermediate feature map of the noise image; calculating the similarity between the first intermediate feature map of the noise image and the second intermediate feature map of the noise image to obtain the feature distillation loss; updating the parameters of the student model based on the feature distillation loss until the feature distillation loss meets the second preset condition.

[0012] Furthermore, when the target loss function is a multi-category adversarial loss, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including: inputting the noise image into the teacher model corresponding to the style hint vector, and outputting the real image; inputting the noise image and the style hint vector into the student model, and outputting the predicted image; inputting the real image and the predicted image into the discriminator to obtain the prediction result; based on the prediction result, calculating the multi-category adversarial loss; updating the parameters of the student model based on the multi-category adversarial loss until the multi-category adversarial loss meets the third preset condition.

[0013] Furthermore, when the target loss function is score distillation, feature distillation and multi-category adversarial loss, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including: weighted summing the loss values ​​of the score distillation loss, feature distillation loss and multi-category adversarial loss to obtain the total training loss; updating the parameters of the student model based on the total training loss; repeatedly executing the process of inputting noise images and style hint vectors, calculating the total training loss, and updating the parameters until the preset training stop condition is reached to obtain the target student model.

[0014] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a diffusion model fusion method is provided, including: obtaining text information uploaded by a client, wherein the text information includes description information of a target image to be generated; in a cloud server, determining a target style hint vector matching the text information, wherein the target style hint vector is used to instruct a target student model to output an image of a target image style, and the target student model is a student model obtained by performing knowledge distillation based on teacher models of multiple different image styles under a distributed training framework, and the distributed training framework deploys teacher models of different image styles on multiple image processing units respectively, and the target student model is trained in parallel on multiple image processing units while sharing parameters; inputting the text information and the target style hint vector into the target student model, and the target student model executes a diffusion generation process and outputs a target image; and returning the target image to the client.

[0015] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a diffusion model fusion device is provided. The device includes: an acquisition unit, which is used to acquire text information, wherein the text information includes description information of the target image to be generated; a first confirmation unit, which is used to determine a target style hint vector that matches the text information, wherein the target style hint vector is used to instruct the target student model to output an image of the target image style, and the target student model is a student model obtained by knowledge distillation based on a plurality of teacher models with different image styles under a distributed training framework, and the distributed training framework deploys teacher models with different image styles on a plurality of image processing units, respectively, and the target student model is trained in parallel on the plurality of image processing units while sharing parameters; an output unit, which is used to input the text information and the target style hint vector into the target student model, and the target student model performs the diffusion generation process and outputs the target image.

[0016] Furthermore, the output unit includes: a conversion module, which is used to input text information into a text embedding layer in the target student model to convert the text information into a text vector; a production module, which is used to generate a target image by modulating a generation mechanism of the target student model based on the text vector and the target style prompt vector.

[0017] Furthermore, the device includes: a deployment unit, used to deploy teacher models of different image styles on multiple image processing units of a distributed training framework; a confirmation unit, used to determine the style hint vector corresponding to the teacher model of each image style, wherein the style hint vector is used to instruct the student model to simulate the generation mechanism of the image style of the corresponding teacher model; an integration unit, used to integrate the knowledge of teacher models of multiple image styles into a student model with shared parameters through knowledge distillation based on the style hint vector corresponding to the teacher model of each image style, so as to obtain a target student model.

[0018] Furthermore, the integration unit includes: a selection module, used to select sample images from a training sample set; an adding module, used to add noise to the sample image according to a predetermined noise scheduling strategy to obtain a noise image; a generation module, used to obtain a target student model by using the style hint vector and the noise image corresponding to the teacher model of each image style, and using the target loss function to constrain the knowledge distillation process between the teacher model and the student model.

[0019] Furthermore, the generation module includes: a first input submodule, used to input the noise image into the teacher model corresponding to the style cue vector, predict the noise of the noise image through the teacher model, and obtain a first score of the noise image; a second submodule, used to input the noise image and the style cue vector into the student model, predict the noise of the noise image through the student model, and obtain a second score of the noise image; a first calculation submodule, used to calculate the difference between the first score of the noise image and the second score of the noise image, and obtain the fractional distillation loss; a first update submodule, used to update the parameters of the student model based on the fractional distillation loss until the fractional distillation loss meets the first preset condition.

[0020] Furthermore, the generation module includes: a third input sub-module, used to input the noise image into the teacher model corresponding to the style hint vector, extract the intermediate feature map of the noise image through the teacher model, and obtain the first intermediate feature map of the noise image; a fourth input sub-module, used to input the noise image into the student model, extract the intermediate feature map of the noise image through the student model, and obtain the second intermediate feature map of the noise image; a second calculation sub-module, used to calculate the similarity between the first intermediate feature map of the noise image and the second intermediate feature map of the noise image, and obtain the feature distillation loss; a second update sub-module, used to update the parameters of the student model based on the feature distillation loss until the feature distillation loss meets the second preset condition.

[0021] Furthermore, the generation module includes: a fifth input submodule, used to input the noise image into the teacher model corresponding to the style hint vector, and output the real image; a sixth input submodule, used to input the noise image and the style hint vector into the student model, and output the predicted image; a seventh input submodule, used to input the real image and the predicted image into the discriminator to obtain the prediction result; a third calculation submodule, used to calculate the multi-category adversarial loss based on the prediction result; and a third update submodule, used to update the parameters of the student model based on the multi-category adversarial loss until the multi-category adversarial loss meets the third preset condition.

[0022] Furthermore, the generation module includes: a weighted summation sub-module, which is used to weighted sum the loss values ​​of the score distillation loss, the feature distillation loss, and the multi-category adversarial loss to obtain the total training loss; a fourth update sub-module, which is used to update the parameters of the student model based on the total training loss; a generation sub-module, which is used to repeatedly execute the process of inputting a noise image and a style hint vector, calculating the total training loss, and updating the parameters until a preset training stop condition is reached to obtain a target student model.

[0023] According to another aspect of the present application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the diffusion model fusion methods.

[0024] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for executing any diffusion model fusion method.

[0025] According to another aspect of the present application, a computer program product is provided, comprising computer instructions, which implement the steps of any one of the above diffusion model fusion methods when executed by a processor.

[0026] In an embodiment of the present application, by obtaining text information, wherein the text information includes description information of the target image to be generated; determining a target style hint vector that matches the text information, wherein the target style hint vector is used to instruct the target student model to output an image of the target image style, the target student model is a student model obtained by knowledge distillation based on a plurality of teacher models of different image styles under a distributed training framework, the distributed training framework deploys teacher models of different image styles on a plurality of image processing units respectively, and the target student model is trained in parallel on a plurality of image processing units while sharing parameters; the text information and the target style hint vector are input into the target student model, the target student model performs a diffusion generation process, and outputs the target image, thereby solving the problem that the fusion model obtained by the existing model fusion method in the related art generates images of different styles, and the generated image quality is low. In the present application, a method of generating a target image based on a target student model is adopted, and a target image having a specified style and consistent with the description of the text information is output through a diffusion model generation process of the target student model, so as to achieve the purpose of accurately generating images of different styles based on the target student model, thereby achieving the technical effect of improving the accuracy of generating images of different styles. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0028] Figure 1 It is a schematic diagram of an application scenario of a multi-style diffusion model fusion method provided according to an embodiment of the present application;

[0029] Figure 2 is a flowchart of a multi-style diffusion model fusion method provided in an embodiment of the present application;

[0030] Figure 3 is a flowchart of constructing a target student model according to an embodiment of the present application;

[0031] Figure 4 It is a schematic diagram of implementing model fusion on a distributed training framework according to an embodiment of the present application;

[0032] Figure 5 is a flowchart of integrating a teacher model into a student model through knowledge distillation according to an embodiment of the present application;

[0033] Figure 6 It is a flowchart of integrating a teacher model into a student model through fractional distillation according to an embodiment of the present application;

[0034] Figure 7 It is a flow chart of integrating a teacher model into a student model through feature distillation according to an embodiment of the present application;

[0035] Figure 8 It is a flowchart of integrating a teacher model into a student model through multi-category adversarial loss according to an embodiment of the present application;

[0036] Fig. 9 It is a flowchart of integrating a teacher model into a student model through score distillation, feature distillation and multi-category adversarial loss according to an embodiment of the present application;

[0037] Fig.10 is a schematic diagram of integrating a teacher model into a student model through score distillation, feature distillation, and multi-category adversarial loss according to an embodiment of the present application;

[0038] Fig.11 is a schematic diagram of a multi-style diffusion model fusion device provided according to an embodiment of the present application;

[0039] Fig.12 is a flowchart of another multi-style diffusion model fusion method provided according to an embodiment of the present application;

[0040] Fig.13It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0042] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0043] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0044] Example 1

[0045] The technical solution provided in this application is mainly implemented using large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than 10 trillion model parameters. The large model can also be called a cornerstone model / basic model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities.

[0046] It should be noted that when the large model is actually used, the pre-trained model can be fine-tuned through a small number of samples so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing, computer vision, speech processing and other fields, and can be specifically applied to computer vision tasks such as visual question answering, image description, image generation, etc. It can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiment of the present application, data processing through the target student model in the Wenshengtu scenario is used as an example for explanation.

[0047] According to an embodiment of the present application, a diffusion model fusion method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0048] Considering the huge number of model parameters of the large model and the limited computing resources of the mobile terminal, the above method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to these. Figure 1 In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface to implement the call of the large model, thereby implementing the method provided in the embodiment of the present application.

[0049] It should be noted that, with the rapid development of high-performance computing units, in other application scenarios, the above method provided in the embodiment of the present application can also be applied to the model all-in-one machine. In an optional embodiment, a plurality of models are built into the model all-in-one machine, and the user can choose to adjust a model as needed to obtain the user's own model, so that the high-performance computing unit built into the model all-in-one machine can directly call the adjusted model to execute the above method provided in the embodiment of the present application. In another optional embodiment, a trained model is built into the large model all-in-one machine, so that the high-performance computing unit built into the model all-in-one machine can directly call the model to execute the above method provided in the embodiment of the present application.

[0050] Furthermore, when users need to train their own models, they can also upload their own data sets through the client, which are sent by the client to the server, so that the server can adjust the pre-trained model with the data set to obtain the user's own model and then deploy it to the production environment. In order to facilitate users' needs for model adjustment, the server can provide complete adjustment tools, development frameworks and processes, and can support multiple adjustment strategies, so that the adjusted model can better adapt to applications in different fields and achieve a high degree of customization.

[0051] Under the above operating environment, this application provides Figure 2 The multi-style diffusion model fusion method shown. Figure 2 This is a flowchart of the multi-style diffusion model fusion method according to Example 1 of the present application.

[0052] S201, obtaining text information, wherein the text information includes description information of a target image to be generated.

[0053] Optionally, the above text information may be description information input by the user indicating to generate a target image of one style or to generate target images of multiple styles. For example, the text information may be “sunset on a summer beach”.

[0054] S202, determining a target style hint vector that matches the text information, wherein the target style hint vector is used to instruct a target student model to output an image of a target image style, and the target student model is a student model obtained by performing knowledge distillation based on a plurality of teacher models with different image styles under a distributed training framework. The distributed training framework deploys teacher models with different image styles on a plurality of image processing units, respectively, and the target student model is trained in parallel on the plurality of image processing units while sharing parameters.

[0055] The teacher model involved in this application can be a pre-trained, high-performance Vincent graph model, which acts as a "teacher" to guide the training of a simpler, smaller model (student model). The teacher model usually has a high accuracy and complex structure, and it can handle complex tasks and generate relatively sophisticated outputs.

[0056] The student model involved in this application learns by imitating the output of the teacher model, thereby achieving higher performance while maintaining a smaller model size. Therefore, the student model is a relatively small and simple model that aims to learn and imitate the behavior or prediction results of another more complex and better performing model (i.e., the teacher model). The design goal of the student model is to reduce computational complexity and resource consumption while maintaining high performance, making it more suitable for deployment in resource-constrained environments.

[0057] For example, there are several teacher models with the following image styles: Renaissance style, street art style, Japanese Ukiyo-e style, and modern photography style. Now we want to create a target student model that can generate images that match the text information when it receives text information. The user enters the text information: "Depict a quiet night, with a full moon hanging high and waves gently lapping on the beach." First, the implicit image style information needs to be identified from the text information. In this example, the scene described in the text information is relatively quiet and poetic, which may be closer to the Renaissance style or modern photography style. However, since there is no direct mention of a specific artistic style in the text, the style that best matches the text information can be determined through semantic understanding and style mapping mechanisms. For example, the keywords "night", "full moon", and "beach" may be associated with a specific vector that encodes the style features of tranquility, nature, and rich details, which is close to the Renaissance style. Through the above mapping mechanism, a target style hint vector that matches the text information can be determined. This vector contains enough information for the target student model to understand the required style and call the style knowledge of the teacher model when generating images to generate images that match the text description.

[0058] S203, inputting the text information and the target style cue vector into the target student model, and the target student model performs a diffusion generation process to output a target image.

[0059] In an optional embodiment, the text information and the target style cue vector are input into the target student model, and the target image is output through the diffusion generation process of the target student model, including:

[0060] Input the text information into the text embedding layer in the target student model to convert the text information into a text vector;

[0061] According to the text vector and the target style cue vector, the target image is generated by modulating the generation mechanism of the target student model.

[0062] Optionally, the text embedding layer that inputs the text information into the target student model can first convert each word in the text information into a vector, and then integrate each vector into a context-aware text vector through a sequence encoder. This vector contains the semantic information of the text information and can guide the details of image generation. The target style cue vector is input to the target student model. This target style cue vector is learned during the process of generating the target image and is used to trigger the target student model's ability to imitate the specified style. Different style cue vectors can correspond to different artistic styles, such as realism, cartoon, comic, impressionism, etc. The text vector and the style cue vector are used together to modulate the generation mechanism of the target student model, jointly affecting the parameters inside the target student model and guiding the target student model on how to generate images from noise. This modulation process ensures that the generated image not only conforms to the text description, but also has the specified style. The diffusion generation process can start from a completely noisy image, and the target student model removes the noise through a step-by-step back diffusion process to generate an image that matches the text information. At each step, the target student model predicts and removes part of the noise based on the current image state (i.e., the current noise level and image details), the text vector, and the style cue vector. This process is repeated until a clear target image is recovered from the noise. Finally, after a series of denoising steps, the target student model will output a clear image that conforms to the text description and the specified style (corresponding to the target image above).

[0063] For example, a user is using the target student model to generate an image. The text information input by the user is "Summer beach sunset" and the target style is "Van Gogh style". The text embedding layer of the target student model converts it into a text vector. This vector contains semantic information about summer, beach and sunset. Enter a "Van Gogh style" hint vector. This vector is learned during training and is used to modulate the target student model generation process to imitate Van Gogh's unique brushstrokes and color application. The text vector and style hint vector affect the generation mechanism of the target student model, guiding the target student model how to gradually recover the "Summer beach sunset" image that meets the description from the noise and make it have Van Gogh style characteristics. The target student model starts with a noisy image and gradually denoises it, and each step is guided by the text vector and style hint vector. During the denoising process, the model gradually recovers the summer beach scene while imitating Van Gogh's style brushstrokes and color application. Finally, the target model generates a clear, summer beach sunset image, and the image shows Van Gogh's style painting characteristics, such as strong colors and obvious brushstrokes.

[0064] Through the content of the above technical solution, combined with text information and style hint vectors, through the diffusion generation process, high-quality images that meet user needs can be generated, thereby improving the accuracy of generating images of different styles.

[0065] In summary, the diffusion model fusion method provided by the embodiment of the present application obtains text information, wherein the text information includes description information of the target image to be generated; determines a target style hint vector matching the text information, wherein the target style hint vector is used to instruct the target student model to output an image of the target image style, and the target student model is a student model obtained by knowledge distillation based on a plurality of teacher models of different image styles under a distributed training framework, and the distributed training framework deploys teacher models of different image styles on multiple image processing units respectively, and the target student model is trained in parallel on multiple image processing units while sharing parameters; inputs the text information and the target style hint vector into the target student model, and the target student model performs a diffusion generation process and outputs the target image, which solves the problem that the fusion model obtained by the existing model fusion method in the related art generates images of different styles, and the generated image quality is low. In the present application, a method of generating a target image based on a target student model is adopted, and a target image having a specified style and consistent with the description of the text information is output through a diffusion model generation process of a target student model, so as to achieve the purpose of accurately generating images of different styles based on the target student model, thereby achieving the technical effect of improving the accuracy of generating images of different styles.

[0066] In an optional embodiment, if Figure 3 As shown, the target student model can be obtained by the following steps:

[0067] S301, deploy teacher models of different image styles on multiple image processing units of the distributed training framework.

[0068] For example, there are three teacher models, each of which is good at generating images in the following three styles: Van Gogh style (Post-Impressionism), Japanese Ukiyo-e style (Ukiyo-e), and Surrealism style (Surrealism). The goal is to use a student model with shared parameters to learn and fuse these three styles so that images of corresponding styles can be generated based on different style cue vectors. Suppose that in a distributed training framework, there are three GPUs (that is, the image processing units mentioned above), and a teacher model is deployed on each GPU, that is, a model that is good at Van Gogh style, a model that is good at Japanese Ukiyo-e style, and a model that is good at Surrealism style. Deploying teacher models of different image styles in a distributed training framework is a technical strategy to efficiently utilize computing resources and accelerate model training and knowledge distillation processes.

[0069] S302, determining a style hint vector corresponding to the teacher model for each image style, wherein the style hint vector is used to instruct the student model to simulate a generation mechanism of the image style of the corresponding teacher model.

[0070] Optionally, before training begins, a specific style hint vector can be assigned to each style of the teacher model. A vector encoding table can be initialized, which contains three style hint vectors corresponding to the three styles of the teacher models mentioned above. These style hint vectors can effectively instruct the student model to imitate the teacher model of the corresponding style during training.

[0071] S303 , based on the style hint vector corresponding to the teacher model of each image style, the knowledge of the teacher models of multiple image styles is integrated into the student model of shared parameters through knowledge distillation to obtain the target student model.

[0072]

[0073] For example, after the training is completed, the target student model not only has the ability to generate images in the style of Van Gogh, but also can generate images in the style of Ukiyo-e and surrealism. It can also flexibly switch between these image styles according to the input style hint vector to achieve the mixing and innovation of image styles. If the input is a vector in the style of Van Gogh, the student model will generate an image with the characteristics of Van Gogh style; if the input is a vector that mixes the styles of Ukiyo-e and surrealism, the student model can generate a new image style that has both the flatness of Ukiyo-e and the strange elements of surrealism.

[0074] In summary, the target student model obtained through the above technical solutions achieves high efficiency and flexibility in multi-style generation. It can not only independently replicate various artistic styles, but also innovatively mix and fuse these styles, bringing a more general and practical image generation solution to the field of literary images.

[0075] Optionally, the embodiment of the present application also provides a framework for implementing model fusion knowledge distillation on a distributed training framework. Figure 4As shown. In the GPU cluster, a teacher model is configured on each GPU, and the student model shares parameters on each GPU, thereby efficiently utilizing the GPU. The network structure of the student model is basically the same as the basic model, except that a small number of additional parameters are introduced to process the input style index i. The prior distributions of different models are maintained as an encoding table of N learnable vectors, and these learnable vectors are called the above-mentioned style cue vectors. These style cue vectors are used to specify the style of the image and modulate the student model to imitate the corresponding teacher model. Based on the style cue vector corresponding to the teacher model of each image style, the knowledge of teacher models of multiple image styles is integrated into the student model with shared parameters through knowledge distillation to obtain the target student model.

[0076] The steps of the process can be as follows Figure 5 shown.

[0077] S501, selecting a sample image from a training sample set.

[0078] For example, it is necessary to fuse the teacher models of three styles: Van Gogh style, Japanese Ukiyo-e style, and surrealist style to obtain the target student model. Sample images can be randomly selected from a large dataset containing a wide range of images. These sample images can be natural scenery, portraits, works of art, etc., which cover a variety of styles and contents to ensure that the target student model can learn rich style features and generation capabilities. You can select Van Gogh's A image, a typical Japanese Ukiyo-e work, and a surrealist painter Dali's B image from an art painting dataset.

[0079] S502: adding noise to the sample image according to a predetermined noise scheduling strategy to obtain a noisy image.

[0080] Optionally, the above-mentioned predetermined noise scheduling strategy can be linear noise scheduling, exponential noise scheduling, adaptive noise scheduling, staged noise scheduling, etc. In linear noise scheduling, the noise level increases or decreases linearly over time, and the linear scheduling strategy is easy to understand and implement. In exponential noise scheduling, the noise intensity changes according to an exponential function over time. Exponential scheduling can accelerate the training and convergence of the model, especially in the early generation stage of the image. In adaptive noise scheduling, the noise level is dynamically adjusted according to the current state of the image, which can more intelligently manage the addition and removal of noise and avoid introducing too much or too little noise in certain parts of the image. In staged noise scheduling, the denoising process is divided into several stages, each stage using different noise intensities and types, which helps the model focus on different features of the image at different stages. A predetermined noise scheduling strategy is used to artificially add noise to the sample image to obtain a noisy image. The degree of noise addition can be adjusted according to the stage and needs of training, with the aim of enabling the model to learn the ability to restore images at different noise levels.

[0081] S503, by using the style cue vector and the noise image corresponding to the teacher model of each image style, and utilizing the target loss function to constrain the knowledge distillation process between the teacher model and the student model, a target student model is obtained.

[0082] Optionally, the above objective loss function can be a combination of one or more of score distillation, feature distillation and multi-category adversarial loss. Score distillation ensures that the student model can learn the score prediction ability of the teacher model, feature distillation promotes the learning of style and features, and multi-category adversarial loss further improves the authenticity of the generated images and the differentiation of styles. By combining these loss functions, the student model can learn more comprehensive and in-depth knowledge in a shorter time, improving the efficiency of knowledge distillation.

[0083] In an optional embodiment, Figure 6 It is a flowchart of integrating the teacher model into the student model through fractional distillation according to an embodiment of the present application. When the target loss function is fractional distillation, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including:

[0084] S601, input the noise image into the teacher model corresponding to the style hint vector, predict the noise of the noise image through the teacher model, and obtain a first score of the noise image.

[0085] For example, to train the target student model to generate Van Gogh-style images, first select an image from the training data set, such as Van Gogh's A image, and then add noise to the image according to the predetermined noise scheduling strategy to obtain a noisy version of A image, which is the above-mentioned noise image. Next, the noise image is input into the Van Gogh-style teacher model, which is a trained model that is good at generating Van Gogh-style images. The teacher model will predict the noise distribution of this noise image and output a score, which is the first score of the above-mentioned noise image, indicating the prediction of the noise distribution.

[0086] S602, inputting the noise image and the style cue vector into the student model, predicting the noise of the noise image by the student model, and obtaining a second score of the noise image.

[0087] For example, the noisy image A and a specified style hint vector (representing the style of Van Gogh) are input into the student model. The student model tries to predict a noise distribution similar to that of the teacher model based on this style hint vector and the noisy image, and obtains the second score of the noisy image mentioned above.

[0088] S603, calculating the difference between the first score of the noisy image and the second score of the noisy image to obtain the score distillation loss.

[0089] For example, after the student model predicts a score, the difference between its predicted score (the second score) and the score of the teacher model (the first score) can be calculated. This difference is expressed as the score distillation loss, which is usually quantified using the mean squared error (MSE) or a more complex loss function. The goal is to make this loss as small as possible, which means that the predicted score of the student model should be as close to the score of the teacher model as possible.

[0090] Alternatively, the fractional distillation loss function can be:

[0091]

[0092] Where N is the number of teacher models, i is the style index, is the training sample image, t is the time step, are the student model parameters, are the teacher model parameters, is the noise score predicted by the model for noisy images.

[0093] S604, updating the parameters of the student model based on the fractional distillation loss until the fractional distillation loss satisfies a first preset condition.

[0094] Optionally, the parameters of the student model are updated based on the fractional distillation loss. The parameters of the student model can be updated through a back-propagation algorithm. During the training process, the student model will continue to learn and adjust to reduce the fractional distillation loss until the loss reaches a first preset condition. The first preset condition can be that the fractional distillation loss is lower than a certain threshold, or that the training reaches a limited number of iterations (for example, 100,000 times). Once this condition is met, the student model has reached the expected level in imitating the style of the teacher model, and it can be considered that knowledge distillation has been completed in the current style.

[0095] For example, during the training process, the score distillation loss value is continuously monitored. If after a certain iteration, the score distillation loss is lower than the threshold (for example, 0.01), it can be considered that the student model has imitated the current style accurately enough, and you can consider stopping the learning of the current style or switching to the learning of the next style. At the same time, it will also check whether the maximum number of iterations (for example, 100,000) has been reached. If it has been reached, the training can be stopped even if the loss value has not yet met the threshold. Once the student model reaches the predetermined score distillation loss threshold for all styles, or the overall number of training iterations reaches the set upper limit, it can be considered that the knowledge distillation process is complete and the student model has the ability to integrate multiple styles.

[0096] In summary, through the above technical solutions, the knowledge distillation process based on fractional distillation can efficiently extract and transfer style knowledge from the teacher model, so that the generated student model has the ability to generate multiple image styles.

[0097] In an optional embodiment, Figure 7 It is a flowchart of integrating the teacher model into the student model through feature distillation according to an embodiment of the present application. When the target loss function is feature distillation, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including:

[0098] S701, input the noise image into the teacher model corresponding to the style hint vector, extract the intermediate feature map of the noise image through the teacher model, and obtain the first intermediate feature map of the noise image.

[0099] Optionally, the intermediate feature map of the noisy image comes from the internal layer of the teacher model, which can be the output of a convolutional layer or a Transformer layer.

[0100] For example, first select an image A of Van Gogh's painting from the training data set, add noise according to the noise scheduling strategy, and obtain the noisy version of image A, which is the noise image mentioned above. Then use the Van Gogh style teacher model to process the noisy image. In this step, the focus is on the middle layer of the teacher model, that is, the intermediate feature map generated by the teacher model when processing the image. These feature maps contain information such as the structure, texture, color, etc. of the image, as well as features specific to the Van Gogh style. After the teacher model processes the noisy image, the feature map of its middle layer is obtained, that is, the first intermediate feature map.

[0101] S702, input the noise image into the student model, extract the intermediate feature map of the noise image through the student model, and obtain a second intermediate feature map of the noise image.

[0102] For example, the above-mentioned noisy image A is input into the student model, and the target architecture of the student model is similar to that of the teacher model. The student model generates its own intermediate feature maps, namely the second intermediate feature maps, which reflect the student model's understanding and processing of the noisy image.

[0103] S703, calculating the similarity between the first intermediate feature map of the noise image and the second intermediate feature map of the noise image to obtain a feature distillation loss.

[0104] Optionally, after obtaining the intermediate feature maps of the teacher model and the student model, the similarity between them is calculated to quantify whether the feature learning effect of the student model is close to that of the teacher model. The similarity calculation method can be mean square error (MSE), structural similarity index (SSIM), cosine similarity, feature matching loss, etc. For example, MSE can be used to measure the difference between two feature maps, and the calculated value is the feature distillation loss. The goal is to make this loss as small as possible to ensure that the feature representation of the student model in the intermediate layer is consistent with or very close to that of the teacher model, so as to learn the style and feature expression ability of the teacher model.

[0105] Optionally, the feature distillation loss function can be:

[0106]

[0107] Where N is the number of teacher models, is the training sample image, t is the time step, are the student model parameters, are the teacher model parameters, is the feature map of the student model, is the feature map of the teacher model, i is the feature map of the student model at layer i, and j is the feature map of the teacher model at layer j.

[0108] S704, updating the parameters of the student model based on the feature distillation loss until the feature distillation loss meets the second preset condition.

[0109] Optionally, a back-propagation algorithm can be used to adjust the parameters of the student model according to the feature distillation loss to reduce the feature difference. The training process continues until the feature distillation loss reaches a second preset condition, which can be that the feature distillation loss is lower than a certain threshold, such as 0.001, or reaches a limited maximum number of iterations, for example, 50,000. Once this condition is met, it is considered that the student model has mastered the style knowledge of the teacher model at the intermediate feature level and completed the feature distillation of the style.

[0110] Through the above technical scheme, the use of feature distillation loss ensures that the target student model not only matches the style of the teacher model in the final generated image, but also the feature representation during the generation process is consistent with the teacher model, enhancing the quality of knowledge transfer and style learning during the knowledge distillation process.

[0111] In an optional embodiment, Figure 8It is a flowchart of integrating the teacher model into the student model through multi-category adversarial loss according to an embodiment of the present application. When the target loss function is multi-category adversarial loss, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including:

[0112] S801, input the noise image into the teacher model corresponding to the style hint vector, and output the real image.

[0113] For example, you can select a real image with noise (a Van Gogh-style image) and then input it into the teacher model along with a style hint vector representing the style of Van Gogh. The teacher model is a trained generative model that can output a real image with a high similarity to the original style based on the input noise and style hint vector. If the input is a Van Gogh-style style hint vector, the teacher model will generate an image in the style of Van Gogh.

[0114] S802, input the noise image and the style cue vector into the student model, and output the predicted image.

[0115] For example, using the same noisy image and style hint vector as input, pass it to the student model. The goal of the student model is to learn and imitate the behavior of the teacher model, so it also tries to generate an image that matches the input style hint vector. In the early stages of training, the predicted images generated by the student model may be very different from the real images generated by the teacher model, but as training progresses, the student model will gradually improve the quality of its generated images.

[0116] S803, input the real image and the predicted image into the discriminator to obtain the prediction result.

[0117] Optionally, in order to evaluate the quality of the images generated by the student model, an additional model - the discriminator is used. The discriminator is a classifier that can determine whether the input image is a real image and which style it belongs to. Assume that there are N teacher models, each corresponding to a specific style. The dimension of the classification head of the discriminator is set to 2N, which means that it can recognize 2N different categories, where the first N categories correspond to the real images of the N target styles, and the last N categories correspond to the predicted images of these N styles (i.e., the images generated by the student model). The cleverness of this design is that it not only allows the discriminator to distinguish between real images and predicted images, but also allows it to further distinguish between real images of different styles, and the difference between real and predicted images of the same style. In this way, the discriminator can learn more detailed discrimination capabilities, while also providing richer feedback to the student model, guiding it to generate images that are more in line with the specified style.

[0118] Optionally, the tasks of the discriminator can include distinguishing between real and fake categories and distinguishing between different styles. Distinguishing between real and fake categories is to learn to distinguish between real images and fake images. For real images of each style, the goal of the discriminator is to classify them into the correct style category in the first N categories; and for images generated by the student model in the same style, the goal of the discriminator is to classify them into the fake category of the corresponding style in the last N categories. In addition to real and fake classification, the discriminator also needs to learn to distinguish between images of different styles. This means that even if two images are real, if their styles are different, the discriminator can correctly classify them into different style categories.

[0119] For example, at this stage, the real image generated by the teacher model and the predicted image generated by the student model are simultaneously input into the discriminator, which outputs the classification results and the real / fake probabilities of the real and predicted images. Then, the multi-category adversarial loss is calculated based on the prediction results of the discriminator. The specific calculation method involves tuning the loss function for each category (real style) and each model (student model, teacher model) to ensure that the image generated by the student model is close to the output of the teacher model in terms of style attributes and authenticity.

[0120] S804, based on the prediction results, calculate the multi-category adversarial loss.

[0121] Optionally, the multi-class adversarial loss function can be:

[0122]

[0123] Where N is the number of teacher models, i is the style index, is the training sample image, t is the time step, are the student model parameters, are the teacher model parameters.

[0124] S805, updating the parameters of the student model based on the multi-category adversarial loss until the multi-category adversarial loss satisfies a third preset condition.

[0125] Optionally, based on the calculated multi-class adversarial loss, the back-propagation algorithm can be used to update the parameters of the student model, with the goal of reducing the difference between the images generated by the student model and the real images, while ensuring that the images generated by the student model can be correctly classified. The training process continues until the multi-class adversarial loss reaches a third preset condition, which can be that the multi-class adversarial loss is lower than a certain threshold (such as 0.005), or the training reaches a predetermined number of iterations (such as 30,000). When the student model performs well on the multi-class adversarial loss, it can be considered that it has been able to generate real images that are similar to the teacher model and can be identified as a specified style, completing the knowledge distillation process.

[0126] By adding the supervision of the discriminator, the above technical solution allows the student model to more efficiently learn and integrate style knowledge from different teacher models, ultimately achieving the goal of multi-style generation in the target student model.

[0127] In an optional embodiment, Fig. 9 It is a flowchart of integrating the teacher model into the student model through score distillation, feature distillation and multi-category adversarial loss according to an embodiment of the present application. When the target loss function is score distillation, feature distillation and multi-category adversarial loss, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including:

[0128] S901, perform weighted summation of the loss values ​​of the score distillation loss, feature distillation loss, and multi-category adversarial loss to obtain the total training loss.

[0129] Optionally, the above three loss functions are used together, and the final optimization goal is a weighted combination of the three:

[0130]

[0131] in, is the fractional distillation loss function, is the feature distillation loss function, is a multi-class adversarial loss function, is the weight corresponding to the feature distillation loss function, Corresponding weights for the multi-category adversarial loss distillation loss function.

[0132] Optionally, through weighted combination, the student model can be guided to learn accurate score predictions to reproduce the generation process of the teacher model; to learn rich intermediate layer features to grasp the deep representation of style; and to generate images that can pass the multi-category adversarial training of the discriminator to improve the realism and style consistency of the generated images. This combined training strategy ensures that the student model can achieve a good state in terms of style fusion, image quality and generation efficiency.

[0133] S902, based on the total training loss, update the parameters of the student model.

[0134] Optionally, the total training loss is obtained, and the student model parameters can be updated based on the total training loss through the back-propagation algorithm. The back-propagation algorithm calculates the gradient of the student model parameters based on the total training loss, and then uses the optimizer to adjust the parameters to reduce the loss. This process is repeated until the parameters of the student model converge or the predetermined training stop condition is reached.

[0135] Optionally, during the training process, the weight coefficients can be dynamically adjusted according to the learning progress and performance of the student model. For example, in the early stages of training, score distillation and feature distillation can be given greater weights to ensure that the student model can accurately learn score predictions and style features from the teacher model; in the later stages of training, the weight of the multi-class adversarial loss can be increased to improve the visual quality and style differentiation of the images generated by the student model.

[0136] Optionally, in order to find a better combination of weights, hyperparameter tuning can be performed, such as using grid search, random search, or Bayesian optimization. The goal is to find a balance between learning multiple capabilities so that the student model can achieve a good state in style fusion, image quality, and generation efficiency.

[0137] Optionally, during the evaluation phase of the training process, you can check the performance metrics of the student model in different styles, such as FID score, PSNR (peak signal-to-noise ratio), SSIM (structural similarity index), etc. to determine whether the weights need to be adjusted.

[0138] S903, repeatedly executing the process of inputting the noise image and the style hint vector, calculating the total training loss, and updating the parameters until the preset training stop condition is reached, and obtaining the target student model.

[0139] Optionally, the training process is an iterative process, where the noise image and style cue vector are continuously input to the student model, the three losses mentioned above are calculated, the total training loss is obtained by weighted summation, and then the parameters of the student model are updated. This process is repeated continuously, and each iteration attempts to reduce the total training loss value, so that the student model can better learn and integrate the knowledge of the teacher model.

[0140] Optionally, the preset training stop condition can be that the training reaches a limited maximum number of iterations (such as 50,000 iterations), the total training loss is lower than a certain threshold (such as 0.005), or the performance of the student model on the validation set reaches a certain standard. When any of the stop conditions is met, the training process ends, and the target student model obtained can integrate the capabilities of multiple teacher models to achieve efficient generation of images in multiple styles, while ensuring image quality, style accuracy and authenticity.

[0141] In summary, the target student model constructed through the above scheme can not only learn and integrate knowledge from multiple teacher models, but also maintain a high degree of style consistency and image authenticity when generating images, which greatly improves the generalization ability and application value of the target student model.

[0142] According to the process of integrating the teacher model into the student model through score distillation, feature distillation and multi-category adversarial loss provided in the embodiment of the present application, the following can be done: Fig.10 As shown in the figure. The noisy images are input to the student model and the teacher model respectively, and the style hint vector is input to the student model. The score distillation loss is used to learn the score prediction ability of the teacher model, ensuring that the student model can reproduce the data distribution modeled by each teacher model, that is, the score prediction ability of images of different styles. The feature distillation loss is used to promote the learning of style and features by comparing the feature maps of the student model and the teacher model at some intermediate layers, thereby enhancing the knowledge transfer effect. The multi-category adversarial loss can introduce the concept of generative adversarial network, which not only requires the generated image to deceive the discriminator to be considered real, but also requires the discriminator to correctly identify the style of the image, thereby further improving the style consistency and authenticity of the image generated by the student model. Finally, by jointly using these three loss functions, the parameters of the student model are updated based on the total training loss. Therefore, by using the score distillation, feature distillation and multi-category adversarial loss in combination, a more powerful and flexible target student model can be constructed, which can not only learn and integrate knowledge from multiple teacher models, but also maintain a high degree of style consistency and image authenticity when generating images, greatly improving the generalization ability and application value of the target student model.

[0143] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0144] Example 2

[0145] The embodiment of the present application also provides a diffusion model fusion device. It should be noted that the diffusion model fusion device of the embodiment of the present application can be used to execute the diffusion model fusion method provided in the embodiment of the present application. The diffusion model fusion device provided in the embodiment of the present application is introduced below.

[0146] According to an embodiment of the present application, a device for implementing the above diffusion model fusion method is also provided, such as Fig.11 As shown, the device includes: an acquisition unit 1101, a first confirmation unit 1102 and an output unit 1103.

[0147] Specifically, the acquisition unit 1101 is used to acquire text information, wherein the text information includes description information of the target image to be generated;

[0148] A first confirmation unit 1102 is used to determine a target style hint vector that matches the text information, wherein the target style hint vector is used to instruct a target student model to output an image of a target image style, and the target student model is a student model obtained by performing knowledge distillation based on a plurality of teacher models with different image styles in a distributed training framework, and the distributed training framework deploys teacher models with different image styles on a plurality of image processing units, respectively, and the target student model is trained in parallel on the plurality of image processing units while sharing parameters;

[0149] The output unit 1103 is used to input the text information and the target style cue vector into the target student model, and the target student model performs the diffusion generation process to output the target image.

[0150] In summary, the diffusion model fusion device provided in the embodiment of the present application obtains text information through the acquisition unit 1101, wherein the text information includes description information of the target image to be generated; the first confirmation unit 1102 determines a target style prompt vector matching the text information, wherein the target style prompt vector is used to instruct the target student model to output an image of the target image style, and the target student model is a student model obtained by performing knowledge distillation based on teacher models of multiple different image styles under a distributed training framework. The distributed training framework deploys teacher models of different image styles on multiple image processing units, and the target student model is trained in parallel on multiple image processing units while sharing parameters; the output unit 1103 inputs the text information and the target style prompt vector into the target student model, and the target student model executes the diffusion generation process to output the target image, thereby solving the problem that the fusion model obtained by the existing model fusion method in the related art generates images of different styles and the generated image quality is low. In the present application, a method of generating a target image based on a target student model is adopted. Through the target student model diffusion model generation process, a target image that is consistent with the text information description and has a specified style is output, thereby achieving the purpose of accurately generating images of different styles based on the target student model, thereby achieving the technical effect of improving the accuracy of generating images of different styles.

[0151] Optionally, in the diffusion model fusion device provided in the embodiment of the present application, the output unit 1103 includes: a conversion module, used to input text information into the text embedding layer in the target student model, and convert the text information into a text vector; a production module, used to generate a target image by modulating the generation mechanism of the target student model based on the text vector and the target style prompt vector.

[0152] Optionally, the diffusion model fusion device provided in the embodiment of the present application includes: a deployment unit, used to deploy teacher models of different image styles on multiple image processing units of a distributed training framework; a confirmation unit, used to determine the style hint vector corresponding to the teacher model of each image style, wherein the style hint vector is used to instruct the student model to simulate the generation mechanism of the image style of the corresponding teacher model; an integration unit, used to integrate the knowledge of teacher models of multiple image styles into a student model with shared parameters through knowledge distillation based on the style hint vector corresponding to the teacher model of each image style, so as to obtain a target student model.

[0153] Optionally, in the diffusion model fusion device provided in an embodiment of the present application, the integration unit includes: a selection module, used to select a sample image from a training sample set; an adding module, used to add noise to the sample image according to a predetermined noise scheduling strategy to obtain a noise image; a generation module, used to obtain a target student model by using the style hint vector and the noise image corresponding to the teacher model of each image style, and using the target loss function to constrain the knowledge distillation process between the teacher model and the student model.

[0154] Optionally, in the diffusion model fusion device provided in the embodiment of the present application, the generation module includes: a first input submodule, used to input the noise image into the teacher model corresponding to the style hint vector, predict the noise of the noise image through the teacher model, and obtain a first score of the noise image; a second submodule, used to input the noise image and the style hint vector into the student model, predict the noise of the noise image through the student model, and obtain a second score of the noise image; a first calculation submodule, used to calculate the difference between the first score of the noise image and the second score of the noise image, and obtain a fractional distillation loss; a first update submodule, used to update the parameters of the student model based on the fractional distillation loss until the fractional distillation loss meets a first preset condition.

[0155] Optionally, in the diffusion model fusion device provided in the embodiment of the present application, the generation module includes: a third input submodule, used to input the noise image into the teacher model corresponding to the style hint vector, extract the intermediate feature map of the noise image through the teacher model, and obtain the first intermediate feature map of the noise image; a fourth input submodule, used to input the noise image into the student model, extract the intermediate feature map of the noise image through the student model, and obtain the second intermediate feature map of the noise image; a second calculation submodule, used to calculate the similarity between the first intermediate feature map of the noise image and the second intermediate feature map of the noise image, and obtain the feature distillation loss; a second update submodule, used to update the parameters of the student model based on the feature distillation loss until the feature distillation loss meets the second preset condition.

[0156] Optionally, in the diffusion model fusion device provided in the embodiment of the present application, the generation module includes: a fifth input submodule, used to input the noise image into the teacher model corresponding to the style hint vector, and output the real image; a sixth input submodule, used to input the noise image and the style hint vector into the student model, and output the predicted image; a seventh input submodule, used to input the real image and the predicted image into the discriminator to obtain the prediction result; a third calculation submodule, used to calculate the multi-category adversarial loss based on the prediction result; and a third update submodule, used to update the parameters of the student model based on the multi-category adversarial loss until the multi-category adversarial loss meets the third preset condition.

[0157] Optionally, in the diffusion model fusion device provided in the embodiment of the present application, the generation module includes: a weighted summation submodule, which is used to weighted sum the loss values ​​of the score distillation loss, the feature distillation loss, and the multi-category adversarial loss to obtain the total training loss; a fourth update submodule, which is used to update the parameters of the student model based on the total training loss; and a generation submodule, which is used to repeatedly execute the process of inputting a noise image and a style hint vector, calculating the total training loss, and updating the parameters until a preset training stop condition is reached to obtain a target student model.

[0158] It should be noted that the acquisition unit 1101, the first confirmation unit 1102 and the output unit 1103 correspond to steps S201 to S203 in Example 1, and the three units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules may also be part of the device and may be run in the computer terminal 10 provided in Example 1.

[0159] Example 3

[0160] According to an embodiment of the present application, another diffusion model fusion method is also provided, such as Fig.12 As shown, the method includes:

[0161] S1201, obtaining text information uploaded by a client, wherein the text information includes description information of a target image to be generated;

[0162] S1202, in the cloud server, determining a target style cue vector that matches the text information, wherein the target style cue vector is used to instruct a target student model to output an image of a target image style, and the target student model is a student model obtained by performing knowledge distillation based on a plurality of teacher models of different image styles in a distributed training framework, and the distributed training framework deploys teacher models of different image styles on a plurality of image processing units, respectively, and the target student model is trained in parallel on the plurality of image processing units while sharing parameters;

[0163] S1203, inputting the text information and the target style cue vector into the target student model, and the target student model performs a diffusion generation process to output a target image;

[0164] S1204: Return the target image to the client.

[0165] In the cloud server, the diffusion model fusion method is the same as the method in the first embodiment, and will not be repeated here.

[0166] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0167] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0168] Example 4

[0169] The embodiment of the present application may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal or an electronic device.

[0170] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0171] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the diffusion model fusion method: obtaining text information, wherein the text information includes description information of the target image to be generated; determining a target style prompt vector that matches the text information, wherein the target style prompt vector is used to instruct the target student model to output an image of the target image style, and the target student model is a student model obtained by performing knowledge distillation based on teacher models of multiple different image styles under a distributed training framework. The distributed training framework deploys teacher models of different image styles on multiple image processing units, and the target student model is trained in parallel on multiple image processing units while sharing parameters; the text information and the target style prompt vector are input into the target student model, and the target student model executes the diffusion generation process to output the target image.

[0172] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the diffusion model fusion method: inputting text information and target style cue vectors into the target student model, and outputting the target image through the diffusion generation process of the target student model, including: inputting text information into the text embedding layer in the target student model, converting the text information into a text vector; generating a target image by modulating the generation mechanism of the target student model based on the text vector and the target style cue vector.

[0173] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the diffusion model fusion method: the target student model is obtained by the following steps: deploying teacher models of different image styles on multiple image processing units of a distributed training framework; determining the style hint vector corresponding to the teacher model of each image style, wherein the style hint vector is used to indicate the student model to simulate the generation mechanism of the image style of the corresponding teacher model; based on the style hint vector corresponding to the teacher model of each image style, integrating the knowledge of teacher models of multiple image styles into a student model with shared parameters through knowledge distillation to obtain the target student model.

[0174] Optionally, the computer terminal can execute the program code of the following steps in the diffusion model fusion method: based on the style cue vector corresponding to the teacher model of each image style, the knowledge of the teacher models of multiple image styles is integrated into the student model of shared parameters through knowledge distillation to obtain the target student model, including: selecting a sample image from the training sample set; adding noise to the sample image according to a predetermined noise scheduling strategy to obtain a noise image; by using the style cue vector and the noise image corresponding to the teacher model of each image style, and using the target loss function to constrain the knowledge distillation process between the teacher model and the student model to obtain the target student model.

[0175] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the diffusion model fusion method: when the target loss function is fractional distillation, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including: inputting the noise image into the teacher model corresponding to the style hint vector, predicting the noise of the noise image through the teacher model, and obtaining a first score of the noise image; inputting the noise image and the style hint vector into the student model, predicting the noise of the noise image through the student model, and obtaining a second score of the noise image; calculating the difference between the first score of the noise image and the second score of the noise image to obtain the fractional distillation loss; updating the parameters of the student model based on the fractional distillation loss until the fractional distillation loss meets the first preset condition.

[0176] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the diffusion model fusion method: when the target loss function is feature distillation, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including: inputting the noise image into the teacher model corresponding to the style hint vector, extracting the intermediate feature map of the noise image through the teacher model, and obtaining the first intermediate feature map of the noise image; inputting the noise image into the student model, extracting the intermediate feature map of the noise image through the student model, and obtaining the second intermediate feature map of the noise image; calculating the similarity between the first intermediate feature map of the noise image and the second intermediate feature map of the noise image to obtain the feature distillation loss; updating the parameters of the student model based on the feature distillation loss until the feature distillation loss meets the second preset condition.

[0177] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the diffusion model fusion method: when the target loss function is a multi-category adversarial loss, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including: inputting the noise image into the teacher model corresponding to the style hint vector, and outputting the real image; inputting the noise image and the style hint vector into the student model, and outputting the predicted image; inputting the real image and the predicted image into the discriminator to obtain the predicted result; based on the predicted result, calculating the multi-category adversarial loss; updating the parameters of the student model based on the multi-category adversarial loss until the multi-category adversarial loss meets the third preset condition.

[0178] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the diffusion model fusion method: when the target loss function is score distillation, feature distillation and multi-category adversarial loss, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained, including: weighted summing the loss values ​​of the score distillation loss, feature distillation loss, and multi-category adversarial loss to obtain the total training loss; based on the total training loss, updating the parameters of the student model; repeatedly executing the process of inputting noise images and style hint vectors, calculating the total training loss, and updating parameters until the preset training stop condition is reached to obtain the target student model.

[0179] In an embodiment of the present application, a solution for diffusion model fusion is provided. By obtaining text information, wherein the text information includes description information of the target image to be generated; determining a target style hint vector that matches the text information, wherein the target style hint vector is used to instruct the target student model to output an image of the target image style, the target student model is a student model obtained by knowledge distillation based on a plurality of teacher models of different image styles under a distributed training framework, the distributed training framework deploys teacher models of different image styles on a plurality of image processing units respectively, and the target student model is trained in parallel on a plurality of image processing units while sharing parameters; the text information and the target style hint vector are input into the target student model, the target student model executes the diffusion generation process, and outputs the target image, thus solving the technical problem of poor fusion effect when using the existing model fusion method to fuse models of different styles in the related technology. In the present application, a method for generating a target image based on a target student model is adopted. Text information is input into the target student model, and a target style prompt vector matching the text information is determined. The target style vector can instruct the target student model to output an image in the style of the target image. The target student model outputs the target image based on the text information and the target style prompt vector, thereby achieving the purpose of accurately generating images of different styles based on the target student model, thereby realizing the technical effect of improving the accuracy of generating images of different styles.

[0180] Optionally, Fig.13 is a structural block diagram of an electronic device according to an embodiment of the present application. Fig.13 As shown, the electronic device may include: one or more ( Fig.13 Only one is shown) processor 1302, memory 1304, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0181] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the diffusion model fusion method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the above-mentioned diffusion model fusion method is realized. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0182] The processor may call the information and application program stored in the memory through the transmission device to execute the above steps in the above diffusion model fusion method.

[0183] By adopting the embodiment of the present application, a solution for diffusion model fusion is provided. By acquiring text information, wherein the text information includes description information of the target image to be generated; determining a target style hint vector that matches the text information, wherein the target style hint vector is used to instruct the target student model to output an image of the target image style, the target student model is a student model obtained by knowledge distillation based on a plurality of teacher models of different image styles under a distributed training framework, the distributed training framework deploys teacher models of different image styles on a plurality of image processing units respectively, and the target student model is trained in parallel on a plurality of image processing units while sharing parameters; the text information and the target style hint vector are input into the target student model, the target student model executes the diffusion generation process, and outputs the target image, thus solving the technical problem of poor fusion effect when using the existing model fusion method to fuse models of different styles in the related technology. In the present application, a method for generating a target image based on a target student model is adopted. Text information is input into the target student model, and a target style prompt vector matching the text information is determined. The target style vector can instruct the target student model to output an image in the style of the target image. The target student model outputs the target image based on the text information and the target style prompt vector, thereby achieving the purpose of accurately generating images of different styles based on the target student model, thereby realizing the technical effect of improving the accuracy of generating images of different styles.

[0184] It can be understood by those skilled in the art that Fig.13 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.13The structure of the electronic device is not limited. Fig.13 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Fig.13 Different configurations are shown.

[0185] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0186] Example 5

[0187] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the diffusion model fusion method provided in the first embodiment.

[0188] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0189] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the steps of the diffusion model fusion method.

[0190] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the program steps of the diffusion model fusion method.

[0191] Example 6

[0192] According to another aspect of an embodiment of the present application, a computer program product is also provided, which is suitable for executing the steps of the diffusion model fusion method when executed on a data processing device.

[0193] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0194] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0195] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0196] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0197] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0198] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program code.

[0199] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A diffusion model fusion method, characterized in that: include: Acquire text information, wherein the text information includes description information of the target image to be generated; Determine a target style hint vector that matches the text information, wherein the target style hint vector is used to instruct a target student model to output an image of a target image style, wherein the target student model is a student model obtained by performing knowledge distillation based on a plurality of teacher models of different image styles in a distributed training framework, wherein the distributed training framework deploys teacher models of different image styles on a plurality of image processing units, respectively, and the target student model is trained in parallel on the plurality of image processing units while sharing parameters; The text information and the target style cue vector are input into the target student model, and the target student model performs a diffusion generation process to output a target image.

2. The method according to claim 1, characterized in that Inputting the text information and the target style cue vector into the target student model, and outputting the target image through the diffusion generation process of the target student model comprises: Inputting the text information into a text embedding layer in the target student model to convert the text information into a text vector; The target image is generated by modulating a generation mechanism of the target student model according to the text vector and the target style hint vector.

3. The method according to claim 1, characterized in that The target student model is obtained by the following steps: Deploying teacher models of different image styles on multiple image processing units of the distributed training framework; Determine a style hint vector corresponding to the teacher model for each image style, wherein the style hint vector is used to instruct the student model to simulate a generation mechanism of the image style of the corresponding teacher model; According to the style cue vector corresponding to the teacher model of each image style, the knowledge of the teacher models of multiple image styles is integrated into the student model of shared parameters through knowledge distillation to obtain the target student model.

4. The method according to claim 3, characterized in that According to the style hint vector corresponding to the teacher model of each image style, the knowledge of the teacher models of multiple image styles is integrated into the student model of shared parameters through knowledge distillation, and the target student model includes: Select sample images from the training sample set; Adding noise to the sample image according to a predetermined noise scheduling strategy to obtain a noise image; The target student model is obtained by using the style cue vector corresponding to the teacher model of each image style and the noise image, and utilizing the target loss function to constrain the knowledge distillation process between the teacher model and the student model.

5. The method according to claim 4, characterized in that When the target loss function is fractional distillation, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained including: Inputting the noise image into the teacher model corresponding to the style hint vector, predicting the noise of the noise image by the teacher model, and obtaining a first score of the noise image; Inputting the noise image and the style cue vector into the student model, predicting the noise of the noise image by the student model, and obtaining a second score of the noise image; Calculating a difference between a first score of the noisy image and a second score of the noisy image to obtain a fractional distillation loss; The parameters of the student model are updated based on the fractional distillation loss until the fractional distillation loss satisfies a first preset condition.

6. The method according to claim 5, characterized in that When the target loss function is feature distillation, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained including: Inputting the noise image into the teacher model corresponding to the style hint vector, extracting the intermediate feature map of the noise image through the teacher model, and obtaining a first intermediate feature map of the noise image; Inputting the noise image into the student model, extracting an intermediate feature map of the noise image through the student model, and obtaining a second intermediate feature map of the noise image; Calculating the similarity between the first intermediate feature map of the noise image and the second intermediate feature map of the noise image to obtain a feature distillation loss; The parameters of the student model are updated based on the feature distillation loss until the feature distillation loss satisfies a second preset condition.

7. The method according to claim 6, characterized in that When the target loss function is a multi-category adversarial loss, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained including: Input the noise image into the teacher model corresponding to the style hint vector, and output a real image; Input the noise image and the style cue vector into the student model and output a predicted image; Inputting the real image and the predicted image into a discriminator to obtain a prediction result; Based on the prediction results, calculating a multi-class adversarial loss; The parameters of the student model are updated based on the multi-class adversarial loss until the multi-class adversarial loss satisfies a third preset condition.

8. The method according to claim 7, characterized in that When the target loss function is score distillation, feature distillation, and multi-category adversarial loss, the target loss function is used to constrain the knowledge distillation process between the teacher model and the student model, and the target student model is obtained including: Performing a weighted summation on the loss values ​​of the score distillation loss, the feature distillation loss, and the multi-category adversarial loss to obtain a total training loss; Based on the total training loss, updating parameters of the student model; The process of inputting the noise image and the style hint vector, calculating the total training loss, and updating the parameters is repeatedly performed until a preset training stop condition is reached to obtain the target student model.

9. A diffusion model fusion method, characterized in that: include: Acquire text information uploaded by the client, wherein the text information includes description information of the target image to be generated; In a cloud server, a target style hint vector matching the text information is determined, wherein the target style hint vector is used to instruct a target student model to output an image of a target image style, and the target student model is a student model obtained by performing knowledge distillation based on a plurality of teacher models of different image styles in a distributed training framework, wherein the distributed training framework deploys teacher models of different image styles on a plurality of image processing units, respectively, and the target student model is trained in parallel on the plurality of image processing units while sharing parameters; Inputting the text information and the target style cue vector into the target student model, and having the target student model perform a diffusion generation process to output a target image; The target image is returned to the client.

10. A diffusion model fusion device, characterized in that: include: An acquisition unit, used for acquiring text information, wherein the text information includes description information of a target image to be generated; A first confirmation unit is used to determine a target style hint vector that matches the text information, wherein the target style hint vector is used to instruct a target student model to output an image of a target image style, and the target student model is a student model obtained by performing knowledge distillation based on a plurality of teacher models with different image styles in a distributed training framework, wherein the distributed training framework deploys teacher models with different image styles on a plurality of image processing units, respectively, and the target student model is trained in parallel on the plurality of image processing units while sharing parameters; The output unit is used to input the text information and the target style hint vector into the target student model, and the target student model performs a diffusion generation process to output a target image.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is run, the device where the computer-readable storage medium is located is controlled to execute the diffusion model fusion method according to any one of claims 1 to 9.

12. An electronic device, characterized in that: include: A memory storing an executable program; A processor is used to run the program, wherein the program executes the diffusion model fusion method described in any one of claims 1 to 9 when running.

13. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the diffusion model fusion method described in any one of claims 1 to 9 are implemented.

Images