Model training method and device based on knowledge distillation, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a knowledge distillation-based model training method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining a first sample data set; the first sample data set comprises a plurality of first labeled texts, and a first sample image corresponding to each first labeled text has different visual effect types and comprises a plurality of objects; calling a pre-trained teacher model, and generating a first generation image based on the first annotation text; generating a second generation image based on the first generation image and the first annotation text through a to-be-trained student model; determining knowledge distillation loss of the student model based on the first generated image and the second generated image; and performing model parameter updating on the student model based on the knowledge distillation loss to obtain a trained student model. According to the method and the device, the images which have different visual effects and contain a plurality of objects can be generated through the trained student model, and the generation quality of the images can be improved.

Description

Technical Field

[0001] The present application relates to the field of image generation technology, and in particular to a model training method, device, electronic device, computer-readable storage medium, and computer program product based on knowledge distillation. Background Art

[0002] The development of deep learning has laid the foundation for text-to-image technology. Deep learning models can effectively process text and image information, mapping text descriptions to image content. People are also keen to incorporate their favorite concepts or characters into generated works by fine-tuning the models. However, related technologies typically only generate one character and one type of visual effect at a time. For scenarios requiring multiple characters and a variety of different visual effects, not only is model training difficult, but the generated images also struggle to achieve the desired effect, resulting in poor image quality. Summary of the Invention

[0003] The embodiments of the present application provide a model training method, device, electronic device, computer-readable storage medium and computer program product based on knowledge distillation. Through this application, not only can images with different visual effects and containing multiple objects be generated through the trained student model, but the image generation quality can also be improved.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] An embodiment of the present application provides a model training method based on knowledge distillation, the method comprising: obtaining a first sample data set; the first sample data set comprising a plurality of first annotated texts, a first sample image corresponding to each first annotated text having a different visual effect type and containing a plurality of objects; calling a pre-trained teacher model to generate a first generated image based on the first annotated text; generating a second generated image based on the first generated image and the first annotated text through a student model to be trained; determining the knowledge distillation loss of the student model based on the first generated image and the second generated image; and updating the model parameters of the student model based on the knowledge distillation loss to obtain a trained student model.

[0006] An embodiment of the present application provides a model training device based on knowledge distillation, including: an acquisition module for acquiring a first sample data set; the first sample data set includes multiple first annotated texts, and the first sample image corresponding to each first annotated text has a different visual effect type and contains multiple objects; a first generation module for calling a pre-trained teacher model to generate a first generated image based on the first annotated text; a second generation module for generating a second generated image based on the first generated image and the first annotated text through a student model to be trained; a determination module for determining the knowledge distillation loss of the student model based on the first generated image and the second generated image; and an update module for updating the model parameters of the student model based on the knowledge distillation loss to obtain a trained student model.

[0007] In the above scheme, the first sample data set is obtained from a pre-constructed sample database; the device also includes a storage module, which is used to obtain multiple second sample images with different visual effect types; each second sample image includes at least one object; the multiple second sample images are respectively subjected to image labeling processing to obtain a second annotated text for each second sample image; a visual effect type label is generated for each second sample image; each second sample image, the visual effect type label of the second sample image, and the second annotated text of the second sample image are mapped and stored in the sample database.

[0008] In the above scheme, the update module is also used to determine the module to be trained in the student model; fix the parameters of the remaining modules in the student model except the module to be trained; and update the parameters of the module to be trained based on the knowledge distillation loss to obtain the trained student model.

[0009] In the above scheme, the determination module is also used to obtain multiple first pixel values ​​of the first generated image and multiple second pixel values ​​of the second generated image; determine the pixel-level loss of the student model based on the first pixel values ​​and the second pixel values; obtain multiple first image features of the first generated image and multiple second image features of the second generated image; determine the feature-level loss of the student model based on the first image features and the second image features; and determine the knowledge distillation loss based on the pixel-level loss and the feature-level loss.

[0010] In the above scheme, the determination module is also used to determine a first weight corresponding to the pixel-level loss and a second weight corresponding to the feature-level loss; the pixel-level loss is weighted using the first weight to obtain a first weighted loss; the feature-level loss is weighted using the second weight to obtain a second weighted loss; and the sum of the first weighted loss and the second weighted loss is determined as the knowledge distillation loss.

[0011] In the above scheme, the device also includes a model training module, which is used to: obtain a second sample data set from the sample database; the second sample data set includes multiple second sample images and second annotated text for each second sample image, and the multiple second sample images have different visual effect types and contain multiple objects; generate a third generated image based on the second annotated text through the teacher model to be trained; determine the loss value of the teacher model based on the second sample image and the third generated image; based on the loss value, update the model parameters of the teacher model to obtain the pre-trained teacher model.

[0012] In the above scheme, the device also includes an evaluation module, which is used to: obtain a third annotated text; perform image generation processing based on the third annotated text through the trained student model to obtain a fourth generated image; perform effect evaluation on the fourth generated image based on the third annotated text to obtain an effect evaluation value; when the effect evaluation value is greater than or equal to a preset evaluation threshold, obtain the model parameters of the trained student model; add the model parameters to a preset model file; and call the trained student model for image generation by loading the model parameters in the model file.

[0013] In the above scheme, the evaluation module is also used to obtain a third sample data set from the sample database when the effect evaluation value is less than a preset evaluation threshold; the third sample data set includes multiple fourth annotated texts, and the third sample image corresponding to each fourth annotated text has a different visual effect type and contains multiple objects; calling the pre-trained teacher model to generate a fifth generated image based on the fourth annotated text; generating a sixth generated image based on the fifth generated image and the fourth annotated text through the trained student model; determining the knowledge distillation loss of the trained student model based on the fifth generated image and the sixth generated image; adjusting the model parameters of the trained student model based on the knowledge distillation loss of the trained student model to obtain an image generation model.

[0014] An embodiment of the present application provides an electronic device, which includes: a memory for storing computer-executable instructions or computer programs; and a processor for executing the computer-executable instructions or computer programs stored in the memory to implement the knowledge distillation-based model training method provided in the embodiment of the present application.

[0015] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which is used to implement the knowledge distillation-based model training method provided in an embodiment of the present application when executed by a processor.

[0016] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the model training method based on knowledge distillation provided in the embodiment of the present application is implemented.

[0017] The embodiments of the present application have the following beneficial effects:

[0018] First, a first sample dataset containing multiple first annotated texts is obtained. Each first annotated text in the first sample dataset corresponds to a first sample image with different visual effects and multiple objects. This first sample dataset provides rich training data for the student model, enabling the student model to learn how to generate images with different visual effects and multiple objects. Next, a pre-trained teacher model is invoked to generate a first generated image based on the first annotated texts. Next, a student model to be trained generates a second generated image based on the first generated image and the first annotated texts. The teacher model now possesses high image generation capabilities. The first generated image generated by the teacher model serves as a reference image during student model training. By learning from the teacher model's generation results, the student model gradually improves its own image generation capabilities. Finally, based on the student model's knowledge distillation loss determined by the first and second generated images, the student model's model parameters are updated to obtain a trained student model. The knowledge distillation loss reflects the difference between the second generated image generated by the student model and the first generated image generated by the teacher model. By minimizing this knowledge distillation loss, the student model can better learn the knowledge and skills of the teacher model. Through the above training process, the trained student model can not only generate images with different types of visual effects and containing multiple objects, but also improve the generation ability and image generation quality of the student model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is an optional flowchart of the model training method based on knowledge distillation provided in an embodiment of the present application;

[0020] Figure 2 This is another optional flowchart of the model training method based on knowledge distillation provided in an embodiment of the present application;

[0021] Figure 3 This is a flow chart of building a sample database according to an embodiment of the present application;

[0022] Figure 4 This is an optional training flow diagram of the teacher model provided in the embodiment of the present application;

[0023] Figure 5 This is a schematic diagram of an implementation process for determining knowledge distillation loss provided in an embodiment of the present application;

[0024] Figure 6 Schematic diagram of the evaluation process for the trained student model provided in an embodiment of the present application;

[0025] Figure 7 Schematic diagram of the structure of the offline knowledge distillation method provided in the embodiment of the present application;

[0026] Figure 8 This is another optional training flow diagram of the teacher model provided in the embodiment of the present application;

[0027] Figure 9 Schematic diagram of the test flow of the teacher model provided in the embodiment of the present application;

[0028] Figure 10 Schematic diagram of the knowledge distillation process between the teacher model and the student model provided in the embodiment of the present application;

[0029] Figure 11 This is a structural block diagram of a model training device based on knowledge distillation provided in an embodiment of the present application;

[0030] Figure 12 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0032] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0034] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0035] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0036] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0037] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0038] 1) In response to: used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed can be in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0039] 2) Human-computer interaction interface, which is used to provide an interface for human-computer interaction functions / an interface for displaying image information.

[0040] For example, graphical user interface (GUI) display, such as augmented reality (AR) interface, virtual reality (VR) interface, voice user interface (VUI), interactive projection interface (using projection technology to display information on a plane), eye movement detection interface (interface controlled by detecting the user's line of sight), holographic interface (three-dimensional hologram formed by projecting images through holographic projection technology, so that stereoscopic images can be seen without wearing special glasses), multimodal interface (interface that combines multiple interaction methods, such as touch, vision, hearing, etc.), brain-machine interface (BMI) interface, etc.

[0041] 3) Low-Rank Adaptation (LoRA): This is a model fine-tuning method specifically designed for large language models. By introducing two small, decomposable matrices with far fewer parameters than the original model, it constructs an efficient bypass structure. Fine-tuning this bypass effectively adjusts the weights of the pre-trained model without affecting its overall structure. Requiring only minimal computing resources, LoRA enables rapid fine-tuning, and the performance of the model fine-tuned with LoRA is comparable to that of a full-model fine-tuning, achieving efficient model optimization.

[0042] 4) Latent Space: It is an abstract multidimensional space formed by the model by compressing data features.

[0043] 5) Variational Autoencoder (VAE): A generative AI method that combines deep learning and probabilistic graphical models. It is primarily used for data generation, feature learning, and latent variable inference. The core idea of ​​VAE is to map data into a latent space through variational inference and generate new data from the latent space. For example, it can generate images (such as faces and handwritten digits), text, and music.

[0044] 6) Variational Autoencoder Encoder (VAE-Encoder): Responsible for mapping the input data to the probability distribution parameters (e.g., mean and variance) of the latent space.

[0045] 7) Prompt: It is a text instruction or question input by the user to the model to guide the model to generate a specific type of response or content.

[0046] In order to better understand the model training method based on knowledge distillation provided in the embodiment of the present application, the model training method based on knowledge distillation in the related art is first described below.

[0047] In related technologies, most image generation models are trained for one character and one type of visual effect, and can usually only generate one character and one type of visual effect at a time. If the scenario requires generating multiple characters and multiple different types of visual effects, not only is the model training difficult, but the generated images are also difficult to achieve people's expected effects. That is, a single model is difficult to generate images with different types of visual effects and containing multiple objects, and the image generation quality is also poor.

[0048] To address the problems existing in the related art, embodiments of the present application provide a model training method based on knowledge distillation. This method uses a well-established model as a teacher model, extracts knowledge from annotated text through the teacher model, and then condenses the extracted knowledge into a student model. Several training sets with different visual effect types and multiple objects can be used to train a student model for generating images with different visual effect types and multiple objects. During model training based on knowledge distillation, a first sample dataset containing multiple first annotated texts is first obtained. Each first sample image corresponding to each first annotated text in the first sample dataset has a different visual effect type and contains multiple objects. This first sample dataset provides rich training data for the student model, enabling the student model to learn how to generate images with different visual effect types and multiple objects. Next, a pre-trained teacher model is invoked to generate a first generated image based on the first annotated text. Next, a second generated image is generated based on the first generated image and the first annotated text using the student model to be trained. The teacher model now possesses a high level of image generation capability. The first generated image generated by the teacher model is used as a reference image during student model training. By learning from the teacher model's generation results, the student model can gradually improve its own image generation capability. Finally, based on the student model's knowledge distillation loss, determined by the first and second generated images, the model parameters of the student model are updated to produce a trained student model. In this way, the knowledge distillation loss reflects the difference between the second generated image generated by the student model and the first generated image generated by the teacher model. By minimizing this knowledge distillation loss, the student model can better learn the knowledge and skills of the teacher model. Through this training process, the trained student model can not only generate images with different visual effects and multiple objects, but also improve the student model's generation capabilities and image quality.

[0049] The model training method based on knowledge distillation provided in the embodiment of the present application can be applied to electronic devices such as laptops, tablet computers, and desktop computers. The embodiment of the present application does not impose any restrictions on the specific type of electronic devices.

[0050] The following describes in detail the model training method based on knowledge distillation provided in the embodiment of the present application in conjunction with the accompanying drawings.

[0051] Figure 1 This is an optional flow chart of the model training method based on knowledge distillation provided in the embodiment of the present application. The method can be applied to an electronic device, which can be a server or a terminal. That is, the model training method based on knowledge distillation in each embodiment of the present application can be executed by a server, or by a terminal, or by interaction between a server and a terminal. The following will be explained exemplarily using the electronic device as a server. Figure 1 As shown, the method includes the following steps S101 to S105:

[0052] Step S101: Acquire a first sample data set.

[0053] In an embodiment of the present application, the first sample data set includes multiple first annotation texts, each of which corresponds to a first sample image having a different visual effect type and containing multiple objects. That is, each first annotation text in the first sample data set corresponds to a first sample image, each of which has a different visual effect type and contains multiple objects. The first annotation text is text describing the first sample image, including information such as the description, location, color, and shape of the objects in the first sample image. The first annotation text can be generated manually or through natural language processing technology.

[0054] The visual effect type can be either the image style or the image category. Style refers to the unique style and characteristics presented by the image, including brushwork and composition, color and tone, expressive techniques, and overall visual experience. Brushwork and composition refer to the style reflected in the brushstrokes, line treatment, color application, and overall composition. Color and tone refer to color matching, light and dark contrast, and the warm and cold tones. Expressive techniques refer to different interpretations and expressions of the same theme, such as realism, cartoon, and abstraction. Overall visual experience refers to the spatial sense, layering, and lighting of the image. Category refers to the result of classifying an image based on its content, features, or purpose. For example, the image category can be natural scene images, animal images, human images, and architectural images. Each first annotated text in the first sample dataset can correspond to one or more objects. The object can be any entity with clear boundaries, shape, features, or semantic meaning, or an abstract element, etc. This is not limited in this embodiment of the present application. For example, the object can be a person, animal, or plant in the image.

[0055] In some embodiments, the first annotated text in the first sample dataset can be obtained based on the determination of the first sample image. Methods for obtaining the first sample image may include: using public datasets, web crawlers, and data synthesis. For example, first sample images with different visual effect types and containing multiple objects can be downloaded from the dataset's official website or related platform. Based on the determination of the first sample image, the first sample image is annotated in text form to obtain the first annotated text in the first sample dataset. In this way, the first annotated text in the first sample dataset also has different visual effect types and contains multiple objects.

[0056] Step S102: calling a pre-trained teacher model to generate a first generated image based on the first annotated text.

[0057] In an embodiment of the present application, a pre-trained teacher model refers to a model that has been trained with a large amount of data and can generate a corresponding image based on an input text description. The pre-trained teacher model generally uses deep learning technology, such as Stable Diffusion XL (SDXL), Stable Diffusion 1.5 (SD1.5), Midjerney or variational autoencoders. The first generated image refers to an image generated by the pre-trained teacher model based on the first annotated text. The first generated image is the teacher model's understanding and expression of the input first annotated text. The first generated image may contain multiple objects and have different types of visual effects, such as color, texture, light and shadow.

[0058] In some embodiments, the process of generating the first generated image may be: loading a pre-trained teacher model into memory, ensuring that the pre-trained teacher model has been fully trained and can generate high-quality images based on the input annotated text. Inputting the first annotated text into the teacher model, the input process generally involves converting the first annotated text into a numerical representation that the teacher model can understand, such as word embedding or sequence encoding. The teacher model generates a first generated image based on the input first annotated text. Through calculation and reasoning within the teacher model, a first generated image that is consistent with the description of the input first annotated text is ultimately output.

[0059] For example, suppose there is a first annotated text as "A brown dog is running on the green grass." This first annotated text can be input into the pre-trained teacher model. The teacher model will generate a first generated image containing a brown dog and green grass based on the text description. This first generated image is the teacher model's understanding and expression of the first annotated text and can be used for subsequent student model training.

[0060] Step S103 : generating a second generated image based on the first generated image and the first annotated text by using the student model to be trained.

[0061] In the embodiment of the present application, the student model to be trained is a model that has not been fully trained. The goal of the student model to be trained is to learn how to generate an image based on the input annotated text. For example, the student model can be a Flux model or a Flux LoRA model. The second generated image is an image generated by the student model based on the first generated image and the first annotated text. The second generated image is the student model's understanding and expression of the input text (i.e., the first annotated text) and the reference image (i.e., the first generated image). The second generated image can contain multiple objects and have different types of visual effects.

[0062] In some embodiments, the process of generating the second generated image may be: loading the student model to be trained into the memory, ensuring that the student model to be trained has been initialized and can start training. The first generated image and the first annotated text generated by the teacher model are input into the student model. The input process generally involves converting the first generated image and the first annotated text into numerical representations that the student model can understand, such as image feature vectors and text embeddings. The student model generates the second generated image based on the input first generated image and the first annotated text. Through calculation and reasoning within the student model, it finally outputs an image that is consistent with the input text description and the reference image, i.e., the second generated image.

[0063] For example, consider a first annotated text: "A brown dog is running on green grass," and a first generated image generated by a teacher model. The first annotated text and the first generated image can be fed into a student model to be trained. Based on the input text and the reference image, the student model generates an image of a brown dog and green grass. The second generated image represents the student model's understanding and representation of the input text and the reference image, and can be used to evaluate the student model's performance and update model parameters.

[0064] Step S104: determining the knowledge distillation loss of the student model based on the first generated image and the second generated image.

[0065] In this embodiment, the knowledge distillation loss refers to the difference between the second generated image generated by the student model and the first generated image generated by the teacher model. The knowledge distillation loss is used to measure the student model's effectiveness in learning the teacher model's knowledge and is an important indicator in the student model training process. The first generated image and the second generated image are generated by the teacher model and the student model, respectively, based on the same first annotated text.

[0066] In some embodiments, the knowledge distillation loss is determined as follows: a suitable loss function is selected to measure the difference between the first generated image and the second generated image. For example, the loss function may be mean squared error (MSE). The selected loss function is used to calculate the difference between the first generated image and the second generated image to obtain the knowledge distillation loss of the student model.

[0067] Step S105: Based on the knowledge distillation loss, the model parameters of the student model are updated to obtain a trained student model.

[0068] In the embodiment of the present application, model parameter updating refers to adjusting the model parameters of the student model according to the calculated knowledge distillation loss to improve the performance of the student model. Model parameter updating can be to update some parameters of the student model, and the updated parameters can be selected or set according to actual conditions to reduce training overhead. For example, only the parameters of the attention layer in the student model can be updated. The trained student model refers to the model after parameter update. The trained student model can better generate high-quality images with different visual effect types and containing multiple objects based on the input annotated text through the knowledge distillation of the teacher model.

[0069] In some embodiments, the process of updating the model parameters may be: selecting a suitable optimization algorithm to update the parameters of the student model. Common optimization algorithms include Stochastic Gradient Descent (SGD), Adam, or Adagrad. Based on the knowledge distillation loss, the gradient of the student model parameters is calculated. The gradient represents the rate of change of the loss function with respect to the model parameters and is used to guide the update direction of the student model parameters. The parameters of the student model are updated using the optimization algorithm based on the calculated gradient. This process usually involves multiplying the gradient with the learning rate and then subtracting this value from the current parameters to reduce the knowledge distillation loss and improve the performance of the student model. The above process is repeated until the performance of the student model reaches the expected level or meets the preset iteration conditions. This process requires multiple iterations, each of which involves forward propagation, calculating loss, calculating gradients, and updating parameters. The preset iteration conditions may be the loss threshold of the knowledge distillation loss, the maximum number of iterations threshold, and the cutoff iteration time. A loss threshold is set in advance. When the knowledge distillation loss is less than the pre-set loss threshold, the iterative training is stopped, that is, the model parameter update is stopped. A maximum iteration number threshold can also be set in advance. When the number of iterations exceeds the maximum iteration number threshold, the model parameter update is stopped. A cutoff iteration time can also be set in advance. When the iteration time reaches the cutoff iteration time, the model parameter update is stopped, and finally the trained student model is obtained.

[0070] In one embodiment of the present application, a first sample dataset containing multiple first annotated texts is first obtained. Each first annotated text in the first sample dataset corresponds to a first sample image with different visual effect types and contains multiple objects. This first sample dataset provides rich training data for the student model, enabling the student model to learn how to generate images with different visual effect types and multiple objects. A pre-trained teacher model is then invoked to generate a first generated image based on the first annotated texts. A second generated image is then generated based on the first generated image and the first annotated texts using the student model to be trained. The teacher model now possesses high image generation capabilities. The first generated image generated by the teacher model serves as a reference image during student model training. By learning from the teacher model's generation results, the student model can gradually improve its own image generation capabilities. Finally, based on the student model's knowledge distillation loss determined by the first and second generated images, the model parameters of the student model are updated to obtain a trained student model. The knowledge distillation loss reflects the difference between the second generated image generated by the student model and the first generated image generated by the teacher model. By minimizing this knowledge distillation loss, the student model can better learn the knowledge and skills of the teacher model. Through the above process, the student model is not only able to generate images with different types of visual effects and containing multiple objects, but also improves the generation ability and image generation quality of the student model.

[0071] The following is an example of an application scenario of the model training method based on knowledge distillation provided in the embodiment of the present application.

[0072] Artists and designers can use the student model (e.g., Flux model) trained using the embodiments of this application to generate creative images, illustrations, and concept art. For example, by inputting a specific text description, a futuristic science fiction scene or a retro-style illustration can be generated. The student model can also be fine-tuned through knowledge distillation by combining sample data sets of various styles, categories, and characters to generate images that conform to a variety of styles, categories, and characters.

[0073] Game developers can use the student model trained by the embodiments of this application to quickly create game assets such as characters, environments, and props, accelerating prototyping and content iteration. They can also train the student model to generate high-quality game assets by collecting a large amount of image data in game styles.

[0074] In film and animation production, the student model trained using the embodiments of this application can be used to generate backgrounds, special effects, or character designs, helping artists and directors realize their visual creativity. The student model can also be trained to generate images that match a variety of styles, categories, and characters by combining sample data from film and television styles.

[0075] The following will describe the model training method based on knowledge distillation in the embodiment of the present application in combination with the above scenario.

[0076] Figure 2 This is another optional flow chart of the model training method based on knowledge distillation provided in the embodiment of the present application, such as Figure 2 As shown, the method includes the following steps S201 to S216:

[0077] Step S201: The terminal receives a knowledge distillation-based model training operation input by a user.

[0078] In an embodiment of the present application, the model training operation based on knowledge distillation includes an input operation, which is used to input a first sample data set, such as a user performing an operation on a terminal (such as a computer, mobile phone, etc.) through an input device such as a keyboard, mouse, and touch screen, such as pressing a button, moving the mouse, or touching the screen.

[0079] Step S202: The terminal generates a knowledge distillation-based model training request in response to the knowledge distillation-based model training operation.

[0080] In an embodiment of the present application, a model training request based on knowledge distillation is used to request a server to perform model training, and the first sample data set can be encapsulated into the model training request based on knowledge distillation.

[0081] In step S203, the terminal sends a knowledge distillation-based model training request to the server.

[0082] In an embodiment of the present application, the terminal sends a model training request based on knowledge distillation to the server to request the server to perform model training.

[0083] Step S204: The server obtains a first sample data set in response to the knowledge distillation-based model training request.

[0084] In this embodiment of the present application, the first sample dataset includes multiple first annotated texts, each of which corresponds to a first sample image having a different visual effect type and containing multiple objects. The server parses the knowledge distillation-based model training request sent by the terminal to obtain the first sample dataset, and then performs subsequent processing operations based on the first sample dataset.

[0085] In some embodiments, the first sample data set may also be obtained from a pre-built sample database; see Figure 3 , Figure 3 It shows that a sample database can be constructed by following steps S301 to S304:

[0086] Step S301: Acquire a plurality of second sample images with different visual effect types.

[0087] In this embodiment of the present application, each second sample image includes at least one object. Methods for obtaining the second sample images may include using a public dataset, web crawling, and data synthesis. For example, multiple second sample images with different visual effect types can be downloaded from the dataset's official website or related platform.

[0088] Step S302 : performing image labeling processing on each of the plurality of second sample images to obtain a second labeled text for each second sample image.

[0089] In an embodiment of the present application, image labeling processing may refer to the process of annotating objects or background areas in a second sample image. The image labeling processing method may include using a target detection model or a scene understanding model to detect or understand the second sample image for labeling, or may use a large language model to label the image by inputting specified prompt words. The embodiment of the present application does not limit this.

[0090] The second annotation text may be the annotation text for the second sample image under multiple representation dimensions. Representation dimensions are a set of features used to describe different aspects or attributes of an object. Each representation dimension represents information about a specific aspect of the object, and together, the multiple representation dimensions constitute a complete representation of the object. For example, if the object is a person, the multiple representation dimensions may include category description, style description, appearance description, action description, and background description. If the object is an animal, the multiple representation dimensions may include physiological characteristics, behavioral characteristics, environmental description, and health status. If the object is an object, the multiple representation dimensions may include geometric characteristics, material characteristics, and functional status. The annotation text may be the text corresponding to each representation dimension. For example, if the object is a girl, the category description annotation text may include person, the style description may include realistic, and the appearance description annotation text may include: female, short black hair, big eyes, wearing glasses, shirt top, skirt bottom, and sneakers. The action description annotation text may include: standing, reaching out, and opening mouth. The background description annotation text may include: park, tree, trash can, rainy day, and daytime.

[0091] In some embodiments, image labeling is performed on multiple second sample images respectively to obtain second annotated text for each second sample image, which can be achieved by the following steps: image labeling is performed on multiple second sample images respectively to obtain annotated text for each second sample image; deduplication of the annotated text of the second sample image is performed to obtain deduplication annotated text; text recognition is performed on the deduplication annotated text to obtain the target dimension to which the deduplication annotated text belongs; according to the target dimension to which the deduplication annotated text belongs, the deduplication annotated text is divided into any one of multiple representation dimensions to obtain the second annotated text for each second sample image.

[0092] In an embodiment of the present application, duplicate annotations are removed from the annotation text of the second sample image to obtain duplicate-free annotation text. For example, the annotation text of the second sample image can be hashed, and annotations with the same hash value can be deleted, retaining one annotation text for each hash value. Alternatively, the annotation text can be vectorized and similarity calculated, and annotation texts of multiple second sample images with similarities greater than a similarity threshold can be deleted, retaining one annotation text. Alternatively, incorrect annotation texts can be deleted to obtain duplicate-free annotation text. For example, the second sample image can be labeled multiple times, and annotation texts that appear less than a threshold number of times can be identified as incorrect annotation texts and deleted. For example, the annotation text of the second sample image is: "Girl, reading book in bed, blue eyes, black hair, blue top, red skirt, yellow top, night, purple pillow, stuffed bunny, bunny." The annotation texts for "blue top" and "yellow top" are the same, but the colors are different. Therefore, one of the annotation texts is incorrect. The image labeling process can be repeated. Since "blue top" only appears once, it can be confirmed that "blue top" is the incorrect annotation text and the incorrect annotation text "blue top" can be deleted. By calculating text similarity, we can confirm that "plush bunny" and "rabbit" are duplicates. We can then delete the annotation "rabbit," leaving the more refined annotation "plush bunny." The resulting deduplicated annotation is: "girl, reading in bed, black hair, red skirt, yellow top, night, purple pillow, plush bunny."

[0093] Next, text recognition is performed on each of the deduplicated annotation texts in turn to obtain the target dimension to which each deduplicated annotation text belongs. For example, the deduplicated annotation text "Reading books in bed" is subjected to text recognition, and the target dimension is obtained as the character behavior dimension, "blue eyes" is the character appearance dimension, "plush bunny" is the background dimension, and so on. If the target dimension is the same as any of the multiple representation dimensions, the deduplicated annotation text can be divided into the representation dimension that is the same as the target dimension among the multiple representation dimensions based on the target dimension to which the deduplicated annotation text belongs. If the target dimension is a sub-dimension of any of the multiple representation dimensions, the deduplicated annotation text can be divided into the representation dimension to which the target dimension belongs among the multiple representation dimensions based on the target dimension to which the deduplicated annotation text belongs. If the target dimension belongs to multiple representation dimensions, the target dimension can be randomly assigned to any of the multiple representation dimensions. After all the deduplicated annotation texts are divided, the second annotation text of each second sample image under multiple representation dimensions can be obtained. For example, the multiple representation dimensions are the character behavior dimension, the character appearance dimension, and the background dimension. The target dimension is the same as any one of the multiple representation dimensions. For example, "reading a book in bed" is a character behavior dimension, "blue eyes" is a character appearance dimension, and "stuffed bunny" is a background dimension. The deduplicated annotated text "girl, reading a book in bed, black hair, red skirt, yellow top, night, purple pillow, stuffed bunny" can be divided according to the target dimension to obtain the annotated text "girl, black hair, red skirt, yellow top" for the character appearance dimension, "reading a book in bed" for the character behavior dimension, and "night, purple pillow, stuffed bunny" for the background dimension. For example, the multiple representation dimensions are the character behavior dimension, the character appearance dimension, and the background dimension. The target dimension is a sub-dimension of any one of the multiple representation dimensions. For example, the target dimension of the character appearance dimension can include the facial dimension, the clothing dimension, and so on. "Black hair" is a facial dimension and a sub-dimension of the character appearance dimension. Therefore, "black hair" can be divided into the character appearance dimension. The division process of other annotated texts can refer to the above description and is not listed here. Finally, the second annotated text of the second sample image in multiple representation dimensions is obtained, namely, the annotated text of the character appearance dimension "girl, black hair, red skirt, yellow top", the annotated text of the character behavior dimension "reading a book in bed", and the annotated text of the background dimension "night, purple pillow, plush rabbit".

[0094] Through the above steps, the redundancy and errors in the annotated text can be eliminated through the deduplication and error correction mechanism to ensure the accuracy of the annotated text. Then, the annotated text is layered to achieve feature decoupling, enabling the image generation model to independently learn features at different semantic levels and enhance the fine-grained learning ability of the image generation model.

[0095] Step S303: Generate a visual effect type label for each second sample image.

[0096] In an embodiment of the present application, the visual effect type label is a label that classifies and describes the visual features of the second sample image, and the visual effect type label corresponds one-to-one to the second sample image. The visual effect type label may include a style label and an image category label of the second sample image. The style label describes the painting style of the second sample image, such as labels such as realistic, abstract, or cartoon; the image category label describes the subject content of the second sample image, such as labels such as landscape, person, and animal. For example, for a landscape painting, its style label can be marked as realistic and its image category label can be marked as landscape; for a cartoon character painting, its style label can be marked as cartoon and its image category label can be marked as person.

[0097] In some embodiments, the visual effect type label of each second sample image can be generated in the following ways: manual labeling can be used, in which a professional or a labeling team observes and analyzes the image, and manually labels the style and image category label according to the visual features of the second sample image. Automatic labeling can also be used, using computer vision technology, such as convolutional neural networks or image classification algorithms, to automatically label the second sample image. Semi-automatic labeling can also be used, combining manual and automatic labeling methods, first using computer vision technology to preliminarily label the second sample image, and then professionals correct and supplement the labeling results. Data mining can also be used to analyze and mine a large amount of labeled image data to discover the correlation between the visual features of the image and the label, thereby generating a new visual effect type label for the second sample image.

[0098] Step S304 : Mapping each second sample image, the visual effect type label of the second sample image, and the second annotated text of the second sample image, and storing the mapping in a sample database.

[0099] In this embodiment of the present application, mapping refers to associating the second sample image, its visual effect type label, and the corresponding second annotation text to form a complete data record. After obtaining the data record corresponding to each second sample image, the data record corresponding to each second sample image is stored in the sample database.

[0100] In some embodiments, first, the collected second sample image, visual effect type label, and second annotation text are combined into a data record, which can be a structured data structure, such as a dictionary, a tuple, or a list, or an object. The mapping relationship between the second sample image, the visual effect type label, and the second annotation text is defined, and the definition process can be implemented by specifying the position of each element in the data record or using a key-value pair. The mapped data record is stored in a sample database, which can be a relational database, such as MySQL or PostgreSQL, or a non-relational database, such as MongoDB or Cassandra. The sample database can be used for the training, verification, and testing of the teacher model and the student model.

[0101] For example, suppose there's a second sample image with a visual effect type label of "realistic scenery" and a corresponding second annotation text of "a realistic landscape painting depicting mountains, rivers, and lakes." These three elements can be combined into a single data record. For example, a dictionary might include: second sample image: "second_sample_image.jpg", visual effect type label: "realistic scenery", and second annotation text: "a realistic landscape painting depicting mountains, rivers, and lakes."

[0102] Through steps S301 to S304, by obtaining multiple second sample images with different visual effect types, and performing image labeling processing and visual effect type label generation on them, a diverse and richly annotated sample database can be constructed. This database contains not only the images themselves, but also the corresponding visual effect type labels and second annotated texts, providing comprehensive data support for the training of teacher models and student models. The diverse visual effect type labels help the model learn the ability to generate images of different styles and themes, while the second annotated text provides the model with a semantic understanding of the image content. This comprehensive data support will enable the generalization ability of the model and the quality of the generated images to be improved in the future, thereby achieving better results in the text-based image task.

[0103] Step S205: The server calls a pre-trained teacher model to generate a first generated image based on the first annotated text.

[0104] In an embodiment of the present application, for calling the pre-trained teacher model and generating a first generated image based on the first annotated text, the specific meaning and specific implementation method involved in this step can be found in the description of step S102 above, and will not be repeated here.

[0105] In some embodiments, see Figure 4 , Figure 4It shows that before the server calls the pre-trained teacher model and generates the first generated image based on the first annotated text, it can also perform the following steps 401 to 404 to obtain the pre-trained teacher model:

[0106] Step 401: Acquire a second sample data set from a sample database.

[0107] In an embodiment of the present application, the second sample data set includes a plurality of second sample images and a second annotated text for each second sample image, wherein the plurality of second sample images have different visual effect types and contain a plurality of objects. First, a data loader or data pipeline can be designed to read data from a sample database. The data loader can randomly or sequentially extract a certain number of second sample images and their corresponding second annotated texts from the sample database to form a batch of data sets, which will be fed into the teacher model to be trained for training.

[0108] Step 402: Generate a third generated image based on the second annotated text using the teacher model to be trained.

[0109] In this embodiment of the present application, the second annotated text is the input text of the teacher model to be trained, and the third generated image is an image generated by the teacher model to be trained based on the second annotated text. The specific meaning and specific implementation method involved in the step of generating the third generated image based on the second annotated text by the teacher model to be trained are similar to those of step S103 above. Please refer to the description of step S103 above and will not be repeated here.

[0110] Step 403: Determine the loss value of the teacher model based on the second sample image and the third generated image.

[0111] In an embodiment of the present application, the loss value of the teacher model refers to the difference between the third generated image generated by the teacher model and the second sample image. The process of determining the loss value of the teacher model is as follows: select a suitable loss function to measure the difference between the third generated image and the second sample image. For example, the loss function can be mean square error (MSE) or cross-entropy loss (Cross-Entropy Loss), etc. Using the selected loss function, calculate the difference between the third generated image and the second sample image to obtain the loss value of the teacher model.

[0112] Step 404: Based on the loss value, update the model parameters of the teacher model to obtain a pre-trained teacher model.

[0113] In the embodiments of the present application, model parameter updating refers to adjusting the model parameters of the teacher model based on the calculated loss value to improve the performance of the teacher model. The pre-trained teacher model refers to a model that has undergone parameter updating. The pre-trained teacher model can generate high-quality images with different visual effects and multiple objects based on the input annotated text.

[0114] In some embodiments, the process of updating the model parameters may be: selecting a suitable optimization algorithm to update the parameters of the teacher model. Common optimization algorithms include stochastic gradient descent, Adam or Adagrad, etc. According to the loss value, the gradient of the teacher model parameters is calculated. The gradient represents the rate of change of the loss function with respect to the model parameters, which is used to guide the update direction of the teacher model parameters. The parameters of the teacher model are updated according to the calculated gradient using the optimization algorithm. This process usually involves multiplying the gradient with the learning rate, and then subtracting this value from the current parameters to reduce the knowledge loss value and improve the performance of the teacher model. The above process is repeated until the performance of the teacher model reaches the expected level or meets the preset iteration conditions. This process requires multiple iterations, each of which involves forward propagation, calculating loss, calculating gradients and updating parameters. The preset iteration conditions may be a loss threshold of the loss value, a maximum number of iterations threshold, and a cutoff iteration time, etc. A loss threshold is set in advance. When the loss value is less than the pre-set loss threshold, the iterative training is stopped, that is, the model parameter update is stopped; a maximum iteration number threshold can also be set in advance. When the number of iterations exceeds the maximum iteration number threshold, the model parameter update is stopped; a cutoff iteration time can also be set in advance. When the iteration time reaches the cutoff iteration time, the model parameter update is stopped, and finally a pre-trained teacher model is obtained.

[0115] Through steps 401 to 404, by obtaining a diverse second sample dataset from the sample database and using the teacher model to be trained to generate a third generated image based on the second annotated text, the teacher model can be effectively trained to understand and generate images of different visual effects. The loss value calculation based on the second sample image and the third generated image can guide the parameter update of the teacher model, so that it can gradually optimize the quality and accuracy of the generated image. Ultimately, the pre-trained teacher model can not only generate high-quality images, but also provide effective knowledge distillation for the subsequent student model, improving the performance and generalization ability of the student model.

[0116] In step S206 , the server generates a second generated image based on the first generated image and the first annotated text using the student model to be trained.

[0117] In an embodiment of the present application, for generating a second generated image based on the first generated image and the first annotated text through the student model to be trained, the specific meaning and specific implementation method involved in this step can be found in the description of step S103 above, and will not be repeated here.

[0118] In step S207 , the server obtains a plurality of first pixel values ​​of the first generated image and a plurality of second pixel values ​​of the second generated image.

[0119] In the embodiment of the present application, the first pixel value refers to the color value (usually RGB value) of each pixel in the first generated image generated by the teacher model. The second pixel value refers to the color value of the corresponding pixel in the second generated image generated by the student model.

[0120] In some embodiments, an image processing library (eg, OpenCV or PIL, etc.) may be used to read pixel data of the first generated image and the second generated image respectively, and obtain a plurality of first pixel values ​​and a plurality of second pixel values ​​accordingly.

[0121] In step S208 , the server determines the pixel-level loss of the student model based on the first pixel value and the second pixel value.

[0122] In the embodiment of the present application, the pixel-level loss is calculated by comparing the color values ​​of corresponding pixels in the first generated image and the second generated image. For example, the pixel-level loss function can be mean square error or absolute error. For each corresponding pixel in the first generated image and the second generated image, the difference between the first pixel value and the second pixel value is calculated, and then these differences are summarized using the selected pixel-level loss function to obtain the pixel-level loss.

[0123] In step S209 , the server obtains a plurality of first image features of the first generated image and a plurality of second image features of the second generated image.

[0124] In an embodiment of the present application, the first image feature refers to the high-level semantic features and structural features extracted from the first generated image generated by the teacher model through the feature extraction network. The second image feature refers to the high-level semantic features and structural features extracted from the second generated image generated by the student model through the same feature extraction network. Semantic features and structural features can be extracted by a pre-trained convolutional neural network model, such as VGG, ResNet or Inception. The feature extraction network is trained on large-scale image data and can learn the deep semantic information and structural information of the image.

[0125] In step S210 , the server determines a feature-level loss of the student model based on the first image feature and the second image feature.

[0126] In the embodiment of the present application, the feature-level loss is calculated by comparing the high-level semantic features and structural features of the first generated image and the second generated image. For example, the feature-level loss function can be mean squared error or perceptual loss. For each corresponding feature map in the first generated image and the second generated image, the difference between the first image features and the second image features is calculated, and then these differences are summarized using the selected feature-level loss function to obtain the feature-level loss.

[0127] In step S211, the server determines the knowledge distillation loss based on the pixel-level loss and the feature-level loss.

[0128] In this embodiment of the present application, the pixel-level loss and feature-level loss can be combined according to a certain weight ratio to obtain the final knowledge distillation loss. The weight ratio can be adjusted according to the actual situation to balance the impact of pixel-level and feature-level losses on model training.

[0129] In some embodiments, see Figure 5 , Figure 5 It is shown that in step S211, the server determines the knowledge distillation loss based on the pixel-level loss and the feature-level loss, which can be achieved by the following steps S2111 to S2114:

[0130] Step S2111: determine a first weight corresponding to the pixel-level loss and a second weight corresponding to the feature-level loss.

[0131] In this embodiment of the present application, the first weight is a coefficient used to adjust the proportion of pixel-level loss in the knowledge distillation loss, and the second weight is a coefficient used to adjust the proportion of feature-level loss in the knowledge distillation loss. The first and second weights can be determined through experimentation or experience to achieve an appropriate balance between pixel-level loss and feature-level loss in the knowledge distillation loss. For example, the first weight can be set to 0.5 and the second weight can be set to 0.5 to indicate that both are equally important.

[0132] Step S2112: weighting the pixel-level loss using a first weight value to obtain a first weighted loss.

[0133] In the embodiment of the present application, weighted processing refers to multiplying the pixel-level loss by the corresponding first weight to reflect the importance of the pixel-level loss in the training process. The first weighted loss is the loss value obtained by multiplying the pixel-level loss by the first weight.

[0134] Step S2113: Use the second weight value to perform weighted processing on the feature-level loss to obtain a second weighted loss.

[0135] In the embodiment of the present application, weighted processing refers to multiplying the feature-level loss by the corresponding second weight to reflect the importance of the feature-level loss in the training process. The second weighted loss is the loss value obtained by multiplying the feature-level loss by the second weight.

[0136] Step S2114: The sum of the first weighted loss and the second weighted loss is determined as the knowledge distillation loss.

[0137] In the embodiment of the present application, the knowledge distillation loss is the sum of the first weighted loss and the second weighted loss, which comprehensively reflects the performance of the student model at the pixel level and the feature level.

[0138] Through steps S2111 to S2114, by weighting the pixel-level loss and feature-level loss and combining them into a knowledge distillation loss, the difference between the images generated by the student model and the images generated by the teacher model can be more comprehensively measured. This comprehensive loss function helps the student model focus on both the pixel-level details of the image and the high-level semantic and structural information of the image when learning the knowledge of the teacher model, thereby improving the quality and realism of the generated image. At the same time, by adjusting the weights, the importance of pixel-level and feature-level losses in the training process can be flexibly controlled according to the requirements of specific tasks, allowing the student model to better adapt to different application scenarios.

[0139] Step S212: The server determines the modules to be trained in the student model.

[0140] In an embodiment of the present application, the module to be trained refers to the part of the student model that needs to update parameters through knowledge distillation. The module to be trained can be the core module in the student model that is mainly responsible for generating images, such as the attention module in the student model.

[0141] In step S213, the server fixes the parameters of the remaining modules in the student model except the module to be trained.

[0142] In this embodiment of the present application, the remaining modules refer to the parts of the student model that do not require parameter updates through knowledge distillation, for example, other pre-trained parts other than the attention module. The parameter freezing technique in the optimization algorithm can be used to set the parameters of the remaining modules to be non-trainable, that is, they are not updated during the training process of the student model.

[0143] In step S214, the server updates the parameters of the training module based on the knowledge distillation loss to obtain a trained student model.

[0144] In the embodiments of the present application, updating refers to adjusting the model parameters according to the knowledge distillation loss by an optimization algorithm (e.g., Adam or SGD) to reduce the knowledge distillation loss and improve the model performance of the student model. The trained student model refers to a model whose parameters have been updated after knowledge distillation training. The optimization algorithm can be used to calculate the gradient of the parameters of the module to be trained according to the knowledge distillation loss, and the parameters of the module to be trained can be updated according to the gradient.

[0145] By determining the modules to be trained and fixing the parameters of the remaining modules through steps S212 to S214, the student model can be trained in a targeted manner, improving its generative capabilities. This training method helps the student model learn effective knowledge from the teacher model while avoiding unnecessary parameter updates, improving training efficiency and model performance. Ultimately, the trained student model can generate more realistic and expected images, meeting the requirements of the text-based image task.

[0146] In some embodiments, see Figure 6 , Figure 6 It shows that after the server updates the model parameters of the student model and obtains the trained student model, it can also perform the following steps S601 to S611:

[0147] Step S601: Obtain the third annotated text.

[0148] In this embodiment of the present application, the third annotation text is an input text description used to generate an image by the trained student model. The third annotation text can be any descriptive text used to guide the trained student model to generate the corresponding image. The number of third annotation texts can be at least one. The third annotation text can be obtained through user input, random sampling from a pre-built sample database, or from a specific sample data set.

[0149] Step S602 : performing image generation processing based on the third annotated text using the trained student model to obtain a fourth generated image.

[0150] In this embodiment of the present application, the image generation process refers to the process of inputting the third annotated text into the trained student model and converting the text into an image through the student model. The fourth generated image is the output image obtained by the image generation process. Ideally, the fourth generated image matches the description of the third annotated text.

[0151] Step S603: Based on the third annotated text, perform effect evaluation on the fourth generated image to obtain an effect evaluation value.

[0152] In an embodiment of the present application, effect evaluation refers to the process of evaluating whether the generated image accurately reflects the description of the input text, and the quality and diversity of the image. The effect evaluation can evaluate whether the fourth generated image contains the categories and roles involved in the third annotation text, and whether it presents the expected style of painting. The effect evaluation value can be the result of a quantitative evaluation of the fourth generated image by a certain evaluation standard (for example, a subjective evaluation or an automatic evaluation index, etc.). The effect evaluation value is usually a numerical value that can be used to indicate the degree of conformity of the fourth generated image with the description of the third annotation text and the quality of the image. If the effect evaluation value is higher, it means that the quality of the generated image is better and the degree of conformity with the text description is higher; if the effect evaluation value is lower, it means that the quality of the generated image is worse and the degree of conformity with the text description is lower.

[0153] In some embodiments, an automatic evaluation index (such as FID or IS) or a manual evaluation method may be used to perform effect evaluation on the fourth generated image to obtain an effect evaluation value.

[0154] Step S604: When the effect evaluation value is greater than or equal to the preset evaluation threshold, the model parameters of the trained student model are obtained.

[0155] In the embodiment of the present application, the preset evaluation threshold is a pre-set value used to determine whether the effect of the fourth generated image meets the expected generation standard. When the effect evaluation value is greater than or equal to the preset evaluation threshold, the server can obtain the model parameters of the trained student model from the training environment. After the training is completed, the server obtains the model parameters from the training environment through an application programming interface (API) call or a file transfer protocol.

[0156] Step S605: Add the model parameters to the preset model file.

[0157] In the embodiments of the present application, the preset model file is a file that stores model parameters and is typically used to save and load the trained student model. The model file can be a container that can contain structural information and optimizer status of the trained student model. The server can add the model parameters of the trained student model obtained to the preset model file.

[0158] Step S606: Load the model parameters in the model file to call the trained student model to generate the image.

[0159] In an embodiment of the present application, the server can use a deep learning framework to load the model parameters in the model file to call the trained student model for image generation. For example, the server can use Python and PyTorch frameworks to load the model file. After loading, the server can initialize a model instance and set the parameters to the model parameters of the trained student model read from the model file. The server then receives the user's text description input, such as "a kitten playing on the grass", and calls the loaded student model to generate the corresponding image.

[0160] Through steps S601 to S606, the model's generation capabilities can be verified by obtaining the third annotated text and using the trained student model to generate images. The quality and reliability of the model can be ensured by evaluating the generated images and saving the model parameters when the evaluation values ​​meet the requirements. This process helps to continuously optimize and improve the performance of the student model, enabling it to generate images that better meet the expectations and meet the needs of practical applications.

[0161] Step S607: when the effect evaluation value is less than the preset evaluation threshold, a third sample data set is obtained from the sample database.

[0162] In this embodiment of the present application, the third sample dataset includes multiple fourth annotated texts, and each third sample image corresponding to each fourth annotated text has a different visual effect type and includes multiple objects. If the effect evaluation value is less than a preset evaluation threshold, the trained student model is unable to generate images that meet the expectations. In this case, the trained student model needs to be retrained until it can generate high-quality images with different visual effect types and multiple objects.

[0163] Step S608: calling a pre-trained teacher model to generate a fifth generated image based on the fourth annotated text.

[0164] In an embodiment of the present application, for calling the pre-trained teacher model and generating the fifth generated image based on the fourth annotated text, the specific meaning explanation and specific implementation method involved in this step are similar to those of step S102 above. Please refer to the description of step S102 above, and no repetition will be given here.

[0165] Step S609 : Generate a sixth generated image based on the fifth generated image and the fourth annotated text using the trained student model.

[0166] In an embodiment of the present application, for the student model after training, a sixth generated image is generated based on the fifth generated image and the fourth annotated text. The specific meaning explanation and specific implementation method involved in this step are similar to those of step S103 above. Please refer to the description of step S103 above, and no repetition will be given here.

[0167] Step S610: Determine the knowledge distillation loss of the trained student model based on the fifth generated image and the sixth generated image.

[0168] In an embodiment of the present application, for determining the knowledge distillation loss of the trained student model based on the fifth generated image and the sixth generated image, the specific meaning explanation and specific implementation method involved in this step are similar to those of step S104 above. Please refer to the description of step S104 above, and no further explanation will be given here.

[0169] Step S611: Based on the knowledge distillation loss of the trained student model, the model parameters of the trained student model are adjusted to obtain an image generation model.

[0170] In an embodiment of the present application, for the knowledge distillation loss based on the trained student model, the model parameters of the trained student model are adjusted to obtain an image generation model. The specific meaning explanation and specific implementation method involved in this step are similar to those of step S105 above. Please refer to the description of step S105 above, and the description will not be repeated here. The specific process of adjusting the model parameters is similar to that of steps S212 to S214 above. Please refer to the description of steps S212 to S214 above, and the description will not be repeated here. The image generation model refers to a student model that is obtained after retraining the trained student model, which can generate high-quality images with different visual effect types and containing multiple objects.

[0171] Through steps S607 to S611, when the image quality generated by the student model does not meet expectations, additional sample datasets can be obtained and retrained to improve the model's generation capabilities and image quality. This approach helps the model learn more diverse and complex visual effects, enhancing its performance in practical applications.

[0172] Step S215: The server sends the trained student model to the terminal.

[0173] Step S216: The terminal outputs the trained student model.

[0174] In the embodiment of the present application, by constructing a diverse and richly annotated sample database, not only comprehensive data support is provided for the teacher model and the student model, but also the generalization ability of the model and the quality of the generated images are improved. Through knowledge distillation technology, the student model can learn the knowledge and skills of generating high-quality images from the teacher model, thereby improving its generation ability and image quality. At the same time, by using a sample data set containing multiple objects and different types of visual effects, the student model can learn more diverse and complex image generation capabilities and enhance its performance in practical applications. In addition, obtaining a third annotated text and using the trained student model to generate images can verify the generation ability of the model. By evaluating the effect of the generated image and saving the model parameters when the evaluation value meets the conditions, the quality and reliability of the model can be ensured. This process helps to continuously optimize and improve the performance of the student model, so that it can generate images that are more in line with expectations and meet the needs of practical applications.

[0175] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0176] The Flux LoRA model, with its superior generation quality and multi-style support, has become a focal point in the open-source image generation field. Currently the best open-source image generation model, the Flux LoRA model leads the way in high-quality image generation and multi-style adaptation. The Flux LoRA model offers advantages such as a large number of parameters, multiple styles, and optimized hand and font effects. However, due to the large number of parameters in the Flux LoRA model, it is very difficult for users to customize and fine-tune the Flux LoRA base model. Whether it is full quantization fine-tuning or LoRA fine-tuning, it is difficult to achieve the desired results. The multimodal knowledge distillation method for training Flux models provided in the embodiments of the present application uses models such as SDXL, SD1.5, or Midjerney as teacher models, pre-learning training data features to provide knowledge distillation for Flux LoRA model training, greatly reducing the difficulty of training the Flux model. Furthermore, most Flux LoRA models in the related art are trained for a single category, character, and style. However, this method allows the use of a single Flux model to generate images of multiple categories, characters, and styles.

[0177] The embodiment of this application adopts Figure 7The offline knowledge distillation method of the structure shown, specifically, in view of the characteristics of the Flux basic model with a large number of parameters, a large amount of preliminary knowledge, and difficulty in LoRA training, adopts the knowledge distillation method, using a model with good LoRA training results as the teacher model 701 to guide the Flux LoRA model (i.e., the student model 702) to better learn the knowledge in the training data set. In addition, a special data annotation method is adopted. Specifically, the special data annotation method is used in the process of labeling the images in the training data set to decouple the features such as people and scenes in the image, and at the same time use specific triggers to fix the categories and people in the image. In addition, a text prompt vocabulary is created to expand the training data set. Specifically, based on the annotation mode of the special data annotation method, a part of the training data for the student model training is expanded on the basis of the teacher model. The text prompt vocabulary contains dozens of categories, several characters, and several styles of painting, providing conditions for learning the generation effect of the commercial model.

[0178] The specific image annotation method adopted in the embodiment of the present application can generate a multi-role prompt vocabulary. First, according to the role situation, several pictures of each role from various angles are collected respectively. Then, the image set is labeled according to the role using a specific image marker, and the features required by the model are added to the prompt (to reduce the features that the model does not expect to appear), and the added features are defined as the unique features of each role, and a special trigger word is given to each role. When using the specific image annotation method for specific annotation, the data is first roughly labeled. A machine labeling tool can be used to automatically label and add a role name trigger (all related files can be placed in a folder named role name). Then, data cleaning is performed to remove duplicate annotations (i.e., annotation text) and retain more detailed annotations. Finally, the annotation format is organized into a unified paragraph format (i.e., preset text format). For example, the first line is the character's name; the second line is the character's appearance description, marking the characteristics of the character's appearance, such as hair, eye color, clothing color and style, hat type, upper body or full body portrait, etc.; the third line is the character's action description, marking the character's action expression; the fourth line is the background description, marking the things that appear in the background, location, day and night or other objects, etc.; the fifth line is the category, marking the category of the image; the sixth line is the style, marking the artistic style of the image.

[0179] In the embodiment of the present application, when training the student model, first, image data is collected and image labeling is performed on the images to produce a training data set (i.e., the above-mentioned pre-built sample database) for the teacher model 701 and the student model 702. Then, based on the first input image (i.e., the above-mentioned second sample image) and the first input prompt word (i.e., the above-mentioned second annotated text) in the training data set, the teacher model 701 is trained to obtain a first output image (i.e., the above-mentioned third generated image); the first loss (i.e., the above-mentioned loss value) of the teacher model is calculated based on the first output image and the first input image. The training process diagram of the teacher model 701 is shown as follows: Figure 8 Next, the teacher model 701 is used to test the training effect. The test flow diagram of the teacher model 701 is shown as follows: Figure 9 As shown in Figure 1, multiple second input prompt words are extracted from the training dataset, and the second input prompt words are input into the trained teacher model 701 to obtain the second output image. Finally, knowledge is extracted from the teacher model 701 through the prompt library to train the student model 702, i.e., the Flux model or Flux LoRA model. The knowledge distillation process between the teacher model 701 and the student model 702 is shown in Figure 1. Figure 10 As shown, the third input prompt word (i.e., the first annotated text mentioned above) is input into the trained teacher model 701, and the third output image (i.e., the first generated image mentioned above) is output by the trained teacher model 701. The third input prompt word is simultaneously input into the student model 702 to be trained, and the student model 702 to be trained outputs a fourth output image (i.e., the second generated image mentioned above); the second loss (i.e., the knowledge distillation loss) of the student model 702 is calculated based on the third output image and the fourth output image, and the model parameters of the student model 702 are fine-tuned based on the second loss to obtain the trained student model 702.

[0180] This knowledge distillation process reduces the difficulty of Flux LoRA model training, enabling a single Flux LoRA model to generate images for multiple categories, characters, and styles. Furthermore, a text prompt vocabulary is created based on a specific annotation scheme to expand the training dataset, addressing the issue of commercial models having limited image sets and failing to meet personalized training requirements.

[0181] Based on the model training method based on knowledge distillation described in the above embodiment, Figure 11A structural block diagram of a model training device based on knowledge distillation provided in an embodiment of the present application is shown. The model training device based on knowledge distillation 100 can be a device in an electronic device (for example, a server). The model training device based on knowledge distillation can be implemented in a software manner, which can be software in the form of programs and plug-ins, including the following software modules: an acquisition module 101, a first generation module 102, a second generation module 103, a determination module 104 and an update module 105. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented.

[0182] Among them, the acquisition module 101 is used to obtain a first sample data set; the first sample data set includes multiple first annotated texts, and the first sample image corresponding to each first annotated text has a different visual effect type and contains multiple objects; the first generation module 102 is used to call the pre-trained teacher model to generate a first generated image based on the first annotated text; the second generation module 103 is used to generate a second generated image based on the first generated image and the first annotated text through the student model to be trained; the determination module 104 is used to determine the knowledge distillation loss of the student model based on the first generated image and the second generated image; the update module 105 is used to update the model parameters of the student model based on the knowledge distillation loss to obtain a trained student model.

[0183] In some embodiments, the first sample data set is obtained from a pre-constructed sample database; the device 100 also includes a storage module, which is used to obtain multiple second sample images with different visual effect types; each second sample image includes at least one object; the multiple second sample images are respectively subjected to image labeling processing to obtain a second annotated text for each second sample image; a visual effect type label is generated for each second sample image; each second sample image, the visual effect type label of the second sample image, and the second annotated text of the second sample image are mapped and stored in the sample database.

[0184] In some embodiments, the update module 105 is also used to determine the module to be trained in the student model; fix the parameters of the remaining modules in the student model except the module to be trained; and update the parameters of the module to be trained based on the knowledge distillation loss to obtain the trained student model.

[0185] In some embodiments, the determination module 104 is further used to obtain multiple first pixel values ​​of the first generated image and multiple second pixel values ​​of the second generated image; determine the pixel-level loss of the student model based on the first pixel values ​​and the second pixel values; obtain multiple first image features of the first generated image and multiple second image features of the second generated image; determine the feature-level loss of the student model based on the first image features and the second image features; and determine the knowledge distillation loss based on the pixel-level loss and the feature-level loss.

[0186] In some embodiments, the determination module 104 is further used to determine a first weight corresponding to the pixel-level loss and a second weight corresponding to the feature-level loss; the pixel-level loss is weighted using the first weight to obtain a first weighted loss; the feature-level loss is weighted using the second weight to obtain a second weighted loss; and the sum of the first weighted loss and the second weighted loss is determined as the knowledge distillation loss.

[0187] In some embodiments, the device 100 also includes a model training module, which is used to: obtain a second sample data set from the sample database; the second sample data set includes multiple second sample images and second annotated text for each second sample image, and the multiple second sample images have different visual effect types and contain multiple objects; generate a third generated image based on the second annotated text through the teacher model to be trained; determine the loss value of the teacher model based on the second sample image and the third generated image; based on the loss value, update the model parameters of the teacher model to obtain the pre-trained teacher model.

[0188] In some embodiments, the device 100 also includes an evaluation module, which is used to: obtain a third annotated text; perform image generation processing based on the third annotated text through the trained student model to obtain a fourth generated image; perform effect evaluation on the fourth generated image based on the third annotated text to obtain an effect evaluation value; when the effect evaluation value is greater than or equal to a preset evaluation threshold, obtain the model parameters of the trained student model; add the model parameters to a preset model file; and call the trained student model for image generation by loading the model parameters in the model file.

[0189] In some embodiments, the evaluation module is also used to obtain a third sample data set from a sample database when the effect evaluation value is less than a preset evaluation threshold; the third sample data set includes multiple fourth annotated texts, and the third sample image corresponding to each fourth annotated text has a different visual effect type and contains multiple objects; calling the pre-trained teacher model to generate a fifth generated image based on the fourth annotated text; generating a sixth generated image based on the fifth generated image and the fourth annotated text through the trained student model; determining the knowledge distillation loss of the trained student model based on the fifth generated image and the sixth generated image; adjusting the model parameters of the trained student model based on the knowledge distillation loss of the trained student model to obtain an image generation model.

[0190] It should be noted that the description of the device embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. For technical details not disclosed in the device embodiment, please refer to the description of the method embodiment of the present application for understanding.

[0191] An embodiment of the present application provides an electronic device, Figure 12 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 12 As shown, the electronic device 130 includes: at least one processor 131 ( Figure 12 Only one is shown), a memory 132, and computer executable instructions 133 stored in the memory 132 and executable on at least one processor 131. When the processor 131 executes the executable instructions 133, the steps in any of the above-mentioned embodiments of the model training method based on knowledge distillation are implemented.

[0192] The electronic device may include but is not limited to a processor 131 and a memory 132. It will be understood by those skilled in the art that Figure 12 This is merely an example of the electronic device 130 and does not constitute a limitation on the electronic device 130 . The electronic device 130 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0193] The processor 131 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPG), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0194] In some embodiments, the memory 132 may be an internal storage unit of the electronic device 130, such as a hard disk or memory of the electronic device 130. In other embodiments, the memory 132 may also be an external storage device of the electronic device 130, such as a plug-in hard disk equipped on the electronic device 130, a smart memory card (SMC, Smart Media Card), a secure digital (SD, Secure Digital) card, a flash card, etc. Furthermore, the memory 132 may include both an internal storage unit of the electronic device 130 and an external storage device. The memory 132 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of a computer program. The memory 132 may also be used to temporarily store data that has been output or is about to be output.

[0195] The present invention provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the model training method based on knowledge distillation described in the present invention.

[0196] The embodiment of the present application provides a computer-readable storage medium in which computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the model training method based on knowledge distillation provided in the embodiment of the present application, for example, Figure 1 The model training method based on knowledge distillation is shown.

[0197] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0198] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0199] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0200] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0201] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A model training method based on knowledge distillation, characterized in that: The method comprises: Acquire a first sample data set; the first sample data set includes a plurality of first annotated texts, and a first sample image corresponding to each first annotated text has a different visual effect type and includes a plurality of objects; Calling a pre-trained teacher model to generate a first generated image based on the first annotated text; generating, by a student model to be trained, a second generated image based on the first generated image and the first annotated text; determining a knowledge distillation loss for the student model based on the first generated image and the second generated image; Based on the knowledge distillation loss, the model parameters of the student model are updated to obtain a trained student model.

2. The method according to claim 1, characterized in that The first sample data set is obtained from a pre-built sample database; the method further includes: Acquire a plurality of second sample images with different visual effect types; each second sample image includes at least one object; performing image labeling processing on each of the plurality of second sample images to obtain a second labeled text for each of the second sample images; generating a visual effect type label for each of the second sample images; Each second sample image, the visual effect type label of the second sample image, and the second annotated text of the second sample image are mapped and stored in the sample database.

3. The method according to claim 1, characterized in that The updating of the model parameters of the student model based on the knowledge distillation loss to obtain a trained student model includes: Determining a module to be trained in the student model; Fixing the parameters of the remaining modules in the student model except the module to be trained; Based on the knowledge distillation loss, the parameters of the module to be trained are updated to obtain the trained student model.

4. The method according to claim 1, wherein Determining the knowledge distillation loss of the student model based on the first generated image and the second generated image includes: Obtaining a plurality of first pixel values ​​of the first generated image and a plurality of second pixel values ​​of the second generated image; determining a pixel-level loss of the student model based on the first pixel value and the second pixel value; acquiring a plurality of first image features of the first generated image and a plurality of second image features of the second generated image; determining a feature-level loss for the student model based on the first image feature and the second image feature; The knowledge distillation loss is determined based on the pixel-level loss and the feature-level loss.

5. The method according to claim 4, characterized in that The determining the knowledge distillation loss based on the pixel-level loss and the feature-level loss includes: Determining a first weight corresponding to the pixel-level loss and a second weight corresponding to the feature-level loss; Performing weighted processing on the pixel-level loss using the first weight value to obtain a first weighted loss; Using the second weight value to perform weighted processing on the feature-level loss to obtain a second weighted loss; The sum of the first weighted loss and the second weighted loss is determined as the knowledge distillation loss.

6. The method according to claim 2, characterized in that Before calling the pre-trained teacher model to generate a first generated image based on the first annotated text, the method further includes: Acquire a second sample data set from the sample database; the second sample data set includes a plurality of second sample images and a second annotated text for each second sample image, the plurality of second sample images having different visual effect types and containing a plurality of objects; generating a third generated image based on the second annotated text using a teacher model to be trained; determining a loss value of the teacher model based on the second sample image and the third generated image; Based on the loss value, the model parameters of the teacher model are updated to obtain the pre-trained teacher model.

7. The method according to any one of claims 1 to 6, characterized in that After updating the model parameters of the student model to obtain a trained student model, the method further includes: Get the third annotation text; Performing image generation processing based on the third annotated text using the trained student model to obtain a fourth generated image; Based on the third annotated text, performing an effect evaluation on the fourth generated image to obtain an effect evaluation value; When the effect evaluation value is greater than or equal to a preset evaluation threshold, obtaining model parameters of the trained student model; Adding the model parameters to a preset model file; The trained student model is called to generate an image by loading the model parameters in the model file.

8. The method according to claim 7, characterized in that The method further comprises: When the effect evaluation value is less than a preset evaluation threshold, obtaining a third sample data set from the sample database; the third sample data set includes a plurality of fourth annotated texts, and each fourth annotated text corresponds to a third sample image having a different visual effect type and including a plurality of objects; Calling the pre-trained teacher model to generate a fifth generated image based on the fourth annotated text; generating a sixth generated image based on the fifth generated image and the fourth annotated text using the trained student model; determining a knowledge distillation loss of the trained student model based on the fifth generated image and the sixth generated image; Based on the knowledge distillation loss of the trained student model, model parameters of the trained student model are adjusted to obtain an image generation model.

9. A model training device based on knowledge distillation, characterized in that: The device comprises: An acquisition module is configured to acquire a first sample data set, wherein the first sample data set includes a plurality of first annotated texts, and a first sample image corresponding to each first annotated text has a different visual effect type and includes a plurality of objects; A first generating module, configured to call a pre-trained teacher model to generate a first generated image based on the first annotated text; a second generating module, configured to generate a second generated image based on the first generated image and the first annotated text by using a student model to be trained; a determination module, configured to determine a knowledge distillation loss of the student model based on the first generated image and the second generated image; An updating module is used to update the model parameters of the student model based on the knowledge distillation loss to obtain a trained student model.

10. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer executable instructions or computer programs; A processor, configured to implement the model training method based on knowledge distillation according to any one of claims 1 to 8 when executing computer-executable instructions or computer programs stored in the memory.

11. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer-executable instructions or computer program are executed by a processor, the model training method based on knowledge distillation according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer-executable instructions or computer program are executed by a processor, the model training method based on knowledge distillation according to any one of claims 1 to 8 is implemented.

Images