A method for transferring knowledge of image generation model and computer-readable storage medium

By filtering and migrating the hierarchical response of the image generation model, combined with the cross-domain consistency loss function, the problem of knowledge migration methods in the existing technology ignore the difference in knowledge is solved, and the rapid and effective knowledge migration of image generation model is achieved, which lowers the threshold for use and cost, improving ease of use and general use.

CN115063644BActive Publication Date: 2024-11-08TONGJI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210469387.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-28
Publication Date
2024-11-08
Estimated Expiration
2042-04-28

AI Technical Summary

Technical Problem

The existing knowledge transfer method of image generation model ignores the knowledge differences learned by different structures, resulting in the inappropriate reuse of knowledge with strong specificity, and it is difficult to design an effective supervision method to match the image distribution of new and old models.

Method used

By obtaining the hierarchical response visualization results of the old model, compute the similarity filtering hierarchy suitable for transfer, and set the loss function based on cross-domain consistency, add the filtered old hierarchy to the learning process of the new model to realize knowledge transfer.

Benefits of technology

It effectively reduces the threshold and cost of use, improves the transfer speed and effect, avoids the interference of sample-specific semantic representation on knowledge transfer, and encourages new and old models to match the learned sample correlation relationship.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115063644B_ABST
    Figure CN115063644B_ABST
Patent Text Reader

Abstract

The present invention relates to a method for transferring knowledge of an image generation model and a computer-readable storage medium, wherein the method for transferring knowledge includes: obtaining a number of old image generation models trained on old tasks and freezing their parameters; obtaining a training image set representing a new task and building an untrained image generation model; obtaining a visualization result of the response of each level of the old model to the output sample of the old model; sequentially calculating the similarity between the visualization result of the response of each level and the output sample of the old model, and selecting the level suitable for transfer according to a preset threshold; setting a loss function based on the cross-domain consistency of the selected level and the corresponding level of the new model; adding the selected old level to the learning process of the new model for the new task based on the set loss function to complete the knowledge transfer. Compared with the prior art, the present invention has the advantages of effectively reducing the threshold and cost of use, fast migration speed, and good effect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for transferring knowledge of an image generation model based on hierarchical response and cross-domain consistency and a computer-readable storage medium. Background Art

[0002] Nowadays, with the rapid development of artificial intelligence (AI) technology, the use of AI technology for image generation has attracted widespread attention. Through AI-based image generation technology, users can obtain any number of high-quality image samples that meet their needs. However, in most actual use scenarios of production and life, for efficiency reasons, a set of generation models often needs to be reused in many different tasks at a high frequency. The types and distributions of data between different tasks are different, and the usage conditions required by different tasks (such as response time, consumed computing resources, etc.) are different. As a result, when switching tasks, it is generally necessary to adjust the parameters of the model itself to adapt to the different specific requirements of different tasks. In order to ensure that the performance of the new model is still better after adjusting the parameters, a common practice is to adjust the new model based on the "knowledge" learned by the old model in the old task, that is, the knowledge transfer of the model.

[0003] Chinese patent CN112990222B discloses a guided semantic segmentation method based on image boundary knowledge transfer, and Chinese patent CN113656692A discloses a product recommendation method based on a knowledge transfer algorithm. The knowledge transfer methods used in these two patents will use all parameters of the entire structure of the old model as knowledge transfer. This approach ignores the differences between the knowledge learned by different structures in the old model: in the image generation task, due to the specificity of the task, the knowledge contained in the model trained based on different training samples is also coupled and has a certain specificity. Among this part of knowledge, only part of the knowledge is suitable for reuse, and other knowledge with too strong specificity is not suitable for reuse.

[0004] Chinese patent CN111047054A discloses an adversarial example defense method based on two-stage adversarial knowledge transfer, in which the supervision methods used in the transfer are designed based on the difference between the output distribution of the new and old models. Similarly, in image generation tasks, due to the specificity of the task training samples themselves, it is difficult to design a supervision method by directly or indirectly matching the distribution between the images generated by the new and old models. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an image generation model knowledge migration method and computer-readable storage medium that effectively reduces the usage threshold and usage cost, has a fast migration speed and good effect.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A method for transferring knowledge of an image generation model, the method comprising:

[0008] Step 1: Obtain several old image generation models trained on old tasks and freeze their parameters;

[0009] Step 2: Get a training image set representing the new task and build an untrained image generation model;

[0010] Step 3: Obtain the visualization results of the response of each layer of the old model to the output samples of the old model;

[0011] Step 4: Calculate the similarity between the response visualization results of each level and the output samples of the old model in turn, and select the level suitable for migration according to the preset threshold;

[0012] Step 5: Set the loss function based on the cross-domain consistency of the selected layers and the corresponding layers of the new model;

[0013] Step 6: Based on the set loss function, add the filtered old layers to the learning process of the new model for the new task to complete the knowledge transfer.

[0014] Preferably, the old image generation model in step 1 is specifically a hierarchical structure model.

[0015] Preferably, the image generation model in step 2 has the same structure as the old image generation model in step 1.

[0016] Preferably, the cross-domain consistency between the level selected in step 5 and the corresponding level of the new model is specifically:

[0017] The correlation between the outputs of the selected levels and the corresponding levels of the new model for the same input variables is consistent.

[0018] Preferably, the step 6 is specifically:

[0019] Based on the set loss function, the filtered old layers are added to the training of the new model for the new task, and are set as additional items so as not to affect the established training paradigm of the new model for the new task.

[0020] Preferably, the step 3 is specifically:

[0021] The visualization algorithm is used to obtain the corresponding visualization results of the old model output samples at each level of the old model.

[0022] More preferably, the visualization algorithm comprises a Grad-CAM algorithm.

[0023] More preferably, the method of obtaining the visualization result by using the Grad-CAM algorithm is specifically as follows:

[0024] get The corresponding visualization results The method can be expressed as follows:

[0025]

[0026]

[0027] Among them, ReLU represents the ReLU activation function; express The corresponding weight of each piece of information on the output feature matrix; express The output feature matrix; y represents the image sample finally output by the generated model.

[0028] Preferably, the step 4 uses cosine similarity to calculate the similarity between each level response visualization result and the output sample of the old model.

[0029] A computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement any of the above-mentioned image generation model knowledge transfer methods.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. Effectively reduce the usage threshold and usage cost: The image generation model knowledge transfer method in the present invention can effectively reduce the usage threshold and usage cost of the deep learning image generation model, solve the common problem that the image generation model is difficult to migrate nowadays, and greatly improve the ease of use and versatility of the image generation model.

[0032] 2. Fast migration speed and good effect: The image generation model knowledge transfer method in the present invention encourages the new and old models to match the correlation of the learned samples instead of directly inheriting all parameters. It also screens the levels of the old model and selects the most suitable level for migration, avoiding the interference of sample-specific semantic representation on the knowledge transfer effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of the process of image generation model knowledge transfer method in the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0035] Example 1

[0036] A knowledge transfer method for image generation model, the process is as follows Figure 1 As shown, including:

[0037] Step 1: Obtain several old image generation models trained on old tasks and freeze their parameters;

[0038] The old image generation model in this embodiment is specifically a hierarchical structure model;

[0039] Step 2: Get a training image set representing the new task and build an untrained image generation model;

[0040] The quality and quantity of the images in the training image set in this step should be sufficient to train an image generation model. The image generation model in step 2 has the same structure as the old image generation model in step 1.

[0041] Step 3: Obtain the visualization result of the response of each level of the old model to the output sample of the old model. The visualization result of the response of each level should be able to reflect the degree of correlation between each position in the output feature map of the level and the output sample;

[0042] Step 4: Calculate the similarity between the response visualization results of each layer and the output samples of the old model in turn, and select the layers suitable for migration according to the preset threshold. The screening process requires selecting the layers that produce feature maps with a high degree of similarity to the output samples of the old model;

[0043] Step 5: Set the loss function based on the cross-domain consistency of the selected layers and the corresponding layers of the new model;

[0044] Step 6: Based on the set loss function, add the filtered old layers to the learning process of the new model for the new task to complete the knowledge transfer.

[0045] Example 2

[0046] A method for transferring knowledge of an image generation model based on hierarchical response and cross-domain consistency includes the following steps:

[0047] Step 1: Get several old image generation models trained on old tasks and freeze their parameters. The old models should be able to generate old task samples of sufficient quality. The structure of the old model should be hierarchical. Take 3 as an example, and record the 3 old models as M1, M2, and M3 respectively. Each model has N layers, and the nth layer of the mth model is Represents, where 1≤m≤3, 1≤n≤N, and m and n are both positive integers.

[0048] Step 2: Get a training image set representing a new task and build an untrained image generation model. The training image set for the new task is required to meet the basic training needs of the new model in terms of image quality and sample quantity. The untrained new image generation model is required to have the same structure as the old model in step 1. The new model is called M'.

[0049] Step 3: Obtain the visualization results of the response of each layer of the old model to the old task samples. The size of the corresponding visualization results obtained should be consistent with the size of the output feature map of this layer; the visualization results should be able to reflect the degree of association between each piece of information on the relevant feature map and the original training samples of the old task. The methods for obtaining the response visualization results here include but are not limited to the Grad-CAM method.

[0050] Taking the Grad-CAM method as an example, we can obtain The corresponding visualization results The method can be expressed as follows:

[0051]

[0052] Among them, ReLU represents the ReLU activation function; express The corresponding weight of each piece of information on the output feature matrix. The larger the weight value, the The higher the correlation between the information at that position and the training sample, the lower the correlation is. express The output feature matrix.

[0053] Specifically, in the above formula The calculation method of is as follows:

[0054]

[0055] Where y represents the image sample of the final output of the generative model. The above formula actually regards the gradient value obtained by the derivative operation as the degree of correlation between the information on the feature matrix and the final output. The above is the Grad-CAM approach. Other feasible approaches to obtain visual responses are not listed one by one in this invention. Taking the visual response results obtained by the Grad-CAM method as an example, the results The areas that are strongly correlated with the final generated samples will be highlighted, while the areas that are weakly correlated will not.

[0056] Step 4: Calculate the similarity between the response visualization results of each level and the output samples of the old model in turn, and select the levels suitable for migration based on the preset threshold.

[0057] The recommended method for calculating similarity is cosine similarity, and the recommended threshold is 0.5. The layer that can generate feature maps with similarity to the output sample higher than the threshold is considered to be suitable as a guiding layer for knowledge transfer. The principle is that a high similarity with the output sample proves that the knowledge learned from the old task at this layer is closer to the semantic structure information of the sample, and this information is more reusable; low similarity means that the knowledge learned at this layer is more inclined to specific texture information, which is not suitable for reuse and transfer.

[0058] Take the hierarchical image generation model SinGAN as an example: the entire SinGAN model is a hierarchical training model. The higher the level, the larger the image size and the more details it is responsible for processing. Similar to other computer vision tasks, in SinGAN, the lower-level sub-GAN model is responsible for the perception of more basic elements (such as corners, edges, etc.). This information is more universal and generalizable, and is more suitable for passing to new models as information guidance; while the higher-level sub-GAN model is responsible for the perception of more details of the image (such as texture). This information is highly specialized, and many of them are strongly related to the specific content of the sample. This information is not suitable for passing to the new model for learning. Otherwise, if the differences between the training samples are large, this highly specific information will hinder the training process of the new model. Therefore, before performing subsequent operations, a necessary step is to identify the parts of the old model that may be helpful to the training of the new model and eliminate the parts that may cause interference, which is a screening process.

[0059] Step 5: Design a loss function based on the cross-domain consistency of the above-screened layers and the corresponding layers of the new model. The cross-domain consistency means that when the input variables are the same, the correlation between the outputs of the screened old layers and the untrained new layers is matched as much as possible. An example of a design method for a cross-domain consistency loss function is given: let a set of random noise inputs be z1, ..., z k, where k>1, k is a positive integer. Taking M1 as an example, M1 has a positive effect on z i The correlation of the output y i It can be calculated by the following formula:

[0060]

[0061] Among them, sim(·) represents cosine similarity; softmax(·) represents softmax calculation.

[0062] Similarly, M' to z i The correlation of the output y i ' can be calculated by the following formula:

[0063]

[0064] The final loss function is used to optimize y i and i 'The distance L between:

[0065]

[0066] Among them, D KL (·) denotes KL divergence.

[0067] The above formula is essentially optimizing the distance between the relative relationship of the samples generated by the original model and the relative relationship of the samples generated by the new model, that is, in the migration process, the new model is encouraged to learn as much as possible about the correlation between samples from the old model. In fact, although there are differences between the learning samples, the basic geometric information inside the samples is suitable for migration and reuse, while the appearance information is not suitable for reuse. Although the screening of the most suitable old task level for knowledge transfer has been completed, the knowledge learned at one level is also complex, and it contains both appearance information that is not suitable for migration and basic geometric information that is suitable for migration. The present invention considers using an implicit method to migrate general geometric structure information. Specifically, the present invention proposes to transfer the correlation between images from the old model as the knowledge to be actually transferred in the knowledge migration. Intuitively speaking, the basic geometric information suitable for migration is contained in the correlation between samples. If the new model can retain the relative similarities and differences between the samples generated in the original model, then it can inherit the knowledge of the sample correlation learned by the original model. The method adopted by the present invention does not directly model the distribution of the target, but converts the difficult method of direct modeling into a more easily implemented modeling of the corresponding relationship. By guiding the new model to automatically learn the correspondence between different samples in the original task, the new model can effectively acquire the knowledge learned from the original model, and ultimately generate diverse and realistic images more easily.

[0068] Step 6: Based on the loss function designed above, add the selected old layers to the learning process of the new model for the new task. The above-mentioned rapid knowledge transfer method is added on the basis of the original and established basic training paradigm of the new model. It guides and gains the training of the task model of the new task without affecting the original training paradigm.

[0069] This embodiment also relates to a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute any of the above methods.

[0070] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for transferring knowledge of an image generation model, characterized in that: The knowledge transfer method includes: Step 1: Obtain several old image generation models trained on old tasks and freeze their parameters; Step 2: Get a training image set representing the new task and build an untrained image generation model; Step 3: Obtain the visualization results of the response of each layer of the old model to the output samples of the old model; Step 4: Calculate the similarity between the response visualization results of each level and the output samples of the old model in turn, and select the level suitable for migration according to the preset threshold; Step 5: Set the loss function based on the cross-domain consistency of the selected layers and the corresponding layers of the new model; Step 6: Based on the set loss function, add the filtered old layers to the learning process of the new model for the new task to complete the knowledge transfer; The step 3 specifically includes: using a visualization algorithm to obtain corresponding visualization results of the old model output samples at each level of the old model; The visualization algorithm includes the Grad-CAM algorithm; The design method of the loss function in step 5 is specifically as follows: Let a set of random noise inputs be z1, ..., z k , where k>1, k is a positive integer; The old model M1 responds to random noise z i The correlation of the output y i Calculated by the following formula: Among them, sim(·) represents cosine similarity; softmax(·) represents softmax calculation; Similarly, the new model M' is effective for random noise z i The correlation of the output y i 'Calculated by the following formula: The final loss function is used to optimize y i and i 'The distance L between: Among them, D KL (·) represents KL divergence; The step 6 is specifically as follows: Based on the set loss function, the filtered old layers are added to the training of the new model for the new task, and are set as additional items so as not to affect the established training paradigm of the new model for the new task.

2. The method for transferring knowledge of an image generation model according to claim 1, characterized in that: The old image generation model in step 1 is specifically a hierarchical structure model.

3. The method for transferring knowledge of an image generation model according to claim 1, characterized in that: The image generation model in step 2 has the same structure as the old image generation model in step 1.

4. The method for transferring knowledge of an image generation model according to claim 1, characterized in that: The cross-domain consistency between the levels selected in step 5 and the corresponding levels of the new model is specifically: The correlation between the outputs of the selected levels and the corresponding levels of the new model for the same input variables is consistent.

5. The method for transferring knowledge of an image generation model according to claim 1, characterized in that: The method of obtaining the visualization result by using the Grad-CAM algorithm is specifically as follows: get The corresponding visualization results The method is expressed as follows: Among them, ReLU represents the ReLU activation function; represents the nth layer of the mth model; express The corresponding weight of each piece of information on the output feature matrix; express The output feature matrix; y represents the image sample finally output by the generated model.

6. The method for transferring knowledge of an image generation model according to claim 1, characterized in that: The step 4 uses cosine similarity to calculate the similarity between the response visualization results of each level and the output samples of the old model.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor according to the method for transferring knowledge of an image generation model as described in any one of claims 1 to 6.