A method and apparatus for deconstructing layer representations in an image generation network

Abstract

The application discloses a method and device for generating a layer representation in a deconstruction image generation network, and belongs to the technical field of generative adversarial networks and explainability application. The method modifies a traditional GAN into an explainable GAN without manually labeling semantic features. The application designs a residual function to optimize the enhancement of model explainability by deconstructing the middle layer representation. The method provided by the application can realize unsupervised learning, enables each group of filters to consciously learn image regions of consistent visual concepts, and avoids manual semantic labeling operation, and is more in line with the real characteristics inside the neural network.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning, generative adversarial neural network and explainability application, and particularly relates to a method and device for deconstructing layer representation in an image generation network. BACKGROUND

[0002] Deep learning is increasingly playing an increasingly important role in many scenarios in people's daily life. Generative adversarial networks (GAN) is a deep learning model, and is one of the most promising methods for unsupervised learning on complex distribution in recent years. Generative adversarial networks have been widely used in the medical field, the field of autonomous driving and other image processing fields. However, the end-to-end training method based on it also has the inherent defect of unexplainability, which limits its application in the fields of finance, medicine and law. At the same time, giving the model explainability also helps to ensure fairness, privacy protection and robustness.

[0003] Deconstructing GAN is a direct method to explore the explainability of GAN. Previous studies mainly focus on two aspects of deconstructing GAN. Some works mainly aim at separating the attributes of generated images. Radford [1] visualizes the receptive field of filters in GAN at the same time, indicating that a specific filter corresponds to generating a specific semantic feature. Voynov [2] focuses on optimizing the parameter space of GAN generator to learn an interpretable representation. Shen [3] explores different attributes of generated images by changing the input latent vector, and Jahania [4] explores the latent vector space of GAN to achieve simple transformation of GAN generated images. Other works mainly focus on the separation of structure and texture of generated images. Singh [5] separates the background, object shape and object appearance to generate images in a hierarchical manner. Collin [6] controls and adjusts the basic features of GAN generated images under unsupervised conditions, realizing the decomposition of object components in generated images, but it cannot ensure that each filter corresponds to a single visual concept independently. However, these studies are limited to exploring the explainability of GAN and do not propose a general method to enhance the explainability of GAN. Zhang [7] proposes an explainable convolutional neural network, which adds a residual function to each convolution kernel to promote the independent representation of the convolution kernel to the target component. Based on previous research, the present application not only explores the explainability of GAN, but also proposes a general method to enhance the explainability of GAN.

[0004] The contributions of the present application can be summarized as follows. We propose a general method to modify the traditional GAN into an explainable GAN without any annotation of visual concepts. In the explainable GAN, each filter in the intermediate layer of the generator always generates the same local visual concept when generating different images. Experiments show that our method can be applied to different types of GANs. SUMMARY

[0005] The present application aims at the deficiencies of the prior art and provides a method and device for deconstructing layer representation in image generation network. The present application makes the intermediate layer of the GAN generator deconstruct local visual concepts, that is, each filter of the intermediate layer of the generation model always generates the same local visual feature concept when generating different images.

[0006] The present application is realized by the following technical scheme: a method for deconstructing layer representation in image generation network, comprising the following steps:

[0007] (1) Given the generator G of the generative adversarial network, a Gaussian mixture model is constructed to group any layer of G, so that the filters of each group independently generate the same image region;

[0008] (2) An energy model is established to represent the authenticity of the generated image; specifically, the energy function is set to be the feature map output by each filter of the selected layer and the feature map points of the group to which it belongs are multiplied and multiplied by the energy model parameters, so as to enhance the authenticity of the GAN generated image;

[0009] (3) Design a residual function so that each filter in the same group of filters can independently generate the same image region and the independence of the image regions generated by different groups of filters; specifically, the energy model parameters are optimized so that when the jth filter belongs to the cth class, the energy model parameter W jc >0, when the jth filter does not belong to the cth class, the energy model parameter W jc <0;

[0010] (4) Overall residual function design, by fusing the loss of GAN training, the loss of optimizing EBM and the loss of making the target layer filter deconstruct, so as to achieve the purpose of optimization;

[0011] (5) Energy model optimization, adopt EBM and GAN alternating training mode, by alternating multiple times of training EBM and GAN, until each class of filter can determine a class of visual concept;

[0012] In the step (3), all filters in a group independently generate the same image region, that is, the filter belongs to the cth class; different filters generate different image regions, that is, the filter does not belong to the cth class.

[0013] Further, the step (1) is specifically:

[0014] The filter of the selected layer of the Gaussian mixture model is regarded as a variable, and the Gaussian mixture model is used to cluster the filter, and the corresponding class corresponds to several peaks of the Gaussian mixture model, and the class to which each filter belongs can be obtained by optimizing the GMM parameters through the EM algorithm; and grouping is performed according to the class to which the filter belongs.

[0015] Further, in the step (2), the energy model represents the authenticity of the generated image; and the step (2) further includes: for the hidden vector noise inputted by the GAN randomly, the hidden vector noise is optimized by using the Langevin dynamics, so as to optimize the image quality.

[0016] Further, the step (5) is specifically:

[0017] The method of training the discriminator D first and then training the generator G is used to train the GAN multiple times; and then the parameters of the energy model and the parameters of the generator G of the GAN from the first layer to the target layer are trained.

[0018] Further, the training of the parameters of the energy model and the parameters of the generator G of the GAN from the first layer to the target layer includes the following steps:

[0019] The modified vector is obtained by calculating the Langevin dynamics for each input hidden vector of the GAN, and the ideal feature map is obtained by forward propagation using the modified vector.

[0020] The EBM parameters are regarded as a learnable parameter, and the EBM parameters and the parameters of the GAN from the first layer to the target layer are updated.

[0021] Further, in the step (2), the generated image is manipulated by directly manipulating the feature map outputted by the filter.

[0022] Further, in the step (2), the translation, rotation and scaling operation of the feature map outputted by the filter is used to manipulate the change of the semantic concept of the generated image.

[0023] Further, in the step (2), the feature maps outputted by different groups of filters are exchanged, so as to achieve the purpose of exchanging visual concepts between the generated images.

[0024] An apparatus for deconstructing layer representation in image generation network, comprising one or more processors for implementing the above-mentioned method for deconstructing layer representation in image generation network.

[0025] A computer readable storage medium, having stored thereon a program, which when executed by a processor, is used to implement the method for generating layer representation in a deconstruction image generation network.

[0026] The present application has the beneficial effects that: the present application considers the process calculation problem and the enhanced explainability problem, adopts a Gaussian mixture model (GMM) to group the model target layer filter, and enhances the explainability of the GMM by designing a new residual function. The method proposed in the present application can realize unsupervised learning, so that each group of filters can generate image regions corresponding to visual concepts. The present application can realize no labeling, so that it can learn the image regions corresponding to the visual concepts itself, and is more consistent with the actual representation inside the neural network. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0028] Figure 1 It is a flowchart of the method of the present application;

[0029] Figure 2 It is a comparison chart of the image generated by the traditional GAN and the image generated by the present application;

[0030] Figure 3 It is a diagram of visualizing the receptive field of the explainable GAN proposed in the present application;

[0031] Figure 4 It is a result chart of exchanging visual concepts between two images by using the explainable GAN proposed in the present application;

[0032] Figure 5 It is a hardware structure diagram of the present application. DETAILED DESCRIPTION

[0033] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0034] The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the present application and the following claims, the singular forms "a," "an" and "the" include plural forms unless the context clearly dictates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0035] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It is also to be understood that the term such as "if" can be interpreted as "when" or "upon" or "in response to determining" depending on the context.

[0036] The application will be described in detail below with reference to the attached drawings. The features of the embodiments and implementation described below can be combined with each other in the case of no conflict.

[0037] As shown in the following figure, one method for disassembling the layer representation in the image generation model based on the generative adversarial network of the present application includes the following steps: Figure 1

[0038] Step 1, given the generator G, use GMM to solve the grouping condition Q so that the filter of each group independently generates the same image region; including the following sub-steps:

[0039] Step 1.1, establish a Gaussian mixture model. The specific formula is as follows:

[0040]

[0041]

[0042] Θ is the established Gaussian mixture model, and then the model parameters can be solved.

[0043] Wherein x represents the filter of the selected target layer; K represents the number of sub-Gaussian models in the mixture model, that is, the number of clusters; k = 1, 2,..., K; p k represents the probability that the selected filter belongs to the kth class, φ(x∣θ k ) represents the distribution of the kth class, which itself satisfies the distribution of a Gaussian model; μ k is the mean of the kth sub-Gaussian model, σ k is the variance of the kth sub-Gaussian model. ​

[0044] Step 1.2, the optimization goal of GMM is the multi-sample clustering likelihood function, which is specifically expressed as follows:

[0045]

[0046] L(Θ) is the establishment of the likelihood function, and the Gaussian mixture model parameters {p k ,μ k ,σ 2}∈Θ are learned by maximizing the likelihood function.

[0047] Where s = 1, 2,... S, S represents the number of sample pictures, j = 1, 2,... N, N represents the number of selected target layer filters, and X j represents the jth filter of the target layer.

[0048] Step 1.3, the GMM parameters are estimated by using the EM algorithm; including the following sub-steps:

[0049] Step 1.3.1, hidden variable explanation

[0050]

[0051]

[0052] Where z j ∈{1, 2,..., K} represents the class to which the jth filter belongs; k ∈{1, 2,..., K} represents the number of filter groups, and Θ is the established GMM model parameter; s = 1, 2,... S, S represents the number of sample pictures; represents the kth filter of the st sample

[0053] Step 1.3.2, E step: calculating expectation and posterior probability

[0054]

[0055] The E step is to calculate the probability that the jth filter of the st sample belongs to the kth class according to the current GMM parameter. Wherein, represents the kth filter of the st sample,

[0056] θ k is the parameter of the kth Gaussian model, represents the probability that the jth filter of the sample picture comes from the kth group under the current model parameter.

[0057] Step 1.3.3, M step: calculating model parameters

[0058]

[0059] where and represent the mean and variance of the k-th Gaussian model, respectively. represents the weight of each Gaussian model. represents the probability of the j-th filter of the sample image coming from the k-th group under the current model parameters. represents the k-th filter of the s-th sample

[0060] s = 1, 2,... S, S represents the number of sample images, j = 1, 2,... N, N represents the number of selected target layer filters, and k = 1, 2,... K represents the total number of groups.

[0061] Step 1.3.4, check whether the number of GMM clustering reaches the maximum step or the change of maximum likelihood estimation is less than the set threshold, if the condition is met, it is considered that the clustering is completed, otherwise return to step 1.3.2.

[0062] Step 2, the energy model can output feature map probability to improve the image authenticity, an energy model is established to represent the authenticity of the generated image, and the loss is designed as follows:

[0063]

[0064] where z represents the filter obtained by optimizing the energy model for the GAN input hidden vector, f G (z) represents the feature map of the target layer, Q represents the clustering grouping result of the GMM, P W (f G (z) | Q) is the energy model after determining the classification Q, λ1 is the weight design, L(χ) is the indicator function, and Z(W) = ∫exp(g W (f G′ (z)))P0(z)dz is used for normalization. where z ~ N(0, σ 2 I d ) represents that z satisfies the Gaussian distribution, I d represents the d-dimensional unit matrix. σ 2 represents the variance.

[0065]

[0066]

[0067] g W (f G (z)) is an energy function, which is a measure index of the authenticity of the generated image.

[0068] where f j represents the feature map output by the j-th filter, The feature map output by the cluster center representing the c-th class, W jc Representing the energy model parameters.

[0069] Step 3, design the residual function so that each filter in the same group can independently generate the same image region, and the independence of the image region generated by different groups of filters is represented as:

[0070]

[0071] In order to represent the interpretability, all filters in a group independently generate the same image region, that is, the j-th filter belongs to the c-th class, in order to f j Approach By setting W jck > 0, g W (f G (z)) so that it is approximate; different filters generate different image regions, that is, the j-th filter does not belong to the c-th class, in order to f j Away By setting W jck < 0, g W (f G (z)) so that it is different.

[0072] Step 4, overall residual function design

[0073] In order to realize the interpretability, the loss is introduced as:

[0074]

[0075] Wherein, λ2, λ3 are weights, In order to realize the grouping of target layer filters, In order to increase the authenticity of the generated image, In order to increase the interpretability.

[0076]

[0077] L is the overall residual function, and λ0 is the weight.

[0078] Wherein The loss of GAN,

[0079] The overall loss is optimized as:

[0080]

[0081] Step 4, model training, adopt EBM and GAN alternating training mode;

[0082] Step 4.1, train GAN multiple times; including the following sub-steps:

[0083] Step 4.1.1, fixing the generator G, training the discriminator D;

[0084]

[0085] where x i is the input image, is the generated image by the generator, θ d is the discriminator parameter, i = 1, 2,... m represents the ith sample participating in the training, D(x i ) represents the output of the discriminator D to the real signal data, represents the output of the discriminator D to the false image generated by the generator G, represents the adversarial loss function between the generator G and the discriminator D. It represents the ability of D to distinguish real data and false data; represents the derivative

[0086] Step 4.1.1, fixing the discriminator D, training the generator G;

[0087]

[0088] where θ g is the generator parameter, represents the adversarial loss function between the generator G and the discriminator D. It represents the ability of D to distinguish real data and false data. G(z i ) represents the output of the generator G inputting the hidden vector.

[0089] Step 4.2, training the sum of EBM parameters and GAN from the first layer to the target layer multiple times; including the following sub-steps:

[0090] Step 4.2.1, for each input GAN hidden vector z i Calculate the Langevin dynamics for MCMC sampling to obtain the correction vector

[0091]

[0092] where τ represents the time step, δ represents the weight of each step, U τ ~ N(0, I d ) represents Gaussian noise, d is the dimension of z, and based on this the corrected hidden vector Using the correction vector Forward propagation to obtain the ideal feature map

[0093] Step 4.2.2, regarding the EBM model parameter W as a learnable parameter, updating its EBM parameters and the parameters of the GAN from the first layer to the explainable layer

[0094]

[0095] The gradient of which can be directly obtained:

[0096]

[0097]

[0098] Step 4.3, repeat step 4.1, step 4.2 until the Loss does not decrease anymore.

[0099] As Figure 2 shown, compared with the traditional GAN, each filter in the explainable GAN consistently represents a meaningful visual concept when generating different images; different filters represent different visual concepts.

[0100] Figure 3 is a visualization of the receptive fields (RFs) corresponding to the filters of StyleGAN enhanced for explainability using the present application and scaled up to image resolution. When generating different images, the filters of the GAN model enhanced for explainability using the present application generate the same image regions corresponding to the same visual concept, and different filters generate different image regions corresponding to different visual concepts. The first column shows the generated images. The second column shows the visualization of the distribution of visual concepts encoded in the intermediate layer filters. Each of the remaining columns in the figure corresponds to a certain class of filters. The visualization results show that each group of filters in the explainable GAN consistently generates image regions corresponding to the same visual concept. Different groups of filters generate image regions corresponding to different visual concepts.

[0101] Figure 4 is to exchange a specific visual concept between the original image and the source image. The second column shows the selected exchange component, marked in red. The fourth column shows the modified image. The fifth column shows the mean square error heat map between the original image and the modified image. The heat map shows that our modification is perceptible and local.

[0102] Figure 5 is to modify the spires on the generated church image by applying geometric transformations to the corresponding feature maps in our explainable GAN. The results show that our method can also be used to geometrically control the appearance of specific visual concepts

[0103] For a pair of images, the face of the original image is replaced with the face of the source image to generate a modified image, and then it is tested whether the face of the modified image and the face of the source image have the same identity. Specifically, 2K pairs of faces are selected, and the ArcFace method is used to test the results. The results show that the method of the present application is superior to other state-of-the-art face swapping methods in maintaining identity after face swapping. As shown in Table 1:

[0104] Table 1: Face verification experiment evaluates the results of face swapping

[0105]

[0106] Corresponding to the foregoing embodiment of the method for disassembling the middle layer representation of the image generation network, the present application also provides an embodiment of a device for disassembling the middle layer representation of the image generation network.

[0107] Referring to Figure 5 , the device for disassembling the middle layer representation of the image generation network provided by the embodiment of the present application comprises one or more processors for implementing the method for disassembling the middle layer representation of the image generation network in the foregoing embodiment.

[0108] The device embodiment for disassembling the middle layer representation of the image generation network can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in logical sense, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory for execution by the processor of the device with data processing capability where it is located. From the hardware level, as shown in Figure 5 , it is a hardware structure diagram of the device for disassembling the middle layer representation of the image generation network in any device with data processing capability where it is located. In addition to the processor, the memory, the network interface, and the non-volatile memory shown in Figure 5 , the device in the embodiment is usually provided with other hardware according to the actual functions of the device with data processing capability where it is located, and the details are not described here.

[0109] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, and is not described here.

[0110] For the apparatus embodiment, since it basically corresponds to the method embodiment, the relevant part is described in the part of the method embodiment. The apparatus embodiment described above is only illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Some or all of the modules can be selected to achieve the purpose of the present application according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0111] The embodiment of the present application also provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the method for generating layer representation in a deconstruction image generation network.

[0112] The computer readable storage medium can be an internal storage unit of any data processing device, such as a hard disk or a memory. The computer readable storage medium can also be any data processing device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both an internal storage unit of any data processing device and an external storage device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0113] The above is only the preferred embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

[0114] The above embodiments are only used to illustrate the design ideas and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and implement it, and the protection scope of the present application is not limited to the above embodiments. Therefore, any equivalent change or modification made according to the disclosed principles and design ideas is within the protection scope of the present application.

[0115] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope of the application being indicated by the following claims.

[0116] It should be understood that the application is not limited to the precise construction here described and illustrated in the drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application.

Claims

1. A method for deconstructing layer representations in an image generation network, the method comprising: Comprising the following steps: (1) Given a generator G of a generative adversarial network, a Gaussian mixture model is constructed to group any layer of G, so that the filters of each group independently generate the same image region; comprising: Step 1.1, establish a Gaussian mixture model; the specific formula is as follows: Θ is the established Gaussian mixture model, and then the model parameters are solved; where x represents the filter of the selected target layer; K represents the number of sub-Gaussian models in the mixture model, that is, the cluster number; k = 1, 2, …, K; p k represents the probability that the selected filter belongs to the kth class, φ(x|θ k ) represents the distribution of the kth class, which itself satisfies a Gaussian model distribution; μ k is the mean of the kth sub-Gaussian model, σ k is the variance of the kth sub-Gaussian model; Step 1.2, the optimization objective of GMM is the multi-sample clustering likelihood function, which is specifically expressed as follows: L(Θ) is to establish a likelihood function, by maximizing the likelihood function to learn Gaussian mixture model parameters {p k ,μ k ,σ 2}∈Θ, wherein s=1,2,…S, S represents the number of sample pictures j=1,2,…,N, N represents the number of selected target layer filters, X j represents the jth filter of the target layer; Step 1.3, estimate the GMM parameters using the EM algorithm; comprising the following sub-steps: Step 1.3.1, hidden variable explanation where z j represents the class to which the jth filter belongs; k e {1,2,..., K} represents the group number of filter grouping; Θ is the established GMM model parameter; s = 1,2,... S, S represents the number of sample pictures; represents the k filters of the st sample Step 1.3.2, E step: calculate the expectation and posterior probability Estep is to calculate the probability that the jth filter of the st sample belongs to the kth class according to the current GMM parameters; wherein, represents the kth filter of the st sample, θ k is a parameter of the kth Gaussian model, denotes the jth filter of the sample picture comes from the kth group of probabilities under the current model parameters; Step 1.3.3, M step: calculate the model parameters wherein respectively represent the mean and variance of the kth Gaussian model; represent the weight of each Gaussian model; denotes the probability that the jth filter of the sample picture comes from the kth group under the current model parameters; denotes the kth filter of the st sample s=1,2,…S, S represents the number of sample pictures j=1,2,…,N, N represents the number of selected target layer filters, k=1,2,…K represents all the number of groups; Step 1.3.4, check whether the GMM clustering number reaches the maximum step or the maximum likelihood estimation change is less than the set threshold, if the conditions are met, it is considered that the clustering is completed, otherwise return to step 1.3.2; (2) An energy model is established to represent the authenticity of the generated image; specifically: set the energy function as the feature map of each filter output of the selected layer and the feature map point multiplication of its own group, and multiply by the energy model parameter, in order to enhance the authenticity of the GAN generated image; The loss is designed as follows: where z represents the filter obtained by optimizing the energy model for the GAN input latent vector f G (z) represents the feature map of the target layer, Q represents the result of GMM clustering grouping, P W (z) represents the feature map of the target layer, Q represents the result of GMM clustering grouping, P G (z) represents the feature map of the target layer, Q represents the result of GMM clustering grouping, P W (z) represents the feature map of the target layer, Q represents the result of GMM clustering grouping, P G (z) represents the feature map of the target layer, Q represents the result of GMM clustering grouping, P where z ~ N(0, σ 2 I d (z) represents that z satisfies the Gaussian distribution, I d represents the d-dimensional unit matrix; σ 2 represents the variance; g W (f G (z)) is an energy function as a measure of the generated image's realism; wherein f j a feature map representing the output of the jth filter, a feature map representing the output of the cluster center of the cth class, W jc representing an energy model parameter; (3) Designing the residual function so that each filter in the same group can independently generate the same image region and the independence of the image region generated by different groups of filters; Specifically, the energy model parameters are optimized so that when the jth filter belongs to the cth class, the energy model parameters W jc > 0, and when the jth filter does not belong to the cth class, the energy model parameters W jc < 0; Specifically, it is represented as: (4) The overall residual function design, by fusing the loss of GAN training and the loss of optimizing EBM and the loss of making the target layer filter deconstruction, so as to realize the optimization purpose; comprising: The introduced loss is: wherein λ2, λ3 are weights, to implement target layer filter grouping, to increase the realism of the generated image, to increase the explainability; L is the overall residual function, and λ0 is the weight; wherein is the loss of the GAN, The overall loss optimization is: (5) Energy model optimization, adopt the alternating training method of EBM and GAN, through alternating multiple training of EBM and GAN, until each type of filter can determine a type of visual concept; In step (3), all filters in a group independently generate the same image region, that is, the filter belongs to the cth class; different filters generate different image regions, that is, the filter does not belong to the cth class.

2. The method of claim 1, wherein, Said step (1) is specifically: The filters of the selected layer of the Gaussian mixture model are regarded as variables, and the Gaussian mixture model is used for clustering, the corresponding class corresponds to several peaks of the Gaussian mixture model, and the GMM parameters are optimized by the EM algorithm to obtain the class to which each filter belongs; then group according to the class to which the filter belongs.

3. The method of claim 1, wherein, In step (2), the energy model represents the authenticity of the generated image; further comprising: for the hidden vector noise randomly input by GAN, the Langevin dynamics is used to optimize the hidden vector noise to optimize the image quality.

4. The method of claim 1, wherein, Said step (5) is specifically: First train the discriminator D and then train the generator G to train the GAN multiple times; then start training the parameters of the energy model and the generator G of the GAN from the first layer to the target layer.

5. The method of claim 4, wherein, Said training the parameters of the energy model and the generator G of the GAN from the first layer to the target layer, comprising the following steps: The MCMC sampling is performed on the Langevin dynamics of the latent vector of each input GAN to obtain a correction vector, and the ideal feature map is obtained by forward propagation using the correction vector; The EBM parameter is regarded as a learnable parameter, and the EBM parameter and the parameters of the GAN from the first layer to the target layer are updated.

6. The method of claim 1, wherein, In the step (2), the generated image is manipulated by directly manipulating the feature maps output by the filters.

7. The method of claim 1, wherein, In the step (2), the generated image is manipulated by translating, rotating and scaling the feature maps output by the filters.

8. The method of claim 1, wherein, In the step (2), the generated image is manipulated by exchanging the feature maps output by different groups of filters.

9. An apparatus for deconstructing layer representations in an image generation network, the apparatus comprising: The program is executed by the processor to implement the method for representing a layer of an image generation network in a deconstruction manner according to any one of claims 1-8.

10. A computer-readable storage medium having stored thereon a program, characterized in that, The program is executed by the processor to implement the method for representing a layer of an image generation network in a deconstruction manner according to any one of claims 1-8.

Images