Explanatable training method of unconditional image generation diffusion probability model

Abstract

The invention relates to the field of image editing, and particularly discloses an explainable training method of an unconditional image generation diffusion probability model, which is used for solving the problem of corresponding confusion of input Gaussian noise and output images of a de-noising diffusion probability model. According to the method, different semantic regions of Gaussian noise in input are regarded as independent players, corresponding games are designed, and in order to enhance the interpretability of a model, a loss function based on a Shapley value is designed, so that the corresponding relation between the input noise and the semantic region of an output image is enhanced; a method for reducing calculation complexity is provided; the method can be applied to various DDPM models, has good generalization ability, can accurately resample the local visual concept of the image, and allows a user to more accurately modify a certain part in the image without being interfered by other areas.

Description

Technical Field

[0001] The present invention relates to the field of image editing, and more particularly to an interpretable training method for an unconditional image generation diffusion probability model. Background Art

[0002] In recent years, the Denoising Diffusion Probabilistic Model (DDPM) has achieved remarkable results in the field of image generation. Whether it is image generation from scratch or detailed rendering based on existing images, DDPM has shown excellent performance. Although the generation effect of DDPM is impressive, the understanding of its internal mechanism, especially the role and principle of each step of the denoising process, is still limited. This understanding is not only crucial for theoretical research, but also for fine image editing in practical applications. When we can understand how each component in DDPM learns representations and how they affect the final generated image, we can process and edit images more effectively.

[0003] Intuitively, since the input Gaussian noise and the image generated during the DDPM denoising process have the same dimension, it can be expected that modifications made to certain specific areas of the input noise should be directly reflected in the corresponding areas of the final image. For example, in the DDPM process of generating a face image, if the part of the input Gaussian noise corresponding to the nose is modified, then the final generated image should show a different nose, while other areas remain unchanged. However, the original DDPM does not have this intuitive interpretability property. In order to solve the above problem, a technical solution is now provided. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides an interpretable training method for an unconditional image generation diffusion probability model, which is used to solve the problem of confusion between the input Gaussian noise and the output image of the existing denoising diffusion probability model, so as to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for interpretable training of a diffusion probability model for unconditional image generation, comprising the following steps:

[0007] Step 1: Problem formalization and semantic region division: By performing image sequence representation and semantic region division on the output images during the DDPM denoising process, the first constraint condition of the desired self-explanatory model is preliminarily determined;

[0008] Step 2, game theory model and Shapley value attribution: treat the semantic regions in the input Gaussian noise as independent "players" and define the game function Used to measure the kth t The benefits of semantic regions;

[0009] Step 3: Design of interpretable loss function: Reduce the computational complexity by simplifying the basic interpretable loss function.

[0010] As a further solution of the present invention, in step 1, the first constraint condition of the desired self-explanatory model is preliminarily determined by performing image sequence representation and semantic region division on the output image during the DDPM denoising process. The specific steps are:

[0011] Step 11: The image sequence output during DDPM denoising is expressed as At x0, x1, …, x T The same semantics as x0 is applied to the region division, x T is the initial Gaussian noise, x1 is the Gaussian noise after the first denoising, and x0 is the final denoised image;

[0012] Step 12, use Represents a one-hot vector of length K, used to indicate the xth t The kth image sequence t semantic regions, and in K is the number of all semantic regions, is the set of all semantic regions, thus we get in, is the One-Hot vector of the k0th semantic region of the output image x0, is the One-Hot vector of the k1th semantic region of the output image x1, The output image x T kth T The One-Hot vector of the semantic region, k0 is the number of the k0th semantic region of the final generated image x0, k1 is the number of the k1th semantic region of the final generated image x1, and k T To finally generate the image x T kth T The number of the semantic region;

[0013] Step 13, based on the Shapley value attribution perspective, preliminarily determine the first constraint condition of the expected self-explanatory model:

[0014]

[0015] Where: I(·) is the indicator function, is the One-Hot vector of the k0th semantic region of the output image x0, is the attribution value calculated based on the Shapley value, is the attribution function based on the Shapley value, is a game function, which represents the average pixel value of the k0th semantic area on x0.

[0016] As a further solution of the present invention, in step 2, the semantic regions in the input Gaussian noise are regarded as independent "players", and the game function is defined Used to measure the kth t The specific steps are:

[0017] Based on the Markov process characteristics of DDPM, the original constraints are transformed into ensuring that at each step of the denoising process, when k t-1 =k t From time point t to t-1, x t-1 k t-1 Semantic regions are independently affected by x t k t The semantic region control is simplified to obtain the following constraints:

[0018]

[0019] Where: I(·) is the indicator function, The output image x t-1 kth t-1 One-Hot vector of semantic regions, is the attribution value calculated based on the Shapley value, which is used to describe the input image x t The semantic region pairs generate the image x t-1 kth t-1 The contribution value of a semantic region is is the attribution function based on the Shapley value, is the game function, which means generating image x t-1 Previous k t-1 The pixel average value of the semantic region.

[0020] As a further solution of the present invention, in step 3, the computational complexity is reduced by simplifying the basic interpretability loss function, and the specific steps are:

[0021] Step 31, use the sampling method in DDIM to ensure that the generation process is completely determined by the input Gaussian noise, and convert the game function The equivalent setting is to return the kth prediction noise t The average value of semantic regions, that is, ∈θ (x t , t) in k t The pixel average value of the semantic area, thus the formula of the basic interpretability loss function is proposed as:

[0022]

[0023] The basic interpretability loss function is understood as x t k t constrains the region to affect only x t-1 k t-1 Area (k t-1 =k t ), and not for other regions (k t-1 ≠k t ) have an impact;

[0024] Step 32, for each step of back propagation, calculating the Shapley value of a region requires considering all possible region subsets, the number of which is 2 K , where K is the current noisy image x t The number of regions, so the time complexity of calculating a single Shapley value is O(2 K );

[0025] Step 33, the calculation of the basic interpretability loss function requires x t K semantic regions and x t-1 The K-1 semantic regions are all calculated, and the overall calculation time complexity is O(K(K-1)2 K ), an equivalent loss function is proposed based on time complexity, which reduces the time complexity of accurately calculating the loss function from O(K(K-1)2 K ) is reduced to O(K2 K ), if for any subset, all have v(SU{i})=v({i}) and v(S)=v(φ), then all players in N\{i} are considered to be virtual players, where N is the set of all players, i is the target player, S is the subset, and the formula of the equivalent loss function is:

[0026]

[0027] Where: L IE is the equivalent loss function, E t is the expected value at time step t, φ is an empty set, When the semantic region k t After adding to the subset S, the kth t-1 The incremental contribution of the region, Generate x for the subset S t-1k t-1 Total contribution of the region.

[0028] The technical effects and advantages of the interpretable training method of the unconditional image generation diffusion probability model of the present invention are as follows: the present invention regards different semantic regions of Gaussian noise in the input as independent "players" and designs corresponding game. In order to enhance the interpretability of the model, a loss function based on Shapley value is designed to enhance the correspondence between the input noise and the semantic region of the output image, and a method for reducing the computational complexity is proposed; the present invention can be applied to a variety of DDPM models, has good generalization ability, can accurately resample the local visual concept of the image, and allows the user to more accurately modify a part of the image without being disturbed by other regions. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A flowchart of an interpretable training method for an unconditional image generation diffusion probability model provided by the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the accompanying drawings in the present invention to clearly and completely describe the technical solution in the present invention. Obviously, the described technical solution is only a part of the present invention, not all of it. Based on the technical solution in the present invention, all other technical solutions obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0031] Figure 1 The flowchart of the interpretable training method of the diffusion probability model for unconditional image generation provided by the present invention is as follows. Figure 1 As shown, an interpretable training method for an unconditional image generation diffusion probability model includes the following steps:

[0032] Step 1: Problem formalization and semantic region division: By performing image sequence representation and semantic region division on the output images during the DDPM denoising process, the first constraint condition of the desired self-explanatory model is preliminarily determined;

[0033] Step 2, game theory model and Shapley value attribution: treat the semantic regions in the input Gaussian noise as independent "players" and define the game function Used to measure the kth t The benefits of semantic regions;

[0034] Step 3: Design of interpretable loss function: Reduce the computational complexity by simplifying the basic interpretable loss function.

[0035] Specifically, in step 1, the first constraint of the desired self-explanatory model is preliminarily determined by performing image sequence representation and semantic region division on the output image during the DDPM denoising process. The specific steps are:

[0036] The image sequence output during the DDPM denoising process is represented as At x0, x1, …, x T The same semantics as x0 is applied to the region division, x T is the initial Gaussian noise, x1 is the Gaussian noise after the first denoising, and x0 is the final denoised image;

[0037] use Represents a one-hot vector of length K, used to indicate the xth t The kth image sequence t semantic regions, and in K is the number of all semantic regions, is the set of all semantic regions, thus we get in, is the One-Hot vector of the k0th semantic region of the output image x0, is the One-Hot vector of the k1th semantic region of the output image x1, The output image x T kth T The One-Hot vector of the semantic region, k0 is the number of the k0th semantic region of the final generated image x0, k1 is the number of the k1th semantic region of the final generated image x1, and k T To finally generate the image x T kth T The number of the semantic region;

[0038] Based on the Shapley value attribution perspective, the first constraint of the expected self-explanatory model is preliminarily determined:

[0039]

[0040] Where: I(·) is the indicator function, is the One-Hot vector of the k0th semantic region of the output image x0, is the attribution value calculated based on the Shapley value, is the attribution function based on the Shapley value, is the game function, which represents the pixel average value of the k0th semantic area on x0; is x obtained based on the Shapley value T Generate the attribution value of the k0th region on x0; for example, is the input Gaussian noise x T Previous k T Scalar Shapley value of the region; game function Refers to the entire denoising process of DDPM, which is used to generate the k0th semantic region on x0, and is set to generate the pixel average of the k0th semantic region on image x0. That is, for any k0th semantic region on x0, it is expected that such a generated result is only composed of x T The corresponding kth T There are k semantic regions contributing T =k0.

[0041] By constraining k0=k1=…=k T , to ensure that the input Gaussian noise x T There is a direct and clear mapping relationship between a certain semantic region in the generated image x0 and the corresponding semantic region. This consistency makes the contribution of each input semantic region clear and independent, enhancing the interpretability of the model. Each semantic region of the output image (such as an object or background) is completely independently generated by the corresponding region in the input noise, avoiding interference between different semantic regions. By clearly defining a certain semantic region k in the input noise, T It only affects the region k0 with the same number in the output. It can make region-level modifications on the input noise to control the content of a specific part of the output image. If a semantic region of the output image needs to be modified, it is only necessary to adjust the corresponding region in the input noise without retraining the model or affecting other regions. The generation process is quantitatively analyzed using the Shapley value attribution function, which clarifies the specific contribution of each input region to the output region, which is helpful for model diagnosis and optimization. Through the attribution constraint of the Shapley value, it is ensured that the generation process strictly satisfies the semantic region correspondence between the input and output, making the model behavior easier to understand. Each semantic region in the output image is only affected by the corresponding input region, avoiding coupling or misleading between different regions. This design can improve the stability and reliability of the model. Since the relationship between input and output is clear, if the noise in a certain input region is large, the output result can be adjusted by controlling the noise level of the region, thereby improving the robustness to noise. Using the input Gaussian noise x T The sub-region design supports independent generation or modification of specific areas of the output image. For example, in the generated image, only a specific object is modified without affecting other parts. Through the semantic consistency of the sub-region mapping, the content of the generated image can be controlled more finely, and the resolution and detail integrity of the generated result can be improved.

[0042] Specifically, in step 2, the semantic regions in the input Gaussian noise are regarded as independent "players" and the game function is defined Used to measure the kth tThe specific steps are:

[0043] Based on the Markov process characteristics of DDPM, the original constraints are transformed into ensuring that at each step of the denoising process, when k t-1 =k t From time point t to t-1, x t-1 k t-1 Semantic regions are independently affected by x t k t The semantic region is controlled. Therefore, the simplified constraint condition can be obtained:

[0044]

[0045] Where: I(·) is the indicator function, The output image x t-1 kth t-1 One-Hot vector of semantic regions, is the attribution value calculated based on the Shapley value, which is used to describe the input image x t The semantic region pairs generate the image x t-1 kth t-1 The contribution value of a semantic region is is the attribution function based on the Shapley value, is the game function, which means generating image x t-1 Previous k t-1 The pixel average value of the semantic region;

[0046] In the denoising process from time step t to t-1, for the generated image x t-1 Previous k t-1 The task of semantic region is expected to be t The following properties of the attribution graph: the corresponding k-th t Regions (i.e., k t =k t-1 ) should have high absolute attribution values, while all other regions (i.e., k t ≠k t-1 ) should be as close to zero as possible.

[0047] By introducing independence constraints based on Shapley value attribution, we can ensure independent mapping between input and output semantic regions, avoid interference between regions, clarify the contribution of the input region to the output region, provide a transparent generation process, and improve the semantic clarity and stability of the generated image.

[0048] Specifically, in step 3, the computational complexity is reduced by simplifying the basic interpretability loss function. The specific steps are:

[0049] Step 31, use the sampling method in DDIM to ensure that the generation process is completely determined by the input Gaussian noise, and convert the game function The equivalent setting is to return the kth prediction noise t The average value of semantic regions, that is, ∈ θ (x t , t) in k t The pixel average value of the semantic area, thus the formula of the basic interpretability loss function is proposed as:

[0050]

[0051] The basic interpretability loss function is understood as x t k t constrains the region to affect only x t-1 k t-1 Area (k t-1 =k t ), and not for other regions (k t-1 ≠k t ) have an impact.

[0052] Step 32, for each step of back propagation, calculating the Shapley value of a region requires considering all possible region subsets, the number of which is 2 K , where K is the current noisy image x t The number of regions, so the time complexity of calculating a single Shapley value is O(2 K );

[0053] Step 33, the calculation of the basic interpretability loss function requires x t K semantic regions and x t-1 The K-1 semantic regions are all calculated, and the overall calculation time complexity is O(K(K-1)2 K ), an equivalent loss function is proposed based on time complexity, which reduces the time complexity of accurately calculating the loss function from O(K(K-1)2 K ) is reduced to O(K2 K ), if for any subset, all have v(SU{i})=v({i}) and v(S)=v(φ), then all players in N\{i} are considered to be virtual players, where N is the set of all players, i is the target player, S is the subset, and the formula of the equivalent loss function is:

[0054]

[0055] Where: L IE is the equivalent loss function, E t is the expected value at time step t, φ is an empty set, When the semantic region k t After adding to the subset S, the kth t-1 The incremental contribution of the region, Generate x for the subset S t-1 k t-1 Total contribution of the region.

[0056] The above loss function will force Each player in becomes a virtual player, thereby achieving the goal of building a self-explanatory model proposed by the present invention. However, it is difficult to accurately calculate L IE The time complexity is O(K2 K ). In order to further solve the problem of high computational time complexity, the present invention uses an optimized sampling algorithm to speed up the calculation. By randomly sampling a subset of players instead of considering all possible combinations, the contribution of adding a specific player to the revenue is calculated for each sampled subset, which represents the marginal contribution of the player. The above process is repeated to average the marginal contributions obtained from all samples to estimate the Shapley value. Since the algorithm only needs to process randomly sampled subsets instead of all possible combinations, the time complexity of the algorithm is significantly reduced. By applying the algorithm, the loss function L is calculated. IE The computational complexity is simplified to O(KC), where C represents the number of sampling times.

[0057] Based on this, the following loss function is designed to train interpretable DDPM:

[0058] L=L DDPM +λ0L IE ;

[0059] Where: λ0 is a positive weight, L DDPM is the traditional DDPM loss function, L IE is an equivalent loss function, which is used to ensure that any semantic region on the input Gaussian noise independently generates the corresponding semantic region on the output image.

[0060] By treating the semantic regions in the input Gaussian noise as independent "players" and constraining the independence of the generation process based on Shapley value attribution and Markov characteristics, we can ensure that the input region and the output region correspond one-to-one, avoid interference between regions, clarify the semantic mapping relationship between input and output, enhance the transparency of model behavior, improve the semantic clarity and authenticity of the generated image, support region-level semantic editing and adjustment, and enhance the tolerance of the generation results to input noise disturbances.

[0061] The embodiment of the present invention regards different semantic areas of Gaussian noise in the input as independent "players" and designs corresponding game. In order to enhance the interpretability of the model, a loss function based on Shapley value is designed to enhance the correspondence between the input noise and the semantic area of ​​the output image, and a method for reducing the computational complexity is proposed. The present invention can be applied to a variety of DDPM models, has good generalization ability, can accurately resample the local visual concept of the image, and allows the user to modify a part of the image more accurately without being disturbed by other areas.

[0062] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0063] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An interpretable training method for an unconditional image generation diffusion probability model, characterized in that: The steps include: Step 1: Problem formalization and semantic region division: By performing image sequence representation and semantic region division on the output images during the DDPM denoising process, the first constraint condition of the desired self-explanatory model is preliminarily determined; Step 2, game theory model and Shapley value attribution: treat the semantic regions in the input Gaussian noise as independent "players" and define the game function Used to measure the kth t The benefits of semantic regions; Step 3: Design of interpretable loss function: Reduce the computational complexity by simplifying the basic interpretable loss function.

2. The interpretable training method of the unconditional image generation diffusion probability model according to claim 1, characterized in that: In step 1, the first constraint of the desired self-explanatory model is preliminarily determined by performing image sequence representation and semantic region division on the output image during the DDPM denoising process. The specific steps are as follows: Step 11: The image sequence output during DDPM denoising is expressed as In x0,x1,…,x T The same semantics as x0 is applied to the region division, x T is the initial Gaussian noise, x1 is the Gaussian noise after the first denoising, and x0 is the final denoised image; Step 12, use Represents a one-hot vector of length K, used to indicate the xth t The kth image sequence t semantic regions, and in K is the number of all semantic regions, is the set of all semantic regions, thus we get in, is the One-Hot vector of the k0th semantic region of the output image x0, is the One-Hot vector of the k1th semantic region of the output image x1, The output image x T kth T The One-Hot vector of the semantic region, k0 is the number of the k0th semantic region of the final generated image x0, k1 is the number of the k1th semantic region of the final generated image x1, and k T To finally generate the image x T kth T The number of the semantic region; Step 13, based on the Shapley value attribution perspective, preliminarily determine the first constraint condition of the expected self-explanatory model: Where: I(·) is the indicator function, is the One-Hot vector of the k0th semantic region of the output image x0, is the attribution value calculated based on the Shapley value, is the attribution function based on the Shapley value, is a game function, which represents the average pixel value of the k0th semantic area on x0.

3. The interpretable training method of the unconditional image generation diffusion probability model according to claim 1, characterized in that: In step 2, the semantic regions in the input Gaussian noise are regarded as independent "players" and the game function is defined Used to measure the kth t The specific steps are: Based on the Markov process characteristics of DDPM, the original constraints are transformed into ensuring that at each step of the denoising process, when k t-1 =k t From time point t to t-1, x t-1 k t-1 Semantic regions are independently affected by x t k t The semantic region control is simplified to obtain the following constraints: Where: I(·) is the indicator function, The output image x t-1 kth t-1 One-Hot vector of semantic regions, is the attribution value calculated based on the Shapley value, which is used to describe the input image x t The semantic region pairs generate the image x t-1 kth t-1 The contribution value of a semantic region is is the attribution function based on the Shapley value, is the game function, which means generating image x t-1 Previous k t-1 The pixel average value of the semantic region.

4. The interpretable training method of the unconditional image generation diffusion probability model according to claim 1, characterized in that: In step 3, the computational complexity is reduced by simplifying the basic interpretability loss function. The specific steps are as follows: Step 31, use the sampling method in DDIM to ensure that the generation process is completely determined by the input Gaussian noise, and convert the game function The equivalent setting is to return the kth prediction noise t The average value of semantic regions, that is, ∈ θ (x t ,t) in k t The pixel average value of the semantic area, thus the formula of the basic interpretability loss function is proposed as: The basic interpretability loss function is understood as x t k t constrains the region to affect only x t-1 k t-1 Area (k t-1 =k t ), and not for other regions (k t-1 ≠k t ) have an impact; Step 32, for each step of back propagation, calculating the Shapley value of a region requires considering all possible region subsets, the number of which is 2 K , where K is the current noisy image x t The number of regions, so the time complexity of calculating a single Shapley value is O(2 K ); Step 33, the calculation of the basic interpretability loss function requires x t K semantic regions and x t-1 The K-1 semantic regions of the computation are all calculated, and the overall computational time complexity is O(K(K-1)2 K ), an equivalent loss function is proposed based on time complexity, which reduces the time complexity of accurately calculating the loss function from O(K(K-1)2 K ) is reduced to O(K2 K ), if for any subset, all have v(S∪{i})=v({i}) and v(S)=v(φ), then all players in N\{i} are considered to be virtual players, where N is the set of all players, i is the target player, S is the subset, and the formula of the equivalent loss function is: Where: L IE is the equivalent loss function, E t is the expected value at time step t, φ is an empty set, When the semantic region k t After adding to the subset S, the kth t-1 The incremental contribution of the region, Generate x for the subset S t-1 k t-1 Total contribution of the region.

Images