Training device and training method

Abstract

A training device according to one aspect of the present disclosure comprises: an input unit that inputs unlabeled data representing appropriate or inappropriate data, and labeled data representing inappropriate data; and a training unit that, on the basis of the unlabeled data and the labeled data, trains a diffusion model that does not generate inappropriate data by minimizing an objective function that includes an approximation of the variational lower bound of transition probabilities in the reverse diffusion process of the diffusion model to appropriate data, and an approximation of the variational lower bound to inappropriate data.

Description

Learning device and learning method

[0001] The present disclosure relates to a learning device and a learning method.

[0002] Generative models are known as one type of machine learning model (e.g., Non-Patent Documents 1 and 2), and are used to generate data in various formats such as images, audio, and text. However, depending on the training data used when training a generative model, a problem can arise in that the trained generative model may generate inappropriate data.

[0003] To address the above problem, it is possible to make a trained generative model forget concepts related to inappropriate data by using a technique for forgetting specific concepts from the trained generative model (e.g., Non-Patent Document 3).

[0004] Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022.Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." Advances in neural information processing systems 33 (2020): 6840-6851.Heng, Alvin, and Harold Soh. "Selective amnesia: A continuous learning approach to forgetting in deep generative models." Advances in Neural Information Processing Systems 36 (2024).

[0005] However, in the prior art, it was not possible to prevent the generation of inappropriate data when the inappropriate data was included in the training dataset as unlabeled data.

[0006] The present disclosure has been made in consideration of the above points, and aims to provide a technology that can learn a generative model that does not generate inappropriate data.

[0007] A learning device according to one aspect of the present disclosure includes an input unit that inputs unlabeled data representing appropriate or inappropriate data and labeled data representing inappropriate data, and a learning unit that learns the diffusion model that does not generate inappropriate data by minimizing an objective function that includes an approximation, for appropriate data, of a variational lower bound of a transition probability in a dediffusion process of a diffusion model and an approximation, for inappropriate data, of the variational lower bound, based on the unlabeled data and the labeled data.

[0008] It is possible to train a generative model that does not generate irrelevant data.

[0009] FIG. 1 is a diagram showing an example of a diffusion model. FIG. 2 is a diagram showing an example of the hardware configuration of a learning device according to the present embodiment. FIG. 3 is a diagram showing an example of the functional configuration of a learning device according to the present embodiment. FIG. 4 is a flowchart showing an example of a learning process according to the present embodiment. FIG. 5 is a diagram showing an example of learning data for an experiment. FIG. 6 is a diagram showing an example of data generated by each technique.

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0011] <Background> Generative models, which are a type of machine learning model, are used to generate data in various formats such as images, audio, text, etc. For example, Stable Diffusion (Non-Patent Document 1) is widely used in various fields because it can generate high-quality images using text as input.

[0012] Generative models are generally trained using unlabeled data as training data, but this training data often contains inappropriate data. This can lead to problems such as a trained generative model generating inappropriate data. For example, a trained generative model may generate sexually explicit images of real people. Examples of inappropriate data include, but are not limited to, data that represents content that is morally or socially inappropriate (e.g., data that represents content that is offensive or disgusting to many people, data that represents content that is inappropriate for the healthy education of children and young children, data that represents content that is discriminatory or insulting to a particular race or gender, or other content that violates public order and morals). For example, inappropriate data may also be data that represents inappropriate content in light of the intended use of the data to be generated by the trained generative model (e.g., data that represents content that is suitable for use for purposes other than the intended use).

[0013] Although it is difficult to list all inappropriate data from unlabeled data, it is possible to list a small amount of inappropriate data, for example, by listing a few images of people that you do not want your generative model to learn.

[0014] Therefore, we propose a method (hereinafter also referred to as the "proposed method") for learning a generative model that does not generate inappropriate data (in other words, a generative model that generates only appropriate data) using a large amount of unlabeled data, which may contain inappropriate data, and a small amount of inappropriate data (labeled data).

[0015] The proposed method focuses on a technique called Positive-Unlabeled (PU) learning (Reference 1) and extends it to apply PU learning to a typical generative model, the Diffusion Model. PU learning is a semi-supervised learning method that uses a small amount of irrelevant data (Positive) and unlabeled data (Unlabeled).

[0016] <Diffusion Model> Before describing the details of the proposed method, we will explain DDPM (Denoising Diffusion Probabilistic Models) (Non-Patent Document 2), which is one of the diffusion models. Hereinafter, DDPM is assumed as the diffusion model.

[0017] The diffusion model consists of a diffusion process and a de-diffusion process (a generation process or a noise removal process). An example of the diffusion model is shown in Figure 1. In the example shown in Figure 1, x ｔ－１ From x ｔ The dashed arrow in the direction indicates the diffusion process, and x ｔ From x ｔ－１ The solid arrows in the direction represent the back-diffusion process.

[0018] In the diffusion process, the given data is divided into x ０ As a result, each data x ｔ The noise following the standard Gaussian distribution is sequentially added to the Ｔ At this time, the transition probability at step t (i.e., the amount of noise added at step t) is q(x ｔ |x ｔ－１ ) The noise added at each step of the diffusion process is set in advance, so there are no parameters to be learned in the diffusion process. In the example shown in Figure 1, the total number of steps in the diffusion process is T.

[0019] On the other hand, in the de-diffusion process, data sampled from a standard Gaussian distribution is Ｔ As a result, each data x ｔ The noise is removed (denoised) sequentially from the original data x ０ At this time, the transition probability at step t (i.e., the amount of noise removed at step t) is p θ (x ｔ－１ |x ｔ ) where p θ (x ｔ－１ |x ｔ ) is a probability distribution modeled by a neural network with a learning parameter θ. In the example shown in Figure 1, the total number of steps in the dediffusion process is T, the same as in the diffusion process.

[0020] In the diffusion model, the data x in the dediffusion process ０ The probability p θ (x ０ ) is estimated by the following equations (1) to (3).

[0021] Furthermore, the probability of the data spread during the spreading process is estimated by the following equations (4) and (5).

[0022] Here, 0≦α ｔ ≦1 is a hyperparameter.

[0023] p θ (x ｔ－１ |x ｔ ) is learned by maximizing the Evidence Lower-Bound (ELBO). The ELBO for one data point is derived by the following equation (6):

[0024] By modifying the ELBO equation shown in the above equation (6), it can be simplified as shown in the following equation (7) (Non-Patent Document 2).

[0025] where ε is the true noise, and t is a value sampled from a uniform distribution of integers between 1 and T. θ (x ｔ , t) has a learning parameter θ and data x ｔ It is a neural network that estimates noise from the

[0026] That is, the data x for each step t ｔ The noise added to the neural network ε θ The diffusion model is trained by learning to be able to estimate with high accuracy.

[0027] <Problems in training a diffusion model> Assume that the following unlabeled dataset U, which is composed of a large amount of unlabeled data, and the following improper dataset S, which is composed of a small amount of improper data (labeled data), are given as training datasets.

[0028] Here, N = |U| is the number of unlabeled data, M = |S| is the number of inappropriate data, and N > M. Note that the unlabeled dataset U may include inappropriate unlabeled data.

[0029] In this case, in a typical diffusion model (Non-Patent Document 2), the diffusion model is trained so as to minimize the sum shown in the following equation (8) using the objective function l(x;θ) shown in the above equation (7).

[0030] However, the unlabeled dataset U may contain irrelevant data, and thus a diffusion model that generates irrelevant data may be trained.

[0031] <Proposed Method> Below, we will explain a method for training a diffusion model that does not generate inappropriate data (in other words, a diffusion model that generates only appropriate data) when an unlabeled dataset U that may contain inappropriate data and an inappropriate dataset S are given as training datasets. Note that the proposed method will also be called a Positive-Unlabeled Diffusion Model (PUDM).

[0032] In order to prevent the diffusion model from generating incorrect data, it is necessary to minimize l(x;θ) for correct data x and maximize l(x;θ) for incorrect data x. Following Reference 2, the conditional probability that data x is correct (y=0) or incorrect (y=1) is defined by the following equation (9):

[0033] By this definition, when l(x;θ) is small, p θ When (y = 0 | x) is large and l(x; θ) is large, p θ (y=1|x) becomes large. The conditional probability p θ Using (y|x), binary cross entropy shown in the following equation (10) is introduced.

[0034] As a naive method, we can learn a diffusion model by assuming that all unlabeled data constituting the unlabeled dataset U are appropriate data and minimizing the following equation (11):

[0035] However, since the unlabeled dataset U may contain irrelevant data, there is a problem in that the effect of maximizing l(x; θ) for the irrelevant dataset S is weakened.

[0036] To solve the above problem, we extend PU learning and apply it to learning of a diffusion model. Hereinafter, inappropriate data will be referred to as positive examples (+) and appropriate data as negative examples (-).

[0037] Let p be the probability distribution of the appropriate data. Ｒ (x), the probability distribution of the incorrect data is p Ｓ (x), the probability distribution of unlabeled data is p Ｕ The unlabeled data contained in the unlabeled dataset U is expressed as a probability distribution p Ｕ (x). Similarly, the improper data contained in the improper data set S is sampled from the probability distribution p Ｓ (x) is sampled. Then, the probability distribution of the unlabeled data p Ｕ (x) can be expressed as shown in the following equation (12) using the incidence rate of inappropriate data 0≦β≦1.

[0038] Therefore, the probability distribution of the appropriate data p Ｒ (x) can be expressed as shown in the following equation (13).

[0039] Note that β is a hyperparameter.

[0040] If p Ｒ If you have access to (x) (i.e., p Ｒ (x)), the diffusion model can be trained by minimizing the ideal objective function shown in equation (14) below:

[0041] However, in reality, p Ｒ Since (x) is not accessible, it is necessary to approximate the second term of the above equation (14). The second term of the above equation (14) can be approximated using the above equation (13) as shown in the following equation (15).

[0042] Therefore, the probability distribution of the appropriate data p Ｒ Even if (x) is unknown, the probability distribution of unlabeled data p Ｕ (x) and the probability distribution of inappropriate data p Ｓ (x) and L shown in the above formula (14) ＰＮ (θ) can be approximated as shown in equation (16) below.

[0043] The above equation (16) can be approximated (Monte Carlo approximation) by the Monte Carlo method using the unlabeled data set U and the irrelevant data set S as shown in the following equation (17).

[0044] In the above equation (17), the part excluding β of the first term on the right side is L Ｓ ＋ (θ), the second term is L Ｕ － (θ), the part excluding the third term -β is L Ｓ － (θ).

[0045] In equation (17) above, the sum of the second and third terms on the right hand side is an approximation of l(x;θ) for the appropriate data, that is, the approximation shown in equation (18) below.

[0046] The left side of the approximation shown in the above formula (18) always takes a value greater than or equal to 0, but the right side can take a negative value. Experimentally, the right side often converges to negative infinity, resulting in a meaningless solution. Therefore, similar to the method described in Reference 1, the right side of the approximation shown in the above formula (18) is adjusted so that it does not become negative.

[0047] As described above, in the proposed method, the diffusion model is trained so as to minimize the objective function shown in the following equation (19).

[0048] The objective function shown in the above equation (19) can be optimized by a known gradient method (e.g., stochastic gradient descent (SGD), batch gradient descent, mini-batch gradient descent, etc.).

[0049] The learning device 10 that learns the diffusion model using the proposed method will be described below.

[0050] <Example of Hardware Configuration of Learning Device 10> An example of the hardware configuration of the learning device 10 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the hardware configuration of the learning device 10 according to this embodiment.

[0051] 2, the learning device 10 according to this embodiment includes an input device 101, a display device 102, an external I / F 103, a communication I / F 104, a random access memory (RAM) 105, a read-only memory (ROM) 106, an auxiliary storage device 107, and a processor 108. Each of these pieces of hardware is connected to each other via a bus 109 so as to be able to communicate with each other.

[0052] The input device 101 is, for example, a keyboard, a mouse, a touch panel, physical buttons, etc. The display device 102 is, for example, a display, a display panel, etc. Note that the learning device 10 does not necessarily have to have at least one of the input device 101 and the display device 102, for example.

[0053] The external I / F 103 is an interface with an external device such as a recording medium 103a. Examples of the recording medium 103a include a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0054] The communication I / F 104 is an interface for connecting to a communication network. The RAM 105 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 106 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 107 is a non-volatile storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The processor 108 is a variety of arithmetic devices such as a central processing unit (CPU) or a graphic processing unit (GPU).

[0055] 2 is an example, and the hardware configuration of the learning device 10 is not limited to this. For example, the learning device 10 may have multiple auxiliary storage devices 107 or multiple processors 108, may not have some of the hardware shown in the figure, or may have various hardware other than the hardware shown in the figure.

[0056] <Example of Functional Configuration of Learning Device 10> An example of the functional configuration of the learning device 10 according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the functional configuration of the learning device 10 according to this embodiment.

[0057] 3, the learning device 10 according to this embodiment includes an input unit 201 and a learning unit 202. These units are realized, for example, by processing in which one or more programs installed in the learning device 10 are executed by the processor 108 or the like.

[0058] The learning device 10 according to this embodiment also includes a learning dataset storage unit 203, a hyperparameter storage unit 204, and a parameter storage unit 205. Each of these storage units is realized, for example, by a storage area of ​​the auxiliary storage device 107. However, at least one of these storage units may be realized by a storage area of ​​a storage device (e.g., a storage device provided in a database server) communicatively connected to the learning device 10.

[0059] The input unit 201 inputs the unlabeled dataset U and the inappropriate dataset S as training datasets from the training dataset storage unit 203 .

[0060] The learning unit 202 learns the diffusion model using the unlabeled dataset U and the inappropriate dataset S input by the input unit 201. That is, the learning unit 202 uses the unlabeled dataset U and the inappropriate dataset S input by the input unit 201 to update the learning target parameter θ of the diffusion model using a known gradient method so as to minimize the objective function shown in the above equation (19).

[0061] The learning dataset storage unit 203 stores an unlabeled dataset U and an irrelevant dataset S.

[0062] The hyperparameter storage unit 204 stores α ｔ The hyperparameters such as α and β are stored. ｔ In addition to β and β, for example, the size K of a mini-batch when using a mini-batch gradient descent method can be used.

[0063] The parameter storage unit 205 stores the learning target parameter θ.

[0064] <Learning Process> An example of the learning process according to this embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of the learning process according to this embodiment. Below, as an example, a case where mini-batch gradient descent is used will be described. However, mini-batch gradient descent is just one example, and the same can be applied even when other gradient methods are used by making appropriate necessary modifications.

[0065] The input unit 201 inputs an unlabeled data set U and an irrelevant data set S as training data sets from the training data set storage unit 203 (step S101). Hereinafter, the training data set will be represented as (U, S).

[0066] The input unit 201 receives the hyperparameter α ｔ , β, and K are input from the hyperparameter storage unit 204 (step S102).

[0067] The learning unit 202 generates a mini-batch B=(U Ｂ , S Ｂ ) is sampled (step S103). Ｂ ⊆U, S Ｂ ⊆S, |U Ｂ |+|S Ｂ |=K.

[0068] The learning unit 202 uses the mini-batch B and the learning target parameter θ to Ｓ ＋ (θ), L Ｕ － (θ), L Ｓ － (θ) (step S104). That is, the learning unit 202 calculates U in the above equation (17) by Ｂ , S to S Ｂ By replacing the above with the first term on the right side (excluding β), and the second and third terms (excluding -β), we can obtain L Ｓ ＋ (θ), L Ｕ － (θ), L Ｓ － Calculate (θ).

[0069] The learning unit 202 uses the L calculated in step S104 above. Ｓ ＋ (θ), L Ｕ － (θ), L Ｓ － (θ) is used to calculate the gradient (step S105). Ｕ － (θ)-βL Ｓ － If (θ)≧0, the learning unit 202 θ (βL Ｓ ＋ (θ) + L Ｕ － (θ)-βL Ｓ － (θ)) is calculated. Ｕ － (θ)-βL Ｓ －If (θ)<0, the learning unit 202 θ (-(L Ｕ － (θ)-βL Ｓ － (θ))) or ∇ θ βL Ｓ － Calculate (θ).

[0070] The learning unit 202 uses the gradient calculated in step S105 to update the learning target parameter θ according to a known parameter update equation so as to minimize the objective function shown in equation (19) above (step S106).

[0071] The learning unit 202 determines whether or not to terminate the learning process (step S107). If the learning unit 202 determines to terminate the learning process, it returns to step S103 above; otherwise, it terminates the learning process. Note that the learning unit 202 may determine to terminate the learning process if a predetermined termination condition is met, for example, or may determine not to terminate the learning process if a predetermined termination condition is not met. Examples of the predetermined termination condition include convergence of the learning target parameter θ, convergence of the value of the objective function, the number of repetitions of steps S103 to S106 above exceeding a predetermined number, and the number of epochs exceeding a predetermined value.

[0072] <Comparison with Other Methods> A comparison was made between a method for training a diffusion model by minimizing the sum shown in the above formula (8) (hereinafter referred to as the "existing method"), a method for training a diffusion model by minimizing the above formula (11) (hereinafter referred to as the "naive method"), and the proposed method. To compare these methods, an experiment was conducted using the training data shown in Figure 5. Note that the dataset consisting of unlabeled data (appropriate) and unlabeled data (inappropriate) in Figure 5 is the unlabeled dataset U, and the dataset consisting of labeled data (inappropriate) is the inappropriate dataset S.

[0073] The data generated by the trained diffusion models trained by each method is shown in Figure 6. As shown in Figure 6, the existing method and the naive method generate inappropriate data. On the other hand, the proposed method does not generate inappropriate data, and only appropriate data is generated.

[0074] As described above, it can be seen that the proposed method is able to train the generative model to generate only appropriate data.

[0075] <Related Research> Research related to the proposed method includes Selective Amnesia (Non-Patent Document 3). Selective Amnesia is a method for forgetting specific concepts from a trained generative model. In this method, data to be remembered and data to be forgotten are generated, and by minimizing l(x;θ) for the data to be remembered and maximizing l(x;θ) for the data to be forgotten, only the specific concepts to be forgotten are forgotten.

[0076] Selective Amnesia is similar to a naive approach. As mentioned above, the naive approach (the approach that minimizes the above equation (11)) cannot handle the case where inappropriate data is included in the unlabeled dataset U. In contrast, the proposed approach can train a diffusion model to generate only appropriate data even when inappropriate data is included in the unlabeled dataset U.

[0077] <Modification> The proposed method uses the objective function l(x ０ ; θ) is expressed as the objective function l(x ０ , c; θ), the data x ０ The conditional probability distribution p θ (x ０ It can be easily extended to |c) In other words, it can also be applied to a generative model that generates images from text, for example.

[0078] where ε θ (x ｔ, c, t) has a learning parameter θ and data x at time t ｔ and a neural network that estimates noise from condition c.

[0079] <Summary> As described above, the learning device 10 according to this embodiment calculates the probability distribution of unlabeled data using p Ｕ (x) = βp Ｓ (x) + (1-β)p Ｒ By assuming that the objective function of the diffusion model can be expressed as (x), the unlabeled data and the inappropriate data are used to approximate the objective function of the diffusion model. This makes it possible to train a diffusion model that generates only appropriate data, even when inappropriate data exists as unlabeled data.

[0080] The learning device 10 according to the present embodiment may include a generation unit that generates data using a trained diffusion model trained by the proposed method. A device having a generation unit may be called, for example, a "generation device" or a "data generation device."

[0081] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims.

[0082] [References] Reference 1: Kiryo, Ryuichi, et al. "Positive-unlabeled learning with non-negative risk estimator." Advances in neural information processing systems 30 (2017). Reference 2: Yamanaka, Yuki, et al. "Autoencoding binary classifiers for supervised anomaly detection." PRICAI 2019: Trends in Artificial Intelligence: 16th Pacific Rim International Conference on Artificial Intelligence, Cuvu, Yanuca Island, Fiji, August 26-30, 2019, Proceedings, Part II 16. Springer International Publishing, 2019.

[0083] 10 Learning device 101 Input device 102 Display device 103 External I / F 103a Recording medium 104 Communication I / F 105 RAM 106 ROM 107 Auxiliary storage device 108 Processor 109 Bus 201 Input unit 202 Learning unit 203 Learning dataset storage unit 204 Hyperparameter storage unit 205 Parameter storage unit

Claims

1. A learning device comprising: an input unit that inputs unlabeled data representing appropriate or inappropriate data and labeled data representing inappropriate data; and a learning unit that learns a diffusion model that does not generate inappropriate data by minimizing an objective function that includes an approximation, for appropriate data, of a variational lower bound of transition probabilities in the dediffusion process of a diffusion model, and an approximation, for inappropriate data, of the variational lower bound, based on the unlabeled data and the labeled data.

2. The learning device according to claim 1, wherein the approximation of the variational lower bound for the appropriate data is an approximation of a weighted sum of the variational lower bound for the unlabeled data and the variational lower bound for the labeled data, with the weight being the occurrence rate of inappropriate data.

3. The learning device described in claim 2, wherein the objective function is expressed as the sum of the product of the occurrence rate and an approximation of the variational lower bound for inappropriate data and a function that outputs an approximation of the weighted sum if the value of the approximation of the weighted sum is 0 or greater, and outputs 0 if the approximation of the weighted sum is less than 0.

4. A learning method performed by a computer, comprising: an input procedure for inputting unlabeled data representing appropriate or inappropriate data and labeled data representing inappropriate data; and a learning procedure for learning the diffusion model that does not generate inappropriate data by minimizing an objective function based on the unlabeled data and the labeled data, the objective function including an approximation, for appropriate data, of a variational lower bound of the transition probability in the inverse diffusion process of the diffusion model, and an approximation, for inappropriate data, of the variational lower bound.

Images