Generative model trained with alignment loss, generative model training method, and morphing generation program, device, and method

Abstract

To provide a generation model capable of changing a positional relation of probability distribution corresponding to a prescribed class in a latent space or bringing the same closer to a prescribed positional relation defined at a distance between distributions.SOLUTION: A generation model causes a computer to function input data as an encoding unit which converts the input data to information related to a probability distribution in a set latent space, corresponding to a class to which the input data belongs, a latent variable generation unit which generates a latent variable from the information related to the probability distribution, and a decoding unit which converts the generated latent variable to output data equivalent to the input data. The generation model is trained using a generation loss containing at least an arrangement loss defined using a distance between probability distributions corresponding to classes different from each other and showing "no loss" or "minimum" when a positional relation in latent spaces of probability distribution is a specified or set positional relation.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technology relating to a generative model in the field of machine learning. [Background technology]

[0002] In the field of machine learning, a generative model that can sample pseudo-data from an estimated probability distribution in a latent space under the premise that "data (x) is generated from a latent probability distribution (p(x|z), z: latent variable)" has been attracting attention. This generative model makes it possible to grasp the distribution of data in detail and, for example, sample new training data or generate morphing data.

[0003] Concerning such a generative model that has attracted attention, for example, Non-Patent Document 1 proposes a variational autoencoder (VAE), which is a generative model using an autoencoder. Here, the VAE is configured such that an encoder outputs parameters of a probability distribution in a latent space from input data, and a decoder generates pseudo data using latent variables sampled from the probability distribution parameters.

[0004] With this VAE, for example, it is possible to perform morphing that expresses continuous data transitions between classes by learning a probability distribution in the latent space for each class of given data, and continuously sampling and decoding latent variables across the probability distributions of different classes. Specifically, in Non-Patent Document 1, a VAE is trained using a data set of handwritten digits consisting of nine classes from "0" to "9", and then decoding is performed using latent variables uniformly sampled from the latent space to generate morphed images of digits that transition between classes.

[0005] Non-Patent Document 2 discloses a technique for associating a class, which is a discrete attribute, with a specific dimension in a latent variable of a VAE. This technique makes it possible to handle a class separately from other continuous attributes, and more specifically, it is possible to obtain an output of a desired class by manipulating and decoding the dimension of the latent variable corresponding to the class.

[0006] Furthermore, Non-Patent Document 3 discloses a technique for generating multiple patterns of morphing data between two predetermined classes via different classes by devising the arrangement of the probability distribution of each class in the latent space of a generative model. In this technique, in order to generate such morphing data, the loss function used for training the generative model is designed so that the probability distribution of each class is arranged at equal intervals on a multidimensional torus in the latent space. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] DP Kingma, M. Welling, “Auto-Encoding Variational Bayes” International Conference on Learning Representations (ICLR) 2014, Conference Track Proceedings,<https: / / arXiv.org / abs / 1312.6114> , 2013 [Non-Patent Document 2] Emilien Dupont, “Learning Disentangled Joint Continuous and Discrete Representations”, Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS) 2018, pp.708-718, 2018 [Non-Patent Document 3] Maciej Mikulski, Jaroslaw Duda, “Toroidal AutoEncoder”,<https: / / arXiv.org / abs / 1903.12286> , 2019 Summary of the Invention [Problem to be solved by the invention]

[0008] In the generative model described above, the arrangement of probability distributions for each class in the latent space has a significant impact on the data sampled thereafter, and is therefore a very important design issue.

[0009] However, in the generative model (VAE) disclosed in Non-Patent Document 1 and Non-Patent Document 2, the arrangement of probability distributions for each class in the latent space, i.e., the positional relationship between the probability distributions, is automatically determined within the model. Therefore, a problem occurs in that it is not necessarily possible to generate morphing data that transitions smoothly between two given classes without passing through other classes and without discontinuity in the direction of change.

[0010] For example, when a handwritten digit dataset consisting of three classes, "0", "1", and "2", is encoded into a latent space using the VAE disclosed in Non-Patent Document 1, it is quite possible that the probability distributions of "0", "2", and "1" are arranged in a straight line in this order. In this case, it is almost impossible to obtain morphing data that transitions smoothly between two given classes, "0" and "1", without passing through "2", for example, by moving along a line segment in the latent space.

[0011] On the other hand, in Non-Patent Document 3, the probability distribution of each class of the generative model is arranged at equal intervals on a multidimensional torus in the latent space. This certainly makes it possible to generate multiple patterns of morphing data between two given classes that pass through different classes. However, even so, there is no guarantee that morphing data can be obtained that transitions smoothly between these two classes without passing through other classes and without discontinuity in the direction of change, and the above problem remains.

[0012] Therefore, the present invention aims to provide a generative model that can bring the positional relationship of probability distributions corresponding to predetermined classes in a latent space into or close to a predetermined positional relationship defined by the distance between the distributions, and a training method thereof. It also aims to provide a morphing generation program, device, and method that can generate morphing data that transitions seamlessly between two predetermined classes without passing through other classes. [Means for solving the problem]

[0013] According to the present invention, there is provided a generative model for generating latent variables corresponding to a class to which a plurality of input data belongs, in a set latent space, from each of a plurality of input data belonging to each of a plurality of classes, the generative model comprising: an encoding unit that converts the input data into information related to a probability distribution in the latent space corresponding to the class to which the input data belongs; a latent variable generation unit that generates the latent variables from information related to the probability distribution; A decoding unit that converts the latent variables into output data corresponding to the input data; to make the computer function, The encoding and decoding units are A placement loss defined using a distance between the probability distributions corresponding to different classes, For at least one combination of two classes that is specified or set among the multiple combinations of two classes generated from the multiple classes, when the two probability distributions corresponding to the two classes that make up the combination of the two classes are adjacent to each other in the latent space, the loss is smaller. The model is trained using generative losses that include at least the configuration loss. And, The information on the probability distribution corresponding to the two classes forming the combination of the two classes specified or set, which is output from the encoding unit, is information in which the two different probability distributions corresponding to the information on this probability distribution are adjacent to each other or close to adjacent to each other in the latent space. A generative model is provided, characterized in that

[0015] In addition, as a preferred embodiment of the configuration loss according to the present invention, it is also preferable that the configuration loss is set to a distance that is defined for each combination of a plurality of two-class combinations generated from a plurality of classes, and indicates that there is no loss or that the loss is minimal when the distance between the two probability distributions corresponding to the two classes that make up the combination is equal to each other between the combinations.

[0016] According to the present invention, there is also provided a generation model for generating, from each of a plurality of input data belonging to a plurality of classes, a latent variable corresponding to the class to which the input data belongs in a set latent space, comprising: an encoding unit that converts the input data into information related to a probability distribution in the latent space corresponding to the class to which the input data belongs; a latent variable generation unit that generates the latent variables from information related to the probability distribution; A decoding unit that converts the latent variables into output data corresponding to the input data; to make the computer function, The encoding unit and the decoding unit are defined using the distance between the probability distributions corresponding to the different classes. placement loss In the latent space, It is shown that there is no loss or the loss is minimized when the distance between the probability distributions satisfies a condition related to the distance that should be satisfied when one of the probability distributions corresponding to each of the multiple classes is taken as the starting point and the remaining probability distributions are arranged in a positional relationship in which they are adjacent to each other in sequence and are further away from the starting point in sequence. It is trained using a generative loss that includes at least a configuration loss, The information on the probability distribution corresponding to each of the plurality of classes output from the encoding unit is information in which the distance in the latent space between different probability distributions corresponding to the information on the probability distribution satisfies a condition related to the distance or approaches a value that satisfies the condition related to the distance. A generative model is provided, characterized in that .

[0017] It is also preferable that the placement loss according to the present invention includes a distance constraint term indicating that the loss becomes smaller as the distance between adjacent probability distributions becomes closer to a set value. Furthermore, according to the present invention, there is provided a generation model for generating, from each of a plurality of input data belonging to a plurality of classes, a latent variable corresponding to the class to which the input data belongs in a set latent space, comprising: an encoding unit that converts the input data into information related to a probability distribution in the latent space corresponding to the class to which the input data belongs; a latent variable generation unit that generates the latent variables from information related to the probability distribution; A decoding unit that converts the latent variables into output data corresponding to the input data; to make the computer function, the encoding unit and the decoding unit are trained using a generation loss including at least a configuration loss that is defined using a distance between the probability distributions corresponding to the different classes, the configuration loss including a distance constraint term indicating that there is no loss or that the loss is minimized when the positional relationship of the probability distributions in the latent space is a specified or set positional relationship, and indicating that the loss becomes smaller as the distance between adjacent probability distributions becomes closer to a set value; The information on the probability distributions corresponding to the different classes output from the encoding unit is information in which the positional relationship in the latent space in the different probability distributions corresponding to the information on the probability distributions is the specified or set positional relationship or is close to the specified or set positional relationship. A generative model is provided, characterized in that .

[0018] Furthermore, it is also preferable that the generation loss according to the present invention is set as a linear combination of a reconstruction loss relating to an expected value of the output data when the input data is input, a distance information amount relating to the distance between the probability distribution and a standard distribution relating to the output of an encoding unit to which the input data is input, and the alignment loss.

[0019] According to the present invention, there is also provided a model training method for training a generative model that generates, from each of a plurality of input data belonging to a plurality of classes, a latent variable in a set latent space that corresponds to the class to which the input data belongs, the method comprising the steps of: The generative model above is an encoding unit that converts the input data into information related to a probability distribution in the latent space corresponding to the class to which the input data belongs; a latent variable generation unit that generates the latent variables from information related to the probability distribution; A decoding unit that converts the latent variables into output data corresponding to the input data; This is a model of how computers function. This model training method is as follows: A placement loss defined using a distance between the probability distributions corresponding to different classes, For at least one combination of two classes that is specified or set among the multiple combinations of two classes generated from the multiple classes, when the two probability distributions corresponding to the two classes that make up the combination of the two classes are adjacent to each other in the latent space, the loss is smaller. setting a placement loss indicating that A step of causing the encoding unit and the decoding unit to learn learning parameters using a generation loss including at least the set alignment loss; There is provided a computer-implemented method for training a model, comprising:

[0020] According to the present invention, further comprising: A sampling means for determining a representative latent variable which is a latent variable representing the probability distribution corresponding to each class for each class using the above-mentioned generative model, and sampling a plurality of latent variables corresponding to points on a line segment or curve segment which sequentially connects the representative latent variables relating to the plurality of classes specified or set including their order in accordance with the order; Decode each of the sampled latent variables. D Input the code to the decoder. D a morphing generating means for generating morphing data between a plurality of classes using a set of the plurality of output data output from the unit; to make the computer function 、 The morphing data is data that realizes a morphing in which no classes other than two adjacent classes are involved during the transition between the two adjacent classes in the order. Characterized by A morphing generator is provided.

[0021] According to the present invention, there is also provided a method for producing a medicament for a medicament comprising the steps of: A sampling means for determining a representative latent variable which is a latent variable representing the probability distribution corresponding to each class for each class using the above-mentioned generative model, and sampling a plurality of latent variables corresponding to points on a line segment or curve segment which sequentially connects the representative latent variables relating to the plurality of classes specified or set including their order in accordance with the order; a morphing generating means for inputting each of the sampled latent variables to a decoding unit and generating a set of the output data output from the decoding unit as morphing data between the classes; to make the computer function, The sampling means samples the plurality of latent variables so that the density of the sampled latent variables on the line or curve segment becomes lower at positions closer to the representative latent variable. R A morphing generation program is provided, which is characterized by .

[0022] According to the present invention, further comprising: A sampling means for determining a representative latent variable which is a latent variable representing the probability distribution corresponding to each class for each class using the above-mentioned generative model, and sampling a plurality of latent variables corresponding to points on a line segment or curve segment which sequentially connects the representative latent variables relating to the plurality of classes specified or set including their order in accordance with the order; Decode each of the sampled latent variables. D Input the code to the decoder. D a morphing generating means for generating morphing data between a plurality of classes using a set of the plurality of output data output from the unit; Have death, The morphing data is data that realizes a morphing in which no classes other than two adjacent classes are involved during the transition between the two adjacent classes in the order. Characterized by A morphing generator is provided.

[0023] According to the present invention, furthermore, A step of determining a representative latent variable, which is a latent variable representing the probability distribution corresponding to each class, for each class using the above-described generative model, and sampling a plurality of latent variables corresponding to points on a line segment or curve segment that sequentially connects the representative latent variables related to the plurality of classes specified or set, including their order, in the specified or set order; Decode each of the sampled latent variables. D Input the code to the decoder. D a step of converting a set of the plurality of output data output from the unit into morphing data between the plurality of classes; Have death, The morphing data is data that realizes a morphing in which no classes other than two adjacent classes are involved during the transition between the two adjacent classes in the order. It is characterized by A computer-implemented morphing generation method is provided. Effect of the Invention

[0024] According to the generative model and the training method of the generative model of the present invention, it is possible to make the positional relationship of the probability distributions corresponding to predetermined classes in the latent space equal or close to a predetermined positional relationship defined by the distance between the distributions. Also, according to the morphing generation program, device, and method of the present invention, it is possible to generate morphing data that transitions seamlessly between two predetermined classes without passing through other classes. [Brief description of the drawings]

[0025] [Figure 1] 1 is a schematic diagram and functional block diagram showing an embodiment of a generative model according to the present invention, and an embodiment of a morphing generating device according to the present invention. [Diagram 2] FIG. 1 is a schematic diagram for explaining an embodiment of a placement loss according to the present invention. [Diagram 3] FIG. 11 is a schematic diagram for explaining another embodiment of the arrangement loss according to the present invention. [Figure 4]11 is a graph and an image for explaining a comparative example of a morphing generation method according to the present invention; [Diagram 5] 3A and 3B are graphs and images for explaining a first embodiment of a morphing generation method according to the present invention. [Figure 6] 6A and 6B are graphs and images for explaining a second embodiment of a morphing generation method according to the present invention. [Figure 7] 11 is a graph for explaining a third embodiment of the probability distribution arrangement process according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0027] [Generative model] FIG. 1 is a schematic diagram and a functional block diagram showing an embodiment of a generative model according to the present invention, and an embodiment of a morphing generating device according to the present invention.

[0028] The generative model 1 of this embodiment shown in FIG. 1 is a model that generates, from each of a plurality of input data belonging to a plurality of classes, a “latent variable” (and information related to the “probability distribution” corresponding to the class) in a set “latent space” that corresponds to the class to which the input data belongs.

[0029] Generally speaking, a generative model is a model that outputs (samples) pseudo-data from an estimated "probability distribution" in a "latent space" spanned by "latent variables." In this type of generative model, the basic premise is that data (x) can generally be generated from a latent "probability distribution" (p(x|z), z: latent variable). With this type of generative model, it is possible to grasp the distribution of data in detail and, for example, (a) Sampling new learning data from the currently acquired learning data to resolve the lack of learning data, (b) To generate morphing data for so-called morphing, such as a continuous change in image content from an image of a dog belonging to the class "dog" to an image of a cat belonging to the class "cat" It is also possible to do so.

[0030] 1, three classes, "0", "1" and "2", are set in advance. The generative model 1 of this embodiment takes handwritten digit "0" image data belonging to class "0" as input data, for example, and ultimately outputs handwritten digit "0" pseudo image data (output data), but also generates a latent variable z(0) corresponding to class "0" and a probability distribution parameter pd(0) (parameters defining the "probability distribution" corresponding to class "0") during the process.

[0031] Similarly, the generative model 1 of this embodiment can generate a latent variable z(1) and a probability distribution parameter pd(1) corresponding to class "1" using image data of the handwritten digit "1" as input data, and can generate a latent variable z(2) and a probability distribution parameter pd(2) corresponding to class "2" from image data of the handwritten digit "2."

[0032] To achieve the above-mentioned functions, Generative Model 1 is specifically (A) an encoding unit 11 that converts input data into information (probability distribution parameters) related to a “probability distribution” corresponding to a class to which the input data belongs in a “latent space”; (B) a latent variable generation unit 12 that generates a “latent variable” from information related to this “probability distribution” (probability distribution information); (C) A decoder 13 that converts this "latent variable" into output data (pseudo data) equivalent to the input data. It serves as a model for how a computer functions.

[0033] In other words, the generative model 1 is a model that corresponds to the variational autoencoder (VAE) disclosed in Non-Patent Document 1.

[0034] Here, the arrangement of the "probability distributions" for each class in the "latent space", i.e., the positional relationship between the "probability distributions", is a very important design issue since it has a significant effect on the data sampled thereafter. However, in conventional generative models including VAE, this positional relationship is usually determined automatically or in advance within the model. Therefore, for example, there was a problem that it was not necessarily possible to generate morphing data that transitions smoothly between two specified classes without going through other classes and without discontinuity in the direction of change (for example, moving along a line segment in the "latent space").

[0035] Specifically, in the conventional generative model, it is quite possible that the probability distribution of the three classes "0", "2", and "1" in the example shown in Figure 1 is automatically arranged in a straight line in this order. In this case, it is generally impossible to obtain morphing data that transitions smoothly between the two specified or set classes "0" and "1" without passing through "2", for example, by moving along a straight line in the "latent space".

[0036] In contrast, the generative model 1 has the following characteristics: (D) The model is trained using a generative loss that includes at least a “configuration loss” that is defined using the distance between “probability distributions” corresponding to different classes, and that indicates that there is no or minimal loss when the positional relationship in the “latent space” of the “probability distributions” is a “specified or set positional relationship” defined by the distance.

[0037] That is, in the generative model 1, by performing training using at least the above-mentioned "configuration loss", the positional relationship of the "probability distribution" corresponding to a predetermined class in the "latent space" can be made to match or approach a predetermined "specified or set positional relationship" defined by the distance between the distributions. This also makes it possible to realize a desired positional relationship in the probability distribution and increase the degree of freedom in sampling of sampled data such as new training data and morphing data.

[0038] Here, as a preferred embodiment of the "placement loss", consider the case where the above-mentioned "specified or set positional relationship" is "a positional relationship in which two predetermined probability distributions are adjacent". In this case, the "placement loss" is set to indicate that the loss is smaller when, for at least one specified or set two-class combination (e.g., a combination of "0" and "1") among a plurality of two-class combinations (a total of 3C2 combinations in FIG. 1) generated from a plurality of classes (three classes in FIG. 1), the two "probability distributions" (e.g., probability distribution "0" and probability distribution "1") corresponding to the two classes forming the combination are adjacent to each other in a positional relationship. Here, "two "probability distributions" being "adjacent" refers to a state in which there is no other "probability distribution" having a "center" such that the distance to a line segment connecting the centers of the two "probability distributions" (the length of a perpendicular line to the line segment) is less than a predetermined threshold.

[0039] This makes it possible to obtain morphing data that transitions smoothly and uninterruptedly between two specified or set classes, for example "0" and "1," without passing through "2," for example by moving along a straight line in the "latent space."

[0040] Incidentally, the uses of the "latent variables" and probability distribution information (information related to "probability distribution") generated by the generative model 1 are not limited to the generation of new learning data or morphed data as described above. For example, it is possible to generate intermediate data having intermediate properties, characteristics, and attributes between two different pieces of data, such as face image data of a husband and face image data of a wife, face image data of a person A when angry and face image data of a person A when laughing, or voice data of a friend B and voice data of a friend C. For example, it is possible to create a "mixed face app" or a "mixed voice app" using the generative model 1.

[0041] [Model configuration] The configuration of the generative model 1 of this embodiment will be described in more detail below. In FIG. 1, the generative model 1 is realized by executing a program installed in a computer, and includes the following functional components: (A) Encoding unit 11, (B) Latent variable generating unit 12, and (C) Decoding unit 13. It has.

[0042] Among these, the encoding unit 11 encodes the input data into a probability distribution in a set low-dimensional (e.g., two-dimensional or three-dimensional) latent space, which in this embodiment is a normal distribution according to the method described in Non-Patent Document 1. Specifically, input data belonging to a certain class (e.g., image data of handwritten digit "0" belonging to class "0") is input, and probability distribution parameters (μ, σ) related to the probability distribution (normal distribution) corresponding to the class to which the input data belongs in the latent space (e.g., class "0") are output. Here, the mean μ and standard deviation σ take vector values ​​according to the number of dimensions of the latent space (latent variables).

[0043] Such an encoding unit 11 can be realized by a neural network algorithm well known in the field of machine learning. For example, the encoding unit 11 may be configured using a multi-layer perceptron (MLP) that receives vector data of a predetermined number of dimensions as input, or a convolutional neural network (CNN) that can receive image data as input.

[0044] The latent variable generation unit 12, also shown in FIG. 1, generates a latent variable related to the class (e.g., class “0”), using the probability distribution parameters (μ, σ) of the class to which the input data belongs (e.g., class “0”), output from the encoding unit 11.

[0045] Here, the latent variable generation unit 12 of this embodiment generates latent variables using a predetermined noise in order to obtain a latent variable that represents a class (class "0") and is slightly fluctuated for learning. Specifically, in accordance with the method described in Non-Patent Document 1, ε is a random number (ε ∼ N(0,I)) that follows a standard normal distribution, and the following equation is used: (1) z=μ+ε*σ (* is the element-wise multiplication operator) A latent variable z (for example, a latent variable z(0) relating to class “0”) may be generated by:

[0046] 1, the decoding unit 13 converts the latent variable z (e.g., z(0)) output from the latent variable generating unit 12 into output data equivalent to the input data (e.g., of the same dimension and structure), that is, pseudo data (e.g., image data of the number "0" belonging to class "0"). This encoding unit may also be embodied by a neural network algorithm, and specifically, it can be configured using MLP or transposed CNN.

[0047] As described above, the generative model 1 that performs a series of encoding processes, latent variable generation (reparameterization) processes, and decoding processes is a model that corresponds to the VAE disclosed in Non-Patent Document 1. That is, in this embodiment, a lower-dimensional (dimensionally compressed) latent variable z is generated from input data, and further, information related to a (lower-dimensional) probability distribution corresponding to the class of the input data can be generated.

[0048] Below, an embodiment will be described in which such a generative model 1 is trained to become a learned model, and then morphing data is generated using this learned generative model 1.

[0049] [Morphing generation device, model training method, morphing generation program, morphing generation method] The morphing generation device 9 of this embodiment shown in Figure 1 is equipped with the generative model 1 described above, trains this generative model 1, and further generates and outputs morphing data using the trained (learned) generative model 1.

[0050] Specifically, as shown in the functional block diagram of Fig. 1, the morphing generation device 9 has an input unit 91, an output unit 95, and a processor / memory. Here, the processor / memory stores one embodiment of a morphing generation program according to the present invention, has a computer function, and executes this morphing generation program to perform a morphing generation process.

[0051] For this reason, the morphing generation device 9 may be a dedicated device for morphing generation, but it can also be a cloud server, a non-cloud type server, a personal computer, a notebook type or tablet type computer, or even a smartphone or various wearable terminals equipped with an embodiment of the morphing generation program according to the present invention.

[0052] The processor memory also has, as functional components, a training unit 92 having a placement loss calculation unit 92a and a generation loss calculation unit 92b, a sampling unit 93, and a morphing generation unit 94. These functional components can be considered as functions realized by executing a morphing generation program according to the present invention stored in the processor memory. The process flow shown by connecting the functional components of the morphing generation device 9 in the functional block diagram of Figure 1 with arrows can also be understood as one embodiment of a morphing generation method according to the present invention.

[0053] Also in the functional block diagram of FIG. 1, an input unit 91 has a communication function and acquires learning data for training the generative model 1, for example, a handwritten digit data set belonging to each of the three classes "0", "1", and "2", from an external learning data management and distribution server, and outputs the data to a training unit 92. Meanwhile, when generating morphing data, the input unit 91 receives, for example, a plurality of input data corresponding to a plurality of classes (for example, classes "0", "1", and "2") between which morphing should be performed, as user input, and outputs the data to a sampling unit 93.

[0054] The training unit 92 trains the generative model 1 using the received learning data (e.g., a handwritten digit data set of classes “0”, “1”, and “2”). Specifically, the training unit 92: (a) A configuration loss calculation unit 92a calculates a "configuration loss" that is defined using a "distance" between probability distributions corresponding to different classes, and indicates that there is no or minimal loss when the positional relationship of these probability distributions in a latent space is a specified or set positional relationship, for each input of learning data and the corresponding output; (b) In the generation loss calculation unit 92b, for each input of learning data and each output associated therewith, a "generation loss" including the above-mentioned "placement loss" (or a "generation loss" which is the "placement loss" itself) is calculated; (c) Using the calculated “generative loss”, we train a generative model 1 using the well-known backpropagation method.

[0055] Here, the "distance" between the two probability distributions in (a) above may be the spatial distance between the means (centers) of the respective probability distributions, or it may be the well-known KL divergence (Kullback-Leibler divergence) between the probability distributions. In general, the KL divergence D between probability distributions P and Q is KL (P||Q) is expressed by the following equation, where H(P,Q) is the cross entropy and H(P) is the information entropy. (2) D KL (P||Q)=H(P,Q)-H(P) =E x~P [log(p(x) / Q(x))] : The expected value of log(p(x) / Q(x)) when x is distributed over P =∫ -∞ +∞ p(x) × log(P(x) / Q(x)) It is a non-negative quantity defined by: and the closer the probability distribution Q is to the probability distribution P (the better it approximates the probability distribution P), the smaller its value is.

[0056] Furthermore, although specific examples of the "placement loss" will be described in detail later using Figures 2 and 3, in this embodiment, for at least one specified or set two-class combination (e.g., a combination of "0" and "1") among multiple two-class combinations (a total of 3C2 combinations in Figure 1) generated from multiple classes (three classes in Figure 1), the "placement loss" is set to indicate that the loss is smaller when the two "probability distributions" (e.g., probability distribution "0" and probability distribution "1") corresponding to the two classes that make up the combination are adjacent to each other.

[0057] As described above, two "probability distributions" being "adjacent" refers to a state in which there is no other "probability distribution" having a "center" such that the distance to a line segment connecting the centers of the two "probability distributions" (the length of a perpendicular line drawn to the line segment) is less than a predetermined threshold.

[0058] Furthermore, the "generation loss" in (b) above may be the "arrangement loss" itself as described above, but in this embodiment, it is expressed by the following formula (3) (Generation loss) = α {-(variation lower limit)} + β (placement loss) =α·{(KL information amount)+(reconstruction loss)}+β·(placement loss) α,β: weights set appropriately That is, in this embodiment, the "generation loss" is set as a linear combination of the "KL divergence", the "reconstruction loss", and the "placement loss".

[0059] Here, the "variational lower bound" (=-(KL divergence)-(reconstruction loss)) in the above formula (3) is the "variational lower bound" defined in Non-Patent Document 1, which will be described in detail later. Furthermore, the "KL divergence" is a distance divergence related to the distance (KL divergence) between the probability distribution related to the output of the encoding unit 11 with the input data as input and the standard distribution (standard normal distribution). Furthermore, the "reconstruction loss" is a loss function well known in the field of machine learning, and is a quantity related to the expected value of the output data (pseudo data) when the input data is input. For example, when the input data is a vector consisting of vector elements having values ​​from 0 to 1, the "reconstruction loss" can be the cross entropy between the input data and the output data (pseudo data).

[0060] In any case, the training unit 92 uses the probability distribution parameters output by the encoding unit 11 and the output data (pseudo data) output by the decoding unit 13 to calculate the “generation loss” based on the above formula (3), and φ (·)) and the decoding unit 13 (D θ (·)), the learning parameters (φ,θ) are learned so that this "generation loss" is minimized. Specifically, for example, the well-known backpropagation method is used to calculate the error gradient for the learning parameters (φ,θ), and the learning parameters (φ,θ) are repeatedly updated based on an appropriate learning rate, thereby obtaining the learning parameters (φ,θ) that minimize the "generation loss" in equation (3) above.

[0061] The “variational lower bound” (=−(KL divergence)−(reconstruction loss)) described in Non-Patent Document 1 will be explained below.

[0062] First, the latent variable z is given by a standard normal distribution N(0,I) with a (hence easy to compute) probability density function p θ (z), and the data x is also normally distributed N(μ D ,σ D 2 I) the (also easy to compute) probability density function p θ (x|z). On the other hand, the probability density function p θ Since (z|x) is not easily calculated, we use the normal distribution N(μ E ,σ E 2 I) the (also easy to compute) probability density function q φ Consider approximating by (z|x).

[0063] Here, the normal distribution N(μ D ,σ D 2 Regarding I), the trained neural network constituting the decoding unit 13 is denoted as D θ (·) (θ is a learning parameter), then (4) D θ (z)=(μ D ,σ D 2 ) holds, and the above normal distribution N(μ E ,σ E 2 Regarding I), the trained neural network constituting the encoding unit 11 is denoted as E φ (·) (φ is a learning parameter), then (5) E φ (x)=(μ E ,σ E 2 ) That is, the learning parameter θ of the neural network of the decoding unit 13 and the learning parameter φ of the neural network of the encoding unit 11 should be trained so that the above expressions (4) and (5) hold.

[0064] Below, the above mentioned probability density function q φ (z|x) is the probability density function p θ As a measure of how well (z|x) is approximated, the log-likelihood function log(p θ Find the variational lower bound as the lower limit of (x).

[0065] Specifically, the variational lower bound L(θ,φ,x) is expressed as follows: (6) L(θ,φ,x)=log(p θ (x))-D KL (q φ (z|x)||p θ (z|x) =log(p θ (x))-D KL (q φ (z|x)||p θ (z) +E qφ(z|x) [p θ (x|z)]-log(p θ (x) =-D KL (q φ (z|x)||p θ (z))+E qφ(z|x) [p θ (x|z)] Here, the first term in the last equation of (6) above (D KL (q φ (z|x)||p θ (z))) is the above-mentioned "KL divergence", specifically, q φ This is equivalent to the KL divergence between (z|x) and the standard normal distribution. In addition, the second term E qφ(z|x) [p θ (x|z)] is the amount obtained by multiplying the above-mentioned “reconstruction loss” by -1. In other words, the above equation (6) can be expressed as follows: (6') (variational lower bound)=-(KL information amount)-(reconstruction loss) It can be expressed as follows.

[0066] Incidentally, in Non-Patent Document 1, in order to facilitate the actual calculation, the SGVB (Stochastic Gradient Variational Bayes) estimator L'(θ,φ,x, (ε (l) ) l=1,..,L ) and apply the latent variable z(=μ+ε*σ) in the above equation (1), ε (l) Let (l=1,...,L (L is a hyperparameter)) be a random number independently extracted from ε, and (7) L'(θ,φ,x,(ε (l) ) l=1,..,L ) =-(K / 2) log2π -(1 / L) Σ l=1 L Σ k=1 K (((x k -μ D,k (l) ) 2 / (σ D,k (l) ) 2 )-logσ D,k (l) ) -(1 / 2) ·Σ j=1 J (1-μ E,j 2 -σ E,j 2 +logσ E,j 2 ) The "variational lower bound" is calculated as follows.

[0067] In the above formula (7), K is the data length of x, k , μ D,k (l) and σ D,k (l) is x, μ D (l) and σ D (l) This is the kth component of (μ D (l) ,(σ D (l) ) 2 )=D θ (μ E +σ E *ε(l) ) where J is the data length of z, and μ E,j and σ E,j is μ E and σ E is the j-th component of

[0068] The above describes the "variational lower bound" (=-(KL divergence)-(reconstruction loss)), which is a well-known part of the "generation loss". Below, we will provide a detailed description of the "placement loss", which is a feature of the present invention.

[0069] FIG. 2 is a schematic diagram for explaining an embodiment of the arrangement loss according to the present invention.

[0070] The alignment loss L_align(k) of this embodiment, which will be described below with reference to Figs. 2(A) to 2(C), is calculated by dividing the combinations of two classes ( k The "distance" d is set to a value that indicates that there is no loss or that the loss is minimized when the "distance" d between the two probability distributions corresponding to the two classes that make up the combination is equal between the combinations. This makes it possible to realize in the latent space a positional relationship in which multiple probability distributions (k probability distributions) corresponding to multiple classes (k classes) are all "adjacent."

[0071] Incidentally, as already mentioned, the “distance” d between the above probability distributions may be the spatial distance between the mean values ​​(centers) of each probability distribution, or the KL divergence between the probability distributions may be adopted.

[0072] First, as shown in Figure 2(A), consider the alignment loss L_align(3) for realizing a positional relationship in which the probability distributions corresponding to three classes (k=3), class 0, class 1, and class 2, are all adjacent to each other. This positional relationship in which the three probability distributions are all adjacent to each other can be realized by setting a two-dimensional latent space and satisfying the condition that the mutual distances d in all the probability distributions are equal. In other words, this can be realized by making the centers of the three probability distributions respectively located at the three vertices of an equilateral triangle.

[0073] Here, the distance between class a and class b is d ab Then, the above condition is expressed as follows: (8) d 01 =d 02 , and d 01 =d 12 Therefore, the alignment loss L_align(3) is expressed by the following equation: (9) L_align(3)=(d 01 -d 02 ) 2 +(d 01 -d 12 ) 2 It can be set as follows.

[0074] Next, as shown in Figure 2(B), consider the alignment loss L_align(4) for realizing a positional relationship in which the probability distributions corresponding to the four classes (k=4) from class 0 to 3 are all adjacent to each other. This positional relationship in which all four probability distributions are adjacent to each other can be realized by setting a three-dimensional latent space and satisfying the condition that the mutual distances d in all probability distributions are equal. In other words, this can be realized by making the centers of the four probability distributions each located at the four vertices of a regular tetrahedron.

[0075] Here, the above condition is expressed by the above formula (8) and the following formula: (10) d 01 =d 03 ,d 01 =d13 , and d 01 =d 23 Therefore, the alignment loss L_align(4) is expressed by the following equation: (11) L_align(4)=L_align(3)+(d 01 -d 03 ) 2 +(d 01 -d 13 ) 2 +(d 01 -d 23 ) 2 It can be set as follows.

[0076] Furthermore, as shown in FIG. 2(C), consider the alignment loss L_align(5) for realizing a positional relationship in which the probability distributions corresponding to the five classes (k=5) from class 0 to 4 are all adjacent to each other. This positional relationship in which all five probability distributions are adjacent to each other can be realized by setting a four-dimensional latent space and satisfying the condition that the mutual distances d in all probability distributions are equal. In other words, it can be realized by making the centers of the five probability distributions each located at the five vertices of a regular pentagon (a five-dimensional regular simplex). Incidentally, a regular pentagon is a regular polyhedron whose cells (three-dimensional surfaces) are regular tetrahedrons in three-dimensional space.

[0077] Here, the above conditions are the above formula (8), the above formula (10), and the following formula: (12) d 01 =d 04 ,d 01 =d 14 ,d 01 =d 24 , and d 01 =d 34 Therefore, the alignment loss L_align(5) is expressed by the following equation: (13) L_align(5) = L_align(4) +(d 01 -d 04 ) 2 +(d 01 -d14 ) 2 +(d 01 -d 24 ) 2 +(d 01 -d 34 ) 2 It can be set as follows.

[0078] As above, the alignment loss was set for k=3, k=4, and k=5. By inductively generalizing these results for k (≧3), the alignment loss L_align(k) in this embodiment is expressed by the following formula: (14) When k=3, L_align(3)=(d 01 -d 02 ) 2 +(d 01 -d 12 ) 2 When k≧4, L_align(k)=L_align(k-1)+Σ m=0 k-2 (d 01 -d m(k-1) ) 2 It can be set as follows.

[0079] In this way, the placement loss in this embodiment is a function that places the centers of the probability distributions at k vertices of a (k-1)-dimensional regular polytope (regular simplex) so that the probability distributions of k (≧3) classes are all adjacent to each other, thereby making the distances d between all the probability distributions equal.

[0080] Of course, the setting of the arrangement loss in this embodiment is not limited to the above formula (14). For example, when k=3, (d 01 -d 02 ) and (d 01 -d 12 ), and for k ≥ 4, (d 01 -d m(k-1) )(m=0~(k-2)) appears (d 01 -**), any quantity that is a monotonically increasing function can be used for the placement loss.

[0081] FIG. 3 is a schematic diagram for explaining another embodiment of the arrangement loss according to the present invention.

[0082] The alignment loss L_align(k) of this embodiment, which will be described below with reference to Figures 3(A) and (B), is set to indicate that there is no loss or that the loss is minimal when the distance d between the probability distributions satisfies a condition related to the distance d that should be satisfied when one of a plurality of probability distributions (k probability distributions) corresponding to a plurality of classes (k classes, k is an integer equal to or greater than 3) is taken as the starting point and the remaining probability distributions are arranged in a positional relationship in which they are adjacent to each other and progressively further away from the starting point (in this embodiment, in a linear manner).

[0083] This makes it possible to realize in the latent space a positional relationship in which multiple probability distributions (k probability distributions) corresponding to multiple classes (k classes) are arranged in a straight line (or a curve with a sufficiently small curvature), such that each distribution is "adjacent" only to the previous and / or next probability distribution.

[0084] First, as shown in Fig. 3(A), consider the alignment loss L_align(3) for realizing a positional relationship in which the probability distributions corresponding to three classes (k=3), class 0, class 1, and class 2, are arranged in a straight line in this order and are adjacent to each other in sequence (the following explanation will be the same even if the classes are in a different order). This positional relationship can be realized by setting a two-dimensional latent space, assuming that the trigonometric inequality of Euclidean geometry also holds in this two-dimensional latent space, and making the mutual distance d in the probability distributions satisfy the condition of the trigonometric inequality.

[0085] Here, the order of the probability distributions is l1 → l2 → l3, that is, l1 = (probability distribution of class 0), l2 = (probability distribution of class 1), and l3 = (probability distribution of class 2). Let d be the distance between the i-th (li) probability distribution and the j-th (lj) probability distribution. li,lj Then, the condition of the above trigonometric inequality is as follows: (15) d l1,l2 +d l2,l3 ≧ d l1,l3 When these three probability distributions are aligned in a straight line, both sides of the above equation (15) are equal. Therefore, the alignment loss L_align(3) is expressed by the following equation: (16) L_align(3)=(d l1,l2 +d l2,l3 )-d l1,l3 It is possible to set it like this.

[0086] Next, as shown in FIG. 3B, we consider the alignment loss L_align(4) for realizing a positional relationship in which the probability distributions corresponding to four classes (k=4), class 0, class 1, class 3, and class 2, are arranged in a straight line in this order and adjacent to each other in sequence (the following explanation will be the same even if these classes are arranged in a different order).

[0087] Here, if the order of the probability distributions is l1 → l2 → l3 → l4, that is, l1 = (probability distribution of class 0), l2 = (probability distribution of class 1), l3 = (probability distribution of class 3), and l4 = (probability distribution of class 2), then, after setting up a two-dimensional latent space, such a positional relationship can be expressed as follows: (a) The distance d between the first, second, and final probability distributions in the ordering satisfies the triangle inequality, and (b) The distance d between the second, (next) third, and (last) fourth probability distributions in the ordering also satisfies the triangle inequality. This can be realized by doing so.

[0088] Specifically, the condition of the above triangle inequality is as follows: (17) d l1,l2 +d l2,l4 ≧ d l1,l4 (18) d l2,l3 +d l3,l4 ≧ d l2,l4 When these four probability distributions are aligned in a straight line, both sides of the above equations (17) and (18) are equal. Here, d l1,l2 By adding the above equation (17), we obtain the following equation: (19) d l1,l2 +d l2,l3 +d l3,l4 ≧ d l1,l2 +d l2,l4 ≧ d l1,l4 , i.e. (20) d l1,l2 +d l2,l3 +d l3,l4 ≧ d l1,l4 In the above equation (20), both sides are equal when the four probability distributions are aligned in a straight line, so the alignment loss L_align(4) is given by the following equation: (21) L_align(4)=(d l1,l2 +d l2,l3 +d l3,l4 )-d l1,l4 It can be set as follows.

[0089] As above, the alignment loss for k=3 and k=4 has been set. By inductively generalizing these results for k (≧3), the alignment loss L_align(k) in this embodiment can be calculated by the following formula: (22) L_align(k)=Σ m=1 k-1 (d lm,l(m+1) )-d l1,lk It can be set as follows.

[0090] Of course, the setting of the alignment loss in this embodiment is not limited to the above formula (22). For example, in order to realize a positional relationship in which the probability distributions to be aligned are not aligned in a straight line but aligned in a curve with a sufficiently small curvature and are "adjacent" only to the previous and / or next probability distribution, the alignment loss L_align(k) can be set to the following formula: (22') L_align(k)=Σ m=1 k-1 (d lm,l(m+1) )-D·d l1,lk Here, D is a coefficient that is slightly greater than 1. In this way, any amount that induces the realization of a positional relationship in which one of the multiple probability distributions is the starting point and the remaining probability distributions are arranged adjacent to each other and further away from the starting point can be used as the placement loss.

[0091] Hereinafter, still another embodiment for setting the placement loss will be described.

[0092] In general, as the number of dimensions of the latent space increases, the possibility of gaps occurring between the probability distributions for each class placed in the latent space increases. D Since unit 13 (FIG. 1) is not trained to generate output data (pseudo data) from latent variables located in such gaps, even if such latent variables are sampled, their use is limited; for example, they are not suitable for normal smooth morphing.

[0093] Therefore, the alignment loss in this embodiment includes a "distance constraint term" indicating that the closer the distance between adjacent probability distributions is to a set value, the smaller the loss. For example, the right side of the above formula (14), the right side of the above formula (22), or the right side of the above formula (22') can be added with a distance constraint term L_distance(k) to be described later, and this can be used as the alignment loss L_align(k). This makes it possible to arrange the probability distributions for each class so that no gaps or almost no gaps occur between the probability distributions, even in a high-dimensional latent space.

[0094] Specifically, let us consider the distance constraint term L_distance(k) to be added to the placement loss (Equation (14) above) for realizing a positional relationship in which all probability distributions for each class are adjacent to each other, as explained with reference to FIG. 2. In this case, it is sufficient to make the distance d between any two probability distributions equal or close to a predetermined set distance δ that is not too large, so the following equation is used: (23) When k=3, L_distance(3)=(d 01 -δ)2 +Σ m=0 1 (d m2 -δ) 2 When k ≧ 4, L_distance(k) = L_distance(k-1) + Σ m=0 k-2 (d m(k-1) -δ) 2 A distance constraint term L_distance(k) expressed as:

[0095] Of course, the setting of the distance constraint term in this case is not limited to the above equation (23). For example, when k=3, (d 01 -δ), (d 02 -δ) and (d 12 -δ) and for k ≥ 4, (d m(k-1) If the quantity is a monotonically increasing function of all (**-δ) that appear, including (**-δ) (m=0~(k-2)), it can be used as a distance constraint term.

[0096] Next, consider the distance constraint term L_distance(k) to be added to the arrangement loss (the above formula (22) or (22')) for realizing a positional relationship in which the probability distributions for each class are arranged in a straight line (or curve) and adjacent to each other, as explained with reference to FIG. 3. In this case, it is sufficient that the distance d between two adjacent probability distributions is equal to or close to a predetermined set distance δ that is not too large, so the following formula is used. (24) L_distance(k)=Σ m=0 k-1 (d lm,l(m+1) -δ) 2 A distance constraint term L_distance(k) expressed as:

[0097] Of course, the setting of the distance constraint term in this case is not limited to the above formula (24). For example, (d lm,l(m+1) Any quantity that is a monotonically increasing function of m = 0 to (k-1) can be used as a distance constraint term.

[0098] Although various possible configuration losses including the configuration loss with the distance constraint term added thereto have been described above, the configuration loss is not limited to these. For example, it is possible to set a configuration loss that takes a minimum value when the distance d between only two of the probability distributions of many classes is within a predetermined range, or a configuration loss that directs the probability distributions of three predetermined classes to be located at the vertices of a triangle of a predetermined shape (taking a minimum value when the distance d between the vertices satisfies the condition that the distance between the vertices satisfies).

[0099] However, the arrangement loss that induces predetermined probability distributions to be "adjacent" as described above is important in generating morphing data that smoothly transitions, for example, between two predetermined classes without interruption without passing through other classes. The generation of such morphing data will be described in detail later as Examples 1 and 2 with reference to Figs. 5 and 6.

[0100] Returning to the functional block diagram of FIG. 1, the sampling unit 93 inputs each of a plurality of input data belonging to a plurality of classes (e.g., classes “0”, “1”, and “2”) between which morphing should be performed, to the learned generative model 1 trained by the training unit 92, (a) for each of a plurality of classes (classes “0”, “1”, and “2”), a representative latent variable is determined which is a latent variable representing a probability distribution corresponding to the class (arranged in a suitable positional relationship in the set latent space); (b) Representative latent variables relating to multiple classes (e.g., classes “0”, “1”, and “2”) that have been specified or set, including their order (e.g., “0” → “1” → “2”), are sequentially connected in that order (e.g., “0” → “1” → “2”), and multiple latent variables corresponding to points on the line or curve are sampled from the line or curve.

[0101] Here, the representative latent variables z_r in (a) above may be specified by the user in any number from each class, and may be determined by z_r=μ+p*σ ​​according to the above formula (1), using the probability distribution parameters (μ,σ) of the probability distribution corresponding to the class (output from the trained encoding unit 11). Here, p is a parameter specified by the user. Alternatively, at least one representative latent variable z_r may be automatically determined by setting p to, for example, a random number (p to N(0,I)). It is also preferable that such a representative latent variable z_r includes the mean μ of the probability distribution corresponding to the class. In any case, when generating morphing data that transitions between classes, one representative latent variable is determined (selected) for each class.

[0102] In addition, the multiple latent variables corresponding to points on the line or curve segment connecting the representative latent variables in (b) above may be latent variables corresponding to a set number of points arranged at intervals of a predetermined internal division on the line segment, or latent variables corresponding to a set number of points arranged at intervals of a predetermined internal division on the curve segment, calculated from the curve segment using a known interpolation method such as spline interpolation.

[0103] Furthermore, it is also preferable that the sampling unit 93 samples a plurality of latent variables so that the density of the sampled latent variables on the line segment or curve segment is lower as the position is closer to the representative latent variable, rather than sampling points (latent variables) on the line segment or curve segment at equal intervals as described above. In other words, it is also preferable to densely sample latent variables in an area close to the class boundary, and sparsely sample latent variables at other positions, particularly in the vicinity of the representative latent variable. For example, the intervals between the internal divisions of the line segment or curve segment may be set to be larger as the position is closer to the representative latent variable, and latent variables corresponding to points arranged at uneven internal division intervals may be sampled.

[0104] By sampling the latent variables non-uniformly in this way, it is possible to eliminate or improve the non-uniformity of the degree of change in morphing that occurs when the latent variables are sampled at equal intervals. In fact, in morphing when the latent variables are sampled at equal intervals, the change is small near the representative latent variable, but often increases rapidly as the class boundary is approached.

[0105] Incidentally, Examples 1 and 2 of the morphing generation method according to the present invention will be explained later with reference to FIGS. 5 and 6, where a specific example of sampling of latent variables in the sampling unit 93 will also be explained.

[0106] Also in the functional block diagram of FIG. 1, a morphing generation unit 94 decodes each of the multiple latent variables sampled by the sampling unit 93. D Enter this in section 13 and D A set of a plurality of output data output from the unit 13 is obtained as morphing data between a plurality of classes. This morphing data may be transmitted from the output unit 95 having a communication function to an external information processing device, for example, and used by this information processing device. The output unit 95 may also have a display, and may receive this morphing data and display the morphing (for example, the transition of images between classes) on this display.

[0107] Hereinafter, with reference to Figures 4 to 7, Examples 1 and 2 of the morphing generation method according to the present invention, Example 3 of the arrangement processing of the probability distribution according to the present invention, and Comparative Examples will be described. Here, Examples 1 to 3 and Comparative Example were implemented using the generative model 1 shown in Figure 1 (however, the Comparative Example does not take into account "arrangement loss"). In addition, both the encoding unit 11 and the decoding unit 13 are configured with a three-layer MLP (activation function is softplus, number of dimensions is 400), and the latent space (latent variables) was set to two dimensions.

[0108] Incidentally, Figures 4(A), 5(A), 6(A) and 7(A)-(C) shown later are scatter plots in which the mean μ of the probability distribution parameters (μ, σ) output from the trained encoding unit 11 (Figure 1) using the corresponding handwritten digit image data as input is plotted on a two-dimensional plane (two-dimensional latent space).

[0109] As the learning data for training, handwritten digit image data of classes "0", "1" and "2" obtained from the MNIST (Modified National Institute of Standards and Technology) database was used. Furthermore, the generation loss used for training in the comparative example was the same as in the conventional method. (Generation loss) = (KL information amount) + (Reconstruction loss) On the other hand, the generating function used for training in Examples 1 to 3 was the same as the above formula (3): (Generation loss) = α {(KL information amount) + (Reconstruction loss)} + β (Placement loss) Here, α was set to 1. β was set to 30.0 in Example 1, 30.0 in Example 2, and multiple positive values ​​up to 300.0 in Example 3. Furthermore, the distance d in the "configuration loss" was the spatial distance between the mean values ​​(centers) of the respective probability distributions.

[0110] [Comparative Example] 4 is a graph and an image for explaining a comparative example of the morphing generation method according to the present invention. Here, in this comparative example, morphing data is generated for morphing from an image of a handwritten digit "0" to an image of a handwritten digit "1" without passing through the "2".

[0111] FIG. 4(A) shows the positional relationship in the latent space of three probability distributions "0", "1", and "2" corresponding to classes "0", "1", and "2" in this comparative example. As can be seen from the figure, probability distribution "2" is positioned between probability distribution "0" and probability distribution "1", which are the targets of morphing. In this way, the positional relationship of probability distributions in the latent space was previously automatically determined within the model.

[0112] In this comparative example, in order to obtain morphing data from a handwritten digit "0" image to a handwritten digit "1" image without passing through the "2", latent variables were sampled along the sampling route of the broken line arrow shown in Fig. 4(A). This sampling route is determined to be as far away as possible from the center of the probability distribution "2" along the way.

[0113] The resulting morphed data is shown in Fig. 4(B). The morphed data in Fig. 4 is obtained by arranging the image data obtained by decoding the sampled latent variables by the decoding unit 13, from the left to the right of the first row, then from the left to the right of the second row, and so on, from the first row to the last row. In this morphed data, it can be seen that a "2" is inserted into the shape of the changing digits between the handwritten digit "0" image and the handwritten digit "1" image, causing a discontinuity (breakdown) in the direction of change.

[0114] In this way, in this comparative example, where the positional relationship of the probability distribution in the latent space cannot be controlled, it is understood that it is not necessarily possible to obtain morphing data that transitions smoothly between two classes without passing through other classes and without interruption in the direction of change.

[0115] [Example 1] FIG. 5 shows graphs and images for explaining the first embodiment of the morphing generation method according to the present invention.

[0116] Here, in this embodiment 1, the setting is to generate morphing data of handwritten digit "0" image → handwritten digit "1" image → handwritten digit "2" image → handwritten digit "0" image, in which no other images intervene between the transition between two classes. In order to make such morphing successful, the above formula (9) (the formula with k=3 in the above formula (14)) is adopted as the "configuration loss" so that the probability distributions "0", "1" and "2" are adjacent to each other in the latent space.

[0117] 5(A) shows the positional relationship in the latent space of three probability distributions "0", "1", and "2" corresponding to classes "0", "1", and "2" in Example 1. According to the figure, it can be seen that the adoption of the above-mentioned "configuration loss" has been successful, and the representative latent variables (selected by the implementer) of the three probability distributions "0", "1", and "2" are approximately located at the three vertices of an equilateral triangle, and these probability distributions are located adjacent to each other.

[0118] Next, in Example 1, (a) From the line segment connecting the representative latent variable (-0.4, 0.1) of probability distribution "0" and the representative latent variable (0.5, 0.2) of probability distribution "1", a total of 64 latent variables are sampled at equal intervals by linear interpolation. (b) From the line segment connecting the representative latent variable (0.5, 0.2) of probability distribution "1" and the representative latent variable (0.0, -0.6) of probability distribution "2", a total of 64 equally spaced latent variables are sampled by linear interpolation. (c) From the line segment connecting the representative latent variable (0.0, -0.6) of probability distribution "2" and the representative latent variable (-0.4, 0.1) of probability distribution "0", a total of 64 equally spaced latent variables are sampled by linear interpolation. A total of 192 latent variables are obtained, and these latent variables are input to the trained decoding unit 13 to generate morphing data consisting of a total of 192 transition images.

[0119] The morphed data thus obtained is shown in Fig. 5(B). In this data, the handwritten digit "0" image is morphed to the handwritten digit "1" image to the handwritten digit "2" image to the handwritten digit "0" image, and a smooth morphing is achieved without any other images being inserted between the transitions between the two classes.

[0120] In this manner, in Example 1, which employs the “configuration loss” to control the positional relationship of the probability distribution in the latent space, morphing data is generated that transitions smoothly between two given classes without going through other classes, without any breaks in the direction of change.

[0121] [Example 2] FIG. 6 shows graphs and images for explaining a morphing generating method according to a second embodiment of the present invention.

[0122] Here, in this second embodiment, the setting is to generate morphing data of a handwritten digit "0" image → a handwritten digit "1" image → a handwritten digit "2" image, in which no other images are interposed between the transition between the two classes. In order to make such morphing successful, the above formula (16) (k=3 in the above formula (22)) is adopted as the "arrangement loss" so that the probability distribution "0", the probability distribution "1", and the probability distribution "2" are linearly arranged in this order in the latent space.

[0123] 6(A) shows the positional relationship in the latent space of three probability distributions "0", "1", and "2" corresponding to classes "0", "1", and "2" in Example 2. According to the figure, the adoption of the above-mentioned "configuration loss" has been successful, and it can be seen that the representative latent variables (selected by the implementer) of the three probability distributions "0", "1", and "2" are each points on approximately one line, and these probability distributions "0", "1", and "2" are positioned in this order in an approximately linear manner.

[0124] Next, in Example 2, (a) From the line segment connecting the representative latent variable (-1.8, 0.0) of probability distribution "0" and the representative latent variable (-0.4, 0.25) of probability distribution "1", a total of 32 latent variables are sampled at equal intervals by linear interpolation. (b) From the line segment connecting the representative latent variable (-0.4, 0.25) of probability distribution "1" and the representative latent variable (1.0, 0.5) of probability distribution "2", a total of 32 equally spaced latent variables are sampled by linear interpolation. A total of 64 latent variables are obtained, and these latent variables are input to the trained decoding unit 13 to generate morphing data consisting of a total of 64 transition images.

[0125] The morphing data thus obtained is shown in Fig. 6(B). In this data, the handwritten digit "0" image is morphed to the handwritten digit "1" image to the handwritten digit "2" image, and it can be seen that a smooth morphing has been achieved without any other images being inserted between the transitions between the two classes.

[0126] In this way, even in this Example 2, which employs the “configuration loss” to control the positional relationship of the probability distribution in the latent space, morphing data is generated that transitions smoothly between two specified classes without passing through other classes and without interruption in the direction of change.

[0127] [Example 3] FIG. 7 is a graph for explaining a third embodiment of the probability distribution arrangement process according to the present invention.

[0128] Here, in this Example 3, as in Example 2, the above formula (16) (k=3 in the above formula (22)) is used as the "arrangement loss" so that the probability distribution "0", the probability distribution "1", and the probability distribution "2" are linearly arranged in this order in the latent space. However, in this Example 3, the generation loss of the above formula (3) used for training, i.e. (Generation loss) = α {(KL information amount) + (Reconstruction loss)} + β (Placement loss) In this example, α is set to 1, while β is set to multiple positive values ​​up to 300.0. Incidentally, the (absolute) position of the probability distribution in Example 3 described later differs from the (absolute) position of the probability distribution in Example 2 because the latent variable generation unit 12 (FIG. 1) generates latent variables that are slightly fluctuated for learning, as shown in the above formula (1). This fluctuation changes every time learning is performed, and the (absolute) position of the probability distribution of each class fluctuates.

[0129] First, in the case of β=34.0 shown in FIG. 7(A), it can be seen that the probability distributions "0", "1" and "2" are neatly arranged in a desired order in a linear manner. Incidentally, it has been confirmed that even with smaller β values, these probability distributions are roughly linearly arranged in a desired order in a linear manner, but with β=34.0, a very neatly arranged positional relationship is realized. Therefore, it can be understood that, for example, when generating morphing data of a handwritten digit "0" image → a handwritten digit "1" image → a handwritten digit "2" image, it is preferable to set β to 34.0.

[0130] On the other hand, as shown in Fig. 7(B) and Fig. 7(C), even in the cases of β=200.0 and β=300.0, the individual probability distributions tend to extend linearly, but the adoption of the "arrangement loss" is effective, and the probability distributions "0", "1", and "2" are neatly arranged in a desired linear order. Here, when comparing the case of β=200.0 (Fig. 7(B)) with the case of β=300.0 (Fig. 7(C)), it can be seen that the (absolute) positional relationship between the probability distributions "0" and "2" in the latent space is swapped. In other words, it can be seen that the (absolute) positional relationship of the probability distributions can be controlled quite freely by setting the value of β.

[0131] Incidentally, even when the "placement loss" has an extremely large effect on the loss calculation, such as when α=1 and β=200.0 or β=300.0, it is possible to obtain the results described above even if only the "placement loss" is used as the generation loss (i.e., even if (generation loss) = (placement loss)), as can be seen from the fact that the desired positional relationship can be realized as described above.

[0132] As can be seen from the results of Example 3, according to the present invention, at least under conditions similar to those of Example 3, it is possible to control the positional relationship of probability distributions corresponding to predetermined classes in the latent space quite freely.

[0133] As described above in detail, according to the present invention, by training a generative model using the configuration loss, the positional relationship of probability distributions corresponding to a predetermined class in the latent space can be made to be or be made to be close to a predetermined (specified or set) positional relationship defined by the distance between the distributions. This also makes it possible to realize a desired positional relationship in the probability distribution and increase the degree of freedom in sampling data such as new learning data and morphing data.

[0134] Furthermore, according to the present invention, it is possible to generate morphing data that transitions seamlessly between two given classes without passing through other classes, using the generative model according to the present invention. It is also possible to generate an intermediate image between two arbitrary images, without the intervention of other image features, so that it is possible to create, for example, a "mixed face app" that generates a face that is a mix of two faces.

[0135] Furthermore, according to the present invention, it is possible to generate an image of an intermediate posture at any stage from an image of a person at the start of a movement and an image of the same person at the end of the movement, making the present invention applicable, for example, to predicting person movements in the field of RaaS (Robotics as a Service).

[0136] In this case, for example, it is possible to widely provide advanced nursing support robot functions, educational support robot functions, work assistance robot functions, etc., that apply more advanced motion prediction processing as RaaS. In other words, according to the present invention, it is also possible to contribute to Goal 3 "Ensure healthy lives and promote well-being for all at all ages," Goal 4 "Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all," and Goal 8 "Promote inclusive and sustainable economic growth, employment and decent work for all" of the Sustainable Development Goals (SDGs) led by the United Nations.

[0137] Regarding the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical idea and perspective of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to be restrictive in any way. The present invention is limited only by the scope of the claims and their equivalents. [Explanation of symbols]

[0138] 1. Generative Model 11 Encoding section 12 Latent variable generation section 13 Decoding section 9 Morphing Generation Device 91 Input section 92 Training Department 92a Placement loss calculation section 92b Generation loss calculation section 93 Sampling section 94 Morphing Generation Section 95 Output section

Claims

1. A generation model that generates, from each of a plurality of input data belonging to a plurality of classes, a latent variable corresponding to the class to which the input data belongs in a set latent space, an encoding unit that converts the input data into information related to a probability distribution in the latent space corresponding to the class to which the input data belongs; a latent variable generation unit that generates the latent variables from information related to the probability distribution; A decoding unit that converts the latent variables into output data corresponding to the input data; to make the computer function, the encoding unit and the decoding unit are trained using a generation loss including at least a configuration loss that is defined using a distance between the probability distributions corresponding to the different classes, and indicates that for at least one combination of two classes designated or set among a plurality of combinations of two classes generated from the plurality of classes, the loss is smaller when the two probability distributions corresponding to the two classes constituting the combination of the two classes are adjacent to each other in the latent space; The information on the probability distribution corresponding to the two classes forming the combination of the two classes specified or set and output from the encoding unit is information in which the two different probability distributions corresponding to the information on the probability distribution are adjacent to each other or close to an adjacent positional relationship in the latent space. A generative model characterized by

2. The generative model according to claim 1, characterized in that the placement loss is a distance defined for each combination of two classes generated from the multiple classes, and is set to a value indicating that there is no loss or that the loss is minimal when the distance between the two probability distributions corresponding to the two classes constituting the combination is equal to each other between the combinations.

3. A generative model that generates, from each of a plurality of input data belonging to a plurality of classes, a latent variable corresponding to the class to which the input data belongs in a set latent space, comprising: an encoding unit that converts the input data into information related to a probability distribution in the latent space corresponding to the class to which the input data belongs; a latent variable generation unit that generates the latent variables from information related to the probability distribution; A decoding unit that converts the latent variables into output data corresponding to the input data; to make the computer function, the encoding unit and the decoding unit are trained using a generation loss including at least a configuration loss that is defined using a distance between the probability distributions corresponding to the different classes, and that indicates that there is no loss or that the loss is minimized when a distance between the probability distributions satisfies a condition related to a distance that should be satisfied when, in the latent space, one of the plurality of probability distributions corresponding to the plurality of classes respectively is a starting point and the remaining probability distributions are arranged in a positional relationship in which the remaining probability distributions are adjacent to each other in sequence and are arranged in a manner that is sequentially farther from the starting point, and The information on the probability distribution corresponding to each of the plurality of classes output from the encoding unit is information in which a distance in the latent space between different probability distributions corresponding to the information on the probability distribution satisfies a condition related to the distance or approaches a value that satisfies the condition related to the distance. A generative model characterized by

4. The generative model of any one of claims 1 to 3, characterized in that the placement loss includes a distance constraint term indicating that the loss becomes smaller the closer the distance between adjacent probability distributions is to a set value.

5. The generation loss is set as a linear combination of a reconstruction loss relating to an expected value of the output data when the input data is input, a distance information amount relating to the distance between the probability distribution and a standard distribution relating to the output of the encoding unit to which the input data is input, and the alignment loss. The generation model according to any one of claims 1 to 4, characterized in that

6. A generative model for generating, from each of a plurality of input data belonging to a plurality of classes, a latent variable corresponding to the class to which the input data belongs in a set latent space, comprising: an encoding unit that converts the input data into information related to a probability distribution in the latent space corresponding to the class to which the input data belongs; a latent variable generation unit that generates the latent variables from information related to the probability distribution; A decoding unit that converts the latent variables into output data corresponding to the input data; to make the computer function, the encoding unit and the decoding unit are trained using a generation loss including at least a configuration loss that is defined using a distance between the probability distributions corresponding to the different classes, the configuration loss including a distance constraint term indicating that there is no loss or that the loss is minimized when the positional relationship of the probability distributions in the latent space is a specified or set positional relationship, and that the loss becomes smaller as the distance between adjacent probability distributions becomes closer to a set value; The information on the probability distributions corresponding to the classes, which are different from each other and output from the encoding unit, is information in which the positional relationship in the latent space in the different probability distributions corresponding to the information on the probability distributions is the specified or set positional relationship or is close to the specified or set positional relationship. A generative model characterized by

7. A model training method for training a generative model that generates latent variables corresponding to a class to which a plurality of input data belongs, in a set latent space, from each of a plurality of input data belonging to each of a plurality of classes, the method comprising: The generative model is an encoding unit that converts the input data into information related to a probability distribution in the latent space corresponding to the class to which the input data belongs; a latent variable generation unit that generates the latent variables from information related to the probability distribution; A decoding unit that converts the latent variables into output data corresponding to the input data; This is a model of how computers function. The model training method includes: setting a configuration loss that is defined using a distance between the probability distributions corresponding to different classes, and indicates that the loss is smaller when, for at least one combination of two classes designated or set among a plurality of combinations of two classes generated from the plurality of classes, the two probability distributions corresponding to the two classes constituting the combination of the two classes are adjacent to each other in the latent space; a step of causing the encoding unit and the decoding unit to learn learning parameters using a generation loss including at least the set alignment loss; 1. A computer-implemented method for training a model, comprising:

8. a sampling means for determining a representative latent variable which is a latent variable representing the probability distribution corresponding to each class for each class by using the generative model according to any one of claims 1 to 5, and sampling a plurality of latent variables corresponding to points on a line segment or curve segment which sequentially connects the representative latent variables relating to the plurality of classes specified or set including their order in accordance with the order; a morphing generating means for inputting each of the sampled latent variables to the decoding unit and generating a set of the output data output from the decoding unit as morphing data between the classes; to make the computer function, The morphing data is data for realizing a morphing in which no classes other than the two classes adjacent to each other in the order are involved during the transition between the two classes. A morphing generation program comprising:

9. a sampling means for determining a representative latent variable which is a latent variable representing the probability distribution corresponding to each class for each class by using the generative model according to any one of claims 1 to 6, and sampling a plurality of latent variables corresponding to points on a line segment or curve segment which sequentially connects the representative latent variables relating to the plurality of classes specified or set including their order in accordance with the order; a morphing generating means for inputting each of the sampled latent variables to the decoding unit and generating a set of the output data output from the decoding unit as morphing data between the classes; to make the computer function, The sampling means samples the plurality of latent variables such that the density of the sampled latent variables on the line segment or curve segment becomes lower at positions closer to the representative latent variable. A morphing generation program comprising:

10. a sampling means for determining a representative latent variable which is a latent variable representing the probability distribution corresponding to each class for each class by using the generative model according to any one of claims 1 to 5, and sampling a plurality of latent variables corresponding to points on a line segment or curve segment which sequentially connects the representative latent variables relating to the plurality of classes specified or set including their order in accordance with the order; a morphing generating means for inputting each of the sampled latent variables to the decoding unit and generating a set of the output data output from the decoding unit as morphing data between the classes; having The morphing data is data for realizing a morphing in which no classes other than the two classes adjacent to each other in the order are involved during the transition between the two classes. A morphing generating device comprising:

11. A step of determining a representative latent variable, which is a latent variable representing the probability distribution corresponding to each class, for each class using the generative model according to any one of claims 1 to 5, and sampling a plurality of latent variables corresponding to points on a line segment or curve segment that sequentially connects the representative latent variables related to the plurality of classes specified or set including their order in the order; inputting each of the sampled latent variables to the decoding unit, and setting a set of the output data output from the decoding unit as morphing data between the classes; having The morphing data is data for realizing a morphing in which no classes other than the two classes adjacent to each other in the order are involved during the transition between the two classes.

13. A morphing generation method implemented by a computer.

Images