Learning system, learning method, and program

Abstract

Improve the accuracy of GAN. [Solution] A learning system (1) trains a GAN (Generative Adversarial Network) generator that allows a user to control multiple features related to a generated image. A partial code acquisition unit (107) acquires multiple partial codes corresponding to multiple features, respectively, based on a latent code for generating the generated image and multiple mapping networks corresponding to the multiple features. A generated image generation unit (105) generates a generated image based on an image synthesis network that generates the generated image using the multiple partial codes. A generator training unit (106) trains a generator including multiple mapping networks and an image synthesis network, based on the generated image and a trained GAN classifier.

Description

[Technical Field]

[0001] The present disclosure relates to a learning system, a learning method, and a program. [Background technology]

[0002] Conventionally, in the field of machine learning, GANs (Generative Adversarial Networks) that generate images are known. Hereinafter, images generated by GANs are referred to as generated images. For example, Non-Patent Documents 1, 2, and 3 describe GANs that allow a user to control multiple features (e.g., degree of smile, age, and hairstyle) related to the generated image. When a user specifies desired features, a latent code generated based on the specified features is input to a generator of a trained GAN. The generator generates a generated image having the features desired by the user based on the latent code. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Tero Karras, Miika Aittala, Samuli Laine, Erik Harkonen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34:852-863, 2021. [Non-patent document 2] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019. [Non-patent document 3] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 8110-8119, 2020. Summary of the Invention [Problem to be solved by the invention]

[0004] However, in each of the technologies in Non-Patent Document 1, Non-Patent Document 2, and Non-Patent Document 3, the GAN generator generates a generated image based on the entire latent code, and therefore it is not possible to accurately learn which part of the latent code corresponds to which feature among multiple features. For this reason, each of the technologies in Non-Patent Document 1, Non-Patent Document 2, and Non-Patent Document 3 was not able to sufficiently improve the accuracy of the GAN.

[0005] One of the goals of this disclosure is to improve the accuracy of GANs. [Means for solving the problem]

[0006] The learning system according to the present disclosure is a learning system that trains a GAN (Generative Adversarial Network) generator in which a user can control multiple features related to a generated image, and includes: a partial code acquisition unit that acquires multiple partial codes corresponding to each of the multiple features based on a latent code for generating the generated image and multiple mapping networks corresponding to each of the multiple features; a generated image generation unit that generates the generated image based on an image synthesis network that generates the generated image using the multiple partial codes; and a generator learning unit that trains the generator, including the multiple mapping networks and the image synthesis network, based on the generated image and a trained classifier of the GAN. [Effects of the Invention]

[0007] The present disclosure can, for example, improve the accuracy of GANs. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of a learning system. [Figure 2] FIG. 1 illustrates an example of a GAN where the user can control the features of the generated image. [Figure 3] FIG. 2 is a diagram illustrating an example of functions realized by the learning system. [Figure 4] FIG. 10 is a diagram illustrating an example of a function for learning a classifier. [Figure 5] FIG. 10 is a diagram illustrating an example of an identification image database. [Figure 6] FIG. 10 is a diagram illustrating an example of a function for training a generator. [Figure 7] FIG. 10 is a diagram illustrating an example of a generated image database. [Figure 8] FIG. 10 is a diagram illustrating an example of a function for training a generator. [Figure 9] FIG. 10 is a diagram illustrating an example of a function for training a generator. [Figure 10]FIG. 2 is a diagram illustrating an example of processing executed in the learning system. [Figure 11] FIG. 2 is a diagram illustrating an example of processing executed in the learning system. DETAILED DESCRIPTION OF THE INVENTION

[0009] [1. Hardware configuration of the learning system] An example of an embodiment of a learning system, learning method, and program according to the present disclosure will be described. Fig. 1 is a diagram showing an example of the hardware configuration of a learning system. For example, the learning system 1 includes a learning terminal 10, a server 20, and a user terminal 30. Each of the learning terminal 10, the server 20, and the user terminal 30 is connected to a network such as the Internet or a LAN (Local Area Network).

[0010] The learning terminal 10 is a computer that performs training using a GAN (Generative Adversarial Network) to generate images. Hereinafter, an image generated by a GAN is referred to as a generated image. For example, the learning terminal 10 is a personal computer, a server computer, a tablet, or a smartphone. For example, the learning terminal 10 includes a control unit 11, a memory unit 12, a communication unit 13, an operation unit 14, and a display unit 15.

[0011] For example, the control unit 11 includes at least one processor. The storage unit 12 includes at least one of a volatile memory such as RAM and a non-volatile memory such as flash memory. The communication unit 13 includes at least one of a communication interface for wired communication and a communication interface for wireless communication. The operation unit 14 is an input device such as a touch panel or a mouse. The display unit 15 is a display such as a liquid crystal or organic electroluminescence (EL).

[0012] The server 20 is a server computer that stores the trained GAN. For example, the server 20 includes a control unit 21, a storage unit 22, and a communication unit 23. The hardware configurations of the control unit 21, the storage unit 22, and the communication unit 23 may be similar to those of the control unit 11, the storage unit 12, and the communication unit 13, respectively.

[0013] The user terminal 30 is a user's computer that uses the trained GAN. For example, the user terminal 30 is a personal computer, a smartphone, a tablet, or a wearable device. For example, the user terminal 30 includes a control unit 31, a storage unit 32, a communication unit 33, an operation unit 34, and a display unit 35. The hardware configurations of the control unit 31, the storage unit 32, the communication unit 33, the operation unit 34, and the display unit 35 may be similar to those of the control unit 11, the storage unit 12, the communication unit 13, the operation unit 14, and the display unit 15, respectively.

[0014] The programs stored in the storage units 12, 22, 32 may be supplied to the learning terminal 10, the server 20, or the user terminal 30 via a network. Also, at least one of a reading unit (e.g., a memory card slot) that reads a computer-readable information storage medium and an input / output unit (e.g., a USB port) for inputting and outputting data to and from an external device may be included in the learning terminal 10, the server 20, or the user terminal 30. For example, a program stored in an information storage medium may be supplied to the learning terminal 10, the server 20, or the user terminal 30 via at least one of the reading unit and the input / output unit.

[0015] Furthermore, the learning system 1 only needs to include at least one computer. The computers included in the learning system 1 are not limited to the example in FIG. 1. For example, the learning system 1 may include only the learning terminal 10. In this case, the server 20 and the user terminal 30 exist outside the learning system 1. The learning system 1 may include only the learning terminal 10 and the server 20. In this case, the user terminal 30 exists outside the learning system 1. The learning system 1 may include only other computers not shown in FIG. 1.

[0016] [2. Overview of the learning system] For example, a GAN generates a generated image based on a latent code, which will be described later. If a GAN generates a generated image based on a completely randomly generated latent code, the GAN may not generate a generated image with the characteristics desired by a user. The features here refer to visual characteristics of the generated image. For example, the shape, appearance, size, brightness, or arrangement of an object (e.g., a person, an animal, a landscape, or a building) shown in the generated image corresponds to a feature. A feature can also be referred to as a style, condition, or attribute of the generated image.

[0017] FIG. 2 is a diagram illustrating an example of a GAN in which a user can control features related to a generated image. For example, research on GANs that generate generated images with user-desired features includes latent conditioned GANs (LC), conditional GANs, StyleGANs, and self-supervised style decomposition (SSD) GANs. Hereinafter, when simply referring to a "feature," it does not refer to a feature vector calculated by a GAN, but rather to a feature to be controlled. In this embodiment, a user can control each of a plurality of features.

[0018] As shown in FIG. 2, a GAN includes a generator that generates a generated image and a classifier that identifies the generated image. The generator is a model that generates a generated image based on a latent code corresponding to features desired by a user. For example, the generator is a convolutional neural network or a fully connected neural network. Other known techniques may also be used for the generator. Recently, a technique called Transformer, which is mainly used in natural language processing, has been used for image generation, so a generator using this technique may also be used.

[0019] A latent code is information that conveys the user's desired features to the generator. For example, the latent code is expressed in vector format. The latent code is also called random noise or a condition vector. The latent code may be expressed in a format other than a vector. For example, in a GAN that generates a synthetic image showing an animal's face, a user can control two features, namely, shape and appearance. When a user specifies these two features, a latent code is generated that corresponds to the shape and appearance specified by the user. The user does not need to specify all features, and may specify only some of the features.

[0020] For example, the generator generates a generated image based on a latent code input thereto. The generator may also generate a generated image based on random noise separate from the latent code. When a latent code is input, the generator may convert the latent code into a more appropriate representation based on a mapping network. The generator outputs a generated image corresponding to the latent code based on its own parameters. The generator parameters are referenced in the process by which the generator outputs a generated image based on the latent code.

[0021] The classifier identifies whether the generated image is authentic or not. In the case of a controllable GAN, the classifier identifies whether the generated image has the features desired by the user. That is, the classifier identifies whether the generated image is an image corresponding to the latent code. For example, the classifier is a convolutional neural network or a fully connected neural network. Other known methods may also be used for the classifier. Recently, a method called Transformer, which is mainly used in natural language processing, has been used for image generation, so the classifier may use such a method.

[0022] For example, when a generated image is input to the classifier, the classifier performs processing such as convolution based on its own parameters to calculate a feature vector for the generated image. The classifier outputs a classification result corresponding to the feature vector. The classification result may be a label indicating the authenticity of the generated image, a score indicating the probability that the generated image is authentic, a label indicating whether the generated image is an image corresponding to the latent code, a score indicating the probability that the generated image is an image corresponding to the latent code, or a combination of these. The series of processes from when the generated image is input to the classifier to when the classifier outputs the classification result may be similar to known processes.

[0023] As described above, a GAN includes a generator and a classifier. In general, the training of a GAN begins with the training of the classifier. After the training of the classifier is completed to a certain extent, the training of the generator is performed. The parameters of the classifier may be fixed during the training of the generator. The training of the classifier and the training of the generator may then be repeated. When a latent code for a user to control multiple features is input to the generator, each part of the latent code corresponds to an individual feature. For example, if a user can control two features, such as shape and appearance, a specific part of the latent code corresponds to the shape. Another part of the latent code corresponds to the appearance. In order for the generator to generate a generated image according to each feature, it is necessary for the generator to learn which feature each part of the latent code corresponds to.

[0024] Therefore, the learning system 1 of this embodiment divides the training latent code into parts corresponding to each of the multiple features. Hereinafter, each part will be referred to as a partial code. The latent code is divided into multiple partial codes. A generator generates a generated image based on each of the multiple partial codes. A classifier estimates the authenticity of the generated image, and the generator is trained. For example, in training the generator, a spectral normalization technique (described below) is used, which makes it easier for the generator to recognize which part of the latent code corresponds to which feature. The learning system 1 also has other functions for training the generator and a function for training the classifier. The learning system 1 improves the accuracy of the GAN by using at least one of these functions. The learning system 1 will be described in detail below.

[0025] [3. Functions realized by the learning system] FIG. 3 is a diagram showing an example of functions realized by the learning system 1. In this embodiment, an example is given in which the main functions for GAN learning are realized by the learning terminal 10. In this embodiment, an example is given of the processing during GAN learning that generates generated images showing animal faces. Furthermore, two examples of user-controllable features are given: shape and appearance. Shape is the shape of the surface of an object. Shape can also be said to be the outline of an object. Appearance is the appearance of an object. For example, appearance can be color, pattern, or brightness.

[0026] It should be noted that the user-controllable features are not limited to shape and appearance. For example, the user-controllable features may be four features: the position and orientation of a camera capturing an object, a global identity, which is the overall appearance of the object, a local identity, which is the local appearance of the object, and color. The user-controllable features may also be other features. For example, the user-controllable features may be facial expression, facial contour, eye color, hair color, brightness, background, or other features. The number of user-controllable features is also not limited to two or four, and may be any number.

[0027] [3-1. Functions for learning classifiers] FIG. 4 is a diagram illustrating an example of a function for training a classifier. The function for training a classifier will be described with reference to FIG. 4, using the diagram illustrating the functions of the learning terminal 10 shown in FIG. 3. As shown in FIG. 3, for example, the learning terminal 10 includes a data storage unit 100, a discriminant image acquisition unit 101, a discriminant vector calculation unit 102, and a discriminant training unit 103 as functions for training a classifier. The data storage unit 100 is realized by the storage unit 12. The discriminant image acquisition unit 101, the discriminant vector calculation unit 102, and the discriminant training unit 103 are each realized by the control unit 11. With these functions, the learning system 1 trains a GAN classifier in which multiple features related to the generated image can be controlled by the user.

[0028] [Data storage section] The data storage unit 100 stores data necessary for training the classifier. For example, the data storage unit 100 stores actual data of the GAN and a classification image database DB1.

[0029] The GAN whose actual data is stored in the data storage unit 100 is a GAN before learning. A GAN before learning is a GAN whose parameters are initial values. The actual data indicates a GAN program and GAN parameters. For example, the data storage unit 100 stores actual data of a classifier included in the actual data of the GAN. The actual data of the classifier indicates a classifier program and classifier parameters. The classifier parameters are referenced by the classifier program. The classifier parameters are adjusted by the classifier learning unit 103. For example, the classifier parameters are weight coefficients and biases. The classifier parameters may be known parameters. For example, the classifier parameters may be the number of hidden layers, the number of units in the hidden layers, or other hyperparameters.

[0030] FIG. 5 is a diagram showing an example of the identification image database DB1. The identification image database DB1 is a database that stores identification images, which are images for training a classifier. In this embodiment, three identification images, namely, an anchor identification image, a shape identification image, and an appearance identification image, are used in training the classifier. Hereinafter, when there is no need to distinguish between the anchor identification image, the shape identification image, and the appearance identification image, they will simply be referred to as identification images. The identification images can also be referred to as training images used as training data for the classifier. The learning terminal 10 can acquire generated images stored in the identification image database DB1 at any time.

[0031] For example, the identification image database DB1 stores image data of an anchor identification image, a shape identification image, and an appearance identification image. Any data may be stored in the identification image database DB1. The data stored in the identification image database DB1 is not limited to the example of FIG. 5. For example, the identification image database DB1 may store information indicating which feature, out of multiple user-controllable features, was changed to generate the shape identification image and the appearance identification image.

[0032] The anchor identification image is an original image with no changes to user-controllable features. For example, the anchor identification image may be a publicly known image distributed on the Internet, either free of charge or for a fee. The anchor identification image is a reference image for distance learning. In this embodiment, when an anchor identification image is hereinafter represented by a symbol, it is written as x. The identification image database DB1 may store any number of anchor identification images. The objects represented by the individual anchor identification images may be the same or different from each other. In the example of FIG. 4, a dog corresponds to the object, but anchor identification images representing multiple animals may be mixed in the identification image database DB1.

[0033] A shape identification image is an image in which the shape of an object has been changed relative to the anchor identification image. For example, a shape identification image may be an image generated by performing image processing such as affine transformation, linear transformation, or spline transformation on the anchor identification image. In learning in the shape feature space described below, a shape identification image is used as a positive identification image. A positive identification image is an image that allows a classifier to recognize an image similar to anchor identification information (an image that belongs to the same cluster as the anchor identification image) in metric learning. On the other hand, in learning in the appearance feature space described below, a shape identification image is used as a negative identification image described below. A negative identification image is an image that allows a classifier to recognize an image different from anchor identification information (an image that does not belong to the same cluster as the anchor identification image) in metric learning. Hereinafter, when a shape identification image is represented by a symbol, it will be referred to as x g In the example of Fig. 4, the shape identification image has a changed shape (outline) of the dog indicated by the anchor identification image.

[0034] The appearance identification image is an image in which the appearance of an object is changed relative to the anchor identification image. For example, the shape identification image may be an image generated by performing image processing such as color conversion, masking, cutting, or texture pasting on the anchor identification image. In learning in the appearance feature space described below, the appearance identification image is used as a positive identification image. In learning in the shape feature space described below, the appearance identification image is used as a negative identification image. Hereinafter, when the appearance identification image is represented by a symbol, it will be referred to as x a In the example of FIG. 4, the appearance of the dog shown in the anchor identification image is changed by masking.

[0035] The data stored in the data storage unit 100 is not limited to the above example. The data storage unit 100 may store any data. For example, the data storage unit 100 may store a classifier learning program that indicates a series of processes in learning a classifier. The classifier learning program includes program codes that indicate the processes of the classification image acquisition unit 101, the classification vector calculation unit 102, part of the processes of the classifier learning unit 103, and the processes of the classifier learning unit 103. For example, the data storage unit 100 may store a feature modification program for modifying each of a plurality of features that can be controlled by GAN.

[0036] [Identification image acquisition section] The identification image acquisition unit 101 acquires an identification image from the identification image database DB1. If an identification image is stored in a database other than the identification image database DB1, a computer other than the learning terminal 10, or an information storage medium, the identification image acquisition unit 101 may acquire the identification image from the other database, computer, or information storage medium. The identification image acquisition unit 101 can acquire any number of identification images.

[0037] For example, the identification image acquisition unit 101 acquires an anchor identification image and a feature identification image in which each of a plurality of features has been changed. The feature identification image is an image in which changes have been made to the anchor identification image in relation to features that can be controlled by the user. The feature identification image is an image in which at least a part of the anchor identification image has been changed. An example will be given in which there are as many feature identification images for one anchor identification image as there are features that can be controlled by the user, but there may also be more feature identification images for one anchor identification image than the number of features that can be controlled by the user.

[0038] In this embodiment, since each of the shape and appearance is changed as a feature, the identification image acquisition unit 101 acquires an anchor identification image, a shape identification image, and an appearance identification image. Each of the shape identification image and the appearance identification image is an example of a feature identification image. Therefore, the description of each of the shape identification image and the appearance identification image can be read as a feature identification image. For example, the identification image acquisition unit 101 acquires two feature identification images, a shape identification image and an appearance identification image, for one anchor identification image.

[0039] In this embodiment, distance learning is used in learning the classifier, so the discriminant image acquisition unit 101 acquires an anchor discriminant image and a plurality of feature discriminant images in which the features of the anchor discriminant image have been changed and which correspond to a plurality of features, respectively. For example, the discriminant image acquisition unit 101 acquires an anchor discriminant image, a shape discriminant image as a positive discriminant image, and an appearance discriminant image as a negative discriminant image for learning in the shape feature space. The discriminant image acquisition unit 101 acquires an anchor discriminant image, an appearance discriminant image as a positive discriminant image, and a shape discriminant image as a negative discriminant image for learning in the appearance feature space.

[0040] In this embodiment, an example is given in which the anchor identification image, shape identification image, and appearance identification image are stored in advance in the identification image database DB1. However, the identification image database DB1 may initially store only the anchor identification image. The identification image acquisition unit 101 may generate the shape identification image and the appearance identification image based on the anchor identification image. In this case, the identification image acquisition unit 101 generates the shape identification image and the appearance identification image by performing image processing on the anchor identification image. It is assumed that a program for image processing is stored in advance in the data storage unit 100.

[0041] For example, the identification image acquisition unit 101 generates a shape identification image by performing image processing (such as the affine transformation described above) on the anchor identification image to change the shape of the object. The identification image acquisition unit 101 generates an appearance identification image by performing image processing (such as the mask processing described above) on the anchor identification image to change the appearance of the object. The identification image acquisition unit 101 stores the generated shape identification image and appearance identification image in the identification image database DB1.

[0042] [Discrimination vector calculation part] The discrimination vector calculation unit 102 calculates a discrimination vector, which is a feature vector of the discrimination image, based on the discriminator. For example, the discrimination vector calculation unit 102 calculates, for each feature space corresponding to each of a plurality of features, an anchor discrimination vector for the anchor discrimination image, a positive discrimination vector for the feature discrimination image corresponding to the feature, and a negative discrimination vector for the feature discrimination image corresponding to another feature, based on the discriminator. When the anchor discrimination vector, the positive discrimination vector, and the negative discrimination vector are not distinguished from one another, they are simply referred to as discrimination vectors. The calculation of these vectors is sometimes called mapping to the feature space.

[0043] The feature space is a multidimensional space. The number of feature spaces is equal to or greater than the number of user-controllable features. The classifier calculates a feature vector in each of multiple feature spaces for an image input thereto. The classifier parameters for calculating a feature vector in one feature space are different from the classifier parameters for calculating a feature vector in another feature space. The number of classifier parameters is equal to or greater than the number of user-controllable features. In this embodiment, the classifier has three parameters: a parameter for authenticity discrimination, a parameter for a feature space corresponding to shape, and a parameter for a feature space corresponding to appearance. Other parameters may also be included in the classifier.

[0044] In this embodiment, an example is given in which a program for calculating a feature vector in one feature space and a program for calculating a feature vector in another feature space are the same. Even if these programs are the same, different feature vectors are calculated for the same image because the parameters for calculating the feature vector in one feature space are different from the parameters for calculating the feature vector in the feature space in the other feature space. Note that these programs may be different from each other. There may be as many classifier programs as there are user-controllable features, or more.

[0045] The anchor identification vector is the feature vector of the anchor identification image. The positive identification vector is the feature vector of the positive identification image. The negative identification vector is the feature vector of the negative identification image. Hereinafter, the set of anchor identification image, positive identification image, and negative identification image will be referred to as the identification image set. For one identification image set, a set of three feature vectors, namely, the anchor identification vector, positive identification vector, and negative identification vector, is calculated. If the number of user-controllable features is X (X is an integer greater than or equal to 2) and the number of identification image sets is Y (Y is an integer greater than or equal to 2), at least X × Y × 3 feature vectors will be calculated.

[0046] In this embodiment, an example is given in which there is a feature space for identifying the authenticity of an input image (e.g., an identification image) and a feature space corresponding to each of a plurality of user-controllable features. In this embodiment, since the number of user-controllable features is two, the classifier calculates a feature vector in each of at least two feature spaces, as shown in FIG. 4. In the example of FIG. 4, the output of the classifier is indicated by the symbol P. Furthermore, the projection head for projecting the identification image into the feature space is indicated by the symbol h. The projection head is a program and parameters for calculating a feature vector in the feature space from the identification image. For example, each layer, such as a fully connected layer and an embedding layer, corresponds to a projection head. Note that the projection head reduces the dimension of the feature vector, but this reduction need not necessarily be performed.

[0047] In the example in Figure 4, h g is a projection head for projecting an identification image into the shape feature space, which is the feature space of the shape. g (x) is the anchor identification vector of the anchor identification image in the shape feature space. The identification vector calculation unit 102 calculates the anchor identification vector h of the anchor identification image based on the anchor identification image in the shape feature space and the projection head in the shape feature space. g Calculate (x). h g (x g ) is a positive discrimination vector of a positive discrimination image (shape discrimination image) in the shape feature space. The discrimination vector calculation unit 102 calculates the positive discrimination vector h of the positive discrimination image based on the positive discrimination image in the shape feature space and the projection head in the shape feature space. g (x g ) is calculated. g (x a ) is a negative discrimination vector of the negative discrimination image (appearance discrimination image) in the shape feature space. The discrimination vector calculation unit 102 calculates the negative discrimination vector h of the negative discrimination image based on the negative discrimination image in the shape feature space and the projection head in the shape feature space. g (xa ) is calculated.

[0048] In the example in Figure 4, h adv is a projection head for projecting an identification image into a true / false feature space, which is a feature space for determining whether an image is true or false. The true / false feature space may be the same as the feature space used in known GANs. In FIG. 4, the true / false feature space is omitted for space reasons. The identification vector calculation unit 102 calculates feature vectors for each of the anchor identification image, shape identification image, and appearance identification image in the true / false feature space as well. For example, the identification vector calculation unit 102 calculates feature vectors in the feature space for true / false identification by reducing the dimension of the output from the classifier using a projection head. Based on these feature vectors, the image input to the classifier is identified as true or false. If the classifier can be trained accurately, the classifier will estimate the anchor identification image as true. The classifier will estimate the shape identification image and appearance identification image as false.

[0049] In the example in Figure 4, h a is a projection head for projecting an identification image into the appearance feature space, which is the appearance feature space. a (x) is the anchor identification vector of the anchor identification image in the appearance feature space. The identification vector calculation unit 102 calculates the anchor identification vector h of the anchor identification image based on the anchor identification image in the appearance feature space and the projection head in the appearance feature space. a Calculate (x). h a (x a ) is a positive discrimination vector of a positive discrimination image (appearance discrimination image) in the appearance feature space. The discrimination vector calculation unit 102 calculates the positive discrimination vector h of the positive discrimination image based on the positive discrimination image in the appearance feature space and the projection head in the appearance feature space. a (x a ) is calculated. a (x g) is the negative discrimination vector of the negative discrimination image (shape discrimination image) in the appearance feature space. The discrimination vector calculation unit 102 calculates the negative discrimination vector h of the negative discrimination image based on the negative discrimination image in the appearance feature space and the projection head in the appearance feature space. a (x g ) is calculated.

[0050] [Classifier training section] The classifier training unit 103 trains the classifier based on the discrimination vector calculated by the discrimination vector calculation unit 102. For example, the classifier training unit 103 causes the classifier to estimate the authenticity of the anchor discrimination image and the authenticity of the generated image generated by the GAN generator. Here, estimation can also be referred to as discrimination. The estimation by the classifier training unit 103 is performed based on the classifier currently being trained. That is, the classifier training unit 103 performs estimation based on the current parameters of the classifier. For example, the classifier training unit 103 inputs the anchor discrimination image to the classifier. The classifier calculates an anchor discrimination vector of the anchor discrimination image in a feature space for discriminating authenticity. The classifier outputs an estimation result of authenticity based on the anchor discrimination vector. The classifier training unit 103 acquires the estimation result of authenticity output from the classifier.

[0051] In this embodiment, since a plurality of anchor identification images are prepared, the classifier learning unit 103 causes the classifier to estimate the authenticity of each of the plurality of anchor identification images. The classifier learning unit 103 inputs each of the plurality of anchor identification images to the classifier one after another, and obtains an estimation result of the authenticity of each anchor identification image from the classifier. The method of estimating the authenticity of each anchor identification image is as described above. The classifier learning unit 103 trains the classifier so that the anchor identification image is estimated to be authentic.

[0052] For example, the classifier training unit 103 trains the classifier so that the anchor identification vector and the positive identification vector are close to each other and the anchor identification vector and the negative identification vector are far apart in the feature space corresponding to each of the multiple features. In other words, the classifier training unit 103 trains the classifier so that the anchor identification vector and the positive identification vector belong to the same cluster and the anchor identification vector and the negative identification vector do not belong to the same cluster in the feature space corresponding to each of the multiple features.

[0053] The learning algorithm itself may be a known algorithm used in metric learning. For example, algorithms such as gradient descent, gradient penalty, stochastic gradient flow, or variational inference may be used to train the classifier. In this embodiment, the relationship between three identified images is used in the learning. Therefore, the classifier training unit 103 trains the classifier based on a triplet margin loss loss function so that the anchor identification vector and the positive identification vector become closer and the anchor identification vector and the negative identification vector become farther apart. The loss function described in this embodiment is an example of a triplet margin loss loss function.

[0054] For example, the classifier training unit 103 performs classifier training by adjusting the parameters of a plurality of features so that the anchor identification vector and the positive identification vector become closer and the anchor identification vector and the negative identification vector become farther apart in a feature space corresponding to each of the plurality of features. The classifier training unit 103 calculates a loss indicating the proximity between the anchor identification vector and the positive identification vector and the distance between the anchor identification vector and the negative identification vector in the feature space corresponding to each of the plurality of features based on a loss function in distance training, and adjusts the parameters of the feature so that the loss becomes smaller.

[0055] In this embodiment, as shown in Fig. 4, the classifier training unit 103 adjusts the shape parameters so that the anchor identification vector of the anchor identification image and the positive identification vector of the shape's positive identification image (shape identification image) become closer to each other and the anchor identification vector becomes farther away from the negative identification vector of the shape's negative identification image (appearance identification image) in the shape feature space, thereby training the classifier. Based on the loss function in distance training, the classifier training unit 103 calculates a loss indicating the proximity between the anchor identification vector and the positive identification vector and the distance between the anchor identification vector and the negative identification vector in the shape feature space, and adjusts the shape parameters so as to reduce the loss.

[0056] 4, the classifier training unit 103 adjusts the appearance parameters so that the anchor identification vector of the anchor identification image and the positive identification vector of the positive appearance identification image (appearance identification image) become closer and the anchor identification vector becomes farther away from the negative identification vector of the negative appearance identification image (shape identification image) in the appearance feature space. Based on the loss function in distance training, the classifier training unit 103 calculates a loss indicating the proximity between the anchor identification vector and the positive identification vector and the distance between the anchor identification vector and the negative identification vector in the appearance feature space, and adjusts the appearance parameters so as to reduce the loss.

[0057] In this embodiment, the classifier training unit 103 calculates, for each anchor identification image, a contrastive classification loss related to the closeness between the anchor identification vector of the anchor identification image and the positive classification vector of the positive classification image of the anchor identification image, and the closeness between the anchor identification vector of the anchor identification image and the negative classification vector of the negative classification image of the anchor identification image. The classifier training unit 103 trains the classifier based on the contrastive classification loss. The classifier training unit 103 trains the classifier so as to reduce the contrastive classification loss.

[0058] The contrastive discrimination loss is a contrastive loss calculated based on a discrimination image. The contrastive loss is a loss used in distance learning (similarity learning). The contrastive loss is a loss used to bring data of the same class closer to each other and to move data of different classes farther apart. A known calculation formula can be used for calculating the contrastive loss. In this embodiment, the discriminator training unit 103 calculates the contrastive discrimination loss based on the following formula 1. The discriminator training unit 103 trains the discriminator so that the contrastive discrimination loss is small. Formula 1 is an example of a formula for evaluating the difference between an anchor discrimination image and a positive discrimination image, and the difference between an anchor discrimination image and a negative discrimination image.

[0059]

number

[0060] For example, the function C in Equation 1 is defined by the following Equation 2. T is the anchor identification vector. + is a positive discriminant vector. - is a negative discrimination vector. τ in Equation 2 is a temperature parameter indicating the influence of the positive discrimination image and the negative discrimination image when calculating the contrast discrimination loss. For example, τ is 0.05. τ may be any value and is not limited to the example of this embodiment.

[0061]

number

[0062] The formula for calculating the contrastive discrimination loss is not limited to Formula 2. For example, learning does not need to be performed so that a certain anchor discrimination image is separated from other anchor discrimination images. Alternatively, for example, a formula without the temperature parameter τ may be used. The formula for calculating the contrastive discrimination loss may be another formula used in the triplet margin loss method. Furthermore, the classifier training unit 103 may calculate a batchwise discrimination loss related to the average of the contrastive discrimination losses of each of a plurality of anchor discrimination images. The batchwise discrimination loss is a loss that takes into account the batch size of the discrimination images (the number of images used in training).

[0063] In this embodiment, the classifier training unit 103 causes the classifier to estimate the authenticity of the anchor identification image and the authenticity of the generated image generated by the generator, and trains the classifier further based on the estimation result of the authenticity of the anchor identification image and the estimation result of the authenticity of the generated image generated by the generator. The classifier training unit 103 trains the classifier so that the probability that the anchor identification image is estimated to be true increases and the probability that the generated image generated by the generator is estimated to be false increases.

[0064] For example, the classifier training unit 103 calculates the oppositional classification loss based on the following formula 3. The oppositional classification loss is a classification loss used in classifier training. The classifier training unit 103 trains the classifier based on the oppositional classification loss. The classifier training unit 103 trains the classifier so that the oppositional classification loss becomes small. D() in formula 3 is the result of estimation of true or false by the classifier (for example, true is 0 and false is 1). For example, D(x) is the result of estimation of the anchor classification image by the classifier. G(z) is a generated image generated by the generator. D(G(z)) is the result of estimation of the generated image by the classifier. The E in the blackboard letters is an expected value. p data ,p z are the distributions of the identification image and the latent code, respectively. z is the latent code. For example, the latent code is random noise that follows a Gaussian distribution. The mean and variance of the Gaussian distribution can be any value, for example, the mean is 0 and the variance is 1.

[0065]

number

[0066] The formula for calculating the opponent classification loss is not limited to Formula 3. The opponent classification loss may be any loss that allows the generator and the classifier to train in a competitive manner. For example, the opponent classification loss may be multiplied by a coefficient according to the batch size. The formula for calculating the opponent classification loss may be any other formula used in known GANs. The opponent classification loss does not have to be used in training the generator.

[0067] In this embodiment, the classifier training unit 103 causes a classifier to estimate the authenticity of each of the multiple anchor identification images, normalizes the estimation results of the authenticity of each of the multiple anchor identification images, and trains the classifier based on the results of the normalization. In this embodiment, an example is given in which R1 normalization is performed, but various known methods can be used for the normalization itself. For example, other normalization methods such as R2 normalization or elastic net regularization may be performed. For example, the classifier training unit 103 calculates the normalized classification loss based on the following equation 4. The classifier training unit 103 trains the classifier based on the normalized classification loss. The classifier training unit 103 trains the classifier so as to reduce the normalized classification loss.

[0068]

number

[0069] In this embodiment, the classifier training unit 103 calculates the final loss based on the following formula 5. The classifier training unit 103 trains the classifier based on the final loss. The classifier training unit 103 trains the classifier so that the final loss becomes small. λ in formula 5 cl and λ R1 are hyperparameters. The hyperparameters are determined according to the relative importance of each loss. For example, λ cl may be 0.05.R1 may be 10. Thus, the coefficient of the contrastive discriminative loss may be smaller than the coefficient of the opposing discriminative loss. Furthermore, the coefficient of the normalized discriminative loss may be larger than the coefficient of the contrastive discriminative loss.

[0070]

number

[0071] [3-2. Functions for Generator Training] 6, 8, and 9 are diagrams showing an example of functions for training a generator. The functions for training a generator will be described with reference to FIGS. 6 to 9, using the diagram showing the functions of the learning terminal 10 shown in FIG. 3. In this embodiment, two examples of generator training methods will be described: a training method using distance learning and a training method using spectral normalization. FIGS. 6 and 7 show an example of the training method using distance learning. FIGS. 8 and 9 show an example of the training method using spectral normalization.

[0072] As shown in FIG. 3, for example, the learning terminal 10 includes, as functions for learning the generator, a data storage unit 100, a latent code acquisition unit 104, a generated image generation unit 105, a generator learning unit 106, and a partial code acquisition unit 107. The data storage unit 100 is realized by the memory unit 12. The latent code acquisition unit 104, the generated image generation unit 105, the generator learning unit 106, and the partial code acquisition unit 107 are each realized by the control unit 11. With these functions, the learning system 1 learns a GAN generator in which multiple features related to the generated image can be controlled by the user.

[0073] [Data storage section] The data storage unit 100 stores data necessary for training the generator. For example, the data storage unit 100 stores actual data of the generator included in the actual data of the GAN and a generated image database DB2. The actual data of the generator indicates the generator program and parameters of the generator. The generator parameters are referenced by the generator program. The generator parameters are adjusted by the generator training unit 106. For example, the generator parameters are weight coefficients and biases. The generator parameters may be known parameters. For example, the generator parameters may be the number of hidden layers, the number of units in the hidden layers, or other hyperparameters.

[0074] FIG. 7 is a diagram showing an example of the generated image database DB2. The generated image database DB2 is a database that stores generated images, which are images for training the generator. In this embodiment, three generated images, namely, anchor generated images, shape generated images, and appearance generated images, are used in training the generator. Hereinafter, when there is no need to distinguish between anchor generated images, shape generated images, and appearance generated images, they will simply be referred to as generated images. The generated images can also be referred to as training images used as training data for the generator. The generated image database DB2 stores generated images generated by the generated image generation unit 105. The learning terminal 10 can acquire the generated images stored in the generated image database DB2 at any time.

[0075] For example, the generated image database DB2 stores image data for each of the anchor generated image, the shape generated image, and the appearance generated image. Any data may be stored in the generated image database DB2. The data stored in the generated image database DB2 is not limited to the example of FIG. 7. For example, the generated image database DB2 may store information indicating which part of the latent code corresponds to which feature was changed to generate the shape identification image and the appearance identification image.

[0076] An anchor-generated image is a generated image generated based on a latent code indicating user-controllable features. Hereinafter, when it is necessary to express an individual anchor-generated image in a mathematical formula, it will be written as G(z). z is the latent code input to the generator. Hereinafter, the latent code for generating an anchor-generated image will be referred to as an anchor latent code. In the example of FIG. 6, in this embodiment, the anchor latent code is divided into parts corresponding to individual features. Each part corresponds to a user-controllable feature. In this embodiment, since two features, shape and appearance, are controlled, the anchor latent code is divided into two. For example, of the total dimensions of the anchor latent code, up to half the dimensions correspond to shape, and the remaining dimensions correspond to appearance.

[0077] The shape generated image is an image generated based on a latent code in which a portion of the anchor latent code corresponding to the shape has been changed. Hereinafter, this latent code will be referred to as a shape latent code. In this embodiment, the shape generated image is an image generated based on a shape latent code in which a portion of the anchor latent code corresponding to one of a plurality of user-controllable features has been changed. However, the shape generated image may also be an image generated based on a shape latent code in which a portion of the anchor latent code corresponding to each of a plurality of features has been changed. Hereinafter, when it is necessary to express each shape latent code in a mathematical formula, z g When it is necessary to express each shape generation image in a mathematical formula, G(z g ) should be written.

[0078] In this embodiment, the shape generating image is used as a positive generating image in the feature space corresponding to the shape. The positive generating image is a generated image that has similar features to the anchor generating image. On the other hand, the appearance generating image is used as a negative generating image in the feature space corresponding to the shape. The negative generating image is a generated image that has different features from the anchor generating image.

[0079] The appearance generated image is an image generated based on a latent code in which a portion of the anchor latent code corresponding to the appearance has been changed. Hereinafter, this latent code will be referred to as the appearance latent code. In this embodiment, the appearance generated image is an image generated based on an appearance latent code in which a portion of the anchor latent code corresponding to one of a plurality of user-controllable features has been changed. However, the appearance generated image may also be an image generated based on an appearance latent code in which a portion of the anchor latent code corresponding to each of a plurality of features has been changed. Hereinafter, when it is necessary to express each appearance latent code in a mathematical formula, z a When it is necessary to express each appearance generation image in a mathematical formula, G(z a In this embodiment, the appearance generating image is used as a positive generating image in the feature space corresponding to the appearance, while the shape generating image is used as a negative generating image in the feature space corresponding to the appearance.

[0080] The data stored in the data storage unit 100 is not limited to the above example. The data storage unit 100 may store any data. For example, the data storage unit 100 may store a generator learning program that indicates a series of processes in learning the generator. Program codes are shown that indicate each of the processes of a part of the classifier learning unit 103, the process of the latent code acquisition unit 104, the process of the generated image generation unit 105, the process of the generator learning unit 106, and the process of the partial code acquisition unit 107. For example, the data storage unit 100 stores a program for changing the latent code.

[0081] [Latent Code Acquisition Section] The latent code acquisition unit 104 acquires a latent code. For example, the latent code acquisition unit 104 acquires randomly generated noise as the latent code. The method of randomly generating noise may be a known method. For example, the latent code acquisition unit 104 may acquire the latent code based on a probability distribution (e.g., Gaussian distribution). The probability distribution may be a normal distribution or may not be a normal distribution. The probability distribution data is assumed to be pre-stored in the data storage unit 100. The latent code acquisition unit 104 may acquire the latent code based on a program that generates random numbers instead of a probability distribution.

[0082] For example, the latent code acquisition unit 104 acquires an anchor latent code and a plurality of feature latent codes, each of which corresponds to a plurality of features and is a feature latent code obtained by modifying a portion of the anchor latent code corresponding to the feature. In this embodiment, two features, shape and appearance, are used, so a shape latent code and an appearance latent code are acquired. The portions describing the shape latent code and the appearance latent code can be read as feature latent codes. Whether the shape latent code and the appearance latent code are acquired as positive latent codes or negative latent codes depends on which feature space is used in training the generator.

[0083] For example, the latent code acquisition unit 104 acquires the anchor latent code based on random noise. The latent code acquisition unit 104 may also acquire the anchor latent code based on a predetermined probability distribution. The latent code acquisition unit 104 acquires the shape latent code by changing a part of the anchor latent code that corresponds to the shape. In the example of FIG. 6, the latent code acquisition unit 104 acquires the shape latent code by changing elements of the first half dimensions of the anchor latent code (for example, if the anchor latent code has 100 dimensions, dimensions 1 to 50).

[0084] For example, the latent code acquisition unit 104 acquires an appearance latent code by changing a part of the anchor latent code corresponding to appearance. In the example of FIG. 6, the latent code acquisition unit 104 acquires an appearance latent code by changing elements of the latter dimension of the anchor latent code (for example, if the anchor latent code has 100 dimensions, dimensions 51 to 100). Note that the number of dimensions of each feature does not need to be the same. For example, if the anchor latent code has 100 dimensions, the part corresponding to shape may have 40 dimensions and the part corresponding to appearance may have 60 dimensions.

[0085] For example, when a feature space corresponding to a shape is used in training the generator, the latent code acquisition unit 104 acquires, as a positive latent code, a shape latent code in which a part corresponding to a shape has been changed among the anchor latent codes, and acquires, as a negative latent code, an appearance latent code in which a part corresponding to an appearance has been changed among the anchor latent codes.

[0086] For example, when a feature space corresponding to appearance is used in training the generator, the latent code acquisition unit 104 acquires, as a positive latent code, an appearance latent code in which a part corresponding to appearance has been changed among the anchor latent codes, and acquires, as a negative latent code, a shape latent code in which a part corresponding to shape has been changed among the anchor latent codes.

[0087] In this embodiment, the case where the latent code acquisition unit 104 generates the positive latent code and the negative latent code is exemplified, but the positive latent code and the negative latent code may be generated by a computer other than the learning terminal 10. The generated image database DB2 may store the positive latent code and the negative latent code generated by another computer. In this case, the learning terminal 10 does not need to have the function of generating the positive latent code and the negative latent code.

[0088] [Image generation section] The generated image generation unit 105 causes a generator to generate a generated image based on the latent code. In a training method using distance learning, at least a part of the generator may be similar to a known architecture, but the training method using distance learning has a novel configuration. The generator architecture in the training method using spectral normalization shown in FIG. 8 has a novel configuration that has not been seen before. Hereinafter, the training method using distance learning will be described first, followed by a training method using spectral normalization.

[0089] For example, the generated image generation unit 105 generates an anchor generated image based on the anchor latent code and the generator. The generated image generation unit 105 generates a plurality of feature generated images based on each of a plurality of feature latent codes and the generator. Whether each of the plurality of feature generated images is used as a positive generated image or a negative generated image depends on which feature space is used in training the generator. A positive generated image is an image generated based on a positive latent code. A negative generated image is an image generated based on a negative latent code.

[0090] In this embodiment, in addition to anchor generated images, shape generated images and appearance generated images are also generated. Each of the shape generated images and appearance generated images is an example of a feature generated image. Therefore, the description of each of the shape generated images and appearance generated images can be read as feature generated images. Whether each of the shape generated images and appearance generated images is acquired as a positive generated image or a negative generated image depends on which feature space is used in training the generator. In the example of FIG. 6, the anchor generated image is indicated by the symbol G(z). The shape generated image is represented by G(z g The appearance generated image is denoted by G(z a ) is indicated by the symbol.

[0091] For example, the generated image generation unit 105 inputs an anchor latent code to a generator undergoing training. The generator converts the anchor latent code into an intermediate anchor latent code as needed based on a mapping network. This conversion does not have to be performed. The generator performs processing such as convolution on the anchor latent code and outputs an image corresponding to the result of the processing as an anchor generated image. This series of processing may be similar to the internal processing of a known GAN generator. The generated image generation unit 105 stores the anchor generated image in a generated image database DB2.

[0092] For example, the generated image generation unit 105 inputs a shape latent code to a generator undergoing training. The generator converts the shape latent code to an intermediate shape latent code as needed based on a mapping network. This conversion is optional. The generator performs processing such as convolution on the shape latent code and outputs an image corresponding to the result of the processing as a shape generated image. The generated image generation unit 105 stores the shape generated image in the generated image database DB2.

[0093] For example, the generated image generation unit 105 inputs the appearance latent code to a generator undergoing training. The generator converts the appearance latent code into an intermediate appearance latent code as needed based on a mapping network. This conversion does not have to be performed. The generator performs processing such as convolution on the appearance latent code and outputs an image corresponding to the result of the processing as an appearance generated image. The generated image generation unit 105 stores the appearance generated image in the generated image database DB2.

[0094] [Generator learning part] The generator training unit 106 trains the generator based on the generated image generated by the generated image generation unit 105 and a trained classifier. In this embodiment, the generator training unit 106 performs training based on the classifier trained by the classifier training unit 103. Note that the generator training unit 106 may also train the generator based on a classifier that has completed training using a method different from the method described for the classifier training unit 103.

[0095] For example, the generator training unit 106 causes a classifier to estimate the authenticity of the anchor-generated image, and further trains the generator based on the estimation result of the authenticity of the anchor-generated image. For example, the generator training unit 106 trains the generator based on the estimation result of the authenticity of each of the multiple anchor-generated images. The generator training unit 106 calculates the opposing generation loss based on the following equation (6). The opposing classification loss is an opposing loss used in training the generator. The generator training unit 106 trains the generator based on the opposing generation loss. The generator training unit 106 trains the generator so that the opposing generation loss is small. The meanings of the symbols included on the right side of the following equation (6) are as described above. The generator training unit 106 trains the generator based on the opposing generation loss so that the generator generates a generated image that deceives the classifier.

[0096]

number

[0097] The formula for calculating the opposing generation loss is not limited to Formula 6. The opposing generation loss may be any loss that allows the generator and the classifier to train in a competitive manner. For example, the opposing generation loss may be calculated taking into account the batch size of generated images. The formula for calculating the opposing generation loss may be any other formula used in known GANs. The opposing generation loss does not have to be used in training the generator.

[0098] For example, the generator training unit 106 calculates, for each feature space corresponding to each of the multiple features, an anchor generation vector for the anchor generation image, a positive generation vector for the feature generation image corresponding to the feature, and a negative generation vector for the feature generation image corresponding to another feature, based on the classifier. The generator training unit 106 performs training so that, in the feature space corresponding to each of the multiple features, the anchor generation vector and the positive generation vector become closer to each other and the anchor generation vector and the negative generation vector become farther apart. In other words, the generator training unit 106 trains the generator so that, in the feature space corresponding to each of the multiple features, the anchor generation vector and the positive generation vector belong to the same cluster and the anchor generation vector and the negative generation vector do not belong to the same cluster.

[0099] The learning algorithm itself may be a known algorithm used in distance learning. For example, algorithms such as gradient descent, gradient penalty, stochastic gradient flow, or variational inference may be used to learn the generator. In this embodiment, the relationship between the three generated images is used in learning, so the generator learning unit 106 learns the generator based on a triplet margin loss loss function so that the anchor generated vector and the positive generated vector become closer and the anchor generated vector and the negative generated vector become farther apart. The loss function described in this embodiment is an example of a triplet margin loss loss function.

[0100] For example, the generator learning unit 106 performs learning of the generator by adjusting the parameters of the generator so that the anchor generating vector and the positive generating vector become closer and the anchor generating vector and the negative generating vector become farther apart in the feature space corresponding to each of the multiple features. The generator learning unit 106 calculates a loss indicating the closeness between the anchor generating vector and the positive generating vector and the farthest between the anchor generating vector and the negative generating vector in the feature space corresponding to each of the multiple features based on a loss function in distance learning, and adjusts the parameters of the generator so that the loss becomes smaller.

[0101] Hereinafter, a feature vector calculated based on a shape-generated image is referred to as a shape-generated vector. A feature vector calculated based on an appearance-generated image is referred to as an appearance-generated vector. Each of the shape-generated vector and the appearance-generated vector is an example of a feature-generated vector, which is a feature vector calculated based on a feature-generated image. Therefore, any description of each of the shape-generated vector and the appearance-generated vector can be read as a feature-generated vector. Whether each of the shape-generated vector and the appearance-generated vector is acquired as a positive or negative generated vector depends on which feature space is used in training the generator.

[0102] For example, in the shape feature space corresponding to the shape, the shape generating image is a positive generating image, and the appearance generating image is a negative generating image. Therefore, the generator learning unit 106 generates a shape generating vector (h in FIG. 6) based on the shape generating image and the projection head of the shape. g (G(z g The generator learning unit 106 calculates the appearance generating vector (h in FIG. 6) based on the appearance generating image and the projection head of the shape. g (G(z a )) is calculated as the negative generating vector of the shape.

[0103] For example, the generator learning unit 106 calculates the anchor generation vector (h in FIG. 6) of the shape based on the anchor generation image and the projection head of the shape. g The generator training unit 106 performs training of the generator so that the anchor generating vector of the shape and the shape generating vector become closer to each other and the anchor generating vector of the shape and the appearance generating vector become farther apart in the shape feature space corresponding to the shape.

[0104] For example, in the appearance feature space corresponding to the appearance, the appearance generated image is a positive generated image, and the shape generated image is a negative generated image. Therefore, the generator learning unit 106 generates an appearance generated vector (h in FIG. 6) based on the appearance generated image and the projection head of the appearance. a (G(z a The generator learning unit 106 calculates a shape generating vector (h in FIG. 6) based on the shape generating image and the projection head of the appearance. a (G(z g )) is calculated as the negative generative vector of the appearance.

[0105] For example, the generator learning unit 106 calculates an appearance anchor generation vector (h in FIG. 6) based on the anchor generation image and the appearance projection head. a The generator training unit 106 performs training of the generator so that the anchor generating vector of the appearance and the appearance generating vector become closer to each other and the anchor generating vector of the appearance and the shape generating vector become farther apart in the appearance feature space corresponding to the appearance.

[0106] In this embodiment, the generator training unit 106 calculates, for each anchor generated image, a contrastive generation loss related to the closeness between the anchor generated vector of the anchor generated image and the positive generated vector of the positive generated image of the anchor generated image, and the closeness between the anchor generated vector of the anchor generated image and the negative generated vector of the negative generated image of the anchor generated image. The generator training unit 106 trains the generator based on the contrastive generation loss. The generator training unit 106 trains the generator so as to reduce the contrastive generation loss.

[0107] The contrast generation loss is a contrast loss calculated based on a generated image. The concept of the contrast generation loss is generally similar to that of the contrast discrimination loss. In this embodiment, the generator training unit 106 calculates the contrast generation loss based on the following formula 7. Formula 7 is an example of a formula for evaluating the difference between the anchor generated image and the positive generated image, and the difference between the anchor generated image and the negative generated image. The meanings of the symbols on the right side of Formula 7 are as described above.

[0108]

number

[0109] The generator training method described above is an example of a training method using distance learning. Hereinafter, a training method using spectral normalization will be described with reference to FIGS. 8 and 9. The training terminal 10 may train the generator by combining a training method using distance learning and a training method using spectral normalization, or may train the generator by using only either a training method using distance learning or a training method using spectral normalization. For simplicity of explanation, the training method using spectral normalization will be described first without assuming the training method using distance learning.

[0110] If distance learning is not used, the latent code acquisition unit 104 does not acquire positive latent codes and negative latent codes. In FIG. 6, the anchor latent code is indicated by the symbol z. However, if the latent code acquisition unit 104 does not acquire positive latent codes and negative latent codes, there is no need to distinguish between the latent codes. Therefore, the symbol z is simply referred to as the latent code. The method for acquiring the latent code by the latent code acquisition unit 104 is as described above. For example, the latent code acquisition unit 104 may acquire the latent code z based on a Gaussian distribution N(0,I). I may be any value (e.g., 1).

[0111] [Partial code acquisition section] The partial code acquisition unit 107 acquires a plurality of partial codes corresponding to the plurality of features, respectively, based on a latent code for generating a generated image and a plurality of mapping networks corresponding to the plurality of features. The partial code is a part of the latent code. The partial code acquisition unit 107 divides the latent code into portions corresponding to the plurality of features, and then inputs and converts each of the portions of the plurality of features into the mapping network corresponding to the feature, thereby acquiring the plurality of partial codes. In this embodiment, two features, shape and appearance, are used, so the partial code acquisition unit 107 acquires a partial code corresponding to shape and a partial code corresponding to appearance based on the latent code, the mapping network corresponding to shape, and the mapping network corresponding to appearance.

[0112] A mapping network is a program for transforming latent code. A mapping network converts latent code into a more meaningful intermediate representation. For example, a mapping network transforms latent code using multiple fully connected layers. The transformed latent code is a partial code. Mapping networks are used in architectures such as StyleGAN. While conventional GANs generate generated images directly from latent code, mapping networks improve the process of generating generated images. Transformation by a mapping network makes it easier to control specific features. For example, a mapping network may be a neural network with multiple layers (e.g., eight layers).

[0113] The example in FIG. 8 illustrates an example of the architecture of the generator. For example, the generator includes a mapping network corresponding to shape and a mapping network corresponding to appearance. The generator includes the same number of mapping networks as the number of user-controllable features. For example, if the user can control three or more features, the generator includes three or more mapping networks. The initially obtained latent code z follows a Gaussian distribution N(0,I). The partial code acquisition unit 107 divides the latent code z into the same number of parts as the number of user-controllable features, and then inputs each divided part into the mapping network.

[0114] For example, the partial code acquisition unit 107 acquires a part of the latent code z (in the example of FIG. 8, the first half of all dimensions. For example, if the latent code z has 100 dimensions, the part from dimension 1 to dimension 50) as a shape partial code, which is a partial code corresponding to the shape. The partial code acquisition unit 107 inputs the shape partial code into a mapping network corresponding to the shape, and acquires the final shape partial code (in FIG. 8, w g As will be described later, the shape part codes may be transformed by a covariance matrix and then input to a shape-corresponding mapping network.

[0115] For example, the partial code acquisition unit 107 acquires the remaining part of the latent code z (in the example of FIG. 8, the latter half of all dimensions. For example, if the latent code z is 100 dimensions, the part from dimension 51 to dimension 100) as an appearance partial code, which is a partial code corresponding to the appearance. The partial code acquisition unit 107 inputs the appearance partial code into a mapping network corresponding to the appearance, and acquires the final appearance partial code (in FIG. 8, w a ) to obtain the appearance part code. As will be described later, the appearance part code may be transformed by a covariance matrix and then input to a mapping network corresponding to the appearance.

[0116] In this embodiment, the partial code acquisition unit 107 acquires a first latent code based on a predetermined probability distribution and converts the first latent code to a second latent code based on parameters that can be adjusted through learning. FIG. 9 illustrates an example of the process of converting a first latent code to a second latent code. The partial code acquisition unit 107 converting a first latent code to a second latent code as shown in FIG. 9 corresponds to acquiring a second latent code that follows a Gaussian distribution N(0,Σ). Σ is a learnable covariance matrix. The learnable covariance matrix is ​​one of the parameters of the generator.

[0117] In this embodiment, the learnable covariance matrix is ​​obtained by eigendecomposition as follows: Σ=UDU T For example, let us consider the case where Σ is decomposed into U. U is a matrix in which the eigenvectors of the covariance matrix are arranged as column vectors. For example, U may include orthonormal vectors. D is a diagonal matrix with the eigenvalues ​​of Σ as diagonal components. U T is a matrix in which the rows and columns of U are swapped (the matrix in which U is transposed). Eigendecomposition of the learnable covariance matrix may be performed by known processing, and the formula for eigendecomposition is not limited to the above example. For example, eigendecomposition may be performed by an inverse matrix rather than the transpose of U.

[0118] For example, the learnable parameters of the generator are the base vectors and lengths of each axis of the vector space, V={v1,...,v k}. Also, vk ={v k 1 ,...,v k k}. d={d1,...,d k}. These can also be said to be parameters in the first layer of the mapping network. These parameters are converted into a square matrix with the same number of rows and columns, and a diagonal matrix in which all elements except for those on the diagonal are 0. The converted parameters are expressed by the following formulas 8 and 9. The blackboard letter R in formulas 8 and 9 means the set of real numbers.

[0119]

number

[0120]

number

[0121] Hereinafter, the second latent code acquired by the Gaussian distribution N(0,Σ) will be referred to as z(bar). Since the format of the specification does not allow a bar to be placed above the symbol, the (bar) in z(bar) is a schematic representation of the bar at the top of the symbol. As shown in FIG. 9, the partial code acquisition unit 107 calculates z(bar)=UD 1 / 2 The first latent code z is converted into the second latent code z(bar) by the formula z. D 1 / 2 is a square root matrix of the diagonal matrix D. The partial code acquisition unit 107 obtains a column vector U and a square root matrix D of the diagonal matrix D. 1 / 2 The second latent code z(bar) is obtained by multiplying z(bar) with the first latent code z(bar) obtained from the Gaussian distribution Z(0,I).

[0122] For example, the partial code acquisition unit 107 acquires multiple partial codes based on the second latent code and multiple mapping networks. A portion of the second latent code z (bar) (for example, the first half of all dimensions; if the second latent code z (bar) has 100 dimensions, the portion from dimension 1 to dimension 50) may be acquired as the shape partial code. The partial code acquisition unit 107 may input the portion of the second latent code z (bar) to a mapping network corresponding to the shape, and acquire the partial code output from the mapping network as the shape partial code.

[0123] For example, the partial code acquisition unit 107 may acquire the remaining part of the second latent code z (bar) (for example, the latter half of all dimensions. If the second latent code z (bar) has 100 dimensions, the part from dimension 51 to dimension 100) as the appearance partial code. The partial code acquisition unit 107 may input the remaining part of the second latent code z (bar) to a mapping network corresponding to the appearance, and acquire the partial code output from the mapping network as the appearance partial code.

[0124] As described above, the generator parameter may be a learnable covariance matrix. The partial code acquisition unit 107 converts the first latent code (z in FIG. 9) into the second latent code (z(bar) in FIG. 9) based on the learnable covariance matrix. For example, the predetermined probability distribution may be an isotropic Gaussian distribution (N(0,I) in FIG. 9). The isotropic Gaussian distribution is a Gaussian distribution with uniform distribution in each direction, as shown in FIG. 9. The partial code acquisition unit 107 acquires the first latent code based on the isotropic Gaussian distribution, and converts the first latent code into a second latent code based on the learnable covariance matrix, thereby acquiring the second latent code that follows an anisotropic Gaussian distribution. The anisotropic Gaussian distribution is expressed as z(bar)=UD in FIG. 9. 1 / 2 Like the z distribution, it is a Gaussian distribution, which varies depending on the direction.

[0125] In this embodiment, the generated image generation unit 105 generates a generated image based on an image synthesis network that generates a generated image using multiple partial codes. FIG. 8 shows an example of the architecture of the image synthesis network. The "4x4x512 Const" at the beginning of the image synthesis network is the tensor the generator initially handles. Here, a tensor with a spatial size of 4x4 and 512 channels is used as an example. "Const" means that the tensor has a fixed value. While the example shows a case where the value of the tensor the generator initially handles is fixed, various known tensors can be used for the tensor itself. For example, the tensor may have a randomly varying value instead of a fixed value. The size and number of channels of the tensor may also be arbitrary and are not limited to those of this embodiment.

[0126] For example, the generated image generation unit 105 generates a generated image by repeatedly performing convolution and upsampling in an image synthesis network based on multiple partial codes and an initial feature map in the generator. In the architecture of FIG. 8, the image synthesis network includes multiple synthesis blocks. Each of the multiple partial codes is input to each synthesis block. Each synthesis block includes a convolution layer and a layer that performs at least one of upsampling. Processing of each synthesis block gradually upsamples tensors to increase spatial resolution. Ultimately, an m×m (m is an arbitrary integer; in the example of FIG. 8, m is an integer that is a multiple of 4)×64 generated image is generated. The size and number of channels of the generated image may also be arbitrary and are not limited to the example of this embodiment.

[0127] In this embodiment, the generator training unit 106 trains a generator including multiple mapping networks and an image synthesis network based on the generated image and a trained GAN classifier. For example, the generator training unit 106 performs training based on a spectral loss function indicating that the smaller the vector distance between multiple partial codes, the smaller the loss. When a learnable covariance matrix is ​​used as a parameter, the generator training unit 106 performs training by adjusting the value of the learnable covariance matrix based on the spectral loss function.

[0128] For example, the spectral loss function is calculated by the following Equation 10. ||d on the right side of Equation 10 g ||1 is the L1 norm (sum of absolute values) of the vector indicated by the shape part code. ||d a ||1 is the L1 norm (sum of absolute values) of the vector indicated by the appearance part code. The generator training unit 106 trains the generator so as to reduce the spectral loss function.

[0129]

number

[0130] The calculation formula for the spectral loss is not limited to Equation 10. The spectral loss may be calculated using a function that reduces the loss as the absolute value of the partial code decreases. For example, ||d g ||1 and ||d a A coefficient may be set to at least one of || 1. The absolute values ​​of the shape part code and the appearance part code may be evaluated by a calculation method other than the L1 norm.

[0131] As described above, the generator may be trained using a training method that utilizes spectral normalization. When a training method that utilizes distance learning and a training method that utilizes spectral normalization are combined, the partial code acquisition unit 107 may acquire a plurality of anchor partial codes that are a plurality of partial codes based on the anchor latent code, and a plurality of feature partial codes that are a plurality of partial codes based on each of the plurality of feature latent codes. The anchor partial codes are the same as the partial codes already described.

[0132] Note that feature partial codes differ from anchor partial codes in that they are obtained based on feature latent codes. However, the method for obtaining feature partial codes from feature latent codes is similar to the method for obtaining anchor partial codes from anchor partial codes. For example, the partial code obtaining unit 107 obtains each of a plurality of feature latent codes as a first latent code and converts each of the plurality of feature latent codes into a second latent code based on a learnable covariance matrix. The partial code obtaining unit 107 divides the second latent codes converted from each of the plurality of feature latent codes into features and inputs the second latent codes into each of a plurality of mapping networks, thereby obtaining a plurality of feature partial codes. The partial code obtaining unit 107 generates a plurality of feature partial codes for each feature latent code. For example, the partial code obtaining unit 107 obtains a shape partial code and an appearance partial code, which are a plurality of feature partial codes corresponding to a shape latent code. The partial code obtaining unit 107 obtains a shape partial code and an appearance partial code, which are a plurality of feature partial codes corresponding to an appearance latent code.

[0133] For example, the generated image generation unit 105 acquires an anchor generated image, which is a generated image based on a plurality of anchor part codes, and a plurality of feature generated images, which are generated images based on a plurality of feature part codes and correspond to a plurality of features, respectively. The anchor generated image is similar to the generated image already described. The feature generated image differs from the anchor generated image in that it is acquired based on a plurality of feature part codes, but the method of acquiring the feature generated image from a plurality of feature part codes is similar to the method of acquiring the anchor generated image from a plurality of anchor part codes.

[0134] For example, the generated image generation unit 105 inputs, for each feature, each of a plurality of feature part codes corresponding to the feature into a synthesis block and repeats upsampling one after another to obtain a feature generated image. The generated image generation unit 105 inputs each of the shape part codes and appearance part codes obtained from the shape latent code into a synthesis block and repeats upsampling one after another to obtain a shape generated image. The generated image generation unit 105 inputs each of the shape part codes and appearance part codes obtained from the appearance latent code into a synthesis block and repeats upsampling one after another to obtain an appearance generated image. The generator learning unit 106 may perform learning based on these generated images.

[0135] In this embodiment, the generator learning unit 106 calculates the final loss based on the following formula 11. The generator learning unit 106 learns the generator based on the final loss. The generator learning unit 106 learns the generator so that the final loss becomes small. λ in formula 11 cl is a hyperparameter. λ in Equation 11 cl and the value of λ in Eq. cl The values ​​of and may be the same or different from each other.

[0136]

number

[0137] [4. Processing performed by the learning system] 10 and 11 are diagrams showing an example of processing executed by the learning system 1. The processing of Fig. 10 and Fig. 11 is executed by the control units 11, 21, and 31 executing programs stored in the storage units 12, 22, and 32, respectively. In Fig. 10 and Fig. 11, processing for training a classifier, processing for training a generator, and processing for utilizing a trained generator are shown as a series of processing, but these may also be performed separately.

[0138] As shown in FIG. 10, the learning terminal 10 acquires an anchor discrimination image, a shape discrimination image, and an appearance discrimination image based on a discrimination image database DB1 (S1). The learning terminal 10 inputs each of the anchor discrimination image, the shape discrimination image, and the appearance discrimination image to the classifier, and calculates an anchor discrimination vector, a positive discrimination vector, and a negative discrimination vector for each feature space corresponding to each of a plurality of features (S2). The learning terminal 10 trains the classifier based on the execution results of the processes of S1 and S2 (S3). In S3, the learning terminal 10 calculates various losses based on Formulas 1 to 4. The learning terminal 10 calculates the final loss based on Formula 5. The learning terminal 10 trains the classifier so that the final loss is small. Through the above processes, the training of the classifier is completed.

[0139] The learning terminal 10 acquires an anchor latent code, a shape latent code, and an appearance latent code (S4). The learning terminal 10 acquires a shape partial code and an appearance partial code corresponding to the anchor latent code based on the anchor latent code and the mapping network (S5). In S5, the learning terminal 10 transforms and divides the anchor latent code based on a learnable covariance matrix, and inputs the resulting code into a mapping network corresponding to each feature, thereby acquiring the shape partial code and the appearance partial code as the anchor partial code. The learning terminal 10 generates an anchor-generated image based on the shape partial code and the appearance partial code acquired from the anchor latent code and the image synthesis network (S6). In S6, the learning terminal 10 inputs anchor partial codes corresponding to the shape and appearance into each synthesis block, and generates an anchor-generated image by successively performing upsampling.

[0140] The learning terminal 10 acquires a shape part code and an appearance part code corresponding to the shape latent code based on the shape latent code and the mapping network (S7). In S7, the learning terminal 10 converts and divides the shape latent code based on a learnable covariance matrix, and inputs the converted code into a mapping network corresponding to each feature to acquire the shape part code and the appearance part code. The learning terminal 10 generates a shape-generated image based on the shape part code and the appearance part code acquired from the shape latent code and the image synthesis network (S8). In S8, the learning terminal 10 inputs the shape part code and the appearance part code into each synthesis block and generates a shape-generated image by successively performing upsampling.

[0141] The learning terminal 10 acquires a shape part code and an appearance part code corresponding to the appearance latent code based on the appearance latent code and the mapping network (S9). In S9, the learning terminal 10 converts and divides the appearance latent code based on a learnable covariance matrix, and inputs the converted code into a mapping network corresponding to each feature, thereby acquiring a shape part code and an appearance part code. The learning terminal 10 generates an appearance-generated image based on the shape part code and appearance part code acquired from the appearance latent code and the image synthesis network (S10). In S10, the learning terminal 10 inputs the shape part code and the appearance part code into each synthesis block, and generates an appearance-generated image by successively performing upsampling.

[0142] The learning terminal 10 inputs each of the anchor generated image, shape generated image, and appearance generated image into the classifier, and calculates an anchor generated vector, a positive generated vector, and a negative generated vector for each feature space corresponding to each of the multiple features (S11). The learning terminal 10 trains the generator based on the execution results of the processes of S4 to S11 (S12). In S12, the learning terminal 10 calculates various losses based on Formulas 7 to 10. The learning terminal 10 calculates the final loss based on Formula 11. The learning terminal 10 trains the generator so that the final loss is small.

[0143] 11, the learning terminal 10 transmits the trained generator to the server 20 (S13). In S13, the learning terminal 10 may transmit the trained classifier together with the trained generator to the server 20. When the server 20 receives the trained generator from the learning terminal 10 (S14), it records the trained generator in the storage unit 22. The user terminal 30 transmits feature data indicating the features specified by the user to the server 20 (S15).

[0144] When the server 20 receives the feature data from the user terminal 30 (S16), it generates a latent code according to the feature indicated by the feature data (S17). The server 20 generates a generated image based on the latent code and the trained generator (S18). The server 20 transmits image data indicating the generated image to the user terminal 30 (S19). The user terminal 30 receives the image data from the server 20 (S20). The user terminal 30 displays the generated image on the display unit 35 based on the image data (S21), and this process ends.

[0145] [5. Summary of embodiments] The learning system 1 of this embodiment trains a GAN generator that allows a user to control multiple features related to a generated image. The learning system 1 acquires multiple partial codes corresponding to multiple features based on a latent code for generating the generated image and multiple mapping networks corresponding to the multiple features. The learning system 1 generates the generated image based on an image synthesis network that generates the generated image using the multiple partial codes. The learning system 1 trains the generator based on the generated image and a trained GAN classifier. This allows the learning system 1 to divide the latent code into more detailed meanings, thereby improving the accuracy of the GAN. The learning system 1 can improve the accuracy of the GAN because the GAN can easily recognize which part of the latent code corresponds to which feature based on each partial code. For example, by learning using the partial codes, the GAN can easily recognize that a specific dimension of the latent code corresponds to a specific feature. Therefore, the learning system 1 can improve the accuracy of a GAN that linearly controls a specific feature of a generated image (a GAN that can emphasize or change a specific feature of a generated image). Such a GAN is sometimes referred to as an LC-GAN. By learning using partial codes, the learning system 1 can create an LC-GAN that allows users to intuitively and efficiently manipulate specific features of generated images.

[0146] The learning system 1 also acquires a first latent code based on a predetermined probability distribution. The learning system 1 converts the first latent code into a second latent code based on parameters that can be adjusted through learning. The learning system 1 acquires multiple partial codes based on the second latent code and multiple mapping networks. The learning system 1 performs learning based on a spectral loss function that indicates that the smaller the vector distance between the multiple partial codes, the smaller the loss. The learning system 1 converts the first latent code acquired using the probability distribution into a second latent code appropriate for controlling individual features, thereby enabling the GAN to generate a generated image based on a more appropriate second latent code. Furthermore, the spectral loss function, which focuses on specific dimensions of the partial code, allows the GAN to more easily recognize that a specific dimension in the partial code corresponds to a specific feature, thereby further improving the accuracy of the GAN. For example, if a certain dimension of the partial code corresponds to a certain feature, the value of the element of that dimension is important for controlling the feature, while the values ​​of the elements of other dimensions are less relevant to controlling the feature. In this regard, if the values ​​of elements in other dimensions are relatively large, the GAN may not be able to recognize dimensions that are important to the feature in question. Therefore, learning system 1 can train a GAN that emphasizes dimensions corresponding to the feature to be controlled by using a spectral loss function that reduces the loss as the distance between vectors related to the partial code decreases. For example, learning system 1 can achieve GAN training through unsupervised learning by utilizing a spectral normalization technique.

[0147] The parameter is a learnable covariance matrix. The learning system 1 converts the first latent code into the second latent code based on the learnable covariance matrix. The learning system 1 performs learning by adjusting the value of the learnable covariance matrix based on a spectral loss function. This allows the learning system 1 to control the characteristics of the GAN using specific values ​​in the covariance matrix, thereby further improving the accuracy of the GAN. The covariance matrix enables the GAN to more appropriately recognize the structure of the latent code.

[0148] Furthermore, the predetermined probability distribution is an isotropic Gaussian distribution. The learning system 1 acquires a first latent code based on the isotropic Gaussian distribution. The learning system 1 acquires a second latent code that follows an anisotropic Gaussian distribution by converting the first latent code into a second latent code based on a learnable covariance matrix. This allows the learning system 1 to acquire a partial code based on the second latent code whose distribution is biased in a specific direction in vector space, making it easier for the GAN to recognize that a specific dimension corresponds to a specific feature. As a result, the learning system 1 can further improve the accuracy of the GAN. The GAN can more appropriately recognize the structure of the latent code.

[0149] Furthermore, the learning system 1 generates a generated image by repeatedly performing convolution and upsampling in the image synthesis network based on multiple partial codes and the initial feature map in the generator. This allows the learning system 1 to reflect each partial code in the generated image, thereby generating a generated image that reflects the features desired by the user.

[0150] The learning system 1 also acquires an anchor latent code and a plurality of feature latent codes corresponding to each of the plurality of features. The learning system 1 acquires a plurality of anchor partial codes and a plurality of feature partial codes based on each of the plurality of feature latent codes. The learning system 1 acquires an anchor generation image based on the plurality of anchor partial codes and a plurality of feature generation images corresponding to each of the plurality of features. The learning system 1 calculates, for each feature space corresponding to each of the plurality of features, an anchor generation vector, a positive generation vector related to the feature generation image corresponding to the feature, and a negative generation vector related to the feature generation image corresponding to the other feature, based on the classifier. The learning system 1 performs learning so that, in the feature spaces corresponding to each of the plurality of features, the anchor generation vector and the positive generation vector become closer to each other and the anchor generation vector and the negative generation vector become farther apart from each other. This allows the learning system 1 to further improve the accuracy of the GAN by utilizing a distance learning technique. By utilizing a distance learning technique, the learning system 1 can reduce the number of images that need to be prepared during learning, thereby reducing the effort required for training the generator.

[0151] Furthermore, the learning system 1 has a classifier estimate the authenticity of the anchor-generated image, and further trains the generator based on the estimation result of the authenticity of the anchor-generated image. This allows the learning system 1 to realize training that takes into account the authenticity estimation result of the classifier, further reducing the effort required for GAN training. The accuracy of the generator is also further improved.

[0152] The learning system 1 also acquires an anchor identification image and multiple feature-identification images corresponding to multiple features. Based on the classifier, the learning system 1 calculates, for each feature space corresponding to each of the multiple features, an anchor identification vector, a positive identification vector for the feature-identification image corresponding to the feature, and a negative identification vector for the feature-identification image corresponding to the other feature. The learning system 1 trains the classifier so that, in the feature space corresponding to each of the multiple features, the anchor identification vector and the positive identification vector become closer and the anchor identification vector and the negative identification vector become farther apart. The learning system 1 performs training based on a trained classifier. This allows the learning system 1 to reduce the effort required for training the classifier. For example, the learning system 1 reduces the effort required for training the classifier by eliminating the need to manually label the identified images or use a pre-trained classification model. The learning system 1 also achieves efficient learning by seamlessly preparing images to be trained by the classifier and training the classifier using the images. Through such learning, the learning system 1 also improves the accuracy of the classifier.

[0153] Furthermore, the learning system 1 causes the classifier to estimate the authenticity of the anchor identification image and the authenticity of the generated image generated by the generator. The learning system 1 further trains the classifier based on the estimation result of the authenticity of the anchor identification image and the estimation result of the authenticity of the generated image generated by the generator. This allows the learning system 1 to realize training that takes into account the authenticity estimation result of the classifier, thereby further reducing the effort required for training the GAN. The accuracy of the classifier is also further improved.

[0154] Furthermore, the learning system 1 causes a classifier to estimate the authenticity of each of the multiple anchor identification images. The learning system 1 performs normalization on the estimation results of the authenticity of each of the multiple anchor identification images. The learning system 1 trains the classifier further based on the results of the normalization. This allows the learning system 1 to realize learning that takes into account the normalized estimation results of the authenticity of the classifier, thereby further reducing the effort required for training the GAN. The accuracy of the classifier is also further improved. The learning system 1 can further reduce the effort required for training the GAN.

[0155] [6. Modifications] The present disclosure is not limited to the above-described embodiments, and may be modified as appropriate without departing from the spirit of the present disclosure.

[0156] For example, the process for training the classifier and the process for training the generator may be executed on separate computers. A first learning terminal 10 may execute the process for training the classifier, and a second learning terminal 10 may execute the process for training the generator. For example, the process described as being executed on the learning terminal 10 may be executed on the server 20, the user terminal 30, or another computer. The process described as being executed on the learning terminal 10 may be shared among multiple computers.

[0157] [7. Notes] For example, the learning system can be configured as follows: (1) A learning system for training a generator of a GAN (Generative Adversarial Network) in which a user can control multiple features related to the generated image, a partial code acquisition unit that acquires a plurality of partial codes corresponding to the plurality of features based on a latent code for generating the generated image and a plurality of mapping networks corresponding to the plurality of features; a generated image generator that generates the generated image based on an image synthesis network that uses the plurality of partial codes to generate the generated image; A generator training unit that trains the generator, which includes the plurality of mapping networks and the image synthesis network, based on the generated image and the trained classifier of the GAN; A learning system including. (2) The partial code acquisition unit obtaining a first latent code based on a predetermined probability distribution; converting the first latent code to a second latent code based on the adjustable parameter; obtaining the plurality of partial codes based on the second latent code and the plurality of mapping networks; the generator learning unit performs the learning based on a spectral loss function indicating that the loss decreases as the distance between vectors related to the plurality of partial codes decreases. (1) A learning system according to the present invention. (3) The parameters are learnable covariance matrices, the partial code acquisition unit converts the first latent code into the second latent code based on the learnable covariance matrix; The generator learning unit performs the learning by adjusting the value of the learnable covariance matrix based on the spectral loss function. (2) A learning system according to the present invention. (4) the predetermined probability distribution is an isotropic Gaussian distribution, the partial code acquisition unit acquires the first latent code based on the isotropic Gaussian distribution, and converts the first latent code into the second latent code based on the learnable covariance matrix, thereby acquiring the second latent code that follows an anisotropic Gaussian distribution. (3) A learning system according to the present invention. (5) The generated image generation unit generates the generated image by causing the image synthesis network to repeatedly perform convolution and upsampling based on the plurality of partial codes and an initial feature map in the generator. A learning system according to any one of (1) to (4). (6) the learning system further includes a latent code acquisition unit that acquires an anchor latent code and a plurality of feature latent codes obtained by changing a portion of the anchor latent code corresponding to the feature, the feature latent codes corresponding to the plurality of features, respectively; the partial code acquisition unit acquires a plurality of anchor partial codes, which are the plurality of partial codes based on the anchor latent code, and a plurality of feature partial codes, which are the plurality of partial codes based on each of the plurality of feature latent codes; the generated image generation unit acquires an anchor generated image that is the generated image based on the plurality of anchor part codes, and a plurality of feature generated images that are the generated image based on the plurality of feature part codes, the feature generated images corresponding to the plurality of features, respectively; The generator learning unit calculating, for each feature space corresponding to each of the plurality of features based on the classifier, an anchor generation vector for the anchor generation image, a positive generation vector for the feature generation image corresponding to the feature, and a negative generation vector for the feature generation image corresponding to another feature; The learning is performed so that, in the feature space corresponding to each of the plurality of features, the anchor generating vector and the positive generating vector become closer to each other and the anchor generating vector and the negative generating vector become farther apart from each other. A learning system according to any one of (1) to (5). (7) the generator learning unit causes the classifier to estimate the authenticity of the anchor-generated image, and further learns the generator based on the estimation result of the authenticity of the anchor-generated image. (6) A learning system according to the present invention. (8) The learning system includes: an identification image acquisition unit that acquires an anchor identification image and a plurality of feature identification images obtained by changing the features of the anchor identification image, the feature identification images corresponding to the plurality of features respectively; a discrimination vector calculation unit that calculates, for each of the feature spaces corresponding to each of the plurality of features based on the discriminator, an anchor discrimination vector for the anchor discrimination image, a positive discrimination vector for the feature discrimination image corresponding to the feature, and a negative discrimination vector for the feature discrimination image corresponding to another feature; a classifier training unit that trains the classifier so that the anchor identification vector and the positive identification vector become closer to each other and the anchor identification vector and the negative identification vector become farther apart in the feature space corresponding to each of the plurality of features; Further comprising: the generator learning unit performs the learning based on the classifier that has been trained by the classifier learning unit. A learning system according to (6) or (7). (9) the classifier learning unit causes the classifier to estimate the authenticity of the anchor identification image and the authenticity of the generated image generated by the generator, and further learns the classifier based on the estimation result of the authenticity of the anchor identification image and the estimation result of the authenticity of the generated image generated by the generator. (8) A learning system according to (8). (10) the classifier learning unit causes the classifier to estimate the authenticity of each of the plurality of anchor identification images, normalizes the estimation result of the authenticity of each of the plurality of anchor identification images, and learns the classifier further based on the execution result of the normalization; A learning system according to (8) or (9). [Explanation of symbols]

[0158] 1 Learning system, 10 Learning terminal, 20 Server, 11, 21, 31 Control unit, 12, 22, 32 Memory unit, 13, 23, 33 Communication unit, 14, 34 Operation unit, 15, 35 Display unit, 30 User terminal, 100 Data memory unit, 101 Identification image acquisition unit, 102 Identification vector calculation unit, 103 Classifier learning unit, 104 Latent code acquisition unit, 105 Generated image generation unit, 106 Generator learning unit, 107 Partial code acquisition unit, DB1 Identification image database, DB2 Generated image database.

Claims

1. A learning system for training a generator of a Generative Adversarial Network (GAN) in which a user can control multiple features related to a generated image, a partial code acquisition unit that acquires a plurality of partial codes corresponding to the plurality of features based on a latent code for generating the generated image and a plurality of mapping networks corresponding to the plurality of features; a generated image generator that generates the generated image based on an image synthesis network that uses the plurality of partial codes to generate the generated image; A generator training unit that trains the generator including the plurality of mapping networks and the image synthesis network based on the generated image and the trained classifier of the GAN; A learning system including.

2. The partial code acquisition unit Obtaining a first latent code based on a predetermined probability distribution; converting the first latent code to a second latent code based on the adjustable parameter; obtaining the plurality of partial codes based on the second latent code and the plurality of mapping networks; the generator learning unit performs the learning based on a spectral loss function indicating that the loss decreases as the distance between vectors related to the plurality of partial codes decreases. The learning system of claim 1 .

3. The parameters are learnable covariance matrices, the partial code acquisition unit converts the first latent code into the second latent code based on the learnable covariance matrix; The generator learning unit performs the learning by adjusting the value of the learnable covariance matrix based on the spectral loss function. The learning system according to claim 2 .

4. the predetermined probability distribution is an isotropic Gaussian distribution, the partial code acquisition unit acquires the first latent code based on the isotropic Gaussian distribution, and converts the first latent code into the second latent code based on the learnable covariance matrix, thereby acquiring the second latent code that follows an anisotropic Gaussian distribution. The learning system according to claim 3 .

5. The generated image generation unit generates the generated image by causing the image synthesis network to repeatedly perform convolution and upsampling based on the plurality of partial codes and an initial feature map in the generator. The learning system according to any one of claims 1 to 4.

6. the learning system further includes a latent code acquisition unit that acquires an anchor latent code and a plurality of feature latent codes obtained by changing a portion of the anchor latent code corresponding to the feature, the feature latent codes corresponding to the plurality of features, respectively; the partial code acquisition unit acquires a plurality of anchor partial codes, which are the plurality of partial codes based on the anchor latent code, and a plurality of feature partial codes, which are the plurality of partial codes based on each of the plurality of feature latent codes; the generated image generation unit acquires an anchor generated image that is the generated image based on the plurality of anchor part codes, and a plurality of feature generated images that are the generated image based on the plurality of feature part codes, the feature generated images corresponding to the plurality of features, respectively; The generator learning unit calculating, for each feature space corresponding to each of the plurality of features based on the classifier, an anchor generation vector for the anchor generation image, a positive generation vector for the feature generation image corresponding to the feature, and a negative generation vector for the feature generation image corresponding to another feature; The learning is performed so that, in the feature space corresponding to each of the plurality of features, the anchor generating vector and the positive generating vector become closer to each other and the anchor generating vector and the negative generating vector become farther apart from each other. The learning system according to any one of claims 1 to 4.

7. the generator learning unit causes the classifier to estimate the authenticity of the anchor-generated image, and further learns the generator based on the estimation result of the authenticity of the anchor-generated image. The learning system of claim 6.

8. The learning system includes: an identification image acquisition unit that acquires an anchor identification image and a plurality of feature identification images obtained by changing the features of the anchor identification image, the feature identification images corresponding to the plurality of features respectively; a discrimination vector calculation unit that calculates, for each of the feature spaces corresponding to each of the plurality of features based on the discriminator, an anchor discrimination vector for the anchor discrimination image, a positive discrimination vector for the feature discrimination image corresponding to the feature, and a negative discrimination vector for the feature discrimination image corresponding to another feature; a classifier training unit that trains the classifier so that the anchor identification vector and the positive identification vector become closer to each other and the anchor identification vector and the negative identification vector become farther apart in the feature space corresponding to each of the plurality of features; Further comprising: the generator learning unit performs the learning based on the classifier that has been trained by the classifier learning unit. The learning system of claim 6.

9. the classifier learning unit causes the classifier to estimate the authenticity of the anchor identification image and the authenticity of the generated image generated by the generator, and further learns the classifier based on the estimation result of the authenticity of the anchor identification image and the estimation result of the authenticity of the generated image generated by the generator. The learning system of claim 8.

10. the classifier learning unit causes the classifier to estimate the authenticity of each of the plurality of anchor identification images, normalizes the estimation result of the authenticity of each of the plurality of anchor identification images, and learns the classifier further based on the execution result of the normalization; The learning system of claim 8.

11. A learning method for training a generator of a Generative Adversarial Network (GAN) in which a user can control a plurality of features related to a generated image, comprising: a partial code acquisition step of acquiring a plurality of partial codes corresponding to the plurality of features based on a latent code for generating the generated image and a plurality of mapping networks corresponding to the plurality of features; a generated image generating step of generating the generated image based on an image synthesis network that generates the generated image using the plurality of partial codes; a generator training step of training the generator including the plurality of mapping networks and the image synthesis network based on the generated image and the trained classifier of the GAN; Learning methods including.

12. A computer that trains a generator of a GAN (Generative Adversarial Network) in which a user can control multiple features related to the generated image, a partial code acquisition unit that acquires a plurality of partial codes corresponding to the plurality of features, based on a latent code for generating the generated image and a plurality of mapping networks corresponding to the plurality of features; a generated image generator that generates the generated image based on an image synthesis network that uses the plurality of partial codes to generate the generated image; a generator training unit that trains the generator, which includes the plurality of mapping networks and the image synthesis network, based on the generated image and a trained classifier of the GAN; A program to function as a

Images