Learning system, learning method, and program

Abstract

To reduce labor for learning of a GAN.SOLUTION: A discrimination image acquisition part (101) acquires, for each feature relating to a generated image generated by a GAN: an anchor discrimination image; a positive discrimination image changed in this feature from the anchor discrimination image; and a negative discrimination image changed in another feature from the anchor discrimination image. A discrimination vector calculation part (102) calculates, for each feature space corresponding to each of the plurality of features, based on a discriminator of the GAN: an anchor discrimination vector relating to the anchor discrimination image of this feature; a positive discrimination vector about the positive discrimination image of the features; and a negative discrimination vector relating to the negative discrimination image of this feature. A discriminator learning part (104) performs learning of the discriminator so that, in the feature space corresponding to each of the plurality of features, the anchor discrimination vector and the positive discrimination vector approach each other, and the anchor discrimination vector and the negative discrimination vector become distant from each other.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a learning system, a learning method, and a program. [Background technology]

[0002] Conventionally, in the field of machine learning, a GAN (Generative Adversarial Network) that generates an image is known. Hereinafter, an image generated by a GAN is referred to as a generated image. For example, Non-Patent Document 1, Non-Patent Document 2, and Non-Patent Document 3 describe a GAN that allows a user to control multiple features (e.g., degree of smile, age, and hairstyle) related to a generated image. When a user specifies a desired feature, a latent code generated based on the specified feature is input to a generator of a trained GAN. The generator generates a generated image having the user's desired feature based on the latent code. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Tero Karras, Miika Aittala, Samuli Laine, Erik Harkonen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34:852-863, 2021. [Non-Patent Document 2] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019. [Non-Patent Document 3] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 8110-8119, 2020. Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the technologies of Non-Patent Document 1, Non-Patent Document 2, and Non-Patent Document 3, the creator of the GAN had to prepare a pre-trained model for labeling the training data or manually label the training data in order to prepare training data for the GAN to learn. For this reason, in the technologies of Non-Patent Document 1, Non-Patent Document 2, and Non-Patent Document 3, training the GAN was very time-consuming.

[0005] One of the goals of this disclosure is to reduce the effort required to train GANs. [Means for solving the problem]

[0006] The learning system according to the present disclosure includes an identification image acquisition unit that acquires, for each feature of a generative adversarial network (GAN) in which a user can control a plurality of features related to a generated image, an anchor identification image, a positive identification image in which the feature of the anchor identification image has been changed, and a negative identification image in which another feature of the anchor identification image has been changed, based on a classifier of the GAN, an anchor identification vector related to the anchor identification image of the feature, a positive identification vector related to the positive identification image of the feature, and a negative identification vector related to the negative identification image of the feature, for each feature space corresponding to each of the plurality of features, based on a classifier of the GAN, and a classifier training unit that trains the classifier so that the anchor identification vector and the positive identification vector are close to each other and the anchor identification vector and the negative identification vector are farther apart in the feature space corresponding to each of the plurality of features. Effect of the Invention

[0007] The present disclosure can, for example, reduce the effort required for training GANs. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of a learning system. [Diagram 2] FIG. 1 is a diagram illustrating an example of a GAN in which the user can control features related to the generated image. [Diagram 3] A diagram showing an example of functions realized by the learning system. [Figure 4] FIG. 13 is a diagram illustrating an example of a function for learning a classifier. [Diagram 5] FIG. 13 is a diagram illustrating an example of a function for learning a classifier. [Figure 6] FIG. 11 is a diagram illustrating an example of an identification image database. [Figure 7] FIG. 13 is a diagram illustrating an example of a function for training a generator. [Figure 8]FIG. 13 is a diagram illustrating an example of a function for training a generator. [Figure 9] FIG. 11 is a diagram illustrating an example of a generated image database. [Figure 10] FIG. 2 is a diagram illustrating an example of processing executed in the learning system. [Figure 11] FIG. 2 is a diagram illustrating an example of processing executed in the learning system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [1. Hardware configuration of the learning system] An example of an embodiment of a learning system, a learning method, and a program according to the present disclosure will be described. Fig. 1 is a diagram showing an example of a hardware configuration of a learning system. For example, the learning system 1 includes a learning terminal 10, a server 20, and a user terminal 30. Each of the learning terminal 10, the server 20, and the user terminal 30 is connected to a network N such as the Internet or a LAN (Local Area Network).

[0010] The learning terminal 10 is a computer that performs learning of a GAN (Generative Adversarial Network) that generates an image. Hereinafter, an image generated by a GAN is referred to as a generated image. For example, the learning terminal 10 is a personal computer, a server computer, a tablet, or a smartphone. For example, the learning terminal 10 includes a control unit 11, a memory unit 12, a communication unit 13, an operation unit 14, and a display unit 15.

[0011] For example, the control unit 11 includes at least one processor. The storage unit 12 includes at least one of a volatile memory such as a RAM and a non-volatile memory such as a flash memory. The communication unit 13 includes at least one of a communication interface for wired communication and a communication interface for wireless communication. The operation unit 14 is an input device such as a touch panel or a mouse. The display unit 15 is a display such as a liquid crystal or organic EL.

[0012] The server 20 is a server computer that stores the trained GAN. For example, the server 20 includes a control unit 21, a storage unit 22, and a communication unit 23. The hardware configurations of the control unit 21, the storage unit 22, and the communication unit 23 may be similar to those of the control unit 11, the storage unit 12, and the communication unit 13, respectively.

[0013] The user terminal 30 is a user's computer that uses the trained GAN. For example, the user terminal 30 is a personal computer, a smartphone, a tablet, or a wearable terminal. For example, the user terminal 30 includes a control unit 31, a storage unit 32, a communication unit 33, an operation unit 34, and a display unit 35. The hardware configurations of the control unit 31, the storage unit 32, the communication unit 33, the operation unit 34, and the display unit 35 may be similar to those of the control unit 11, the storage unit 12, the communication unit 13, the operation unit 14, and the display unit 15, respectively.

[0014] The programs stored in the storage units 12, 22, 32 may be supplied to the learning terminal 10, the server 20, or the user terminal 30 via the network N. At least one of a reading unit (e.g., a memory card slot) that reads a computer-readable information storage medium and an input / output unit (e.g., a USB port) for inputting and outputting data to and from an external device may be included in the learning terminal 10, the server 20, or the user terminal 30. For example, a program stored in an information storage medium may be supplied to the learning terminal 10, the server 20, or the user terminal 30 via at least one of the reading unit and the input / output unit.

[0015] Furthermore, the learning system 1 may include at least one computer. The computers included in the learning system 1 are not limited to the example of FIG. 1. For example, the learning system 1 may include only the learning terminal 10. In this case, the server 20 and the user terminal 30 exist outside the learning system 1. The learning system 1 may include only the learning terminal 10 and the server 20. In this case, the user terminal 30 exists outside the learning system 1. The learning system 1 may include only other computers not shown in FIG. 1.

[0016] [2. Overview of the learning system] For example, the GAN generates a generated image based on random noise. If the GAN generates a generated image based entirely on random noise, the GAN may not generate a generated image having the characteristics desired by the user. The features here are visual features of the generated image. For example, the shape of an object (e.g., a person, an animal, a landscape, or a building) shown in the generated image, the color of the object, the size of the object, the brightness of the object, or the arrangement of the object correspond to the features. The features may also be referred to as the style, condition, or attribute of the generated image.

[0017] FIG. 2 is a diagram showing an example of a GAN in which the user can control the features of a generated image. For example, research on conditional GAN, StyleGAN, or SSD (Self-supervised Style Decomposition)-GAN has been conducted as research on GANs that generate generated images having features desired by the user. Hereinafter, when the word "feature" is simply written, it does not mean a feature vector calculated by the GAN, but a feature to be controlled. In this embodiment, the user can control each of a plurality of features.

[0018] As shown in FIG. 2, the GAN includes a generator G that generates a generated image and a classifier D that identifies the generated image. The generator G is a model that generates a generated image based on a latent code according to the features desired by the user. For example, the generator G is a convolutional neural network or a fully connected neural network. The generator G may use other known methods. Recently, a method called Transformer, which is mainly used for natural language processing, is sometimes used for image generation, so the generator G may use this method.

[0019] The latent code is information for conveying the user's desired features to the generator G. For example, the latent code is expressed in a vector format. The latent code is sometimes called a condition vector. The latent code may be expressed in a format other than a vector. For example, in a GAN that generates a generated image showing a human face, a user can control three features, such as the degree of smile, age, and hairstyle. When the user specifies these three features, a latent code is generated according to each of the degree of smile, age, and hairstyle specified by the user. The user does not need to specify all the features, and may specify only some of the features.

[0020] For example, the generator G generates a generated image based on a latent code input thereto. The latent code may include random noise. The generator G may generate a generated image based on random noise separately from the latent code. The processing of the generator G after the latent code is input may be a publicly known processing. For example, the generator G may convert the latent code into a high-dimensional representation. The generator G outputs a generated image corresponding to the latent code based on its own parameters. The parameters of the generator G are referenced in the processing for converting the latent code into a generated image. A series of processing until the generator G outputs a generated image may be a publicly known processing.

[0021] The classifier D identifies whether the generated image is genuine or false. In the case of a controllable GAN, the classifier D identifies whether the generated image has a feature desired by the user. That is, the classifier D identifies whether the generated image is an image corresponding to the latent code. For example, the classifier D is a convolutional neural network or a fully connected neural network. The classifier D may use other known methods. Recently, a method called Transformer, which is mainly used for natural language processing, is sometimes used for image generation, so the classifier D may use this method.

[0022] For example, when a generated image is input to the classifier D, the classifier D executes processing such as convolution based on its own parameters to calculate a feature vector for the generated image. The classifier D outputs a classification result according to the feature vector. The classification result may be a label indicating whether the generated image is true or false, a score indicating the probability that the generated image is true or false, a label indicating whether the generated image is an image corresponding to the latent code, a score indicating the probability that the generated image is an image corresponding to the latent code, or a combination of these. A series of processes from when the generated image is input to the classifier D to when the classifier D outputs the classification result may be known processes.

[0023] As described above, the GAN includes a generator G and a discriminator D. In general, the learning of the GAN is performed first by learning the discriminator D. After the learning of the discriminator D is completed to a certain extent, the learning of the generator G is performed. Then, the learning of the discriminator D and the learning of the generator G are repeated. For a GAN in which a plurality of features can be controlled by a user as shown in FIG. 2, it is necessary to learn images having various features. For example, in order to create a GAN in which three features, such as the degree of smile, age, and hairstyle, can be controlled by a user, it is necessary to prepare images of various degrees of smile, images of various ages, and images of various hairstyles.

[0024] For example, it is very time-consuming for the creator of the GAN to manually label images in order to prepare images to be trained by the GAN. To avoid the creator having to manually label images, it is possible to use a classification model to classify images to be trained by the GAN, but preparing the classification model is time-consuming. Conventionally, the preparation of images to be trained by the GAN and the training of the GAN using the images are performed separately, so this is also not efficient learning. Furthermore, if there are few images to be trained by the GAN, the accuracy of the GAN may decrease.

[0025] Therefore, the learning system 1 of the present embodiment employs a learning method specific to GANs that allows users to control multiple features, thereby reducing the effort required for learning GANs without manually labeling identification images or using pre-trained classification models. In addition, the learning system 1 realizes efficient learning by seamlessly performing advance preparation of images to be learned by the GAN and learning of the GAN using the images. The learning system 1 also improves the accuracy of the GAN through such learning. Hereinafter, the learning system 1 will be described in detail.

[0026] [3. Functions realized by the learning system] 3 is a diagram showing an example of functions realized by the learning system 1. In this embodiment, a case where main functions for GAN learning are realized by the learning terminal 10 is taken as an example.

[0027] In this embodiment, the learning process of a GAN that generates a generated image showing an animal's face is taken as an example. Furthermore, four user-controllable features are taken as examples: viewpoint, global identity, local identity, and color. The viewpoint is at least one of the viewpoint position and orientation. The viewpoint can also be called a camera. The global identity is the overall appearance of an object. The local identity is the local appearance of an object. The color is the overall color of an object.

[0028] [3-1. Functions for learning classifiers] 4 and 5 are diagrams showing an example of a function for learning the classifier D. The function for learning the classifier D will be described with reference to FIGS. 4 and 5, using the diagram showing the functions of the learning terminal 10 shown in FIG. 3. As shown in FIG. 3, for example, the learning terminal 10 includes a data storage unit 100, a discrimination image acquisition unit 101, a discrimination vector calculation unit 102, an estimation unit 103, and a classifier learning unit 104 as functions for learning the classifier D. The data storage unit 100 is realized by the storage unit 12. Each of the discrimination image acquisition unit 101, the discrimination vector calculation unit 102, the estimation unit 103, and the classifier learning unit 104 is realized by the control unit 11.

[0029] [Data storage section] The data storage unit 100 stores data necessary for training the classifier D. For example, the data storage unit 100 stores actual data of the GAN and a classification image database DB1.

[0030] The GAN whose actual data is stored in the data storage unit 100 is a GAN before learning. The GAN before learning is a GAN whose parameters are initial values. The actual data indicates the program of the GAN and the parameters of the GAN. For example, the data storage unit 100 stores the actual data of a classifier D included in the actual data of the GAN. The actual data of the classifier D indicates the program of the classifier D and the parameters of the classifier D. The parameters of the classifier D are referenced by the program of the classifier D. The parameters of the classifier D are adjusted by the classifier learning unit 104. For example, the parameters of the classifier D are weight coefficients and biases. The parameters of the classifier D may be known parameters. For example, the parameters of the classifier D may be the number of hidden layers, the number of units of the hidden layer, or other hyperparameters.

[0031] 6 is a diagram showing an example of the identification image database DB1. The identification image database DB1 is a database in which identification images, which are images for training the classifier D, are stored. In this embodiment, three identification images, namely, an anchor identification image, a positive identification image, and a negative identification image, are used in training the classifier D. Hereinafter, when there is no need to distinguish between the anchor identification image, the positive identification image, and the negative identification image, they are simply referred to as identification images. The identification images can also be referred to as training images used as training data for the classifier D.

[0032] For example, the identification image database DB1 stores feature identification information and image data of the anchor identification image, the positive identification image, and the negative identification image. Any data may be stored in the identification image database DB1. The data stored in the identification image database DB1 is not limited to the example of FIG. 6. For example, the identification image database DB1 may store information indicating which feature among a plurality of user-controllable features was changed to generate the negative identification image.

[0033] The feature identification information is information that can identify any one of a plurality of user-controllable features. In this embodiment, the feature identification information indicates any one of a viewpoint, a global identity, a local identity, or a color. For each feature identification information indicating a certain feature, a set of three identification images, an anchor identification image, a positive identification image, and a negative identification image, is stored in the identification image database DB1. Hereinafter, the set is referred to as an identification image set. For feature identification information indicating a certain feature, multiple identification image sets may be stored in the identification image database DB1.

[0034] The number of features that can be controlled by the user is not limited to four as in this embodiment. The number of features may be more than one. For example, the number of features may be two or three, or may be five or more. The specific content of the features is also not limited to the example of this embodiment. The features may be those used in known image generation. For example, the features may be facial expressions, facial contours, eye color, hair color, brightness, background, or other features. In this embodiment, an example is given of a case where the user can control multiple features, but a GAN that can control only one feature may also be trained in the same way as in this embodiment.

[0035] The anchor identification image is an original image whose user-controllable features have not been changed. For example, the anchor identification image may be a publicly known image distributed on the Internet for free or for a fee. The anchor identification image is a reference image in distance learning. In this embodiment, the anchor identification image is defined by x in the following formula 1. B is the batch size. i is a numerical value for identifying each anchor identification image. i is an arbitrary numerical value between 1 and B. x i is an individual anchor identification image. x is a collection of anchor identification images. For example, the image corresponding to x in FIG. 4 is an anchor identification image. The original data set of anchor identification images can be written as x∈X.

[0036]

number

[0037] A positive identification image is an image in which at least one of a plurality of user-controllable features is changed with respect to an anchor identification image. In this embodiment, a positive identification image is an image in which any of a plurality of user-controllable features is changed with respect to an anchor identification image, but a positive identification image may be an image in which each of a plurality of user-controllable features is changed with respect to an anchor identification image. A positive identification image is an image that is used by the classifier D to recognize an image similar to anchor identification information (an image belonging to the same cluster as the anchor identification image) in distance learning. Hereinafter, when it is necessary to express each positive identification image in a mathematical formula, τ + (x i ) is written as τ + means a positive change. For example, τ + The image corresponding to (x) is the positive identification image.

[0038] In this embodiment, a positive identification image of a certain feature identification information is an image in which the feature indicated by the feature identification information has been changed with respect to the anchor image of the feature identification information. For example, a positive identification image of feature identification information indicating a viewpoint is an image in which a viewpoint change has been made with respect to the anchor identification image. A positive identification image of feature identification information indicating a global identity is an image in which a global identity change has been made with respect to the anchor identification image. A positive identification image of feature identification information indicating a local identity is an image in which a local identity change has been made with respect to the anchor identification image. A positive identification image of feature identification information indicating a color is an image in which a color change has been made with respect to the anchor identification image.

[0039] A negative identification image is an image in which other features different from those of the positive identification image have been changed with respect to the anchor identification image. In this embodiment, a negative identification image is an image in which one other feature different from that of the positive identification image has been changed with respect to the anchor identification image among a plurality of features controllable by the user, but the negative identification image may be an image in which each of a plurality of other features has been changed with respect to the anchor identification image. A negative identification image is an image that is used by the classifier D in distance learning to recognize an image different from the anchor identification information (an image that does not belong to the same cluster as the anchor identification image). Hereinafter, when it is necessary to express each negative identification image in a mathematical formula, τ - (x i ) is written as τ - means a negative change. - is τ + For example, the τ - The image corresponding to (x) is the negative identification image.

[0040] In this embodiment, a negative identification image of a certain feature identification information is an image in which features other than the feature indicated by the feature identification information are changed with respect to the anchor image of the feature identification information. For example, a negative identification image of feature identification information indicating a viewpoint is an image in which features other than the viewpoint (any of a global identity, a local identity, or a color) are changed with respect to the anchor identification image. A negative identification image of feature identification information indicating a global identity is an image in which features other than the global identity (any of a viewpoint, a local identity, or a color) are changed with respect to the anchor identification image.

[0041] For example, a negative identification image of feature identification information indicating a local identity is an image in which a feature other than the local identity (either viewpoint, global identity, or color) has been changed from the anchor identification image. A negative identification image of feature identification information indicating color is an image in which a feature other than the color (either viewpoint, global identity, or local identity) has been changed from the anchor identification image.

[0042] The data stored in the data storage unit 100 is not limited to the above example. The data storage unit 100 may store any data. For example, the data storage unit 100 may store a classifier learning program indicating a series of processes in learning the classifier D. The classifier learning program indicates program codes indicating each of the processes of the discrimination image acquisition unit 101, the discrimination vector calculation unit 102, a part of the processes of the estimation unit 103, and the classifier learning unit 104. For example, the data storage unit 100 may store a feature change program for changing each of a plurality of features controllable by the GAN.

[0043] [Identification image acquisition section] The identification image acquisition unit 101 acquires, for each GAN feature in which the user can control a plurality of features related to the generated image, an anchor identification image, a positive identification image in which the feature of the anchor identification image is changed, and a negative identification image in which another feature of the anchor identification image is changed. For example, the identification image acquisition unit 101 acquires an identification image set for each of a plurality of features that are controllable by the user. The identification image acquisition unit 101 may acquire multiple identification image sets for one feature.

[0044] In this embodiment, pairs of feature identification information and anchor identification images are stored in advance in the identification image database DB1. For example, a plurality of such pairs may be stored in the identification image database DB1. The positive identification image and the negative identification image are not initially stored in the identification image database DB1. The identification image acquisition unit 101 acquires anchor identification images of features indicated by each of the plurality of pairs by referring to the identification image database DB1. The identification image acquisition unit 101 acquires each of the positive identification image and the negative identification image of the feature by generating each of the positive identification image and the negative identification image of the feature. The identification image acquisition unit 101 stores each of the positive identification image and the negative identification image of the feature in the identification image database DB1 in association with the pair.

[0045] In this embodiment, the program by which the identification image acquisition unit 101 changes each of the multiple features of the anchor identification image is stored in the data storage unit 100. Hereinafter, this program is referred to as a feature change program. A feature change program for a certain feature indicates a process for changing the feature of the anchor identification image. The feature change program may be a publicly known image processing program. The identification image acquisition unit 101 generates each of the positive identification image and the negative identification image by executing image processing for changing the feature of the anchor identification image based on the feature change program for each of the multiple features.

[0046] In this embodiment, feature modification programs for each of the four features, i.e., viewpoint, global identity, local identity, and color, are stored in the data storage unit 100. The identification image acquisition unit 101 generates a positive identification image and a negative identification image for each feature based on each of these four feature modification programs.

[0047] For example, the viewpoint characteristic change program indicates a process of performing a geometric change. The geometric process is a transformation, a rotation, a resize, or a combination thereof. The geometric process may be any process used in the field of image processing. For example, the geometric process may be an affine transformation, a projective transformation, a warp process, a rotation, a resize, or a combination thereof. The identification image acquisition unit 101 changes the viewpoint of the anchor identification image based on the viewpoint characteristic change program. The extent to which the viewpoint is changed may be determined randomly or may be predetermined.

[0048] For example, the global identity feature change program refers to a process of changing the overall appearance of the object represented by the anchor identification image. The process of changing the overall appearance refers to a process of changing the contour, size, color, or a combination of these of the object. The process of changing the overall appearance may be other processes such as a blurring process or a masking process. The process of changing the overall appearance may be any process used in the image processing field. The identification image acquisition unit 101 changes the global identity of the anchor identification image based on the global identity feature change program. The extent to which the global identity is changed may be determined randomly or may be predetermined.

[0049] For example, the local identity feature change program refers to a process of changing the local appearance of an object represented by an anchor identification image. The process of changing the local appearance refers to a process of changing the outline, size, color, or a combination of these of a part of an object. The process of changing the local appearance may be other processes such as a blurring process or a masking process. The process of changing the local appearance may be any process used in the image processing field. The identification image acquisition unit 101 changes the local identity of the anchor identification image based on the local identity feature change program. The extent to which the local identity is changed may be determined randomly or may be predetermined.

[0050] For example, the color feature change program indicates a process of changing the color of an object indicated by an anchor identification image. The color change process indicates a process of changing a color filter, a hue change, a color balance adjustment, or a combination of these. The color change process may be a process of changing brightness or transparency, etc. The color change process may be any process used in the image processing field. The identification image acquisition unit 101 changes the color of the anchor identification image based on the color feature change program. The extent to which the color is changed may be determined randomly or may be predetermined.

[0051] Hereinafter, the change in the feature is defined by the following Equation 2. v is a change of perspective. τ g is a global identity change. l is a local identity change. τ c is a color change. T is a collection of these changes. Hereinafter, when it is necessary to express the feature change in a mathematical formula, these symbols will be used. In this embodiment, the identification image acquisition unit 101 generates a positive identification image and a negative identification image for each feature based on each of the four feature change programs described above. In the case of a GAN that can control features other than the viewpoint, global identity, local identity, and color, it is assumed that the feature change program for the other feature is stored in the data storage unit 100. The identification image acquisition unit 101 may generate at least one of a positive identification image and a negative identification image by changing the anchor identification image based on the feature change program for the other feature.

[0052]

number

[0053] For example, when certain feature identification information indicates a viewpoint, the identification image acquisition unit 101 generates a positive identification image by changing the viewpoint of the anchor identification image for the feature identification information based on a viewpoint feature change program. The identification image acquisition unit 101 generates a negative identification image by changing other features of the anchor identification image for the anchor identification image for which the feature identification information indicates a viewpoint based on a feature change program for other features (global identity, local identity, or color) other than the viewpoint. The identification image acquisition unit 101 stores the generated positive identification image and negative identification image in the identification image database DB1 in association with the feature identification information and the anchor identification image.

[0054] For example, when certain feature identification information indicates a global identity, the identification image acquisition unit 101 generates a positive identification image by changing the global identity of the anchor identification image for the feature identification information based on a feature change program for the global identity. The identification image acquisition unit 101 generates a negative identification image by changing other features of the anchor identification image for the anchor identification image for which the feature identification information indicates a global identity based on a feature change program for other features (viewpoint, local identity, or color) other than the global identity. The identification image acquisition unit 101 stores the generated positive identification image and negative identification image in the identification image database DB1 in association with the feature identification information and the anchor identification image.

[0055] For example, when certain feature identification information indicates a local identity, the identification image acquisition unit 101 generates a positive identification image by changing the local identity of the anchor identification image for the feature identification information based on a feature change program for the local identity. The identification image acquisition unit 101 generates a negative identification image by changing other features of the anchor identification image for the anchor identification image for which the feature identification information indicates a local identity based on a feature change program for other features (viewpoint, global identity, or color) other than the local identity. The identification image acquisition unit 101 stores the generated positive identification image and negative identification image in the identification image database DB1 in association with the feature identification information and the anchor identification image.

[0056] For example, when certain feature identification information indicates a color, the identification image acquisition unit 101 generates a positive identification image by changing the color of the anchor identification image for the feature identification information based on a color feature change program. The identification image acquisition unit 101 generates a negative identification image by changing other features of the anchor identification image for the anchor identification image for which the feature identification information indicates a color based on a feature change program for other features (perspective, global identity, or local identity) other than color. The identification image acquisition unit 101 stores the generated positive identification image and negative identification image in the identification image database DB1 in association with the feature identification information and the anchor identification image.

[0057] In the case where the plurality of features are three or more features as in the present embodiment, the identification image acquisition unit 101 may acquire a plurality of negative identification images in which other features different from each other are changed for each feature. For example, the identification image acquisition unit 101 may generate two negative identification images, a negative identification image in which the global identity is changed and a negative identification image in which the local identity is changed, for the anchor identification image of the feature identification information indicating the viewpoint. Furthermore, the identification image acquisition unit 101 may generate three negative identification images by generating a negative identification image in which the color is changed for the anchor identification image. Similarly, the identification image acquisition unit 101 may generate a plurality of negative identification images for features other than the viewpoint.

[0058] In this embodiment, multiple anchor identification images are prepared for each feature. At least one positive identification image and at least one negative identification image are generated from each anchor identification image, so that the identification image acquisition unit 101 acquires multiple anchor identification images, multiple positive identification images, and multiple negative identification images for each feature. The acquisition method of each anchor identification image, each positive identification image, and each negative identification image is as described above.

[0059] In the present embodiment, the case where the identification image acquisition unit 101 generates the positive identification image and the negative identification image is taken as an example, but the positive identification image and the negative identification image may be generated by a computer other than the learning terminal 10. The identification image database DB1 may store the positive identification image and the negative identification image generated by the other computer. In this case, the learning terminal 10 may not have the function of generating the positive identification image and the negative identification image.

[0060] [Discrimination vector calculation part] The discrimination vector calculation unit 102 calculates, for each feature space corresponding to each of a plurality of features, an anchor discrimination vector for the anchor discrimination image of the feature, a positive discrimination vector for the positive discrimination image of the feature, and a negative discrimination vector for the negative discrimination image of the feature, based on the GAN discriminator D. The calculation of these vectors is sometimes called mapping to the feature space.

[0061] The feature space is a multidimensional space. The number of feature spaces is equal to or greater than the number of user-controllable features. The classifier D calculates a feature vector in each of a plurality of feature spaces for an image input thereto. The parameters of the classifier D for calculating a feature vector in a certain feature space are different from the parameters of the classifier D for calculating a feature vector in another feature space. The number of parameters of the classifier D is equal to or greater than the number of user-controllable features. In this embodiment, the classifier D has five parameters, including a parameter for identifying the authenticity of an image, a parameter for a feature space corresponding to a viewpoint, a parameter for a feature space corresponding to a global identity, a parameter for a feature space corresponding to a local identity, and a parameter for a feature space corresponding to a color. Other parameters may be present in the classifier D.

[0062] In this embodiment, a case is taken as an example where a program for calculating a feature vector in a feature space is the same as a program for calculating a feature vector in another feature space. Even if these programs are the same, parameters for calculating a feature vector in a feature space are different from parameters for calculating a feature vector in another feature space, so that different feature vectors are calculated for the same image. Note that these programs may be different from each other. There may be as many programs for the classifier D as the number of features that can be controlled by the user, or more.

[0063] The anchor identification vector is the feature vector of the anchor identification image. The positive identification vector is the feature vector of the positive identification image. The negative identification vector is the feature vector of the negative identification image. For one identification image set, a set of three feature vectors is calculated: the anchor identification vector, the positive identification image, and the negative identification image. If the number of user-controllable features is n (n is an integer equal to or greater than 2) and the number of identification image sets is m (m is an integer equal to or greater than 2), then at least n x m x 3 feature vectors are calculated.

[0064] In this embodiment, an example is given in which there exists a feature space for identifying the authenticity of an image (e.g., an identification image) input to the device, and a feature space corresponding to each of a plurality of features controllable by the user. In this embodiment, since the number of features controllable by the user is four, as shown in FIG. 4, the classifier D calculates a feature vector in each of five feature spaces. In the example of FIG. 4, the outputs of the classifier D are indicated by symbols P0 to P4. Each of the outputs P0 to P4 may be information necessary for calculating a feature vector in the feature space. Each of the outputs P0 to P4 may be the feature vector itself in the feature space, or may be information input to a calculation formula called a projection head for reducing the dimension.

[0065] The feature space for authenticity discrimination may be the same as the feature space adopted in a known GAN. For example, when an image (e.g., an identified image) is input to the classifier D, the discrimination vector calculation unit 102 calculates a feature vector of the image in the feature space based on parameters corresponding to the feature space for authenticity discrimination. In the example of FIG. 4, the discrimination vector calculation unit 102 calculates a feature vector in the feature space for authenticity discrimination by reducing the dimension of the output P0 from the classifier D by a projection head. Based on the feature vector, the authenticity of the image input to the classifier D is discriminated.

[0066] Hereafter, when we need to express the viewpoint feature space, the global identity feature space, the local identity feature space, and the color feature space in mathematical terms, we use f v , f g , f l , and f c In the example of FIG. 4, a set of distinguished images of the viewpoint is input to the classifier D. Therefore, the feature space f v The output P1 surrounded by a solid line is used to calculate the feature vector in the feature space f v Since it is not used in the feature vector in, it is surrounded by a dotted line in the example of Figure 4.

[0067] In this embodiment, the discrimination vector calculation unit 102 calculates an anchor discrimination vector in a feature space corresponding to a certain feature based on an anchor discrimination image of the feature and parameters of the feature. The discrimination vector calculation unit 102 calculates a positive discrimination vector in a feature space corresponding to the certain feature based on a positive discrimination image of the feature and parameters of the feature. The discrimination vector calculation unit 102 calculates a negative discrimination vector in a feature space corresponding to the certain feature based on a negative discrimination image of the feature and parameters of the feature.

[0068] Hereafter, the features selected as positive identification images from among T shown in Equation 2 are called τ + Among the features of the negative identification image, those selected from T shown in Equation 2 are called τ - Let τ + and τ - and are different from each other. It is assumed that random noise is generated according to a normal distribution z ∼ N (0, 1). It is assumed that the anchor identification image, the positive identification image, and the negative identification image in the feature space are defined by the following Equation 3, Equation 4, and Equation 5, respectively. D τ+ is the selected change τ + is a mapping (computation of a feature vector) in the feature space corresponding to

[0069]

number

[0070]

number

[0071]

number

[0072] For example, the discrimination vector calculation unit 102 calculates an anchor discrimination vector in a feature space corresponding to a viewpoint based on an anchor discrimination image of the viewpoint and a parameter of the viewpoint. The discrimination vector calculation unit 102 calculates a positive discrimination vector in a feature space corresponding to a viewpoint based on a positive discrimination image of the viewpoint and a parameter of the viewpoint. The discrimination vector calculation unit 102 calculates a negative discrimination vector in a feature space corresponding to a viewpoint based on a negative discrimination image of the viewpoint and a parameter of the viewpoint.

[0073] For example, the discriminant vector calculation unit 102 calculates an anchor discriminant vector in the feature space corresponding to the global identity based on the anchor discriminant image of the global identity and the parameters of the global identity. The discriminant vector calculation unit 102 calculates a positive discriminant vector in the feature space corresponding to the global identity based on the positive discriminant image of the global identity and the parameters of the global identity. The discriminant vector calculation unit 102 calculates a negative discriminant vector in the feature space corresponding to the global identity based on the negative discriminant image of the global identity and the parameters of the global identity.

[0074] For example, the discrimination vector calculation unit 102 calculates an anchor discrimination vector in the feature space corresponding to the local identity based on the anchor discrimination image of the local identity and the parameters of the local identity. The discrimination vector calculation unit 102 calculates a positive discrimination vector in the feature space corresponding to the local identity based on the positive discrimination image of the local identity and the parameters of the local identity. The discrimination vector calculation unit 102 calculates a negative discrimination vector in the feature space corresponding to the local identity based on the negative discrimination image of the local identity and the parameters of the local identity.

[0075] For example, the discrimination vector calculation unit 102 calculates an anchor discrimination vector in a feature space corresponding to the color based on an anchor discrimination image of the color and a color parameter. The discrimination vector calculation unit 102 calculates a positive discrimination vector in a feature space corresponding to the color based on a positive discrimination image of the color and a color parameter. The discrimination vector calculation unit 102 calculates a negative discrimination vector in a feature space corresponding to the color based on a negative discrimination image of the color and a color parameter.

[0076] In addition, when a plurality of negative identification images in which different features are changed are acquired for each feature, the identification vector calculation unit 102 may calculate the negative identification vector of each of the plurality of negative identification images of the feature for each feature space corresponding to each of the plurality of features. In other words, when a plurality of negative identification images exist for one anchor identification image, the identification vector calculation unit 102 may calculate the negative identification vector of each of the plurality of negative identification images. The method of calculating each negative identification vector is as described above.

[0077] In this embodiment, since multiple anchor identification images are prepared for each feature, the identification vector calculation unit 102 calculates, for each feature space corresponding to each of the multiple features, an anchor identification vector for each of the multiple anchor identification images of the feature, a positive identification vector for each of the multiple positive identification images of the feature, and a negative identification vector for each of the multiple negative identification images of the feature. The calculation methods for each anchor identification vector, each positive identification vector, and each negative identification vector are as described above.

[0078] [Estimation part] The estimation unit 103 causes the classifier D to estimate the authenticity of the anchor identification image and the authenticity of the generated image generated by the generator G of the GAN. The estimation here can also be called classification. The estimation by the estimation unit 103 is performed based on the classifier D undergoing training. That is, the estimation unit 103 performs estimation based on the current parameters of the classifier D. For example, the estimation unit 103 inputs the anchor identification image to the classifier D. The classifier D calculates a feature vector of the anchor identification image in a feature space for identifying authenticity. The classifier D outputs an estimation result of authenticity based on the feature vector. The estimation unit 103 acquires the estimation result of authenticity output from the classifier D.

[0079] In this embodiment, since a plurality of anchor identification images are prepared, the estimation unit 103 causes the classifier D to estimate the authenticity of each of the plurality of anchor identification images. The estimation unit 103 inputs each of the plurality of anchor identification images to the classifier D one after another, and obtains an estimation result of the authenticity of each anchor identification image from the classifier D. The method of estimating the authenticity of each anchor identification image is as described above.

[0080] For example, the estimation unit 103 acquires a generated image generated by a generator G. This generator G is a generator G currently undergoing training. This generator G may be an initial generator G, or may be a generator G for which training by a generator training unit 108 described below has been temporarily completed. The estimation unit 103 inputs the generated image to a classifier D. The classifier D calculates a feature vector of the generated image in a feature space for discriminating between true and false. The classifier D outputs an estimation result of true and false based on the feature vector. The estimation unit 103 acquires the estimation result of true and false output from the classifier D.

[0081] [Classifier learning part] The classifier training unit 104 trains the classifier D so that the anchor identification vector and the positive identification vector are close to each other and the anchor identification vector and the negative identification vector are far apart in the feature space corresponding to each of the multiple features. In other words, the classifier training unit 104 trains the classifier D so that the anchor identification vector and the positive identification vector belong to the same cluster and the anchor identification vector and the negative identification vector do not belong to the same cluster in the feature space corresponding to each of the multiple features.

[0082] The learning algorithm itself may be a known algorithm used in distance learning. For example, an algorithm such as gradient descent may be used for learning the classifier D. In this embodiment, the mutual relationship between the three classified images is used in learning, so the classifier learning unit 104 learns the classifier D based on the loss function of triplet margin loss so that the anchor classification vector and the positive classification vector are closer to each other and the anchor classification vector and the negative classification vector are farther away from each other. The loss function described in this embodiment is an example of the loss function of triplet margin loss.

[0083] For example, the classifier training unit 104 adjusts parameters of a plurality of features so that the anchor identification vector and the positive identification vector become closer to each other and the anchor identification vector and the negative identification vector become farther apart in a feature space corresponding to each of the plurality of features, thereby training the classifier D. The classifier training unit 104 calculates a loss indicating the closeness between the anchor identification vector and the positive identification vector and the farthest between the anchor identification vector and the negative identification vector in a feature space corresponding to each of the plurality of features based on a loss function in distance learning, and adjusts parameters of the feature so as to reduce the loss.

[0084] For example, the classifier training unit 104 adjusts the parameters of the viewpoint so that the anchor identification vector of the anchor identification image of the viewpoint and the positive identification vector of the positive identification image of the viewpoint become closer to each other and the anchor identification vector and the negative identification vector of the negative identification image of the viewpoint become farther apart in the feature space corresponding to the viewpoint, thereby training the classifier D. The classifier training unit 104 calculates a loss indicating the closeness of the anchor identification vector and the positive identification vector and the farthest distance between the anchor identification vector and the negative identification vector in the feature space corresponding to the viewpoint based on a loss function in distance learning, and adjusts the parameters of the viewpoint so as to reduce the loss.

[0085] In the example of FIG. 5, the state of the feature space corresponding to the viewpoint is shown. For example, by learning, the anchor identification image x 1 and the positive identification image of the viewpoint τ + (x 1 ) and , the distance between them becomes closer and they belong to the same cluster. 1 and the negative discrimination image of the viewpoint τ - (x 1 ) and , the distance between them becomes greater, and they do not belong to the same cluster. 1 and other anchor identification image x of the viewpoint 2 ~x 4The distance between each of and will become greater, and they will no longer belong to the same cluster. Similar learning is performed for other feature spaces other than the feature space corresponding to the viewpoint.

[0086] For example, the classifier training unit 104 adjusts the parameters of the global identity so that the anchor identification vector of the anchor identification image of the global identity and the positive identification vector of the positive identification image of the global identity become closer to each other and the anchor identification vector becomes farther away from the negative identification vector of the negative identification image of the global identity in the feature space corresponding to the global identity, thereby training the classifier D. The classifier training unit 104 calculates a loss indicating the closeness of the anchor identification vector and the positive identification vector and the farthest distance between the anchor identification vector and the negative identification vector in the feature space corresponding to the global identity based on a loss function in distance learning, and adjusts the parameters of the global identity so as to reduce the loss.

[0087] For example, the classifier training unit 104 adjusts the parameters of the local identity so that the anchor identification vector of the anchor identification image of the local identity and the positive identification vector of the positive identification image of the local identity become closer to each other and the anchor identification vector becomes farther away from the negative identification vector of the negative identification image of the local identity in the feature space corresponding to the local identity, thereby training the classifier D. The classifier training unit 104 calculates a loss indicating the closeness of the anchor identification vector and the positive identification vector and the farthest distance between the anchor identification vector and the negative identification vector in the feature space corresponding to the local identity based on a loss function in distance learning, and adjusts the parameters of the local identity so as to reduce the loss.

[0088] For example, the classifier training unit 104 adjusts the color parameters so that the anchor classification vector of the anchor classification image of the color and the positive classification vector of the positive classification image of the color become closer to each other and the anchor classification vector and the negative classification vector of the negative classification image of the color become farther apart in the feature space corresponding to the color, thereby training the classifier D. The classifier training unit 104 calculates a loss indicating the closeness of the anchor classification vector and the positive classification vector and the farthest distance between the anchor classification vector and the negative classification vector in the feature space corresponding to the color based on a loss function in distance learning, and adjusts the color parameters so as to reduce the loss.

[0089] In this embodiment, the classifier learning unit 104 calculates, for each anchor identification image, a contrastive classification loss relating to the closeness between the anchor identification vector of the anchor identification image and the positive classification vector of the positive classification image of the anchor identification image, and the closeness between the anchor identification vector of the anchor identification image and the negative classification vector of the negative classification image of the anchor identification image. The classifier learning unit 104 trains the classifier D based on the contrastive classification loss. The classifier learning unit 104 trains the classifier D so that the contrastive classification loss is small.

[0090] The contrastive discrimination loss is a contrastive loss calculated based on a discrimination image. The contrastive loss is a loss used in similarity learning. The contrastive loss is a loss for making data of the same class closer to each other and making data of different classes farther apart. The calculation formula for the contrastive loss itself can be a known calculation formula. In this embodiment, the discriminator training unit 104 calculates the contrastive discrimination loss based on the following formula 6. Formula 6 is an example of a formula for evaluating each of the difference between the anchor discrimination image and the positive discrimination image, the difference between the anchor discrimination image and the negative discrimination image, and the difference between the anchor discrimination image and another anchor discrimination image.

[0091]

number

[0092] In Equation 6, ε is a temperature parameter that indicates the influence of the positive and negative classification images in calculating the contrast classification loss. For example, ε is 0.1. α is the anchor classification vector of another anchor classification image, ζ j - In this embodiment, α is smaller than ε. For example, α is 0.01. ε and α may be any value and are not limited to the example of this embodiment.

[0093] For example, the classifier learning unit 104 learns the classifier D so that, in a feature space corresponding to each of the multiple features, the anchor identification vector of each of the multiple anchor identification images and the positive identification vector of the positive identification image of the anchor identification image become closer to each other, the anchor identification vector of the anchor identification image and the negative identification vector of the negative identification image of the anchor identification image become farther apart, and the anchor identification vector of the anchor identification image and the anchor identification vector of another anchor identification image become farther apart. In the example of Formula 6, the part "so that the anchor identification vector of the anchor identification image and the anchor identification vector of another anchor identification image become farther apart" is specified by the part after α.

[0094] The formula for calculating the contrast loss is not limited to Formula 6. For example, learning may not be performed so that a certain anchor identification image and another anchor identification image are separated from each other. In this case, a formula that does not include the coefficient α of Formula 6 may be used. Another formula that does not include the temperature parameter ε may be used. The formula for calculating the contrast loss may be another formula used in the triplet margin loss method.

[0095] For example, the classifier training unit 104 calculates a batchwise classification loss related to an average of the contrastive classification losses of each of the multiple anchor classification images. The batchwise classification loss is a loss that takes into account the batch size. For example, the classifier training unit 104 calculates the batchwise classification loss based on the following Equation 7. The classifier training unit 104 trains the classifier D based on the batchwise classification loss. The classifier training unit 104 trains the classifier D so that the batchwise classification loss is small.

[0096]

number

[0097] The formula for calculating the batchwise classification loss is not limited to Formula 7. For example, a weighting coefficient may be determined for each batch. The batchwise classification loss may be a sum instead of an average as in Formula 7. The classifier training unit 104 may perform training of the classifier D based on the contrastive loss without calculating the batchwise classification loss.

[0098] In this embodiment, the classifier training unit 104 trains the classifier D further based on the estimation result of the authenticity of the anchor identification image and the estimation result of the authenticity of the generated image generated by the generator G. The classifier training unit 104 trains the classifier D so that the probability that the anchor identification image is estimated to be true increases and the probability that the generated image generated by the generator G is estimated to be false increases. For example, the classifier training unit 104 calculates the opposing classification loss based on the following formula 8. The opposing classification loss is a competition loss used in training the classifier D. The classifier training unit 104 trains the classifier D based on the opposing classification loss. The classifier training unit 104 trains the classifier D so that the opposing classification loss becomes small.

[0099]

number

[0100] The formula for calculating the opposing classification loss is not limited to Formula 8. The opposing classification loss may be any loss for training the generator G and the classifier D to compete with each other. For example, in Formula 8, the average of the batch size is taken, but the sum may be used as the opposing classification loss without taking the average. The formula for calculating the opposing classification loss may be another formula used in known GANs. The opposing classification loss may not be used in training the generator G.

[0101] In this embodiment, the classifier learning unit 104 performs normalization on the estimation result of the authenticity of each of the multiple anchor identification images, and further performs learning of the classifier D based on the result of the normalization. In this embodiment, the case where R1 normalization is performed is taken as an example, but various known methods can be used for the normalization itself. For example, other normalization such as R2 normalization or Elastic Net regularization may be performed.

[0102] For example, the classifier training unit 104 calculates the normalized classification loss based on the following formula 9. The classifier training unit 104 trains the classifier D based on the normalized classification loss. The classifier training unit 104 trains the classifier D so that the normalized classification loss is small.

[0103]

number

[0104] In this embodiment, the classifier training unit 104 calculates the final loss based on the following formula 10. The classifier training unit 104 trains the classifier D based on the final loss. The classifier training unit 104 trains the classifier D so that the final loss is small. λ in formula 10 cl and λ R1 is a hyperparameter. For example, λ cl is 1.5. R1 is 5.0. Thus, the coefficient of the batchwise classification loss may be larger than that of the adversarial classification loss. Furthermore, the coefficient of the regularized classification loss may be larger than that of the batchwise classification loss.

[0105]

number

[0106] In addition, when multiple negative classification images in which different features are changed are prepared for each feature, the classifier training unit 104 may train the classifier D so that the anchor classification vector and the negative classification vector of each of the multiple negative classification images are separated from each other in the feature space corresponding to each of the multiple features.

[0107] [3-3. Functions for learning the generator] 7 and 8 are diagrams showing an example of a function for learning the generator G. The function for learning the generator G will be described with reference to FIG. 7 and FIG. 8, using the diagram showing the function of the learning terminal 10 shown in FIG. 3. As shown in FIG. 3, for example, the learning terminal 10 includes a data storage unit 100, a latent code acquisition unit 105, a generated image generation unit 106, a generated vector calculation unit 107, and a generator learning unit 108 as functions for learning the generator G. The data storage unit 100 is realized by the storage unit 12. Each of the latent code acquisition unit 105, the generated image generation unit 106, the generated vector calculation unit 107, and the generator learning unit 108 is realized by the control unit 11.

[0108] [Data storage section] The data storage unit 100 stores data necessary for learning the generator G. For example, the data storage unit 100 stores the actual data of the generator G included in the actual data of the GAN and the generated image database DB2. The actual data of the generator G indicates the program of the generator G and the parameters of the generator G. The parameters of the generator G are referenced by the program of the generator G. The parameters of the generator G are adjusted by the generator learning unit 108. For example, the parameters of the generator G are weight coefficients and biases. The parameters of the generator G may be known parameters. For example, the parameters of the generator G may be the number of hidden layers, the number of units in the hidden layer, or other hyperparameters.

[0109] 9 is a diagram showing an example of the generated image database DB2. The generated image database DB2 is a database in which generated images, which are images for training the generator G, are stored. In this embodiment, three generated images, namely, an anchor generated image, a positive generated image, and a negative generated image, are used in training the generator G. Hereinafter, when there is no need to distinguish between the anchor generated image, the positive generated image, and the negative generated image, they are simply referred to as generated images. The generated images can also be referred to as training images used as training data for the generator G.

[0110] For example, the generated image database DB2 stores feature identification information and image data of each of the anchor generated image, the positive generated image, and the negative generated image. Any data may be stored in the generated image database DB2. The data stored in the generated image database DB2 is not limited to the example of FIG. 9. For example, the generated image database DB2 may store information indicating which part of the latent code corresponding to which feature was changed to generate the negative generated image.

[0111] An anchor generated image is a generated image generated based on a latent code indicating a user-controllable feature. Hereinafter, when it is necessary to express each positive generated image in a mathematical formula, it is written as G(z). z is the latent code input to the generator G. As shown in FIG. 7, in this embodiment, the anchor latent code is divided into parts corresponding to each feature. In the example of FIG. 7, the part corresponding to the viewpoint is z v The part that corresponds to the global identity is z g The part that corresponds to the local identity is z l The part that corresponds to the color is z c In this embodiment, the number of dimensions of each part is the same as each other, but the number of dimensions of each part may be different from each other. When they need to be converted into intermediate latent codes by a mapping network, w v ,wg ,w l ,w c Hereinafter, the latent code for generating an anchor generated image is referred to as an anchor latent code.

[0112] A positive generated image is an image generated based on a latent code in which a portion of the anchor latent code of an anchor generated image that corresponds to at least one of a plurality of user-controllable features has been changed. Hereinafter, this latent code will be referred to as a positive latent code. In this embodiment, an example will be given in which a positive generated image is an image generated based on a positive latent code in which a portion of the anchor latent code that corresponds to any one of a plurality of user-controllable features has been changed, but a positive generated image may also be an image generated based on a positive latent code in which a portion of the anchor latent code that corresponds to each of a plurality of features has been changed. Hereinafter, when it is necessary to express each positive latent code in a mathematical formula, z + When it is necessary to express each positive generated image in a mathematical formula, G(z + ) should be written.

[0113] A negative generated image is an image generated based on a latent code in which a part of the anchor latent code that is different from the positive generated image has been changed. Hereinafter, this latent code will be referred to as a negative latent code. In this embodiment, an example will be given of a negative generated image generated based on a negative latent code in which one part of the anchor latent code that is different from the positive generated image has been changed, but the negative generated image may also be an image generated based on a negative latent code in which each of multiple other parts of the anchor latent code has been changed. Hereinafter, when it is necessary to express each positive latent code in a mathematical formula, z - When it is necessary to express each positive generated image in a mathematical formula, G(z - ) should be written.

[0114] The data stored in the data storage unit 100 is not limited to the above example. The data storage unit 100 may store any data. For example, the data storage unit 100 may store a generator learning program indicating a series of processes in learning the generator G. Program codes indicating each of a part of the process of the estimation unit 103, the process of the latent code acquisition unit 105, the process of the generated image generation unit 106, the process of the generated vector calculation unit 107, and the process of the generator learning unit 108 are shown. For example, the data storage unit 100 stores a program for changing the latent code.

[0115] [Latent Code Acquisition Section] The latent code acquisition unit 105 acquires, for each GAN feature in which a user can control multiple features related to a generated image, an anchor latent code, a positive latent code in which a part of the anchor latent code corresponding to the feature has been changed, and a negative latent code in which a part of the anchor latent code corresponding to another feature has been changed.

[0116] For example, the latent code acquisition unit 105 acquires an anchor latent code of a certain feature based on random noise. The latent code acquisition unit 105 acquires a positive latent code of the feature by changing a part of the anchor latent code that corresponds to the feature. The latent code acquisition unit 105 acquires a positive latent code of the feature by changing a part of the anchor latent code that corresponds to a feature other than the feature.

[0117] In the example of FIG. 7, the latent code acquisition unit 105 shows a process of acquiring an anchor latent code of a viewpoint, a positive latent code of a viewpoint, and a negative latent code of a viewpoint. For example, if the latent code has 200 dimensions, the first dimension to the 50th dimension is a part corresponding to the viewpoint. The 51st dimension to the 100th dimension is a part corresponding to the global identity. The 101st dimension to the 150th dimension is a part corresponding to the local identity. The 151st dimension to the 200th dimension is a part corresponding to the color. The number of dimensions of each feature does not have to be the same. For example, the part corresponding to the viewpoint may be 60 dimensions, and the part corresponding to the global identity may be 40 dimensions.

[0118] For example, the latent code acquisition unit 105 acquires a positive latent code of a viewpoint by changing a part of the anchor latent code of the viewpoint that corresponds to the viewpoint, the part being from the 1st dimension to the 50th dimension. This change may be performed randomly or a predetermined change may be performed. The latent code acquisition unit 105 acquires a negative latent code of a viewpoint by changing a part of the anchor latent code of the viewpoint other than the part being from the 1st dimension to the 50th dimension that corresponds to the viewpoint (in the example of FIG. 7, the part being from the 101st dimension to the 150th dimension that corresponds to the local identity). This change may also be performed randomly or a predetermined change may be performed.

[0119] For example, the latent code acquisition unit 105 acquires a positive latent code of a global identity by changing a part of the anchor latent code of the global identity from the 51st dimension to the 100th dimension corresponding to the global identity. This change may be performed randomly or may be a predetermined change. The latent code acquisition unit 105 acquires a negative latent code of a global identity by changing a part of the anchor latent code of the global identity other than the part of the 51st dimension to the 100th dimension corresponding to the global identity. This change may also be performed randomly or may be a predetermined change.

[0120] For example, the latent code acquisition unit 105 acquires a positive latent code of the local identity by changing a part of the anchor latent code of the local identity from the 101st dimension to the 150th dimension corresponding to the local identity. This change may be performed randomly or may be a predetermined change. The latent code acquisition unit 105 acquires a negative latent code of the local identity by changing a part of the anchor latent code of the local identity other than the part of the 101st dimension to the 150th dimension corresponding to the local identity. This change may also be performed randomly or may be a predetermined change.

[0121] For example, the latent code acquisition unit 105 acquires a positive latent code of a color by changing a part of the anchor latent code of the color that corresponds to the color from the 151st dimension to the 200th dimension. This change may be performed randomly or may be a predetermined change. The latent code acquisition unit 105 acquires a negative latent code of a color by changing a part of the anchor latent code of the color other than the part of the 151st dimension to the 200th dimension that corresponds to the color. This change may also be performed randomly or may be a predetermined change.

[0122] In this embodiment, multiple anchor latent codes are obtained for each feature. At least one positive latent code and at least one negative latent code are generated from each anchor latent code, so the latent code obtaining unit 105 obtains multiple anchor latent codes, multiple positive latent codes, and multiple negative latent codes for each feature. The methods for obtaining each anchor latent code, each positive latent code, and each negative latent code image are as described above.

[0123] In the present embodiment, the case where the latent code acquisition unit 105 generates the positive latent code and the negative latent code is exemplified, but the positive latent code and the negative latent code may be generated by a computer other than the learning terminal 10. The generated image database DB2 may store the positive latent code and the negative latent code generated by another computer. In this case, the learning terminal 10 may not have the function of generating the positive latent code and the negative latent code.

[0124] [Generated image generation section] The generated image generating unit 106 generates, for each feature, an anchor generated image corresponding to the anchor latent code of the feature, a positive generated image corresponding to the positive latent code of the feature, and a negative generated image corresponding to the negative latent code of the feature based on a generator G of the GAN. The anchor generated image is an image generated by inputting an anchor latent code to the generator G. The positive generated image is an image generated by inputting a positive latent code to the generator G. The negative generated image is an image generated by inputting a negative latent code to the generator G.

[0125] For example, the generated image generating unit 106 inputs an anchor latent code of a certain feature to the generator G during learning. The generator G converts the anchor latent code of the feature into an intermediate anchor latent code as necessary. This conversion does not have to be performed. The generator G executes a process such as convolution on the anchor latent code, and outputs an image according to the result of the process as an anchor generated image. This series of processes may be similar to the internal processes of the generator G of a known GAN.

[0126] For example, the generated image generating unit 106 inputs a positive latent code of a certain feature to the generator G during learning. The generator G converts the positive latent code of the feature into an intermediate positive latent code as necessary. This conversion does not have to be performed. The generator G performs a process such as convolution on the positive latent code, and outputs an image according to the result of the process as a positive generated image. This series of processes may be similar to the internal processes of the generator G of a known GAN.

[0127] For example, the generated image generating unit 106 inputs a negative latent code of a certain feature to the generator G during learning. The generator G converts the negative latent code of the feature into an intermediate negative latent code as necessary. This conversion does not have to be performed. The generator G performs a process such as convolution on the negative latent code, and outputs an image according to the result of the process as a negative generated image. This series of processes may be similar to the internal processes of the generator G of a known GAN.

[0128] In this embodiment, since the multiple features are three or more features, the generated image generating unit 106 may generate multiple negative generated images in which different feature changes are performed for each feature. For example, the generated image generating unit 106 inputs each of the multiple negative latent codes obtained for one anchor latent code to the generator G. The generator G generates a negative generated image based on each of the multiple negative latent codes. The generated image generating unit 106 obtains the multiple negative generated images output from the generator G.

[0129] In this embodiment, since multiple anchor latent codes are obtained for each feature, the generated image generating unit 106 generates, for each feature, an anchor generated image corresponding to each of the multiple anchor latent codes of the feature, a positive generated image corresponding to each of the multiple positive latent codes of the feature, and a negative generated image corresponding to each of the multiple negative latent codes of the feature. That is, the generated image acquiring unit acquires, for each feature, multiple anchor generated images, multiple positive generated images, and multiple negative generated images. The method of generating each of the individual anchor generated images, individual positive generated images, and individual negative generated images is as described above.

[0130] [Generated Vector Calculation Part] Generator vector calculation unit 107 calculates, for each feature space corresponding to each of a plurality of features, an anchor generator vector for the anchor generated image of the feature, a positive generator vector for the positive generated image of the feature, and a negative generator vector for the negative generated image of the feature, based on classifier D trained by classifier training unit 104. The processing of generator vector calculation unit 107 is similar to the processing of classification vector calculation unit 102, although the images input to classifier D differ between classification images and generated images.

[0131] For example, the generator vector calculation unit 107 inputs an anchor generated image of a certain feature to the classifier D. The generator vector calculation unit 107 acquires an anchor generated vector in the feature space corresponding to the feature based on the output from the classifier D. The generator vector calculation unit 107 inputs a positive generated image of a certain feature to the classifier D. The generator vector calculation unit 107 acquires a positive generated vector in the feature space corresponding to the feature based on the output from the classifier D. The generator vector calculation unit 107 inputs a negative generated image of a certain feature to the classifier D. The generator vector calculation unit 107 acquires a negative generated vector in the feature space corresponding to the feature based on the output from the classifier D.

[0132] In this embodiment, the anchor generated image, the positive generated image, and the negative generated image in the feature space are defined by the following formulas 11, 12, and 13, respectively. τ+ Similarly to Equation 3, Equation 4, and Equation 5, τ + is a mapping (computation of a feature vector) in the feature space corresponding to

[0133]

number

[0134]

number

[0135]

number

[0136] In addition, when a plurality of negative generated images in which different features are changed are obtained for each feature, the generating vector calculation unit 107 may calculate a negative generating vector for each of the plurality of negative generated images of the feature for each feature space corresponding to each of the plurality of features. In other words, when there are a plurality of negative generated images for one anchor generated image, the generating vector calculation unit 107 may calculate a negative generating vector for each of the plurality of negative generated images.

[0137] In this embodiment, since multiple anchor latent codes are obtained for each feature, the generating vector calculation unit 107 may calculate, for each feature space corresponding to each of the multiple features, an anchor generating vector of an anchor generated image corresponding to each of the multiple anchor latent codes of the feature, a positive generating vector of each of the multiple positive generated images of the feature, and a negative generating vector of each of the multiple anchor generated images of the feature. That is, the generating vector calculation unit 107 may calculate, for each feature space corresponding to each of the multiple features, an anchor generating vector of each of the multiple anchor generated images of the feature, a positive generating vector of each of the multiple positive generated images of the feature, and a negative generating vector of each of the multiple negative generated images of the feature. The calculation method of each of the individual anchor generating vectors, each of the individual positive generating vectors, and each of the individual negative generating vectors is as described above.

[0138] [Estimation part] The estimation unit 103 causes a classifier D to estimate the authenticity of each of the multiple anchor generated images. For example, the estimation unit 103 inputs the anchor generated image to the classifier D. The classifier D calculates a feature vector of the anchor generated image in a feature space for authenticity discrimination. The classifier D outputs an authenticity estimation result based on the feature vector. The estimation unit 103 acquires the authenticity estimation result output from the classifier D. The authenticity estimation method itself may be a publicly known method.

[0139] [Generator learning part] The generator learning unit 108 learns the generator G so that the anchor generation vector and the positive generation vector are closer to each other and the anchor generation vector and the negative generation vector are farther away from each other in the feature space corresponding to each of the multiple changes. In other words, the generator learning unit 108 learns the generator G so that the anchor generation vector and the positive generation vector belong to the same cluster and the anchor generation vector and the negative generation vector do not belong to the same cluster in the feature space corresponding to each of the multiple features.

[0140] The learning algorithm itself may be a known algorithm used in distance learning. For example, an algorithm such as gradient descent may be used for learning the generator G. In this embodiment, the relationship between the three generated images is used in learning, so the generator learning unit 108 learns the generator G based on the loss function of triplet margin loss so that the anchor generated vector and the positive generated vector are closer to each other and the anchor generated vector and the negative generated vector are farther away from each other. The loss function described in this embodiment is an example of the loss function of triplet margin loss.

[0141] For example, the generator learning unit 108 adjusts the parameters of the generator G so that the anchor generation vector and the positive generation vector become closer and the anchor generation vector and the negative generation vector become farther apart in the feature space corresponding to each of the multiple features, thereby learning the generator G. The generator learning unit 108 calculates a loss indicating the closeness of the anchor generation vector and the positive generation vector and the farthest between the anchor generation vector and the negative generation vector in the feature space corresponding to each of the multiple features based on a loss function in distance learning, and adjusts the parameters of the generator G so that the loss becomes smaller.

[0142] For example, the generator learning unit 108 adjusts the parameters of the generator G so that, in a feature space corresponding to the viewpoint, the anchor generation vector of the anchor generation image of the viewpoint and the positive generation vector of the positive generation image of the viewpoint are closer to each other, and the anchor generation vector and the negative generation vector of the negative generation image of the viewpoint are farther apart. The generator learning unit 108 calculates a loss indicating the closeness of the anchor generation vector and the positive generation vector, and the farthest between the anchor generation vector and the negative generation vector, in the feature space corresponding to the viewpoint, based on a loss function in distance learning, and adjusts the parameters of the generator G so as to reduce the loss.

[0143] In the example of FIG. 8, the state of the feature space corresponding to the viewpoint is shown. For example, by learning, the anchor generation image G(z 1 ) and the positive generated image G(z 1 + ) and , become closer to each other and belong to the same cluster. 1 ) and the negative generated image G(z 1 - ) and , and so they do not belong to the same cluster. 1 ) and the other anchor generated image G(z 2 )~G(z 4 ) and , and so they no longer belong to the same cluster. Similar learning is performed for feature spaces other than the feature space corresponding to the viewpoint.

[0144] For example, the generator learning unit 108 adjusts the parameters of the generator G so that the anchor generation vector of the anchor generation image of the global identity and the positive generation vector of the positive generation image of the global identity become closer to each other and the anchor generation vector and the negative generation vector of the negative generation image of the global identity become farther apart in the feature space corresponding to the global identity. The generator learning unit 108 calculates a loss indicating the closeness of the anchor generation vector and the positive generation vector and the farthest distance between the anchor generation vector and the negative generation vector in the feature space corresponding to the global identity based on a loss function in distance learning, and adjusts the parameters of the generator G so that the loss becomes smaller.

[0145] For example, the generator learning unit 108 adjusts the parameters of the generator G so that, in the feature space corresponding to the local identity, the anchor generation vector of the anchor generation image of the local identity and the positive generation vector of the positive generation image of the local identity are closer to each other and the anchor generation vector and the negative generation vector of the negative generation image of the local identity are farther apart. The generator learning unit 108 calculates a loss indicating the closeness of the anchor generation vector and the positive generation vector and the farthest distance of the anchor generation vector and the negative generation vector in the feature space corresponding to the local identity based on a loss function in distance learning, and adjusts the parameters of the generator G so that the loss is reduced.

[0146] For example, the generator learning unit 108 adjusts the parameters of the generator G so that the anchor generation vector of the anchor generation image of the color and the positive generation vector of the positive generation image of the color are closer to each other and the anchor generation vector and the negative generation vector of the negative generation image of the color are farther apart in the feature space corresponding to the color, based on the loss function in distance learning, and adjusts the parameters of the generator G so that the loss is smaller.

[0147] In this embodiment, the generator training unit 108 calculates, for each anchor generated image, a contrastive generation loss related to the closeness between the anchor generated vector of the anchor generated image and the positive generated vector of the positive generated image of the anchor generated image, and the closeness between the anchor generated vector of the anchor generated image and the negative generated vector of the negative generated image of the anchor generated image. The generator training unit 108 trains the generator G based on the contrastive discrimination loss. The generator training unit 108 trains the generator G so that the contrastive discrimination loss is small.

[0148] The contrastive generation loss is a contrastive loss calculated based on the generated image. The contrastive loss is a loss used in similarity learning. The contrastive loss is a loss for making data of the same class closer to each other and making data of different classes farther apart. The calculation formula for the contrastive loss itself can be a known calculation formula. In this embodiment, the generator learning unit 108 calculates the contrastive generation loss based on the following formula 14. Formula 14 is an example of a formula for evaluating each of the difference between the anchor generated image and the positive generated image, the difference between the anchor generated image and the negative generated image, and the difference between the anchor generated image and the other anchor generated image. Note that the values ​​of ε and α in Formula 14 and the values ​​of ε and α in Formula 6 may be the same or different from each other.

[0149]

number

[0150] The generator learning unit 108 learns the generator G so that, in a feature space corresponding to each of the multiple features, the anchor generation vector of each of the multiple anchor generated images and the positive generated vector of the positive generated image of the anchor generated image approach each other, the anchor generation vector of the anchor generated image and the negative generated vector of the negative generated image of the anchor generated image move away from each other, and the anchor generation vector of the anchor generated image and the anchor generation vector of the other anchor generated images move away from each other. In the example of Equation 14, the part "so that the anchor generation vector of the anchor generated image and the anchor generation vector of the other anchor generated images move away from each other" is specified by the part after α.

[0151] For example, the generator training unit 108 calculates a batchwise generation loss related to an average of the contrastive generation loss of each of the multiple anchor generated images. The batchwise generation loss is a loss that takes into account the batch size. For example, the generator training unit 108 calculates the batchwise generation loss based on the following Equation 15. The generator training unit 108 trains the generator G based on the batchwise generation loss. The generator training unit 108 trains the generator G so that the batchwise generation loss is small.

[0152]

number

[0153] The calculation formula for the batchwise generation loss is not limited to Equation 15. For example, a weighting coefficient may be determined for each batch. The batchwise generation loss may be a sum instead of an average as in Equation 15. The generator training unit 108 may perform training of the generator G based on the contrastive loss without calculating the batchwise generation loss.

[0154] In this embodiment, the generator learning unit 108 learns the generator G further based on the estimation result of the authenticity of the anchor generated image. For example, the generator learning unit 108 learns the generator G further based on the estimation result of the authenticity of each of the multiple anchor generated images. The generator learning unit 108 calculates the opposing generation loss based on the following formula 16. The opposing discrimination loss is an opposing loss used in training the generator G. The generator learning unit 108 learns the generator G based on the opposing generation loss. The generator learning unit 108 learns the generator G so that the opposing generation loss is small.

[0155]

number

[0156] The formula for calculating the opposing generation loss is not limited to Formula 16. The opposing generation loss may be any loss for training the generator G and the discriminator D to compete with each other. For example, in Formula 16, the average of the batch size is taken, but the average may not be taken and the total may be used as the opposing generation loss. The formula for calculating the opposing generation loss may be another formula used in known GANs. The opposing generation loss may not be used in the training of the generator G.

[0157] In this embodiment, the generator learning unit 108 calculates the final loss based on the following formula 17. The generator learning unit 108 learns the generator G based on the final loss. The generator learning unit 108 learns the generator G so that the final loss becomes small. λ in formula 17 cl is a hyperparameter. λ in Equation 17 cl The value of and λ in Eq. cl The value of λ may be the same as that of λ. In this case, the coefficient of the opposing generation loss may be larger than the coefficient of the batchwise classification loss. cl The value of and λ in Eq. cl The values ​​of and may be different from each other.

[0158]

number

[0159] In addition, when multiple negative generated images are generated for each feature in which other different features are changed, the generator learning unit 108 may train the generator G so that the anchor generated vector and the negative generated vector of each of the multiple negative generated images become separated in the feature space corresponding to each of the multiple features.

[0160] [4. Processing performed by the learning system] Figures 10 and 11 are diagrams showing an example of processing executed by the learning system 1. The processing in Figures 10 and 11 is executed by the control units 11, 21, and 31 executing programs stored in the storage units 12, 22, and 32, respectively. In Figures 10 and 11, the processing for training the discriminator D, the processing for training the generator G, and the processing for utilizing the trained generator G are shown as a series of processing, but these may be performed separately.

[0161] 10, the learning terminal 10 acquires an anchor identification image, a positive identification image, and a negative identification image for each feature based on the identification image database DB1 (S1). The learning terminal 10 calculates an anchor identification vector, a positive identification vector, and a negative identification vector for each feature space corresponding to each of the multiple features (S2).

[0162] The learning terminal 10 trains the discriminator D so that the anchor discrimination vector and the positive discrimination vector become closer to each other and the anchor discrimination vector and the negative discrimination vector become farther apart in a feature space corresponding to each of a plurality of features (S3). In S3, the learning terminal 10 calculates various losses based on Formulas 6 to 9. The learning terminal 10 calculates a final loss based on Formula 10. The learning terminal 10 trains the discriminator D so that the final loss becomes smaller.

[0163] The learning terminal 10 acquires an anchor latent code, a positive latent code, and a negative latent code for each feature (S4). The learning terminal 10 generates an anchor generated image, a positive generated image, and a negative generated image for each feature based on the anchor latent code, the positive latent code, and the negative latent code and the generator G (S5). The learning terminal 10 calculates an anchor generated vector, a positive generated vector, and a negative generated vector for each feature space corresponding to each of the multiple features based on the classifier D (S6).

[0164] The learning terminal 10 performs learning of the generator G so that the anchor generation vector and the positive generation vector become closer to each other and the anchor generation vector and the negative generation vector become farther apart in a feature space corresponding to each of a plurality of features (S7). In S7, the learning terminal 10 calculates various losses based on Formulas 14 to 16. The learning terminal 10 calculates the final loss based on Formula 17. The learning terminal 10 performs learning of the generator G so that the final loss becomes smaller.

[0165] 11, the learning terminal 10 transmits the trained generator G to the server 20 (S8). In S8, the learning terminal 10 may transmit the trained discriminator D together with the trained generator G to the server 20. When the server 20 receives the trained generator G from the learning terminal 10 (S9), it records the trained generator G in the storage unit 22. The user terminal 30 transmits feature data indicating features designated by the user to the server 20 (S10).

[0166] When the server 20 receives the feature data from the user terminal 30 (S11), it generates a latent code according to the feature indicated by the feature data (S12). The server 20 generates a generated image based on the latent code and the trained generator G (S13). The server 20 transmits image data indicating the generated image to the user terminal 30 (S14). The user terminal 30 receives the image data from the server 20 (S15). The user terminal 30 displays the generated image on the display unit 35 based on the image data (S16), and the process ends.

[0167] [5. Summary of the embodiment] The learning system 1 of this embodiment acquires an anchor identification image, a positive identification image, and a negative identification image for each feature. Based on the classifier D, the learning system 1 calculates an anchor identification vector of the feature, a positive identification vector of the feature, and a negative identification vector of the feature for each feature space corresponding to each of the multiple features. The learning system 1 trains the classifier D so that the anchor identification vector and the positive identification vector are close to each other and the anchor identification vector and the negative identification vector are farther apart in the feature space corresponding to each of the multiple features. This allows the learning system 1 to reduce the effort required for training the classifier D. For example, the learning system 1 reduces the effort required for training the classifier D without manually labeling the identification image or using a classification model that has been trained in advance. In addition, the learning system 1 realizes efficient learning by performing the advance preparation of images to be trained by the classifier D and the training of the classifier D using the images in a single go. The learning system 1 also improves the accuracy of the classifier D through such learning.

[0168] Furthermore, the learning system 1 acquires, for each feature, a plurality of negative identification images in which other mutually different features have been changed. The learning system 1 calculates, for each feature space corresponding to each of the plurality of features, a negative identification vector for each of the plurality of negative identification images of the feature. The learning system 1 trains the classifier D so that the anchor identification vector and the negative identification vector for each of the plurality of negative identification images are separated in the feature space corresponding to each of the plurality of features. This allows the learning system 1 to generate a larger number of negative identification images, thereby further reducing the effort required for training the GAN. The accuracy of the classifier D is also further improved.

[0169] Furthermore, the learning system 1 calculates a contrastive discrimination loss for each anchor identification image. The learning system 1 calculates a batchwise discrimination loss related to the average of the contrastive discrimination losses for each of the multiple anchor identification images. The learning system 1 trains the classifier D based on the batchwise discrimination loss. This allows the learning system 1 to perform GAN training based on the batchwise discrimination loss that comprehensively takes into account the multiple anchor identification images. As a result, the accuracy of the classifier D is improved.

[0170] In addition, the learning system 1 trains the classifier D so that, in a feature space corresponding to each of the multiple features, the anchor identification vector of each of the multiple anchor identification images and the positive identification vector of the positive identification image of the anchor identification image are closer to each other, the anchor identification vector of the anchor identification image and the negative identification vector of the negative identification image of the anchor identification image are farther away from each other, and the anchor identification vector of the anchor identification image and the anchor identification vector of the other anchor identification images are farther away from each other. This allows the learning system 1 to realize learning that takes into account the relationship between the anchor identification images, thereby further reducing the effort required for learning the GAN. The accuracy of the classifier D is also improved.

[0171] Furthermore, the learning system 1 causes the classifier D to estimate the authenticity of the anchor identification image and the authenticity of the generated image generated by the generator G of the GAN. The learning system 1 trains the classifier D further based on the estimation result of the authenticity of the anchor identification image and the estimation result of the authenticity of the generated image generated by the generator G. This allows the learning system 1 to realize learning that takes into account the estimation result of the authenticity of the classifier D, thereby further reducing the effort required for training the GAN. The accuracy of the classifier D is also further improved.

[0172] Furthermore, the learning system 1 causes the classifier D to estimate the authenticity of each of the multiple anchor identification images. The learning system 1 performs normalization on the estimation result of the authenticity of each of the multiple anchor identification images. The learning system 1 performs training of the classifier D further based on the result of the normalization. This allows the learning system 1 to realize training that takes into account the normalized estimation result of authenticity of the classifier D, thereby further reducing the effort required for training the GAN. The accuracy of the classifier D is also further improved. The learning system 1 can further reduce the effort required for training the GAN.

[0173] Further, the learning system 1 acquires an anchor latent code, a positive latent code, and a negative latent code for each feature. Based on the generator G, the learning system 1 generates, for each feature, an anchor generated image corresponding to the anchor latent code of the feature, a positive generated image corresponding to the positive latent code of the feature, and a negative generated image corresponding to the negative latent code of the feature. Based on the trained classifier D, the learning system 1 calculates, for each feature space corresponding to each of the multiple features, an anchor generated vector related to the anchor generated image of the feature, a positive generated vector related to the positive generated image of the feature, and a negative generated vector related to the negative generated image of the feature. The learning system 1 trains the generator G so that the anchor generated vector and the positive generated vector are closer to each other and the anchor generated vector and the negative generated vector are farther apart in the feature spaces corresponding to each of the multiple changes. This allows the learning system 1 to further reduce the effort required for training the generator G. For example, the learning system 1 reduces the effort required for training the generator G without manually preparing multiple latent codes or using a pre-trained classification model. The learning system 1 also realizes efficient learning by seamlessly preparing images to be trained by the generator G and training the generator G using the images. The learning system 1 also improves the accuracy of the generator G through such learning.

[0174] Furthermore, the learning system 1 generates a plurality of negative generated images in which different feature modifications have been performed for each feature. The learning system 1 calculates a negative generation vector for each of the plurality of negative generated images for each feature space corresponding to each of the plurality of features. The learning system 1 trains the generator G so that the anchor generation vector and the negative generation vector of each of the plurality of negative generated images are separated in the feature space corresponding to each of the plurality of features. This allows the learning system 1 to generate a larger number of negative generated images, thereby further reducing the effort required for training the GAN. The accuracy of the generator G is also improved.

[0175] Furthermore, the learning system 1 calculates a contrastive generation loss for each anchor-generated image. The learning system 1 calculates a batchwise generation loss related to the average of the contrastive generation losses for each of the multiple anchor-generated images. The learning system 1 trains the generator G based on the batchwise generation loss. This allows the learning system 1 to perform GAN training based on the batchwise generation loss that comprehensively takes into account the multiple anchor-generated images. As a result, the accuracy of the generator G is improved.

[0176] In addition, the learning system 1 trains the generator G so that, in a feature space corresponding to each of the multiple features, the anchor generation vector of each of the multiple anchor generation images and the positive generation vector of the positive generation image of the anchor generation image approach each other, the anchor generation vector of the anchor generation image and the negative generation vector of the negative generation image of the anchor generation image move away from each other, and the anchor generation vector of the anchor generation image and the anchor generation vector of the other anchor generation image move away from each other. This allows the learning system 1 to realize learning that takes into account the relationship between the anchor generation images, thereby further reducing the effort required for learning the GAN. The accuracy of the generator G is also improved.

[0177] Furthermore, the learning system 1 causes the classifier D to estimate the authenticity of the anchor-generated image. The learning system 1 trains the generator G based on the estimation result of the authenticity of the anchor-generated image. This allows the learning system 1 to realize training that takes into account the estimation result of the authenticity of the classifier D, thereby further reducing the effort required for training the GAN. The accuracy of the generator G is also further improved.

[0178] [6. Modifications] The present disclosure is not limited to the above-described embodiments. The present disclosure can be modified as appropriate without departing from the spirit and scope of the present disclosure.

[0179] For example, the process for training the discriminator D and the process for training the generator G may be executed on separate computers. A first learning terminal 10 may execute the process for training the discriminator D, and a second learning terminal 10 may execute the process for training the generator G. For example, the process described as being executed on the learning terminal 10 may be executed on the server 20, the user terminal 30, or another computer. The process described as being executed on the learning terminal 10 may be shared among multiple computers.

[0180] [7. Notes] For example, the learning system can be configured as follows. (1) A discrimination image acquisition unit that acquires an anchor discrimination image, a positive discrimination image in which the feature of the anchor discrimination image is changed, and a negative discrimination image in which other features of the anchor discrimination image are changed, for each feature of a GAN (Generative Adversarial Network) in which a user can control a plurality of features related to a generated image; A discrimination vector calculation unit that calculates, for each feature space corresponding to each of the plurality of features based on the GAN discriminator, an anchor discrimination vector for the anchor discrimination image of the feature, a positive discrimination vector for the positive discrimination image of the feature, and a negative discrimination vector for the negative discrimination image of the feature; a classifier training unit that trains the classifier so that the anchor identification vector and the positive identification vector become closer to each other and the anchor identification vector and the negative identification vector become farther apart in the feature space corresponding to each of the plurality of features; A learning system including: (2) The plurality of features is three or more features, The identification image acquisition unit acquires, for each of the features, a plurality of the negative identification images in which the other features different from each other are changed, the discrimination vector calculation unit calculates, for each of the feature spaces corresponding to each of the plurality of features, the negative discrimination vector of each of the plurality of negative discrimination images of the feature; the classifier training unit trains the classifier so that the anchor classification vector and the negative classification vector of each of the plurality of negative classification images become separated from each other in the feature space corresponding to each of the plurality of features; A learning system as described in (1). (3) The identification image acquisition unit acquires a plurality of the anchor identification images, a plurality of the positive identification images, and a plurality of the negative identification images for each of the features, the discrimination vector calculation unit calculates, for each of the feature spaces corresponding to each of the plurality of features, the anchor discrimination vector of each of the plurality of anchor discrimination images of the feature, the positive discrimination vector of each of the plurality of positive discrimination images of the feature, and the negative discrimination vector of each of the plurality of negative discrimination images of the feature; The classifier learning unit Calculate a contrastive discrimination loss for each of the anchor identification images, the closeness between the anchor identification vector of the anchor identification image and the positive discrimination vector of the positive discrimination image of the anchor identification image, and the closeness between the anchor identification vector of the anchor identification image and the negative discrimination vector of the negative discrimination image of the anchor identification image; Calculate a batchwise discrimination loss relative to the average of the contrastive discrimination losses for each of the plurality of anchor discrimination images; training the classifier based on the batchwise classification loss; A learning system according to (1) or (2). (4) The identification image acquisition unit acquires a plurality of the anchor identification images, a plurality of the positive identification images, and a plurality of the negative identification images for each of the features, the discrimination vector calculation unit calculates, for each of the feature spaces corresponding to each of the plurality of features, the anchor discrimination vector of each of the plurality of anchor discrimination images of the feature, the positive discrimination vector of each of the plurality of positive discrimination images of the feature, and the negative discrimination vector of each of the plurality of negative discrimination images of the feature; The classifier learning unit learning the classifier so that, in the feature space corresponding to each of the plurality of features, the anchor identification vector of each of the plurality of anchor identification images and the positive identification vector of the positive identification image of the anchor identification image are close to each other, the anchor identification vector of the anchor identification image and the negative identification vector of the negative identification image of the anchor identification image are separated from each other, and the anchor identification vector of the anchor identification image and the anchor identification vector of another anchor identification image are separated from each other. A learning system according to any one of (1) to (3). (5) The learning system further includes an estimation unit that causes the classifier to estimate the authenticity of the anchor identification image and the authenticity of the generated image generated by the generator of the GAN, The classifier learning unit further learns the classifier based on an estimation result of authenticity of the anchor identification image and an estimation result of authenticity of the generated image generated by the generator. A learning system according to any one of (1) to (4). (6) The learning system further includes an estimation unit that causes the classifier to estimate the authenticity of each of the plurality of anchor identification images, The classifier learning unit performs normalization on the estimation result of the authenticity of each of the plurality of anchor identification images, and performs learning of the classifier based on the result of the normalization. A learning system according to any one of (1) to (5). (7) A latent code acquisition unit that acquires, for each feature of a generative adversarial network (GAN) that allows a user to control a plurality of features related to a generated image, an anchor latent code, a positive latent code in which a portion of the anchor latent code corresponding to the feature has been changed, and a negative latent code in which a portion of the anchor latent code corresponding to another feature has been changed; A generated image generator that generates, for each of the features based on the generator of the GAN, an anchor generated image corresponding to the anchor latent code of the feature, a positive generated image corresponding to the positive latent code of the feature, and a negative generated image corresponding to the negative latent code of the feature; A generator vector calculation unit that calculates, for each feature space corresponding to each of the plurality of features based on a classifier trained by the classifier training unit described in (1), an anchor generator vector for the anchor generated image of the feature, a positive generator vector for the positive generated image of the feature, and a negative generator vector for the negative generated image of the feature; a generator learning unit that learns the generator so that the anchor generating vector and the positive generating vector become closer to each other and the anchor generating vector and the negative generating vector become farther apart in the feature space corresponding to each of the plurality of changes; A learning system including: (8) The plurality of features is three or more features, the generated image generating unit generates a plurality of the negative generated images in which the other feature changes different from each other are performed for each of the features; the generating vector calculation unit calculates, for each of the feature spaces corresponding to each of the plurality of features, the negative generating vector of each of the plurality of negative generated images of the feature; The generator learning unit performs learning of the generator so that the anchor generated vector and the negative generated vector of each of the negative generated images become separated in the feature space corresponding to each of the plurality of features. A learning system as described in (7). (9) the latent code acquisition unit acquires, for each of the features, a plurality of anchor latent codes, a plurality of positive latent codes, and a plurality of negative latent codes; the generated image generating unit generates, for each of the features, the anchor generated image corresponding to each of the plurality of anchor latent codes of the feature, the positive generated image corresponding to each of the plurality of positive latent codes of the feature, and the negative generated image corresponding to each of the plurality of negative latent codes of the feature; the generating vector calculation unit calculates, for each of the feature spaces corresponding to each of the plurality of features, the anchor generating vector of the anchor generating image corresponding to each of the plurality of anchor latent codes of the feature, the positive generating vector of each of the plurality of positive generating images of the feature, and the negative generating vector of each of the plurality of anchor generating images of the feature; The generator learning unit is For each anchor generated image, calculate a contrast generation loss for the closeness of the anchor generated vector of the anchor generated image to the positive generated vector of the positive generated image of the anchor generated image, and for the closeness of the anchor generated vector of the anchor generated image to the negative generated vector of the negative generated image of the anchor generated image; calculating a batchwise generation loss relative to an average of the contrast generation loss for each of the plurality of anchor generated images; training the generator based on the batchwise generation losses; A learning system according to (7) or (8). (10) the generated image acquisition unit acquires, for each of the features, a plurality of the anchor generated images, a plurality of the positive generated images, and a plurality of the negative generated images; the generating vector calculation unit calculates, for each of the feature spaces corresponding to each of the plurality of features, the anchor generating vector of each of the plurality of anchor generated images of the feature, the positive generating vector of each of the plurality of positive generated images of the feature, and the negative generating vector of each of the plurality of negative generated images of the feature; The generator learning unit is learning the generator so that, in the feature space corresponding to each of the plurality of features, the anchor generation vector of each of the plurality of anchor generated images and the positive generated vector of the positive generated image of the anchor generated image are closer to each other, the anchor generation vector of the anchor generated image and the negative generated vector of the negative generated image of the anchor generated image are separated from each other, and the anchor generation vector of the anchor generated image and the anchor generation vector of another anchor generated image are separated from each other. A learning system according to any one of (7) to (9). (11) The learning system further includes an estimation unit that causes the classifier to estimate authenticity of the anchor generation image, The generator learning unit further learns the generator based on the estimation result of the authenticity of the anchor generated image. A learning system according to any one of (7) to (10). [Explanation of symbols]

[0181] 1 Learning system, 10 Learning terminal, 20 Server, 11, 21, 31 Control unit, 12, 22, 32 Memory unit, 13, 23, 33 Communication unit, 14, 34 Operation unit, 15, 35 Display unit, 30 User terminal, 100 Data memory unit, 101 Discrimination image acquisition unit, 102 Discrimination vector calculation unit, 103 Estimation unit, 104 Classifier learning unit, 105 Latent code acquisition unit, 106 Generated image generation unit, 107 Generated vector calculation unit, 108 Generator learning unit, DB1 Discrimination image database, DB2 Generated image database, D Classifier, G Generator, N Network.

Claims

1. A discrimination image acquisition unit that acquires, for each feature of a generative adversarial network (GAN) in which a user can control a plurality of features related to a generated image, an anchor discrimination image, a positive discrimination image in which the feature of the anchor discrimination image is changed, and a negative discrimination image in which other features of the anchor discrimination image are changed; a discrimination vector calculation unit that calculates, for each feature space corresponding to each of the plurality of features based on the GAN discriminator, an anchor discrimination vector for the anchor discrimination image of the feature, a positive discrimination vector for the positive discrimination image of the feature, and a negative discrimination vector for the negative discrimination image of the feature; a classifier training unit that trains the classifier so that the anchor identification vector and the positive identification vector become closer to each other and the anchor identification vector and the negative identification vector become farther apart in the feature space corresponding to each of the plurality of features; A learning system including:

2. The plurality of features is three or more features, The identification image acquisition unit acquires, for each of the features, a plurality of the negative identification images in which the other features different from each other are changed, the discrimination vector calculation unit calculates, for each of the feature spaces corresponding to each of the plurality of features, the negative discrimination vector of each of the plurality of negative discrimination images of the feature; the classifier training unit performs training of the classifier so that the anchor classification vector and the negative classification vector of each of the plurality of negative classification images become separated from each other in the feature space corresponding to each of the plurality of features; The learning system according to claim 1 .

3. The identification image acquisition unit acquires a plurality of the anchor identification images, a plurality of the positive identification images, and a plurality of the negative identification images for each of the features; the discrimination vector calculation unit calculates, for each of the feature spaces corresponding to each of the plurality of features, the anchor discrimination vector of each of the plurality of anchor discrimination images of the feature, the positive discrimination vector of each of the plurality of positive discrimination images of the feature, and the negative discrimination vector of each of the plurality of negative discrimination images of the feature; The classifier learning unit Calculate a contrastive discrimination loss for each of the anchor identification images, the closeness between the anchor identification vector of the anchor identification image and the positive discrimination vector of the positive discrimination image of the anchor identification image, and the closeness between the anchor identification vector of the anchor identification image and the negative discrimination vector of the negative discrimination image of the anchor identification image; Calculate a batchwise discrimination loss relative to the average of the contrastive discrimination losses for each of the plurality of anchor discrimination images; training the classifier based on the batchwise classification loss; A learning system according to claim 1 or 2.

4. The identification image acquisition unit acquires a plurality of the anchor identification images, a plurality of the positive identification images, and a plurality of the negative identification images for each of the features, the discrimination vector calculation unit calculates, for each of the feature spaces corresponding to each of the plurality of features, the anchor discrimination vector of each of the plurality of anchor discrimination images of the feature, the positive discrimination vector of each of the plurality of positive discrimination images of the feature, and the negative discrimination vector of each of the plurality of negative discrimination images of the feature; The classifier learning unit learning the classifier so that, in the feature space corresponding to each of the plurality of features, the anchor identification vector of each of the plurality of anchor identification images and the positive identification vector of the positive identification image of the anchor identification image are close to each other, the anchor identification vector of the anchor identification image and the negative identification vector of the negative identification image of the anchor identification image are separated from each other, and the anchor identification vector of the anchor identification image and the anchor identification vector of another anchor identification image are separated from each other. A learning system according to claim 1 or 2.

5. The learning system further includes an estimation unit that causes the classifier to estimate the authenticity of the anchor identification image and the authenticity of the generated image generated by the generator of the GAN; The classifier learning unit further learns the classifier based on an estimation result of authenticity of the anchor identification image and an estimation result of authenticity of the generated image generated by the generator. A learning system according to claim 1 or 2.

6. The learning system further includes an estimation unit that causes the classifier to estimate the authenticity of each of the plurality of anchor identification images, The classifier learning unit performs normalization on the estimation result of the authenticity of each of the plurality of anchor identification images, and performs learning of the classifier based on the result of the normalization. A learning system according to claim 1 or 2.

7. a latent code acquisition unit that acquires, for each feature of a generative adversarial network (GAN) in which a user can control a plurality of features related to a generated image, an anchor latent code, a positive latent code in which a portion of the anchor latent code corresponding to the feature has been changed, and a negative latent code in which a portion of the anchor latent code corresponding to another feature has been changed; a generated image generator that generates, for each of the features based on a generator of the GAN, an anchor generated image corresponding to the anchor latent code of the feature, a positive generated image corresponding to the positive latent code of the feature, and a negative generated image corresponding to the negative latent code of the feature; a generator vector calculation unit that calculates, for each feature space corresponding to each of the plurality of features based on a classifier trained by the classifier training unit according to claim 1, an anchor generator vector for the anchor generated image of the feature, a positive generator vector for the positive generated image of the feature, and a negative generator vector for the negative generated image of the feature; a generator learning unit that learns the generator so that the anchor generating vector and the positive generating vector become closer to each other and the anchor generating vector and the negative generating vector become farther apart in the feature space corresponding to each of the plurality of changes; A learning system including:

8. The plurality of features is three or more features, the generated image generating unit generates a plurality of the negative generated images in which the other feature changes different from each other are performed for each of the features; the generating vector calculation unit calculates, for each of the feature spaces corresponding to each of the plurality of features, the negative generating vector of each of the plurality of negative generated images of the feature; The generator learning unit performs learning of the generator so that the anchor generated vector and the negative generated vector of each of the negative generated images become separated in the feature space corresponding to each of the plurality of features. The learning system according to claim 7.

9. the latent code acquisition unit acquires, for each of the features, a plurality of anchor latent codes, a plurality of positive latent codes, and a plurality of negative latent codes; the generated image generating unit generates, for each of the features, the anchor generated image corresponding to each of the plurality of anchor latent codes of the feature, the positive generated image corresponding to each of the plurality of positive latent codes of the feature, and the negative generated image corresponding to each of the plurality of negative latent codes of the feature; the generating vector calculation unit calculates, for each of the feature spaces corresponding to each of the plurality of features, the anchor generating vector of the anchor generating image corresponding to each of the plurality of anchor latent codes of the feature, the positive generating vector of each of the plurality of positive generating images of the feature, and the negative generating vector of each of the plurality of anchor generating images of the feature; The generator learning unit is For each anchor generated image, calculate a contrast generation loss for the closeness of the anchor generated vector of the anchor generated image to the positive generated vector of the positive generated image of the anchor generated image, and for the closeness of the anchor generated vector of the anchor generated image to the negative generated vector of the negative generated image of the anchor generated image; calculating a batchwise generation loss relative to an average of the contrast generation loss for each of the plurality of anchor generated images; training the generator based on the batchwise generation losses; A learning system according to claim 7 or 8.

10. the generated image acquisition unit acquires, for each of the features, a plurality of the anchor generated images, a plurality of the positive generated images, and a plurality of the negative generated images; the generating vector calculation unit calculates, for each of the feature spaces corresponding to each of the plurality of features, the anchor generating vector of each of the plurality of anchor generated images of the feature, the positive generating vector of each of the plurality of positive generated images of the feature, and the negative generating vector of each of the plurality of negative generated images of the feature; The generator learning unit is learning the generator so that, in the feature space corresponding to each of the plurality of features, the anchor generation vector of each of the plurality of anchor generated images and the positive generated vector of the positive generated image of the anchor generated image are closer to each other, the anchor generation vector of the anchor generated image and the negative generated vector of the negative generated image of the anchor generated image are separated from each other, and the anchor generation vector of the anchor generated image and the anchor generation vector of another anchor generated image are separated from each other. A learning system according to claim 7 or 8.

11. The learning system further includes an estimation unit that causes the classifier to estimate authenticity of the anchor generation image, The generator learning unit further learns the generator based on the estimation result of the authenticity of the anchor generated image. A learning system according to claim 7 or 8.

12. The computer A discrimination image acquisition step of acquiring an anchor discrimination image, a positive discrimination image in which the feature of the anchor discrimination image is changed, and a negative discrimination image in which other features of the anchor discrimination image are changed, for each of the features of a generative adversarial network (GAN) in which a user can control a plurality of features related to a generated image; a discrimination vector calculation step of calculating, for each feature space corresponding to each of the plurality of features based on the GAN discriminator, an anchor discrimination vector for the anchor discrimination image of the feature, a positive discrimination vector for the positive discrimination image of the feature, and a negative discrimination vector for the negative discrimination image of the feature; a classifier training step of training the classifier so that the anchor identification vector and the positive identification vector become closer to each other and the anchor identification vector and the negative identification vector become farther apart in the feature space corresponding to each of the plurality of features; Learn how to do it.

13. The computer a latent code acquisition step of acquiring, for each feature of a generative adversarial network (GAN) in which a user can control a plurality of features related to a generated image, an anchor latent code, a positive latent code in which a portion of the anchor latent code corresponding to the feature has been changed, and a negative latent code in which a portion of the anchor latent code corresponding to another feature has been changed; a generated image generation step of generating, for each of the features based on a generator of the GAN, an anchor generated image corresponding to the anchor latent code of the feature, a positive generated image corresponding to the positive latent code of the feature, and a negative generated image corresponding to the negative latent code of the feature; a generating vector calculation step of calculating, for each feature space corresponding to each of the plurality of features, an anchor generating vector for the anchor generating image of the feature, a positive generating vector for the positive generating image of the feature, and a negative generating vector for the negative generating image of the feature, based on a classifier trained by the classifier training unit according to claim 12; a generator learning step of learning the generator so that the anchor generating vector and the positive generating vector become closer to each other and the anchor generating vector and the negative generating vector become farther apart in the feature space corresponding to each of the plurality of changes; Learn how to do it.

14. a discrimination image acquisition unit that acquires, for each feature of a generative adversarial network (GAN) in which a user can control a plurality of features related to a generated image, an anchor discrimination image, a positive discrimination image in which the feature of the anchor discrimination image is changed, and a negative discrimination image in which another feature of the anchor discrimination image is changed; a discrimination vector calculation unit that calculates, for each feature space corresponding to each of the plurality of features based on the GAN discriminator, an anchor discrimination vector for the anchor discrimination image of the feature, a positive discrimination vector for the positive discrimination image of the feature, and a negative discrimination vector for the negative discrimination image of the feature; a classifier training unit that trains the classifier so that the anchor identification vector and the positive identification vector become closer to each other and the anchor identification vector and the negative identification vector become farther apart in the feature space corresponding to each of the plurality of features; A program that makes a computer function as a

15. a latent code acquisition unit that acquires, for each feature of a generative adversarial network (GAN) in which a user can control a plurality of features related to a generated image, an anchor latent code, a positive latent code in which a portion of the anchor latent code corresponding to the feature has been changed, and a negative latent code in which a portion of the anchor latent code corresponding to another feature has been changed; a generated image generator that generates, for each of the features based on a generator of the GAN, an anchor generated image corresponding to the anchor latent code of the feature, a positive generated image corresponding to the positive latent code of the feature, and a negative generated image corresponding to the negative latent code of the feature; a generator vector calculation unit that calculates, for each feature space corresponding to each of the plurality of features, an anchor generator vector for the anchor generated image of the feature, a positive generator vector for the positive generated image of the feature, and a negative generator vector for the negative generated image of the feature, based on a classifier trained by the classifier training unit according to claim 14; a generator learning unit that learns the generator so that the anchor generating vector and the positive generating vector become closer to each other and the anchor generating vector and the negative generating vector become farther apart in the feature space corresponding to each of the plurality of changes; A program that makes a computer function as a

Images