Data generation method, data processing device, and program

Abstract

To provide a technique for selecting data generated by a generation model according to the uniqueness of the data.SOLUTION: A data generation method for generating new data includes: a density estimation step of estimating the density of the new data in embedded space using a correspondence relation between a reference latent variable, which is a latent variable serving as a source of reference data generated previously by a generation model for generating data using the latent variable as an input, and density of the reference data in the embedded space, and a new latent variable, which is a latent variable serving as a source of the new data; and an evaluation step of evaluating the new latent variable according to the density of the new data estimated in the density estimation step.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a data generation method, a data processing device, and a program. [Background technology]

[0002] Patent Document 1 discloses a technology for outputting an output image in which an input image of a natural object is multiplied with an artificial object, based on a learning model that has been trained to output an image in which an input image of a natural object is multiplied with an artificial object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-168078 Summary of the Invention [Problem to be solved by the invention]

[0004] In a technology for outputting data generated by a generative model that generates data based on input data, as in Patent Document 1, the data generated by the generative model may be similar to each other, resulting in a lower diversity of the output data than desired. Conversely, for example, when lower diversity data is desired, the diversity of the output data may be higher than desired. Therefore, a technology for selecting data generated by a generative model based on the uniqueness of the data is desired. [Means for solving the problem]

[0005] According to a first aspect of the present disclosure, there is provided a data generation method, comprising: a density estimation step of estimating the density of new data in an embedding space using a correspondence between a reference latent variable, which is a latent variable that is a source of reference data previously generated by a generative model that generates data using latent variables as input, and the density of the reference data in an embedding space, and a new latent variable, which is the latent variable that is a source of new data; and an evaluation step of evaluating the new latent variable according to the density of the new data estimated in the density estimation step.

[0006] According to a second aspect of the present disclosure, there is provided a data processing device including a processor that executes a density estimation process of estimating a density of new data in an embedding space using a correspondence between a reference latent variable, which is a latent variable that is a source of reference data previously generated by a generative model that generates data using latent variables as input, and a density of the reference data in an embedding space, and a new latent variable, which is a latent variable that is a source of new data, and an evaluation process of evaluating the new latent variable according to the density of the new data estimated by the density estimation process.

[0007] According to a third aspect of the present disclosure, there is provided a program that causes a computer to implement a density estimation function that estimates the density of new data in an embedding space using a correspondence between a reference latent variable, which is a latent variable that is the source of reference data previously generated by a generative model that generates data using latent variables as input, and a density in an embedding space of the reference data, and a new latent variable, which is a latent variable for generating the new data, and a function that evaluates the new latent variable according to the density of the new data estimated by the density estimation function. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration of a data processing system. [Figure 2]A conceptual diagram explaining the learning method of the learning model. [Figure 3] 10 is a flowchart of pre-processing. [Figure 4] 4 is a flowchart of a main process in the first embodiment. [Figure 5] FIG. 4 is an explanatory diagram showing an example of a probability density function calculated by a calculation unit. [Figure 6] FIG. 10 is a diagram showing an example of new data. [Figure 7] 10 is a flowchart of a main process in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] A. First embodiment: FIG. 1 is a block diagram showing a schematic configuration of a data processing system 50 according to the first embodiment. The data processing system 50 is used to generate new generated data using a generative model 170. The generated data may be various types of data, such as image data, spectral data, or character string data. Hereinafter, the new generated data may also be simply referred to as "new data." The new data is used for various purposes by a user Ur. For example, if the new data is image data, the new data may be used to obtain new design ideas.

[0010] The data processing system 50 is also used to select new data based on its uniqueness. The uniqueness of data represents the degree to which other data similar to the data in the data group to which the data belongs are scarce. The greater the number of other data similar to the data in the data group to which the data belongs, above a predetermined level, the higher the uniqueness of the data. Generally, whether or not data are similar to each other can be determined based on the distance between the data or the similarity between the data. In this case, for example, Euclidean distance or Manhattan distance can be used as the distance between the data. Furthermore, for example, cosine similarity or similarity based on cross entropy can be used as the similarity. Furthermore, the uniqueness of each data correlates with the diversity of each data. Specifically, a data group containing more unique data means that the diversity of each data included in the data group is higher. Hereinafter, a data group with more diverse data and a data set with more diverse data will be simply referred to as a "highly diverse data group" or a "highly diverse data set."

[0011] The data processing system 50 includes a data processing device 100. The data processing device 100 in this embodiment is configured by a computer including one or more processors 101, a storage unit 102 including ROM and RAM, an input / output interface 103, and an internal bus 104. The processor 101, the storage unit 102, and the input / output interface 103 are connected via the internal bus 104 to enable bidirectional communication. An output device 105 and an input device 106 are connected to the input / output interface 103. The storage unit 102 in this embodiment stores a program 155, a learning model 160, reference embedded data 165, a generative model 170, and a regression model 175. By executing the program 155 stored in the memory unit 102, the processor 101 realizes various functions including the functions of a data generation unit 108, an embedded data generation unit 115, a calculation unit 120, a regression model generation unit 122, an estimation unit 123, an evaluation unit 125, an evaluation condition reception unit 135, and a learning execution unit 140.

[0012] The input device 106 is configured by, for example, a mouse and a keyboard. The data processing device 100 receives various inputs from the user Ur via the input device 106.

[0013] The output device 105 outputs various data processing results obtained by the data processing device 100. The data processing results include result information, which will be described later. In this embodiment, the output device 105 is configured as a display device that outputs the data processing results as visual information. The display device is configured, for example, by a liquid crystal panel or an organic EL panel. The output device 105 as a display device may be configured, for example, as a touch panel that can accept touch operations from the user Ur. In this case, the output device 105 may also function as the input device 106, for example. In other embodiments, the output device 105 may be configured, for example, as a speaker that outputs the data processing results as audio information, or as a device that outputs the data processing results by transmitting them to an external computer or recording medium.

[0014] The learning model 160 is trained to project input data into an embedding space. In the embedding space, each piece of data is projected so that the feature vectors of more similar data are closer in distance. The embedding space in this embodiment is a 64-dimensional vector space. The embedding space is also called a feature space. The feature vectors in the embedding space are also called embedding vectors or embedded representations.

[0015] In this embodiment, the learning model 160 has been trained by self-supervised learning (SSL). The learning model 160 is configured as a machine learning model using Resnet. More specifically, the learning model 160 is configured as a Resnet model that is a backbone included in an encoder in a network structure that has been trained using SimSiam. Details of the learning model 160 will be described later.

[0016] The generative model 170 is also called generative AI (generative artificial intelligence). In this embodiment, the generative model 170 is configured as a trained model that uses latent variables as input and generates generated data. The training of the generative model 170 may be performed in the data processing device 100 or outside the data processing device 100.

[0017] In particular, a generative adversarial network (GAN) that has been trained to generate image data using latent variables as input is used as the generative model 170 in this embodiment. In the GAN, a generator and a discriminator are trained using a training dataset for training the GAN. Specifically, the generator is trained to be able to generate fake data that approximates true data, and the discriminator is trained to be able to discriminate data generated by the generator. In this embodiment, a projected GAN is used as the generative model 170.

[0018] The data generation unit 108 generates generated data using latent variables. The generated generated data is stored in the storage unit 102. Note that, prior to generating the generated data using the generative model 170, the data generation unit 108 selects latent variables to be used in generating the generated data. The data generation unit 108 selects the latent variables based on, for example, random numbers. As the random numbers for selecting the latent variables, for example, random numbers following a probability distribution such as a normal distribution are used. The latent variables selected in this manner are so-called noise data.

[0019] The generated data can be used as new data, as well as reference data or learning data, which will be described later. Hereinafter, the latent variables that are the source of generated data as new data will also be referred to as new latent variables. Furthermore, the latent variables that are the source of generated data as reference data will also be referred to as reference latent variables. As will be described later, in this embodiment, in the evaluation step, it is evaluated whether or not the selected new latent variables will actually be used to generate new data. If it is evaluated that the new latent variables will be used, the data generation unit 108 generates new data by inputting the new latent variables into the generation model 170. Hereinafter, the step and process of generating new data will also be referred to as the new data generation step and the new data generation process, respectively.

[0020] The embedding data generation unit 115 generates reference embedding data 165 in which multiple reference data are projected into the embedding space. The reference data is generated by the generative model 170 as described above, and serves as a reference for obtaining correspondences, which will be described later. In the reference embedding data 165, the projected reference data is represented as a vector in the embedding space. Hereinafter, this vector in which the reference data is projected into the embedding space is also referred to as a reference embedding vector. Since the reference embedding data 165 is generated based on each reference data generated by the generative model 170, the reference embedding data 165 reflects the tendency of data characteristics that are likely to be generated by the generative model 170. Hereinafter, the process and operation for generating the reference embedding vector are also referred to as a reference generation process and a reference generation process, respectively.

[0021] The calculation unit 120 calculates the density of the reference data in the embedding space using the reference embedding vector. Note that the "density of the reference data" here refers to the density of the reference embedding data 165 at the coordinates where the reference data is located in the embedding space into which each reference data included in the reference dataset is projected. The reference dataset is composed of multiple reference data used to generate the reference embedding data 165.

[0022] The calculation unit 120 in this embodiment calculates the density of the reference data using kernel density estimation (KDE). Specifically, the calculation unit 120 first calculates a probability density function representing the probability distribution of each reference data in the embedding space using KDE based on the reference embedding data 165. The calculated probability density function represents the distribution density of data points of each reference data in the embedding space. The calculation unit 120 then calculates the density of each reference data based on the calculated probability density function and the vector coordinates of each reference data projected into the embedding space.

[0023] In this embodiment, the calculation unit 120 calculates a normalized density, which is a normalized density, as the density of the new data. In this disclosure, "normalization" includes so-called min-max normalization and "standardization." Standardization is also called Z-score normalization. In this embodiment, the calculation unit 120 calculates a density normalized by Z-score normalization as the density of the new data. More specifically, the calculation unit 120 uses each reference data as a population and normalizes each density in the population by Z-score normalization so that the average of the density in the population is 0 and the variance is 1. As a result, a normalized density is calculated as the density of each reference data.

[0024] The regression model generation unit 122 generates a regression model 175 using a reference latent variable and the density of the reference data. The regression model 175 is a trained model that has learned the correspondence between the reference latent variable and the density of the reference data. Note that the correspondence in this embodiment represents the correspondence between the reference latent variable and the normalized density of the reference data. In this embodiment, the regression model 175 is a kernel regression model, which is a ridge regression model that uses a Gaussian kernel as the kernel function. In other embodiments, the regression model 175 may be various nonlinear regression models or various linear relational models. Furthermore, the regression model 175 may be a deep learning model.

[0025] The estimation unit 123 estimates the density of new data in the embedding space using the correspondence and the selected new latent variable. Specifically, the estimation unit 123 uses the regression model 175 and the new latent variable to estimate the density of new data generated using the new latent variable as input. Hereinafter, the density estimated by the estimation unit 123 will also be referred to as the estimated density. Furthermore, the process, processing, and function for estimating the estimated density will also be referred to as the density estimation process, density estimation processing, and density estimation function, respectively. As described above, the correspondence in this embodiment is the correspondence between the reference latent variable and the normalized density of the reference data, and therefore the estimated density reflects the result of normalizing the density of the reference data.

[0026] The evaluation unit 125 evaluates the selected new latent variables according to the estimated density. Here, for example, when relatively unique new data is projected into the embedding space, the projected new data is located farther from each reference data in the embedding space. As a result, the density of the new data in the embedding space is relatively low. Conversely, when relatively unique new data is projected into the embedding space, the projected new data is located closer to each reference data in the embedding space. As a result, the density of the new data in the embedding space is relatively high. As such, since the density and uniqueness of the new data are favorably correlated, the evaluation unit 125 can evaluate the new latent variables from the perspective of the level of uniqueness of the new data generated based on the new latent variables. Hereinafter, the process and step of evaluating the new latent variables according to the estimated density are also referred to as the evaluation process and evaluation process, respectively.

[0027] In detail, in this embodiment, the evaluation unit 125 evaluates whether or not to use a new latent variable to generate new data according to the estimated density. Specifically, when the estimated density of the new data is equal to or greater than a predetermined threshold, the evaluation unit 125 evaluates not to use the new latent variable that is the source of the new data. Furthermore, when the estimated density of the new data is less than the threshold, the evaluation unit 125 evaluates to use the new latent variable that is the source of the new data. In other words, the evaluation process in this embodiment can be said to be a process of determining whether or not to use a new latent variable according to the estimated density.

[0028] As described above, the data generation unit 108 inputs the new latent variables evaluated for use by the evaluation unit 125 into the generative model 170, and generates new data based on the new latent variables. As a result, new data is generated that is highly likely to be unique.

[0029] The evaluation condition receiving unit 135 receives, from the user Ur, a designation of the evaluation conditions for the evaluation process. In this embodiment, the evaluation condition receiving unit 135 receives, as the evaluation conditions, the designation of the first threshold value and the designation of the second threshold value. Specifically, the evaluation condition receiving unit 135 receives the designation from the user Ur, for example, by displaying a dialog box for designating the first threshold value and the second threshold value on the display unit of the output device 105. The user Ur designates the evaluation conditions by, for example, inputting the evaluation conditions via the input device 106.

[0030] The learning execution unit 140 executes machine learning of the learning model 160. In this embodiment, the learning execution unit 140 uses at least a portion of a reference dataset as a training dataset to train the learning model 160. The training dataset is a dataset for training the learning model 160. The training dataset includes a plurality of pieces of generated data generated by the generative model 170. The training dataset in this embodiment includes reference data. Specifically, the training dataset includes the entire reference dataset. In other words, it can be said that a portion of the training dataset and the entire reference dataset are the same data.

[0031] FIG. 2 is a conceptual diagram illustrating a learning method of the learning model 160 in this embodiment. FIG. 2 shows a model structure SM for realizing learning by SimSiam. The model structure SM includes an encoder En and a predictor Pd. The encoder En is configured by a neural network. The encoder En includes a backbone Bb and a projector Pj as a head. The projector Pj is configured by a fully connected layer including an output layer among the layers of the neural network that configures the encoder En. The predictor Pd is a neural network configured by two fully connected layers.

[0032] When input data Dt, such as image data, is input to a model structure SM, first and second data augmentations DA, which are different data augmentations, are performed on the input data Dt. The first data Dt1, which is the input data Dt subjected to the first data augmentation, and the second data Dt2, which is the input data Dt subjected to the second data augmentation, are each input to an encoder En. When the input data Dt is image data, the data augmentation DA may be, for example, vertical flipping, horizontal flipping, rotation, enlargement, reduction, cropping, translation, color jitter, brightness jitter, contrast jitter, saturation jitter, hue jitter, grayscaling, Gaussian blur, partial masking, etc. "Different data augmentations" means that the types or numbers of data augmentations DA to be performed are different. The types and numbers of data augmentations to be performed as the first and second data augmentations are each determined probabilistically.

[0033] The encoder En outputs a feature vector z1 of the input first data Dt1 and a feature vector z2 of the input second data Dt2.

[0034] The predictor Pd converts and outputs the input feature vector so that the feature vector based on the first data Dt1 matches the feature vector based on the second data Dt2. The feature vector z1 and feature vector z2 output from the encoder En are each input to the predictor Pd alternatively. FIG. 2 shows a situation in which the feature vector z1 is input to the predictor Pd. When the feature vector z1 is input to the predictor Pd, the predictor Pd outputs a feature vector p1 based on the feature vector z1. On the other hand, when the feature vector z2 is input to the predictor Pd, the predictor Pd outputs a feature vector p2 based on the feature vector z2. Note that the feature vector p2 is not shown in FIG. 2.

[0035] In the similarity evaluation SA, the similarity between a feature vector based on the first data Dt1 and a feature vector based on the second data Dt2 is calculated. Specifically, in the similarity evaluation SA, the similarity D(p1, z2) between feature vector p1 and feature vector z2 and the similarity D(p2, z1) between feature vector z1 and feature vector p2 are calculated. The similarity D(p1, z2) is expressed by the following formula (1). Note that the similarity D(p2, z1) is expressed by a formula obtained by replacing p1 with p2 and z2 with z1 in formula (1).

[0036]

number

[0037] In the model structure SM, learning is performed so as to minimize the loss function L. The loss function L is expressed by the following equation (2).

[0038]

number

[0039] In learning in the model structure SM, the parameters of the encoder En and the predictor Pd are updated using gradient descent and backpropagation so that the loss function L is minimized. However, in learning in the model structure SM, a gradient stopping operation GS is executed. That is, the above formula (2) can be expressed by the following formula (3). In formula (3), stopgrad represents gradient stopping.

[0040]

number

[0041] That is, in updating the parameters of the encoder En, the feature vector z1 in the similarity D(p2, z1) and z2 in the similarity D(p1, z2) are treated as constants. Note that Fig. 2 shows how the gradient stopping operation GS is performed for the second data Dt2. Specifically, in Fig. 2, the encoder En receives the gradient Gd for the first data Dt1, but does not receive the gradient Gd for the second data Dt2.

[0042] Generally, collapse can be a problem in SSL. Collapse means that the learning model is trained so that a fixed feature vector is output from the encoder regardless of the data input to the encoder. In SimSiam, it is known that collapse is suppressed by arranging a predictor Pd corresponding to one of the first data Dt1 and the second data Dt2 and executing a gradient stopping operation GS corresponding to the other data. Therefore, by using SimSiam to train the learning model 160, the learning model 160 can be trained effectively without using negative samples or a momentum encoder.

[0043] As the learning model 160, a part or all of the encoder En can be used. In this embodiment, the backbone Bb of the encoder En is used as the learning model 160. That is, when data is input to the learning model 160, a feature vector based on the data is output by the learning model 160. The feature vector thus output corresponds to embedded data obtained by projecting the input data into the embedding space.

[0044] The type of data augmentation DA performed during training of the learning model 160 affects the factors that contribute to the uniqueness of new data. For example, when grayscaling is used as the data augmentation DA, differences in hue contribute less to the uniqueness of new data than when grayscaling is not used. Similarly, when left-right flipping is used as the data augmentation DA, differences in left-right orientation between image data contribute less to the uniqueness of new data. Therefore, for example, by setting the type of data augmentation DA according to desired factors, new latent variables can be more effectively evaluated.

[0045] Fig. 3 is a flowchart of the pre-processing in this embodiment. For example, a predetermined input from the user Ur via the input device 106 is used as a trigger to cause the processor 101 to start the pre-processing shown in Fig. 3. The data processing method in this embodiment is realized by the pre-processing in Fig. 3 and the main processing described later.

[0046] In steps S105 to S115, the processor 101 executes a learning process for training the learning model 160 by machine learning. First, in step S105, the data generation unit 108 generates a generation data group consisting of multiple pieces of generation data using the generation model 170, and stores the group in the storage unit 102. In step S110, the learning execution unit 140 acquires a training dataset from the generation data group stored in the storage unit 102. Note that the learning execution unit 140 may acquire, for example, the entire generation data group as the training dataset. In step S115, the learning execution unit 140 executes machine learning of the learning model 160 using the training dataset acquired in step S110.

[0047] In step S120, the embedding data generation unit 115 acquires a reference dataset. As described above, in this embodiment, the reference dataset is included in the training dataset. Note that the embedding data generation unit 115 may acquire, for example, the entire training dataset as the reference dataset. In step S125, the embedding data generation unit 115 generates reference embedded data 165 using the training model 160 trained in step S115 and the reference dataset acquired in step S120. The embedding data generation unit 115 stores the reference embedded data 165 thus generated in the storage unit 102. The process of generating the reference embedded data 165, as in step S125, is also referred to as a pre-generation process. The pre-generation process is executed prior to a density calculation process, which will be described later.

[0048] In step S130, the calculation unit 120 calculates the density of each reference data using the reference embedding vector, specifically, the reference embedding data 165 generated in step S125. The step and process of calculating the density of the reference data as in step S130 are also referred to as a density calculation step and a density calculation process, respectively.

[0049] In step S135, the regression model generation unit 122 generates a regression model 175 that has learned the correspondence through machine learning using the density calculated in step S130 and each reference latent variable. The step and process of generating the regression model 175, such as step S135, are also referred to as a regression step and a regression process, respectively.

[0050] 4 is a flowchart of the main processing in this embodiment. The main processing is started by the processor 101, for example, after the completion of the preliminary processing, when a predetermined input by the user Ur via the input device 106 is used as a trigger.

[0051] In step S205, the data generation unit 108 selects a new latent variable. In step S210, the estimation unit 123 estimates an estimated density using the new latent variable selected in step S205 and the trained regression model 175 generated in step S135 of FIG. 3.

[0052] In step S215, the evaluation condition receiving unit 135 receives the evaluation condition specification from the user Ur. The step and process of receiving the evaluation condition specification from the user Ur, as in step S215, are also referred to as the receiving step and the receiving process, respectively. Note that the receiving step may be executed at any timing before the evaluation step is started.

[0053] In step S220, the evaluation unit 125 evaluates the new latent variable according to the estimated density estimated in step S210. Specifically, in step S220, the evaluation unit 125 evaluates whether or not to use the new latent variable to generate new data according to the estimated density. If the new latent variable is evaluated as being used in step S220, the data generation unit 108 generates new data using the new latent variable in step S225. If the new latent variable is evaluated as not being used in step S220, the data generation unit 108 returns to step S205 without generating new data. Note that, in the re-execution of step S205, a latent variable different from that selected in the previously executed step S205 is likely to be selected based on a random number. Furthermore, the processor 101 may present the new data generated in step S225 to the user Ur. Furthermore, the processor 101 may present the evaluation result of the new latent variable in step S220 to the user Ur. That is, the data generation method may include a step of presenting the generated new data to the user Ur and a step of presenting the evaluation result of the new latent variable to the user Ur. Furthermore, the processor 101 may store the generated new data in a recording medium, for example, without presenting the generated new data to the user Ur, or while presenting the data to the user Ur. That is, the data generation method may include a storage step of storing the generated new data. In this way, for example, by sequentially storing multiple pieces of new data in a recording medium and compiling the stored new data, a data set including a large amount of highly unique new data can be created.

[0054] The above main process may be repeatedly executed until a predetermined number of new data items are generated, for example. In this case, the number of new data items to be generated may be specified by the user Ur via the input device 106, for example.

[0055] Fig. 5 is an explanatory diagram showing an example of a probability density function calculated by the calculation unit 120. Fig. 6 is a diagram showing an example of new data.

[0056] The probability density function in FIG. 5 and each new data in FIG. 6 were obtained using the learning model 160, the reference embedding data 165, and the generative model 170 prepared under the following conditions. The generative model 170 was trained to generate image data as generated data using a dataset consisting of multiple image data. Each image data included in the dataset for training the generative model 170 is classified into an "airplane class." In other words, the generative model 170 was trained to generate image data that approximates images in the airplane class. The reference embedding data 165 and the reference embedding data 165 were obtained by inputting each image data generated by the generative model 170 as reference data into the trained learning model 160.

[0057] In Fig. 5, the probability density function, which is actually calculated as a function representing the density of data points in a 64-dimensional embedding space, is schematically represented in a two-dimensional XY space having mutually orthogonal X and Y coordinates. In Fig. 5, areas of the XY space with higher density are indicated by darker hatching.

[0058] FIG. 6 shows a data group Gr1 including only new data based on new latent variables with low evaluation scores, and a data group Gr2 including only new data based on new latent variables with high evaluation scores. FIG. 6 shows 10 image data as examples of new data included in each data group. In FIG. 6, colored portions of pixels included in each image data that differ from the white background are hatched. All of the image data included in data group Gr1 were image data in which the colored portions were located near the vertical center of each image, more specifically, image data in which the entire body of an airplane was depicted near the vertical center. On the other hand, the image data included in data group Gr2 varied, for example, in the location and area of ​​the colored portions compared to data group Gr1. In other words, the data in data group Gr2 had higher uniqueness and diversity than the data in data group Gr1.

[0059] According to the data generation method of the present embodiment described above, new latent variables that are the source of new data are evaluated according to their estimated densities. Since the estimated densities are favorably correlated with the uniqueness of new data based on the new variables, new data can be selected according to their uniqueness using the evaluation results of the evaluation step.

[0060] It is known that the pieces of generated data generated by the generative model 170 often end up being similar to each other due to various factors. In this embodiment, for example, even when the pieces of generated data generated by the generative model 170 tend to be similar to each other, new data can be selected so that the uniqueness of the selected new data is further improved. Note that in other embodiments, it is also possible to select new data so that the uniqueness of the selected new data is further reduced, for example.

[0061] Furthermore, in this embodiment, the evaluation step evaluates whether or not a new latent variable will be used to generate new data, and if it is evaluated that the new latent variable will be used, the new latent variable is input into the generative model 170 to generate new data. This allows for a preliminary evaluation of whether or not new data will be generated using the new latent variable, and new data can be generated based on the new latent variable that is evaluated to be used. Therefore, new data can be effectively selected based on its uniqueness. Specifically, for example, compared to a configuration in which the uniqueness of new data is evaluated after it is generated, it is possible to prevent the generation of unnecessary new data that will not be selected. Furthermore, compared to a configuration in which the new latent variable that was the source of the new data is evaluated after it is generated, it is possible to prevent the generation of unnecessary new data that will not be selected.

[0062] Furthermore, in this embodiment, prior to the evaluation step, a receiving step is executed in which a designation of evaluation conditions is received from the user Ur. Therefore, the evaluation step can be executed according to the desired evaluation conditions designated by the user Ur.

[0063] In this embodiment, the correspondence relationship represents the relationship between the reference latent variable and the normalized density of the reference data. In this way, the result of normalizing the density of the reference data is reflected in the estimated density estimated using the correspondence relationship, making it easier to execute the evaluation process.

[0064] In this embodiment, the regression model 175 is generated by executing the reference generation step, density calculation step, and regression step, and the generated regression model 175 can be used to estimate the estimated density in the density estimation step.

[0065] In this embodiment, the learning model 160 has been trained using at least a portion of the reference data set. This allows the learning model 160 to more appropriately output feature vectors of the reference data when the reference data is input to the learning model 160, compared to when no reference data set is used at all for training the learning model 160. As a result, the learning model 160 can more appropriately project the reference data into the embedding space and generate more appropriate reference embedded data 165. This allows the density calculation process and evaluation process to be more appropriately performed.

[0066] Furthermore, in this embodiment, the learning model 160 has been trained by self-supervised learning. By training the learning model 160 in this way by self-supervised learning, the learning model 160 can be properly trained so as to realize data projection into an embedding space in which feature vectors of more similar data are closer in distance, without using labeled training data. In this embodiment, by using the learning model 160 properly trained in this way, new latent variables can be more properly evaluated in the evaluation process. Furthermore, the effort required to prepare a training dataset for training the learning model 160 can be reduced. Specifically, in this embodiment, for example, compared to when the learning model 160 is trained by supervised learning, there is no need to prepare a training dataset including labeled data, and therefore the effort required to prepare the training dataset can be reduced.

[0067] B. Second embodiment: 7 is a flowchart of the main processing in the second embodiment. Unlike the first embodiment, in the evaluation step, the evaluation unit 125 evaluates that a new latent variable whose estimated density is a first density will be used with a higher probability than a new latent variable whose estimated density is a second density. The second density is a density greater than the first density. The data processing device 100 in this embodiment is similar to the first embodiment in the respects that are not specifically described.

[0068] The evaluation step in this embodiment includes step S217 in addition to step S220b. In step S217, the evaluation unit 125 determines the use probability of using a new latent variable according to the estimated density estimated in step S210. Specifically, the evaluation unit 125 determines the use probability of a new latent variable whose estimated density is the second density to be the first probability, and determines the use probability of a new latent variable whose estimated density is the first density to be the second probability. The first probability is higher than the second probability. As a result, it is determined that a new latent variable whose estimated density is the first density will be used with a higher probability than a new latent variable whose estimated density is the second density.

[0069] In addition, in the reception process of step S215 in this embodiment, the evaluation condition reception unit 135 may receive, as an evaluation condition, a designation of a first probability or a second probability in addition to or instead of thresholds such as the first threshold or the second threshold.

[0070] In step S220b, the evaluation unit 125 evaluates whether or not to use a new latent variable. However, in step S220b, unlike step S220 in Fig. 4, the evaluation unit 125 evaluates whether or not to use a new latent variable according to the use probability determined in step S217.

[0071] According to the data processing method of the present embodiment described above, in the evaluation step, new latent variables with an estimated density of the first density are evaluated as being used with a higher probability than new latent variables with an estimated density of the second density. In this way, new latent variables with a higher probability of generating new data with higher uniqueness are primarily used, while new latent variables with a higher probability of generating new data with lower uniqueness can also be used. Therefore, new data can be generated more efficiently compared to a mode in which new latent variables with an estimated density of the second density are not used at all.

[0072] In another embodiment in which a new latent variable is evaluated according to a use probability, the evaluation unit 125 may evaluate the new latent variable not in two stages as described above, but in three or more stages, for example. In this case, for example, each use probability may be determined so that the use probability differs in stages according to each stage.

[0073] C. Other Embodiments: (C-1) In each of the above embodiments, the evaluation step is performed before new data is generated, but it may also be performed after the new data is generated. That is, after new data is generated based on a new latent variable, the new latent variable may be evaluated ex post. In this case, the evaluation results of the new latent variable may be output using the output device 105 and presented to the user Ur. In this way, the user Ur can select new data by referring to the evaluation results. Even in this form, new data can be selected according to its uniqueness.

[0074] (C-2) In the above embodiment, a Projected GAN is used for the generative model 170, but this is not limited to this. For example, various GANs such as a Conditional GAN ​​(CGAN), a Deep Convolutional GAN ​​(DCGAN), a CycleGAN, or a StyleGAN are used for the generative model 170. Furthermore, the generative model 170 is not limited to a GAN, and for example, a Variational Autoencoder (VAE), a Flow-based Generative Model, or a Diffusion Model may also be used.

[0075] (C-3) In the above embodiment, the learning model 160 has been trained using SimSiam. In contrast, the learning model 160 may be trained using various methods other than SimSiam, as long as it has been trained to project input data into an embedding space. For example, the learning model 160 may be trained using SimCLR, BYOL, or SwAV. Furthermore, the learning model 160 may be trained using various learning methods, such as metric learning, other than SSL. In this case, the learning model 160 may be trained using, for example, unsupervised learning, supervised learning, semi-supervised learning, or reinforcement learning. Specifically, the learning model 160 may be a model trained by supervised learning or the like to perform class classification of input data.

[0076] (C-4) In each of the above embodiments, the training dataset includes the entire reference dataset. In contrast, the training dataset may include, for example, only a portion of the reference dataset, or may not include the reference dataset at all. Furthermore, all or a portion of the training dataset may be identical to all or a portion of the reference dataset. Furthermore, all or a portion of the training dataset may be identical to new data. Furthermore, all or a portion of the reference dataset may be identical to new data.

[0077] (C-5) In each of the above embodiments, the data processing method includes a pre-generation step, but the pre-generation step may not be included. In this case, for example, the reference embedding data 165 may be prepared in advance by a computer external to the data processing device 100.

[0078] (C-6) In each of the above embodiments, the data processing method includes a regression step, but it does not necessarily have to include a regression step. In this case, for example, the regression model generation unit 122 may be prepared in advance in a computer external to the data processing device 100.

[0079] (C-7) In each of the above embodiments, the learning process is performed in the data processing device 100. However, the learning process does not have to be performed in the data processing device 100. In this case, machine learning of the learning model 160 may be performed, for example, on a computer external to the data processing device 100.

[0080] (C-8) In each of the above embodiments, the correspondence represents the correspondence between the reference latent variable and the normalized density of the reference data. In contrast, the correspondence may represent, for example, the correspondence between the reference latent variable and the non-normalized density.

[0081] (C-9) In each of the above embodiments, KDE is used in the calculation step. However, instead of KDE, other non-parametric density estimation methods such as histogram density estimation (HDE) or a parametric density estimation method may be used in the calculation step.

[0082] (C-10) In each of the above embodiments, the evaluation condition receiving unit 135 may be configured to be able to receive designation of various evaluation conditions, not limited to thresholds and use probabilities. For example, the evaluation condition receiving unit 135 may receive designation of whether or not to execute an evaluation step according to a use probability, as an evaluation condition.

[0083] D. Other forms: The present disclosure is not limited to the above-described embodiments and can be realized in various forms without departing from the spirit thereof. For example, the present disclosure can also be realized in the following forms. The technical features in the above embodiments corresponding to the technical features in each form described below can be appropriately replaced or combined to solve some or all of the problems of the present disclosure or to achieve some or all of the effects of the present disclosure. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted.

[0084] (1) According to a first aspect of the present disclosure, there is provided a data generation method, comprising: a density estimation step of estimating the density of new data in an embedding space using a correspondence between a reference latent variable, which is a latent variable that is a source of reference data previously generated by a generative model that generates data using latent variables as input, and the density of the reference data in an embedding space, and a new latent variable, which is the latent variable that is a source of new data; and an evaluation step of evaluating the new latent variable according to the density of the new data estimated in the density estimation step. According to this aspect, the density of the estimated new data correlates favorably with the uniqueness, and therefore, by using the evaluation results of the evaluation step, the new data can be selected according to the uniqueness.

[0085] (2) In the above embodiment, the evaluation step may further include a new data generation step in which the evaluation step evaluates whether or not the new latent variable will be used to generate the new data, and if the evaluation step evaluates that the new latent variable will be used, inputs the new latent variable into the generative model to generate the new data. According to this embodiment, it is possible to evaluate in advance whether or not new data will be generated using a new latent variable, and new data can be generated based on a new latent variable that has been evaluated as being used. Therefore, new data can be effectively selected according to its uniqueness.

[0086] (3) In the above embodiment, a receiving step of receiving, from a user, designation of evaluation conditions for the evaluation step may be provided prior to the evaluation step. According to this embodiment, the evaluation step can be executed according to the desired evaluation conditions designated by the user.

[0087] (4) In the above embodiment, when the density of the estimated new latent variable is a first density, the evaluation step may evaluate the use of the new latent variable with a higher probability than when the density of the estimated new latent variable is a second density greater than the first density. According to this embodiment, while primarily using new latent variables with a higher probability of generating new data with higher uniqueness, new latent variables with a higher probability of generating new data with lower uniqueness can also be used. Therefore, new data can be generated more efficiently than in an embodiment in which new latent variables with a second density are not used at all.

[0088] (5) In the above aspect, a step of presenting the generated new data to a user may be provided.

[0089] (6) In the above embodiment, the correspondence relationship may represent a correspondence relationship between the reference latent variable and the normalized density of the reference data. According to this embodiment, the result of normalizing the density of the reference data is reflected in the estimated density estimated using the correspondence relationship, making it possible to more easily execute the evaluation process.

[0090] (7) In the above aspect, a step of presenting the evaluation result of the new latent variable in the evaluation step to a user may be provided.

[0091] (8) In the above aspect, the method includes a reference generation step of generating a reference embedding vector in which the reference data is projected onto the embedding space by inputting the reference data into a learning model that projects features of input data onto the embedding space, a density calculation step of calculating a density of the reference data in the embedding space using the reference embedding vector, and a regression step of generating a regression model that has learned the correspondence using the reference latent variable and the density of the reference data calculated in the density calculation step, where the learning model is a learning model that has been machine-learned using a learning dataset that includes multiple data generated by the generative model, and the density estimation step may estimate the density of the new latent variable using the regression model. According to this aspect, the density estimation step can estimate the density of the new latent variable using the regression model generated by executing the reference generation step, the density calculation step, and the regression step.

[0092] (9) In the above aspect, the training data set may include the reference data. According to this aspect, more appropriate reference embedding data can be generated by the training model. Therefore, the density calculation step and the evaluation step can be performed more appropriately.

[0093] (10) In the above embodiment, the learning model may be trained by self-supervised learning. According to this embodiment, by using a learning model that has been properly trained to realize data projection into an embedding space in which feature vectors of more similar data are closer in distance, new latent variables can be more appropriately evaluated in the evaluation step.

[0094] (11) A second aspect of the present disclosure provides a data processing device including a processor that executes a density estimation process for estimating a density of new data in an embedding space using a correspondence between a reference latent variable, which is a latent variable that is a source of reference data previously generated by a generative model that generates data using latent variables as input, and a density of the reference data in an embedding space, and a new latent variable, which is a latent variable that is a source of new data, and an evaluation process for evaluating the new latent variable according to the density of the new data estimated by the density estimation process.

[0095] (12) According to a third aspect of the present disclosure, there is provided a program that causes a computer to implement a density estimation function that estimates the density of new data in the embedding space using a correspondence between a reference latent variable, which is a latent variable that is the basis of reference data previously generated by a generative model that generates data using latent variables as input, and a density in the embedding space of the reference data, and a new latent variable, which is the latent variable for generating the new data, and a function that evaluates the new latent variable according to the density of the new data estimated by the density estimation function.

[0096] In addition to the above-described embodiments, the present disclosure can be realized in the form of, for example, a data processing system, a non-transitory recording medium on which a program is recorded, a program product, etc. Note that the program product may be provided as a recording medium on which a program is recorded, or may be provided as a program product that can be distributed via a network, for example. [Explanation of symbols]

[0097] 50...data processing system, 100...data processing device, 101...processor, 102...memory unit, 103...input / output interface, 104...internal bus, 105...output device, 106...input device, 108...data generation unit, 115...embedded data generation unit, 120...calculation unit, 122...regression model generation unit, 123...estimation unit, 125...evaluation unit, 135...evaluation condition reception unit, 140...learning execution unit, 155...program, 160...learning model, 165...reference embedded data, 170...generative model, 175...regression model

Claims

1. 1. A data generation method for generating new data, comprising: a density estimation step of estimating the density of the new data in the embedding space using a correspondence between a reference latent variable, which is a latent variable that is a source of reference data previously generated by a generative model that generates data using latent variables as input, and the density of the reference data in the embedding space, and a new latent variable, which is a latent variable that is a source of the new data; an evaluation step of evaluating the new latent variables according to the density of the new data estimated in the density estimation step.

2. 2. The data generation method according to claim 1, In the evaluation step, it is evaluated whether or not the new latent variable is to be used to generate the new data; The data generation method further comprises a new data generation step of inputting the new latent variables into the generative model to generate the new data if the new latent variables are evaluated to be usable in the evaluation step.

3. 2. The data generation method according to claim 1, The data generation method further comprises, prior to the evaluation step, a receiving step of receiving from a user a designation of an evaluation condition for the evaluation step.

4. 3. The data generation method according to claim 2, In the evaluation step, when the density of the estimated new latent variable is a first density, it is evaluated that the new latent variable will be used with a higher probability than when the density of the estimated new latent variable is a second density that is greater than the first density.

5. 3. The data generation method according to claim 2, A data generation method comprising a step of presenting the generated new data to a user.

6. 2. The data generation method according to claim 1, The data generation method, wherein the correspondence relationship represents a correspondence relationship between the reference latent variable and a normalized density of the reference data.

7. 2. The data generation method according to claim 1, A data generation method comprising a step of presenting to a user an evaluation result of the new latent variable in the evaluation step.

8. 8. A data generation method according to any one of claims 1 to 7, comprising: a reference generation step of generating a reference embedding vector in which the reference data is projected onto the embedding space by inputting the reference data into a learning model that projects the features of input data onto the embedding space; a density calculation step of calculating a density of the reference data in the embedding space using the reference embedding vector; a regression process of generating a regression model that has learned the correspondence relationship by using the reference latent variable and the density of the reference data calculated in the density calculation process, The learning model is a learning model that has been machine-learned using a learning dataset that includes a plurality of data generated by the generative model, In the density estimation step, the density of the new latent variable is estimated using the regression model.

9. 9. The data generation method according to claim 8, A data generation method, wherein the training dataset includes the reference data.

10. 9. The data generation method according to claim 8, A data generation method, wherein the learning model has been trained by self-supervised learning.

11. 1. A data processing device for generating new data, comprising: It has a processor, The processor: a density estimation process for estimating the density of the new data in the embedding space using a correspondence between a reference latent variable, which is a latent variable that is a source of reference data previously generated by a generative model that generates data using latent variables as input, and the density of the reference data in the embedding space, and a new latent variable, which is a latent variable that is a source of the new data; an evaluation process for evaluating the new latent variables according to the density of the new data estimated in the density estimation process; Data processing device.

12. A program for generating new data, a density estimation function that estimates the density of the new data in the embedding space using a correspondence between a reference latent variable, which is a latent variable that is the source of reference data generated in advance by a generative model that generates data using latent variables as input, and a density in the embedding space of the reference data, and a new latent variable, which is a latent variable for generating the new data; and a function of evaluating the new latent variables according to the density of the new data estimated by the density estimation function.

Images