Generative model evaluation device for evaluating fidelity and diversity in generative model and generative model evaluation method using the same

Abstract

A generative model evaluation device for evaluating the quality and diversity of a generative model according to the present disclosure may include a feature embedding unit that converts input data into a numerical feature vector, a probability distribution estimation unit that estimates a data distribution based on the feature vector, a confidence band estimation unit that estimates a confidence band, and an evaluation unit that evaluates the generative model based on TopP and TopR. The evaluation unit may evaluate the generative model topologically and statistically in terms of quality (Fidelity) and diversity.

Description

Technology Field

[0001] The present disclosure relates to a generative model evaluation technique, and more specifically, to a generative model evaluation device using a metric program capable of topologically and statistically evaluating an AI-based generative model in terms of quality and diversity, and a generative model evaluation method using the same. Background Technology

[0002] Many generative models have been studied recently, and in parallel, various methods for evaluating their performance have been proposed. Among the metrics, the most widely used indicators for evaluating generative models currently include the Inception Score (IS), Frechet Inception Distance (FID), Improved Precision and Recall (P&R), and Density & Coverage (D&C).

[0003] Evaluation metrics commonly approximate the image distribution in the feature space and are characterized by presenting evaluation scores based on this data distribution. However, research has confirmed that IS has the disadvantage of failing to provide accurate evaluations for images with added noise, and that FID fails to accurately evaluate the performance of generative models by assuming the data distribution in the feature space follows a multivariate normal distribution.

[0004] It can be observed that P&R type metrics fail to accurately approximate the data distribution due to their high sensitivity to outliers or noisy data. Specifically, P&R metrics have a problem in that their evaluation scores become even less reliable when the actual data contains artifacts (e.g., mislabeling, adversarial examples).

[0005] Therefore, current research on generative models requires an evaluation matrix that is robust against noisy data and capable of accurately approximating the data distribution. Prior art literature

[0006] Korean Published Patent No. 10-2023-0159667 “Artificial Intelligence-based Generative Design Method, Device and Computer Program” The problem to be solved

[0007] One objective of the present disclosure is to provide a generative model evaluation device that can evaluate generative models topologically and statistically in terms of quality and diversity, and contributes to the advancement of generative models by evaluating generative models more robustly than conventional matrices.

[0008] Another objective of the present disclosure is to provide a generative model evaluation method that can evaluate generative models topologically and statistically in terms of quality and diversity, and contributes to the advancement of generative models by evaluating generative models more robustly than conventional matrices.

[0009] However, the problems that this disclosure aims to solve are not limited to those mentioned above, and other unmentioned problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem

[0010] A generative model evaluation device according to the present disclosure for evaluating the quality and diversity of a generative model to achieve the aforementioned technical objectives may include a feature embedding unit that converts input data into a numerical feature vector, a probability distribution estimation unit that estimates a data distribution based on the feature vector, a confidence band estimation unit that estimates a confidence band, and an evaluation unit that evaluates the generative model based on TopP and TopR. The evaluation unit may evaluate the generative model topologically and statistically in terms of quality (Fidelity) and diversity.

[0011] In one embodiment, the probability distribution estimator can estimate the data distribution using only the real signal of the data based on the persistent homology of topological data analysis.

[0012] In one embodiment, when the confidence band estimator approximates the distribution of an image dataset containing noise including at least one of outliers, IID perturbation, and non-IID perturbation, it can estimate the data distribution that is robust against the noisy signal based on a newly defined confidence interval.

[0013] In one embodiment, the evaluation unit evaluates the quality and diversity of an image based on an estimated data distribution and can provide an evaluation matrix that presents a consistent evaluation in a plurality of embedding spaces.

[0014] In one embodiment, the evaluation unit can evaluate the ranking of the generation model based on the performance of the generation model through a single evaluation score including at least one of IS and FID.

[0015] In addition, a generative model evaluation method according to the present disclosure for evaluating the quality and diversity of a generative model may include an operation of converting input data into numerical feature vectors, an operation of estimating a data distribution based on said feature vectors, an operation of estimating a confidence band, and an operation of evaluating the generative model based on TopP and TopR. The operation of evaluating the generative model may evaluate the generative model topologically and statistically in terms of quality (Fidelity) and diversity.

[0016] In one embodiment, the operation of estimating the data distribution based on the feature vector can estimate the data distribution using only the real signal of the data based on the persistent homology of topological data analysis.

[0017] In one embodiment, the operation of estimating the confidence band can estimate the data distribution that is robust against the noisy signal based on a newly defined confidence interval when approximating the distribution of an image dataset containing noise that includes at least one of outliers, IID perturbation, and non-IID perturbation.

[0018] In one embodiment, the operation of evaluating the generation model may evaluate the quality and diversity of the image based on the estimated data distribution and provide an evaluation matrix that presents a consistent evaluation in a plurality of embedding spaces.

[0019] In one embodiment, the operation of evaluating the generation model can evaluate the ranking of the generation model based on the performance of the generation model through a single evaluation score including at least one of IS and FID.

[0020] In addition to this, a computer program stored on a computer-readable recording medium for implementing the present disclosure may be further provided.

[0021] In addition to this, a computer-readable recording medium for recording a computer program for implementing the present disclosure may be further provided. Effects of the invention

[0022] According to the means for solving the problem described above in the present disclosure, the generative model evaluation device and the generative model evaluation method using the same of the present disclosure can evaluate generative models topologically and statistically in terms of quality and diversity, and can contribute to the development of generative models by evaluating generative models more robustly than conventional matrices.

[0023] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below. Brief explanation of the drawing

[0024] FIG. 1 is a block diagram showing the configuration of a generation model evaluation device of the present disclosure. FIG. 2 is a flowchart illustrating the operation of the generation model evaluation device of the present disclosure. FIG. 3 is a diagram showing the approximation through boost strapping of the confidence band of the generation model evaluation device of the present disclosure. FIG. 4 is a diagram showing an approximation of a robust data distribution of a generative model evaluation device of the present disclosure. FIG. 5 is a diagram showing the steps of the generation model evaluation device of the present disclosure, and the calculation of an evaluation score through TopP&R. FIG. 6 is a diagram showing the first algorithm of the confidence band estimation unit of the generative model evaluation device of the present disclosure. Figure 7 is a diagram showing the evaluation trend of the evaluation matrix for outliers of the generated data distribution. Figure 8 is a diagram showing the evaluation trend of the evaluation matrix when sequential mode drop and simultaneous mode drop occur. Figure 9 is a chart comparing the rankings of generative models. Specific details for implementing the invention

[0025] Throughout this disclosure, the same reference numerals denote the same components. This disclosure does not describe all elements of the embodiments, and general content in the art to which this disclosure pertains or content that overlaps between embodiments is omitted. The terms 'part, module, component, block' as used in the specification may be implemented in software or hardware, and depending on the embodiments, a plurality of 'parts, modules, components, blocks' may be implemented as a single component, or a single 'part, module, component, block' may include a plurality of components.

[0026] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are directly connected but also cases where they are indirectly connected, and indirect connections include connections made via a wireless communication network.

[0027] Furthermore, when it is stated that a part "includes" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0028] Throughout the specification, when it is stated that a component is located "on" another component, this includes not only cases where a component is in contact with another component, but also cases where another component exists between the two components.

[0029] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0030] Singular expressions include plural expressions unless there is an obvious exception in the context.

[0031] In each step, identification codes are used for convenience of explanation and do not describe the order of the steps; the steps may be performed differently from the specified order unless a specific order is clearly indicated in the context.

[0032] The operating principles and embodiments of the present disclosure will be described below with reference to the attached drawings.

[0033] In this specification, the term "device according to the present disclosure" includes all various devices capable of performing computational processing and providing results to a user. For example, the device according to the present disclosure may include all of a computer, a server device, and a portable terminal, or may be in the form of any one of these.

[0034] Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

[0035] The above server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

[0036] The above portable terminal may include, for example, all types of handheld-based wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminals, smartphones, etc., as well as wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted devices (HMDs).

[0037] Functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (Digital Signal Processors), graphics-dedicated processors such as GPUs and VPUs (Vision Processing Units), or artificial intelligence-dedicated processors such as NPUs. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if the one or more processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0038] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a basic artificial intelligence model is trained using a number of training data by a learning algorithm, thereby creating predefined rules of operation or artificial intelligence models configured to perform desired characteristics (or objectives). Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0039] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. Artificial neural networks may include deep neural networks (DNNs), such as Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), RecuREnt Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional RecuREnt Deep Neural Networks (BRDNNs), or Deep Q-Networks, but are not limited to the examples mentioned above.

[0040] According to an exemplary embodiment of the present disclosure, a processor can implement artificial intelligence. Artificial intelligence refers to a machine learning method based on an artificial neural network that enables a machine to learn by mimicking human biological neurons. Methodologies of artificial intelligence can be classified according to the learning method into supervised learning, where input and output data are provided together as training data and the solution (output data) to the problem (input data) is predetermined; unsupervised learning, where only input data is provided without output data and the solution (output data) to the problem (input data) is not predetermined; and reinforcement learning, where a reward is given from an external environment whenever an action is taken from the current state, and learning proceeds in a direction that maximizes such reward. In addition, artificial intelligence methodologies can be classified according to the architecture, which is the structure of the learning model. The architectures of widely used deep learning technologies can be classified into Convolutional Neural Networks (CNN), RecuREnt Neural Networks (RNN), Transformers, and Generative Adversarial Networks (GAN).

[0041] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0042] FIG. 1 is a block diagram showing the configuration of the generation model evaluation device (1) of the present disclosure, and FIG. 2 is a flowchart showing the operation of the generation model evaluation device (1) of the present disclosure.

[0043] Referring to FIGS. 1 and 2, the generative model evaluation device (1) of the present disclosure may include a feature embedding unit (10) that converts input data into a numerical feature vector, a probability distribution estimation unit (20) that estimates a data distribution based on the feature vector, a confidence band estimation unit (30) that estimates a confidence band, and an evaluation unit (40) that evaluates the generative model based on TopP and TopR.

[0044] The generative model evaluation device (1) of the present disclosure can estimate the data distribution using only the real signal of the data based on the persistent homology of topological data analysis (TDA).

[0045] In addition, when the generation model evaluation device (1) approximates the distribution of an image dataset containing various noises (e.g., outliers, IID & Non-IID perturbations), it can estimate a data distribution that is robust against noisy signals based on a newly defined confidence interval.

[0046] The generation model evaluation device (1) can accurately evaluate the clarity and diversity of an image based on the estimated data distribution and can provide an evaluation matrix that can present a consistent evaluation in various embedding spaces.

[0047] In addition, the generation model evaluation device (1) can provide a method to evaluate the ranking of the generation model according to the performance of the generation model through a single evaluation score such as IS or FID.

[0048] Specifically, as shown in FIG. 2, the generative model evaluation device (1) can convert input data into a numerical feature vector (operation 210), estimate a data distribution based on the feature vector (operation 220), estimate a confidence band (operation 230), and evaluate the generative model based on TopP and TopR (operation 240).

[0049] According to one example, in operation 210, the generation model evaluation device (1) can convert input data into a numerical feature vector. For example, the feature embedding unit (10) can convert input data (e.g., image, text) into a numerical feature vector.

[0050] The feature embedding unit (10) can convert input data into a feature vector and transmit the feature vector to the probability distribution estimation unit (20), thereby helping the probability distribution estimation unit (20) understand the complex pattern of the input data and generate new samples based on it. The feature embedding unit (10) can extract key features of the input data and subsequently help the evaluation unit (40) evaluate the generation model based on TopP and TopR.

[0051] According to one example, in operation 220, the generation model evaluation device (1) can estimate the data distribution based on the feature vector. For example, the probability distribution estimation unit (20) can approximate what data distribution the generated sample has.

[0052] For example, the probability distribution estimator (20) can estimate the set of values ​​that the generated data can take, that is, the possible data range. The probability distribution estimator (20) can estimate how wide a range of data the generating model can generate. The probability distribution estimator (20) can estimate how well the generated data reflects the actual data distribution.

[0053] According to one example, in operation 230, the generation model evaluation device (1) can estimate a confidence band. For example, the confidence band estimation unit (30) can estimate and evaluate the statistical diversity among the generated samples.

[0054] For example, the confidence band estimator (30) can estimate and analyze whether the generative model can generate various types of samples. Specifically, the confidence band estimator (30) can use a statistical approach to identify topologically significant features in the data and obtain them from the noise data.

[0055] Confidence bands can represent a specific statistical range of data. They indicate the probabilistic range in which data points possess certain significant characteristics (e.g., mean, median) and can be defined under a specific significance level.

[0056] The confidence level is a concept used in statistical hypothesis testing that can represent the upper limit of the probability of drawing a false conclusion. Generally, values ​​such as 0.05 (5%) or 0.01 (1%) may be used. For example, the lower the confidence level, the higher the reliability of the conclusion.

[0057] The bootstrap method may be a method used to evaluate the distribution of statistical estimates. The bootstrap method generates multiple datasets by repeatedly sampling from the original dataset, thereby obtaining the distribution of statistical estimates. For example, Algorithm 1 refers to a specific algorithm that implements this bootstrap method, which can be used here to define a confidence band for the bandwidth obtained through bootstrapping.

[0058] The confidence band estimator (30) can identify topologically important features within the data (e.g., important patterns or clusters of the data structure). For example, the confidence band estimator (30) is applied to both the actual dataset and the generated dataset, and can distinguish from noise data by identifying important features within the defined confidence band.

[0059] The confidence band estimator (30) can play an important role in extracting significant statistical features from a complex dataset and evaluating the quality of the data based on them, or improving the performance of a machine learning model. The confidence band estimator (30) can help to understand the intrinsic characteristics of the input data more deeply and to build more accurate analysis and prediction models.

[0060] According to one example, in operation 240, the generation model evaluation device (1) can evaluate the generation model based on TopP and TopR. For example, the evaluation unit (40) can precisely measure the performance of the generation model based on TopP (Top Precision) evaluation and TopR (Top Recall) evaluation. TopP evaluation and TopR evaluation can play an important role in evaluating the quality and diversity of the data generated by the generation model.

[0061] For example, the evaluation unit (40) can evaluate how accurate the top percentage of data samples closest to the actual data distribution is among the data samples generated through TopP evaluation. TopP evaluation can be used as an indicator to measure how high-quality results the generating model produces that are similar to the actual data.

[0062] For example, the evaluation unit (40) can evaluate how well the generating model covers the actual data distribution through TopR evaluation. TopR can indicate how well the generating model reflects the full range of the actual data. That is, TopR can be used as an important indicator to measure the diversity of the generated data.

[0063] The performance of the generative model can be comprehensively evaluated in terms of two main factors: 'accuracy (fidelity)' and 'diversity'. For example, TopP can indicate how accurately the model generates data, and TopR can indicate how well the model captures the diversity of the data. The evaluation unit (40) can quantitatively determine the overall quality and effectiveness of the generative model through the two indicators of TopP and TopR.

[0064] In this way, the generative model evaluation device (1) can utilize a robust estimation technique that is not affected by noisy information by utilizing the Topological Data Analysis (TDA) method. The persistent homology of the Topological Data Analysis (TDA) used by the generative model evaluation device (1) can be one of the methods to verify the topological characteristics of the data distribution. The generative model evaluation device (1) can adopt an approach that approximates the data distribution using only the real signals of the data based on persistent homology.

[0065] Meanwhile, the generative model evaluation device (1) of the present disclosure may include an artificial intelligence model. The artificial intelligence model may be a single artificial intelligence model or may be implemented as a plurality of artificial intelligence models. The artificial intelligence model may be composed of a neural network (or artificial neural network) and may include a statistical learning algorithm that mimics biological neurons in machine learning and cognitive science. A neural network may refer to a model that possesses problem-solving capabilities by changing the strength of synaptic connections through learning, in which artificial neurons (nodes) forming a network through synaptic connections. The neurons of the neural network may include a combination of weights or biases. The neural network may include one or more layers composed of one or more neurons or nodes. For example, the device may include an input layer, a hidden layer, and an output layer. The neural network constituting the device can infer a result (output) to be predicted from an arbitrary input by changing the weights of the neurons through learning.

[0066] At least one processor included in the generative model evaluation device (1) can generate a neural network, train or learn a neural network, perform operations based on received input data, generate an information signal based on the results of the operation, or retrain the neural network. The neural network models may include, but are not limited to, various types of models such as Convolutional Neural Networks (CNN), Region with Convolutional Neural Networks (R-CNN), Region Proposal Networks (RPN), RecuRent Neural Networks (RNN), Stacking-based Deep Neural Networks (S-DNN), State-Space Dynamic Neural Networks (S-SDNN), Deconvolution Networks, Deep Belief Networks (DBN), Restructured Boltzmann Machines (RBM), Fully Convolutional Networks, Long Short-Term Memory Networks (LSTM), and Classification Networks, such as GoogleNet, AlexNet, and VGG Network. The processor may include one or more processors for performing operations according to the neural network models. There is. For example, neural networks can include deep neural networks.

[0067] Neural networks include CNN (Convolutional Neural Network), RNN (RecuREnt Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), LSTM (Long Short Term Memory), GRU (Gated RecuREnt Unit), AE (Auto Encoder), VAE (Variational Auto Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Depp Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics) Network), GAN (Generative Adversarial Network), LSM (Liquid State Machine), ELM (Extreme Learning Machine), A person skilled in the art will understand that any neural network may be included, but is not limited to, ESN (Echo State Network), DRN (Deep Residual Network), DNC (Differentiable Neural Computer), NTM (Neural Turning Machine), CN (Capsule Network), KN (Kohonen Network), and AN (Attention Network).

[0068] According to an exemplary embodiment of the present disclosure, at least one processor included in a generative model evaluation device (1) comprises a Convolutional Neural Network (CNN) such as GoogleNet, AlexNet, VGG Network, Region with Convolutional Neural Network (R-CNN), Region Proposal Network (RPN), RecuREnt Neural Network (RNN), Stacking-based Deep Neural Network (S-DNN), State-Space Dynamic Neural Network (S-SDNN), Deconvolution Network, Deep Belief Network (DBN), Restructured Boltzmann Machine (RBM), Fully Convolutional Network, Long Short-Term Memory Network (LSTM), Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT, SP-BERT, MRC / QA for natural language processing, Text Analysis, Dialog System, GPT-3, GPT-4, Visual Analytics, Visual Understanding, Video Synthesis for vision processing, Anomaly Detection for ResNet data intelligence, Various artificial intelligence structures and algorithms, such as prediction, time-series forecasting, optimization, recommendation, and data creation, may be used, but are not limited thereto.

[0069] FIG. 3 is a diagram showing an approximation through boost strapping of a confidence band of the generative model evaluation device (1) of the present disclosure, FIG. 4 is a diagram showing an approximation for a robust data distribution of the generative model evaluation device (1) of the present disclosure, FIG. 5 is a diagram showing the calculation of an evaluation score through TopP&R, a step of the generative model evaluation device (1) of the present disclosure, and FIG. 6 is a diagram showing a first algorithm used by the confidence band estimation unit (30) of the generative model evaluation device (1) of the present disclosure.

[0070] Referring to FIGS. 3 to 6, the generative model evaluation device (1) can evaluate the performance of a generative model from a statistically conservative perspective. The generative model evaluation device (1) can evaluate the performance of a generative model based on a metric that uses data having topologically important structures or statistically reliable data.

[0071] The generative model evaluation device (1) can observe input data at various resolutions through topological data analysis (TDA) utilizing persistent homology. The generative model evaluation device (1) can quantify and extract topologically important features of the input data through the process of observing the input data at various resolutions.

[0072] Filtration is a set of subspaces that approximate a data point at various resolutions, and generally written as When It satisfies a relationship like that.

[0073] The probability distribution estimation unit (20) included in the generation model evaluation device (1) is a function that approximates the data distribution You can define filtration using . function It uses KDE based on a compact cosine kernel to directly approximate the distribution, and filtration It can be defined as. Here, the parameter It is used as a threshold to exclude noise data points.

[0074] As shown in FIG. 3, the probability distribution estimator (20) is a KDE function related to filtration. Data distribution utilizing It can approximate. Since the data constituting the data distribution is determined by the lifespan of important topological structures, the generative model evaluation device (1) has a threshold that filters out data that allows important topological features to be well maintained. can calculate.

[0075] Persistent homology is a method of representing topological features (homology) that can be visualized in a two-dimensional persistent diagram. Filtration Using this, we can record the process of topological features (e.g., 0-dimension: connected component, 1-dimension: loop, 2-dimension: cavity, ...) being born and dying in k-dimension. Filtration at Homology depending on the increase or decrease of It originates from and It disappears from. Coordinates obtained through filtration is space It is represented by the points above, and this is called a persistent diagram. Here, The lifespan of a homology can be calculated through this, and longer-lived homologs possess topologically important features in the dataset, and the shorter the lifespan (e.g., diagonal The closer it is to) it corresponds to a topologically noisy feature.

[0076] The confidence band estimation unit (30) included in the generation model evaluation device (1) is a statistical indicator, the significance level A threshold that can define important topological features based on It can approximate.

[0077] As shown in FIG. 4, the confidence band estimator (30) is defined in the probability distribution estimator (20). A data distribution that has a topologically important structure and is robust against noise signals using It can be estimated (or approximated).

[0078] Specifically, the confidence band estimator (30) has a significance level When given, is the bootstrap bandwidth obtained through the first algorithm (e.g., the first algorithm of Fig. 6). It can be defined as a confidence band for.

[0079] As defined earlier Is It can satisfy [this]. The confidence band defined in this way serves to classify data that guarantees topologically important features from noisy data. The generative model evaluation device (1) has a significance level Each actual dataset under and generated dataset The confidence band defined for and It can be written as.

[0080] KDE kernel bandwidth is a hyperparameter that requires approximation. The generative model evaluation device (1) for approximating the kernel bandwidth, Changes in the topological characteristics of the data due to change can be observed through a persistent diagram. The generative model evaluation device (1) is an indicator representing the topological importance of the i-th homology in the persistent diagram. Using the survival period of the i-th homology can be calculated. Confidence band Given, having the optimal value It can maximize any one of the following [Equation 1].

[0081] [Formula 1]

[0082]

[0083] The generation model evaluation device (1) can obtain a score that quantitatively evaluates the clarity and diversity of images generated by the generation model by calculating TopP and TopR.

[0084] For example, as shown in FIG. 5, the generative model evaluation device (1) can measure how close the generated data distribution is to the actual data distribution when measuring the fidelity of the generative model through precision. In addition, the generative model evaluation device (1) can measure whether the distribution of the generated data of the generative model has a distribution similar to the diversity or variance of the actual data distribution when measuring diversity through recall.

[0085] The generation model evaluation device (1) is a real space Data distribution defined above Precision and recall can be redefined using the distribution (support) it possesses.

[0086] The generation model evaluation device (1) all data and About is the center and radius It can be defined as a circle that does not contain a boundary having . Real space Data distribution defined above If we mathematically define the support it possesses, It can be written as follows.

[0087] data , kernel functions for data , and kernel bandwidth Given , the kernel density estimator (KDE) that approximates the distribution can be defined as shown in Equation 2 below.

[0088] [Equation 2]

[0089]

[0090] Here, the average of the KDE is It can satisfy... The generation model evaluation device (1) is generation data that may include noise data. The probability distribution that It is denoted as, and the actual data The probability distribution that has is It can be defined as follows.

[0091] The evaluation unit (40) included in the generation model evaluation device (1) can apply precision and recall by modifying the existing method. Precision is It can be defined as follows. Here, if precision is defined for data points, it is as shown in [Equation 3] below.

[0092] [Equation 3]

[0093]

[0094] Similar to how precision was defined above, recall can be defined as shown in [Equation 4] below.

[0095] [Equation 4]

[0096]

[0097] In fact, when defining precision and recall, and Since is not given, an approximation may be necessary. The above distribution (support) is the previously defined KDE Wow, Confidence Band It can be calculated using .

[0098] for example, In The distribution of It is approximated as such, and in this process, data presumed to be noise are excluded because the KDE value is low. Similarly, support of It can be defined as such. Finally, if the precision and recall above are defined using the approximate distributions of P and Q, the topological precision and topological recall can be obtained as shown in [Equation 5] and [Equation 6] below.

[0099] [Formula 5]

[0100]

[0101] [Equation 6]

[0102]

[0103] The hyperparameters of [Equation 5] and [Equation 6] above and It can be approximated by a balloon estimator.

[0104] TopP and TopR generated in the evaluation unit (40) have the advantage of having scores with clear upper and lower limits, and have the advantage of showing consistent performance under conditions such as noisy data, outliers, and long-tailed distributions.

[0105] In this way, the generative model evaluation device (1) of the present disclosure can evaluate generative models topologically and statistically in terms of quality (Fidelity) and diversity, and can contribute to the development of generative models by evaluating generative models more robustly than conventional matrices.

[0106] Figure 7 is a diagram showing the evaluation trend of the evaluation matrix for outliers of the generated data distribution.

[0107] Referring to Fig. 7, the evaluation trends of the evaluation metrics regarding whether the multiple metrics proposed on the synthetic dataset can robustly evaluate the generative model in data containing outliers are illustrated. The x-axis of Fig. 7 represents the mean of the generative distribution. Represents the value. For example, in the test of Fig. 7, 10,000 generated samples are generated. It can be used by randomly sampling from.

[0108] As shown in Fig. 7, It depends on the location of the generation distribution It can vary to values ​​between these. Outliers are both in the actual distribution and the generated distribution. One sample was considered at the location. Through the test results, it can be confirmed that the existing P&R, D&C, and GCA matrices fail to provide a normal evaluation when outliers are present. On the other hand, the generative model evaluation device (1) of the present disclosure can provide a normal evaluation even when outliers are present.

[0109] Figure 8 is a diagram showing the evaluation trend of the evaluation matrix when sequential mode drop and simultaneous mode drop occur.

[0110] The x-axis of Fig. 8 represents the sample density of a distribution (mode) centered at 0.

[0111] FIG. 8 shows whether the matrix proposed in the generative model evaluation device (1) of the present disclosure can provide a sensitive and accurate quantitative evaluation compared to existing matrices by simulating sequential mode drops and simultaneous mode drops.

[0112] Generative models can be trained to generate diverse images corresponding to various classes by mimicking real-world image distributions that possess multiple modes. Poorly trained generative models may exhibit a characteristic called mode dropping, where they fail to generate images of specific classes effectively; therefore, evaluation metrics must assign appropriate scores to generative models that display this tendency.

[0113] As shown in FIG. 8, when reducing the number of samples in a specific mode of the generation distribution, the same number of samples can be added to the mode with a center of 0 so that the fidelity is always fixed at 1. In particular, when looking at the evaluation trend regarding the diversity of the matrices, it can be seen that the TopR of the generation model evaluation device (1) of the present disclosure is the most sensitive and accurately follows the trend that the actual evaluation value should have. On the other hand, it can be seen that some other matrices do not return evaluation values ​​normally in the simultaneous mode drop simulation.

[0114] Figure 9 is a chart comparing the rankings of generative models.

[0115] Specifically, Figure 9 illustrates a comparison between the ranking of the generative model through FID and the F1 scores of TopP&R, D&C, and P&R, as well as the ranking of the generative model calculated through FID, KID, and MTD. InceptionV3, VGG16, and SwAV may be used as embedding networks. The numbers in parentheses in the figure may represent the ranking of the generative model evaluated by each metric.

[0116] As shown in FIG. 9, existing metrics such as D&C and P&R, which evaluate quality (Fidelity) and diversity respectively, are difficult to rank various generative models because they cannot reliably approximate the accurate data distribution and D&C has an evaluation value with no upper limit. On the other hand, the metric of the generative model evaluation device (1) of the present disclosure is robust against various perturbations and accurately approximates the data distribution, so it can be confirmed that it gives a ranking consistent with FID to generative models in various embedding network settings.

[0117] To determine whether the ranking of the generative model assigned by the F1 score defined by each metric is consistent across various embedding networks, the Hamming Distance (HD) can be calculated. For example, as shown in FIG. 9, the HD result values ​​for TopP&R, P&R, D&C, and MTD are calculated as 1.33, 2.66, 3.0, and 3.33, respectively. That is, it can be confirmed that TopP&R according to the generative model evaluation device (1) of the present disclosure gives the most consistent ranking across various embedding network settings.

[0118] In this way, the generative model evaluation device (1) of the present disclosure can evaluate generative models topologically and statistically in terms of quality (Fidelity) and diversity, and can contribute to the development of generative models by evaluating generative models more robustly than conventional matrices.

[0119] However, as this has been explained above, a redundant explanation thereof will be omitted.

[0120] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0121] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0122] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents. Therefore, other implementations, other embodiments, and equivalents to the claims below are also within the scope of the claims. Explanation of the symbols

[0123] 1: Generative Model Evaluation Device 10: Feature Embedding Section 20: Probability distribution estimator 30: Confidence band estimation section 40: Evaluation Department

Claims

Claim 1 A generative model evaluation device for evaluating the quality and diversity of a generative model comprises: a feature embedding unit that converts input data into a numerical feature vector; a probability distribution estimation unit that estimates a data distribution using only real signals excluding noise data points by utilizing a kernel density estimator (KDE) using a cosine kernel based on persistent homology of topological data analysis as a filtration function based on the feature vector; and a confidence band estimation unit that calculates a sup-norm error between a KDE estimator and a bootstrap KDE estimator by repeatedly performing bootstrap sampling based on the data distribution, and estimates a confidence band (c_α) for the sup-norm error under a significance level α. A generative model evaluation device comprising an evaluation unit that approximates the support (supp(P)) of the actual data distribution and the support (supp(Q)) of the generated data distribution using the confidence band (c_α) as a threshold value, respectively, and calculates TopP and TopR using hyperparameters approximated by a balloon estimator, thereby evaluating the generative model topologically and statistically in terms of fidelity and diversity, and providing an evaluation score with fixed upper and lower limits in the range [0, 1]. Claim 2 delete Claim 3 In claim 1, the confidence band estimator estimates the data distribution robust to a noisy signal based on a newly defined confidence interval when approximating the distribution of an image dataset containing noise including at least one of an outlier, an IID perturbation, and a non-IID perturbation, a generative model evaluation device. Claim 4 A generative model evaluation device according to claim 1, wherein the evaluation unit evaluates the quality and diversity of an image based on an estimated data distribution and provides an evaluation matrix that presents a consistent evaluation in multiple embedding spaces. Claim 5 In claim 1, the evaluation unit evaluates the ranking of the generation model based on the performance of the generation model through a single evaluation score including at least one of IS and FID. Claim 6 A generative model evaluation method for evaluating the quality and diversity of a generative model performed by a generative model evaluation device comprises: an operation of converting input data into a numerical feature vector; an operation of estimating a data distribution using only real signals excluding noise data points by utilizing a kernel density estimator (KDE) using a cosine kernel based on persistent homology of topological data analysis as a filtration function based on the feature vector; and an operation of calculating a sup-norm error between a KDE estimator and a bootstrap KDE estimator by repeatedly performing bootstrap sampling based on the data distribution, and estimating a confidence band (c_α) for the sup-norm error under a significance level α. A method for evaluating a generative model, comprising the operation of approximating the support (supp(P)) of the actual data distribution and the support (supp(Q)) of the generated data distribution, respectively, using the confidence band (c_α) as a threshold, and calculating TopP and TopR using hyperparameters approximated by a balloon estimator, thereby evaluating the generative model topologically and statistically in terms of fidelity and diversity, while providing an evaluation score with fixed upper and lower bounds in the range [0, 1]. Claim 7 delete Claim 8 In claim 6, the operation of estimating the confidence band is a generative model evaluation method that estimates the data distribution robust to a noisy signal based on a newly defined confidence interval when approximating the distribution of an image dataset containing noise including at least one of outliers, IID perturbation, and non-IID perturbation. Claim 9 In claim 6, the operation of evaluating the generative model evaluates the quality and diversity of an image based on an estimated data distribution and provides an evaluation matrix that presents a consistent evaluation in multiple embedding spaces. Claim 10 In claim 6, the operation of evaluating the generation model evaluates the ranking of the generation model based on the performance of the generation model through a single evaluation score including at least one of IS and FID.

Images