TRAINING OF ML MODELS FOR SYNTHETIC DATA GENERATION

Abstract

A method for training a set of one or more machine learning (ML) models to generate synthetic data is disclosed, comprising: providing a dataset called the original dataset, which includes entries, each entry of the entries comprising values ​​of the set of attributes; determining at least one objective function and an evaluation function for training the set of one or more ML models; using the original dataset to train each ML model of the set of ML models to optimize a respective objective function of the at least one objective function, the evaluation function being used to evaluate the ML model during the training of the ML model or after the training of the set of one or more ML models; providing at least one of the set of trained ML models.

Description

AREA OF INVENTION

[0001] The disclosure relates to the generation of synthetic data and, in particular, to a method for training a machine learning model to generate synthetic data. BACKGROUND

[0002] Synthetic data can refer to a reproduction of data that represent a particular field and are not obtained through direct measurement. SUMMARY

[0003] Exemplary embodiments provide a method (training method) for training a set of one or more machine learning (ML) models to generate synthetic data, wherein the method comprises: providing a data set, referred to as the original data set, comprising entries, each entry comprising values ​​of a set of attributes; determining at least one objective function and one evaluation function for training the set of one or more ML models;Using the original dataset to train each ML model of the set of one or more ML models to optimize a respective objective function of the at least one objective function, wherein the evaluation function is used to evaluate the ML model during training or after training the set of one or more ML models, the training being performed such that the ML model generates a synthetic dataset; providing at least one trained ML model of the set of trained ML models. The set of trained ML models can, for example, be configured to generate a synthetic dataset of the desired size, e.g., the desired number of entries.

[0004] Exemplary embodiments provide a computer system for training a set of one or more machine learning (ML) models to generate synthetic data, wherein the computer system comprises the set of one or more ML models and an original data set with entries, each entry of the entries comprising values ​​of the set of attributes, wherein the computer system is configured for: determining at least one objective function and one evaluation function for training the set of one or more ML models;Using the original dataset to train each ML model of a set of one or more ML models to optimize a respective objective function of the at least one objective function, wherein the evaluation function is used to evaluate the ML model during the training of the ML model or after the training of the set of one or more ML models, wherein the training is performed such that the ML model generates a synthetic dataset; providing at least one trained ML model of the set of trained ML models.

[0005] Exemplary embodiments include a computer program containing instructions to induce a computer system to perform the training procedure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The following examples describe in more detail with reference to the drawings, in which: Fig. Figure 1 is a flowchart for generating synthetic data according to an example from the present subject. Fig. Figure 2 is a flowchart of a training procedure for training a set of multiple machine learning models to generate synthetic data according to an example of the present subject matter. Fig. Figure 3 is a block diagram of an exemplary computer system for implementing at least part of the present method in accordance with an example of the present subject matter. DETAILED DESCRIPTION

[0007] In the following description, specific details such as particular architectures, interfaces, techniques, etc., are presented for illustrative purposes, not as limitations, to facilitate a comprehensive understanding of the examples. However, it will be clear to those skilled in the art that the disclosed subject matter can also be implemented in other examples that differ from these specific details. In some cases, detailed descriptions of known devices and / or methods are omitted to avoid cluttering the description with unnecessary details.

[0008] The present subject matter enables the systematic and accurate generation of synthetic data using machine learning models. It allows for the flexible generation of synthetic data; for example, the procedure can be executed as many times as needed to generate the synthetic data. Thus, the generated synthetic data can be deleted if a user runs out of storage space, and new synthetic data can be generated as required, exhibiting the same statistical properties as the deleted data. The present subject matter enables the generation of synthetic data in desired quantities and for desired purposes. For example, a large amount of synthetic data can be used to train a machine learning model or to perform reliable data analysis.

[0009] A dataset can be provided. The provided dataset can be referred to as the original dataset. The original dataset comprises multiple entries with a set of attributes. Each entry in the original dataset can contain values ​​from the attribute set (or fields). The attributes can be of type Numeric, String, or Boolean, for example. Each entry in the original dataset can represent a corresponding entity; for example, the entries can represent students, with the set of attributes including age, student address, and student ID, or the entries can represent bank accounts, with the set of attributes including account balance, account holder, bank name, and so on. The original dataset can, for example, consist of a table with a number of columns, each representing the set of attributes. The original dataset can have a field such as...This can represent a financial field, an industry field, or any other field that can be represented by the set of attributes. The terms "data entry" and "entry" are used interchangeably.

[0010] In one example, the entries of the original dataset can be derived from entries originating from one or more data sources. These data sources could be, for example, databases. The computer system could, for instance, send a query to one or more data sources and receive entries of the original dataset in response. The computer system can then create the table from the received datasets. Alternatively or additionally, the entries of the original dataset can be derived from entries created or derived by processing unstructured data such as web pages and PDF files.

[0011] The received entries and / or derived entries can be referred to as original entries. The entries of the original dataset can be derived from the original entries. In one example, the entries of the original dataset could be the original entries themselves. Alternatively or additionally, the entries of the original dataset can be derived by preprocessing the original entries. Each original entry can have zero or more attribute values ​​set to NULL to indicate that the value of one or more attributes is unknown, missing, or unreadable. Each attribute of the attribute set can have a NULL value in one or more of the original entries, or it can have no NULL values ​​in the original entries.

[0012] A set of one or more machine learning (ML) models can be provided. The set of one or more ML models can, for example, consist of multiple ML models. Alternatively, the set of one or more ML models can consist of a single ML model. Each ML model in the set of one or more ML models is configured to receive a dataset and generate a synthetic dataset (e.g., with a specific number of entries). Each ML model in the set of one or more ML models can be trained using the original dataset. Training can be performed, for example, through unsupervised learning. The training of the ML model is performed so that the ML model generates a synthetic dataset containing a desired number of entries. The training of the ML model can be performed, for example, using one or more functions.The function can be, for example, an objective function or an evaluation function. The objective function can represent an optimization goal, while the evaluation function can be a measure of how well the machine learning (ML) model is performing. The objective function can be used during the ML model's training process to guide its optimization. The training of any ML model can depend on its type. For example, the number of training iterations the ML model can perform to optimize its parameters, as well as the number of samples used in each iteration, can be specified. These parameters can be determined based on the type of ML model.

[0013] The set of one or more machine learning (ML) models can be derived from one or more ML algorithms. For example, each ML model in the set can be derived from a different ML algorithm. Alternatively, one or more subsets of the set of ML models can be derived from the same ML algorithm using a different set of hyperparameters for each model. For instance, the set could comprise two subsets, each derived from two different ML algorithms.

[0014] In one example, the training of each machine learning (ML) model from a set of one or more ML models can be performed using an objective function and an evaluation function. The evaluation function can be the same for the set of one or more ML models. Alternatively, the training of each ML model in the set can be performed using a specific objective function, either unique to or related to each model, and the evaluation function. The objective function could, for example, be a loss function. Each ML model in the set of one or more ML models can have learnable parameters (e.g., weights in the case of a neural network). The evaluation function can be defined by one or more metrics. The value of the evaluation function can be related to the values ​​of these one or more metrics.The metric can be descriptive of the generated synthetic dataset. For example, the metric can represent a comparison between the synthetic dataset and the original dataset. The metric can, for example, indicate a difference between a property of the synthetic dataset and that property of the original dataset. In another example, the metric can indicate the level of data protection and / or confidentiality ensured by the synthetic data.

[0015] The evaluation function can also be defined with a target criterion. The target criterion can require that each metric of the evaluation function has a specific target value or that each metric lies within a specific target value range. Alternatively, the target criterion can require that a combination of the metrics of the evaluation function has a target value or that the combination lies within a corresponding target value range. If the evaluation function consists of two metrics, the combination could, for example, be the ratio of the two metrics. In another example, the combination could be the sum of the metrics of the evaluation function. In yet another example, the combination could be the weighted sum of the metrics of the evaluation function, where each metric is assigned a corresponding weight. These weights could, for example, be user-defined.This can, for example, allow the owner of the original dataset to assign more weight to one metric compared to the others. In one example, the target criterion can be defined by the owner of the original dataset. This can be particularly advantageous if one or more users belonging to the owner require synthetic data that can be generated by the trained machine learning models provided under this subject matter. The term "user" can refer to an entity, such as a person, a computer, or an application running on a computer. The owner can, for example, receive a quantified statement about the evaluation function of the machine learning model or the synthetic data generated from it and relate the metric of the evaluation function to a desired / required relationship.

[0016] In an initial training example, entries from the original dataset can be fed into the machine learning (ML) model in each iteration to generate synthetic entries. The objective function can then be evaluated using both the input entries and the synthetic entries. If the objective function fails to meet a convergence criterion, the learning parameters can be updated or modified (e.g., based on the objective function's value), and the next training iteration can be performed. If the loss function meets the convergence criterion, the resulting trained ML model can be deployed. The convergence criterion might, for example, require the objective function to exceed a threshold. In one example implementation of updating the learning parameters, backpropagation can be performed to update the ML model's learning parameters.The update of the learnable parameters can be done, for example, using gradient descent. If the convergence criterion is not met and all entries of the original dataset have been processed, entries can be added to the original dataset and / or the convergence criterion can be changed (e.g., the convergence criterion can be relaxed to make it less strict by changing the threshold), and the training can be repeated.

[0017] The evaluation function can be used after the training of the set of one or more machine learning (ML) models is complete. The evaluation function can be used after training to assess the performance of the ML models. After training each ML model from the set of one or more ML models, the evaluation function can be used. Each trained ML model from the set of one or more ML models can be used to generate a synthetic dataset. The evaluation function can be used to assess each generated synthetic dataset using the original dataset. For example, the set of one or more ML models can be ranked according to the values ​​of the evaluation function. The first n ranked ML models can then be provided for use in the inference phase, where n is an integer greater than or equal to one.In one example, n=1, n=2, or n=3. In another example, only the subset of ML models whose evaluation function values ​​satisfy the target criterion can be ranked according to the evaluation function values. The first n highest-ranked ML models can then be made available for use in the inference phase. If none of the ML models satisfy the target criterion, the training of the set of ML models can be repeated by adding entries and / or changing the convergence criterion and / or changing the target criterion and / or adjusting hyperparameters of the underlying ML model (depending on the algorithm used). This repetition can be performed until one or more trained ML models are able to satisfy the target criterion.In one example, the owner of the original dataset can obtain the values ​​of the evaluation function of the set of trained machine learning (ML) models and relate these values ​​to a desired / required value or ratio that represents the owner's target criterion. If all trained ML models are rejected by the owner, the training of one or more ML models can be repeated by adding entries and / or changing the convergence criterion and / or changing the target criterion, for example, to the owner's defined target criterion. This repetition can be performed until one or more trained ML models can satisfy the target criterion.

[0018] In a second training example, the evaluation function can be used during the training of the ML model. In this case, the training of each ML model from the set of ML models can be performed as follows. In each iteration of the ML model's training, entries from the original dataset can be fed into the ML model to generate the synthetic entries. Both the objective function and the evaluation function can be evaluated against the input entries and the synthetic entries. If the objective function fails to meet a convergence criterion and / or the evaluation function fails to meet a target criterion, the learnable parameters can be updated or modified (e.g., based on the value of the objective function), and the next training iteration can be performed. If the loss function meets the convergence criterion and the evaluation function meets the target criterion, the resulting trained ML model can, for example,will be made available for inference.

[0019] The synthetic dataset, generated by each trained machine learning model (ML) of the set of one or more ML models, can contain entries with a set of attributes. Each entry in the synthetic dataset can contain values ​​from this set of attributes.

[0020] The training according to the first and second training examples can be carried out on the computer system.

[0021] The present subject can provide various techniques for preprocessing the original entries in order to create the original dataset. This can improve the training process.

[0022] In a first preprocessing example, the NULL values ​​present in the original entries can be replaced with non-NULL values. This ensures that the dataset used to train the machine learning models does not contain any missing values ​​that could impair the models' accuracy. For example, the NULL values ​​can be replaced with predefined reference values ​​for the respective attributes. Thus, each attribute in the set of attributes can be assigned a reference value. The NULL value of a specific attribute in the set of attributes can then be replaced by the reference value of that attribute.In another example, any NULL value of a given attribute in the set of attributes can be replaced by a random sample of non-NULL values ​​of the given attribute in the original entries without altering the underlying distribution of the sample values. For instance, if a particular attribute in the set of attributes has a NULL value, a first distribution of the non-NULL values ​​of that particular attribute present in the original entries can be determined, and the NULL value can be replaced by a sample value obtained by sampling the non-NULL values ​​of that particular attribute. The substitution is performed such that a second distribution of the non-NULL values ​​(e.g., including the sample value) of the given attribute after the substitution may be similar to the first distribution, e.g.,The first and second distributions can be similar if the difference between them is less than a threshold. This sample can specifically target the attribute values ​​that are not NULL, resulting in the capture of a sample value. The value used as a substitute for the NULL value is the sampled value. This sample can be used to maintain data integrity and ensure that the trained machine learning models can be used effectively in various applications and scenarios. It also ensures that the substitute value is representative of the attribute's existing values. For categorical attributes, the NULL value can be treated as an additional category; for example, the attribute "fruit" has the three categories {"apple", "banana", "none"}.

[0023] For example, if the dataset contains customer information, including their age, income, and purchase history, certain fields may be missing for some customers, resulting in NULL values. To address this, preprocessing can replace these NULL values ​​with predefined values, such as the mean or median of a numeric field in the dataset. This ensures that the dataset is complete and ready for training machine learning models. Alternatively, instead of using predefined values, the NULL values ​​can be replaced using imputation methods, such as regression imputation or K-nearest neighbor imputation. These methods estimate the missing values ​​based on the values ​​of other features in the dataset or similar instances within the dataset. This can provide an accurate replacement of the NULL values.This ensures that the replacement value is representative of the existing values ​​of the attribute.

[0024] In one example, the first preprocessing example can optionally include the creation of an additional column or attribute to track the replaced NULL values. In this example, the attributes in the set of attributes containing the NULL values ​​can be continuous or numeric. For instance, an additional column can be added to the original dataset, resulting in an extended dataset. The additional column is associated with an additional attribute that has two distinct values: a first value (e.g., 1) and a second value (e.g., 0). If an entry in the extended dataset has at least one NULL value, the first value can be assigned to the additional attribute in that entry; otherwise, the second value can be assigned to the additional attribute in that entry.After each trained machine learning model has generated the synthetic data, entries in the synthetic data can be identified that correspond to the entries with the first value in the extended dataset. The synthetic data values ​​of any one or more attributes that have NULL values ​​in the extended dataset can then be replaced with NULL values. For example, if a given entry in the extended dataset has two attributes, at1 and at2, that have NULL values ​​before the replacement, the additional column for the given entry will have the first value. And for the entry in the synthetic dataset that corresponds to the given entry, the values ​​of attributes at1 and at2 can be replaced with NULL values, respectively.In this way, for example, NULL values ​​in numeric attributes can be masked as a new binary attribute (1 = NULL value, 0 = other value), allowing them to be synthesized and enabling the model to learn correlations with other attributes. By replacing them with uniformly distributed random values ​​in the original attribute, the underlying (normalized) distribution cannot be distorted.

[0025] According to an example, the procedure further includes: creating an additional column in the original dataset, resulting in an extended dataset, where the additional column is associated with an additional attribute with two distinct values: first value and second value; if an entry in the extended dataset has at least one NULL value, assigning the first value to the additional attribute in that entry, otherwise assigning the second value to the additional attribute in that entry; after generating the synthetic data by each of the provided trained ML models, identifying entries in the synthetic data that correspond to the entries with the first value in the extended dataset, and replacing values ​​of one or more attributes that have NULL values ​​in the extended dataset with NULL values ​​in the synthetic data, where the attributes that have the NULL valuesThese are numerical or continuous attributes. An entry in the synthetic data can match an entry in the augmented dataset if they have a similarity score that exceeds a threshold, where the similarity score can be obtained by applying a similarity metric to the compared entries. In this example, the synthetic data can accurately represent the original data from which it was generated. It can capture the patterns, correlations, and statistical properties of the original data to ensure it is a suitable substitute for training, testing, or analysis purposes.

[0026] In a second preprocessing example, outliers present in the original entries can be processed using a predefined outlier procedure to detect and remove them. In one example, the outlier processing procedure can use k-anonymization (where k can be determined during optimization) to identify and remove outliers. Removing outliers can be advantageous because they can interfere with re-identification later in the synthesis.

[0027] In a third preprocessing example, each of the original entries can contain the set of attributes or a larger set of attributes. If the original entries have a larger set of attributes, the set of attributes can be selected from this larger set using a selection criterion. For example, the selection criterion might require user input specifying the set of attributes to be selected. In another example, the selection criterion might require that the set of attributes be of one or more predefined types, where the predefined type could be, for example, a numeric type. The set of attributes could, for example, include attributes of a numeric type (numeric attributes).

[0028] In a fourth example of preprocessing, preprocessing can include steps such as data cleaning, data normalization, and feature engineering of the original entries. Data cleaning can involve removing irrelevant or duplicate data points and handling missing values. Data normalization can be performed to scale the data to a standard range, which can improve the performance of machine learning models. Feature engineering can involve creating new features or transforming existing features of the original entries to improve the predictive power of the models. Once the original entries have been preprocessed, they can be used to train the machine learning models.

[0029] The present subject can utilize various types of machine learning (ML) models. For example, each ML model in the set of one or more ML models could be a Tabular Variational Autoencoder (TVAE), Conditional Tabular Generative Adversarial Network (CTGAN), or Private Data Release via Bayesian Networks (PrivBayes). These ML models can offer different advantages depending on the type and size of the input data.

[0030] The training method can provide various beneficial evaluation functions. For example, the evaluation function might include a data privacy metric and a utility metric. The security level of a generated synthetic dataset, for instance, could be a value of a data privacy metric evaluated against the synthetic dataset. These metrics can play a role in determining the effectiveness and applicability of the trained machine learning (ML) models. The data privacy metric can be used to assess the degree of data protection provided by the trained ML models. It can measure the extent to which the models protect sensitive or personal information during their use. By including this metric, the disclosed method can ensure that the provided ML models prioritize data privacy and comply with applicable data protection regulations.The utility metric and the privacy metric are typically negatively correlated, so a trade-off or threshold for the relationship can be established for the target criterion. The privacy metric can measure the level of privacy protection provided by the generated synthetic data, while the utility metric can assess the usefulness or accuracy of the synthetic data in representing the original data. Various techniques can be used to quantify the privacy metric and determine its level of privacy. One approach is to measure the degree of information disclosure or identifiability within the synthetic data. This can be done by calculating the re-identification risk or the probability of re-identifying an individual from the synthetic data.Another approach to quantifying data protection metrics can be to assess the differential data protection of the synthetic data, which quantifies the data protection guarantee provided by the data generation process. Utility metrics, on the other hand, can focus on the quality and usefulness of the synthetic data. They can be measured by comparing statistical properties or characteristics of the synthetic data with those of the original data. For example, utility metrics can include measures such as mean squared error or correlation coefficient. In another example, utility metrics can encompass the accuracy of a machine learning model (e.g., a classification model) trained on the synthetic data.

[0031] To illustrate the application of these metrics, let's consider an example where the trained machine learning models are used in a healthcare setting. The data privacy metric can assess how well the models protect sensitive patient data, such as medical records or genetic information, during the prediction or analysis process. The utility metric, on the other hand, can evaluate the accuracy and reliability of the models in diagnosing diseases or predicting patient outcomes.

[0032] In one example, the evaluation function could include a metric such as accuracy, which measures the percentage of correctly generated data, and precision, which measures the proportion of correctly generated data. The target criterion could be set at 90% accuracy and 80% precision. The machine learning model can then be evaluated using both synthetic and original data. If the calculated metrics meet the target criterion, the machine learning model is considered to have achieved the desired performance level.

[0033] Furthermore, the evaluation function can be adapted to include different metrics and target criteria depending on the application or scenario. For example, when generating synthetic data in a healthcare system, the evaluation function can include metrics such as the precision metric.

[0034] According to an example, the evaluation function is defined by one or more metrics and a target criterion, where the evaluation of the ML model using the evaluation function includes: evaluating the one or more metrics using synthetic data generated by the ML model and the original dataset, and determining whether the evaluated one or more metrics meet the target criterion.

[0035] To evaluate the performance of machine learning (ML) models, for example, an evaluation function is defined. This function is determined by one or more metrics and a target criterion. The metrics serve as a measure of the ML model's performance, while the target criterion defines the desired performance level the ML model should achieve. Synthetic data is used in the evaluation to assess the ML model's performance. This synthetic data is generated by the ML model itself based on the original dataset. During the evaluation process, one or more metrics are calculated using the synthetic data and the original dataset. These metrics can include accuracy, precision, recognition, or other relevant performance indicators. The calculated metrics are then compared to the target criterion to determine whether the ML model achieves the desired performance level.

[0036] According to one example, the target criterion requires that a combination of one or more metrics of the evaluation function has a target value or that the combination lies within a target range of values, where the combination is a weighted sum of the metrics of the evaluation function using weights, with each metric associated with a corresponding weight of the weights.

[0037] The target criterion might, for example, specify that the combination of metrics such as accuracy, precision, and recall should have a target value of 0.9. This combination can be a weighted sum. The weights assigned to the metrics in the evaluation function determine their relative importance in achieving the target criterion. The set of weights can be adjusted based on the specific requirements of the application or scenario. For example, in a predictive analytics application, the weights can be adjusted to prioritize certain metrics depending on the desired outcome, such as minimizing accuracy or maximizing precision. Furthermore, in alternative examples, different mathematical functions can be used to calculate the combination of metrics.Instead of a weighted sum, other mathematical operations such as weighted means, geometric means or even nonlinear functions can be used to determine the combination.

[0038] For example, at least one objective function is a singular objective function. This singular objective function is used to train the set of one or more machine learning (ML) models. This singular objective function can serve as a guiding principle during the training process, enabling the ML models to learn and adapt to different applications and scenarios.

[0039] For example, the set of machine learning models can comprise multiple machine learning models. The at least one objective function is a set of objective functions. Each objective function in the set of objective functions is associated with a corresponding machine learning model within the set of machine learning models.

[0040] In this example, each objective function can be specifically tailored to a particular machine learning (ML) model within the group. The objective functions serve as a guide for the ML models, providing them with specific goals and criteria for optimizing their performance. By linking each ML model to its own objective function, the method enables a more targeted and specialized approach to training and using the ML models. Beyond the described example, other approaches can be considered. For instance, instead of linking each ML model to a single objective function, a combination of objective functions can be used. This can allow for a more comprehensive evaluation of the ML models' performance, as multiple criteria are considered simultaneously. Furthermore, the objective functions themselves can be adapted and adjusted based on the specific requirements of the application or scenario.Various weighting schemes or constraints can be integrated into the objective functions to further optimize the performance of the ML models.

[0041] According to an example, the set of one or more ML models comprises several ML models, wherein the at least one objective function consists of several objective functions, with each objective function of the several objective functions being associated with a corresponding ML model or several ML models of the set of ML models.

[0042] According to one example, providing the original dataset involves at least one of the following steps: receiving original entries from one or more data sources containing structured data; or deriving original entries from one or more data sources containing unstructured data; or extracting the entries of the original dataset from the original entries. This can enable a comprehensive dataset containing structured and / or unstructured data. The unstructured data, such as natural language text, can be processed to obtain the original entries in a structured format. Extracting the original entries from the text can be achieved, for example, by identifying and classifying named entities within the text into predefined categories such as names, organizations, locations, dates, and other specific attributes.This process can be carried out using a natural language processing (NLP) technique.

[0043] In a healthcare application, structured data might include patient demographics, medical history, and lab results, while unstructured data could include physician notes, radiology reports, and pathology reports. By combining both data types, trained machine learning models can leverage the rich information contained within the unstructured data. This example can involve consolidating data from various sources, such as databases, APIs, or file systems, into a unified dataset. The example can ensure that all relevant entries are included in the original dataset, regardless of their source or format. In a financial application, for example, the original dataset might contain transaction data from various banking systems, credit card statements, and customer feedback from social media platforms.This example can ensure that all these entries are collected and organized into a coherent dataset for training the ML models.

[0044] According to an example, the procedure further includes ranking the set of trained ML models using the respective values ​​of the evaluation function, selecting the highest-ranked n ML models from the set of ML models, where the provided at least one trained ML model is the selected ML model, where n is an integer greater than or equal to one. Alternatively, the provided at least one trained ML model can be all of the trained one or more ML models.

[0045] By ranking the machine learning (ML) models, the training method can ensure that the most effective and reliable models are selected for use. For example, if the set of trained ML models consists of five models: Model A, Model B, Model C, Model D, and Model E, each trained on a different dataset and exhibiting different performance levels, the evaluation function calculates a score for each model based on its performance metric(s). After ranking the models using the evaluation function, the method selects the ML models with the highest "n" rank from the set. The value of "n" can be determined based on the specific requirements of the application or scenario. For example, if "n" is set to 3, the method selects the three highest-ranked models for use.If the evaluation function ranks the models as follows, continuing the previous example: Model C (highest score), Model A, Model E, Model B and Model D (lowest score) and “n” is set to 2, the method would select Model C and Model A as the provided trained ML models.

[0046] In another example, the selection of trained machine learning (ML) models can be tailored to the user's specific needs. For instance, the user might prefer models with higher accuracy or those that perform well with particular data types. The flexibility of this method allows for the selection of ML models that best meet the requirements of the application or scenario.

[0047] As an example, a trained machine learning (ML) model can be used to generate synthetic data. In this example, the trained ML model can be used to generate synthetic data that can be used for various applications and scenarios. The synthetic data is generated by inputting a desired number of entries into the trained ML model and obtaining corresponding output data with the same number of entries. To generate the synthetic data, the trained ML model can first be loaded into a computer system or platform capable of running the model. The desired number of entries is then provided to the model. The model processes the input and generates output data based on the learned patterns and relationships.The synthetic data generated by the trained machine learning (ML) model can be used for a variety of purposes. For example, it can be used to extend existing datasets for training and testing other ML models. Generating synthetic data allows the dataset to be expanded with a larger number of examples, which can improve the performance and generalizability of the ML models trained on that data. Furthermore, the synthetic data can be used for data augmentation. Additionally, the synthetic data can be used for simulation and testing. By generating synthetic data that mimics real-world scenarios, the trained ML model can be evaluated and validated in a controlled environment before being deployed in production.In this way, potential problems or limitations of the model can be identified and necessary improvements or adjustments can be made.

[0048] According to an example, the evaluation function is defined by one or more metrics and a target criterion, where the target criterion specifies a target value or target range for the one or more metrics.The procedure comprises: creating a synthetic data generation module, wherein the data generation module includes at least one ML model from the set of trained ML models in conjunction with the respective target values; receiving a request to generate synthetic data at the synthetic data generation module; selection of a trained ML model by the synthetic data generation module based on the target value associated with the trained ML model and the received request; use of the selected ML model by the synthetic data generation module to generate synthetic data; provision of the synthetic data by the synthetic data generation module to a requester of the synthetic data.

[0049] In one example, the entries of the original data set are obtained through a preprocessing process of the original entries.

[0050] The present subject matter may include the following examples.

[0051] Example 1. A method for training a set of one or more machine learning (ML) models to generate synthetic data comprising a number of entries, wherein the method comprises: providing a dataset called the original dataset, comprising entries, each entry comprising values ​​of the set of attributes; using the original dataset to train each ML model of the set of one or more ML models to optimize a respective objective function of at least one objective function, using an evaluation function to evaluate the ML model during training or after training the set of one or more ML models; providing at least one trained ML model of the set of trained ML models.

[0052] Example 2. The procedure from Example 1, wherein providing the original data set includes creating the original data set.

[0053] A portion or all of the trained set of one or more machine learning models can be used to generate synthetic data according to an inference method. An example implementation of the inference method can be described in the following sections, which follow the description of the first figure ( Fig. 1) precede. However, it is not limited to this inference method, as other inference methods can also use the trained set of ML models. Example of the implementation of the inference method

[0054] Advantageously, the inference method can solve the problem of efficiently selecting the most suitable machine learning model to generate synthetic data that precisely fulfills a user's specific request. Advantageously, the inference method can leverage the robust capabilities of machine learning to generate synthetic data in response to user requests.

[0055] In general, the inference method can enable the generation of synthetic data using trained machine learning models. For this purpose, the inference method can provide a set of one or more trained machine learning models. Each trained ML model in the set is associated with a corresponding target value of an evaluation function. This evaluation function is used to assess the set of ML models in order to perform the training of the set of ML models. The inference procedure can further involve receiving a request from a user to generate synthetic data. The term "user" can refer to an entity, such as a person, a computer, or an application running on a computer.The query and the one or more target values ​​are used to select a trained machine learning (ML) model from the set of one or more trained ML models that satisfies the query. The selected ML model can then be used to generate synthetic data. In one example, the selected ML model can be executed to generate the synthetic data with a predefined default number of entries. In another example, the selected ML model can receive an input, causing it to generate the synthetic data according to that input, which can be the number of desired entries in the synthetic data. The synthetic data could, for example, consist of a table with a number of columns, each representing a set of attributes.

[0056] The inference process can involve the synthetic data generator providing the generated synthetic data to the user. The synthetic data generator can be a computer system. For example, after generating the synthetic data, the generator can send it to the user or store it in a user-accessible repository. This allows for the immediate provision of synthetic data to the user. The timely return of the requested synthetic data ensures that the user can access the information promptly. This can be particularly important if the data is time-critical or required for critical decision-making processes.Alternatively, the synthetic data can be stored in the synthetic data generator or in a shared database accessible to the user and / or other users. Storing the synthetic data in the shared database can enable centralized data management, allowing all users to access and utilize the data from a single source, thus ensuring data consistency and reducing the risk of data duplication or inconsistency. Furthermore, the shared database can offer superior data security compared to distributed data storage. A single storage location can provide robust security features such as user authentication, access control, and encryption to protect the data from unauthorized access or manipulation. For example, the synthetic data in the shared database can be stored according to a specific security feature.This security feature can, for example, require user authentication and / or access control for accessing the synthetic data. Storage according to this security feature can be based, for example, on the target value of the machine learning model used to generate the synthetic data. Storage according to this security feature may be required, for example, if the generated synthetic data could contain sensitive information according to the security level associated with the selected machine learning model. If the generated synthetic data does not contain sensitive information, it can be stored without this security feature. The synthetic data can be provided, for example, in a structured format such as a table.

[0057] The target value can represent a specific desired outcome that the evaluation function of a trained machine learning model should achieve or approximate. The target value can be a value within the evaluation function itself. For example, if the evaluation function comprises only one metric, the target value can be a value of that single metric. If the evaluation function comprises two or more metrics, the target value can be a vector of the respective values ​​of those metrics. Alternatively, if the evaluation function comprises two or more metrics, the target value can also be a combination of the metric values. This combination could be, for example, a ratio, a concatenation, or a weighted sum.

[0058] The set of one or more machine learning models can be derived from one or more machine learning algorithms. For example, each machine learning model in the set can be derived from a different machine learning algorithm. Alternatively, one or more subsets of the set of machine learning models can be derived from the same machine learning algorithm; that is, the set can comprise two subsets, each derived from two different machine learning algorithms. For instance, the same machine learning algorithm can produce different machine learning models for each subset, each associated with its own target value for the evaluation function.

[0059] In a sample implementation, the set of one or more machine learning (ML) models can be provided as follows. For example, the set of one or more ML models can comprise a number N of ML models, where N is an integer greater than or equal to one, N>=1. A data structure (e.g., a table) can be provided, where the data structure contains a mapping between the N ML models and their corresponding target values ​​(tV). For example, each ML model from the set of ML models can be associated with a target value as follows: ML1 - tV1, ML2 - tV2,...,MLN - tVN. Based on the request for synthetic data, one of the target values, e.g., tV1...tVN, that satisfies the request can be selected, and the ML model associated with that target value can be selected to generate the synthetic data.In one example, if more than one target value satisfies the query, the corresponding machine learning (ML) models can be ranked according to the target values, and the ML model with the highest rank can be selected. Alternatively, an ML model can be randomly selected from the multiple ML models. For example, the query might contain a desired value of the evaluation function that is equidistant from two target values, tV1 and tV2. In this case, ML1 and ML2 can be ranked based on tV1 and tV2, and the ML model with the highest ranking can be selected.

[0060] The target value and the corresponding machine learning (ML) model that fulfill the query can be selected using various techniques. For example, the query might contain a desired value for the evaluation function. This desired value can be compared to the target values ​​to select the target value and its associated ML model. The comparison can be performed, for example, by calculating the similarity between the desired value and each target value. The comparison can result in a similarity score. The similarity could be, for example, a distance similarity. If there are multiple target values, the target value that matches the desired value can be the one that is most similar to the desired value. The most similar target value can be the one that has the highest similarity score compared to the desired value. If there is only one target value, the target value matches the desired value if the similarity score exceeds a certain threshold.If there is no match with the target value, the request can be rejected or the similarity criterion, e.g. the threshold, can be relaxed until a match with the target value is found.

[0061] Alternatively, the inference method can automatically assign a target value to the user, where the assigned target value can be the selected target value. In this case, the target value that satisfies the request can be the target value assigned to the user. This assignment can be random or based on predefined criteria. The criteria can assign target values ​​based on the user's location and / or the time the request is received and / or other attributes that describe the user and / or the request. The user attribute can be, for example, the user's location, network connection type, or role. For instance, if the target value is a security level, an administrator who sent the request can be assigned a low security requirement (e.g.,(which corresponds to the target value tV1), while a regular user who sent the request can be assigned a stricter security requirement (e.g., which corresponds to the target value tVN); thus, the ML model ML1, which corresponds to the target value tV1, can be selected for the administrator, and the ML model MLN, which corresponds to the target value tVN, for the regular user. In another example, the highest security level can be assigned to the user if the target values ​​are security levels and the user is located in the same location as the synthetic data generator; otherwise, the user can be assigned a different, lower security level.

[0062] In one example, the query specifies both a desired value for the evaluation function and a desired number of entries. In this example, the selected machine learning (ML) model is the ML model whose target value matches the desired value. Furthermore, the inference process can include the generation of synthetic data that may contain the desired number of entries. The target value that matches the desired value can be the one that is most similar to the desired value. Alternatively, the target value matches the desired value if the similarity value exceeds a certain threshold.

[0063] This query allows for a more precise selection of the trained machine learning (ML) model to be used for generating the synthetic data. To explain how the selection process works, the inference method uses the target values ​​associated with each trained ML model in the set. These target values ​​represent the performance of the ML models as evaluated by the evaluation function during the training phase. By comparing the desired value specified in the query with the target values, the inference method identifies the ML model that best fulfills the user's query. Once the ML model with the appropriate target value has been identified and selected, the inference method proceeds with generating the synthetic data. The synthetic data is generated in a quantity corresponding to the desired number of entries specified in the query.This ensures that the user receives precisely the required number of synthetic entries. For example, a user requests the generation of synthetic data with a desired evaluation function value of 0.8 and a desired number of 100 entries. The inference method can search the set for the trained machine learning (ML) model with a target value closest to 0.8. Once the ML model with the closest target value is identified, the inference method can generate 100 synthetic entries using that selected ML model.

[0064] According to one example, selecting the trained ML model involves: ranking the set of trained ML models using the respective target values ​​of the evaluation function; determining an appropriate rank for the user based on the user-defined access rights; and selecting the ML model associated with the determined rank.

[0065] The evaluation function can be designed, for example, to rank models based on the accuracy of the trained machine learning models in generating synthetic data, the computational resources required for each model, or the time needed to generate the synthetic data. Considering multiple criteria allows the selection process to be optimized to meet the specific needs and preferences of the user. To determine an appropriate rank for the user, the inference method can take into account the user's predefined access rights. These predefined access rights can be specified as a real number, for example. Access rights can be defined based on various factors, such as the user's role, expertise, or specific queries.For example, a user with higher access rights might have access to more accurate or specialized machine learning (ML) models, while a user with lower access rights has access to a broader range of models, but potentially with lower accuracy. Determining the user's rank can be based on a predefined mapping between access rights and ranks. Once the user's rank has been determined, the inference method selects the ML model associated with that rank. This selection ensures that the user is provided with a trained ML model that matches their access level and meets their specific requirements. By selecting the appropriate ML model, the inference method ensures that the generated synthetic data is tailored to the user's needs and preferences.It should be noted that selecting the trained ML model based on its rank is only one possible approach. Other methods can also be used to select the ML model, such as a probabilistic approach where the probability of selecting a particular model is based on its rank, or using a machine learning algorithm that learns the user's preferences and selects the most suitable model accordingly.

[0066] For example, the target value assigned to each trained machine learning (ML) model indicates a security level for the synthetic data generated by that model. The inference process further involves providing user categories, each associated with a security level, and determining which category a user belongs to in order to select the appropriate ML model. The data security level can, for instance, represent the degree of sensitivity of the information contained within the data. Sensitivity can be determined based on the nature of the attributes in the data, as certain types of information are inherently more sensitive than others. For example, personally identifiable information (PII), financial data, health data, trade secrets, or classified information may be considered more sensitive than others.

[0067] This target value serves as an indicator of the security level of the synthetic data generated by the machine learning (ML) model. To increase the flexibility and adaptability of the inference method, user categories are introduced. Each user category is associated with a specific security level. To determine the appropriate ML model for generating synthetic data, the inference method includes identifying the user category. This categorization can be used to select the ML model that meets the user's security requirements. Therefore, the inference method includes the step of selecting the ML model based on the user category and its associated security level.For example, if user categories are defined as "low security," "medium security," and "high security," and each category corresponds to a different level of sensitivity or confidentiality required for the synthetic data, when a user requests synthetic data, the inference method can determine the user's category based on their security preferences or access permissions. If the user belongs to the "low security" category, the inference method selects a machine learning (ML) model associated with a target value indicating a lower security level. This ML model is then trained to generate synthetic data that meets the requirements of users with lower security needs.If, on the other hand, the user falls into the "high security" category, the inference method selects a machine learning model that is linked to a target value indicating a higher security level. This machine learning model is trained to generate synthetic data that can meet stricter security standards.

[0068] In addition to the user categories mentioned above, alternative categorization schemes can be implemented based on specific requirements or industry standards. For example, instead of the categories "low security," "medium security," and "high security," the inference method can use categories such as "public data," "internal data," and "confidential data" to reflect different levels of data sensitivity. Furthermore, the inference method can consider factors beyond security levels to determine the appropriate machine learning model. For instance, the inference method can consider factors such as data complexity, data volume, or specific data attributes.By taking these factors into account, the inference method can select an ML model that not only meets the user's security requirements but also aligns with other relevant considerations.

[0069] In one example, each ML model in the set of one or more ML models can be a Tabular Variational Autoencoder (TVAE), Conditional Tabular Generative Adversarial Networks (CTGAN), or Private Data Release via Bayesian Networks (PrivBayes).

[0070] According to one example, the set of one or more ML models includes multiple ML models.

[0071] For example, the evaluation function is defined by one or more metrics, where the one or more metrics include a privacy metric and a utility metric, and the target value of the evaluation function is a pair of values ​​from the privacy metric and the utility metric. This can enable the secure handling of synthetic data.

[0072] The privacy metric can measure the level of privacy protection offered by the generated synthetic data, while the utility metric can assess the usefulness or accuracy of the synthetic data in representing the original data. This can facilitate the secure handling of synthetic data. Various techniques can be employed to quantify the level of privacy protection offered by the privacy metric. One approach is to measure the degree of information disclosure or identifiability within the synthetic data. This can be done by calculating the re-identification risk or the probability of re-identifying an entity from the synthetic data. Another approach is to assess the differential confidentiality of the synthetic data, which quantifies the confidentiality ensured by the data generation process.The utility metric, on the other hand, can focus on the quality and usefulness of the synthetic data. It can be measured by comparing statistical properties or characteristics of the synthetic data with those of the original data. For example, the utility metric can include measures such as the mean squared error, the correlation coefficient, or the accuracy of classification models trained on the synthetic data. To determine the target value of the evaluation function, the privacy and utility metrics can be combined into a pair of values. The specific combination method can vary depending on the user's needs and preferences. For instance, in another example, a weighted summation approach can be used, where the privacy and utility metrics are weighted differently to reflect their relative importance.Alternatively, a threshold-based approach can be used, where the target value is determined based on predefined thresholds for data privacy and utility metrics. By linking each trained machine learning (ML) model to a corresponding target value of the evaluation function, the inference method enables the selection of a suitable model that fulfills the user's request to generate synthetic data. The user's request, along with the target values, can be used as a criterion for selecting the trained ML model from the set of available models. This ensures that the generated synthetic data meets the user's specified data privacy and utility requirements. In practice, the inference method can be applied in various fields and scenarios.In healthcare, for example, inference can be used to create synthetic medical records that protect patient privacy while possessing the statistical properties required for research or analysis. In finance, synthetic transaction data can be generated to facilitate the development and testing of fraud detection algorithms without exposing real customer data. These are just a few examples, and inference can be adapted and customized to suit various applications and data types.

[0073] The evaluation function can enable the selection of the most suitable model for a given user request. The metric of the evaluation function can be, for example, accuracy, which is evaluated for different task types such as classification tasks. This means that each machine learning model can be associated with its accuracy in performing a specific task. The request can include the desired task. The request can also include a desired value for the evaluation function, or it may not. For example, if a user requests synthetic data for a classification task with a desired accuracy, the inference method could select a model that exhibits the appropriate accuracy for similar classification tasks. If a user requests synthetic data for a regression task, the system could select a model that exhibits low error for similar regression tasks.When a user requests synthetic data for a clustering task, the system could select a model that has a high silhouette score in similar clustering tasks. Model selection could be based, for example, on information about the models, which could be provided by data consumers.

[0074] The generated synthetic data should be as accurate and useful as possible. For example, in a classification task, the synthetic data might contain a balanced distribution of classes to ensure the model is not biased toward any one class. In a regression task, the synthetic data might contain a wide range of values ​​to ensure the model can handle different input scenarios. In a clustering task, the synthetic data might contain different clusters to ensure the model can distinguish between different groups.

[0075] For example, the inference process is performed according to a client-server configuration (or client-server model), where the inference process is executed on a server system, and the request is received from the user of a client system. The server system could, for example, be the generator of synthetic data. The client system is connected to the server system. The client system is connected to the server system via one or more networks, where the one or more networks could be a local area network (LAN), a radio access network (RAN), a metropolitan area network (MAN), a wide area network (WAN), worldwide interoperability for microwave access (WiMAX), a wireless local area network (WLAN), all or part of the internet, any other communication system, or a combination thereof.

[0076] Data exchange between the client system and the server system can occur, for example, via a client-server protocol. This client-server protocol can be platform-independent, meaning that the clients and the server can run on different operating systems. Standardized protocols enable seamless communication between clients and servers, regardless of the underlying technology.

[0077] In a sample implementation, the client system can initiate a connection with the server system using a communication protocol such as TCP / IP. Once the connection is established, the client system can send a request for synthetic data to the server system. This request could, for example, be in the form of an HTTP request. The server system receives the request and processes it to select the machine learning (ML) model and generate the synthetic data using that model. The server system might, for example, generate a response containing the requested synthetic data or a reference to a location where the synthetic data is stored. The response could be formatted according to the protocol used, such as an HTTP response. The server system can then send the response back to the client system over the existing connection. The response might include information about the requested synthetic data.The client system receives the response from the server system and can process it accordingly. This might involve displaying the synthetic data to the user or processing the synthetic data, for example, by training a classifier or performing an analysis. Finally, either the client or the server system can initiate the termination of the connection. This frees up resources, allowing both parties to handle other requests or connections.

[0078] The client-server configuration enables centralized management of inference data, with machine learning models stored and managed on the server system. This centralized approach simplifies data access, backup, and security measures. Access restrictions can be enforced through security policies, facilitating the protection of the machine learning models. This configuration also allows for horizontal scalability, enabling an increase in the number of client machines, as well as vertical scalability, allowing the entire process to be moved to more powerful servers or a multi-server configuration. This flexibility allows the system to meet growing demands and handle increased workloads.

[0079] According to an example, the set was trained by one or more ML models with an original dataset, where the original dataset is extended with an additional column, where the additional column is associated with an additional attribute that has two distinct values: a first value and a second value; if an entry of the extended dataset has at least one NULL value, the first value is assigned to the additional attribute in that entry, otherwise the second value is assigned to the additional attribute in that entry;wherein the inference procedure further comprises: identifying entries in the synthetic data that correspond to the entries with the first value in the extended dataset, and replacing values ​​of one or more attributes that have NULL values ​​in the extended dataset with NULL values ​​in the synthetic data, wherein the attributes having the NULL values ​​are numeric or continuous attributes. An entry in the synthetic data can match an entry in the extended dataset if they have a similarity value that exceeds a threshold, wherein the similarity value can be obtained by applying a similarity metric to the compared entries.

[0080] According to one example, the inference process further includes: Deleting the synthetic data. The user can request the deletion of the synthetic data. Alternatively, the synthetic data can be deleted automatically if a data retention criterion is not met. The data retention criterion might, for example, require a minimum amount of free disk space on the computer system where the synthetic data is stored. The data retention criterion might, for example, stipulate that the synthetic data be deleted immediately after a successful transfer. If, for example, the minimum disk space is not available on the computer system, the synthetic data generator can at least delete the synthetic data to free up space. The user can send another request to generate synthetic data at a later time (e.g., when the synthetic data is needed).If the user needs synthetic data days after deleting the synthetic data, the other request and one or more target values ​​can be used to select a trained ML model from the set of one or more trained ML models that satisfies the other request. The selected ML model can then be used to generate synthetic data.

[0081] As an example, the set of one or more machine learning (ML) models comprises multiple ML models. The inference process further involves using the query and the one or more target values ​​to select another trained ML model from the set of one or more trained ML models that satisfies the query. The other selected ML model can be used to generate further synthetic data. This can make it possible to obtain a sufficient quantity of synthetic data. For example, multiple ML models might satisfy the query. In this case, this example can leverage this to increase the amount of synthetic data generated. Following the example above, the query might include a desired value of the evaluation function that is equidistant from two target values, tV1 and tV2. In this case, ML1 and ML2 can be used to generate the synthetic data.

[0082] According to one example, the inference process further includes using the synthetic data to train a machine learning model. The training can be performed, for instance, on a classification or regression task. Training can be carried out using attributes from the synthetic data's set of attributes as both input and target for supervised learning. In this case, the user-submitted request can specify the type of task required. Alternatively, the synthetic data can also be used to perform data analysis.

[0083] Fig. Figure 1 is a flowchart of an inference procedure for generating synthetic data according to an example from the present subject matter. The procedure from Fig. 1 can, for example, be from the computer system Fig. 3 will be executed.

[0084] In step 101, a set of one or more trained ML models can be provided. The set of one or more trained ML models can be trained using the training procedure described here or using other training techniques. Each trained ML model in the set of one or more trained ML models is associated with a corresponding target value of an evaluation function, which was used to evaluate the set of one or more ML models for training. The set of one or more ML models can, for example, be stored in the computer system of Fig. 3. The set of one or more trained ML models provided in step 101 can be the ML model(s) used in step 207 of Fig. 2 were provided. Alternatively, the set of one or more trained ML models provided in step 101 can be all models generated by the procedure of Fig. 2 were trained.

[0085] A request to generate synthetic data can be received from a user in step 103. The request and the target value(s) can be used in step 105 to select a trained ML model from the set of trained ML models that satisfies the request. The selected ML model can be used in step 107 to generate synthetic data. The synthetic data can be made available to the user in step 109.

[0086] In one example, steps 103 through 109 can be repeated for each subsequent request received from the same or a different user. In another example, steps 105 through 109 can be performed automatically in response to receiving the request.

[0087] Fig. Figure 2 is a flowchart of a training procedure for training a set of multiple machine learning models to generate synthetic data according to an example from the present subject matter. In an example, the set of one or more machine learning models can be a single ML model. Alternatively, the set of one or more machine learning models can consist of multiple ML models. In step 201, a dataset (called the original dataset) can be provided. The original dataset comprises entries, each of which includes values ​​from the attribute set. The original dataset can, for example, be stored in the computer system referred to in Fig. 3 is described. The set of one or more machine learning models can be implemented in the computer system of Fig. 3 be stored. The procedure of Fig. 2 can be done, for example, by the computer system of Fig. 3 will be carried out.

[0088] In step 203, at least one objective function and one evaluation function can be determined for training the set of one or more machine learning models. For example, in step 203, user input can be received, where the user input specifies the at least one objective function and the evaluation function. In another example, the at least one objective function and the evaluation function can be selected (e.g., randomly) from predefined objective and evaluation functions associated with the machine learning models (e.g., in a file stored on the computer system).

[0089] The original dataset can be used in step 205 to train each machine learning (ML) model from the set of one or more ML models. Training each ML model is performed to optimize one of the objective functions associated with that ML model. Training each ML model can be used to generate synthetic data. The evaluation function can be used to assess the ML model during training or after training the set of one or more ML models (upon receiving the trained set of one or more ML models).

[0090] At least one trained machine learning (ML) model from the set of trained ML models can be provided in step 207 to generate synthetic data (corresponding to the original dataset). This at least one trained ML model can, for example, be stored in the computer system. The computer system can then, for instance, receive requests to generate synthetic data and, in response, use a selected model from the at least one trained ML model to generate the synthetic data.

[0091] In an example, the procedural steps of the process can be described as follows: Fig. Step 2 can be executed automatically, for example, upon receipt of the original dataset or upon receipt of a request. Providing the original dataset in step 201 can include creating the original dataset. This allows training to be automated based on the original dataset. The provided trained machine learning model can then be configured to generate synthetic data in the desired quantity on demand.

[0092] The at least one trained ML model provided in step 207 can be the set of trained ML models. Alternatively, the at least one trained ML model provided in step 207 can be selected from the set of trained ML models using a provisioning criterion. The provisioning criterion can, for example, require specific values ​​of the evaluation function.

[0093] Fig. Figure 3 is a block diagram of an exemplary computer system for carrying out the present procedure according to an example of the present subject matter.

[0094] The components of the 602 computer system can include, among other things, one or more processors or processing units 603, a storage system 611, a storage unit 605, and a bus 607 that connects various system components, including the storage unit 605, to the processor 603. The storage system 611 can, for example, include a hard disk drive (HDD). The storage unit 605 can contain computer-readable media in the form of volatile memory, such as random-access memory (RAM) and / or cache memory.

[0095] The Computer System 602 can also communicate with one or more external devices, such as a keyboard, pointing device, display 613, etc.—devices that allow a user to interact with the Computer System 602—and / or with any devices (e.g., network card, modem, etc.) that allow the Computer System 602 to communicate with one or more other computer devices. Such communication can occur via the I / O interface(s) 619. Furthermore, the Computer System 602 can communicate with one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or a public network (e.g., the Internet), via a network adapter 609. As shown in the figure, the network adapter 609 communicates with the other components of the Client System 602 via the bus 607.

[0096] The 605 memory unit is configured to store applications that can be executed on the 603 processor. The 605 memory unit can, for example, contain an operating system and one or more application programs. The application programs contain instructions that, when executed, carry out the operation described in the 603 processor. Fig. 1 or Fig. The 2 described procedures enable this.

[0097] The present subject matter may include the following clauses regarding the aspect of conclusion.

[0098] Clause 1. Method for generating synthetic data, wherein the method comprises: providing a set of one or more trained machine learning (ML) models, wherein each trained ML model of the set of one or more trained ML models is trained to generate synthetic data, wherein each trained ML model of the set of one or more trained ML models is associated with a respective target value of an evaluation function that was used to evaluate the set of one or more ML models for training; receiving a request to generate synthetic data from a user; using the request and the one or more target values ​​to select a trained ML model from the set of one or more trained ML models that satisfies the request; using the selected ML model to generate synthetic data.

[0099] Clause 2. Procedure according to sentence 1, wherein the query specifies a desired value of the evaluation function and a desired number of entries, wherein the selected ML model is the ML model whose target value matches the desired value, and wherein the generated synthetic data comprise entries in the desired number.

[0100] Clause 3. The procedure according to Clause 1 or 2, wherein the target value assigned to each trained ML model specifies a security level of the synthetic data generated by the ML model, the procedure further comprising: providing user categories, each user category being assigned a security level, and determining to which user category the user belongs in order to select the appropriate ML model.

[0101] Clause 4. Procedure according to sentence 1 or 2, wherein the selection of the trained ML model comprises: ranking the set of trained ML models using the respective target values ​​of the evaluation function; determining an appropriate rank for the user based on the privilege access predefined by the user; and selecting the ML model associated with the determined rank.

[0102] Clause 5. The procedure according to any of the preceding clauses, wherein the evaluation function is defined by one or more metrics, the one or more metrics comprising a data protection metric and a utility metric, wherein the target value of the evaluation function is a pair of values ​​from the data protection metric and the utility metric.

[0103] Clause 6. The procedure according to any of the preceding sections, performed in accordance with a client-server configuration, wherein the procedure is performed on a server system, wherein the request is received from the user of a client system, and wherein the client system is remotely connected to the server system.

[0104] Clause 7. The procedure according to any of the preceding sections shall further include repeating the selection, generation and provision for each further request received of synthetic data from the user or any other user.

[0105] Clause 8. Method according to any of the preceding sections, wherein the set has been trained by one or more ML models on an original dataset, wherein the original dataset is extended with an additional column, the additional column being associated with an additional attribute having two distinct values: a first value and a second value; if an entry of the extended dataset has at least one NULL value, the first value is assigned to the additional attribute in that entry, otherwise the second value is assigned to the additional attribute in that entry;wherein the method further comprises: identifying entries in the synthetic data that correspond to the entries with the first value in the extended dataset, and replacing values ​​of one or more attributes that have NULL values ​​in the extended dataset with NULL values ​​in the synthetic data, wherein the attributes that have the NULL values ​​are numeric or continuous attributes.

[0106] Clause 9. The procedure according to any of the preceding sections further includes the use of the synthetic data to train a machine learning model, wherein the machine learning model uses attributes from the set of attributes as input and target for training, or to perform a data analysis.

[0107] Clause 10. The procedure according to any of the preceding sections, further comprising: deleting the synthetic data in response to the deletion of the synthetic data; receiving a further request from the user to generate synthetic data; using the further request and the one or more target values ​​to select a trained ML model from the set of one or more trained ML models that satisfies the further request; and using the selected ML model to generate synthetic data.

[0108] Clause 11. The procedure according to any of the preceding clauses, wherein the set of one or more ML models comprises multiple ML models, and wherein the procedure further comprises: using the query and the target values ​​to select another trained ML model from the set of trained ML models that satisfies the query; and using the other selected ML model to generate further synthetic data.

[0109] Clause 12. The procedure according to any of the preceding sections, wherein the use of the request and the one or more target values ​​to select the trained ML model that satisfies the request comprises: assigning one of the target values ​​to the user based on a predefined criterion, wherein the criteria assign a target value based on a user's location and / or on the time at which the request is received and / or on the basis of other attributes describing the user and / or the request; wherein the assigned target value is the target value that satisfies the request.

[0110] Clause 13. The procedure under any of the preceding clauses further includes the provision of the synthetic data to the user.

[0111] Clause 14. The procedure according to any of the preceding clauses, wherein the set of one or more ML models comprises multiple ML models.

[0112] Clause 15. Synthetic Data Generator for generating synthetic data, wherein the synthetic data generator comprises a set of one or more trained machine learning (ML) models, each trained ML model of the set of one or more trained ML models being trained to generate synthetic data, each trained ML model of the set of one or more trained ML models being associated with a respective target value of an evaluation function that was used to evaluate the set of one or more ML models for training; wherein the synthetic data generator is configured to: receive a request to generate synthetic data from a user; use the request and the one or more target values ​​to select a trained ML model from the set of one or more trained ML models that satisfies the request;Using the selected ML model to generate synthetic data.

[0113] As will be clear to those skilled in the art, aspects of the present invention can be embodied as a device, a method, a computer program, or a computer program product. Accordingly, aspects of the present invention can take the form of a purely hardware variant, a purely software variant (including firmware, resident software, microcode, etc.), or a variant that combines software and hardware aspects, which may be generally referred to here as a "circuit," "module," or "system." Furthermore, aspects of the present invention can take the form of a computer program product embodied in one or more computer-readable media containing computer-executable code. A computer program comprises the computer-executable code or "program instructions."

[0114] The term "computer system" refers to data processing hardware and encompasses all types of devices, apparatus, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The device may also be or include specialized logic circuitry, such as a central processing unit (CPU), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In some implementations, the data processing device and / or specialized logic circuitry may be hardware-based and / or software-based. The device may optionally include code that creates an execution environment for computer programs, such as code representing processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.The present disclosure considers the use of data processing equipment with or without conventional operating systems, e.g. LINUX, UNIX, WINDOWS, MAC OS, ANDROID, IOS or any other suitable conventional operating system.

[0115] Any combination of one or more computer-readable media can be used. The computer-readable medium can be a computer-readable storage medium. A "computer-readable storage medium," as used here, includes any tangible storage medium capable of storing instructions that can be executed by a processor of a computing device. The computer-readable storage medium can be referred to as a computer-readable non-transitory storage medium. The computer-readable storage medium can also be referred to as a concrete computer-readable medium. In some embodiments, a computer-readable storage medium may also be capable of storing data that can be accessed by the processor of the data processing system.

[0116] "Computer memory" or "memory" is an example of a computer-readable storage medium. Computer memory is any memory that a processor can directly access. Another example of a computer-readable storage medium is "computer memory" or "memory." Computer memory is any non-volatile, computer-readable storage medium. In some embodiments, computer memory can also be computer storage, or vice versa.

[0117] A "processor," as used here, comprises an electronic component capable of executing a program, a machine-executable instruction, or computer-executable code. References to the data processing system comprising "a processor" should be understood to mean that it may contain more than one processor or processing core. The processor may, for example, be a multi-core processor. A processor may also refer to a collection of processors within a single computer system or distributed across multiple computer systems. The term "computing device" should also be interpreted to mean that it may refer to a collection or network of computing devices, each comprising one or more processors.The executable computer code can be executed by multiple processors, which may be located in the same computer device or even distributed across multiple computer devices.

[0118] Computer-executable code may comprise machine-executable instructions or a program that causes a processor to execute an aspect of the present invention. Computer-executable code for performing operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, Smalltalk, C++, or similar languages, and conventional procedural programming languages ​​such as the programming language "C" or similar languages, and compiled into machine-executable instructions. In some cases, the computer-executable code may be in the form of a high-level language or in pre-compiled form and used in conjunction with an interpreter that generates the machine-executable instructions on the fly.

[0119] In general, program instructions can be executed on one processor or on multiple processors. In the case of multiple processors, they can be distributed across several different units. Each processor could execute a portion of the instructions intended for that unit. Therefore, when referring to a system or process involving multiple units, the computer program or program instructions are to be understood as being capable of being executed by a processor assigned to or associated with the respective unit.

[0120] Although the invention is illustrated and described in detail in the drawings and the foregoing description, these illustrations and descriptions are to be regarded as illustrative or exemplary and not limiting; the invention is not limited to the disclosed examples.

Claims

[1] A method for training a set of one or more machine learning (ML) models to generate synthetic data, the method comprising: Providing (201) a data set, referred to as the original data set, comprising entries, each entry of the entries comprising values ​​of a set of attributes; Determining (203) at least one objective function and one evaluation function for training the set of one or more ML models; Using (205) the original data set to train each ML model of the set of one or more ML models to optimize a respective objective function of the at least one objective function, wherein the evaluation function is used to evaluate the ML model during training of the ML model or after training of the set of one or more ML models; Providing (207) at least one trained ML model from the set of trained ML models. [2] The method of claim 1, wherein the at least one objective function is an objective function used to train the set of one or more ML models. [3] The method of claim 1, wherein the set of one or more ML models comprises a plurality of ML models, wherein the at least one objective function is a set of objective functions each associated with the set of ML models. [4] The method of claim 1, wherein the set of one or more ML models comprises a plurality of ML models, wherein the at least one objective function is a plurality of objective functions, each objective function of the plurality of objective functions being associated with a corresponding one or more ML models of the set of ML models. [5] A method according to any one of the preceding claims, wherein the evaluation function is defined by one or more metrics and a target criterion, wherein evaluating the ML model using the evaluation function comprises: evaluating the one or more metrics using synthetic data generated by the ML model and the original dataset, determining whether the evaluated one or more metrics meet the target criterion. [6] The method of claim 5, wherein the one or more metrics comprise a privacy metric and a utility metric. [7] The method of claim 5 or 6, wherein the target criterion requires that a combination of the one or more metrics of the evaluation function has a target value or the combination is within a target range of values, the combination being a weighted sum of the metrics of the evaluation function using weights, each metric being associated with a corresponding one of the weights. [8] The method of any preceding claim, wherein the set of one or more ML models comprises multiple ML models. [9] The method according to any one of the preceding claims, wherein providing the original data set comprises Carrying out at least one of the following activities: Receiving original entries from one or more data sources with structured data; or Deriving original entries from one or more data sources with unstructured data; Extracting the entries of the original dataset from the original entries. [10] The method of claim 9, wherein the determining is performed such that the entries of the original data set are the original entries. [11] The method according to any one of the preceding claims, wherein the entries of the original data set are obtained by preprocessing the original entries. [12] The method of claim 11, wherein the preprocessing comprises: Replacing NULL values ​​in the original entries with predefined values ​​and using the resulting original entries as the entries of the original dataset. [13] The method of claim 12, wherein replacing each NULL value comprises: identifying an attribute from the set of attributes that has the NULL value; sampling the values ​​of the identified attribute that are not NULL values ​​to obtain a sampled value, wherein the predefined value replacing the NULL value is the sampled value. [14] The method of claim 13, wherein the sampling is performed such that the distribution of non-NULL values ​​of the identified attribute before replacing the NULL value is similar to the distribution of non-NULL values ​​of the identified attribute after replacing the NULL value. [15] The method according to claim 13 or 14 further comprises: Creating an additional column in the original dataset, resulting in an extended dataset, where the additional column is associated with an additional attribute with two different values: first value and second value; if an entry of the extended data set has at least one NULL value, assigning the first value to the additional attribute in this entry, otherwise assigning the second value to the additional attribute in this data set; after generating the synthetic data by each trained ML model of the provided trained ML models, identifying entries of the synthetic data corresponding to the entries with the first value in the augmented dataset, and replacing the synthetic data values ​​of the one or more attributes that have NULL values ​​in the augmented dataset with NULL values. [16] A method according to any one of the preceding claims 11 to 15, wherein the preprocessing comprises: detecting and removing outliers in the original entries using a predefined outlier processing method. [17] The method of claim 16, wherein the method for processing outliers uses k-anonymization. [18] Method according to one of the preceding claims, wherein the provided at least one trained ML model is the set of trained ML models. [19] The method of any one of the preceding claims 1 to 17, further comprising: ranking the set of trained ML models using the respective values ​​of the evaluation function, selecting the highest ranked n ML models from the set of ML models, wherein the provided at least one trained ML model is the selected ML model, where n is an integer greater than or equal to one. [20] Method according to one of the preceding claims, wherein a trained ML model of the provided at least one trained ML model is used to generate synthetic data. [21] A method according to any one of the preceding claims, wherein the evaluation function is defined by one or more metrics and a target criterion, the target criterion specifying a target value or a target range for the one or more metrics, the method comprising: Creating a synthetic data generation module, wherein the data generation module comprises the at least one ML model from the set of trained ML models in association with the respective target values; Receipt of a request to generate synthetic data by the synthetic data generation module; Selection of a trained ML model by the synthetic data generation module based on the target value associated with the trained ML model and the received request; Use of the selected ML model by the synthetic data generation module to generate synthetic data; Provision of the synthetic data by the synthetic data generation module to a synthetic data requester. [22] A computer program containing instructions for causing a computer system to perform the method of claim 1. [23] A computer system (602) for training a set of one or more machine learning (ML) models to generate synthetic data, the computer system comprising the set of one or more ML models and an original data set having entries, each entry of the entries comprising values ​​of a set of attributes, the computer system being configured to: Determining (203) at least one objective function and one evaluation function for training the set of one or more ML models; Using (205) the original data set to train each ML model of the set of one or more ML models to optimize a respective objective function of the at least one objective function, wherein the evaluation function is used to evaluate the ML model during training of the ML model or after training of the set of one or more ML models; Providing (207) at least one trained ML model from the set of trained ML models.

Images