Training and generating synthetic data using (continuous) normalizing flow that preserves privacy

Abstract

A computer-implemented method for developing a differential privacy model is provided. The method includes collecting a private and personal dataset comprising private and / or personal data and training the differential privacy model via backpropagation to optimize an expected accuracy of an adversarial loss and a privacy loss. The differential privacy model is associated with a continuous normalizing flow. The method further includes outputting the trained differential privacy model. The trained differential privacy model is configured to generate new synthetic datasets that are used to train one or more downstream tasks. The method has applications including, but not limited to, use cases in medicine / healthcare such as Electronic Health Record (EHR) generation, single and bulk cell sequencing data generation, and pre-training large multimodal language models (LLMs) associated with clinical data, and can further for example, be used to optimize machine learning tasks or to support decision making.

Description

Attorney Docket No.820675 (Client Ref. NLE-1352-23-WO) TRAINING AND GENERATING SYNTHETIC DATA USING (CONTINUOUS) NORMALIZING FLOW THAT PRESERVES PRIVACY CROSS-REFERENCE TO PRIOR APPLICATION

[0001] Priority is claimed to European Patent Application No. EP24155113.4, filed on January 31, 2024, the entire contents of which is hereby incorporated by reference herein. FIELD

[0002] The present disclosure relates to Artificial Intelligence (AI) and machine learning (ML), and in particular to a method, system, data structure, computer program product and computer-readable medium for training and generating synthetic data for healthcare applications and biomedical fields using continuous normalizing flow. BACKGROUND

[0003] Generating synthetic data for healthcare applications is critical for developing artificial intelligence (AI) models and / or for testing existing tools. Data privacy laws regulate the use of personal data. In some instances, the data can be anonymized, but in other situations such as with genetic data, it is not possible for such data to be anonymized, which makes it extremely difficult to develop AI models and test AI tools. Therefore, synthetic data can be used to develop and test the AI models. However, the generation of synthetic data can be difficult. In fact, conventional approaches for generating synthetic data have many limitations including the requirement for the original dataset to generate the anonymized dataset. Furthermore, these conventional approaches are single modality (e.g., tabular data or images) and they typically only describe continuous data (e.g., pixel color or tabular entry). SUMMARY

[0004] In an embodiment, the present disclosure provides a computer-implemented method for developing a differential privacy model. For example, the method comprises collecting a private and personal dataset comprising private and / or personal data. The method further comprises training the differential privacy model via backpropagation to optimize an expected accuracy of an adversarial loss and a privacy loss and outputting the trained differential privacy model. The differential privacy model is associated with a continuous normalizing flow and the trained differential privacy model is configured to generate new synthetic datasets that are used to train one or more downstream tasks. The method has applications including, but not limited to, use cases in medicine / healthcare such as Electronic Health Record (EHR) generation, in single and bulk cell sequencing data generation, and in pre-training large multimodal language models (LLMs) in various fields, including LLMs associated with clinical data. The methodAttorney Docket No.820675 (Client Ref. NLE-1352-23-WO) can, for example, be used to optimize machine learning tasks and / or to support decision making in these applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Embodiments of the present disclosure will be described in even greater detail below based on the exemplary figures. The present disclosure is not limited to the exemplary embodiments. All features described and / or illustrated herein can be used alone or combined in different combinations in embodiments of the present disclosure. The features and advantages of various embodiments of the present disclosure will become apparent by reading the following detailed description with reference to the attached drawings which illustrate the following:

[0006] FIG.1 is an overview of training a generative flow model according to an embodiment of the present disclosure.

[0007] FIG.2 is an example of an implementation of the straight-through gradient estimation for integer variables used for the categorical case according to an embodiment of the present disclosure;

[0008] FIG.3 is an overview of performing a downstream prediction task using a generative flow according to an embodiment of the present disclosure;

[0009] FIG.4 is an overview of computing a predictive task loss for training generative and privacy preserving flow according to an embodiment of the present disclosure;

[0010] FIG.5 is an overview of verification of the privacy level according to an embodiment of the present disclosure;

[0011] FIG.6 is an overview of graph generative flow according to an embodiment of the present disclosure;

[0012] FIG.7 is a comparison of the histograms of the minimum distance of each sample from the original dataset to either the original dataset (excluding the point itself) or the synthetic dataset of the same dimension;

[0013] FIG.8 is another comparison of the histograms of the minimum distance of each sample from the original dataset to either the original dataset (excluding the point itself) or the synthetic dataset of the same dimension; and

[0014] FIG.9 is a block diagram of an exemplary processing system, which can be configured to perform any and all operations disclosed herein. DETAILED DESCRIPTION

[0015] As mentioned above, generating synthetic data for healthcare applications can be critical for developing AI models and for testing existing tools. Synthetic data is data that is generated either manually, based on one or more hand-crafted rules, or by a model trained on real data. The generation of data using predefined rules might not guarantee that the dataAttorney Docket No.820675 (Client Ref. NLE-1352-23-WO) includes similar statistics as the original data, and may produce inaccurate predictions if used to train or test AI models. In the case where synthetic data is generated from real data (e.g., based on using a model such as an AI model), there is a requirement that the personal data is not exposed.

[0016] To comply with this requirement, differential privacy has been introduced, which defines the privacy of individual datasets. For instance, differential privacy requires that a dataset that includes specific personal data is to be statistically indistinguishable (up to a predefined value) from a second dataset that does not include the same specific personal data. A standard approach to perform differential privacy is to introduce enough noise in the data that is proportional to the level of privacy required, and privacy loss is used to measure how much information is still revealed by the dataset even with the addition of noise.

[0017] Differential privacy also includes the concept of an algorithm, where it is required that the test is performed on the output of the AI model after training on the two datasets. In this case, a common approach (see, e.g., Abadi, M. et al. Deep Learning with Differential Privacy. in Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security 308–318 (2016). doi:[10.1145 / 2976749.2978318], which is hereby incorporated by reference herein) is to clip and add noise to the gradients while learning.

[0018] However, as mentioned above, conventional approaches suffer from many limitations. As such, embodiments of the present disclosure described below provide aspects that address at least some of these limitations. For instance, embodiments of the present disclosure provide methods and systems for privacy preserving generation of synthetic data for various applications including, but not limited to generation of biomedical and / or healthcare synthetic data (e.g., Electronic Health Records (EHR) synthetic data). According to one or more embodiments, the method can include using an extended loss with adversarial and privacy preserving terms to achieve privacy. According to one or more embodiments, the method can include using normalizing flow to generate the synthetic data and training the normalizing flow from samples using either end-to-end training and / or a flow matching approach. As such, embodiments of the present disclosure can include a method that is an infinite data generator and the method can be used in isolation or in a federated learning setting, where the flow training is divided by federated users and the parameters can be aggregated centrally (e.g., as in federated learning).

[0019] By using embodiments of the present disclosure, one or more advantages can be achieved, including but not limited to, guaranteeing of privacy based on monitoring the loss during training and testing, and a possible infinite generative model that can generate an arbitrary number of different datasets. For instance, as mentioned above, generating syntheticAttorney Docket No.820675 (Client Ref. NLE-1352-23-WO) data can be critical to developing AI models (e.g., sufficiently training the AI models) as actual data (e.g., genetic data or EHR data) might not be available for use due to various limitations (e.g., legal requirements to not share such data). As such, embodiments of the present disclosure describe a process that generates synthetic data that provides additional guarantees for privacy and further can generate an arbitrary number of different datasets. By accomplishing this, embodiments of the present disclosure is capable of generating synthetic data that is useful for training AI models to perform a variety of prediction tasks, which would otherwise be unable to be trained and utilized without such synthetic data.

[0020] According to a first aspect, the present disclosure provides a computer-implemented machine learning method for developing a differential privacy model. The method comprises: collecting a private and personal dataset comprising private and / or personal data; training the differential privacy model via backpropagation to optimize an expected accuracy of an adversarial loss and a privacy loss, wherein the differential privacy model is associated with a continuous normalizing flow; and outputting the trained differential privacy model, wherein the trained differential privacy model is configured to generate new synthetic datasets that are used to train one or more downstream tasks.

[0021] According to a second aspect, the method according to the first aspect further comprises prior to training the differential privacy model, choosing the continuous normalizing flow and loss parameters associated with the adversarial loss and the privacy loss.

[0022] According to a third aspect, the method according to any of the first or the second aspect further comprises that training the differential privacy model is based on using a categorical gradient estimator for categorical variables associated with the private and personal dataset, and wherein the method further comprises choosing categorical gradient parameters associated with the categorical gradient estimator.

[0023] According to a fourth aspect, the method according to any of the first to third aspects further comprises using the trained differential privacy model to generate the new synthetic datasets that guarantee differential privacy based on using a verification loss.

[0024] According to a fifth aspect, the method according to any of the first to fourth aspects further comprises the verification loss is computed based on comparing data from a new synthetic dataset from the synthetic datasets and original data from the private and personal dataset.

[0025] According to a sixth aspect, the method according to any of the first to fifth aspects further comprises that training the differential privacy model comprises minimizing the privacy loss using the continuous normalizing flow that allows for computation of a probability distribution up to a constant term.Attorney Docket No.820675 (Client Ref. NLE-1352-23-WO)

[0026] According to a seventh aspect, the method according to any of the first to sixth aspects further comprises defining a predictive loss associated with the one or more downstream tasks, and wherein training the differential privacy model is based on using the predictive loss.

[0027] According to an eighth aspect, the method according to any of the first through seventh aspects further comprises that defining the predictive loss comprises: splitting features from the private and personal dataset into an input set and an output set; training the one or more downstream tasks based on the input set and the output set, wherein the one or more downstream tasks comprises one or more downstream models; and obtaining the predictive loss based on the one or more trained downstream tasks.

[0028] According to an ninth aspect, the method according to any of the first through eighth aspects further comprises that the differential privacy model comprises: a first model for generating a structure of a graph based on a number of nodes; and a second model for generating attributes associated with the structure of the graph.

[0029] According to a tenth aspect, the method according to any of the first through ninth aspects further comprises that the trained differential privacy model is used in a biomedical field for electronic healthcare data.

[0030] According to an eleventh aspect, the method according to any of the first through tenth aspects further comprises that the private and / or personal data is associated with electronic health record (EHR) data, and wherein the new synthetic datasets generated by the trained differential privacy model comprises synthetic EHR data.

[0031] According to a twelfth aspect, a computer system is provided for developing a differential privacy model, the system comprising one or more hardware processors, which, alone or in combination, are configured to provide for execution of the following steps: collecting a private and personal dataset comprising private and / or personal data; training the differential privacy model via backpropagation to optimize an expected accuracy of an adversarial loss and a privacy loss, wherein the differential privacy model is associated with a continuous normalizing flow; and outputting the trained differential privacy model, wherein the trained differential privacy model is configured to generate new synthetic datasets that are used to train one or more downstream tasks.

[0032] According to an thirteenth aspect, the computer system according to the twelfth aspect further configured to prior to training the differential privacy model, choose the continuous normalizing flow and loss parameters associated with the adversarial loss and the privacy loss.

[0033] According to a fourteenth aspect, the computer system according to the twelfth or thirteenth aspects further configured to train the differential privacy model based on using aAttorney Docket No.820675 (Client Ref. NLE-1352-23-WO) categorical gradient estimator for categorical variables associated with the private and personal dataset, and wherein the computer system is further configured to choose categorical gradient parameters associated with the categorical gradient estimator.

[0034] According to a fifteenth aspect, the computer system according to any of the twelfth through fourteenth aspects further configured to use the trained differential privacy model to generate the new synthetic datasets that guarantee differential privacy based on using a verification loss.

[0035] According to a sixteenth aspect of the present disclosure, the computer system according to any of the twelfth through fifteenth aspects further configured to compute the verification loss based on comparing data from a new synthetic dataset from the synthetic datasets and original data from the private and personal dataset.

[0036] A seventeenth aspect of the present disclosure provides a tangible, non-transitory computer-readable medium having instructions thereon, which, upon being executed by one or more processors, provides for execution of the method according to any of the first to the fourteenth aspects and / or the method comprising the following: collecting a private and personal dataset comprising private and / or personal data; training the differential privacy model via backpropagation to optimize an expected accuracy of an adversarial loss and a privacy loss, wherein the differential privacy model is associated with a continuous normalizing flow; and outputting the trained differential privacy model, wherein the trained differential privacy model is configured to generate new synthetic datasets that are used to train one or more downstream tasks.

[0037] FIG.1 is an overview 100 of training a generative flow model according to an embodiment of the present disclosure. For instance, the overview 100 describes a method according to an embodiment of the present disclosure which is also referred to as a PrivacyFlow method 110 for training a generative flow model (e.g., the generative flow 112) in a privacy preserving way that guarantees that the samples satisfy the differential privacy. From the trained PrivacyFlow method 110, multiple datasets (e.g., privacy dataset 1120 to privacy dataset N 122) can be generated from the differential privacy flow model 118 (e.g., a differential privacy model).

[0038] In some examples, a computing platform can be used to perform one or more embodiments of the present disclosure (e.g., train the generative flow model and / or use the generative flow model to generate synthetic datasets). The computing platform is and / or includes, but is not limited to, a desktop, laptop, server, repository, database, computer system and / or other types of computing entities that generally comprises one or more communication components, one or more processing components, and one or more memory components.Attorney Docket No.820675 (Client Ref. NLE-1352-23-WO)

[0039] For instance, the computing platform can perform one or more aspects described in overview 100. For instance, the computing platform can use the PrivacyFlow method 110 to train the generative flow 112, and use the trained generative flow model (e.g., the differential privacy flow model 118) to generate privacy datasets such as the first privacy dataset 120 and the N privacy dataset 122. In operation, the computing platform can obtain private and personal dataset 102, perform training 104 and the PrivacyFlow method 110, compute losses 108 based on performing the PrivacyFlow method 110, compute gradients 106, use a differential privacy flow model 118 (e.g., the trained generative flow model 112 after performing the PrivacyFlow method 110), and / or generate privacy datasets 120-122. For instance, the computing platform can perform the PrivacyFlow method 110 to train a generative flow model 112 that performs generative flow. For example, the PrivacyFlow method 110 can include using a generative flow model 112 (generative flow), clipping and adding noise 144, and computing adversarial and privacy loss 116. The training 104 can include providing gradients 106 back to the PrivacyFlow method 110. Furthermore, the training 104 can include calculating losses 108 based on performing the adversarial and privacy loss 116. Subsequently, a differential privacy flow model 118 (e.g., the trained generative flow model 112) can be utilized to generate a plurality of privacy datasets (e.g., the first privacy dataset 120 (privacy dataset 1) through the final privacy dataset 122 (privacy dataset N)). The overview 100 will be described in further detail below.

[0040] For instance, the conditional continuous normalizing flow is first described. For example, in a situation where privacy is important (e.g., medical or healthcare fields and / or applications), there can be a desire to be able to generate synthetic data that is representative of actual data, but that also does not reveal the individual data (e.g., actual data associated with actual people). Therefore, according to one or more embodiments of the present disclosure, the following four aspects / setup are considered and described in further detail below. For instance, first, normalizing flows (e.g., continuous normalizing flows (CNFs) that generate the synthetic data (e.g., categorical and / or continuous data) with conditional flow matching and / or discrete differential models are considered. The discrete differentiable models are described below when discussing the discrete gradients. Second, adversarial training with noise injection in the hidden features to avoid revealing the individual data is considered. Third, regularization to compress the flow is considered. Fourth, conditional flow to verify that the classification / regression task is still possible and augmented loss to promote reconstruction of the data from partial information is considered.

[0041] Adversarial training flows are described below. For instance, according to one or more embodiments, the PrivacyFlow method 110 can include training a discrete CNF to generate theAttorney Docket No.820675 (Client Ref. NLE-1352-23-WO) data. The PrivacyFlow method 110 can proceed by first using conditional flow on the discrete / categorical data and then refining to the differentiable variables.

[0042] For instance, according to one or more embodiments, the PrivacyFlow method 110 can use adversarial training to enforce privacy, which is described by the below expression: 1 mi ^n^ᇱ^ฮ ௫భି௫భฮmஸఋa,௫xబ∼ே^^,ூ^^^ℓ^ℎ^^^^|^^௧^, ^^^^ ௩ ^^ᇲ^

[0043] For example, adversarial training solving a minimax-lossproblem where ℎ is the hypothesis sample size, ^^^is afeature, ℓ is a loss function, ^^௧ is the flow of the CNF, ^^^ is a starting sample, ^^^ ∼ ^^^0, ^^^indicates that the starting sample has a distribution of N(0,1), where N(0,1) can indicate a normal distribution with a mean of 0 and a standard deviation of 1, ^^^ᇱare the final points generated from ^^^from the flow ^^௧, and ^^ is a parameter or term associated with the CNF. In other words, the above can be used to the train the flow model ^^௧with the adversarial loss, such that the sample generated by the flow (e.g., ^^^ᇱ) does not copy the original data, but instead distorts the generated sample by a term ^^. This can be performed to avoid memorization. For instance, even if all of the samples can be within the distribution, the peak of the distribution would still not be the exact same as the input sample because it would be displaced by ^^. Insome embodiments, ^^^ ൌ ℎ^^^^|^^௧^, which indicates that the CNF parameterized by the flow ^^௧generates a sample ^^^, starting from the sample ^^^. In this case, ℎ can be the integration operator.

[0044] To perform adversarial training, it is desired to minimize the error in the classification of the generated samples, which is described by the below expression: 1 ^ฮ^^^^ െ ℎ^^^^^|^^௧^ฮ ^^

[0045] For instance, asintegral operator that is used to obtain ^^^^from ^^^^ using the flow ^^௧and can be represented by the below expression: ^ ^^^ ൌ ^ ^^௧൫^^^^^^൯^^^^ , ^^^^^^ℎ ^^^^^ ൌ 0^ ൌ ^^^^ where t is the time. Therefore, ฮ^^^^ െ ℎ^^^^^|^^௧^ฮfrom the above can be represented by the initial expression of the loss: ℓ൫^^^^^ ൌ ℎ൫^^୧^ |^^௧൯, ^^^^൯ ൌ ฮ^^^^^ െ ^^^^ฮAnd, moving to thebe the index of each sample.Then, dividing by the total number of samples N, the loss is averaged over the input of samples. Further, ^^^^^can be the estimated sample and ^^^^can be the true sample.Attorney Docket No.820675 (Client Ref. NLE-1352-23-WO)

[0046] Thus, the below are noted: 1. The updates of the parameters of the generative flow model 112 are clipped (e.g., based on the clip and adding noise block 114): a. For each batch of size B, stochastic gradient descent (SGD) is applied for N epochs: i. Δ^ା^ ൌ ^^^^^^ே^ℓ^ℎ^^^^|^^௧^, ^^ᇱ^^,^^௧^ െ ^^௧ii. Δ^ା^ ൌ min^Δ௧ା^,^^^ ൌ Δ௧ା^ min^1,^^ / ‖Δ௧ା^‖^ with C the clipping and the minis on the norm or for each dimension b. The updates of the parameters are modified with small Gaussian noise i. ^^ ^ ൌ ଶ௧ା ^^௧ ^ Δ௧ା^ ^ ^^^0,^^ ^^^

[0047] parameters. Δ^ା^is the change in theparameters^^^ᇱ^,^^௧^ is the Stochastic Gradient Descent algorithm applied “N” times, based on the loss ℓ between the prediction ℎ^^^^|^^௧^ given the current model ^^௧with parameters ^^௧, with respect the current sample ^^^ᇱand ^^^. ^^^0,^^ଶ^^^ is a noise of variance ^^ଶand zero mean added to the update of the parameters. ‖Δ௧ା^‖ the norm of the change and ^^ is a clipping variable.

[0048] The privacy loss is described below. The ^^, δ^ differential privacy may be defined,extending previous definitions described in He, F., Fu, S., Wang, B. & Tao, D. Robustness, Privacy, and Generalization of Adversarial Training. Preprint at https: / / doi.org / 10.48550 / arXiv.2012.13573 (2020) (“He”) and Nasr, M., Shokri, R. & Houmansadr, A. Machine Learning with Membership Privacy using Adversarial Regularization. Preprint at https: / / doi.org / 10.48550 / arXiv.1807.05852 (2018) (“Nasr”), which hereby incorporated by reference herein, as ^^^^^ ∈ ^^^ െ ^^log ^^^ ^ ^^with B representing any subset anda given data sample while X' arethe generated samples, and with ^, δ being factors that are defined and described in He and Nasr.Then, according to an example, the following protocol with the below six steps may be derived.

[0049] At a first step, samples ^^ ∼ ^^^^^^^ from the prior distribution ^^ ∼ ^^^^^^^ can begenerated, where ^^ ൌ ^^^^^,^^^ is the flow generative process. As used previously, ^^ can be ^^^from the above, and ^^^^^,^^^can be ℎ^^^^|^^௧^. In some embodiments, prior to the first step, pre- training on the data can be performed. In other embodiments, pre-training might not be utilized.

[0050] For example, using the generative flow model 112, the computing platform can generate samples ^^ that can approximate to the distribution ^^^^^^^ using the flow generative process. For instance, Z and X can be random variables that are related by a mapping function ^^ (e.g., a flow generative process ^^). Thus, based on the generative flow model 112 that utilizesAttorney Docket No.820675 (Client Ref. NLE-1352-23-WO) the flow generative process ^^, the computing platform can generate samples ^^ ∼ ^^^^^^^ from aprior distribution ^^ ∼ ^^^^^^^ (e.g., a distribution obtained based on a previous iteration ofperforming the method 110 and training 104).

[0051] At a second step, given a batch of data samples ^^ ∼ ^^, with D being the givenprivate / personal dataset samples, the adversarial loss ^^^ௗ௩^^^^^^^^^^^^ can be computed using: ^^^ௗ௩^^^^^^^^^^^^ ൌ mఏin ^^^∼^బ^௭^,^ୀ^^^,ఏ^ ^^ᇲ: ‖ m^ିa^xᇱ‖ஸఋ ℓ^^^^^^,^^^,^^′^^

[0052] ^^ from the private / using the above expression, the computing platform can compute an adversarial loss for the batch of data samples ^^. For instance, the adversarial loss ^^^ௗ௩^^^^^^^^^^^^can be the loss that is minimized based on the perturbed distance. The inner term (e.g., the max term ∑^ maxℓ^^^^^^,^^^,^^′^) can indicate a different sample from the original sample that ^ᇲ: ‖^ି^ᇱ‖ஸఋ outer problem (e.g., the min term min ^^^∼^ ^௭^,^ୀ^^^) can minimize ఏబ ,ఏ^the flow parameters ^^. Therefore, the produced flow can approximate the original sample up to a distortion ^^.

[0053] At a third step, a subset of points ^^ ∼ ^^ from the dataset 102 are sampled, which canbe used to provide privacy protection.

[0054] For example, the computing platform can sample the private / personal dataset 102 again to obtain a subset of points ^^. Then, in the following (e.g., step 4), the computing platform can use the subset of points ^^ for privacy protection.

[0055] At a fourth step, an additional ^^, δ^ differential privacy loss can be defined, where^^ ∼ ^^^^^^^,^^^^ ^ ^^ ^^^ ^ ^^ ^^ଶ ^^^^^௩^^௬ ఏ ^ െ ^^^^ ା^^ఏcan be computed using the change of variable formula (e.g., as shown above, by integrating the divergence of the vector field for the continuous normalizing flow), using the determinant of the Jacobian for the normalizing flow, and / or the integral of the divergence for the continuous normalizing flow.

[0056] Thus, the privacy loss ^^^^^௩^^௬^^^,^^^ can seek to directly minimize the margin of the (end) differential privacy by computing the probability of the samples ^^ ^ ^^^^ and ^^ (e.g.,Attorney Docket No.820675 (Client Ref. NLE-1352-23-WO)^^ఏ^^^ ^ ^^^^^ and ^^ఏ^^^^). Since the log probabilities can be accessed due to the properties of theflows, the log probability of ^^^can be: ^ log ^^ఏ^^^^^ ൌ log^^^^^^^ െ ^ ∇ ⋅ ^^௧,ఏ^^^^^^^^^^where ∇ ⋅ ^^௧^^^^^ indicates the divergence of the flow ^^௧. As such, the privacy loss providesadvantages as embodiments of the present disclosure can guarantee that the privacy loss is minimized by a direct loss. For instance, the privacy loss can be computed based on the definition of the differentiable privacy, as the difference of the margin ^ and the log of the fraction of the probability of the samples with a specific sample ^^ minus the δ parameters, divided by the probability without the sample. Thus, embodiments of the present disclosure can seek to improve the δ parameters and the ^ margin using a direct loss. The above can be minimized by the gradient descent algorithm. Additionally, and / or alternatively, embodiments of the present disclosure can also implement different strategies to improve the loss, for example with simulated annealing or one or more additional and / or alternative optimization tools and / or strategies.

[0057] At a fifth step, the combination of the two losses can be optimized, with ^^ standing for proper weight of privacy: ^^௧^௧^^^^^,^^, ^^^ ൌ ^^^ௗ௩^^^^^^^^^^^^ ^ ^^ ^^^^^௩^^௬^^^,^^^

[0058] For example, after computing the adversarial loss ^^^ௗ௩^^^^^^^^at the second step and the ^^^^^௩^^௬^^^,^^^at the fourth step, the computing platform can use a scalar weight ^^ to provide a proper weight for the ^^^^^௩^^௬^^^,^^^. Then, the computing platform can compute atotal loss ^^௧^௧^^^^^,^^, ^^^ based on the adversarial loss ^^^ௗ௩^^^^^^^^, the ^^^^^௩^^௬^^^,^^^, and thescalar weight ^^.

[0059] At a sixth step, the continuous normalizing flow is trained on the data based on theconditional flow matching, where the start and end points ^^ ∼ ^^^^^^^,^^ ∼ ^^ are sampled, andthen an interpolation is generated between these two points as a function of t, such thatI^z, x, t ൌ 0^ ൌ z, I^z, x, t ൌ 1^ ൌ x, and then the regression loss is trained:^^^^^^^^^^^^^^^,^^^ ൌ ^ ‖ ^^௧^^^^^, ^^, ^^^ െ ^^^^^௧, ^^, ^^^‖where ^^^^^,^^, ^^^ is the flow and ^^௧ ൌ ^^^^^,^^^ ൌ ^௧ ^^^^^^ఛ,^^, ^^^ ^^^^, ^^^ ൌ ^^, ^^ ∈ ^0,1^ and ^^ ൌ ^^^.

[0060] For instance,mentioned above, ^^ ∼ ^^^^^^^ is the prior distribution and ^^ are the generated samples from theflow generative process. The computing platform samples the prior distribution and theAttorney Docket No.820675 (Client Ref. NLE-1352-23-WO) distribution of the samples from the flow generative process and then generates an interpolation of these two points as a function of t. Then, the computing platform computes a regression lossand uses the regression loss to train the generative flow model 112. For instance, ^^^^^, ^^, ^^^ canbe an interpolation function, and can be represented by ^^^^^, ^^, ^^^ ൌ ^^^^ ^ ^1 െ ^^^^^. Theregression loss can be enforced such that at a specific time t, and the standing provides the two samples. Further, the vector field ^^௧can be closed to the linear interpolation where the flow is given by the ^^௧^^ for the interpolation model.

[0061] The efficient computation is described below. For instance, using normalizing flow or continuous normalizing flow, it can be efficient to compute the determinant of the Jacobian, and respectively the divergence of the flow, and thus compute the probability of the generated point. For example, the normalizing flow is given by the change of variable formula (as mentioned in step 4 above). This is shown below. ^^^^^^^^^^ି^^^^^^^^^ ൌ ^^^൫^^ି^^^^^൯ห^^൫^^ି^^^^^൯ห, ^^൫^^ି^^^^^൯ ൌ ^^^^^ൌ^^^^, |^^| ൌ |^^^^^^^^^^|isgenerated by a transformation defined by the function ^^. ^^൫^^ ^^ ൯ can be the determinant of the Jacobian ^^, which can equal ^^ ^^ି^^^^^. |^^| is the of the matrix A, and thisbe for the meaning of the formula elements. The standard normalizing flows are discrete in time such that the above expression for the change in distribution is used.

[0063] Discrete gradient for categorical data is now described. According to one or more embodiments, it may be provided to consider the use of a discrete variable for categorical data, where the gradient is estimated with the Gumbel Softmax trick, which is described in Paulus, M., Choi, D., Tarlow, D., Krause, A. and Maddison, C.J., 2020. Gradient estimation with stochastic softmax tricks. Advances in Neural Information Processing Systems, 33, pp.5691- 5704 and is incorporated by reference herein. For instance, if the data is represented as integer, the computing platform can consider using the straight through gradient estimation.

[0064] For example, the computing platform can use a discrete variable for categorical data where the gradient can be estimated using the Gumbel Softmax approach. Additionally, and / or alternatively, based on the data being represented as an integer, the computing platform can consider the straight-through gradient estimation.

[0065] FIG.2 is an example of an implementation of the straight-through gradient estimation for integer variables used for the categorical case in accordance with an embodiment of the present disclosure. For instance, FIG.2 shows code 200 for the straight-through gradient estimation for integer variables that can be used for the categorical case. In some examples, the code 200 can be implemented in a programming language, such as PYTORCH.Attorney Docket No.820675 (Client Ref. NLE-1352-23-WO)

[0066] Technical examples are described below. For instance, in one or more embodiments, alternative approaches can use a diffusion model and / or a variational autoencoder (VAE). In some embodiments, the diffusion model and VAE can have disadvantages since there is not an easy way to compute the change in phase volume and thus no “exact” probability can be computed. This is most likely either computational prohibitive or not possible (e.g., it can be difficult for the diffusion model to perform this efficiently and accurately).

[0067] In one or more embodiments (e.g., a simplified version of one or more embodiments descried above), the computing platform can use and apply the adversarial loss (e.g., only use the adversarial loss described in step 2 above) to discrete score matching models, since it might not possible to easily compute the change in distribution. Discrete score matching models can be described in Meng, C., Choi, K., Song, J. and Ermon, S., 2022. Concrete score matching: Generalized score matching for discrete data. Advances in Neural Information Processing Systems, 35, pp.34532-34545, which is incorporated by reference herein.

[0068] Conditional Generation with diffusion models or conditional continuous normalizing flow models is described below. One advantage of diffusion models is that it is possible to train a generic and unconditional model that models the probability p(x), and then conditional data can be generated that has access to a classification model p(y|x). The conditional generative model is p(x|y) in such instances. The classification model is used during generation to guide the generation towards the conditional distribution p(x|y) rather than p(x).

[0069] A similar approach can be used for conditional continuous normalizing flow. For instance, examples (e.g., samples) that are drawn from the conditional distribution p(x|y) for the specific class to guide the generation process can be close to the examples that can be used by the computing platform. The computing platform can obtain this by using samples from the prior distribution and then enforcing the initial velocity in the direction of the examples. For example, if ^^^is the sample from the prior distribution ^^^^^^^, and ^^^is the sample from p(x|y), with yfixed to the target class, then ^^^௧ୀ^^ ൌ ^^^ ⋀ ^^^^௧ୀ^^ ൌ ^^^^^^ െ ^^^^ can be set with a scalingparameter ^^.

[0070] The classification and regression constraints with predictive loss is described below. For instance, while seeking to preserve privacy of the starting dataset D (e.g., the private and personal dataset 102), it can also be desired that the generated dataset preserves the relationship among features such that it is possible to train classification and regression models on top (e.g., downstream tasks).

[0071] For instance, FIG.3 is an overview 300 of performing a downstream prediction task (e.g., a plurality of prediction tasks such as prediction tasks 312 through 314 using different privacy datasets such as privacy datasets 308-310) using a generative flow 304 and a differentialAttorney Docket No.820675 (Client Ref. NLE-1352-23-WO) privacy flow model 306 according to an embodiment of the present disclosure. For instance, in order to make the generated dataset be able to be trained for predictive tasks, a prediction loss can be computed. For instance, to perform a plurality of prediction tasks (e.g., the first prediction task 312 (prediction task 1) through the final prediction task 314 (prediction task N)), the computing platform can train the generative flow 304 using a prediction loss to obtain the differential privacy flow model 306. As such, the overview 300 shows a generative flow 304 that can be trained similar to the overview 100 above (e.g., the private and personal dataset 302, the differential privacy flow model 306, and the privacy datasets 308 (privacy dataset 1) through the final privacy dataset 310 (privacy dataset N) can be similar to the private and personal dataset 102, the differential privacy flow model 118, and the plurality of privacy datasets from FIG.1) except for the generative flow 304 is further trained utilizing a prediction loss that is based on the prediction tasks 312-314. As such, the end result, the differential privacy flow model 306 that is used to generate the privacy datasets such as the datasets 308-310 is a model 306 that is trained using the prediction loss along with the adversarial and / or privacy loss described above. The training of the generative flow 304 using the prediction loss is described in FIG.4.

[0072] FIG.4 is an overview 400 of computing a predictive task loss for training generative and privacy preserving flow according to an embodiment of the present disclosure. For instance, the overview 400 also includes a private and personal dataset 402, a differential privacy flow model 418, and a plurality of privacy datasets (e.g., privacy dataset 420 (privacy dataset 1) and privacy dataset 422 (privacy dataset N)), which can be similar to FIG.1 described above. However, the training of the generative flow 416 can further utilize blocks 404-414, which can compute and use a predictive loss for training the generative flow 416.

[0073] For instance, since it might not be known, a priori, which regression or classification tasks are to be implemented on top of the data, in an example, a class of possible classification and regression tasks can be considered and a loss function can be added during training (e.g., training of the generative flow 416 to obtain the differential privacy flow model 418) that preserves the capacity.

[0074] Thus, the computing platform can consider the generic classification and regression task (or mixed) where the data’s features are split into two sets: 1) the input set and 2) the output set. Depending on the output, the computing platform can obtain a) a classification task based on the target being discrete, b) a regression based on the features being continuous, and c) a mixed problem otherwise. For instance, at 404, the computing platform can split features into input / output sets. For example, the computing platform can obtain features from the private and personal dataset 402, and split the features into an input set and an output set.Attorney Docket No.820675 (Client Ref. NLE-1352-23-WO)

[0075] If there are only few features, the computing platform can create the set of all prediction tasks (e.g., regression, classification or mixed), but in some embodiments, this can become too complex. Thus, in an embodiment, the computing platform can consider a subset of all possible predictive tasks (e.g., the discrete and continuous features are first split, and then subsets of the two classes are created). For each subset, the computing platform generates a prediction task and trains on the original data for the prediction task. Then, the loss during the training of the predictive task can be used during the training of the generative flow 416 to obtain the differential privacy flow model 418.

[0076] For instance, at 406, the computing platform performs training of the predictive task. For example, using the split features from 404 and the original data (e.g., from the private and personal dataset 402), the computing platform trains the predictive tasks and obtains the trained predictive tasks 408. The losses during the training of the predictive task can be used to train the generative flow 416 to obtain the differential privacy flow model 418.

[0077] For example, assuming that ^^^^,^^^^ is the dataset generated by splitting the feature into input (^^^) and output (^^^) (e.g., at 404), then the computing platform trains a model ^^^: ^^^→ ^^^(e.g., at 406) and the loss for each of these trained models is added, with the total loss isdescribed above (e.g., the total loss that is computed based on the adversarial and described above in steps 2, 4, and 5). This is shown in the below expression: ^^^௨^^^^௧^ௗ^^^,^^, ^^, ^^, ^^^^ ,^^^ ,^^_^^^^ ൌ ^^௧^௧^^^^^,^^, ^^^ ^ ^^^ ^^ ^^^ௗ^^௧^௩^^ ^^^^ ,^^^ ,^^^^402 into an input (^^^) and an output (^^^). Then, at 406, the computing platform trains a model (e.g., predictive task) ^^^using the input set and the output set, and at 408, obtains the trained models as well as the losses from the trained models / predictive tasks. At 412, the computing platform computes the losses for each of these trained models and also computes the total loss based on the above. For example, the computing platform computes a predictive loss ^^^^^^ௗ^^௧^௩^^^^^,^^^,^^^^, and aggregates the predictive loss for all of the models ^^^(e.g.,a scalar weight ^^ and the total loss, the computing platform computes an augmented loss ^^^௨^^^^௧^ௗ. Following, using the augmented loss, the computing platform performs training 414 of the generative flow 416. In addition, noise 410 can be provided to the generative flow 416.

[0079] In some embodiments, the computing platform can compute the predictive loss ^^^^^^ௗ^^௧^௩^^^^^,^^^,^^^^ using the 2 norm process. For instance, with ^^^trainable functions and where the computing platform can use a loss function for each prediction task, the computingAttorney Docket No.820675 (Client Ref. NLE-1352-23-WO) platform can use the 2 norm as the predictive loss for the regression task. This is described by the expression below: ^^^ௗ^^௧^௩^^1 ^^^ ^^^ ,^^^ ,^^^^ ൌ^^^ฮ^^^,^ െ ^^^൫^^^,^൯ฮ^ ଶ^ with ^^ ,^,^^^^,^, the input

[0080] In other embodiments, the computing platform can use other regression losses such as, but not limited to, mean square error (MSE), root mean square error (RMSE), and / or mean absolute error (MAE). For classification tasks, the computing platform can use the cross-entropy loss or any other classification loss (e.g., Hamming distance).

[0081] The privacy verification is described below. For instance, while a theoretical guarantee that the privacy is preserved is described above, the computing platform can provide an additional a verification step, where the privacy of the generative model and the dataset generated is maintained. This is described in FIG.5.

[0082] For example, FIG.5 is an overview 500 of verification of the privacy level according to an embodiment of the present disclosure. For instance, after training, the privacy verification 510 is performed on the differential privacy flow model 506 and the privacy dataset 508 (privacy dataset i). Based on the privacy not being maintained, then, the computing platform further trains the generative flow 504. For instance, the private and personal dataset 502, the training of the generative flow 504, the differential privacy flow model, and a privacy dataset 508 (privacy dataset i) can be similar to the elements described above. But, the computing platform can perform privacy verification 510 based on the data from the private and personal dataset 502 and / or data from the privacy dataset 508 to verify that the privacy is maintained. Based on the verification, the computing platform can determine whether to retrain the differential privacy flow model 506 using the generative flow 504.

[0083] For example, the loop shown in FIG.5 can be implemented also end-to-end where the gradient is obtained on the verification loss, which is shown in the below expression: ^^ఏ^^^^ ൌ min^^^^^ ^^,^^ ^ ^^^^where ^^ ൌ ^^ఏ,థ^^^^ aremodel 504 and thecomputing platform seeks to maximize the minimum distance, which is described by the below expression: max థ^^ఏ,థ^^^^ ൌ mథaxmin ∈^ ^^^^^^^^ , ^^^^^^ ^ where ^^,^^ are two setthe other to regulate the privacy, as for example the level of noise.Attorney Docket No.820675 (Client Ref. NLE-1352-23-WO)

[0084] For instance, the verification loss attempts to maximize the minimum distance between the generative model (^^^ and the original dataset to further impose the privacy.

[0085] The graph data generation is described below. For instance, graphs are composed of edges and vertices. Edges and vertices can have attributes that can be either continuous or discrete. A generative model can first generate the structure (the edges) given the number of nodes and then generate the attributes on the edges or nodes. Furthermore, both processes (e.g., generating the structure and then the attributes) can be modeled using continuous or discrete flows, and trained similarly.

[0086] For instance, the generative model can be composed of two models (or flows) with the first generating the structure given the number of nodes (e.g., the first model can decide whether an edge is present or not). Then, based on the output of the first model, the second model, conditioned to the graph structure, generates the attributes. The flow itself can be modeled as a graph or as a transformer. For example, when generating graph data, the graph itself can be generated, including generating the features attached to the nodes and edges depending on the dataset. In some examples, embodiments of the present disclosure can divide the graph generation steps. This is described in FIG.6.

[0087] FIG.6 is an overview 600 of graph generative flow according to an embodiment of the present disclosure. The overview 600 includes the components 102-120 from FIG.1. In addition, the differential privacy flow model 118 includes the two models described above. For instance, the first model 602 is an edges flow that generates the structure (e.g., the edges) given the number of nodes. Then, the second model 604 is a nodes and edges attributes flow that generates the attributes.

[0088] In summary, one or more embodiments of the present disclosure can describe PrivacyFlow, which can be a method and system that is used for training and generating synthetic data using (continuous) normalizing flow in accordance with one or more embodiments described herein. For instance, one or more embodiments can describe a method to train a model that generates differential privacy preserving dataset. The model itself can be used to generate multiple datasets.

[0089] Three embodiments are described below. In the first embodiment, synthetic EHR record generation is described (e.g., as in European project ACCELERATE where Hospitals give access under restrictive condition to the EHR). In the second embodiment, synthetic Single / Bulk cell RNA Sequencing data generation is described (e.g., either start from single cell or bulk RNA sequencing data or from genetic string). In the third embodiment, clinical trials are described (e.g., virtual patients and data is generated to test the expected result).Attorney Docket No.820675 (Client Ref. NLE-1352-23-WO)

[0090] For instance, embodiments of the present disclosure can be used for synthetic Electronic Health Record (EHR) generation. For example, for a use case, one of the fundamental problems in management of Health is to maintain privacy when dealing with critical personal data. EHR includes different information regarding patients, which are typically stored in DATA LAKE, e.g., the database of the Hospital. EHR include demographic (e.g., age, sex, and so on), clinical (e.g., lab tests, biological signals monitoring), and can be of different natures (e.g., categorical, continuous). While strict access is implemented for the access of this data, this causes major obstacles when such data needs to be aggregated for training large models for the development and studies that are beneficial to advance possible cures. In this case, it is ideal to have a large set of synthetic data that can be generated starting by the repository, and then gathered on a specific server (which can be for example on cloud premises), and download from anyone (research institution, company) who needs to train the artificial intelligence (AI) algorithm. The data source for such embodiments can be a dataset of EHR data and the parameter for the privacy. The method can include, according to the above, a training to samples from the privacy dataset, where the information can include data that is either private or sensitive. The user can set the privacy level allowed and then is able to verify the level of privacy and the reconstruction error. More concretely, once the dataset for a specific use case (e.g., prediction of cardiovascular disease development) has been decided, the synthetic dataset is generated and a downstream predictive model is trained on it. The output can be the generative privacy model and / or specific anonymized datasets, together with a trained downstream ML model for predicting the outcome of interest.

[0091] Embodiments of the present disclosure can also be used for generating synthetic genetic data, genetic expression data, and molecular data. For instance, a use case can include Single / Bulk cell Sequencing data generation and Gene data privacy dataset. The dataset can also be represented as expression graph. The data source can be a data set that includes the genetic information, either in form of genetic expression (e.g., single cell sequencing RNA or bulk cell sequencing RNA) or directly genetic information, either using nucleotide or three- dimensional (3D) structure. Molecular data (including biological data) is characterized by number of atoms, atom type, presence, and type of bond and for 3D data the coordinates. Lastly, some properties can be measured from experiment of the molecules (e.g., binding energy, mechanical and thermo-dynamical properties, or toxicity). This information can be either continuous or categorical. The synthetic data may be used in conjunction with downstream tasks to predict for example the cell composition (deconvolution). The method can include, according to the concepts disclosed herein a training to generate the data and preserve differential privacy using one or many of these data. The data may be trained to generate the source either isolatedAttorney Docket No.820675 (Client Ref. NLE-1352-23-WO) or jointly. The output can be a deep network for generating the anonymized data or directly the anonymized dataset, coupled with a deconvolution algorithm to reconstruct the cell type distribution of the patients of interest.

[0092] Embodiments of the present disclosure can also be used for a dataset for pre-trained large multimodal language model (LLM) for generic agent and for clinical data. The use case can include training data for LLMs as this is a critical component. For instance, LLMs are now ubiquitous, but the use of private dataset is a big issue, either because the data includes privacy related information (e.g., medical data as report from physicians or personal information as tax declarations) or because the data include business critical data (e.g., company information). Use of anonymized data is key to be able to train models without violation of privacy and confidentiality. In this use case, the network can be trained to generate the data samples (e.g., sentences from medical report for example) from noise. The network can be based on the transformer or its variant such that it does not depend on the size of the input. The data source can include a data set including text and private messages and other language source, but also images and / or audio data. Using social data (e.g., information datasets) to train LLM has ethical and privacy implications that can both limit the prosecution of the project due to the impossibility of anonymize the information easily. In this context, a system that generates safe synthetic documents can be used for LLM training. The method can include, according to the concepts disclosed herein, a training to generate the data and preserve differential privacy. The output can include a deep network for generating the multimodal LLM or / and the anonymized dataset.

[0093] In an embodiment, the present disclosure provides a system and method for 1) learning an end-to-end privacy preserving trainable flow-based machine-learning model (e.g., the differential privacy flow model 118). The model described herein (e.g., the differential privacy flow model 118) is able to generate multiple datasets (e.g., the datasets 120-122) that guarantee the differential privacy. Further, the flow generative model (e.g., the differential privacy flow model 118) is the main output that can generate multiple datasets (e.g., the datasets 120-122). In addition, directly minimizing for the privacy loss using the continuous normalizing flow can be performed (e.g., using the adversarial and privacy loss 116 during the method 110), which allows to compute the probability distribution (up to constant term). 2) Including discrete-continuous gradient estimator, such that the method can effectively model both continuous and categorical data. 3) Defining a predictive loss to keep the performance of future downstream predictive tasks. For instance, contrary to standard generative models that try to reproduce the dataAttorney Docket No.820675 (Client Ref. NLE-1352-23-WO) but not to maintain specific relationships, the computing platform can explicitly add in the loss during training, which helps preserve these relationships (cause / effect).

[0094] Embodiments of the present disclosure provide methods and systems for developing a fully differentiable deep flow network and its training, comprising one or more of the following steps / components: 1) Collecting a dataset (e.g., the dataset 102) including private or personal data; 2) Choosing a normalizing flow model (e.g., the generative flow model 112) and the parameters of the loss and continuous and / or discrete gradient estimator; 3) training, using PrivacyFlow method 110, via backpropagation to optimize the expected accuracy error of the adversarial and privacy loss (e.g., the adversarial and privacy losses 116) and using categorical gradient for the categorical variables. The output is the trained flow model (e.g., the differential privacy flow model 118) that can generate new synthetic dataset that can be used to train a downstream task, as the deconvolution.

[0095] Embodiments of the present disclosure provide for the following improvements and technical advantages over existing technology: 1) guaranteeing of the privacy by monitoring the loss during training and testing, and 2) a possible infinite generative model that can generate an arbitrary number of different datasets.

[0096] The following is an example using the UCI (University of California Irvine Machine Learning Repository) dataset and the generated samples (Y) resampled without copying the original samples (X).

[0097] X = tensor([[ 2.00, 15.00, 0.46, 0.36, 0.10, 0.51, 0.22, 0.10, 0.15], [ 2.00, 7.00, 0.35, 0.27, 0.09, 0.23, 0.10, 0.05, 0.07], [ 0.00, 9.00, 0.53, 0.42, 0.14, 0.68, 0.26, 0.14, 0.21], [ 2.00, 10.00, 0.44, 0.36, 0.12, 0.52, 0.22, 0.11, 0.15], [ 1.00, 7.00, 0.33, 0.26, 0.08, 0.20, 0.09, 0.04, 0.06], [ 1.00, 8.00, 0.42, 0.30, 0.10, 0.35, 0.14, 0.08, 0.12], [ 0.00, 20.00, 0.53, 0.41, 0.15, 0.78, 0.24, 0.14, 0.33], [ 0.00, 16.00, 0.55, 0.42, 0.12, 0.77, 0.29, 0.15, 0.26], [ 2.00, 9.00, 0.47, 0.37, 0.12, 0.51, 0.22, 0.11, 0.17], [ 0.00, 19.00, 0.55, 0.44, 0.15, 0.89, 0.31, 0.15, 0.32]] Y = tensor([[ 1.00, 10.00, 0.75, 0.42, -0.11, 0.85, 0.13, 0.18, 0.18], [ 2.00, 9.00, 0.68, 0.57, 0.33, 0.65, 0.44, 0.25, 0.21], [ 1.00, 9.00, 0.51, 0.06, -0.04, 0.94, 0.29, 0.35, 0.28], [ 0.00, 12.00, 0.74, 0.46, 0.20, 0.78, 0.33, 0.53, 0.45],Attorney Docket No.820675 (Client Ref. NLE-1352-23-WO) [ 2.00, 14.00, 0.37, 0.23, 0.22, 0.86, 0.17, 0.19, 0.42], [ 1.00, 8.00, 0.71, 0.18, -0.29, 0.41, 0.65, 0.06, 0.36], [ 0.00, 11.00, 0.58, 0.66, 0.27, 0.82, 0.46, 0.04, 0.35], [ 0.00, 9.00, 0.62, 0.27, -0.25, 0.78, -0.01, 0.53, 0.47], [ 1.00, 12.00, 0.85, 0.75, 0.11, 1.37, 0.72, 0.16, 0.19], [ 1.00, 13.00, 0.72, 0.39, 0.18, 1.41, 0.58, 0.46, 0.40]] Minimum distance from original to original and synthetic dataset.

[0098] Embodiments of the present disclosure thus provide for general improvements to computers in machine learning systems to generate synthetic data. Moreover, as described above, embodiments of the present disclosure can be practically applied to use cases to effect further improvements in a number of technical fields including, but not limited to, medical and healthcare (e.g., digital medicine, personalized healthcare, drug or vaccine development, prescription, treatment, etc.), and / or other technical fields that include and / or use sensitive and / or personal data.

[0099] FIG.7 is a comparison 700 of the histograms of the minimum (min) distance of each sample from the original dataset to either the original dataset (excluding the point itself) or the synthetic dataset of the same dimension. For instance, the counts are shown for the histograms. The comparison is between a minimum distance between a sample from the original dataset to the original dataset or between a sample from the original dataset to the synthetic dataset. This was based on training the differential privacy flow model 118 based on setting the training to 400 epochs. For example, the histogram within the dotted shape shows the minimum distance of each sample from the original dataset to the synthetic dataset. The histogram within the rectangle shows the minimum distance of each sample from the original dataset to the original dataset. As shown, the entries for the histogram for the original dataset to the synthetic dataset has a wider width than the entries for the histogram for the original dataset to original dataset, and this is also shown by the overlap between the two histograms.

[0100] FIG.8 is another comparison 800 of the histograms of the minimum (min) distance of each sample from the original dataset to either the original dataset (excluding the point itself) or the synthetic dataset of the same dimension. In contrast to FIG.7, this comparison was performed after training the differential privacy flow model 118 for 2000 iterations. The structure is similar to FIG.7, with minimal overlap in the histogram. For example, the histogram within the dotted shape shows the minimum distance of each sample from the original dataset to the synthetic dataset. The histogram within the rectangle shows the minimum distance of each sample from the original dataset to the original dataset. As shown, the entries for theAttorney Docket No.820675 (Client Ref. NLE-1352-23-WO) histogram for the original dataset to the synthetic dataset has a wider width than the entries for the histogram for the original dataset to original dataset, and this is also shown by the overlap between the two histograms.

[0101] Referring to FIG.9, a processing system 900 can include one or more processors 902, memory 904, one or more input / output devices 906, one or more sensors 908, one or more user interfaces 910, and one or more actuators 912. Processing system 900 can be representative of each computer system disclosed herein.

[0102] Processors 902 can include one or more distinct processors, each having one or more cores. Each of the distinct processors can have the same or different structure. Processors 902 can include one or more central processing units (CPUs), one or more graphics processing units (GPUs), circuitry (e.g., application specific integrated circuits (ASICs)), digital signal processors (DSPs), and the like. Processors 902 can be mounted to a common substrate or to multiple different substrates.

[0103] Processors 902 are configured to perform a certain function, method, or operation (e.g., are configured to provide for performance of a function, method, or operation) at least when one of the one or more of the distinct processors is capable of performing operations embodying the function, method, or operation. Processors 902 can perform operations embodying the function, method, or operation by, for example, executing code (e.g., interpreting scripts) stored on memory 904 and / or trafficking data through one or more ASICs. Processors 902, and thus processing system 900, can be configured to perform, automatically, any and all functions, methods, and operations disclosed herein. Therefore, processing system 900 can be configured to implement any of (e.g., all of) the protocols, devices, mechanisms, systems, and methods described herein.

[0104] For example, when the present disclosure states that a method or device performs task “X” (or that task “X” is performed), such a statement should be understood to disclose that processing system 900 can be configured to perform task “X”. Processing system 900 is configured to perform a function, method, or operation at least when processors 902 are configured to do the same.

[0105] Memory 904 can include volatile memory, non-volatile memory, and any other medium capable of storing data. Each of the volatile memory, non-volatile memory, and any other type of memory can include multiple different memory devices, located at multiple distinct locations and each having a different structure. Memory 904 can include remotely hosted (e.g., cloud) storage.

[0106] Examples of memory 904 include a non-transitory computer-readable media such as RAM, ROM, flash memory, EEPROM, any kind of optical storage disk such as a DVD, a Blu-Attorney Docket No.820675 (Client Ref. NLE-1352-23-WO) Ray® disc, magnetic storage, holographic storage, a HDD, a SSD, any medium that can be used to store program code in the form of instructions or data structures, and the like. Any and all of the methods, functions, and operations described herein can be fully embodied in the form of tangible and / or non-transitory machine-readable code (e.g., interpretable scripts) saved in memory 904.

[0107] Input-output devices 906 can include any component for trafficking data such as ports, antennas (i.e., transceivers), printed conductive paths, and the like. Input-output devices 906 can enable wired communication via USB®, DisplayPort®, HDMI®, Ethernet, and the like. Input-output devices 906 can enable electronic, optical, magnetic, and holographic, communication with suitable memory 904. Input-output devices 906 can enable wireless communication via WiFi®, Bluetooth®, cellular (e.g., LTE®, CDMA®, GSM®, WiMax®, NFC®), GPS, and the like. Input-output devices 906 can include wired and / or wireless communication pathways.

[0108] Sensors 908 can capture physical measurements of environment and report the same to processors 902. User interface 910 can include displays, physical buttons, speakers, microphones, keyboards, and the like. Actuators 912 can enable processors 902 to control mechanical forces.

[0109] Processing system 900 can be distributed. For example, some components of processing system 900 can reside in a remote hosted network service (e.g., a cloud computing environment) while other components of processing system 900 can reside in a local computer system. Processing system 900 can have a modular design where certain modules include a plurality of the features / functions shown in FIG.9. For example, I / O modules can include volatile memory and one or more processors. As another example, individual processor modules can include read-only-memory and / or local caches.

[0110] The following are also incorporated by reference herein in their entirety: 1) Alaa, Ahmed, Boris Van Breugel, Evgeny S. Saveliev, and Mihaela van der Schaar. "How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models." In International Conference on Machine Learning, pp.290-306. PMLR, 2022 (“Alaa”). 2) U.S. Application No.17 / 782,551, titled “Generating Synthetic Patient Heath Data” filed December 4, 2020 (“the ’551 Application”). 3) Ping, H., Stoyanovich, J., & Howe, B. (2017). DataSynthesizer: Privacy-Preserving Synthetic Datasets. Proceedings of the 29th International Conference on Scientific and Statistical Database Management (“Ping”).Attorney Docket No.820675 (Client Ref. NLE-1352-23-WO) 4) Liu, Fan & Cheng, Zhiyong & Wei, Yinwei & Nie, Liqiang & Kankanhalli, Mohan. (2022). Privacy-Preserving Synthetic Data Generation for Recommendation Systems. 10.48550 / arXiv.2209.13133 (“Liu”).

[0111] For example, the above four references do not train a flow based generative model that is based on the losses described above. For instance, Alaa proposes an evaluation pipeline to measure if the sample is contained in the synthetic dataset, but does not describe a generative model. The ’551 Application describes a generative adversarial model (generative adversarial network (GAN)) architecture that is not flow based. Ping describes a data synthesizer for tabular data based on high-level statistical description of the features distribution. Each feature of Ping is generated independently with a selection process based on maximal mutual information with the source data. The model of Ping can only work when the original data is available. Liu describes a specialized model based on GAN for temporal data (e.g., sequential data).

[0112] While subject matter of the present disclosure has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. Any statement made herein characterizing the invention is also to be considered illustrative or exemplary and not restrictive as the invention is defined by the claims. It will be understood that changes and modifications can be made, by those of ordinary skill in the art, within the scope of the following claims, which can include any combination of features from different embodiments described above.

[0113] The terms used in the claims should be construed to have the broadest reasonable interpretation consistent with the foregoing description. For example, the use of the article “a” or “the” in introducing an element should not be interpreted as being exclusive of a plurality of elements. Likewise, the recitation of “or” should be interpreted as being inclusive, such that the recitation of “A or B” is not exclusive of “A and B,” unless it is clear from the context or the foregoing description that only one of A and B is intended. Further, the recitation of “at least one of A, B and C” should be interpreted as one or more of a group of elements consisting of A, B and C, and should not be interpreted as requiring at least one of each of the listed elements A, B and C, regardless of whether A, B and C are related as categories or otherwise. Moreover, the recitation of “A, B and / or C” or “at least one of A, B or C” should be interpreted as including any singular entity from the listed elements, e.g., A, any subset from the listed elements, e.g., A and B, or the entire list of elements A, B and C.

Claims

Attorney Docket No.820675 (Client Ref. NLE-1352-23-WO) CLAIMS What is claimed is:

1. A computer-implemented method for developing a differential privacy model, comprising: collecting a private and personal dataset comprising private and / or personal data; training the differential privacy model via backpropagation to optimize an expected accuracy of an adversarial loss and a privacy loss, wherein the differential privacy model is associated with a continuous normalizing flow; and outputting the trained differential privacy model, wherein the trained differential privacy model is configured to generate new synthetic datasets that are used to train one or more downstream tasks.

2. The computer-implemented method of claim 1, further comprising: prior to training the differential privacy model, choosing the continuous normalizing flow and loss parameters associated with the adversarial loss and the privacy loss.

3. The computer-implemented method of claim 2, wherein training the differential privacy model is based on using a categorical gradient estimator for categorical variables associated with the private and personal dataset, and wherein the method further comprises choosing categorical gradient parameters associated with the categorical gradient estimator.

4. The computer-implemented method of any of the preceding claims, further comprising: using the trained differential privacy model to generate the new synthetic datasets that guarantee differential privacy based on using a verification loss.

5. The computer-implemented method of claim 4, wherein the verification loss is computed based on comparing data from a new synthetic dataset from the synthetic datasets and original data from the private and personal dataset.

6. The computer-implemented method of any of the preceding claims, wherein training the differential privacy model comprises minimizing the privacy loss using the continuous normalizing flow that allows for computation of a probability distribution up to a constant term.

7. The computer-implemented method of any of the preceding claims, further comprising: defining a predictive loss associated with the one or more downstream tasks, and wherein training the differential privacy model is based on using the predictive loss.

8. The computer-implemented method of claim 7, wherein defining the predictive loss comprises: splitting features from the private and personal dataset into an input set and an output set; training the one or more downstream tasks based on the input set and the output set, wherein the one or more downstream tasks comprises one or more downstream models; andAttorney Docket No.820675 (Client Ref. NLE-1352-23-WO) obtaining the predictive loss based on the one or more trained downstream tasks.

9. The computer-implemented method of any of the preceding claims, wherein the differential privacy model comprises: a first model for generating a structure of a graph based on a number of nodes; and a second model for generating attributes associated with the structure of the graph.

10. The computer-implemented method of any of the preceding claims, wherein the trained differential privacy model is used in a biomedical field for electronic healthcare data.

11. The computer-implemented method of any of the preceding claims, wherein the private and / or personal data is associated with electronic health record (EHR) data, and wherein the new synthetic datasets generated by the trained differential privacy model comprises synthetic EHR data.

12. A computer system for developing a differential privacy model, the computer system comprising one or more hardware processors, which, alone or in combination, are configured to provide for execution of the following steps: collecting a private and personal dataset comprising private and / or personal data; training the differential privacy model via backpropagation to optimize an expected accuracy of an adversarial loss and a privacy loss, wherein the differential privacy model is associated with a continuous normalizing flow; and outputting the trained differential privacy model, wherein the trained differential privacy model is configured to generate new synthetic datasets that are used to train one or more downstream tasks.

13. The computer system of claim 12, wherein the one or more hardware processors, which, alone or in combination, are further configured to provide for execution of the following steps: prior to training the differential privacy model, choosing the continuous normalizing flow and loss parameters associated with the adversarial loss and the privacy loss.

14. The computer system of claim 13, wherein training the differential privacy model is based on using a categorical gradient estimator for categorical variables associated with the private and personal dataset, and wherein the one or more hardware processors, which, alone or in combination, are further configured to provide for execution of the following steps: choosing categorical gradient parameters associated with the categorical gradient estimator.

15. The computer system of any of claims 12-14, wherein the one or more hardware processors, which, alone or in combination, are further configured to provide for execution of the following steps:Attorney Docket No.820675 (Client Ref. NLE-1352-23-WO) using the trained differential privacy model to generate the new synthetic datasets that guarantee differential privacy based on using a verification loss.

16. The computer system of claim 15, wherein the verification loss is computed based on comparing data from a new synthetic dataset from the synthetic datasets and original data from the private and personal dataset.

17. A tangible, non-transitory computer-readable medium having instructions thereon which, upon being executed by one or more processors, alone or in combination, provide for execution of a method for developing a differential privacy model, the method comprising the following steps: collecting a private and personal dataset comprising private and / or personal data; training the differential privacy model via backpropagation to optimize an expected accuracy of an adversarial loss and a privacy loss, wherein the differential privacy model is associated with a continuous normalizing flow; and outputting the trained differential privacy model, wherein the trained differential privacy model is configured to generate new synthetic datasets that are used to train one or more downstream tasks.

Images