Private synthetic time series generation

Abstract

A method and system for generating synthetic data is provided, in which longitudinal time series data is retrieved and a neural network is trained to generate synthetic time series data that meets a privacy metric based on the longitudinal time series data, where the longitudinal time series data is unlabeled and univariate.

Description

[Technical Field]

[0001] The present disclosure relates to data processing systems, and more particularly, to private synthetic time series data generation for data processing systems. Summary of the Invention [Problem to be solved by the invention]

[0002] Sharing longitudinal patient medical time series data can enable improved treatment development and technological advances. For example, sharing time series data of measured analytes of patients can contribute to understanding relevant disease mechanisms and developing technologies to improve the quality of life of these patients. Of course, there are serious legal and privacy issues that arise when sharing longitudinal patient medical time series data, such as those outlined by the Health Insurance Portability and Accountability Act of 1996 (known as HIPAA). [Brief explanation of the drawings]

[0003] [Figure 1] FIG. 1 is a block diagram of an exemplary system for generating synthetic data, according to an embodiment of the present disclosure. [Figure 2] 1 is an exemplary artificial neural network (ANN), according to an embodiment of the present disclosure. [Figure 3A] 1A-1C are different diagrams of an exemplary recurrent neural network (RNN), according to embodiments of the present disclosure. [Figure 3B] 1A-1C are different diagrams of an exemplary recurrent neural network (RNN), according to embodiments of the present disclosure. [Figure 3C]1A-1C are different diagrams of an exemplary recurrent neural network (RNN), according to embodiments of the present disclosure. [Figure 3D] 1A-1C are different diagrams of an exemplary recurrent neural network (RNN), according to embodiments of the present disclosure. [Figure 3E] FIG. 10 illustrates an exemplary data flow diagram for a hidden recurrent module, according to an embodiment of the present disclosure. [Figure 4A] FIG. 1 is a diagram of an exemplary long short-term memory (LSTM) network, according to an embodiment of the present disclosure. [Figure 4B] FIG. 10 is an exemplary data flow diagram for an LSTM cell, according to an embodiment of the present disclosure. [Figure 4C] FIG. 10 is an exemplary data flow diagram for an LSTM cell, according to an embodiment of the present disclosure. [Figure 5] FIG. 1 illustrates an exemplary dataflow diagram for a differential-privacy generative adversarial network (DP-GAN), according to an embodiment of the present disclosure. [Figure 6] FIG. 6 is an exemplary loss function diagram for training the DP-GAN shown in FIG. 5 , according to an embodiment of the present disclosure. [Figure 7] FIG. 6 is an exemplary data flow diagram for generating batched raw data for training the DP-GAN shown in FIG. 5, according to an embodiment of the present disclosure. [Figure 8A] FIG. 6 is an exemplary data flow diagram for generating synthetic data by the DP-GAN shown in FIG. 5, according to an embodiment of the present disclosure. [Figure 8B] FIG. 6 is an exemplary data flow diagram for generating synthetic data by the DP-GAN shown in FIG. 5, according to an embodiment of the present disclosure. [Figure 9A] FIG. 10 is an exemplary data flow diagram for a motif causality module, according to an embodiment of the present disclosure. [Figure 9B] FIG. 9B is an exemplary data flow diagram for generating motif sequence blocks for training the motif causality module depicted in FIG. 9A according to an embodiment of the present disclosure. [Figure 10A] 9B depicts a data flow diagram for a motif network within the motif causality module depicted in FIG. 9A according to an embodiment of the present disclosure. [Figure 10B] FIG. 10B is an exemplary data flow diagram for training a neural network within the motif network shown in FIG. 10A according to an embodiment of the present disclosure. [Figure 11A] FIG. 1 illustrates an exemplary motif causality matrix, according to an embodiment of the present disclosure. [Figure 11B] FIG. 10 illustrates exemplary motif time series data for two motif causality matrix entries, according to an embodiment of the present disclosure. [Figure 12A] FIG. 1 illustrates conventional time-series data generation. [Figure 12B] FIG. 1 illustrates motif causal time series data generation according to an embodiment of the present disclosure. [Figure 13] FIG. 1 illustrates a comparison of longitudinal and synthetic time series data, according to an embodiment of the present disclosure. [Figure 14] 1 is a flowchart illustrating functions associated with generating synthetic data according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0004] This application expressly incorporates U.S. Provisional Patent Application No. 63 / 481,431 (filed January 25, 2023) herein as if fully set forth in its entirety below, and for all applicable purposes.

[0005] One potential technical solution to the problem of sharing patients' longitudinal medical time series data is to generate synthetic (fake) time series data based on the patient's original (real) time series data, such as the patient's measured glucose traces. However, the synthetic time series data must provide strong privacy guarantees and protect the privacy of the patient's longitudinal medical time series data while emulating certain important properties of the original time series data. Privacy guarantees refer to the degree to which sensitive data, such as a patient's medical data, is protected. A formal notion of strong privacy guarantees ensures that the probability of exposing sensitive data is extremely small (e.g., close to zero).

[0006] Various methodologies, such as machine learning (ML) techniques, neural networks (NNs), and artificial neural networks (ANNs), can be used to generate synthetic time series data. These methods use training data that may include additional information, such as labels (i.e., labeled data), which are the results or labeled portions of the traces that guide the generation of synthetic data, or multiple variables per time step (i.e., multivariate data), metadata, or auxiliary features (information calculated during model training). For example, generative adversarial networks (GANs) can be used to generate synthetic data based on original data. While GANs can also be trained to generate synthetic time series data based on original time series data, these GANs do not inherently protect the privacy of the original time series data.

[0007] Synthetic time series data that preserve the privacy of patients' longitudinal medical time series data can be publicly shared and integrated into many practical applications, such as blood glucose prediction, artificial pancreas systems, computer-based medical diagnostic methods, and population-level medical research.

[0008] Embodiments of the present disclosure advantageously provide a differentially private generative adversarial network (DP-GAN) architecture that includes a motif causality module as well as an autoencoder module, a generator module, and a discriminator module. The autoencoder module includes an embedding module and a recovery module. Each module may include one or more ANNs, such as RNNs, LSTM networks, among others, as described below.

[0009] Furthermore, embodiments of the present disclosure advantageously provide a DP-GAN training method that includes original data, motif data, and synthetic data processing techniques, an integrated differential privacy metric, and a loss function that characterizes relationships between significant motifs in the original time series data, as described below. A motif is a short, ordered sequence of time steps from a time series (or trace) that characterizes significant events in the time series data, such as peaks, troughs, etc. In the context of the present disclosure, motifs are not temporally dependent and do not form repeated temporal patterns.

[0010] Importantly, certain embodiments of the present disclosure relate to training a DP-GAN advantageously using unlabeled univariate original data without any auxiliary (additional) information.

[0011] FIG. 1 illustrates a block diagram of a system 100 for generating synthetic data according to an embodiment of the present disclosure.

[0012] Generally, system 100 includes a computer, server, etc. having one or more single-core or multi-core processors, special-purpose processors, etc. configured to train a neural network based on longitudinal time series data to generate synthetic time series data that meets a privacy metric.

[0013] More specifically, system 100 includes a computer 110 coupled to one or more networks 172, one or more I / O devices 182, and one or more displays 192. Computer 110 includes one or more processors 130, a storage element or memory 160, one or more communication interfaces 170, one or more I / O interfaces 180, and a bus 120 coupled to display interface 190. In many embodiments, computer 110 also includes one or more special-purpose processors, such as, for example, graphics processing units (GPU) 140, neural processing unit (NPU) 150, etc. Generally, communication interface 170 is coupled to network 172 using a wired or wireless connection, I / O interface 180 is coupled to I / O devices 182 using a wired or wireless connection, and display interface 190 is typically coupled to display 192 using a wired connection.

[0014] Bus 120 is a communication system that transfers data between processor 130, memory 160, communication interface 170, I / O interface 180, and display interface 190. In many embodiments, bus 120 also transfers data between these components and GPU 140 and / or NPU 150, as well as other components not shown in FIG.

[0015] Processor 130 may include one or more general-purpose or application-specific microprocessors that execute instructions to perform functions such as control, computation, and input / output for computer 110. Each processor 130 may include a single integrated circuit, such as a microprocessing device, or multiple integrated circuit devices and / or circuit boards that cooperate to achieve the appropriate functionality. In addition, processor 130 may execute computer programs or modules, such as an operating system 162 and software modules 164, stored in memory 160. For example, software modules 164 may include neural networks, including one or more artificial neural networks (ANNs), recurrent neural networks (RNNs), long-short-term memory (LSTM) networks, convolutional neural networks (CNNs), etc.

[0016] Generally, memory 160 stores instructions and data for execution by processor 130. Memory 160 may include a variety of non-transitory computer-readable media that can be accessed by processor 130 and other components. In various embodiments, memory 160 may include volatile and non-volatile media, non-removable media, and / or removable media. For example, memory 160 may include any combination of random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), read only memory (ROM), flash memory, cache memory, and / or any other type of non-transitory computer-readable media.

[0017] Memory 160 includes various components for retrieving, presenting, modifying, and storing data 166. For example, memory 160 stores software modules 164 that provide functionality when executed by processor 130. Operating system 162 provides operating system functionality for computer 110. Software modules 164 provide various functionality, as described above. Data 166 may include data related to operating system 162, software modules 164, etc.

[0018] Communications interface 170 is configured to send and receive data to and from one or more networks 172 using one or more wired and / or wireless connections. Network 172 may include, for example, one or more local area networks, wide area networks, the Internet, etc., which may implement various network protocols such as wired and / or wireless Ethernet, Bluetooth, etc. Network 172 may also include various combinations of wired and / or wireless physical layers, such as, for example, copper or coaxial cable networks, fiber optic networks, Bluetooth wireless networks, WiFi wireless networks, CDMA, FDMA, and TDMA cellular wireless networks, etc.

[0019] I / O interface 180 is configured to transmit and / or receive data from I / O device(s) 182. I / O interface 180 enables a connection between processor 130, memory 160, and I / O device(s) 182 by encoding data to be transmitted from processor 130 or memory 160 to I / O device(s) 182 and decoding data received from I / O device(s) 182 for processor 130 or memory 160. In general, data may be transmitted via wired and / or wireless connections. For example, I / O interface 180 may include one or more wired communication interfaces, such as USB, Ethernet, etc., and / or one or more wireless communication interfaces, such as WiFi, Bluetooth, cellular, etc., coupled to one or more antennas.

[0020] Generally, I / O devices 182 provide input to and / or output from computer 110. As described above, I / O devices 182 are operably connected to computer 110 using wired and / or wireless connections. I / O devices 182 may include a local processor coupled to a communication interface configured to communicate with computer 110 using wired and / or wireless connections. For example, I / O devices 182 may include a keyboard, a mouse, a touchpad, a joystick, etc.

[0021] Display interface 190 is configured to transmit image data from computer 110 to a monitor or display 192 .

[0022] As mentioned above, the software module 164 may include a neural network, including one or more ANNs, RNNs, LSTMs, etc.

[0023] An ANN uses a network of interconnected nodes that are trained through a learning process to model the relationship between input data or signals and output data or signals. The nodes are arranged in various layers, including, for example, an input layer, one or more hidden layers, and an output layer. The input layer receives input data, such as image data or sensor time series data, and the output layer provides output data, such as the probability that the image data contains a known object or a medical condition. Each hidden layer provides at least a partial transformation of the input data to the output data. A DNN has multiple hidden layers to model complex nonlinear relationships between input data and output data.

[0024] In a fully connected feedforward ANN, each node is connected to all of the nodes in the preceding layer and all of the nodes in the subsequent layer. For example, each input layer node is connected to each hidden layer node, each hidden layer node is connected to each input layer node and each output layer node, and each output layer node is connected to each hidden layer node. Additional hidden layers are similarly interconnected. Each connection has a weight value, and each node has an activation function, such as a linear function, a step function, a sigmoid function, a hyperbolic or tanh operation, or a rectified linear unit (ReLu) function, that determines the node's output based on a weighted sum of the node's inputs. Input data propagates from the input layer nodes through their respective connection weights to the hidden layer nodes, and then to the output layer nodes through their respective connection weights. For any given input, the sigmoid and ReLu functions output a number between 0 and 1, and the tanh operation outputs a number between -1 and 1.

[0025] More specifically, at each input node, input data is provided to that node's activation function, and the output of the activation function is then provided to each hidden layer node as input data values. At each hidden layer node, the input data values ​​received from each input layer node are multiplied by their respective connection weights, and the resulting products are summed or accumulated into an activation signal value provided to the activation function for that node. The output of the activation function is then provided to each output layer node as input data values. At each output layer node, the output data values ​​received from each hidden layer node are multiplied by their respective connection weights, and the resulting products are summed or accumulated into an activation signal value provided to the activation function for that node. The output of the activation function is then provided as output data. Additional hidden layers may be similarly configured to process data.

[0026] FIG. 2 illustrates an ANN 200 according to an embodiment of the present disclosure.

[0027] ANN 200 comprises an input layer 210 and one or more hidden layers, e.g., hidden layers 2101, 2102, ..., 220 N and an output layer 230. The input layer 210 includes one or more input nodes, e.g., nodes I,1 ,node I,2 ,...,node I,i The hidden layer 2201 includes one or more hidden nodes, e.g., nodes 1,1 ,node 1,2 ,...,node 1,j The hidden layer 2202 includes one or more hidden nodes, e.g., nodes 2,1 ,node 2,2 ,...,node 2,k Hidden layer 220 N is one or more hidden nodes, e.g., node N,1 ,node N,2 ,...,node N,n The output layer 230 includes one or more output nodes, e.g., nodes O,1 ,node O,2 ,...,node O,oIn the example shown in Figure 2, there are N hidden layers: input layer 210 contains "i" nodes, hidden layer 2301 contains "j" nodes, hidden layer 2202 contains "k" nodes, and hidden layer 230 N contains "n" nodes and the output layer 230 contains "o" nodes.

[0028] In a particular embodiment, N equals 3, "i" equals 3, "j", "k", and "n" equals 5, and "o" equals 3. Input Node I,1 ,node I,2 , and node I,3 are hidden nodes, respectively. 1,1 ,node 1,2 ,node 1,3 ,node 1,4 , and node 1,5 Hidden nodes 1,1 ,node 1,2 ,node 1,3 ,node 1,4 , and node 1,5 are hidden nodes, respectively. 2,1 ,node 2,2 ,node 2,3 ,node 2,4 , and node 2,5 Hidden nodes 2,1 ,node 2,2 ,node 2,3 ,node 2,4 , and node 2,5 are hidden nodes, respectively. 3,1 ,node 3,2 ,node 3,3 ,node 3,4 , and node 3,5 Hidden nodes 3,1 ,node 3,2 ,node 3,3 ,node 3,4 , and node 3,5 are the output nodes, respectively. O,1 ,node O,2 ,node O,3 is combined with

[0029] Many other variations of input, hidden, and output layers are clearly possible, including hidden layers that are locally connected to each other rather than fully connected.

[0030] Training an ANN involves optimizing the connection weights between nodes by minimizing the prediction error of the output data until the ANN achieves a certain level of accuracy. One method is backpropagation or backward error propagation, which iteratively and recursively determines the gradient (i.e., partial derivative of the error function) for each weight and then adjusts each weight to improve the network's performance.

[0031] A multi-layer perceptron (MLP) is a fully connected ANN with an input layer, an output layer, and one or more hidden layers. MLPs can be used to process time-series data in natural language processing, machine translation, speech recognition, etc. Other ANNs include RNNs, LSTM networks, and CNNs.

[0032] FIG. 3A shows one diagram of an RNN 300 according to an embodiment of the present disclosure.

[0033] Generally, RNNs process input sequence data and generate output sequence data, and may be used for many different applications, such as, for example, natural language processing applications (e.g., sentiment analysis, speech recognition, reading comprehension, summarization, and translation), image processing (e.g., image captioning, video classification), etc. RNNs may be programmed to process many different types of input data and output data, such as, for example, fixed input data and fixed output data, such as for image classification, fixed input data and sequential output data, such as for image captioning, sequential input data and fixed output data, such as for sentence "sentiment" classification, sequential input data and sequential output data, such as for machine translation, and synchronized sequential input data and sequential output data, such as for video classification.

[0034] RNN 300 includes an input layer 310, one or more hidden layers, such as hidden recurrent layer 320, and an output layer 330. In general, an RNN may include one to four hidden recurrent layers, although other numbers of hidden recurrent layers are also supported.

[0035] The input layer 310 is a node I,1 and node I,2 RNN 300 includes one or more input nodes, such as x, y ...

[0036] The hidden recurrent layer 320 may be, for example, a node R,1 ,node R,2 ,node R,3 ,node R,4 ,...,node R,r RNN 300 is a fully connected recurrent layer that includes hidden recurrent nodes such as . Each hidden recurrent node maintains or stores the state of a hidden state vector h for that layer, which is updated at each time step of RNN 300. In other words, hidden state vector h includes the state of each hidden recurrent node in hidden recurrent layer 320. In many embodiments, the size of hidden state vector h ranges from tens or hundreds to thousands of elements, e.g., 64, 256, 4096, etc. elements. In certain embodiments, hidden state vector h may be subsampled to reduce processing requirements.

[0037] One or more additional fully-connected hidden recurrent layers may follow hidden recurrent layer 320. Each successive hidden recurrent layer contains hidden recurrent nodes and a corresponding hidden state vector h. The final hidden layer, e.g., hidden recurrent layer 320 shown in FIG. 3A, presents the hidden state vector h to output layer 330.

[0038] The output layer 330 includes one or more output nodes, e.g., nodes O,1 In particular embodiments, each output node provides an output such as a predicted class score, a probability of a word, sentence, etc., a predicted data value, a predicted correlation value, etc. A normalization function such as a Softmax function may be applied to the output by the output layer 330, or alternatively, by an additional fully connected layer inserted between the last hidden layer and the output layer 330.

[0039] FIG. 3B shows another diagram of an RNN 300 according to an embodiment of the present disclosure.

[0040] The input layer 310 is shown as a single element 310′ containing input data X, the hidden recurrent layer 320 is shown as a single element, module, or cell 320′ containing hidden state vector h, and the output layer 330 is shown as a single element 330′ containing output data Y.

[0041] FIG. 3C shows another view of an RNN 300 according to an embodiment of the present disclosure.

[0042] The diagram of the RNN 300 shown in FIG. 3B shows the processing configuration of the RNN 300 at time step t, i.e., the input data X t and an input layer 310′ including a hidden state vector h t and a hidden recurrent module 320′ including output data Y t , and an output layer 330′ containing the input data X t is the hidden state vector h t is a vector with the same dimensions as

[0043] FIG. 3D shows another view of the RNN 300, according to an embodiment of the present disclosure.

[0044] As mentioned above, the RNN 300 typically executes a loop such that hidden recurrent modules 320' may process input data X and update the hidden state vector h at each time step. In the diagram shown in Figure 3D, the loop is "unrolled" to show three time steps: t-1, t, and t+1. Thus, the RNN 300 can be viewed as a chain of repeated hidden recurrent modules or cells 320'.

[0045] At time step t-1, the input data X t-1 , the input hidden state vector h from the previous time step t-2 , the hidden state vector h t-1 , and output data Y t-1 At time step t, the input data X t , the input hidden state vector h from the previous time step t-1 , the hidden state vector h t , and output data Y t At time step t+1, the input data X t+1 , the input hidden state vector h from the previous time step t , the hidden state vector h t+1 , and output data Y t+1 is shown.

[0046] In general, the hidden state vector h t is the hidden state vector h from the previous time step, as given by Eq. t-1 weight vector W multiplied by state and the input data X t weight vector W multiplied by data The sum of these functions is the activation function f c can be updated by applying h t =f c (W state h t-1 +W data X t ) Equation 1

[0047] Activation function f ccan be a nonlinear activation function, e.g., tanh(), ReLu, etc., applied to each element of the hidden state vector h. In a titration embodiment, the bias b c is the activation function f c The output data Y t is the weight vector W as given by Equation 2. output The hidden state vector h t is the product multiplied by Y t =W output h t formula 2

[0048] In a particular embodiment, the output data Y as given by Equation 3 t To generate the weight vector W output and the hidden state vector h t and an activation function f such as tanh() or ReLu o may be applied. Y t =f o (W output h t ) Equation 3

[0049] In certain embodiments, the bias b o is the activation function f o may be added to the product before application of

[0050] FIG. 3E illustrates a data flow diagram 302 of a hidden recurrent module 320' according to an embodiment of the present disclosure.

[0051] A hidden recurrent module 320′ at time step t is shown. The hidden recurrent module 320′ includes a tanh or sigmoid layer 322, which is the hidden state vector h t-1 and the input data vector X t receives the hidden state vector h from the previous time step. t-1 The weight vector W multiplied by state and the input data X t The weight vector W multiplied by dataApplying the tanh operation to the sum of and gives the hidden state vector h as given by Equation 1. t Generate the hidden state vector h t is output to the output layer 330 and provided to the next time step or stored for use by the next time step.

[0052] Similar to an ANN, training an RNN involves optimizing the weights by minimizing the prediction error of the output data until the RNN achieves a certain level of accuracy. As described above, backpropagation through time can be used to iteratively and recursively determine the gradient (i.e., the partial derivative of the error function) for each weight and then adjust each weight to improve the RNN's performance. However, if the gradient for one or more of the weights becomes too small (i.e., the gradient "vanishes"), those weights will not be adjusted and training will eventually stop. This problem is known as the vanishing gradient problem.

[0053] LSTM networks are a variant of RNNs that, among other advantages, address the vanishing gradient problem by increasing the complexity of each hidden recurrent module or cell to generate and maintain more information, i.e., only the hidden state vector h, i.e., the cell state vector C. LSTM networks also avoid the long-range dependency problem of RNNs.

[0054] FIG. 4A shows a diagram of an LSTM network 400 according to an embodiment of the present disclosure.

[0055] The LSTM network 400 also typically implements a loop so that an LSTM module or cell 420 can process each time step. In the diagram shown in Figure 4A, the loop is "unrolled" to show three time steps, similar to the diagram of the RNN 300 shown in Figure 3D. Thus, the LSTM network 400 can also be viewed as a chain of repeating LSTM cells 420.

[0056] At time step t-1, the input data Xt-1 , the input hidden state vector h from the previous time step t-2 and the input cell state vector C t-2 , the hidden state vector h t-1 , cell state vector C t-1 , and output data Y t-1 At time step t, the input data X from the previous time step is t , the input hidden state vector h from the previous time step t-1 and the input cell state vector C t-1 , the hidden state vector h t , cell state vector C t , and output data Y t At time step t+1, the input data X t+1 , the input hidden state vector h from the previous time step t and the input cell state vector C t , the hidden state vector h t+1 , cell state vector C t+1 , and output data Y t+1 is shown.

[0057] FIG. 4B illustrates a data flow diagram 402 of an LSTM cell 420 according to an embodiment of the present disclosure.

[0058] The LSTM cell 420 is denoted at time step t. The LSTM cell 420 receives the cell state vector C from the previous time step. t-1 and the hidden state vector h t-1 , and receives or obtains input data X from the input layer 410 for the current time step. t and processes these data to generate the cell state vector C for the current time step. t and the hidden state vector h t and generates the hidden state vector h for the current time step. t to the output layer 430, and for the next time step, the hidden state vector h t and the cell state vector C t transmit or store the

[0059] LSTM cell 420 includes, among other things, a cell memory (not shown for clarity), a forget gate 440, an input gate 450, an output gate 460, and a cell state update segment 470. LSTM cell 420 may be implemented by a software module, process, routine, etc., by a hardware component, circuit, etc., by a combination of hardware and software components, etc.

[0060] The forget gate 440 is used to calculate the hidden state vector h t-1 and input data X t Based on this, the cell state vector C t-1 The input gate 450 determines which elements of the hidden state vector h should be discarded (i.e., "forgotten") or retained (i.e., "memorized"). t-1 and input data X t Based on this, the cell state vector C t-1 The cell state update segment 470 generates new information to be added to the cell state vector C based on the outputs of the forget gate 440 and the input gate 450. t-1 and update the cell state vector C t The output gate 460 generates the hidden state vector h t-1 , input data X t , and the updated cell state vector C t Based on this, the hidden state vector h t Generate.

[0061] FIG. 4C illustrates a data flow diagram 404 of an LSTM cell 420 according to an embodiment of the present disclosure.

[0062] The LSTM cell 420 is shown at time step t. The hidden state vector h t-1 and the input data vector X t is provided to a forget gate 440, an input gate 450, and an output gate 460. In many embodiments, the concatenation operation 422 computes the hidden state vector h t-1 and the input data vector X tto form the concatenated input vector [h t-1 ,X t ]. In another embodiment, the hidden state vector h t-1 and the input data vector X t are provided separately to the forget gate 440, the input gate 450, and the output gate 460.

[0063] The forget gate 440 receives the concatenated input vector [h t-1 ,X t ] and the weight vector W f Concatenate the input vector [h t-1 ,X t ] to obtain the weighted concatenated input vector W f ·[h t-1 ,X t ] and apply the sigmoid function to the weighted concatenated input vector W as given by Equation 4. f ·[h t-1 ,X t ] to obtain the activation vector f t The sigmoid layer 442 generates the connection weight vector W f is the hidden state vector h t-1 weight vector and input data vector X t is a concatenation of the weight vectors of f t =σ(W f ·[h t-1 ,X t ]) Formula 4

[0064] In certain embodiments, the bias b f is the weighted concatenated input vector W before the application of the sigmoid function σ. f ·[h t-1 ,X t ]. The sigmoid layer 442 applies the activation vector f t to provide.

[0065] Input gate 450 includes a sigmoid layer 452, a tanh layer 454, and an element-wise multiplication operation 456. Sigmoid layer 452 receives the concatenated input vector [h t-1 ,X t ] and the weight vector W i Concatenate the input vector [h t-1 ,X t ] to obtain the weighted concatenated input vector W i ·[h t-1 ,X t ] and apply the sigmoid function to the weighted concatenated input vector W as given by Equation 5. i ·[h t-1 ,X t ] to obtain the activation vector i t Generate. i t =σ(W i ·[h t-1 ,X t ]) Formula 5

[0066] In certain embodiments, the bias b i is the weighted concatenated input vector W before the application of the sigmoid function σ. i ·[h t-1 ,X t ]. The sigmoid layer 452 performs element-wise multiplication operations 456 on the activation vector i t to provide.

[0067] The Tanh layer 454 receives the concatenated input vector [h t-1 ,X t ] and the weight vector W C Concatenate the input vector [h t-1 ,X t ] to obtain the weighted concatenated input vector W C ·[h t-1 ,X t ] and apply the tanh operation to the weighted concatenated input vector W as given by Equation 6. C ·[h t-1 ,X t ] to obtain the activation vector

[0068]

number

[0069]

number

[0070] In certain embodiments, prior to application of the tanh operation, the weighted concatenated input vector W C ·[h t-1 ,X t ] bias b C The Tanh layer 454 can be added to the activation vector

[0071]

number

[0072]

number

[0073] Output gate 460 includes a sigmoid layer 462, an element-wise multiplication operation 466, and an element-wise tanh operation 464. Sigmoid layer 462 receives the concatenated input vector [h t-1 ,X t ] and the weight vector W o Concatenate the input vector [h t-1 ,X t ] to obtain the weighted concatenated output vector W o ·[h t-1 ,X t ] and apply the sigmoid function to the weighted concatenated output vector W o ·[h t-1 ,X t] to obtain the activation vector o t Generate. o t =σ(W o ·[h t-1 ,X t ]) Formula 7

[0074] In certain embodiments, the bias b o is the weighted concatenated input vector W before the application of the sigmoid function σ. o ·[h t-1 ,X t ]. The sigmoid layer 462 can be added to the activation vector o t to the element-wise multiplication operation 466.

[0075] The tanh operation 464 receives the cell state vector Ct, applies the tanh operation to the cell state vector Ct, and provides the result to an element-wise multiplication operation 466, which multiplies the output of the sigmoid layer 462 with the output of the tanh operation 464 to produce the hidden state vector h, as given by Equation 8. t Generate. h t =o t tanh(C t ) Equation 8

[0076] The element-wise multiplication operation 476 in the cell state update segment 470 multiplies the cell state vector C t-1 receives the activation vector f t and the cell state vector C t-1 and 456 to generate an intermediate vector product, which is provided to an element-wise addition operation 478. The intermediate vector products generated by the element-wise multiplication operation 476 and the element-wise multiplication operation 456 are added to the cell state vector C, as given by Equation 9. t are added to produce

[0077]

number

[0078] hidden state vector h t is output to the output layer 430. The hidden state vector h t and the cell state vector C t is provided to the next time step or stored for use by the next time step.

[0079] FIG. 5 illustrates a data flow diagram 502 for a DP-GAN 500, according to an embodiment of the present disclosure.

[0080] As described above, GANs can be used to generate synthetic data based on original data. Generally, GANs include a generator neural network and a discriminator neural network. The generator neural network learns from the original data and operates to generate synthetic data. The discriminator neural network receives samples of both the original (real) data and the synthetic (fake) data and "guesses" whether each sample is real or fake. The generator neural network and the discriminator neural network are trained adversarially, i.e., against each other. The generator neural network attempts to trick the discriminator neural network into guessing that the synthetic data is real, and the discriminator neural network attempts to become very good at guessing which samples are actually real or fake. If the training is successful, the generator neural network becomes very good at generating synthetic data that tricks the discriminator neural network into guessing that the synthetic data is real.

[0081] Differential privacy is a formal concept of privacy that limits the risk to anyone who provides data for subsequent processing. In DP-GANs, noise is drawn from carefully designed distributions and applied to the weights of the generator neural network and the discriminator neural network to protect the privacy of individuals associated with the data. From one perspective, the addition of noise prevents the generator neural network and the discriminator neural network of a DP-GAN from remembering or disclosing sensitive or personal information from the original data.

[0082] The DP-GAN 500 includes a motif causality module 510, an autoencoder module 520, a generator module 530, a discriminator module 540, a pre-processor module 526, and a post-processor module 528. The autoencoder module 520 includes an embedding module 522 and a recovery module 524. Each module may include, among other things, one or more neural networks, such as RNNs, LSTM networks, etc., implemented as software modules, processes, routines, etc. In many embodiments, these software components may be executed by the processor 130. In particular embodiments, at least some of these software components may be executed by the GPU 140 or the NPU 150. In other embodiments, these software components may be executed by a combination of the processor 130 and the GPU 140, the processor 130 and the NPU 150, or the processor 130, the GPU 140, and the NPU 150.

[0083] In many embodiments, medical longitudinal time series data (traces) for a population of patients can be divided into two sets: Set A and Set B. Each set contains several traces for different groups of patients within the population. Set A can contain the same number of traces as Set B, fewer traces than Set B, or more traces than Set B. In certain embodiments, the medical longitudinal time series data can be completely divided into Set A and Set B, while in other embodiments, the medical longitudinal time series data can be partially divided into Set A and Set B based on selection criteria. For example, a patient's medical longitudinal time series data can be selected based on quality. As an example, unselected patient's medical longitudinal time series data can exhibit undesirable characteristics such as measurement noise, data dropout, etc. Generally, a patient's medical longitudinal time series data is bounded and contains at least 50 time steps. Certain medical longitudinal time series data may include 100 or more time steps, such as 288 time steps (i.e., 12 measurements per hour) for 24 hours of continuous glucose monitoring (CGM) data.

[0084] The original longitudinal time series data (Set A) 550 is the original data x m (data 553) to the motif causality module 510. The original longitudinal time series data (set B) 560 is provided to the pre-processor module 526 as raw data x (data 563).

[0085] The motif causality module 510 uses the original data x m , and generates several non-overlapping motif data partitions. Each motif data partition is provided to a different motif network 512. Each motif network 512 generates an aggregate motif causality matrix M A and provided to the generator module 530. iDuring training, the motif network 512 learns the relationships between motifs and represents these relationships in a causality matrix, which is an aggregate motif causality matrix M A The motif causality module 510 is described in more detail below.

[0086] The pre-processor module 526 pre-processes the original data x to generate batched original data x and provides the batched original data x to the embedding module 522 .

[0087] The embedding module 522 reduces the dimensionality of the batched raw data x to generate a set of embedded traces, i.e., the embedded raw data x e (data 523). In many embodiments, the embedding module 522 includes a neural network having an input layer, at least one hidden layer (e.g., an RNN layer (e.g., hidden recurrent layer 320, hidden recurrent module 320′, etc.) or an LSTM layer (e.g., LSTM cell 420, etc.), and an output layer. In particular embodiments, the neural network may be a CNN. Other neural network architectures are also supported.

[0088] The generator module 530 generates the embedded synthetic data

[0089]

number

[0090] The post-processor module 528 processes the synthetic data embedded in the original data space.

[0091]

number

[0092]

number

[0093] The recovery module 524 recovers the embedded original data x e Rebuild the original data

[0094]

number

[0095] The discriminator module 540 discriminates the source data x e Receive the original data x e Similarly, the discriminator module 540 also uses the embedded synthetic data

[0096]

number

[0097]

number

[0098] During training, carefully calibrated noise (not shown for clarity) is added to the weights of the autoencoder module 520 (i.e., the embedding module 522 and the restoration module 524), the generator module 530, and the discriminator module 540 to ensure that each network maintains differential privacy, e.g., satisfies a privacy metric, to protect patient privacy. To generate synthetic data, a weight noise generator 580 generates weight noise (Z) 583, which is received as input to the generator module 530 and is then fed to the restoration module 524 for generation of the final synthetic data.

[0099]

number

[0100] Embedding source data x e and embedded synthetic data

[0101]

number

[0102]

number

[0103] FIG. 6 illustrates a loss function diagram 600 for training the DP-GAN 500 shown in FIG. 5 according to an embodiment of the present disclosure.

[0104] In many embodiments, the modules of the DP-GAN 500 may be trained on a specific sequence. First, the motif causality module 510 trains the original data x m (data553) and the motif causality matrix513 (M A ) is generated. The remaining modules of the DP-GAN 500 then generate the original data x (data 563) and the aggregate motif causality matrix 513 (M A ) in turn (e.g., within each epoch) and

[0105]

number

[0106] In many embodiments, six loss functions are used to train the autoencoder module 520, the generator module 530, and the discriminator module 540, and the loss functions are the reconstruction loss (L R ) 610 and stepwise loss (L S ) 620 and the distribution loss (L D )630 and motif causality loss (L M ) 640 and the fake adversarial loss (LAf ) 650 and genuine hostile losses (L Ar ) 660. Other loss functions, as well as subsets of these loss functions, are also supported.

[0107] Reconstruction loss (L R )610 is the original data x and the recovered data

[0108]

number

[0109]

number

[0110] Stepwise loss (L S )620 is the original data x e Batches and Embedding Synthetic Data

[0111]

number

[0112] Distribution loss (L D )630 is the distribution of the original data x and the synthetic data

[0113]

number

[0114] Motif causality loss (L M )640 is the original data M x The motif causality matrix calculated above and the synthetic data

[0115]

number

[0116]

number

[0117]

number

[0118]

number

[0119] Fake adversarial loss (L Af )650 is synthetic data

[0120]

number

[0121] Real hostile losses (L Ar ) 660 is the BCE between the discriminator guess on the original data x, i.e., the embedding guess 543, and the ground truth, i.e., a vector of 0s.

[0122] To avoid over-specialization, the autoencoder module 520 uses the reconstruction loss (L R ) 610 and step loss (L S ) 620 (where α is a weight hyperparameter).

[0123]

number

[0124] The generator module 530 generates the step loss (L S )620, distributed loss (L D )630, motif causality loss (L M ) 640, and the fake adversarial loss (L Af ) 650 (η is a weight hyperparameter). S ) 620 allows dual training of the autoencoder module 520 and the generator module 530.

[0125]

number

[0126] The discriminator module 540 calculates the fake adversarial loss (LAf ) 650 and genuine hostile losses (L Ar ) 660.

[0127]

number

[0128] In one embodiment, α is 0.1 and η is 10, although other values ​​are supported. These training objectives and loss functions train DP-GAN 500 to generate high-quality, long-term synthetic time series data.

[0129] FIG. 7 illustrates a data flow 700 for generating batched raw data for training DP-GAN 500 shown in FIG. 5 according to an embodiment of the present disclosure.

[0130] In the exemplary embodiment shown in FIG. 7, the raw data 710 includes 100 traces, namely, traces 7121, . . . , 712 100 Each trace contains 288 data values ​​(time steps). For example, trace 7121 contains the original data X 1,1 , ..., X 1,288 Includes trace 712 100 is the original data X 100,1 , ..., X 100,288 In many embodiments, the pre-processor module 526 (as described above) pre-processes the raw data x (data 563) to generate batched raw data x.

[0131] The pre-processor module 526 applies a sliding window (width 24, stride length 1) to each trace in the raw data 710 to expand each trace into a batched data slice containing 264 time chunks, each containing 24 data values ​​(time steps). For trace 7121, the sliding window is applied to the first 24 data values, i.e., data value sequence 7141, to produce X 1,1 , X 1,2, ..., X 1,23 , X 1,24 The sliding window is then shifted one data value position to the right and applied to the next 24 data values, i.e., data value sequence 7142, to generate a time chunk 7241 of the batched data slice 7221 containing X 1,2 , X 1,3 , ..., X 1,24 , X 1,25 , and so on. 263 is X 1,263 , X 1,264 , ..., X 1,286 , X 1,287 A batched data slice 7221 containing time chunks 724 263 and generates a data value sequence 714 264 is X 1,264 , X 1,265 , ..., X 1,287 , X 1,288 A batched data slice 7221 containing time chunks 724 264 The remaining traces are processed in the same way, and finally, trace 712 is generated. 100 is the batched data slice 722 100 The batched raw data 720 is divided into batched data slices 722. i All of the batched data slices 7221, ..., batched data slices 7222 100 Other methods for generating batched raw data 720 are also supported.

[0132] The embedding module 522 then reduces the dimensionality of the batched raw data x to produce a set of embedded traces, i.e., the embedded raw data x e In an exemplary embodiment, the embedding module 522 generates the embedded data x e The number of chunks in each batched data slice may be reduced from 264 to 128 to generate

[0133] 8A and 8B illustrate a data flow 800 for generating synthetic data 830 by the DP-GAN 500 shown in FIG. 5 according to an embodiment of the present disclosure.

[0134] In many embodiments, the post-processor module 528 processes the embedded synthetic data in the original data space (as described above).

[0135]

number

[0136]

number

[0137] In the exemplary embodiment shown in FIGS. 8A and 8B, the embedded synthetic data 810 is divided into 100 traces, namely, traces 8121, . . . , 812 100 Each trace contains 128 time chunks, and each time chunk contains 24 data values ​​(time steps). For example, trace 8121 contains time chunks 8241, ..., time chunk 824 128 and time chunk 8241 contains embedded synthetic data X h 1,1 , ..., X h 1,24 , ..., including time chunk 824 128 is the embedded synthetic data X h 128,1 , ..., X h 128,24 Contains (X h in Figs. 8A and 8B

[0138]

number

[0139] The post-processor module 528 first processes each trace 812 of the embedded synthetic data 810. i are serialized into a single row of the re-formed embedded synthetic data 820. For example, time chunk 8241 is first placed in the first row of the re-formed embedded synthetic data 820, time chunk 8242 is placed in the first row of the re-formed embedded synthetic data 820 after time chunk 8241, and so on, so that trace 8121 is formed into serialized trace 8221. 128 Time chunk 824 127 , and repeat until the first row of the re-formed embedded synthetic data 820 is placed after the first row, thereby completing the formation of the serialized trace 8221. The elements of the serialized trace 8221 are indexed by the index of the time chunk / time step (e.g., X h 1,1 , ..., X h 1,24 , X h 128、1 , ..., X h 128,24 ) to trace / time step based index (e.g., X h 1,1 , ..., X h 1,3072 , X h 100,1 , ..., X h 100,3072 ) and the remaining trace 812 of the embedded synthetic data 810. i is similarly serialized and traced 100 Serialized trace from 822 100 Conclude with the formation of

[0140] The post-processor module 528 then generates a serialized trace 822 for each of the re-formed embedded synthetic data 820. i, and reconstruct the embedded synthetic data into the original space of 100 traces, each with 288 data values ​​(time steps). In general, the inverse sliding window is applied to each serialized trace 822 i A group of data values ​​in the window are averaged based on the window width (t time steps) and stride length (s time steps) to form each composite trace 832 i For example, the serialized trace 8221 generates a reverse sliding window at data value X h 1,1 , ..., X h 1,3072 is applied to the data value X h 1,1 , ..., X h 1,288 and so on. Finally, the serialized trace 822 100 is the inverse sliding window over the data values ​​X h 100,1 , ..., X h 100,3072 By applying 100 and the data value X h 100,1 , ..., X h 100,288 The synthetic data 830 generates synthetic traces 8321, ..., 832 100 Other methods of generating synthetic data 830 are also supported.

[0141] FIG. 9A illustrates a data flow diagram 900 of the motif causality module 510 according to an embodiment of the present disclosure.

[0142] The motif causality module 510 connects the data processing module 910 with a number (N) of motif networks 5121, 5122, ..., 512 Nand a motif causality matrix aggregation module 940. During training of the motif causality module 510, the data processing module 910 generates a number (N) of non-overlapping motif data partitions 9201, 9202, ..., 920 N the original data x m (data 553). Each motif data partition 920 i contains data for different patients from the original longitudinal time series data (set A) 550. Each motif network 5121, 5122, ..., 512 N The different motif data partitions 920 i , i.e., motif network 5121 receives motif data partition 9201, motif network 5122 receives motif data partition 9202, and so on.

[0143] Each motif network 512 i is the motif data partition 920 i Based on the motif causality matrix 930 i (M i ), i.e., motif network 5121 generates motif causality matrix 9301 (M1) based on motif data partition 9201, motif network 5122 generates motif causality matrix 9302 (M2) based on motif data partition 9202, and so on. As described above, each motif causality matrix 930 i (M i ) is a motif network 512 i represents the relationships between motifs learned during training. In general, the motif causality matrix 930 i (M i ) contains motif causality values ​​c and has width ≦ m and height ≦ m (where m is the number of motifs analyzed in the data partition, as explained below). Each causality coefficient c j,krepresents the strength of the relationship between two motifs (e.g., motif j and motif k) and can have a value between 0 (i.e., indicating a weak relationship) and 1 (i.e., indicating a strong relationship). Other values ​​are also supported.

[0144] The motif causality aggregation module 940 generates motif causality matrices 9301 (M1), 9302 (M2), ..., 930 N (M N ) is the aggregate motif causality matrix 513(M A ) to protect patient privacy, i.e., satisfy the privacy metric. A ) is the original data x e are provided to the generator module 530 during its training in order to focus the generator module 530 on preserving important motifs (events) within the traces.

[0145] FIG. 9B illustrates a dataflow diagram 902 for generating batched motif data 960 for training the motif causality module 510 shown in FIG. 9A, according to an embodiment of the present disclosure.

[0146] The data processing module 910 processes each motif data partition 9201, 9202, ..., 920 N After generating the motif data partitions 9201, 9202, ..., 920 N are further processed to generate respective batched motif data 960.

[0147] 9B, the motif data partition 920 includes 100 traces, each containing 288 data values ​​(time steps). For example, the first trace contains the original data X 1,1 , ..., X 1,288 The final trace is the original data X 100,1 , ..., X 100,288Other numbers of traces and other numbers of data values ​​(time steps) are supported.

[0148] The motif data partition 920 can be conceptually divided into motif blocks, one motif block for each motif being analyzed. Three motif blocks are shown, each containing 96 data values ​​(time steps) for each trace: motif block 921 for motif 1, motif block 922 for motif 2, and motif block 923 for motif 3. The motif block 921 contains the data value X 1,1 , ..., X 1,96 , ..., X 100,1 , ..., X 100,96 The motif block 922 contains the data value X 1,97 , ..., X 1,192 , ..., X 100,97 , ..., X 100,192 The motif block 923 contains the data value X 1,193 , ..., X 1,288 , ..., X 100,193 , ..., X 100,288 The motif data partition 920 may be divided into two motif blocks, but the motif data partition 920 is typically divided into three or more motif blocks.

[0149] The data processing module 910 divides the motif blocks 921, 922, and 933 into separate motif blocks for each trace, and then stacks the separate motif blocks into a motif block stack 950. For the first trace, the motif block 9211 is 1,1 , ..., X 1,96 and motif block 9221 contains data value X 1,97 , ..., X 1,192 and motif block 9231 contains data value X 1,193 , ..., X 1,288 and so on. For the last trace, motif block 921100 is the data value X 100,1 , ..., X 100,96 Includes motif block 922 100 is the data value X 100,97 , ..., X 100,192 Includes motif block 923 100 is the data value X 100,193 , ..., X 100,288 Therefore, the motif block stack 950 includes 300 motif blocks.

[0150] The data processing module 910 then applies a sliding window (width 24, stride length 1) to each motif block in the motif block stack 950 to expand each motif block into a motif sequence block containing 72 overlapping motif sequences, where each motif sequence contains 24 data values ​​(time steps).

[0151] For motif block 9211, a sliding window is applied to the first 24 data values ​​to obtain X 1,1 , X 1,2 , ..., X 1,23 , X 1,24 The sliding window is then moved one data value position to the right and applied to the next 24 data values ​​to produce a motif sequence 9641 of the motif sequence block 9621 containing X 1,2 , X 1,3 , ..., X 1,24 , X 1,25 and so on. For example, motif sequence 964 71 is X 1,71 , X 1,72 , ..., X 1,94 , X 1,95 Contains motif sequence 964 72 is X 1,72 , X 1,73 , ..., X 1,95 , X 1,96 Includes.

[0152] For motif block 9221, the sliding window scans the data value X to generate motif sequence block 9622. 1,97 , ..., X 1,192 For motif block 9231, the sliding window is applied to data value X 1,193 , ..., X 1,288 to generate motif sequence block 9623 (not shown for clarity), and similarly for the remaining motif blocks. For example, motif block 923 100 , the sliding window is 100,193 , ..., X 100,288 applied to motif sequence block 962 300 Other methods for generating the batched motif data 960 are also supported.

[0153] FIG. 10A illustrates a motif network 512 according to an embodiment of the present disclosure. i 10 shows a data flow diagram 1000 of the above.

[0154] Motif Network 512 i is a set of m neural networks 10101, 10102, ..., 1010 m and a weight combination module 1030. The number m is the number of motifs to be analyzed, as described above. Neural network 10101 includes a weight matrix 10201 (W1), neural network 10102 includes a weight matrix 10202 (W2), and so on. m is the weight matrix 1020 m (W m ) is included.

[0155] Each neural network 1010 j As will be described later, the motif data partition 920 i The weight combination module 1030 is trained using the motif causality matrix 930 iTo generate the weight matrices 10201, 10202, ..., 1020 m Generally, each weight matrix 1020 j contains weights w and has width equal to the sliding window width (e.g., 24 time steps) and height equal to m.

[0156] FIG. 10B illustrates the motif network 512 shown in FIG. 10A in accordance with an embodiment of the present disclosure. i Neural Networks in 1010 j 10 shows a data flow diagram 1002 for training

[0157] In certain embodiments, the loss module 1040 and the weight adjustment module 1050 adjust the motif network 512 i Each neural network in 1010 j In other embodiments, the loss module 1040 and the weight adjustment module 1050 may be provided to the motif network 512. i Each neural network is provided with 1010 j can be used to train

[0158] In many embodiments, the neural network 1010 j includes an input layer, at least one hidden layer, such as an RNN layer (e.g., hidden recurrent layer 320, hidden recurrent module 320′, etc.) or an LSTM layer (e.g., LSTM cell 420, etc.), and an output layer. In particular embodiments, the output layer may be preceded by a convolutional layer. Other neural network architectures are also supported.

[0159] In general, neural networks 1010 j is a specific "ground truth" motif sequence block 962 within the batched motif data 960 j It is trained on ground truth motif sequence blocks 962 jand all other motif sequence blocks in the batched motif data 960. More specifically, the neural network 1010 j generates predicted motif sequence blocks 1062 based on the batched motif data 960. The loss module 1040 compares the predicted motif sequence blocks 1062 with the ground truth motif sequence blocks 962 using a loss function, e.g., MSE, RMSE, etc. j By comparing with neural network 1010 j The weight (W j ) should be adjusted.

[0160] FIG. 11A illustrates a motif causality matrix 1100 according to an embodiment of the present disclosure.

[0161] Motif causality matrix 1100 is a 10x10 matrix presenting motif causality values ​​for 100 pairs of motifs. X-axis 1102 contains 10 motif bins, Y-axis 1104 contains 10 motif bins, and scale 1106 ranges from 0 (i.e., no causality between the motifs) to 1 (strong causality between the motifs). For example, motif causality element 1120 has a value of 0.382, indicating some causality between motif 100 (i.e., bin 5 on the X-axis) and motif 281 (i.e., bin 7 on the Y-axis). Motif causality element 1120 has a value of 0.424, indicating a slightly higher causality between motif 140 (i.e., bin 6 on the X-axis) and motif 297 (i.e., bin 9 on the Y-axis).

[0162] FIG. 11B illustrates motif comparisons 1120 and 1130 for motif causality elements 1108 and 1110, respectively, according to an embodiment of the present disclosure.

[0163] Motif comparison 1120 includes graph 1122 showing time series data 1124 (i.e., glucose values ​​vs. time) for motif 100, graph 1126 showing time series data 1128 (i.e., glucose values ​​vs. time) for motif 281, and motif causality element 1108 with a value of 0.382. Similarly, motif comparison 1130 includes graph 1132 showing time series data 1134 (i.e., glucose values ​​vs. time) for motif 140, graph 1136 showing time series data 1138 (i.e., glucose values ​​vs. time) for motif 297, and motif causality element 1110 with a value of 0.424.

[0164] In the case of longitudinal medical time series data, the sequence of significant motifs provides more information than previous individual time steps. Traditional time series data generation methods, such as autoregressive models, assume that the time series depends on all previous time steps within a window and generate a value of x at time t based on the sequence of previous values ​​of x. Importantly, these methods only preserve temporal relationships (e.g., information from previous time steps within the window) and ignore other potentially useful relationships within the same time series (e.g., x) or between different time series.

[0165] FIG. 12A illustrates conventional time series data generation 1200.

[0166] The value of x at time step t (i.e., x value 1206) depends on the values ​​of x at time steps t-1 (i.e., x value 1201), t-2 (i.e., x value 1202), t-3 (i.e., x value 1203), t-4 (i.e., x value 1204), and t-5 (i.e., x value 1205). While previous values ​​for x may be weighted in a linear combination, subject to dropout, etc., traditional methods are highly dependent on the window size and miss long-term relationships between different time series.

[0167] FIG. 12B illustrates motif causal time series data generation 1210 according to an embodiment of the present disclosure.

[0168] 12B, the value of x4 at time step t (i.e., x4 value 1220) depends on the values ​​of x1 at time t-3 (i.e., x1 value 1213) and t-5 (i.e., x1 value 1215), the values ​​of x2 at time t-1 (i.e., x2 value 1221) and t-2 (i.e., x2 value 1222), and the value of x3 at time t-4 (i.e., x1 value 1234). Similarly, the value of x5 at time step t (i.e., x5 value 1230) depends on the value of x1 at time t-1 (i.e., x1 value 1211), the value of x2 at time t-4 (i.e., x2 value 1224), and the value of x3 at time t-2 (i.e., x3 value 1232) and t-3 (i.e., x3 value 1233).

[0169] Advantageously, motif causality time series data generation 1210 uses only previous lags that have causal influence and finds relationships across motifs from different time series, allowing DP-GAN 500 to learn relationships (patterns) between sequences of significant events in traces that contribute to time series construction.

[0170] This is particularly advantageous for long time series, as networks can easily become overwhelmed when trained to learn every previous time step. By instead preserving only the relationships associated with sequences of significant motifs, DP-GAN500 learns to more quickly output realistic sequences of time steps in the trace.

[0171] For example, for a glucose trace, to predict the next glucose value at time t, a large glucose peak (e.g., a hyperglycemic event) six or more time steps in the past (e.g., before t-6) is more informative than the previous five time steps (e.g., t-1, t-2, t-3, t-4, t-5). This is due to the strong influence of the event (e.g., knowing that the glucose value should fall from the peak regardless of whether the previous glucose value was 330 → 329 or 290 → 289). As a result, patterns between these types of events can be exploited (e.g., if a large peak motif is seen, then a decreasing slope motif will emerge).

[0172] FIG. 13 illustrates a comparison 1300 of longitudinal and synthetic time series data according to an embodiment of the present disclosure.

[0173] The longitudinal time series data 1310 includes measured glucose values ​​(mg / dL) for 288 time steps. The synthetic time series data 1320 includes synthetic glucose values ​​(mg / dL) for 288 time steps generated by the DP-GAN 500. As can be seen from the sample synthetic traces, the patterns in the traces look very authentic (e.g., have an overall structure very similar to the real traces in terms of sequences of peaks, troughs, etc.).

[0174] FIG. 14 shows a flowchart 1400 illustrating functions associated with generating synthetic data according to an embodiment of the present disclosure.

[0175] Longitudinal time series data is received at 1410. In many embodiments, the longitudinal time series data is unlabeled and univariate.

[0176] As described above, the time series data may be medical longitudinal time series data. Generally, a patient's medical longitudinal time series data is bounded and includes at least 50 time steps. Particular medical longitudinal time series data may include 100 or more time steps, such as 288 time steps (i.e., 12 measurements per hour) for 24 hours of continuous glucose monitoring (CGM) data.

[0177] At 1420, a neural network is trained based on the longitudinal time series data to generate synthetic time series data that meets the privacy metric.

[0178] In many embodiments, the neural network may be a DP-GAN, such as, for example, DP-GAN 500. Training DP-GAN 500 is described above with reference to FIGS. 6-11C. Additionally, as described above, a privacy-preserving aggregate motif causality matrix 513 (M A ) is the original data x e The noise is provided to the generator module 530 during its training to focus the generator module 530 on preserving important motifs (events) within the traces of the embedding module 522, the recovery module 524, the generator module 530, and the discriminator module 540. Noise may also be added to the weights of the embedding module 522, the recovery module 524, the generator module 530, and the discriminator module 540 to ensure that each network maintains differential privacy.

[0179] The many features and advantages of the present disclosure will be apparent from the detailed description, and thus, it is intended by the appended claims to cover all such features and advantages of the present disclosure that fall within the scope of the present disclosure. Further, because numerous modifications and variations will readily occur to those skilled in the art, it is not desired to limit the disclosure to the exact construction and operation illustrated and described, and therefore, all suitable variations and equivalents may be employed as being within the scope of the present disclosure.

Claims

1. 1. A method for generating synthetic data, comprising: Extracting unlabeled, univariate longitudinal time series data; training a neural network to generate synthetic time series data that meets a privacy metric based on the longitudinal time series data.

2. The method of claim 1 , wherein the longitudinal time-series data comprises at least 50 measured glucose levels from each of a plurality of individuals.

3. The method of claim 1 , wherein the privacy metric defines an upper bound on the amount of privacy loss that is tolerated.

4. The neural network is a differentially private generative adversarial network (DP-GAN), and training the neural network includes: In the motif causality module, receiving a first portion of the longitudinal time series data for a first group of individuals; generating an aggregate motif causality matrix based on the first portion of the longitudinal time series data, the aggregate motif causality matrix identifying causal relationships between motifs within the first portion of the longitudinal time series data.

5. the motif causality module includes a plurality of motif networks, and generating the aggregate motif causality matrix comprises: dividing the first portion of the longitudinal time series data into data partitions, each data partition associated with a different motif network and including multiple motifs, each motif being an ordered sequence of data values ​​from the first portion of the longitudinal time series data; generating, for each motif network, a motif causality matrix from the associated data partitions; aggregating the motif causality matrices into the aggregate motif causality matrix based on the privacy metric.

6. 6. The method of claim 5, wherein each motif network includes multiple recurrent neural networks (RNNs), each RNN receiving motif data from the associated data partition for a different motif.

7. Training the neural network comprises: In the embedded module: receiving a second portion of the longitudinal time series data for a second group of individuals different from the first group of individuals; generating embedded time series data based on the second portion of the longitudinal time series data, the embedded time series data having a lower dimensionality than the second portion of the longitudinal time series data; In the generator module: generating embedded synthetic time series data based on the aggregate motif causality matrix and the embedded time series data; In the recovery module: generating recovered longitudinal time series data based on the embedded time series data; generating synthetic time series data based on the embedded synthetic time series data, the synthetic time series data having the same number of dimensions as the second portion of the longitudinal time series data; In the discriminator module, determining whether each data value in the embedded time series data is real or synthetic; determining whether each data value in the embedded synthetic time series data is real or synthetic; and training the embedding module, the restoration module, the generator module, and the discriminator module to satisfy a performance metric and the privacy metric based on a plurality of loss functions.

8. Training the embedding module, the restoration module, the generator module, and the discriminator module includes: The method of claim 7 , comprising adding noise to weights associated with the embedding module, the restoration module, the generator module, and the discriminator module based on the privacy metric.

9. Training the embedding module, the restoration module, the generator module, and the discriminator module includes: training the embedding module and the recovery module based on a reconstruction loss and a stepwise loss; training the generator module based on at least one of the stepwise loss, distribution loss, motif loss, and synthetic data adversarial loss; and training the discriminator module based on the synthetic data adversarial loss and the embedded data adversarial loss.

10. The method of claim 9 , wherein the motif loss is associated with a data sequence pattern within the second portion of the longitudinal time series data.

11. The method of claim 7 , wherein the embedding module, the restoration module, the generator module, and the discriminator module each comprise an RNN.

12. 1. A system for generating synthetic data, comprising: a memory configured to store unlabeled, univariate longitudinal time series data; at least one processor coupled to the memory, at least one processor configured to train a neural network to generate synthetic time series data that meets a privacy metric based on the longitudinal time series data.

13. The system of claim 12 , wherein the privacy metric defines an upper bound on the amount of privacy loss that is tolerated.

14. The neural network is a differentially private generative adversarial network (DP-GAN) including a motif causality module having a plurality of motif networks, an embedding module, a generator module, a discriminator module, and a recovery module; the motif causality module is trained based on a first portion of the longitudinal time series data for a first group of individuals; 14. The system of claim 13, wherein the embedding module, the recovery module, the generator module, and the discriminator module are trained based on a second portion of the longitudinal time series data for a second group of individuals different from the first group of individuals.

15. Each motif network includes multiple recurrent neural networks (RNNs), 15. The system of claim 14, wherein each RNN receives motif data for a different motif from an associated data partition of the first portion of the longitudinal time series data.

16. the motif causality module generates an aggregate motif causality matrix based on the first portion of the longitudinal time series data; 16. The system of claim 15, wherein the aggregate motif causality matrix identifies causal relationships between motifs within the first portion of the longitudinal time series data.

17. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to: Extract unlabeled, univariate longitudinal time series data, a non-transitory computer-readable medium for training a neural network to generate synthetic time series data that satisfies a privacy metric based on the longitudinal time series data;

18. 20. The non-transitory computer-readable medium of claim 17, wherein the privacy metric defines an upper bound on an amount of privacy loss that is acceptable.

19. The neural network is a differentially private generative adversarial network (DP-GAN) including a motif causality module having a plurality of motif networks, an embedding module, a generator module, a discriminator module, and a recovery module; the motif causality module is trained based on a first portion of the longitudinal time series data for a first group of individuals; 20. The non-transitory computer-readable medium of claim 18, wherein the embedding module, the retrieval module, the generator module, and the discriminator module are trained based on a second portion of the longitudinal time-series data for a second group of individuals different from the first group of individuals.

20. Each motif network includes multiple recurrent neural networks (RNNs), 20. The non-transitory computer-readable medium of claim 19, wherein each RNN receives motif data for a different motif from an associated data partition of the first portion of the longitudinal time-series data.

Images