Synthetic data generation using deep reinforcement learning
Through a deep neural network system with parallel competition, the data sharing problem under privacy and security concerns is solved, efficient machine learning model training is realized, and data privacy and security are ensured.
Patent Information
- Application Number
- CN202380028502.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-13
- Filing Date
- 2023-01-28
- Publication Date
- 2024-11-08
AI Technical Summary
Due to privacy and security concerns, large data sets containing personally identifiable information cannot be transmitted or shared with machine learning models in different geographic locations, resulting in limited model training.
Adopting a deep neural network system with parallel competition, the synthetic data set is generated through the generator network and identified by the discriminator network. The generator network is optimized to ensure that the synthetic data set is similar to the original data set. At the same time, the discriminator network is becoming more and more distinctive, Ensure the privacy and security of synthetic datasets.
The generated synthetic dataset can be effectively used to train external ML models, maintaining privacy and security, avoiding artificial intervention and performance degradation, and improving the robustness and training efficiency of the model.
Smart Images

Figure CN118922838A_ABST
Abstract
Description
Background Art
[0001] Large datasets are used to train machine learning (ML) models. However, due to privacy, security concerns, and privacy regulations, some large datasets should not and / or are not allowed to be shared with ML models that can benefit from using these datasets, especially datasets that contain personally identifiable information. This is because the ML models are executed in a different physical location than where the data is stored or retrieved, such as a device or server in a different geographic location. Due to the aforementioned privacy and security concerns, these datasets cannot be transferred or shared to other physical locations where ML models are executed and trained. Summary of the invention
[0002] This Summary is provided to introduce a selection of concepts in a simplified form, which are further described below in the Detailed Description. This Summary is not intended to identify key features or potential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
[0003] Examples and implementations disclosed herein are directed to systems and methods for generating synthetic data using deep reinforcement learning. The system includes a memory, a processor, a first neural network, and a second neural network implemented on a server. The first neural network generates a synthetic data set based on an original data set. The original data set and the generated synthetic data set are provided to a second neural network that generates a prediction identifying the original data set and the generated synthetic data set. Based at least in part on the prediction of the generated synthetic data set that was incorrectly identified, the generated synthetic data set is derived. BRIEF DESCRIPTION OF THE DRAWINGS
[0004] The present description will be better understood by reading the following detailed description in light of the accompanying drawings, in which:
[0005] Figure 1 is a block diagram illustrating an example computing device for implementing various examples of the present disclosure;
[0006] Figure 2 is a block diagram illustrating an example system for implementing deep reinforcement learning according to various examples of the present disclosure;
[0007] Figure 3 is a flowchart illustrating the operation of a computer-implemented method for deep reinforcement learning according to various examples of the present disclosure;
[0008] Figure 4A is a flowchart illustrating the operation of a computer-implemented method for generating synthetic data and performing deep reinforcement learning according to various examples of the present disclosure;
[0009] Figure 4Bis a flowchart illustrating the operation of a computer-implemented method for predicting original and synthetic data sets and performing deep reinforcement learning according to various examples of the present disclosure; and
[0010] Figure 5 is an operational flow chart illustrating a computer-implemented method for deep reinforcement learning according to various examples of the present disclosure.
[0011] Corresponding reference characters indicate corresponding parts throughout the drawings. Figures 1 to 5 In the drawings, the system is shown schematically. The drawings may not be to scale. DETAILED DESCRIPTION
[0012] Various implementations and examples will be described in detail with reference to the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to represent the same or similar parts. References to specific examples and implementations involved throughout this disclosure are provided for illustrative purposes only, but, unless otherwise indicated, are not intended to limit all examples.
[0013] As described herein, ML models and other deep neural networks are trained using large datasets. However, due to various privacy and security concerns, these datasets may not be able to be transferred to the ML models for use. As a result, some ML models are restricted from using some data, causing them to learn less. Therefore, various implementations of the present disclosure recognize and take into account the need to provide synthetic datasets that represent the original datasets for ML models to learn as effectively as possible. Current solutions are unable to ensure the privacy and security of the original data while providing sufficiently robust synthetic data for effectively training ML models. Current solutions include anonymized data, which can be decoded and cannot provide strict privacy guarantees. Other examples include implementing known ML models for generating synthetic data.
[0014] Various examples in the present disclosure address the above challenges by introducing two competing deep neural networks, one that generates synthetic data based on an original dataset, and the other that attempts to identify the original dataset and the synthetic dataset. Generating a synthetic dataset based on the original dataset (but different from the original dataset) enables external ML models to be trained based on real-world data and can be securely exported and transmitted without the risk of violating the privacy and security issues that exist with the original dataset. The first competing neural network and the second competing neural network continue to learn and optimize in parallel, so that the first deep neural network generates synthetic data that is increasingly similar to the original dataset, and the second deep neural network increasingly effectively distinguishes the original data from the synthetic data. Finally, a robust discriminator neural network is obtained, which, despite its robustness, cannot correctly identify the generated synthetic dataset from the original synthetic dataset, indicating that the generated synthetic dataset sufficiently mimics the original dataset and can therefore be exported and used to train another ML model.
[0015] The input data to the generator network, along with a 'reward', forms a 'state'. The reward is a measure of how well the discriminator network failed to distinguish between real data (i.e. the original dataset) and fake examples (i.e. the synthetic dataset). The 'action' taken by the generator network is a distortion of the original input data. The next state is the next set of examples. Therefore, the tuple <state,action,reward,next state > stands for reinforcement learning.
[0016] Therefore, the system provided by the present disclosure operates in an unconventional manner, which generates a realistic synthetic dataset based on the original dataset by introducing parallel competing neural networks. The system provides several advantages. In addition to the performance improvement, the system also eliminates human bias due to the lack of human intervention. As a complex nonlinear process, privacy is preserved. The training of ML models or supervised classifiers does not achieve performance degradation between the original dataset and the synthetic dataset.
[0017] Thus, the systems and methods of the present disclosure provide technical solutions to inherent technical problems by improving the generation of synthetic data sets by optimizing the parameters of a first neural network that generates a synthetic data set in response to analysis performed by a second competing neural network that predicts whether the synthetic data set is synthetic data or original data. After each iteration of synthetic data generation and prediction (also referred to herein as an epoch), both the first neural network and the second neural network calculate losses and optimize their respective parameters for more efficient generation of synthetic data and predictions, respectively. Thus, the system generates a robust synthetic data set that can be exported for training another ML model without violating privacy and security considerations.
[0018] Various implementations in the present disclosure employ stochastic gradient descent to train one or both of the competing neural networks. Gradient descent is an optimization algorithm that finds the local minimum of a particular function. Stochastic gradient descent is an iterative algorithm that mathematically minimizes a loss (such as a cross entropy loss). By iteratively minimizing the loss, the parameters of the neural network are also iteratively improved. By applying stochastic gradient descent to competing neural networks (such as a generator network ML model that generates synthetic data and a discriminator network ML model that predicts whether the synthetic data is original or synthetic), as described herein, the generated synthetic data is iteratively improved to a degree that is sufficiently similar to the original data, and is therefore eligible to be applied to train additional ML models that cannot or will not use the original data due to privacy and security issues.
[0019] Figure 1 An example computing device 100 for implementing various aspects of the present disclosure is shown and is generally identified as computing device 100. Computing device 100 is only an example of a suitable computing environment and is not intended to provide any limitation on the scope of use or functionality of the examples of the present disclosure. Computing device 100 should not be interpreted as having any dependency or requirement on any one or combination of components / modules shown.
[0020] The examples disclosed herein may be described in the general context of computer code or machine or computer executable instructions (such as program components) executed by a computer or other machine. Program components include routines, programs, objects, components, data structures, etc. that involve code, perform specific tasks, or implement specific abstract data types. The disclosed examples may be practiced in a variety of system configurations, including servers, personal computers, laptops, smart phones, servers, VMs, mobile tablets, handheld devices, consumer electronics, special computing devices, etc. The disclosed examples may also be practiced in a distributed computing environment, where tasks are performed by remote devices connected by a communication network.
[0021] The computing device 100 includes a bus 110 that directly or indirectly couples the following devices: computer storage memory 112, one or more processors 114, one or more display components 116, I / O ports 118, I / O components 120, power supply 122, and network components 124. Although the computing device 100 is described as a single device, multiple computing devices 100 can work together and share the device resources described. For example, the memory 112 is distributed across multiple devices, and (multiple) processors 114 are located in different devices. The bus 110 represents one or more buses (such as an address bus, a data bus, or a combination of the two). Although for clarity, Figure 1 The various boxes in the example are shown with lines, but alternative representations may be used to depict the various components. For example, components such as display devices are shown as I / O components in some examples, and processors in other examples have their own memory. No distinction is made between categories such as "workstation", "server", "laptop", "handheld device", etc., because these are considered Figure 1 The scope shown is within the scope and is introduced herein as a "computing device".
[0022] The memory 112 can take the form of a computer storage memory device referenced below, and can be operable to provide storage of computer-readable instructions, data structures, program modules, and other data for the computing device 100. In some examples, the memory 122 stores one or more of an operating system (OS), a general-purpose application platform, or other program modules and program data. The memory 122 is therefore capable of storing and accessing data 112a and instructions 112b, which can be executed by the processor 114 and configured to perform various operations disclosed herein. In some examples, the memory 112 stores executable computer instructions for the OS and various software applications. The OS can be any OS designed to control the functions of the computing device 100, including, for example, but not limited to: Developed by By Apple from Cupertino, Calif., Developed by MAC By Google from Mountain View, California, Developed by ANDROID TM , open source wait.
[0023] As an example and not limitation, computer-readable media include computer storage memory devices and communication media. Computer storage memory devices may include volatile, non-volatile, removable, non-removable or other memory for storing information (such as computer-readable instructions, data structures, program modules, etc.) implemented in any method or technology. Computer storage memory devices are tangible and mutually exclusive with communication media. Computer storage memory devices are implemented in hardware and do not include carrier waves and propagation signals. The computer storage memory devices used for the purposes of this disclosure are not signals themselves. Example computer storage memory devices include hard disks, flash drives, solid-state memories, phase change random access memories (PRAM), static random access memories (SRAM), dynamic random access memories (DRAM), other types of random access memories (RAM), read-only memories (ROM), electrically erasable programmable read-only memories (EEPROM), flash memory or other storage technologies, compact disk read-only memories (CD-ROM), digital versatile disks (DVD) or other optical memories, cassettes, tapes, disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible to a computing device. In contrast, communication media typically embodies computer readable instructions, data structures, program modules, etc. in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
[0024] Computer executable instructions can be organized into one or more computer executable components or modules. In general, program modules include but are not limited to routines, programs, objects, components, and data structures that perform specific tasks or implement specific abstract data types. Aspects of the present disclosure can be implemented with any number of these components or modules. For example, aspects of the present disclosure are not limited to the specific computer executable instructions or specific components or modules shown in the accompanying drawings and described in the text. Other examples of the present disclosure may include different computer executable instructions or components, which have more or less functions than those shown and described in the text. In examples involving general-purpose computers, aspects of the present disclosure convert general-purpose computers into special-purpose computing devices, CPUs, GPUs, ASICs, systems on chips (SoCs), etc., for providing new VMs when configuring to execute instructions described herein.
[0025] The processor(s) 114 may include any number of processing units that read data from various entities (such as memory 112 or I / O components 120). Specifically, the processor(s) 114 are programmed to execute computer executable instructions for implementing various aspects of the present disclosure. These instructions may be executed by a single processor 114, multiple processors 114 in a computing device 100, or an external processor of a client computing device 100. In some examples, the processor(s) 114 are programmed to execute those instructions shown in the flowcharts discussed below and depicted in the accompanying drawings. In addition, in some examples, the processor(s) 114 represent an implementation of simulation techniques for performing the operations described herein. For example, operations are performed using a simulated client computing device 100 and / or a digital client computing device 100.
[0026] Presentation component(s) 116 present data indications to a user or other device. Example presentation components include a display device, a speaker, a printing component, a vibrating component, and the like. Those skilled in the art will understand and appreciate that computer data can be presented in a variety of ways, such as visually in a graphical user interface (GUI), audibly through a speaker, wirelessly between computing devices 100, through a wired connection, or otherwise. I / O ports 118 allow computing device 100 to be logically coupled to other devices, including I / O components 120, some of which may be built-in. Example I / O components 120 include, for example, but are not limited to, microphones, joysticks, game controllers, satellite dishes, scanners, printers, wireless devices, and the like.
[0027] The computing device 100 can communicate on the network 130 via the network component 124 using a logical connection with one or more remote computers. In some examples, the network component 124 includes a network interface card and / or a computer executable instruction (such as a driver) for operating the network interface card. The communication between the computing device 100 and other devices can be carried out using any protocol or mechanism through any wired or wireless connection. In some examples, the network component 124 is operable to communicate data between short-range wireless communication technology devices (such as near-field communication (NFC), Bluetooth brand communication, etc.) or a combination thereof using a transmission protocol in a public, private or hybrid (public and private) manner. The network component 124 communicates with the cloud environment 128 across the network 130 via a wireless communication link 126 and / or a wired communication link 126a. Various different examples of communication links 126 and 126a include wireless connections, wired connections, and / or dedicated links, and in some examples, at least a portion is routed through the Internet.
[0028] The network 130 may include any computer network or a combination thereof. Examples of computer networks that may be configured to operate as the network 130 include, but are not limited to, wireless networks; landline telephones; cable lines; digital subscriber lines (DSL): fiber optic lines; cellular networks (such as 3G, 4G, 5G, etc.); local area networks (LAN); wide area networks (WAN); metropolitan area networks (MAN), etc. However, the network 130 is not limited to connections that couple independent computer units. Instead, the network 130 may also include a subsystem that transmits data before a server or computing device. For example, the network 130 may also include a point-to-point connection, the Internet, an Ethernet, a circuit bus, a neural network, or other internal systems. Such network architectures are well known and do not need to be discussed in depth here. As described herein, the computing device 10 can be implemented as one or more servers. As described in more detail below, the computing device 100 can be implemented as an electronic device in the system 200.
[0029] Figure 2 200 is a block diagram illustrating an example system for implementing deep reinforcement learning according to various examples of the present disclosure. System 200 can include computing device 100. In some implementations, system 200 includes a cloud-implemented server that includes each of the components of system 200 described herein.
[0030] The system 200 includes a memory 232 and a processor 226 that executes instructions 234 stored on the memory 232. In some implementations, the memory 232 is the memory 112, the instructions 234 stored on the memory 232 are instructions 112b, and the processor 226 is the processor 114. The system 200 also includes a generator network 204 and a discriminator network 216. The generator network 204 and the discriminator network 216 can be executed by the processor 226 that executes the instructions 234 stored on the memory 232. The generator network 204 and the discriminator network 216 will be described in more detail below.
[0031] In some implementations, the memory 232 also stores the original input data 202. In some implementations, the original input data 202 includes one or more data sets. The original input data 202 can include any type of data. In some implementations, the original input data 202 can include personal identification information of one or more users, consumers, customers, employees, suppliers, etc. in combination with one or more data fields. However, this example should not be interpreted as limiting. Various implementations are possible. In some implementations, the original input data is provided in a table containing up to hundreds or thousands of rows and up to hundreds or thousands of columns of data. For example, the original input data 202 can be a data set X including m rows and n columns.
[0032] In some implementations, the raw input data 202 is subject to restrictions on the manner and location of collection, transmission and storage. For example, the raw input data 202 may be subject to restrictions on security and privacy issues, which restrict the raw input data 202 from being transmitted and stored to locations other than the location where it is obtained. The location may include a physical location (such as a facility), or a geographical location (such as a county, city, state, country, etc.). In accordance with these requirements and best practices, the raw input data 202 is co-located with the generator network 204 and the discriminator network 216 that use the raw input data 202 as input. In other words, the raw input data 202 is stored in the same location of the memory 232 as the generator network 204 and the discriminator network 216. In some implementations, the raw input data 202 is stored in the same physical location (such as a facility) as the generator network 204 and the discriminator network 216. In other implementations, the raw input data 202 can be stored in a physical location (such as a different facility) different from the generator network 204 and the discriminator network 216 but in the same geographical range, so that local security and privacy rules and best practice guidelines are followed.
[0033] Each of the generator network 204 and the discriminator network 216 is a deep neural network, also referred to herein as a deep learning network. In some examples, the generator network 204 is referred to herein as a first neural network and the discriminator network 216 is referred to herein as a second neural network. However, the use of primary and secondary descriptions is only for ease of explanation and should not be construed as limiting. Various examples may refer to the discriminator network 216 because the generator network 204 and the discriminator network 216 are executed in parallel to optimize the synthetic data iteratively generated by the generator network 204.
[0034] The generator network 204 includes a data generator 206 and a machine learning (ML) model 208. This ML model 208 may be referred to herein as a first ML model. This data generator 206 iteratively generates a synthetic data set, such as a synthetic data set 214, corresponding to the original input data 202. The ML model 208 includes a loss calculator 210 and a parameter optimizer 212. The loss calculator 210 calculates the loss of the synthetic data 214 based on receiving the labeled input data 230 described in more detail below. Based on the calculated loss, the parameter optimizer 212 adjusts, updates, or optimizes the parameters of the data generator 206, which then generates the next iteration of the synthetic data 214 based on the original input data 202. In some implementations, the ML model 208 is trained using stochastic gradient descent.
[0035] The discriminator network 216 receives iterations of the synthetic data 214 and the original input data 202 simultaneously. In some implementations, in order to mask which data set is original and which data set is synthetic from the discriminator network 216, the synthetic data 214 and the original input data 202 are randomly assigned a label, such as 0 and 1. The discriminator network 216 includes a data classifier 218 and a machine learning (ML) model 220. The ML model can be referred to as a second ML network. In some implementations, the ML model is trained using stochastic gradient descent.
[0036] The data classifier 218 generates a prediction that can predict which of the original input data 202 and the synthesized data 214 is the original data and which is the synthesized data. In other words, the prediction output by the data classifier 218 either identifies the original input data 202 as the original input data and the synthesized data 214 as the synthesized data, or identifies the original input data 202 as the synthesized data and the synthesized data 214 as the original data. Accordingly, the data classifier 218 either correctly predicts the synthesized data and the original data, or incorrectly predicts the synthesized data and the original data.
[0037] To output a prediction, the data classifier 218 labels the input raw data 202 and synthesized data 214. For example, the data classifier 218 may use a binary labeling system, with predicted raw data labeled as 0 and predicted synthesized data labeled as 1, or vice versa. The processor 226 compares the prediction output by the data classifier 218 with the synthesized data 214 and raw input data 202 labels, which are randomly assigned before the discriminator network 216 receives the data. In an implementation where the discriminator network 216 correctly predicts the raw input data 202 and the synthesized data 214, the processor output will return an all-0 label 228 to each generator network 204 and the discriminator network 216. In an implementation where the discriminator network 216 incorrectly predicts the raw input data 202 and the synthesized data 214, the processor output will return an all-1 label 228 to each generator network 204 and the discriminator network 216.
[0038] The label 228 provides an indication of the predictions generated by the data classifier 218 for each of the generator network 204 and the discriminator network 216. In the implementation, a label 228 of 0 indicates that the data classifier 218 correctly predicted the original and synthetic data, and the label 228 will be returned to the discriminator network 216 as positive feedback indicating a correct prediction. The label 228 is also attached to the synthetic data 214 as the labeled data 230 and returned to the generator network 204 as negative feedback, because the iteration of the synthetic data 214 is not realistic enough to fool the discriminator network 216 into predicting that it is the original data. Conversely, in the implementation, a label 228 of 1 indicates that the data classifier 218 incorrectly predicted the original and synthetic data, and the label 228 will be returned to the discriminator network 216 as negative feedback indicating an incorrect prediction. Label 228 is also appended to the synthesized data 214 as labeled data 230 and returned to the generator network 204 as positive feedback, because the iterations of the synthesized data 214 are realistic enough to fool the discriminator network into predicting it as original data.
[0039] The labeled data 230 includes the synthesized data 214 indicating a particular iteration of the synthesized data, and a label 228 generated by the processor 226, which indicates whether the data classifier correctly or incorrectly predicted the synthesized data 214 and the original input data 202. The label 228 can be applied using any suitable method of labeling data. For example, the label 228 can be applied as a header of the synthesized data 214, attached to the front or end of the synthesized data 214, and the file name of the synthesized data 214 can be changed to include this label 228 or other suitable methods can be used.
[0040] The generator network 204 and the discriminator network 216 execute the ML models 208 and 220 after receiving the labeled data 230 and the labels 228, respectively. As described in more detail below, the loss calculator 210 calculates a loss for the synthetic data 214 based on the labeled data 230. Based on the calculated loss, the parameter optimizer 212 adjusts the parameters of the data generator 206, which then generates the next iteration of the synthetic data 214 based on the original input data 202. Similarly, the loss calculator 222 calculates a loss for the predictions of the data classifier 218 based on the labels 228. Based on the loss, the parameter optimizer 224 adjusts, updates, or optimizes the parameters of the data classifier 218, which then generates predictions for the next iteration of the received synthetic data 214 and the original input data 202.
[0041] The system 200 also includes a data exporter 236. After one or more iterations of generating the synthetic data 214 and processing the synthetic data 214 through the discriminator network 216, the synthetic data 214 is deemed to be sufficiently similar to the original input data 202 to be used for training the external ML model. The synthetic data 214 can be automatically determined to be sufficiently similar to the original input data 202 after being predicted as original data by the data classifier 218 more than a threshold number of times. In some implementations, this threshold is one. In other words, the first time the synthetic data 214 is predicted as original data, the data exporter 236 exports the synthetic data 214 for training the external ML model. In other implementations, the threshold is greater than one. In other words, the synthetic data 214 should be predicted as original data by the data classifier 218 multiple times before being identified as realistic enough to be exported by the data exporter 236.
[0042] The external ML model described herein can be any ML model trained using synthetic data 214. The external ML model is different from ML model 208 and ML model 220. Although described as an external ML model, this example should not be construed as limiting. The external ML model can be included in system 200, but is different from ML model 208 and ML model 220.
[0043] In some implementations, system 200 is referred to as a deep reinforcement learning system. Deep reinforcement learning includes states, actions, and rewards. For example, a state is the value of an input feature in an observation (such as raw input data 202). Actions are continuous and include synthetic data 214, and actions include all changes to the original input data 202, such as distortion, noise introduction, synthetic data values, etc. Rewards are generated based on the competitive generator network 204 and the discriminator network 216, which produces an output of labeled data 230 and is returned to the generator network 204 for additional optimization of the parameters for generating synthetic data 214.
[0044] Figure 3 is a flowchart illustrating the operation of a computer-implemented method for deep reinforcement learning according to various examples of the present disclosure. Figure 3 The operations shown are for illustration only and should not be construed as limiting. Various examples of operations may be used without departing from the scope of the present disclosure. The operations in flowchart 300 may be performed by one or more components of system 200, including generator network 204, discriminator network 216, and processor 226.
[0045] Method 300 begins with a generator network 204 that receives raw input data 202 in operation 301. The raw input data 202 can be provided in various formats, including but not limited to .csv files, .xml files, .xlsx files, .txt files or any other type of files. In some implementations, the raw input data 202 is provided in a file format that includes one or more columns and one or more rows. In a specific example, the raw input data 202 can be a data set X that includes m rows and n columns. The raw input data set 202 can be provided as a combination of text data, digital data, or text and digital data. For example, some data can be provided as digital data and other data can be provided as text data, such as personal identity information followed by digital data including birthday, social security number, financial account information, card number, etc. Other data can include a combination of text and digital data in a single data field, such as address.
[0046] In operation 303, the generator network 204 generates synthetic data based on or corresponding to the original input data 202. For example, the synthetic data 214 includes data similar to the original input data 202, including the same type of data fields, the same number of data fields, and similar patterns of data fields as the original input data 202. The generator network 204 distorts the feature values of the original input data 202 to introduce noise and produces synthetic data values that are close to those of the original input data 202 without making any meaningful reconstruction of the original data. The original input data 202 forms the state of the deep reinforcement learning process. As described above, the data set X includes X cat and X cont , are the classification part and continuous part of the dataset X respectively. cat The variables in are one-hot encoded. The original input data 202 is distorted by passing it through a series of alternating affine and nonlinear activation functions. The affine transformation has a linear form W.X+b, where W is the weight matrix and X is the input data. The nonlinear transformation is a ReLU function, which has the form: ReLU(X)=X if X>0 and ReLU(X)=0if X≤0.
[0047] For the distorted input, the output of the generator network 204 is represented by G(X). For example, in a generator network 204 having multiple layers, each layer is represented by L i Represented, and nonlinear activation is performed after applying affine transformation. Each layer L i has the same shape as the previous layer, and each layer undergoes a nonlinear transformation after applying an affine transformation at each step:
[0048] G X =ReLU(W k (ReLU(W k-1(…ReLU(W0X+B0)
[0049] Therefore, the output has the same form as the input and is distorted by applying multiple affine and non-linear activations at different layers. There is a fully connected layer between the reconstructed input and a single node in the output. This output tries to minimize the accuracy of the discriminator network. This output is represented as:
[0050] Generator X =σ(W final .G X +b X )
[0051] For example, the original input data 202 includes consumer information indicating an original age and an original purchase history, while the authentic version of the synthetic data 214 may include an age that is similar to but different from the original age, and a purchase history that is similar to but different from the original purchase history. This data is similar to the original input data, including the same pattern of age ranges and purchase history, and therefore can be used to train external ML models as effectively as the original input data 202, and privacy and security issues are maintained because personal information is not directly used. However, it should be understood that in some implementations, the generator network 204 may perform multiple iterations on generating the synthetic data 214 before the synthetic data 214 is considered authentic. For example, an unauthentic version of the synthetic data 214, especially an early version, may generate synthetic data indicating an age that is younger or older than the original age, and generate synthetic data indicating a purchase history that is very different from the original purchase history, but this synthetic data has the same distribution, so any ML model built on the synthetic data generates similar results, thereby serving as an example of privacy protection.
[0052] In some implementations, the generator network 204 generates the synthesized data 214 based on the segmented original input data 202. For example, in the case where the original input data 202 includes multiple rows and multiple columns, the generator network 204 may generate one row or one column of synthesized data 214 at a time. In other words, in operation 303, the generator network 204 generates a row of synthesized data 214 corresponding to the first row of the original input data 202, and iteratively performs operations 303 to 315 until the synthesized data 214 is determined to be sufficiently realistic. Once the first row of the original input data 202 is synthesized, the generator network 204 continues to the second row of the original input data 202 and iteratively performs operations 303 to 315. Accordingly, the method 300 is iteratively performed for each row of the original input data 202 until the original input data 202 is fully synthesized, that is, until sufficient synthesized data 214 is generated for all of the original input data 202. Although the operations 303 to 315 described here are row-by-row iterations of the original input data 202, this example should not be interpreted as limiting. In some implementations, operations 303 to 315 may be iterated on the original input data 202 column by column.
[0053] In operation 305, the discriminator network 216 receives a labeled version of each of the original input data 202 and the synthesized data 214. For example, the discriminator network receives the original input data 202 labeled as 0 and the synthesized data 214 labeled as 1. In another example, the discriminator network receives the original input data 202 labeled as 1 and the synthesized data 214 labeled as 0. The 0 and 1 labels are used for the discriminator network in order to mask which data set is the original data and which is generated synthetic. In some implementations, the labels are applied by the processor 226. The 0 or 1 label can be appended to the front or end of the data as a header of the data, the file name of the data can be changed, or any other suitable method can be used to include the label. In some implementations, the input of the discriminator network 216 is provided as X and G X Connections such as X,G X =[X|G X ].
[0054] In operation 307, the discriminator network 216 generates predictions that identify the original input data 202 and the synthesized data 214. There are two possible choices for the predictions. The predictions can be correct and accurately identify the original input data 202 and the synthesized data 214, or incorrect and identify the synthesized data 214 as the original data. The predictions are generated by passing the original input data 202 and the synthesized data 214 through a classifier (such as data classifier 218), which distinguishes between 0 and 1 of the labeled data passed to the discriminator network 216. In some implementations, the discriminator network 216 calculates the forward propagation, i.e., the generated predictions, such as The generated prediction is pushed to processor 226 .
[0055] In operation 309, processor 226 labels synthetic data 214 with label 228 based on the prediction generated by discriminator network 216. As described above, label 228 is a binary representation of the accuracy of the generated prediction, where 0 indicates a correct prediction and 1 indicates an incorrect prediction. Label 228 is appended to synthetic data 214 to generate labeled data 230.
[0056] In operation 311, processor 226 returns labeled data 230 to generator network 204 and discriminator network 216. Based on labeled data 230, each of generator network 204 and discriminator network 216 executes an ML model, ML model 208 and ML model 220, respectively, to improve functionality. More specifically, ML model 208 receives labeled data 230 as feedback and estimates or calculates a loss for an iteration of synthesized data 214 in operation 313a. For example, generator network 204 calculates a loss such as:
[0057]
[0058] Similarly, the ML model 220 receives the labeled data 230 as feedback and estimates or calculates the prediction loss for the iterative generation of the synthetic data 214 in operation 313b. For example, the discriminator network 216 calculates the loss as:
[0059]
[0060] The losses of the generator network 204 and the discriminator network 216 can be calculated simultaneously, involving the unknown W k , W final , H k and all bias terms B. Although the losses are referred to herein as being calculated simultaneously, it should be understood that there may be some variation in the time required to calculate the losses. For example, synchronously should be understood to mean occurring at approximately the same time, such that overlap in time can be expected. Computing the loss of the generator network 204 may take more or less time than computing the loss of the discriminator network 216 without departing from the scope of this disclosure.
[0061] In operation 315a, the calculated loss is then used to improve the parameters of the data generator 206. In operation 315b, the calculated loss is then used to improve the parameters of the data classifier 218. For example, the parameter optimizer 212 may back-propagate the loss by subtracting the gradient of the loss, thereby minimizing W in the direction opposite to the gradient. k , W final , H k and the values of all bias terms B. Similarly, the parameter optimizer 224 can backpropagate the loss by subtracting the gradient of the loss, thereby minimizing W in the direction opposite to the gradientk , W final , H k and the values of all bias terms B.
[0062] It should be understood that operations 313a, 313b, 315a, and 315b may be performed in any order. For example, the generator network 204 may perform operations 313a and 315a before the discriminator network 216 performs operations 313b and 315b, or the discriminator network 216 may perform operations 313b and 315b before the generator network 204 performs operations 313a and 315a. In other implementations, the generator network 204 performs operations 313a and 315a, and the discriminator network 216 performs operations 313b and 315b simultaneously. Figure 4A Operations 313a and 315b are described in more detail, and reference is made to Figure 4B Operations 313b and 315b are described in more detail.
[0063] After the generator network 204 completes execution of operation 315a, the method 300 returns to operation 303, and the generator network 204 generates the next iteration of the synthetic data 214 based on the parameters improved in operation 315a. This is the beginning of the second epoch. The operation of the method 300 continues until the generator network 204 generates synthetic data 214 that the discriminator network 216 cannot accurately distinguish from the original input data 202.
[0064] Thus, the generator network 204 and the discriminator network 216 operate in parallel as competing neural networks. The generator network 204 iteratively generates synthetic data that is continually improved at each iteration of the method 300 through the performance of operations 313a and 315a. Likewise, the discriminator network 216 iteratively generates synthetic data 214 and predictions of the original input data 202 that are continually improved at each iteration of the method 300 through the performance of operations 313b and 315b. The result is that the first neural network generates improved synthetic data, and the second neural network generates improved predictions about the synthetic data, which in turn continually improve the first neural network.
[0065] As described herein, operations 303 to 315 are iteratively repeated for each row or column of the original input data 202. When each row (or column) is completed, the data exporter 236 appends the completed row (or column) to the generated synthetic data set. After each row (or column) is synthesized, the data exporter 236 exports the data to an external ML model, which will be trained using the generated synthetic data.
[0066] Each iteration of generating a single dataset of synthetic data 214, processing the synthetic data 214 through the discriminator network 216, returning predictions to the generator network 204 and the discriminator network 216, and executing the ML model 208 and the ML model 220 to calculate the loss and optimize the parameters of the data generator 206 and the data classifier 218, respectively, is referred to as an epoch. Batches of raw input data 202 can refer to states passing through the system 200. The generation of synthetic data 214 is referred to as an action, and the labeled data 230 is referred to as a reward.
[0067] In some implementations, method 300 initializes the parameters of generator network 204 and discriminator network 216 to random values. In other words, all W k , W final , H k The values of and all bias terms B are initialized to random values. The number of epochs (num_epochs) can be 100, the batch size (batch_size) can be 32, and the number of batches (num_batches) = the number of observations (num_observations) divided by the batch size (batch_size).
[0068] At each epoch in the range (1: num_epochs), operations 303 to 315 are performed. In other words, the forward propagation of the generator network 204 (i.e., the synthesized data 214) is calculated, the labeled original input data 202 and the synthesized data 214 are input to the discriminator network 216, the forward propagation (i.e., prediction) of the discriminator network 216 is generated, the loss of the generator network 204 and the discriminator network 216 is calculated, the loss is derived and the parameters of each network in the generator network 204 and the discriminator network 216 are optimized.
[0069] Figure 4A is an operational flow chart illustrating a computer-implemented method for generating synthetic data and performing deep reinforcement learning according to various examples of the present disclosure. Figure 4A The operations shown in the flowchart 400 are for illustration only and should not be construed as limiting. Various examples of operations may be used without departing from the scope of the present disclosure. The operations of the flowchart 400 may be performed by one or more components of the system 200, including the generator network 204.
[0070] The method 400 begins with the generator network 204 receiving raw input data 202 in operation 401. As described herein, the raw input data 202 can be provided in a variety of file forms and include one or more rows and one or more columns that create data fields that include text data, numeric data, or a combination of text and numeric data. In a specific example, the raw input data 202 can be a data set X that includes m rows and n columns.
[0071] In operation 403, data generator 206 generates synthetic data 214 based on or corresponding to the original input data 202. The data generator 206 may generate the synthetic data 214 in segments, such as row by row or column by column. By generating the synthetic data 214 in segments, the data generator 206 is able to receive more accurate feedback for each iteration of the synthetic data 214 and more effectively generate a complete synthetic data set that can be exported for use by an external ML model. The generator network 204 distorts the eigenvalue of the original input data 202 to introduce noise and creates synthetic data values similar to the original input data 202 without any meaningful reconstruction of the original data. The original input data 202 forms the state aspect of the deep reinforcement learning process.
[0072] As described above, the data set X includes X cat and X cont , which are the categorical part and the continuous part of the data set X respectively. The variables in X cat are one-hot encoded. Distortion is applied to the original input data 202 by passing the original input data 202 through a series of alternating affine and non-linear activation functions. The affine transformation has the linear form W.X + b, where W is the weight matrix and X is the input data. The non-linear transformation is the ReLU function, in the form of: ReLU(X) = X if X > 0 and ReLU(X) = 0 if X ≤ 0.
[0073] In some implementations, the original input data 202 includes text data in addition to numerical data, or the text data replaces the numerical data. For example, the text data may indicate a name, an address, or any other text data. To generate the synthetic data 214 corresponding to the text data, the data generator 206 utilizes a mapping or a hash function that generates a hash based on the original text data. Synthetic text data can be generated by embedding, where an m-dimensional input space is mapped to a k-dimensional embedding space such that k << m. In this way, the text features are hashed into a lower representation, and the same weight W k can be learned as part of the stochastic gradient descent.
[0074] In operation 405, the data generator 206 outputs the synthetic data 214. The output for the distorted input can be represented by G X . In some implementations, the generator network 204 includes several layers, each layer represented by L i , which apply an affine transformation and then a non-linear activation. The shape of each layer is the same as the previous layer, and each layer applies an affine transformation at each step and then a non-linear transformation. Thus, the synthetic data 214 G X can be represented as G X = ReLU(W k(ReLU(W k-1 (…ReLU(W0X+B0). Therefore, the output (i.e., synthesized data 214) has the same form as the input (i.e., original input data 202) and is distorted by applying several affine and nonlinear activations at different layers. The reconstructed input has a fully connected layer with a single node at the output. This output attempts to minimize the accuracy of the discriminator network 216. This output can be represented as Generator X =σ(W final .G X +b X ). As described herein, the synthetic data 214 is output and randomly labeled and provided to the discriminator network 216 along with the original input data 202.
[0075] In operation 407, the generator 204 receives the labeled data 230 from the processor 226. The labeled data 230 includes the synthesized data 214 and the label 228 indicating whether the synthesized data 214 is correctly identified as synthesized or incorrectly predicted as original data by the discriminator network 216. Figure 4B The process by which the discriminator network 216 generates predictions about the synthetic data 214 is described in more detail in the description of FIG.
[0076] In operation 409, the loss calculator 210 calculates a loss based on the received labeled data 230. In some implementations, the loss is a cross entropy loss of the generator network 204. The calculated loss indicates the accuracy of the synthesized data 214. The accuracy can be measured from 0 to 1, where a high loss (i.e., 1) indicates that the accuracy of the synthesized data 214 is low, and a low loss (i.e., close to 0) indicates that the accuracy of the synthesized data 214 is high. As described herein, the accuracy of the synthesized data 214 refers to the degree of similarity of the synthesized data 214 to the original input data 202. The lower the calculated loss, the more accurate or closer the degree of similarity of the synthesized data 214 to the original input data 202. In some implementations, the loss is defined as:
[0077]
[0078] For example, log(1) = 0. Therefore, in the case where actual = predicted, the loss becomes 0, indicating high accuracy and the synthesized data 214 is very similar to the original input data 202. However, in the case where actual = 1, the loss becomes 1, indicating low accuracy and the synthesized data 214 is not similar to the original input data 202.
[0079] In some implementations, the loss calculator 210 compares the calculated loss to a threshold value for determining whether the synthetic data 214 can be used to train the external ML model. In the event that the loss is not within the threshold value, the method 400 proceeds to operation 413. In operation 413, the parameter optimizer 212 optimizes the parameters used by the data generator 206 to generate the next iteration of the synthetic data 214. For example, the parameter optimizer 212 may back-propagate the loss by subtracting the gradient of the loss, thereby minimizing W in the direction opposite to the gradient. k , W final , H k and the values of all bias terms B.
[0080] In implementations where the loss is determined to be within the loss threshold, the synthetic data 214 is deemed usable for training the external ML model and the method 400 terminates. Although the synthetic data 214 is described herein as being usable for training the external ML model after one iteration where the loss is within the threshold, various implementations are possible. For example, in order to more accurately confirm that the iteration of the synthetic data 214 is sufficiently accurate, the loss of a single iteration of the synthetic data 214 may be required to be below the threshold a specific number of times. In some implementations, the iteration of the synthetic data 214 should have a loss below the threshold two, three, five, or any other suitable number of times determined by the processor 226. In other implementations, the iteration of the synthetic data 214 should indicate that the loss is below the threshold a specific percentage of times after passing through the discriminator network 216. For example, the iteration of the synthetic data 214 may pass through the discriminator network 216 a specific number of times, such as two, five, ten times, etc. In order to determine that the synthetic data 213 is accurate, in operation 411, the loss should be below the threshold a certain percentage of times.
[0081] Figure 4B An operational flow chart of a computer-implemented method for predicting original and synthetic data sets and performing deep reinforcement learning according to various examples of the present disclosure is shown. Figure 4B The operations shown in flowchart 450 are for illustration only and should not be construed as limiting. Various examples of operations may be used without departing from the scope of the present disclosure. The operations in flowchart 450 may be performed by one or more components of system 200, including discriminator network 216.
[0082] The method 450 begins with the discriminator network 216 receiving the original input data 202 and the synthetic data 214 based on the original input data 202 in operation 451. In some implementations, the synthetic data 214 and the original input data 202 are randomly assigned labels (e.g., 0 and 1) to facilitate masking from the discriminator network 216 which data set is original and which data set is synthetic. For example, the discriminator network 216 may receive the original data input 202 with a label of 0, i.e., data set X, and the synthetic data 214 with a label of 1, i.e., G XThe inputs to the discriminator network 216 can be represented as X and G X The connection of X, G X =[X|G X ],in It should be understood that X and G X The connection of X, G X =[X|G X ] is labeled as 1 for the pseudo example and 0 for the original example.
[0083] In operation 453, the data classifier 218 predicts which of the original input data 202 and the synthesized data 214 is the original data and which is the fake data. The data classifier 218 alternates affine and nonlinear activation functions to reduce the dimensionality to the last layer. The weights of the discriminator network 216 are represented by H k , where k is the number of layers in the discriminator network 216. The output of the data classifier 218 is the generated prediction. The prediction of the discriminator network 216 is labeled as 1 if the synthetic data is correctly identified, otherwise it is labeled as 0. The probability score between 0 and 1 indicates the confidence of the discriminator network 216 in making the correct prediction. The loss function is designed so that there is a high penalty when the prediction output of the discriminator network 216 is wrong.
[0084] In operation 455, the discriminator network 216 receives an indication from the processor 226 regarding the generated prediction result. The result may be provided in the form of a label 228. For example, the label may be a binary indication, such as 0 indicating that the prediction is correct, or 1 indicating that the prediction is wrong.
[0085] In operation 457, the loss calculator 222 calculates the loss of the generated prediction. The loss function of the discriminator network 216 is designed to have a larger loss if the discriminator network 216 mispredicts the original input data 202 and the synthesized data 214. For example, when the actual label is 1 and the discriminator network 216 generates a prediction of 0, the prediction is wrong. D The second term of is 0, and the first term becomes -1*log(0), that is, -1*-Inf=Inf, which is a very large value. Similarly, when the actual label is 0 and the discriminator network 216 generates a prediction of 1, the prediction is also wrong. Loss D The first term is 0, and the second term becomes -(1-0)*log(0), which is also infinite.
[0086] When both the actual label and the predicted label are 1, the discriminator network 216 correctly predicts which data is the synthetic data and which data is the original data. Dbecomes 0 because the second term is an irrelevant term and the first term is -1*log 1=-1*0=0, indicating no loss. Similarly, when both the actual label and the predicted label are 0, the discriminator network 216 correctly predicts which data is the synthetic data and which data is the original data. Here, the first term is 0 and the second term becomes 0, indicating no loss.
[0087] In operation 459, the parameter optimizer 224 optimizes the parameters of the data classifier 218 based on the calculated loss. For example, the parameter optimizer 224 may back-propagate the loss by subtracting the gradient of the loss, thereby minimizing W in the direction opposite to the gradient. k , W final , H k and the values of all bias terms B.
[0088] Figure 5 is a flowchart illustrating the operation of a computer-implemented method for deep reinforcement learning according to various examples of the present disclosure. Figure 5 The operations shown are for illustration only and should not be construed as limiting. Various examples of operations may be used without departing from the scope of the present disclosure. The operations in flowchart 500 may be performed by one or more components of system 200, including generator network 204, discriminator network 216, processor 226, and data exporter 236. When method 500 is performed, the first neural network and the second neural network, i.e., generator network 204 and discriminator network 216, are physically co-located or located within the same geographic range.
[0089] Flowchart 500 begins with data generator 206 generating a synthetic data set (e.g., synthetic data 214) based on an original data set (e.g., original input data 202) received by generator network 204. In some implementations, generator network 204 is referred to as a first neural network. The original input data 202 may be provided as a data set X including m rows and n columns. Synthetic data 214 is generated by distorting feature values of the original input data 202 to introduce noise, and creating synthetic data values that are close to the original input data 202 without any meaningful reconstruction of the original data, and the synthetic data may identify individuals, consumers, groups, etc., like the original input data 202.
[0090] In operation 503, the processor 226 provides the original input data 202 and the synthesized data 214 to the discriminator network 216. In some implementations, the discriminator network 216 is referred to as a second neural network. The provided original input data 202 and the synthesized data 214 may be labeled to mask which is the original data and which is the synthesized data version. For example, the processor may randomly assign a label to each of the original data 202 and the synthesized data 214, and provide the labeled original data 202 and the synthesized data 214 to the discriminator network 216. The label may include 0 or 1. In other words, either the original data 202 is labeled as 1 and the synthesized data 214 is labeled as 0, or the original data 202 is labeled as 0 and the synthesized data 214 is labeled as 1.
[0091] In operation 505, the discriminator network 216 generates predictions about which of the original input data 202 and the synthesized data 214 is the original data, and which of the original input data 202 and the synthesized data 214 is the synthesized data. The discriminator network 216 generates predictions by alternating affine and non-linear activation functions, as described herein, which reduces the dimensionality to the last layer. The generated predictions are output to the processor 226.
[0092] In operation 507, the processor 226 determines whether the generated prediction is correct. In other words, the processor 226 determines whether the discriminator network 216 correctly identifies the original data set and the synthetic data set. In the implementation of generating a correct prediction, the flowchart 500 continues to operation 509. In operation 509, each of the ML models 208 and ML models 220 calculates the loss by subtracting the gradient of the calculated loss and minimizing the value in the opposite direction of the gradient. In operation 511, each of the generator network 204 and the discriminator network 216 updates parameters to more accurately generate the synthetic data 214 and generate predictions, respectively. Because the discriminator network 216 correctly identifies the original input data 202 and the synthetic data 214, the synthetic data 214 is considered not realistic enough to be used for training of the external ML model. The flowchart 500 then returns to operation 501 and generates an updated synthetic data set based on the updated parameters.
[0093] In an implementation where an erroneous prediction is generated, the flowchart continues to operation 513. In operation 513, each of the ML models 208 and ML models 220 calculates a loss by subtracting the gradient of the calculated loss and minimizing the value in the opposite direction of the gradient. In operation 515, each of the ML models 208 and ML models 220 updates parameters to more accurately generate the synthetic data 214 and generate predictions, respectively. Because the discriminator network 216 misidentified the original data and the synthetic data 214, the synthetic data 214 is deemed to be sufficiently realistic to be used for training of an external ML model. The flowchart 500 then continues to operation 517, where the data exporter 236 exports the synthetic data 214 to the external ML model for use in training the ML model.
[0094] The following provides pseudo code corresponding to flowchart 500 in some implementations. The parameter values of the discriminator network and the generator network are randomly initialized, that is, all W k , W final , H k The values of and all bias terms B are initialized to random values.
[0095] num_epochs = 100
[0096] batch_size=32
[0097] num_batches=num_observations / batch_size
[0098] for epoch in range(1:num_epochs):
[0099] for batch in range(1:num_batches):
[0100] 1. Calculate the forward propagation of the generator network:
[0101] G X =ReLU(W k (ReLU(W k-1 (…ReLU(W0X+B0)
[0102] Generator X =σ(W final .G X +b X )
[0103] 2. Input to the discriminator network as X and G X Connection, X, G X =[X|G X ]
[0104] 3. Calculate the forward propagation for the discriminator:
[0105]
[0106] 4. Calculate the discriminator loss:
[0107]
[0108] 5. Calculate the generator loss:
[0109]
[0110] 6. At the same time, about the unknown W k , W final and H k And the loss of all bias terms B is derived
[0111] 7. By subtracting the gradient of the loss and thereby minimizing W in the opposite direction of the gradient k , W final and H k And the value of all bias terms B to backpropagate the loss Additional Examples
[0112] Some examples herein are directed to a deep reinforcement learning method, as shown in a flowchart 500. The method (500) includes generating (501) a synthetic data set (214) based on an original data set (202) by a first neural network (204) implemented by a processor (226), providing (503) the original data set and the generated synthetic data set to a second neural network (216) implemented on the processor, and generating (505) a prediction identifying the original data set and the generated synthetic data set by the second neural network, and deriving (517) the generated synthetic data set based at least in part on the prediction of the incorrectly identified generated synthetic data set.
[0113] In some examples, the method also includes executing, by the first neural network, a machine learning model to calculate (509) a loss for the generated synthetic data and to update (511) parameters of the generated synthetic data set based at least in part on the prediction of the correctly identified generated synthetic data set, and generating, by the first neural network, a second synthetic data set.
[0114] In some examples, the method also includes updating, by the first neural network, a parameter by subtracting the calculated gradient of the loss and minimizing a value in the opposite direction of the gradient.
[0115] In some examples, the method also includes executing, by the second neural network, a machine learning algorithm to calculate (513) a loss for the second neural network and update (515) parameters of the second neural network based at least in part on the predictions of the synthetic data set generated by the misidentification.
[0116] In some examples, the method also includes updating the parameters by the second neural network by subtracting the calculated gradient of the loss and minimizing the value in the opposite direction of the gradient.
[0117] In some examples, the method also includes randomly assigning a label to each of the original dataset and the generated synthetic dataset, and providing the labeled original dataset and the generated synthetic dataset to a second neural network.
[0118] In some examples, to generate the predictions, the method further includes alternating affine and non-linear activation functions.
[0119] In some examples, to generate the synthetic dataset, the method also includes distorting feature values of the original dataset to introduce noise.
[0120] In some examples, the first neural network and the second neural network are physically co-located or located in the same geographic area.
[0121] Although described in conjunction with the example computing device 100 and system 200, the examples of the present disclosure can be implemented with many other general or special computing system environments, configurations or devices. Examples of known computing systems, environments and / or configurations suitable for various aspects of the present disclosure include, but are not limited to, servers, smart phones, mobile tablets, mobile computing devices, personal computers, server computers, handheld or laptop devices, multiprocessor systems, game consoles, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, wearable or accessory mobile computing and / or communication devices (such as watches, glasses, headphones or earphones), network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices and holographic devices, etc. Such systems or devices can accept input from users in any manner, including from input devices (such as keyboards or pointing devices), through gesture input, proximity input (such as through hovering) and / or through voice input.
[0122] Examples of the present disclosure may be described in the general context of computer executable instructions, such as program modules executed by software, firmware, hardware, or a combination thereof, of one or more computers or other devices. Computer executable instructions may be organized into one or more computer executable components or modules. In general, program modules include, but are not limited to, routines, programs, objects, components, and data structures that perform specific tasks or implement specific abstract data types. Aspects of the present disclosure may be implemented with such components or modules in any number and organizational form. For example, aspects of the present disclosure are not limited to specific computer executable instructions or specific components or modules shown in the figures and described herein. Other examples of the present disclosure may include different computer executable instructions or components with more or less functions than those shown or described herein. In examples involving general-purpose computers, when configured to execute instructions described herein, aspects of the present disclosure convert general-purpose computers into special-purpose computing devices.
[0123] As an example and not limitation, computer-readable media include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable, non-removable memory implemented by any method and technology, which are used for the storage of information (such as computer-readable instructions, data structures, program modules, etc.). Computer storage media are tangible and mutually exclusive with communication media. Computer storage media are implemented in hardware and do not include carrier waves and propagation signals. The computer storage media used for the purpose of this disclosure are not signals themselves. Example computer storage media include hard disks, flash drives, solid-state memories, phase change random access memories (PRAM), static random access memories (SRAM), dynamic random access memories (DRAM), other types of random access memories (RAM), read-only memories (ROM), electrically erasable programmable read-only memories (EEPROM), flash memory or other storage technologies, compact disk read-only memories (CD-ROM), digital versatile disks (DVD) or other optical memories, cassettes, tapes, disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible to a computing device. In contrast, communication media typically embodies computer readable instructions, data structures, program modules, etc. in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
[0124] The order in which the operations of the examples shown and described in the present disclosure are performed is not necessary and can be performed in different orders in various examples. For example, performing or running a specific operation before, simultaneously or after another specific operation is considered to be within the scope of the present disclosure. When introducing the elements of aspects of the present disclosure or their examples, the articles "one", "an", "the" and "said" are intended to mean that there are one or more elements. The phrases "comprise", "include" and "have" are intended to be inclusive and mean that in addition to the listed elements, there may be other elements. The phrase "exemplary" is intended to mean "a certain example". The phrase "one or more of the following: A, B and C" means "at least one A and / or at least one B and / or at least one C".
[0125] Having described aspects of the disclosure in detail, it will be apparent that modifications and variations are possible without departing from the scope of the disclosure as defined in the appended claims. As changes may be made to the above-described constructions, products, and methods without departing from the scope of the disclosure, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not limiting.
[0126] Although aspects of the present disclosure do not track personally identifiable information, examples have been described with reference to data monitored and / or collected from users. In some examples, the user may be provided with a notification of the data collection (e.g., through a dialog box or preference setting) and the user may be given an opportunity to consent or decline the monitoring and / or collection. Consent may be in the form of opt-in consent or opt-out consent.
[0127] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
[0128] It should be understood that the benefits and advantages described above may relate to one example, or may relate to several examples. These examples are not limited to examples that solve any or all of the problems described, nor are they limited to examples that have any or all of the benefits and advantages described. It will also be understood that the reference to the "one" term refers to one or more of these items.
[0129] The term “comprising” is used in this specification to indicate including the following feature(s) or behavior(s), and does not preclude the existence of one or more additional features or behaviors.
[0130] In some examples, the operations shown in the figures may be implemented as software instructions encoded on a computer-readable medium, hardware programmed or designed to perform the operations, or a combination of both. For example, aspects of the present disclosure may be implemented as a system on a chip or other circuit comprising a large number of interconnected conductive elements.
[0131] Unless otherwise noted, the execution or running order of the operations in the disclosed examples shown and described herein is not essential. That is, unless otherwise noted, the operations can be performed in any order, and the examples of the present disclosure may include additional operations or operations less than the present disclosure. For example, performing or running a specific operation before, simultaneously with, or after another operation is considered to be within the scope of the present disclosure.
Claims
1. A system for deep reinforcement learning, the system comprising: processor; a first neural network implemented on the processor; a second neural network implemented on the processor; as well as a memory storing instructions that, when executed by the processor, cause the processor to: controlling the first neural network to generate a synthetic data set based on the original data set, providing the original data set and the generated synthetic data set to the second neural network, controlling the second neural network to generate predictions identifying the original data set and the generated synthetic data set, and The generated synthetic data set is derived based at least in part on the prediction of the generated synthetic data set that was incorrectly identified.
2. The system of claim 1 , wherein the instructions further cause the processor to: Based at least in part on the prediction that the generated synthetic data was correctly identified, controlling the first neural network to execute a machine learning model to calculate a loss for the generated synthetic data set and update parameters; controlling the first neural network using the updated parameters to generate a second synthetic data set; as well as The first neural network updates the parameters by subtracting the calculated gradient of the loss and minimizing the value in the opposite direction of the gradient.
3. The system of claim 1 , wherein the instructions further cause the processor to: controlling the second neural network to execute a machine learning algorithm to calculate a loss for the second neural network and update parameters based at least in part on the predictions of the synthetic data set generated by the error identification; and The second neural network updates the parameters by subtracting the calculated gradient of the loss and minimizing the value in the opposite direction of the gradient.
4. The system of claim 1 , wherein the instructions further cause the processor to: Randomly assigning a label to each of the original dataset and the generated synthetic dataset, providing the labeled original data set and the generated synthetic data set to the second neural network, and The prediction generated by the second neural network is received.
5. The system of claim 1 , wherein the instructions further cause the first neural network to generate the synthetic data set by: The feature values of the original data set are distorted to introduce noise.
6. The system of claim 1, wherein the instructions further cause the second neural network to: The predictions are generated by alternating affine and non-linear activation functions.
7. The system of claim 1, wherein the first neural network and the second neural network are physically co-located or located within the same geographic area.
8. A computer-implemented method for deep reinforcement learning, the method comprising: generating, by a first neural network implemented on the processor, a synthetic data set based on the original data set; providing the original data set and the generated synthetic data set to a second neural network implemented on the processor; generating, by the second neural network, predictions identifying the original data set and the generated synthetic data set; as well as The generated synthetic data set is derived based at least in part on the prediction of the generated synthetic data set that was incorrectly identified.
9. The computer-implemented method of claim 8, further comprising: executing, by the first neural network, a machine learning model to calculate a loss for the generated synthetic data set and to update parameters for the generated synthetic data set based at least in part on the prediction that the generated synthetic data set is correctly identified; generating a second synthetic data set by the first neural network; as well as The parameters are updated by the first neural network by subtracting the calculated gradient of the loss and minimizing a value in the opposite direction of the gradient.
10. The computer-implemented method of claim 8, further comprising: executing, by the second neural network, a machine learning algorithm to calculate a loss for the second neural network and to update parameters for the second neural network based at least in part on the predictions of the synthetic data set generated by the error identification; as well as By the second neural network, the parameters are updated by subtracting the calculated gradient of the loss and minimizing the value in the opposite direction of the gradient.
11. The computer-implemented method of claim 8, further comprising: Randomly assigning labels to each of the original dataset and the generated synthetic dataset, and The labeled original data set and the generated synthetic data set are provided to the second neural network.
12. The computer-implemented method of claim 8, wherein generating the prediction further comprises: Alternating affine and non-linear activation functions.
13. The computer-implemented method of claim 8, wherein generating the synthetic data set further comprises: The feature values of the original data set are distorted to introduce noise.
14. The computer-implemented method of claim 8, wherein the first neural network and the second neural network are physically co-located or located within the same geographic area.
15. One or more computer storage memory devices containing executable instructions which, when executed by a processor, cause the processor to: Receive the original data set; controlling a first neural network to generate a first synthetic data set based on the original data set; Providing the original data set and the generated first synthetic data set to a second neural network; controlling the second neural network to generate a first prediction identifying the original data set and the generated first synthetic data set; The first prediction of the synthetic data set generated based at least in part on correctly identifying: controlling the first neural network to execute a first machine learning (ML) model to calculate a loss for the generated first synthetic data set, update parameters, and generate a second synthetic data set using the updated parameters, wherein the generated second synthetic data set is a secondary iteration of the first synthetic data set generated based on the original data set, and controlling the second neural network to execute a second ML model to calculate a loss for the second neural network and update parameters; Providing the original data set and the generated second synthetic data set to the second neural network; controlling the second neural network to generate a second prediction identifying the original data set and the generated first synthetic data set; as well as The generated second synthetic data set is derived based at least in part on the second prediction of the second synthetic data set generated by the error identification.
Citation Information
Patent Citations
Pre-training system for self-learning agent in virtualized environment
US20200065673A1
Distributed generative adversarial networks suitable for privacy-restricted data
WO2021242956A1
