Dual machine learning pipe for converting data and

Abstract

An end-to-end cloud-based machine learning platform provides computer simulation recommendations. A data lineage is generated for all converted data for feature extraction, conversion, and loading (ETL) for generation to a machine learning model. The data is used to understand the performance of the simulation recommendation model. To this end, the platform provides the life cycle of the converted data and compares it to the life cycle of the user interaction, with understanding the performance of the recommendations. By comparing the two lifecycles, recommendations can be returned as to which models are related and which models are not related.

Description

[0001] This application is a divisional application of the invention whose applicant is Sony Interactive Entertainment LLC, whose application date is July 8, 2022, whose application number is 202010649979.0, and whose name is “Dual machine learning pipeline for converting data and optimizing data conversion”. Technical Field

[0002] The present application generally relates to technically creative and unconventional solutions that are necessarily rooted in computer technology and produce specific technical improvements. Specifically, the present application relates to an end-to-end cloud-based machine learning platform that provides personalized game player predictions. Background Art

[0003] Machine learning can be divided into different tasks, such as supervised learning, unsupervised learning, deep learning, reinforcement learning, and recommender systems, which can be used in a variety of computer applications, such as online personalized recommendations, computer vision, speech recognition, Internet search engines, personal assistants, and self-driving cars.

[0004] As understood herein, machine learning systems resist analysis from formal methods. Verification techniques (such as model checking or theorem proving) cannot be applied to machine learning systems due to their complexity. Therefore, the behavior of machine learning models must be determined empirically through controlled experiments to support, refute, or verify market hypotheses.

[0005] The importance of validating market hypotheses in computer simulations such as computer games is complicated by the fact that game developers are constantly innovating by reimagining how people play games, while gamer behavior in how they interact, consume, and purchase games is constantly changing. This means that many machine learning personalization models must be developed and validated in a relatively short life cycle. Summary of the invention

[0006] As understood herein, machine learning is a serialized process that involves identifying, obtaining, and preparing raw data sets, feature engineering and generating extraction, transformation, and loading (ETL) services that convert raw data into features of machine learning models, and designing and developing machine learning models on their own. The model must be "trained" in computationally intensive tasks, so computationally intensive tasks involve distributed computing architectures running on a large number of central processing units (CPUs), graphics processing units (GPUs), and emerging optimized chips for running specific algorithms such as deep learning. Then, the performance and latency of the model must be evaluated. After the model is trained, its prediction quality and the time it takes for the model to provide the prediction need to be evaluated. Then, the "good" model can be deployed in a production environment to provide inferences in online services (such as web stores, user news feeds, in-game activities or skill systems) to help game users play the game better.

[0007] As also understood herein, in order to free data scientists from having to build their own modeling tools to automate these workflows, machine learning pipelines can be used to automate the above workflows from start to finish. This end-to-end pipeline transforms raw data into features, which are then transformed into trained models and deployed into production to generate inferences. The machine learning pipeline ensures reliable, uniform, and repeatable data transformations for generating feature ETL, training models, and providing production model inferences.

[0008] The present principles further understand that machine learning pipelines can have two different types of data processing infrastructure. The first is an infrastructure that supports high data throughput. Machine learning models must be trained using extremely computationally intensive algorithms, such as batch gradient descent in particular. The purpose of gradient descent is to minimize the cost or loss function between the labeled data of the training set and the model predictions. Finding the minimum of the cost function requires a large number of computational iterations, especially for training deep learning systems. Therefore, the training service of the pipeline is typically run on a large number of CPUs, GPUs, and emerging optimized chips for running specific algorithms such as deep learning, and the training can be evenly distributed across a large number of clusters. Although some models must be trained online for some use cases, the training service is primarily an offline pipeline.

[0009] The second type of data processing infrastructure must support low data latency. When a model is trained, it serves its predictions to a large number of requests with very low latency in the range of a few milliseconds. Inference serving is typically online, in that model predictions are served to applications on demand, but can also be offline, in that model predictions are scheduled to run weekly, daily, or hourly.

[0010] The present principles address the technical needs recognized herein to remove human decisions from initial model creation, which are decisions about which model to train and when to train it, which model to deploy in production and why, and which model succeeds in experiments and why. Further, the present principles allow non-data scientists to train models by specifically automating the search for model hyperparameters and selecting algorithms known to work with open source datasets.

[0011] Thus, this principle involves how machine learning pipelines transform data. Raw user interaction data and raw item data are transformed into generated features. Models use these features to learn how to provide personalized software experiences, such as recommending computer games to a given user, recommending tips to a user to play a game better, recommending gaming activities to a user, etc. Model predictions are used to evaluate various experiments on these recommendations. Each experiment has multiple variables.

[0012] Thus, a device includes at least one processor and at least one computer memory, the at least one computer memory being non-transient and including instructions executable by the processor to receive data representing inputs made by a plurality of simulated players to a computer simulation. The instructions are executable to input the data to a training service of a first pipeline of a model generation computerized service to train a plurality of personalized user experience models, and to generate predictions of those personalized user experiences using an inference service of the first pipeline. The instructions are also executable to provide those model predictions from the inference service to an experimentation service of the first pipeline to test the recommended personalized user experiences. Further, the instructions are executable to provide a training service of a second pipeline of a model generation computerized service to train a new model using the output generated by the experimentation service to select a subset of the personalized user experience models from the first pipeline using at least one key performance indicator (KPI). The instructions are executable to provide this new model trained in the training service of the second pipeline to the inference service of the second pipeline to deliver a recommendation for a new personalized user experience model to be trained in the training service of the first pipeline.

[0013] In an example, the instructions may be executable to learn how to classify the model selected from the first pipeline using the training service of the second pipeline, and provide those classifications to the second pipeline. In such an example, the instructions may be executable to execute a reinforcement learning model (RL) of the second pipeline to identify at least the first model as a "good" model at least in part by maximizing a reward for a first model prediction from the first pipeline. The maximization may be performed at least in part by making a recommendation associated with time "t" equal to the product of the reward associated with the time "t" plus a discount factor and a recommendation associated with time t+1.

[0014] In certain implementations, the instructions may be executable to execute an evolution strategy model (ES) that may use the selected model classified in the second pipeline to identify future models to be trained by the first pipeline. The instructions may be executable to execute the ES to learn model metadata based on the classification and generate the future models based at least in part on their metadata.

[0015] In another aspect, a system includes a first plurality of computers implementing a first pipeline for training models and inferring those models; a second plurality of computers implementing a second pipeline for receiving the models from the first pipeline, identifying at least a first model of the models from the first pipeline as a good model, and feeding a new model back to the first pipeline to enable the first pipeline to generate a new model.

[0016] In another aspect, a method includes: using a first computerized pipeline for model training and model inference; using a second computerized pipeline to identify at least one best model from models deployed in an inference service of the first pipeline; and feeding information associated with the best model back to the first pipeline. The method includes outputting model recommendations using at least the best model from the models of the first pipeline that provide computer simulation recommendations.

[0017] The details of the present application, both as to its structure and operation, may be best understood by reference to the accompanying drawings, in which like reference numerals refer to like parts, and in which: BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a block diagram of the network architecture of a cloud-based machine learning platform in accordance with the present principle;

[0019] Figure 1A is a block diagram of a data center architecture that complies with this principle;

[0020] Figure 2 is a schematic diagram illustrating an automated workflow of a dual machine learning pipeline;

[0021] Figure 3 is a schematic diagram showing data transformations running on two pipelines;

[0022] Figure 4 is a schematic diagram showing feature metadata stored in a feature memory and model metadata stored in a model memory of a first pipeline P1;

[0023] Figure 5 is a schematic diagram showing the communication flow between two pipelines using RL, CL and ES models;

[0024] Figure 6 is a flow chart of exemplary logic associated with an RL model algorithm;

[0025] Figure 7 is a schematic diagram showing the RL model architecture of the second pipeline P2;

[0026] Figure 8 is a schematic diagram showing an RL deep Q-network;

[0027] Fig. 9 is a schematic diagram showing the classification of models provided by the model CL in the second pipeline P2;

[0028] Fig.10 is a schematic diagram showing the ES model architecture of the second pipeline P2; and

[0029] Fig.11 is a schematic diagram showing the ES deep learning system. DETAILED DESCRIPTION

[0030] The present disclosure relates generally to a computer ecosystem including aspects of a network of consumer electronic (CE) devices such as, but not limited to, distributed computer gaming networks, augmented reality (AR) networks, virtual reality (VR) networks, video broadcasting, content delivery networks, virtual machines, machine learning, and artificial neural network applications.

[0031] The system herein may include server and client components connected via a network so that data can be exchanged between the client and server components. The client component may include one or more computing devices, including AR headsets, VR headsets, game consoles (such as Sony ) and related motherboards, game controllers, portable televisions (e.g., smart TVs, Internet-enabled TVs), portable computers (such as laptops and tablet computers), and other mobile devices (including smart phones and additional examples discussed below). These client devices can operate in a variety of operating environments. For example, some of the client computers may use, for example, Orbis or Linux operating systems, operating systems from Microsoft, or Unix operating systems, or operating systems produced by Apple or Google. These operating environments can be used to execute one or more programs / applications, such as browsers made by Microsoft or Google or Mozilla, or other browser programs that can access websites hosted by Internet servers discussed below. In addition, an operating environment according to the present principles can be used to execute one or more computer game programs / applications and other programs / applications that employ the present principles.

[0032] The server and / or gateway may include one or more processors executing instructions that configure the server to receive and transmit data over a network such as the Internet. Additionally or alternatively, the client and server may be connected via a local intranet or a virtual private network. The server or controller may be provided by a game console and / or one or more motherboards thereof (such as a Sony ), personal computers, etc.

[0033] Information may be exchanged between the client and the server over the network. To this end and for security, the server and / or client may include firewalls, load balancers, temporary storage and proxies, and other network infrastructure for reliability and security. One or more servers may form a device that implements a method according to the present principles to provide network users with a secure community such as an online social networking site or a video game site to communicate crowdsourced.

[0034] As used herein, instructions refer to computer-implemented steps for processing information in a system. Instructions may be implemented in software, firmware, or hardware, and include any type of programmed steps undertaken by components of the system.

[0035] The processor may be any conventional general purpose single chip processor or multi-chip processor capable of executing logic with the aid of various lines such as address, data and control lines, as well as registers and shift registers.

[0036] The software modules described by the flowcharts and user interfaces herein may include various subroutines, programs, etc. Without limiting the present disclosure, the logic stated as being performed by a particular module may be redistributed to other software modules and / or combined together in a single module and / or made available in a sharable library.

[0037] As indicated above, the present principles described herein may be implemented as hardware, software, firmware, or a combination thereof; accordingly, the illustrative components, blocks, modules, circuits, and steps are described in terms of their functionality.

[0038] In addition to what has been mentioned above, the logic blocks, modules and circuits described below may be implemented or executed with a general purpose processor, a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic devices designed to perform the functions described herein (such as application specific integrated circuits (ASICs), discrete gate or transistor logic, discrete hardware components or any combination thereof). The processor may be implemented by a combination of a controller or a state machine or a computing device.

[0039] The functions and methods described below can be implemented in hardware circuits or software circuits. When implemented in software, the functions and methods can be written in appropriate languages ​​such as, but not limited to, Python, Scala, Java, C#, or C++, and can be stored on or transmitted by computer-readable storage media, such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disk storage (such as digital versatile disk (DVD)), disk storage or other magnetic storage devices including removable thumb drives, etc. A computer-readable medium can be established by connection. Such connections can include, for example, hard-wired cables, including optical fibers and coaxial cables, and digital subscriber lines (DSL) and twisted pairs. Such connections can include wireless communication connections, including infrared and radio.

[0040] The components included in one embodiment may be used in other embodiments in any appropriate combination. For example, any of the various components described herein and / or depicted in the accompanying drawings may be combined, interchanged or excluded from other embodiments.

[0041] “A system having at least one of A, B, and C” (similarly, “a system having at least one of A, B, or C” and “a system having at least one of A, B, C”) includes systems having only A, only B, only C, both A and B, both A and C, both B and C, and / or both A, B, and C, etc.

[0042] Reference now Figure 1, an exemplary system is shown, which may include one or more of the exemplary devices mentioned herein for performing the techniques described in this document according to the present principles. As shown, the system may include one or more regional data centers 10, each of which includes one or more servers with accompanying processors and solid-state memory and disk storage. The servers establish a cloud-based system for executing the above-mentioned machine learning platform in which machine learning models are executed on multiple data centers. The regional data center communicates with one or more computerized front-end devices 18 (such as a computer executing a web browser (laptop, PC, etc.), a mobile application executed on a mobile device, or a computer simulation console (such as a game console)) via a wired link and / or a wireless link 16.

[0043] Therefore, if Figure 1 As shown, a dual machine learning pipeline is provided from the "cloud". The machine learning pipeline is executed on a large number of computer clusters in a cloud backend hosted by a regional data center 10. All model training and model inference occur on the backend. Model predictions are provided to front-end applications, such as any of the front-end applications running on a device 18 including a web browser, a game console, or a mobile application. All front-end applications communicate with the cloud backend through an Internet network 16. Each cluster may include a large number of computer servers physically located in one or more server farms, which are located in one or more geographically dispersed data centers.

[0044] A typical architecture may include a large number of regional data centers distributed around the world, with one or more regional data centers located in North America, EMEA, Japan, and other parts of the world.

[0045] like Figure 1A As shown, each data center may include one or more load balancers 20 to manage network requests from front-end applications 18 to multiple server farms 23, and one or more threat detection and firewall tools 22 to protect the servers 25 from external security threats. Although most data is mobile and generated and can be stored in large file systems (such as Apache Hadoop Distributed File System (HDFS)) in various data formats (such as Apache Parquet) that can be easily accessed and manipulated by clusters in the server farm, some data may be static and physically stored on a network attached storage device (NAS) 24 in the data center.

[0046] This principle fully automates the machine learning pipeline. The automation is not only horizontal, for example, the pipeline automates the workflow from model training to model inference and from model inference to model experimentation, but also vertical, for example, it eliminates the need for a team of data scientists to decide which model to train, which model to deploy in production, and which model to experiment with.

[0047] As shown in the accompanying figure and described more fully below, the development of machine learning models includes many workflows, including acquiring and preparing data, feature engineering and ETL, designing models, training models on large server clusters located in data centers that provide machine learning cloud services, evaluating model performance and latency, and deploying models in production as a service.

[0048] For this reason, Figures 2 to 5 As shown, two machine learning pipelines (called P1 and P2) provide model training and model inference without human intervention. The first pipeline P1 generates a model for personalizing the game player experience.

[0049] like Figure 2 As shown, the first pipeline P1 includes a training service 200 that can be online or offline. In some embodiments, the training service 200 includes multiple computers arranged in a distributed computer architecture that provides parallel processing for computationally intensive training using large data sets. Figure 3 As shown in 300, the training service 200 of the first pipeline P1 uses one or more computers that execute machine learning algorithms to learn how to provide recommendations for personalized gaming experience. The training service 200 provides feature generation ETL services and model training services, and the model training service uses a distributed computing architecture to parallelize computationally intensive training tasks with large data sets.

[0050] The first pipeline P1 also includes an online or offline inference service 202. The inference service 202 includes multiple computers arranged in a high-performance architecture to ensure the lowest deterministic latency of model predictions. In other words, the inference service 202 quickly processes the trained model from the training service 200, as further described below. Figure 3 As shown at 302 in , the inference service 202 provides model predictions.

[0051] The model predictions of the inference service 202 are sent to the experiment service 204 in the first pipeline P1 to Figure 3 Experiments are performed on those predictions as shown in 304. Experimentation service 204 includes a computer with an output device that provides a user interface to define experimental hypotheses and input key performance indicators (KPIs) and metrics for the experiments.

[0052] The output of the first pipeline P1 is sent to Figure 2 The output of the first pipeline P1 includes metadata about the models in training and inference, as well as metadata about the features generated from the ETL. As further described below, the reinforcement learning model (RL) learns in the training service 206 of the P2 pipeline how to select the "best" model from the P1 pipeline for experimentation, such as Figure 3 As shown in 306.

[0053] As explained further below, the classification is learned at state 307, the output of which is sent to an online or offline inference service 208 in the P2 pipeline, which employs one or more computers executing machine learning algorithms to be used by the various models further described below to learn which models to train and provide, such as Figure 3 As shown in 308 , the indication of 308 is fed back to the training service 200 of the P1 pipeline.

[0054] Thus, the P2 pipeline has both a training service and an inference service. These two services are used by the three P2 models that will be described briefly, namely the RL model plus the CL model and the ES model. Each model uses the training service and the inference service in order to be trained and in order to provide its predictions. For example, the RL model selects the best model, the CL model classifies the model selected by the RL model, and the ES model recommends the model to the first pipeline P1 for training. Thus, the RL model provides its predictions to the CL model, which in turn provides its predictions to the ES model. The ES model provides its predictions to the first pipeline P1.

[0055] Therefore, it can be understood that the second pipeline P2 generates a model that helps P1 generate a better model.

[0056] Considering the above points, it can be further understood that the data transformation in the first pipeline P1 can be considered as the data plane of the entire machine learning pipeline including both pipelines P1 and P2, and the second pipeline P2 that optimizes the data transformation from the first pipeline P1 can be considered as the control plane of the entire machine learning system. In fact, the second pipeline P2 is a superposition of the first pipeline P1.

[0057] As further described below, the model from the first pipeline P1 is trained using recommended system techniques to provide personalized user experience, such as logistic regression, matrix factorization, factorization machine, deep learning, and reinforcement learning. The model from the second pipeline P2 is trained using reinforcement learning, clustering, and evolutionary strategy techniques.

[0058] Regarding the experimental service 204 of the first pipeline P1, a large number of models are subjected to experiments to test multiple personalized recommendations for different user segments to evaluate the user's response to those various types of recommendations. The system is capable of providing a large number of experiments simultaneously or sequentially. Each experiment will also have multiple variables. Each experiment usually focuses on testing a specific hypothesis, such as user segmentation, game features, recommendation background or recommendation environment. If necessary, these variables can be run simultaneously. Multiple models can be proposed for each variable, each model providing personalized recommendations. All models of a given experiment should output their recommendations within a given time interval. It is also expected that the system is well scaled to provide multiple experiments with multiple variables online at the same time, and multiple models can be proposed for the variables. It should be noted that the requirements for proportions can change at any time according to the amount and speed of experimental data.

[0059] The success of an experimental hypothesis can be quantitatively measured by a key performance indicator (KPI). Generally, a primary KPI is chosen that is related to the long-term business goal. This primary KPI can have multiple secondary metrics to capture the entire scope of the experiment in order to analyze what exactly happened and why it happened. Secondary metrics can often be used by the experimental product analyst to provide further analysis about the experiment to the market analyst. Figure 4 Additional details of the above pipelines related to features and model metadata are provided. The first pipeline P1 generates a model and deploys the model in production and for experiments, and automates the end-to-end machine learning workflow "horizontally". Figure 4 As shown, the training service 200 of the first pipeline generates feature metadata in the feature data storage 400, and the training service 200 and the inference service 202 generate model metadata in the model data storage 402. Raw data (such as user data, game data) and user interactions (such as user-game interactions (play, purchase, view, stay, like, comment, forward, etc.) and user-user interactions (friends, follow, share, etc.)) are used to generate feature ETL.

[0060] The characteristics of the first pipeline P1 may be generated offline from data in a data store or online while streaming data from a publish and subscribe source.

[0061] As described above, the training service 200 of the first pipeline P1 generates feature ETL for personalized gaming experience models going to the inference service 202. The training service 200 trains those personalized gaming experience models online or offline and deploys them in a production environment in the inference service 202.

[0062] After training the models in the training service 200, the inference service 202 of the first pipeline P1 provides inferences of the personalized gaming experience models, after which those models are ready for experimentation in the experimentation service 204. It should be noted that the first pipeline P1 is able to provide model predictions online when online applications request predictions in real time or offline, and make those predictions available in the data store of the offline application.

[0063] The first pipeline P1 generates model and feature metadata. Feature metadata is generated after each ETL is generated in the pipeline. Model metadata is generated after the model is trained in the training service 200 and after the model provides its predictions in the inference service 202.

[0064] The model metadata mentioned above may include the model algorithm, model architecture (especially for deep learning models such as convolutional neural networks (CNN) or recurrent neural networks (RNN)), model parameters or weights, model hyperparameters, model learning rates, and model features. Metadata may also include information about dependencies on the following: such as when the training or prediction of the model depends on one or more other models, the state of the model prediction in inference (because the model can provide online or offline model inference), and standard algorithm metrics, such as (for regression) root mean square error and (for classification) accuracy, recall, and F1 score (the harmonic mean of precision and recall).

[0065] Additionally, model metadata may include the relevance of the predictions to the recommendations as measured by a metric such as the "WPAE" metric set forth in the present assignee's US Patent Application Serial No. 16 / 424,193, which is incorporated herein by reference.

[0066] Feature metadata, on the other hand, can include standard statistical measures (mean, average, maximum, minimum, and standard deviation) as well as the feature's relationship to other features and models.

[0067] As described above, the second pipeline P2 helps the first pipeline P1 produce better models by helping P1 generate better models to "vertically" automate the end-to-end machine learning workflow. Typically, the second pipeline P2 learns how to select the best model from the experimental KPIs for experiments through reinforcement learning. Models are classified according to experiments. The second pipeline P2 also learns how to recommend which model the first pipeline P1 should train by recommending model metadata via evolution strategies, and how to select the best model from the experimental KPIs for experiments via reinforcement learning.

[0068] Figure 5Further explanation. The training service 206 and the inference service 208 of the second pipeline P2 can be used by the online or offline reinforcement model (RL) 502 that provides the first data transformation. In addition, the second data transformation (classification at 307 above) is provided by the clustering model CL 504, and the third data transformation is provided by the evolution strategy model ES 506, and both data transformations are provided in the above-mentioned training service 206 and inference service 208.

[0069] RL 502 analyzes the best performing model from the models in production from the first pipeline P1 to conduct experiments. In an exemplary non-limiting embodiment, reinforcement learning can be implemented as described in "Playing Atari with Deep Reinforcement Learning" by Mnih et al., published in www.arXiv.org under the following number 1312.5602, which is disclosed in the instant paper history and incorporated herein by reference.

[0070] The first pipeline P1 shares all its models deployed in its production environment for inference with the RL 502 of the second pipeline P2. And, for each model, the RL 502 is able to access the model metadata. The first pipeline P1 also shares each experiment run on its experiment service with the RL 502 of the second pipeline P2, especially the number of experimental KPIs and variables.

[0071] The RL 502 performs the experiment by analyzing the best performing model among the models in production from the first pipeline P1. In one embodiment, the RL 502 uses a Markov decision process (MDP) defined by the following parameters:

[0072] The environment of RL is the experiment, the action of RL is the selection of models for the experimental variables, the state of RL is the state of its model selection for the experiment, the reward of RL is measured by increasing the KPI of the experiment, the policy of RL is to recommend the model that produces the highest KPI for the experiment, and the Q-value of RL is the expected cumulative reward from selecting models for each variable of a given experiment.

[0073] The model selection of RL 502 of the second pipeline P2 maximizes the discounted future reward starting from time point t, which is expressed as:

[0074] R t =r t +r t+1 +…+r t+i +…+r t+n

[0075] Assuming γ is a discount factor chosen to find a suitable balance between exploration (e.g., taking suboptimal actions immediately to collect additional experimental information) and exploitation (e.g., maximizing future rewards), the discounted future reward can be expressed as:

[0076] R t =r t +γR t+1

[0077] Brief reference Figure 6 To explain RL 502, at block 600, assume that the RL agent is in state "s". Moving to block 602, it performs action "a" in this state. Proceeding to block 604, it observes a reward r from its action a. t , at block 606, the RL is moved to a new state "s'", which generates a new action "a'" at block 608 to obtain a new Q-value at block 610, which in one implementation is according to the equation:

[0078] Q(s,a)=r t +γmax a 'Q(s',a')

[0079] Therefore, the Q-learning algorithm can be stated as:

[0080] Arbitrarily initialize Q(s,a), observe the initial state s, repeat, select and execute action a, observe the reward r and the new state Q(s,a) = Q(s,a) + α(r t +γmaxa'Q(s',a')–Q(s,a)), s=s' until termination.

[0081] In the above equation, "α" represents the learning rate, which controls how much of the difference between the previous Q-value and the newly proposed Q-value is taken into account. The Q-value of RL 502 can be estimated using a function approximator that implements a deep Q-learning algorithm. The function approximator can be, for example, a deep Q-network, such as a deep learning system with parameters θ such that:

[0082] Q(s,a;θ)~Q(s,a).

[0083] The deep Q-network can be a deep neural network with three hidden layers and ten features. The features of the neural network can be model metadata as previously defined, i.e., model algorithm, model architecture, model parameters, model hyperparameters, model features, model learning rate, model dependencies, inference status, model metrics, model WPAE. The output of the neural network is a Q value: Q(s,a).

[0084] It should be understood that the loss function used for network training can be given by:

[0085]

[0086] ○Among them is the expected value

[0087] ○ And among them

[0088] And the training parameters θ of the network i The gradient update is:

[0089] ●

[0090] Network parameters θ I Can be initialized randomly.

[0091] A deep Q-network can be trained using experience replay that stores all transitions (s, a, r, s'). When training the network, a random sample from the replay memory is used to replace the most recent transition.

[0092] Through experience replay, the Q network algorithm becomes:

[0093] Initialize playback memory M;

[0094] Initialize the Q-value function with random weights;

[0095] Observe the initial state "s";

[0096] repeat;

[0097] Action "a" is chosen by selecting a random action with probability ε, otherwise

[0098] Choose a = arg maxa'Q(s,a');

[0099] Execute action "a";

[0100] Observe the reward r and the new state s';

[0101] Store the experience (s, a, r, s') in the replay memory M;

[0102] Sample random transitions (ss, aa, rr, ss') from the replay memory M;

[0103] Calculate the target for each mini-batch transformation;

[0104] If ss' is a terminal state, then uu=rr, otherwise uu=rr+γmaxa'Q(ss',aa');

[0105] Use (uu-Q(ss,aa)) 2 Used as loss to train the Q network;

[0106] s=s'

[0107] Until termination.

[0108] The deep Q network selects the best model from the first pipeline P1 for a given experiment. If the experimental product analyst requires multiple variables, after selecting a model from the models available in production in the first pipeline P1, the deep Q network selects the best model from the remaining models.

[0109] The number of variables may be entered by an experiment product analyst in a user interface of the experiment service 204 of the first pipeline P1 .

[0110] Figure 7 Details of an example RL 502 architecture are shown. As shown, an RL agent (deep Q-network) 700 selects a model 702 from the first pipeline P1 for a particular variable 704 of an experiment. The agent selects this model 702 to maximize the reward 708 of the increased KPI for the variable 704. Therefore, the RL agent 700 moves to a new state with a new model selection 706.

[0111] Figure 8 Schematically shows Figure 7 A deep Q network 700 is provided to the network 700 as described above, including a model algorithm 800, a model architecture 802, model parameters 804, model hyperparameters 806, model features 808, a model learning rate 810, model dependencies 812, an inference state 814, a model metric 816, and a model WPAE 818 for processing by a deep learning layer 820 as described above.

[0112] The RL 502 of the second pipeline P2 is linked to the evolution strategy model (ES) 506 of the second pipeline P2 through a classification model (CL) 504 for classifying model metadata according to experiments. Fig. 9 As shown, CL 504 clusters the best performing models selected by RL 502 according to the type of experiment 900 from the lowest to the highest KPI, as shown in FIG. Fig. 9 The classification can be performed by a clustering algorithm such as K-means or hierarchical clustering.

[0113] ES 506 then learns how to recommend which model to train by recommending model metadata via an evolution strategy. ES model 506 learns how to recommend which new models to train to the training service 200 of the first pipeline P1 from the models classified by the CL 504 according to the experiment. To do this, ES 506 samples the population of models classified by CL 504 and allows the best performing models to dictate the distribution of future generations of models that the first pipeline P1 should train.

[0114] ES 506 learns how to generate new models from the metadata of each model selected for experimentation by RL 502, in particular how to recommend model features, recommend model algorithms, recommend model architectures especially for deep learning models, recommend model parameters, hyperparameters and learning rates, recommend model dependencies on other models, and recommend whether model inference should be offline or online.

[0115] Evolution strategy techniques differ from reinforcement learning techniques in that they do not compute Q-values ​​to find the policy that will generate the highest reward, and evolution strategy deep learning systems are trained without backpropagation. In the example, the evolutionary measure can be used in Salimans et al. www.arXiv.org The method is implemented using the principles of "Evolution Strategies as a Scalable Alternative to Reinforcement Learning" published under No. 1703.03864, which is disclosed in the instant file history and incorporated herein by reference.

[0116] Fig.10 and Fig.11 Details of an exemplary implementation of ES 506 are shown. ES 506 learns how to generate new model metadata from the model metadata of each model classified by CL 504 for a given experiment. This forms the basis of an initial policy 1000 of ES agent 1002 that can be implemented by a deep learning system. After obtaining this initial policy, ES 506 generates new models defined by its new metadata that can provide the same or better experimental KPIs.

[0117] More specifically and as Fig.10 As shown, the ES deep learning system may be an agent 1002, whose action 1004 would be to recommend a model to the first pipeline P1 based on the policy 1000. If the policy successfully increases the reward defined by the experiment KPI, the agent 1002 is rewarded as indicated by 1006. The policy is evaluated by the evaluation module 1008, which updates 1010 the policy and feeds this information back to the agent 1002.

[0118] As previously described, RL 502 works by using its deep Q-network to find a policy that will maximize the experiment KPI by computing Q-values ​​through exploration and exploitation. Conversely, given an initial policy, ES 506 generates a population of new policies by applying random perturbations to the weights of its deep learning network. It then evaluates all of these new policies, computing stochastic gradient estimates to explore which policy looks more likely to generate the highest reward.

[0119] Go to accordingly Fig.11, a deep learning system 1100 with four hidden layers may be used to model ES 506. The input of ES 506 may include metadata 1102 of each model classified by CL 504 for each experiment. ES 506 outputs metadata of the new model then trained by the training service 200 of the first pipeline P1.

[0120] The ES policy denoted as π0 recommends the model metadata to the training service 200 of the first pipeline P1. Using the model metadata, the training service 200 of the first pipeline P1 trains a new model generated from the metadata.

[0121] Assume F is the objective function, π θ is the ES strategy, θ i is the parameter of the ES deep learning system, and the ε sample is from the strategy π θ For the sample drawn, with standard deviation σ, we get:

[0122]

[0123] where stochastic gradient ascent is used to optimize θ using the following score function estimator: i :

[0124]

[0125] Therefore, the ES algorithm includes inputs including the initial ES strategy parameters θ i , objective function F, strategy sample ε i , population size n, standard deviation σ and ES learning rate α. The algorithm can be stated as:

[0126] For t = 0, 1, 2, ...

[0127] Sampling strategies;

[0128] Sample ε1,...ε n ~N(0,I)

[0129] Evaluation strategies;

[0130] Calculate the return value F i =F(θ t +σε i )where i=1,...,n

[0131] Update strategy

[0132] set up

[0133] Until convergence

[0134] ES506 shares all recommended model metadata with the first pipeline P1. In turn, the first pipeline P1 utilizes the model metadata to automatically generate training for a new model based on the metadata. The training service 200 of the first pipeline P1 is automated so that a new model can be trained using defined model features, defined model algorithms, defined model architectures, defined model parameters, hyperparameters and learning rates, defined dependencies of the model on other models, and model inference states defined as offline or online.

[0135] Therefore, the second pipeline P2 collaborates with the first pipeline P1 to produce better models for experimentation. At each cycle, RL502 selects a new model with better personalized recommendations from the first pipeline P1 for experimentation. ES 506 then learns how to define new model metadata that will outperform the previous model selected by RL 502 from the first pipeline P1, and shares with the first pipeline P1 the metadata of the model that will maximize any experimental KPI. The first pipeline P1 then trains new models defined by their metadata as recommended by ES 506. Through a large number of experiments, RL 502 gets better at learning how to select the best performing models, while ES 506 gets better at learning how to recommend models to be trained to P1's pipeline. The advantageous result is that the first pipeline P1 can train models and provide model inferences with the help of the second pipeline P2 without manual intervention.

[0136] It will be appreciated that while the present principles have been described with reference to certain exemplary embodiments, these embodiments are not intended to be limiting and that various alternative arrangements may be used to implement the subject matter claimed herein.

Claims

1. A device, comprising: at least one processor; as well as at least one computer memory, the at least one computer memory being not a transient signal and comprising instructions executable by the at least one processor to: receiving data representing inputs made to a computer simulation by a plurality of simulated players; inputting the data into a training service of a first pipeline of a model generation computerized service to train a plurality of recommendation models; using the inference service of the first pipeline to generate recommendations based on a model trained using the training service in the first pipeline; providing the output of the inference service to an experimentation service of the first pipeline to test the recommendations to select a subset of the models using at least one key performance indicator (KPI); Use the training service and inference service of the second pipeline to provide recommendations for models to be trained; as well as Providing recommendations for models to be trained generated by the second pipeline to the training service of the first pipeline.

2. The apparatus of claim 1, wherein the instructions are executable to: The model is classified in the second pipeline to generate a classification.

3. The apparatus of claim 2, wherein the instructions are executable to: A reinforcement learning model (RL) is executed to use the training service and the inference service of the second pipeline to identify at least the first model as a good model at least in part by maximizing a reward for first model predictions from the first pipeline.

4. An apparatus as described in claim 3, wherein the maximization is performed at least in part by making the recommendation associated with time "t" equal to the reward associated with the time "t" plus the product of the discount factor and the recommendation associated with time t+1.

5. The apparatus of claim 3, wherein the instructions are executable to: An evolution strategy model (ES) is executed to use the training service of the second pipeline and an inference service to identify a future model to be trained by the first pipeline using at least the first model identified by the training service of the second pipeline.

6. The apparatus of claim 5, wherein the instructions are executable to execute the ES to learn model metadata based on the classification; and The future model is generated based at least in part on the metadata.

7. The apparatus of claim 1, wherein the instructions are executable to execute at least one of the models to provide recommendations for new computer simulations to provide to simulation players.

8. A system, comprising: a first plurality of computers implementing a first pipeline for training a model and providing model predictions; as well as a second plurality of computers implementing a second pipeline for receiving the models from the first pipeline, identifying at least a first model of the models from the first pipeline as a good model, and feeding the first model back to the first pipeline to enable the first pipeline to generate a new model.

9. The system of claim 8, wherein the first plurality of computer access instructions are: receiving data representing inputs made to a computer simulation by a plurality of simulated players; Inputting the data into a training service of the first pipeline to train a plurality of recommendation models; using the inference service of the first pipeline to generate recommendations based on the model trained in the training service of the first pipeline; providing the recommendation to an experimental service to test the recommendation; as well as The output of the experimentation service is provided to the second pipeline to select at least the first model using at least one key performance indicator (KPI).

10. The system of claim 9, wherein the second plurality of computer access instructions are: providing output from the training service using the second pipeline to a model that uses the training service and the inference service of the second pipeline to provide recommendations for models to train; and The recommendation for a model to be trained is provided to the first pipeline.

11. The system of claim 10, wherein the instructions are executable by the second plurality of computers to: classifying the model learned by the training service using the second pipeline to generate a classification; and The classification is provided to a model of an inference service employing the second pipeline.

12. The system of claim 11, wherein the instructions are executable by the second plurality of computers to: A reinforcement learning model (RL) is executed in the second pipeline to identify at least the first model as a good model at least in part by maximizing a reward for the first model predictions from the first pipeline.

13. The system of claim 12, wherein the maximization is performed at least in part by making the recommendation associated with time "t" equal to the reward associated with the time "t" plus the product of the discount factor and the recommendation associated with time t+1.

14. The system of claim 12, wherein the instructions are executable by the second plurality of computers to: An evolution strategy model (ES) is executed in the second pipeline to identify a future model to be trained by the first pipeline using at least the first model identified by using the training service and the inference service of the second pipeline.

15. The system of claim 14, wherein the instructions are executable by the second plurality of computers to execute the ES to learn model metadata based on the classification; and The future model is generated based at least in part on the metadata.

16. The system of claim 10, wherein the instructions are executable by the second plurality of computers to execute at least one of the models to provide recommendations for new computer simulations to provide to simulated players.

17. A method comprising: training a predictive model using a first pipeline, the first pipeline being computerized; identifying at least one best model from among the predictive models of the first pipeline using a second pipeline, the second pipeline being computerized; feeding information associated with the best model back to the first pipeline; and At least a best model among the predictive models is used to output a recommendation, the recommendation comprising a computer simulated recommendation.

18. The method of claim 17, comprising executing a reinforcement learning model (RL) in the second pipeline to identify at least the best model at least in part by maximizing a reward for predictions of the first model.

19. The method of claim 18, wherein the maximization is performed at least in part by making the recommendation associated with time "t" equal to the reward associated with the time "t" plus the product of the discount factor and the recommendation associated with time t+1.

20. The method of claim 18, comprising executing an Evolution Strategy model (ES) in the second pipeline to use at least the best model to identify future models to be trained by the first pipeline.

Images