Risk assessment techniques for controlling access to computing systems using dynamically selected machine learning models

Abstract

A system can generate a risk assessment associated with a target entity. The system can receive a request for a risk indicator. At a first time and in a first environment, the system can train a first model using a first training data set. At the first time and in a second environment, the system can train a second model using a second training data set. At a second time, the system can select the second model based on a comparison of metrics associated with each model. The system can deploy the second model to the first environment and can determine the risk indicator using the second model. The system can transmit, to a remote computing device, a responsive message including the risk indicator to control access of the target entity to one or more interactive computing environments.

Description

Attorney Docket No.: 096923-1439763 RISK ASSESSMENT TECHNIQUES FOR CONTROLLING ACCESS TO COMPUTING SYSTEMS USING DYNAMICALLY SELECTED MACHINE LEARNING MODELS TECHNICAL FIELD

[0001] The present disclosure relates generally to controlling interactions between computing systems. More specifically, but not by way of limitation, this disclosure relates to risk assessment techniques for controlling access to computing systems using dynamically selected machine learning models. BACKGROUND

[0002] Various systems use machine learning models to predict an amount of risk associated with an entity. Testing and deploying these models can take a long time and use a large number of resources to ensure the machine learning model can be safely deployed to a production environment. Additionally, lags in the deployment of machine learning models can lead to deployment of models that have been trained on now stale data sets. This leads to systems relying on outdated models, which can yield inaccurate risk predictions, leading to vulnerability of secured systems relying on such risk predictions for access control. SUMMARY

[0003] Various aspects of the present disclosure provide systems and methods for risk assessment using a risk indicator. The system can receive a request for a risk indicator associated with a target entity. In some aspects, at a first time and in a first environment, the system can train a first model to determine the risk indicator using a first training data set. At the first time and in a second environment, the system can train a second model to determine the risk indicator using a second training data set. In some aspects, the system can generate, for each of the first model and the second model, a set of metrics. At a second time, the system can select the second model based on a comparison of the set of metrics associated with the first model and the set of metrics associated with the second model. The system can deploy the second model to the first environment. The system can determine the risk indicator using the second model. The system can US2008302003991Attorney Docket No.096923-1439763 transmit, to a remote computing device, a responsive message comprising at least the risk indicator for use in controlling access of the target entity to one or more interactive computing environments.

[0004] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification, any or all drawings, and each claim.

[0005] The foregoing, together with other features and examples, will become more apparent upon referring to the following specification, claims, and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG. 1 is a block diagram depicting an example of an operating environment in which a risk assessment computing system can be used to provide a risk assessment associated with a target entity according to some aspects of the present disclosure.

[0007] FIGs.2A and 2B are block diagrams depicting a process for selecting and deploying a model in a production environment for risk assessment according to some aspects of the present disclosure.

[0008] FIG.3 is a flow chart illustrating a method for generating a risk assessment associated with a target entity according to some aspects of the present disclosure.

[0009] FIG. 4 is a block diagram depicting an example of a computing device, which can be used to implement the embodiments described herein according to some aspects of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] Disclosed systems and methods relate to risk assessment techniques for controlling access to computing systems based on dynamically selected machine learning models. For example, an entity may wish to control access to a secure computer system by a target entity based on a prediction of an amount of risk associated with the target entity. The risk can be predicted using a machine learning model trained on a training data set. Controlling access to computing US2008302003991Attorney Docket No.096923-1439763 systems, such as providing access to a secure resource or computing environment, is important to the security of such resources and computing environments. However, systems relying on static models or that train models on historic data are vulnerable as the machine learning model may yield inaccurate predictions as a result of being trained on stale data. Further, systems can suffer due to the lead time involved in deploying a machine learning model to production, resulting in a model that is outdated.

[0011] Certain aspects described herein for performing risk assessments on target entities using a dynamically selected model can improve existing systems by seamlessly training multiple models in a production environment on current data and dynamically selecting one of the models for servicing requests based on the model’s performance. By dynamically selecting a model based on performance, disclosed systems and methods ensure that the most accurate and up-to-date model is being used to generate a risk indicator to predict risk. In some examples, the risk indicator can be a numerical or binary indicator of a level of risk associated with the target entity. In other aspects, the risk indicator can indicate a lack of risk associated with a target entity (e.g., an indication that the target entity is verified or is trustworthy).

[0012] A risk assessment computing system can include a production environment and a testing environment. A first model can be deployed in the production environment and can be trained to generate a risk indicator in response to a request from a client computing system for a risk indicator associated with a target entity. A second model can be developed and trained in the testing environment for generating a risk indicator. The second model can then be deployed in the production environment to run on production data in the background, while the first model serves risk indicator requests. The risk assessment computing system can monitor the performance of both models and generate one or more metrics. The metrics can include an error rate, false positive rate, accuracy, runtime, and the like. After a period of time, the risk assessment computing system can compare the metrics associated with the first model and the second model to determine which model performs better. For example, the risk assessment computing system can compare an accuracy of the first model with an accuracy of the second model. The model having the higher accuracy can be selected to serve requests for a risk indicator. The model that is not selected can be archived or can be deployed in the testing environment for further training and development. US2008302003991Attorney Docket No.096923-1439763

[0013] In some examples, the risk assessment computing system can receive a request for a risk indicator associated with a target entity. The request can include an identifier, such as an SSN, associated with the target entity. In additional examples, an identifier can be a serial number or other unique identifier of a system, organization, or individual. The risk assessment computing system can generate a risk indicator for the target entity using the model that is deployed in the production environment and that is configured to serve requests for risk indicators. In some examples, the model for serving the request can be dynamically selected in response to receiving the request, based on real-time analysis of the performance of any models running in the production environment.

[0014] The system can then transmit the risk indicator to a remote computing system. In some examples, this may be the system from which the risk indicator was requested. The risk indicator can be used to control access of the target entity to an interactive computing environment. For example, the risk indicator can be included in a responsive message to the request for evaluating the target entity such that the responsive message can be used to allow, challenge, or deny access to the target entity. For example, if the risk indicator is below a predefined threshold, a request by the target entity to access the interactive computing environment may be automatically denied or flagged for manual review.

[0015] Certain aspects described herein, which can include dynamically selecting a model for serving requests for risk indicators and providing a responsive message indicating a risk associated with a target entity, can improve at least the technical fields of controlling interactions between computing environments, access control for a computing environment, or a combination thereof. For instance, by continually training models and running the trained models in the production environment, the most accurate and most up-to-date model can be selected to generate risk indicators and thus, the risk assessment computing system can cause access to a computing system to be controlled more accurately. The risk assessment computing system can mitigate delays in deployment of models to a production environment by running models in the production environment while also training models on current data. In this manner, each of the production models can be evaluated for performance and accuracy, such that the risk assessment computing system can seamlessly switch to the most accurate and efficient model to generate a risk indicator. Additionally or alternatively, by using the techniques described herein, a risk assessment computing system may provide legitimate access to the interactive computing environment more US2008302003991Attorney Docket No.096923-1439763 efficiently and using fewer computing resources compared to other risk assessment systems or techniques. For example, the risk assessment computing system can determine a risk indicator or an actionable response message efficiently thereby reducing the (i) memory usage, (ii) processing time, (iii) network bandwidth usage, (iv) response time, and the like for controlling access to the interactive computing. Accordingly, the risk assessment computing system improves the access control for computing environment by reducing memory usage, processing time, network bandwidth consumption, response time, and the like with respect to controlling access to the interactive computing environment using at least the system architecture and techniques described herein.

[0016] These illustrative examples are given to introduce the reader to the general subject matter discussed here and are not intended to limit the scope of the disclosed concepts. The following sections describe various additional features and examples with reference to the drawings in which like numerals indicate like elements, and directional descriptions are used to describe the illustrative examples but, like the illustrative examples, should not be used to limit the present disclosure. Operating Environment Example for Generating a Risk Indicator associated with a Target Entity

[0017] Referring now to the drawings, FIG. 1 is a block diagram depicting an example of an operating environment in which a risk assessment computing system can be used to provide a risk assessment associated with a target entity according to some aspects of the present disclosure. FIG. 1 depicts examples of hardware components of a risk assessment computing system 102, according to some aspects. The risk assessment computing system 102 can be a specialized computing system that may be used for processing large amounts of data using a large number of computer processing cycles. In other examples, the risk assessment computing system 102 may be or include a general- purpose computing system. The risk assessment computing system 102 can include a risk assessment server 104 for performing a risk assessment (e.g., predicting future risk associated with the target entity, predicting the legitimacy of the target entity, etc.) with respect to a target entity, such as a target individual or a user computing device. The risk assessment can be used by a client computing system 124 to evaluate a risk associated with a target entity.

[0018] The risk assessment server 104 can include one or more processing devices that can execute program code, such as a risk assessment application 106. The program code can be stored US2008302003991Attorney Docket No.096923-1439763 on a non-transitory computer-readable medium or other suitable medium. The risk assessment application 106 can include one or more modules or components executing software code to complete one or more steps for determining a risk indicator. For example, the risk assessment application 106 can include: a model processing module 108; a model evaluation module 110; a decisioning module 112; and a risk module 114. The model processing module 108 can collect performance data and model attributes in real time. The machine learning model monitored by the model processing module 108 can be a machine learning model deployed to a production environment of the risk assessment computing system 102. The model evaluation module 110 can access performance data associated with the deployed model. In some aspects, the model evaluation module 110 can test a deployed model and one or more additional models. The model evaluation module 110 can collect performance data on the deployed model and one or more additional models. The decisioning module 112 can analyze the performance data to determine a model having the highest performance (e.g., the lowest error rate, greatest accuracy, etc.). In some aspects, the risk module 114 can analyze the retrieved records to determine a risk indicator associated with the target entity based on entity data 120 stored in the data repository 116.

[0019] The risk assessment server 104 can perform risk assessment operations or access control operations for validating or otherwise authenticating the target entity, for example using other suitable modules, models, components, etc. of the risk assessment server 104. The risk assessment server 104 can receive data associated with the target entity from data repository 116, an external data source, or any suitable combination thereof. In some aspects, the risk assessment application 106 can authenticate or deny a request for an interaction involving the target entity by generating a risk indicator using the target entity data retrieved from the data repository 116.

[0020] In some aspects, the target entity data can be determined or stored in one or more network-attached storage units on which various repositories, databases, or other structures are stored. An example of these data structures can include the data repository 116. Additionally or alternatively, entity data 120 can be stored in the data repository 116. In some examples, the entity data 120 can be associated with a number of entities and can be searchable using identifying information associated with each entity. For example, the entity 118 can be searched using a unique identifier or unique set of identifiers such as personally identifiable information (PII) or, in other examples, serial or model numbers. US2008302003991Attorney Docket No.096923-1439763

[0021] Network-attached storage units may store a variety of different types of data organized in a variety of different ways and from a variety of different sources. For example, the network- attached storage unit may include storage other than primary storage located within the risk assessment server 104 that is directly accessible by processors located therein. In some aspects, the network-attached storage unit may include secondary, tertiary, or auxiliary storage, such as large hard drives, servers, and virtual memory, among other types of suitable storage. Storage devices may include portable or non-portable storage devices, optical storage devices, and various other mediums capable of storing and containing data. A machine-readable storage medium or computer-readable storage medium may include a non-transitory medium in which data can be stored and that does not include carrier waves or transitory electronic signals. Examples of a non- transitory medium may include, for example, a magnetic disk or tape, optical storage media such as a compact disk or digital versatile disk, flash memory, memory devices, or other suitable media.

[0022] Furthermore, the risk assessment computing system 102 can communicate with various other computing systems. The other computing systems can include user computing systems 122, such as smartphones, personal computers, etc., client computing systems 124, and other suitable computing systems. For example, user computing systems 122 may transmit, such as in response to receiving input from the target entity, requests for accessing the interactive computing environment 126 to the client computing systems 124. In response, the client computing systems 124 can send authentication queries to the risk assessment server 104, and the risk assessment server 104 can receive data associated with the target entity used in the request and generate a risk indicator associated with the target entity. In some aspects, the risk module 114 can retrieve data associated with the target entity from entity data 120 and apply the data to a machine learning model deployed in a production environment of the risk assessment computing system 102. While FIG.1 illustrates that the risk assessment computing system 102 and the client computing systems 124 are separate systems, the risk assessment computing system 102 and the client computing systems 124 can be one system. For example, the risk assessment computing system 102 can be a part of the client computing systems 124, or vice versa.

[0023] As illustrated in FIG. 1, the risk assessment computing system 102 may interact with the client computing systems 124, the user computing systems 122, or a combination thereof via one or more public data networks 128 to facilitate interactions between users of the user computing systems 122 and the interactive computing environment 126. For example, the risk assessment US2008302003991Attorney Docket No.096923-1439763 computing system 102 can facilitate the client computing systems 124 providing a user interface to the user computing system 122 for receiving various data from the user. The risk assessment computing system 102 can transmit validated risk assessment data, for example similarity- preserving hashes, comparisons or scores determined therefrom, etc., to the client computing systems 124 for providing, challenging, or rejecting, etc. access of the target entity to the interactive computing environment 126. In some examples, the risk assessment computing system 102 can additionally communicate with third-party systems to receive risk assessment data, entity data, and the like, through the public data network 128. In some examples, the third-party systems can provide real-time (e.g., streamed) data about the target entity, historical data about the target entity, etc. to the risk assessment computing system 102.

[0024] Each client computing system 124 may include one or more devices such as individual servers or groups of servers operating in a distributed manner. A client computing system 124 can include any computing device or group of computing devices operated by a seller, lender, or other suitable entity that can provide products or services. The client computing system 124 can include one or more server devices. The one or more server devices can include or can otherwise access one or more non-transitory computer-readable media.

[0025] The client computing system 124 can further include one or more processing devices that can be capable of providing an interactive computing environment 126, such as a user interface, etc., that can perform various operations. The interactive computing environment 126 can include executable instructions stored in one or more non-transitory computer-readable media. The instructions providing the interactive computing environment 126 can configure one or more processing devices to perform the various operations. In some aspects, the executable instructions for the interactive computing environment 126 can include instructions that provide one or more graphical interfaces. The graphical interfaces can be used by a user computing system 122 to access various functions of the interactive computing environment 126. For instance, the interactive computing environment 126 may transmit data to and receive data, such as via the graphical interface, from a user computing system 122 to shift between different states of the interactive computing environment 126, where the different states allow one or more electronic interactions between the user computing system 122 and the client computing system 124 to be performed. US2008302003991Attorney Docket No.096923-1439763

[0026] In some examples, the client computing system 124 may include other computing resources associated therewith (e.g., not shown in FIG. 1), such as server computers hosting and managing virtual machine instances for providing cloud computing services, server computers hosting and managing online storage resources for users, server computers for providing database services, and others. The interaction between the user computing system 122, the client computing system 124, and the risk assessment computing system 102, or any suitable sub-combination thereof may be performed through graphical user interfaces, such as the user interface, presented by the risk assessment computing system 102, the client computing system 124, other suitable computing systems of the computing environment 100, or any suitable combination thereof. The graphical user interfaces can be presented to the user computing system 122. Application programming interface (API) calls, web service calls, or other suitable techniques can be used to facilitate interaction between any suitable combination or sub-combination of the client computing system 124, the user computing system 122, and the risk assessment computing system 102.

[0027] A user computing system 122 can include any computing device or other communication device that can be operated by a user or entity, such as the user entity, which may include a consumer or a customer. The user computing system 122 can include one or more computing devices such as laptops, smartphones, and other personal computing devices. A user computing system 122 can include executable instructions stored in one or more non-transitory computer-readable media. The user computing system 122 can additionally include one or more processing devices configured to execute program code to perform various operations. In various examples, the user computing system 122 can allow a user to access certain online services or other suitable products, services, or computing resources from a target entity, such as the client computing system 124, to engage in mobile commerce with the client computing system 124, to obtain controlled access to electronic content, such as the interactive computing environment 126, hosted by the client computing system 124, etc.

[0028] In some examples, the user or a target entity can use the user computing system 122 to engage in an electronic interaction with the client computing system 124 via the interactive computing environment 126. The risk assessment computing system 102 can receive a request, for example from the user computing system 122, to access the interactive computing environment 126 and can use target entity data or any other suitable data or signals determined therefrom, to determine whether to provide access, to challenge the request, to deny the request, etc. An US2008302003991Attorney Docket No.096923-1439763 electronic interaction between the user computing system 122 and the client computing system 124 can include, for example, the user computing system 122 being used to request a financial loan or other suitable services or products from the client computing system 124, and so on. An electronic interaction between the user computing system 122 and the client computing system 124 can also include, for example, one or more queries for a set of sensitive or otherwise controlled data, accessing online financial services provided via the interactive computing environment 126, submitting an online credit card application or other digital application to the client computing system 124 via the interactive computing environment 126, operating an electronic tool within the interactive computing environment 126 (e.g., a content-modification feature, an application- processing feature, etc.), etc.

[0029] In some aspects, an interactive computing environment 126 implemented through the client computing system 124 can be used to provide access to various online functions. As a simplified example, a user interface or other interactive computing environment 126 provided by the client computing system 124 can include electronic functions for requesting computing resources, online storage resources, network resources, database resources, or other types of resources. In another example, a website or other interactive computing environment 126 provided by the client computing system 124 can include electronic functions for obtaining one or more financial services, such as an asset report, management tools, credit card application and transaction management workflows, electronic fund transfers, etc.

[0030] A user computing system 122 can be used to request access to the interactive computing environment 126 provided by the client computing system 124. The client computing system 124 can submit a request, such as in response to a request made by the user computing system 122 to access the interactive computing environment 126, for risk assessment to the risk assessment computing system 102 and can selectively grant or deny access to various electronic functions based on risk assessment performed by the risk assessment computing system 102. Based on the request, or continuously or substantially contemporaneously, the risk assessment computing system 102 can determine one or more risk signals or risk indicators for data associated with the target entity, which may submit or may have submitted the request via the user computing system 122. Based on a risk indicator determined from the risk module 114, the risk assessment computing system 102, the client computing system 124, or a combination thereof can determine whether to grant the access request of the user computing system 122 to certain features of the interactive US2008302003991Attorney Docket No.096923-1439763 computing environment 126. The risk assessment computing system 102, the client computing system 124, or a combination thereof can use the risk indicator for other suitable purposes such as identifying a manipulated identity, controlling a real-world interaction, and the like.

[0031] In a simplified example, the system illustrated in FIG. 1 can configure the risk assessment server 104 to be used for controlling access to the interactive computing environment 126. The risk assessment server 104 can retrieve data associated with the target entity in response to a request to access the interactive computing environment 126. The data may, for example, be retrieved based on identity information (e.g., information collected by the client computing system 124 via a user interface provided to the user computing system 122) provided by the client computing system 124 or received via other suitable computing systems. The risk assessment server 104 can retrieve the data associated with the target entity from the data repository 116. The data repository 116 can store, for example, transaction data associated with one or more entities. The risk assessment server 104 can determine a risk indicator associated with the target entity using a model deployed in a production environment of the risk assessment server 104. The risk assessment server 104 can transmit the risk indicator, or any inference derived therefrom, to the client computing system 124 for use in controlling access to the interactive computing environment 126.

[0032] The risk indicator associated with the target entity, or any suitable score or comparison determined therefrom, can be used, for example by the risk assessment computing system 102, the client computing system 124, etc., to determine whether the risk associated with the target entity accessing a good or a service provided by the client computing system 124 using exceeds a threshold, thereby granting, challenging, or denying access by the target entity to the interactive computing environment 126. For example, if the risk assessment computing system 102 determines that the risk indicator indicates that risk associated with the identity element is lower than a threshold value, then the client computing system 124 associated with the service provider can generate or otherwise provide access permission to the user computing system 122 that requested the access. The access permission can include, for example, cryptographic keys used to generate valid access credentials or decryption keys used to decrypt access credentials. The client computing system 124 can also allocate resources to the target entity and provide a dedicated web address for the allocated resources to the user computing system 122, for example, by adding the user computing system 122 in the access permission. With the obtained access credentials or the US2008302003991Attorney Docket No.096923-1439763 dedicated web address, the user computing system 122 can establish a secure network connection to the interactive computing environment 126 hosted by the client computing system 124 and access the resources via invoking API calls, web service calls, HTTP requests, other suitable mechanisms or techniques, etc.

[0033] In some examples, the risk assessment computing system 102 may determine whether to grant, challenge, or deny the access request made by the user computing system 122 for accessing the interactive computing environment 126. For example, based on the risk indicator associated with the target entity, the risk assessment computing system 102 can determine that the target entity is a legitimate entity that made the access request and may authenticate the request. In other examples, the risk assessment computing system 102 can challenge or deny the access attempt if the risk assessment computing system 102 determines that the target entity may not be a legitimate entity.

[0034] In some examples, the risk indicator used to determine access to the interactive computing environment 126 may be determined at least in part based on output from one or more machine learning models deployed in a production environment of the risk assessment computing system 102. For example, the risk module 114 can retrieve target entity data from entity data 120 in the data repository 116 and can apply the target entity data to a trained machine learning model to determine a risk indicator.

[0035] Each communication within the computing environment 100 may occur over one or more data networks, such as a public data network 128, a network 130 such as a private data network, or some combination thereof. A data network may include one or more of a variety of different types of networks, including a wireless network, a wired network, or a combination of a wired and wireless network. Examples of suitable networks include the Internet, a personal area network, a local area network (“LAN”), a wide area network (“WAN”), or a wireless local area network (“WLAN”). A wireless network may include a wireless interface or a combination of wireless interfaces. A wired network may include a wired interface. The wired or wireless networks may be implemented using routers, access points, bridges, gateways, or the like, to connect devices in the data network.

[0036] The number of devices illustrated in FIG. 1 is provided for illustrative purposes. Different numbers of devices may be used. For example, while certain devices or systems are US2008302003991Attorney Docket No.096923-1439763 shown as single devices in FIG.1, multiple devices may instead be used to implement these devices or systems. Similarly, devices or systems that are shown as separate may be instead implemented in a signal device or system. Process for Generating a Risk Indicator associated with a Target Entity

[0037] FIGs. 2A and 2B are block diagrams depicting an example process 200 for generating a risk assessment associated with a target entity according to some aspects of the present disclosure. The process 200 can be implemented by any of the components as described above with reference to FIG. 1. For example, the steps described with respect to FIG. 2 can be executed by the risk assessment server 102 via one or both of public network 128 and network 130. Other implementations or architectures, however, are possible. The process 200 can occur over a number of subsequent time intervals (e.g., days, weeks, months, etc.). In some aspects, certain steps described with respect to process 200 can occur simultaneously.

[0038] At a time T=1, Model 1 can be deployed in a production environment 202 of the risk assessment computing system 102. Model 1 can be a machine learning model trained to identify fraudulent transactions or to generate a risk indicator associated with a target entity. Model 1 can run in the production environment 202 to produce Model 1 output 206. As Model 1 runs in the production environment 202, the model processing module 108 can access or receive performance data associated with execution of Model 1. In some aspects, the model processing module 108 can also receive Model 1 output 206.

[0039] Storage 204 can be a part of the data repository 116, or can be a separate database of the risk assessment system 102. In some aspects, the storage 204 can be a cloud-based storage system. The storage 204 can store the performance data (e.g. fraud performance data and production model attribute data) collected by the model processing module 108. The storage 204 can also access and store entity data, transaction data, device data, and, in some aspects, credit data. Entity data can include information about an entity, such a PII (e.g., a name, DOB, address, phone number, etc.). In some examples, entity information can be system information such as a serial number, operating system version, etc. Transaction data can be information about a transaction involving the entity, such as a transaction amount, a transaction location, a transaction datetime, etc. Device data can include information about the device or system used to make the transaction, such as a device type, operating system, IP address, etc. Credit data can include US2008302003991Attorney Docket No.096923-1439763 information about the entity. For example, credit data can be sensitive data stored in a secure database (e.g., data repository 116) that can include financial or demographic data associated with the entity.

[0040] At time T=1, the risk assessment computing system 102 can receive requests from the client computing system 124 for a risk indicator or an indicator that a transaction is fraudulent. In response, the risk assessment application 106 can use the risk module 114 to access data associated with a target entity or target transaction and apply the accessed data to Model 1 to generate a risk indicator. The risk indicator can be provided to the client computing system 124 as Model 1 output 206. In some examples, the client computing system 124 can provide feedback to the risk assessment computing system 102 to be stored in the storage 204.

[0041] At time T=2, Model 1 can continue to run in the production environment 202 and can continue to provide risk indicators in response to requests from the client computing system 124. In some examples, the model processing module 108 can provide performance data to the decisioning module 112. The decisioning module 112 can analyze the performance data to generate a strategy recommendation that can be provided to the client computing system 124.

[0042] Also at time T=2, a second model, Model 2, can be developed and trained in a training environment 208. For example, Model 2 can be trained using feedback data 210 accessed from the storage 204, as well as configuration data generated by the configuration module 115. The configuration module 115 can, for example, develop and train Model 2 using the feedback data 210, as well as training data sets 118. The configuration module 115 can determine, for example, parameters or hyperparameters associated with Model 2.

[0043] Model 2 can run in the training environment 208 to generate Model 2 output 212. For example, entity data 120 can be applied to Model 2 to generate Model 2 output 212. The Model 2 output can include a set of risk indicators for entities included in the entity data 120. In some examples, the Model 2 output 212 can include a set of indicators, indicating whether each transaction in the entity data 120 is fraudulent.

[0044] The model evaluation module 110 can receive the Model 2 output 212 and can generate one or more metrics. The one or more metrics can include, for example, a hit rate (e.g., rate of transactions identified as fraudulent), an error rate, a runtime, etc. In some examples, the risk assessment computing system 102 can compare the Model 2 output 212 against a set of baseline US2008302003991Attorney Docket No.096923-1439763 data. For example, the model evaluation module 110 can apply a set of data including known fraudulent transactions to the Model 2 and can determine how many transactions were correctly identified, or how many false positives were generated. In another example, the model evaluation module 110 can compare the Model 2 output 212 to a set of risk indicators. The model evaluation module 110 can determine an average amount by which the risk indicator for each entity differs from a baseline risk indicator for each entity.

[0045] At block 214, the risk assessment system 102, or a component of the decisioning module 112, can determine whether to deploy Model 2 to a test phase in the production environment 202. For example, the risk assessment system 102 can analyze the metrics generated by the model evaluation module 110 by comparing the metrics to a set of thresholds. For example, if the error rate is below an error rate threshold, Model 2 can be promoted to the testing phase in the production environment 202. In another example, if the risk indicators generated by Model 2 are within a threshold number of standard deviations of the baseline set of risk indicators, then Model 2 can be deployed for testing. If a model is not deployed for testing, it can be retrained or parameters may be reconfigured by configuration module 115.

[0046] At time T=3, if, at block 214, Model 2 is promoted to the test phase, Model 2 can be deployed to the production environment 202 for testing. At time T=3, the risk assessment computing system 102 can receive a request from the client computing system 124 for a risk indicator associated with a target entity. In some examples, the client computing system 124 can request a fraud indicator for one or more transactions associated with one or more target entities. The risk assessment system 102 can retrieve entity or transaction data from entity data 120 and can apply the data to both Model 1 and Model 2, such that both models produce a risk indicator or fraud indicator. However, only Model 1 output 206 is provided to the client computing system 124.

[0047] The model processing module 108 can collect performance data and output of both models, which can be stored in the storage 204. In some aspects, as described with respect to the training environment 028 at time T=2, the risk assessment system 102 can train a Model 3. Model 3 can be trained and configured in the training environment 208 based on performance and output data collected by the model processing module 108. The model evaluation module 112 can analyze the model 3 output to determine whether to promote Model 3 to the testing phase at time T=4. US2008302003991Attorney Docket No.096923-1439763

[0048] In some examples, at time T=3, the performance metrics determined by the model processing module 108 can be provided to a decisioning module 110. The decisioning module 110 can provide a strategic recommendation based on the performance metrics. For example, the decisioning module 110 can determine, based on the performance metrics, whether one or more parameters of Model 1 need to be reconfigured or adjusted to improve model performance and outcomes.

[0049] In some aspects, the Model 1 output 206 may also be provided to the model evaluation module 112 in as part of a continuous integration and continuous deployment (CI / CD) pipeline 216. The model evaluation module 112 can also receive, from the model processing module 108, the Model 2 output 212. The model evaluation module 112 can generate metrics by which to evaluate Model 1 and Model 2. For example, the model evaluation module 112 can determine an accuracy of each model. The model evaluation module 112 can compare the accuracy to determine which model has the greater accuracy. If it is Model 2, for example, at block 218, the risk assessment model 102 archive Model 1 and promote Model 2 to serve requests from the client computing system 124. In another example, if Model 1 performs better than Model 2 according to the calculated metrics, Model 1 can continue to serve requests from the client computing system 124. In some examples, the model evaluation module 112 can calculate and compare error rates, runtimes, false positive rates, or other model performance metrics. When a model is not selected to serve requests from the client computing system 124, that model can be, for example, archived in the data repository 116 or another database. In another example, the model that is not selected can return to a training phase in the training environment 208 for further training and parameter adjustment.

[0050] At time T=4, after being deployed to the production environment 202, Model 2 can be used to serve requests from the client computing system 124. The model processing module 108 can collect performance data about Model 2. The storage 204 can access the performance data and store the performance data as well as output of Model 2. The information stored by the storage 204 can be used to train and develop one or more additional models. In some aspects, as Model 2 generates risk indicators in response to requests from the client computing system 124, the risk assessment system 102 can provide Model 2 output 212 (e.g., the generated risk indicators) to the client computing system 124. In some examples, the risk indicators can be batched and transmitted US2008302003991Attorney Docket No.096923-1439763 to the client computing system 124. In other examples, the risk indicators can be provided on- demand, in response to each request from the client computing system 124.

[0051] As discussed above, the decisioning module 110 can determine a strategy recommendation based on performance data generated by the model processing module 108. For example, the decisioning module 110 can analyze performance data associated with Model 2 and can generate a strategy recommendation that can include, for example, parameter modifications or adjustments to improve the functioning of Model 2. The strategy recommendation can be provided to the client computing system 124, which can elect whether or not to implement the strategy recommendations.

[0052] The process 200 described with reference to FIGs. 2A and 2B can run continuously, such that models and new models are continuously developed, trained, deployed, and evaluated. Thus the risk assessment computing system 102 can dynamically and seamlessly switch between models deployed in the production environment, such that a highest performing model is used to serve risk indicator requests. Accordingly, the risk assessment computing system 102 can mitigate lags and delays associated with switching between a model in a testing environment and a model in a production environment by having multiple developed and trained models ready for use in the production environment. Techniques for Generating a Risk Indicator associated with a Target Entity

[0053] FIG. 3 is a flow chart illustrating an example of a process 300 for generating a risk assessment associated with a target entity according to some aspects of the present disclosure. In some examples, the operations of the process 300, or any subset thereof, may be performed by the risk assessment computing system 102 via the risk assessment server 104, but other suitable systems, devices, or subsets or combinations thereof may perform one or more operations described with respect to the process 300. For illustrative purposes, the process 300 is described with reference to certain examples depicted in the figures. Other implementations, however, are possible.

[0054] At block 302, the process 300 involves receiving a request for a risk indicator associated with a target entity. The request can include an identifier associated with the target entity. The identifier can be, for example, PII such as an SSN, or a combination of PII such as a US2008302003991Attorney Docket No.096923-1439763 name and DOB. The request may be generated as part of an authentication process initiated when the target entity attempts to access an interactive computing environment 126.

[0055] At block 304, the process 300 involves generating, by the risk assessment application 106, a set of metrics for a first model and a set of metrics for a second model. The set of metrics can be, for example, performance metrics such as an accuracy, error rate, runtime, etc. In some examples, the first model can be a machine learning model deployed in a production environment and trained on a first set of data (e.g., on production data). The first model can be used to serve risk indicator requests. In some examples, the second model can be a model trained using a second training data set (e.g., historic data) in a testing environment. In some examples, the second set of data can be updated periodically based on the first set of data. For example, if data is older than an expiration threshold, it can be archived to ensure the second model is not trained on stale data. The performance metrics can be determined periodically or in real time in response to the request for a risk indicator.

[0056] At block 306, the process 300 involves, selecting, by the risk assessment application 106, the second model based on a comparison of the set of metrics associated with the first model with the set of metrics associated with the second model. For example, the risk assessment application 106 can compare an accuracy of the first model with an accuracy of the second model. If the accuracy of the second model is higher than the accuracy of the first model, the risk assessment application 106 can select the second model for serving risk indicator requests. In some examples, if the first model performs higher than the second model according to one or more metrics, the risk assessment application 106 can continue to use the first model to serve requests for risk indicators. In such an example, the second model can undergo additional testing or training in the testing environment.

[0057] At block 308, the process 300 involves in response to selecting the second model, deploying the second model to the production environment. In some examples, the second model may already be deployed in the production environment and may be running in the background. In such an example, the risk assessment application 106 can switch to the second model for serving risk indicator requests. The first model can be archived or can be deployed in the testing environment for further development and training. US2008302003991Attorney Docket No.096923-1439763

[0058] At block 310, the process 300 involves determining, by the risk assessment application 106, a risk indicator for the target entity using the second model. For example, the deployed second model can be trained on production data (e.g., current data). Data associated with the target entity can be retrieved from the data repository 116 and can be applied to the second model to generate a risk indicator associated with the target entity. In some aspects, the risk assessment application 106 can select which model to use to generate the risk indicator at the time the request is received. In other examples, the risk assessment application 106 can compare the performance metrics of the models at predetermined intervals. The risk assessment application 106 can continually monitor the performance of the second model and any models deployed in the production environment, or in the testing environment.

[0059] At block 312, the process 300 involves transmitting, to a remote computing device (e.g., the client computing device 124), a responsive message comprising at least the risk indicator for use in controlling access of the target entity to one or more interactive computing environments. For example, the risk indicator can be used in controlling an interaction involving a target entity or access of the target entity to a restricted system (e.g., the interactive computing environment 126).

[0060] In some examples, the process 300 can continually run such that a third model can be trained and compared with the second model, or with both the first and second model, and so on. Accordingly, a model that is deployed and trained on current data is available for selection for serving risk indicator requests. In some aspects, this continuous approach ensures that a current model is available and that the current model can be selected depending on performance of the model in the production environment.

[0061] Systems and methods described herein provide advantages over traditional methods for managing machine learning models in a production environment. For example, described systems and methods can dynamically and seamlessly switch between a model and one or more other “on deck” models based on model performance to generate risk indicators. By continuously evaluating model performance in a production environment, the risk assessment computing system can ensure that a current, high performing model trained on current data is available to serve requests for a risk indicator. In some examples, the risk assessment system 102 can provide explorable metrics and strategy recommendations, allowing a user to review and select a model deployed in a US2008302003991Attorney Docket No.096923-1439763 production environment to use to generate a risk indicator. Additionally, disclosed systems and methods facilitate accurate risk predictions, enabling an entity to better control access to a secure environment. Example of Computing System

[0062] Any suitable computing system or group of computing systems can be used to perform the operations for the techniques described herein. For example, FIG. 4 is a block diagram depicting an example of a computing device 400, which can be used to implement the risk assessment server 104. The computing device 400 can include various devices for communicating with other devices in the computing environment 100, as described with respect to FIG. 1. The computing device 400 can include various devices for performing one or more operations, such as risk assessment operations, described above with respect to FIGs.1-3.

[0063] The computing device 400 can include a processor 402 that can be communicatively coupled to a memory 404. The processor 402 can execute computer-executable program code stored in the memory 404, can access information stored in the memory 404, or both. Program code may include machine-executable instructions that may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, among others.

[0064] Examples of a processor 402 can include a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or any other suitable processing device. The processor 402 can include any suitable number of processing devices, including one. The processor 402 can include or communicate with a memory 504. The memory 404 can store program code that, when executed by the processor 402, causes the processor 402 to perform the operations described herein.

[0065] The memory 404 can include any suitable non-transitory computer-readable medium. The computer-readable medium can include any electronic, optical, magnetic, or other storage device capable of providing a processor with computer-readable program code or other program US2008302003991Attorney Docket No.096923-1439763 code. Non-limiting examples of a computer-readable medium can include a magnetic disk, memory chip, optical storage, flash memory, storage class memory, ROM, RAM, an ASIC, magnetic storage, or any other medium from which a computer processor can read and execute program code. The program code may include processor-specific program code generated by a compiler or an interpreter from code written in any suitable computer-programming language. Examples of suitable programming language can include Hadoop, C, C++, C#, Visual Basic, Java, Python, Perl, JavaScript, ActionScript, etc.

[0066] The computing device 400 may also include a number of external or internal devices such as input or output devices. For example, the computing device 400 is illustrated with an input / output interface 408 that can receive input from input devices or provide output to output devices. A bus 406 can also be included in the computing device 400. The bus 406 can communicatively couple one or more components of the computing device 400.

[0067] The computing device 400 can execute program code 414 that can include risk assessment application 106. The program code 414 for the risk assessment application 106 may be resident in any suitable computer-readable medium and may be executed on any suitable processing device. For example, and as illustrated in FIG. 4, the program code 414 for the risk assessment application 106 can reside in the memory 404 at the computing device 400 along with the program data 416 associated with the program code 414. Executing the risk assessment application 106 can configure the processor 402 to perform at least a portion of the operations described herein.

[0068] In some aspects, the computing device 400 can include one or more output devices. One example of an output device can be or include the network interface device 410 illustrated in FIG. 4. A network interface device 410 can include any device or group of devices suitable for establishing a wired or wireless data connection to one or more data networks described herein. Non-limiting examples of the network interface device 410 can include an Ethernet network adapter, a modem, etc.

[0069] Another example of an output device can include the presentation device 412 depicted in FIG. 4. A presentation device 412 can include any device or group of devices suitable for providing visual, auditory, or other suitable sensory output. Non-limiting examples of the presentation device 412 can include a touchscreen, a monitor, a speaker, a separate mobile US2008302003991Attorney Docket No.096923-1439763 computing device, etc. In some aspects, the presentation device 412 can include a remote client- computing device that communicates with the computing device 400 using one or more data networks described herein. In other aspects, the presentation device 412 can be omitted.

[0070] The foregoing description of some examples has been presented only for the purpose of illustration and description and is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Numerous modifications and adaptations thereof will be apparent to those skilled in the art without departing from the spirit and scope of the disclosure. US2008302003991

Claims

Attorney Docket No.096923-1439763 Claims What is claimed is:

1. A computer-implemented method comprising: receiving, by a processor, a request for a risk indicator associated with a target entity; at a first time and in a first environment, training, by the processor, a first model to determine the risk indicator using a first training data set; at the first time and in a second environment, training, by the processor, a second model to determine the risk indicator using a second training data set; generating, by the processor and for each of the first model and the second model, a set of metrics; at a second time, selecting, by the processor, the second model based on a comparison of the set of metrics associated with the first model and the set of metrics associated with the second model; deploying, by the processor, the second model to the first environment; determining, by the processor, the risk indicator using the second model; and transmitting, by the processor to a remote computing device, a responsive message comprising at least the risk indicator to control access of the target entity to one or more interactive computing environments.

2. The method of claim 1, wherein the second training data set comprises at least a portion of the first training data set.

3. The method of claim 2, wherein the method further comprises: generating, by the processor, the second training data set, in part, by comparing data of the first training data set to an expiration threshold and discarding data having an age exceeding the expiration threshold. US2008302003991Attorney Docket No.096923-1439763 4. The method of claim 2, wherein the first training data set comprises data associated with a set of entities and wherein the method further comprises: generating, by the processor, the second training data set, in part, by updating data associated with at least one entity in the first training data set.

5. The method of claim 1, wherein the first environment comprises a production environment and the second environment comprises a test environment.

6. The method of claim 1, wherein the set of metrics comprises an accuracy metric and wherein selecting the second model comprises: determining, by the processor, an accuracy of the first model and an accuracy of the second model; and selecting, by the processor, the second model based on the accuracy of the second model being greater than the accuracy of the first model.

7. The method of claim 1, wherein the method further comprises: at a third time and in the second environment, training, by the processor, a third model using a third training data set; generating, by the processor, for each of the second model and the third model, a second set of metrics; at a fourth time, selecting, by the processor, the third model based on a comparison of the second set of metrics associated with the second model and the second set of metrics associated with the third model; deploying, by the processor, the third model to the first environment; determining, by the processor, the risk indicator using the third model; and transmitting, by the processor to the remote computing device, a second responsive message comprising at least the risk indicator to control access of the target entity to one or more interactive computing environments.

8. A system comprising: a processor; and US2008302003991Attorney Docket No.096923-1439763 a non-transitory computer-readable medium comprising instructions that are executable by the processor for causing the processor to perform operations comprising: receiving a request for a risk indicator associated with a target entity; at a first time and in a first environment, training a first model to determine the risk indicator using a first training data set; at the first time and in a second environment, training a second model to determine the risk indicator using a second training data set; generating, for each of the first model and the second model, a set of metrics; at a second time, selecting the second model based on a comparison of the set of metrics associated with the first model and the set of metrics associated with the second model; deploying the second model to the first environment; determining the risk indicator using the second model; and transmitting, to a remote computing device, a responsive message comprising at least the risk indicator to control access of the target entity to one or more interactive computing environments.

9. The system of claim 8, wherein the second training data set comprises at least a portion of the first training data set.

10. The system of claim 9, wherein the operations further comprise: generating the second training data set, in part, by comparing data of the first training data set to an expiration threshold and discarding data having an age exceeding the expiration threshold.

11. The system of claim 9, wherein the first training data set comprises data associated with a set of entities and wherein the operations further comprise: generating the second training data set, in part, by updating data associated with at least one entity in the first training data set. US2008302003991Attorney Docket No.096923-1439763 12. The system of claim 8, wherein the first environment comprises a production environment and the second environment comprises a test environment.

13. The system of claim 8, wherein the set of metrics comprises an accuracy metric and wherein selecting the second model comprises: determining an accuracy of the first model and an accuracy of the second model; and selecting the second model based on the accuracy of the second model being greater than the accuracy of the first model.

14. The system of claim 8, wherein the operations further comprise: at a third time and in the second environment, training a third model using a third training data set; generating, for each of the second model and the third model, a second set of metrics; at a fourth time, selecting the third model based on a comparison of the second set of metrics associated with the second model and the second set of metrics associated with the third model; deploying the third model to the first environment; determining the risk indicator using the third model; and transmitting, to the remote computing device, a second responsive message comprising at least the risk indicator to control access of the target entity to one or more interactive computing environments.

15. A non-transitory computer-readable storage medium having program code that is executable by a processor device to cause a computing device to perform operations, the operations comprising: receiving a request for a risk indicator associated with a target entity; at a first time and in a first environment, training a first model to determine the risk indicator using a first training data set; at the first time and in a second environment, training a second model to determine the risk indicator using a second training data set; generating, for each of the first model and the second model, a set of metrics; US2008302003991Attorney Docket No.096923-1439763 at a second time, selecting the second model based on a comparison of the set of metrics associated with the first model and the set of metrics associated with the second model; deploying the second model to the first environment; determining the risk indicator using the second model; and transmitting, to a remote computing device, a responsive message comprising at least the risk indicator to control access of the target entity to one or more interactive computing environments.

16. The non-transitory computer-readable storage medium of claim 15, wherein the second training data set comprises at least a portion of the first training data set.

17. The non-transitory computer-readable storage medium of claim 16, wherein the operations further comprise: generating the second training data set, in part, by comparing data of the first training data set to an expiration threshold and discarding data having an age exceeding the expiration threshold.

18. The non-transitory computer-readable storage medium of claim 16, wherein the first training data set comprises data associated with a set of entities and wherein the operations further comprise: generating the second training data set, in part, by updating data associated with at least one entity in the first training data set.

19. The non-transitory computer-readable storage medium of claim 15, wherein the first environment comprises a production environment and the second environment comprises a test environment.

20. The non-transitory computer-readable storage medium of claim 15, wherein the set of metrics comprises an accuracy metric and wherein selecting the second model comprises: determining an accuracy of the first model and an accuracy of the second model; and US2008302003991Attorney Docket No.096923-1439763 selecting the second model based on the accuracy of the second model being greater than the accuracy of the first model. US2008302003991

Images