Risk assessment techniques for controlling access to computing systems using dynamically deployed machine learning models

Abstract

A system can generate a risk assessment associated with a target entity. The system can access a request for a risk indicator associated with a target entity. The system can also access a model in a test environment and run an ATP tool on the model. The ATP tool can execute a set of deployment steps using the model and determine a completion status of each of the set of deployment steps. The system can deploy the model to a production environment when the set of deployment steps is successful. The system can determine the risk indicator using the model. The system can transmit, to a remote computing device, a responsive message comprising at least the risk indicator to control access of the target entity to one or more interactive computing environments.

Description

Attorney Docket No.: 096923-1441781 RISK ASSESSMENT TECHNIQUES FOR CONTROLLING ACCESS TO COMPUTING SYSTEMS USING DYNAMICALLY DEPLOYED MACHINE LEARNING MODELS TECHNICAL FIELD

[0001] The present disclosure relates generally to controlling interactions between computing systems. More specifically, but not by way of limitation, this disclosure relates to risk assessment techniques for controlling access to computing systems using dynamically deployed machine learning models. BACKGROUND

[0002] Various systems use machine learning models to predict an amount of risk associated with an entity. Testing and deploying these models can take a long time and use a large number of resources to ensure the machine learning model can be safely deployed to a production environment. Additionally, lags in the deployment of machine learning models can increase a product’s time-to-market and reduce the product’s competitive advantage in the marketplace. Further, the lengthy and stringent testing and deployment process can use significant computing and personnel resources. The use of older and less current machine learning models can yield inaccurate risk predictions, leading to vulnerability of secured systems relying on such risk predictions for access control. SUMMARY

[0003] Various aspects of the present disclosure provide systems and methods for risk assessment using a risk indicator. The system can receive a request for a risk indicator associated with a target entity. In some aspects, the system access a model in a test environment. In some aspects, the system can run an analytics-to-production (ATP) tool on the model. The ATP tool can be configured to: execute one or more steps of a set of deployment steps using the model; store analytics information associated with each step; and determine a completion status of each of the set of deployment steps for the model. The system can deploy the model to a production environment when the completion status of each of the set of deployment steps is successful. The system can determine the risk indicator using the model. The system can transmit, to a remote 1 US2008305628361Attorney Docket No.096923-1441781 computing device, a responsive message comprising at least the risk indicator for use in controlling access of the target entity to one or more interactive computing environments.

[0004] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification, any or all drawings, and each claim.

[0005] The foregoing, together with other features and examples, will become more apparent upon referring to the following specification, claims, and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG. 1 is a block diagram depicting an example of an operating environment in which a risk assessment computing system can be used to provide a risk assessment associated with a target entity according to some aspects of the present disclosure.

[0007] FIG. 2 is a block diagrams depicting an architecture of a system for dynamically deploying a model according to some aspects of the present disclosure.

[0008] FIG. 3 is a flow chart illustrating a process for interacting with an analytics-to- production (ATP) tool according to some aspects of the present disclosure.

[0009] FIG. 4 is an illustration of a graphical user interface (GUI) according to some aspects of the present disclosure.

[0010] FIG.5 is a flow chart illustrating a method for generating a risk assessment associated with a target entity according to some aspects of the present disclosure.

[0011] FIG 6 is a block diagram depicting an example of a computing device, which can be used to implement the embodiments described herein according to some aspects of the present disclosure. 2 US2008305628361Attorney Docket No.096923-1441781 DETAILED DESCRIPTION OF THE INVENTION

[0012] Disclosed systems and methods relate to risk assessment techniques for controlling access to computing systems based on dynamically deployed machine learning models. For example, an entity may wish to control access to a secure computer system by a target entity based on a prediction of an amount of risk associated with the target entity. The risk can be predicted using a machine learning model trained on a training data set. Controlling access to computing systems, such as providing access to a secure resource or computing environment, is important to the security of such resources and computing environments. However, testing and deployment are often multi-step processes requiring communication between resources and personnel. These processes can cause lags in the deployment of a model leading to delays in product releases.

[0013] Certain aspects described herein for performing risk assessments on target entities using a dynamically selected model can improve existing systems by facilitating automated and dynamic deployment of machine learning models. Thus, current versions of models can be tested and deployed efficiently and with little oversight, resulting in the most recent version of a machine learning model being deployed more quickly. The most current version of a machine learning model can contain important updates for security or accuracy, and delays in deployment can result in a sub-optimal product being used by customers. Having the most up-to-date version of a machine learning model can be important in risk assessment application to reduce system vulnerabilities.

[0014] A risk assessment computing system can include a production environment to which machine learning models (e.g., risk assessment machine learning models) are deployed. Disclosed systems and methods describe an end-to-end automated solution for deploying machine learning models to a production environment. In some examples, the disclosed systems and methods can seamlessly interface with existing systems to automate the testing and deployment processes. For example, an analytics-to-production (ATP) tool can provide a secure and compliant, self-service platform through which models can be tested and deployed. Further, the ATP tool can provide visibility and analytic data on each step in testing and deployment.

[0015] Once a model is deployed, the risk assessment computing system can receive a request for a risk indicator associated with a target entity. The request can include an identifier, such as a social security number (SSN), associated with the target entity. In additional examples, an 3 US2008305628361Attorney Docket No.096923-1441781 identifier can be a serial number or other unique identifier of a system, organization, or individual. The risk assessment computing system can generate a risk indicator for the target entity using the model that is deployed in the production environment and that is configured to serve requests for risk indicators.

[0016] The system can then transmit the risk indicator to a remote computing system. In some examples, this may be the system from which the risk indicator was requested. The risk indicator can be used to control access of the target entity to an interactive computing environment. For example, the risk indicator can be included in a responsive message to the request for evaluating the target entity such that the responsive message can be used to allow, challenge, or deny access to the target entity. For example, if the risk indicator is below a predefined threshold, a request by the target entity to access the interactive computing environment may be automatically denied or flagged for manual review.

[0017] Certain aspects described herein, which can include dynamically selecting a model for serving requests for risk indicators and providing a responsive message indicating a risk associated with a target entity, can improve at least the technical fields of controlling interactions between computing environments, access control for a computing environment, or a combination thereof. For instance, the risk assessment computing system, via the ATP tool, can mitigate delays in deployment of models to a production environment by reducing the number of interactions required to test and approve a model for deployment to a production environment. In this manner, delays in releasing the most comprehensive and up-to-date version of a model to generate a risk indicator are reduced. Additionally or alternatively, by using the techniques described herein, a risk assessment computing system may provide legitimate access to the interactive computing environment more efficiently and using fewer computing resources compared to other risk assessment systems or techniques. For example, the risk assessment computing system can determine a risk indicator or an actionable response message efficiently thereby reducing the (i) memory usage, (ii) processing time, (iii) network bandwidth usage, (iv) response time, and the like for controlling access to the interactive computing. Accordingly, the risk assessment computing system improves the access control for computing environment by reducing memory usage, processing time, network bandwidth consumption, response time, and the like with respect to controlling access to the interactive computing environment using at least the system architecture and techniques described herein. US2008305628361Attorney Docket No.096923-1441781

[0018] These illustrative examples are given to introduce the reader to the general subject matter discussed here and are not intended to limit the scope of the disclosed concepts. The following sections describe various additional features and examples with reference to the drawings in which like numerals indicate like elements, and directional descriptions are used to describe the illustrative examples but, like the illustrative examples, should not be used to limit the present disclosure. Operating Environment Example for Generating a Risk Indicator associated with a Target Entity

[0019] Referring now to the drawings, FIG. 1 is a block diagram depicting an example of an operating environment in which a risk assessment computing system can be used to provide a risk assessment associated with a target entity according to some aspects of the present disclosure. FIG. 1 depicts examples of hardware components of a risk assessment computing system 102, according to some aspects. The risk assessment computing system 102 can be a specialized computing system that may be used for processing large amounts of data using a large number of computer processing cycles. In other examples, the risk assessment computing system 102 may be or include a general- purpose computing system. The risk assessment computing system 102 can include a risk assessment server 104 for performing a risk assessment (e.g., predicting future risk associated with the target entity, predicting the legitimacy of the target entity, etc.) with respect to a target entity, such as a target individual or a user computing device. The risk assessment can be used by a client computing system 124 to evaluate a risk associated with a target entity. The risk assessment computing system 102 can also include an automation-to-production (ATP) server 106 for managing end-to-end testing and deployment of machine learning models (e.g., machine learning model 108) for use by the risk assessment server 104.

[0020] The risk assessment server 104 can include one or more processing devices that can execute program code, such as a risk assessment application 110. The program code can be stored on a non-transitory computer-readable medium or other suitable medium. The risk assessment application 110 can include one or more modules or components executing software code to complete one or more steps for determining a risk indicator, including a machine learning model 108. The machine learning model 108 can be trained to output a risk indicator for use in access control decisions. For example, the machine learning model 108 can be trained using training data sets 120 stored in the data repository 112. 5 US2008305628361Attorney Docket No.096923-1441781

[0021] The risk assessment server 104 can perform risk assessment operations or access control operations for validating or otherwise authenticating the target entity, for example using other suitable modules, models, components, etc. of the risk assessment server 104. The risk assessment server 104 can receive data associated with the target entity from data repository 112, an external data source, or any suitable combination thereof. In some aspects, the risk assessment application 110 can authenticate or deny a request for an interaction involving the target entity by generating a risk indicator using the target entity data retrieved from the data repository 112.

[0022] The ATP server 106 can include one or more processing devices that can execute program code, such as an ATP application 114. The program code can be stored on a non-transitory computer-readable medium or other suitable medium. The ATP application 114 can include one or more modules or components executing software code to complete one or more steps for automating the deployment of the machine learning model 108 to a production environment. For example, the ATP application 114 can include a workflow orchestrator 116 to manage one or more workflows associated with testing and deploying the machine learning model 108.

[0023] In some aspects, target entity data can be determined or stored in one or more network- attached storage units on which various repositories, databases, or other structures are stored. An example of these data structures can include the data repository 112. In some examples, the entity data 118 stored in the data repository 112 can be associated with a number of entities and can be searchable using identifying information associated with each entity. For example, the entity data 118 can be searched using a unique identifier or unique set of identifiers such as personally identifiable information (PII) or, in other examples, serial or model numbers.

[0024] Network-attached storage units may store a variety of different types of data organized in a variety of different ways and from a variety of different sources. For example, the network- attached storage unit may include storage other than primary storage located within the risk assessment server 104 that is directly accessible by processors located therein. In some aspects, the network-attached storage unit may include secondary, tertiary, or auxiliary storage, such as large hard drives, servers, and virtual memory, among other types of suitable storage. Storage devices may include portable or non-portable storage devices, optical storage devices, and various other mediums capable of storing and containing data. A machine-readable storage medium or computer-readable storage medium may include a non-transitory medium in which data can be 6 US2008305628361Attorney Docket No.096923-1441781 stored and that does not include carrier waves or transitory electronic signals. Examples of a non- transitory medium may include, for example, a magnetic disk or tape, optical storage media such as a compact disk or digital versatile disk, flash memory, memory devices, or other suitable media.

[0025] Furthermore, the risk assessment computing system 102 can communicate with various other computing systems. The other computing systems can include user computing systems 122, such as smartphones, personal computers, etc., client computing systems 124, and other suitable computing systems. For example, user computing systems 122 may transmit, such as in response to receiving input from the target entity, requests for accessing the interactive computing environment 126 to the client computing systems 124. In response, the client computing systems 124 can send authentication queries to the risk assessment server 104, and the risk assessment server 104 can receive data associated with the target entity used in the request and generate a risk indicator associated with the target entity. In some aspects, the risk assessment application 110 can retrieve data associated with the target entity from entity data 118 and apply the data to a machine learning model deployed in a production environment of the risk assessment computing system 102. While FIG. 1 illustrates that the risk assessment computing system 102 and the client computing systems 124 are separate systems, the risk assessment computing system 102 and the client computing systems 124 can be one system. For example, the risk assessment computing system 102 can be a part of the client computing systems 124, or vice versa.

[0026] As illustrated in FIG. 1, the risk assessment computing system 102 may interact with the client computing systems 124, the user computing systems 122, or a combination thereof via one or more public data networks 128 to facilitate interactions between users of the user computing systems 122 and the interactive computing environment 126. For example, the risk assessment computing system 102 can facilitate the client computing systems 124 providing a user interface to the user computing system 122 for receiving various data from the user. The risk assessment computing system 102 can transmit validated risk assessment data, for example similarity- preserving hashes, comparisons or scores determined therefrom, etc., to the client computing systems 124 for providing, challenging, or rejecting, etc. access of the target entity to the interactive computing environment 126. In some examples, the risk assessment computing system 102 can additionally communicate with third-party systems to receive risk assessment data, entity data, and the like, through the public data network 128. In some examples, the third-party systems US2008305628361Attorney Docket No.096923-1441781 can provide real-time (e.g., streamed) data about the target entity, historical data about the target entity, etc. to the risk assessment computing system 102.

[0027] Each client computing system 124 may include one or more devices such as individual servers or groups of servers operating in a distributed manner. A client computing system 124 can include any computing device or group of computing devices operated by a seller, lender, or other suitable entity that can provide products or services. The client computing system 124 can include one or more server devices. The one or more server devices can include or can otherwise access one or more non-transitory computer-readable media.

[0028] The client computing system 124 can further include one or more processing devices that can be capable of providing an interactive computing environment 126, such as a user interface, etc., that can perform various operations. The interactive computing environment 126 can include executable instructions stored in one or more non-transitory computer-readable media. The instructions providing the interactive computing environment 126 can configure one or more processing devices to perform the various operations. In some aspects, the executable instructions for the interactive computing environment 126 can include instructions that provide one or more graphical interfaces. The graphical interfaces can be used by a user computing system 122 to access various functions of the interactive computing environment 126. For instance, the interactive computing environment 126 may transmit data to and receive data, such as via the graphical interface, from a user computing system 122 to shift between different states of the interactive computing environment 126, where the different states allow one or more electronic interactions between the user computing system 122 and the client computing system 124 to be performed.

[0029] In some examples, the client computing system 124 may include other computing resources associated therewith (e.g., not shown in FIG. 1), such as server computers hosting and managing virtual machine instances for providing cloud computing services, server computers hosting and managing online storage resources for users, server computers for providing database services, and others. The interaction between the user computing system 122, the client computing system 124, and the risk assessment computing system 102, or any suitable sub-combination thereof may be performed through graphical user interfaces, such as the user interface, presented by the risk assessment computing system 102, the client computing system 124, other suitable computing systems of the computing environment 100, or any suitable combination thereof. The 8 US2008305628361Attorney Docket No.096923-1441781 graphical user interfaces can be presented to the user computing system 122. Application programming interface (API) calls, web service calls, or other suitable techniques can be used to facilitate interaction between any suitable combination or sub-combination of the client computing system 124, the user computing system 122, and the risk assessment computing system 102.

[0030] A user computing system 122 can include any computing device or other communication device that can be operated by a user or entity, such as the user entity, which may include a consumer or a customer. The user computing system 122 can include one or more computing devices such as laptops, smartphones, and other personal computing devices. A user computing system 122 can include executable instructions stored in one or more non-transitory computer-readable media. The user computing system 122 can additionally include one or more processing devices configured to execute program code to perform various operations. In various examples, the user computing system 122 can allow a user to access certain online services or other suitable products, services, or computing resources from a target entity, such as the client computing system 124, to engage in mobile commerce with the client computing system 124, to obtain controlled access to electronic content, such as the interactive computing environment 126, hosted by the client computing system 124, etc.

[0031] In some examples, the user or a target entity can use the user computing system 122 to engage in an electronic interaction with the client computing system 124 via the interactive computing environment 126. The risk assessment computing system 102 can receive a request, for example from the user computing system 122, to access the interactive computing environment 126 and can use target entity data or any other suitable data or signals determined therefrom, to determine whether to provide access, to challenge the request, to deny the request, etc. An electronic interaction between the user computing system 122 and the client computing system 124 can include, for example, the user computing system 122 being used to request a financial loan or other suitable services or products from the client computing system 124, and so on. An electronic interaction between the user computing system 122 and the client computing system 124 can also include, for example, one or more queries for a set of sensitive or otherwise controlled data, accessing online financial services provided via the interactive computing environment 126, submitting an online credit card application or other digital application to the client computing system 124 via the interactive computing environment 126, operating an electronic tool within the 9 US2008305628361Attorney Docket No.096923-1441781 interactive computing environment 126 (e.g., a content-modification feature, an application- processing feature, etc.), etc.

[0032] In some aspects, an interactive computing environment 126 implemented through the client computing system 124 can be used to provide access to various online functions. As a simplified example, a user interface or other interactive computing environment 126 provided by the client computing system 124 can include electronic functions for requesting computing resources, online storage resources, network resources, database resources, or other types of resources. In another example, a website or other interactive computing environment 126 provided by the client computing system 124 can include electronic functions for obtaining one or more financial services, such as an asset report, management tools, credit card application and transaction management workflows, electronic fund transfers, etc.

[0033] A user computing system 122 can be used to request access to the interactive computing environment 126 provided by the client computing system 124. The client computing system 124 can submit a request, such as in response to a request made by the user computing system 122 to access the interactive computing environment 126, for risk assessment to the risk assessment computing system 102 and can selectively grant or deny access to various electronic functions based on risk assessment performed by the risk assessment computing system 102. Based on the request, or continuously or substantially contemporaneously, the risk assessment computing system 102 can determine one or more risk signals or risk indicators for data associated with the target entity, which may submit or may have submitted the request via the user computing system 122. Based on a risk indicator determined from the risk assessment application 110 (e.g., using the machine learning model 108), the risk assessment computing system 102, the client computing system 124, or a combination thereof can determine whether to grant the access request of the user computing system 122 to certain features of the interactive computing environment 126. The risk assessment computing system 102, the client computing system 124, or a combination thereof can use the risk indicator for other suitable purposes such as identifying a manipulated identity, controlling a real-world interaction, and the like.

[0034] In a simplified example, the system illustrated in FIG. 1 can configure the risk assessment server 104 to be used for controlling access to the interactive computing environment 126. The risk assessment server 104 can retrieve data associated with the target entity in response 10 US2008305628361Attorney Docket No.096923-1441781 to a request to access the interactive computing environment 126. The data may, for example, be retrieved based on identity information (e.g., information collected by the client computing system 124 via a user interface provided to the user computing system 122) provided by the client computing system 124 or received via other suitable computing systems. The risk assessment server 104 can retrieve the data associated with the target entity from the data repository 112. The data repository 112 can store, for example, transaction data associated with one or more entities. The risk assessment server 104 can determine a risk indicator associated with the target entity using a model deployed in a production environment of the risk assessment server 104. The risk assessment server 104 can transmit the risk indicator, or any inference derived therefrom, to the client computing system 124 for use in controlling access to the interactive computing environment 126.

[0035] The risk indicator associated with the target entity, or any suitable score or comparison determined therefrom, can be used, for example by the risk assessment computing system 102, the client computing system 124, etc., to determine whether the risk associated with the target entity accessing a good or a service provided by the client computing system 124 using exceeds a threshold, thereby granting, challenging, or denying access by the target entity to the interactive computing environment 126. For example, if the risk assessment computing system 102 determines that the risk indicator indicates that risk associated with the identity element is lower than a threshold value, then the client computing system 124 associated with the service provider can generate or otherwise provide access permission to the user computing system 122 that requested the access. The access permission can include, for example, cryptographic keys used to generate valid access credentials or decryption keys used to decrypt access credentials. The client computing system 124 can also allocate resources to the target entity and provide a dedicated web address for the allocated resources to the user computing system 122, for example, by adding the user computing system 122 in the access permission. With the obtained access credentials or the dedicated web address, the user computing system 122 can establish a secure network connection to the interactive computing environment 126 hosted by the client computing system 124 and access the resources via invoking API calls, web service calls, HTTP requests, other suitable mechanisms or techniques, etc.

[0036] In some examples, the risk assessment computing system 102 may determine whether to grant, challenge, or deny the access request made by the user computing system 122 for 11 US2008305628361Attorney Docket No.096923-1441781 accessing the interactive computing environment 126. For example, based on the risk indicator associated with the target entity, the risk assessment computing system 102 can determine that the target entity is a legitimate entity that made the access request and may authenticate the request. In other examples, the risk assessment computing system 102 can challenge or deny the access attempt if the risk assessment computing system 102 determines that the target entity may not be a legitimate entity.

[0037] In some examples, the risk indicator used to determine access to the interactive computing environment 126 may be determined at least in part based on output from one or more machine learning models deployed in a production environment of the risk assessment computing system 102. For example, the risk assessment application 110 can retrieve target entity data from entity data 118 in the data repository 112 and can apply the target entity data to a trained machine learning model (e.g., machine learning model 108) to determine a risk indicator. The machine learning model 108 can be deployed to a production environment of the risk assessment computing system 102 from a test environment of the risk assessment computing system 102 using the ATP application 114. Thus, the workflow orchestrator 116 can automatically run one or more testing and deployment steps, after which the machine learning model 108 can be deployed to production and used to serve requests for risk indicators.

[0038] Each communication within the computing environment 100 may occur over one or more data networks, such as a public data network 128, a network 130 such as a private data network, or some combination thereof. A data network may include one or more of a variety of different types of networks, including a wireless network, a wired network, or a combination of a wired and wireless network. Examples of suitable networks include the Internet, a personal area network, a local area network (“LAN”), a wide area network (“WAN”), or a wireless local area network (“WLAN”). A wireless network may include a wireless interface or a combination of wireless interfaces. A wired network may include a wired interface. The wired or wireless networks may be implemented using routers, access points, bridges, gateways, or the like, to connect devices in the data network.

[0039] The number of devices illustrated in FIG. 1 is provided for illustrative purposes. Different numbers of devices may be used. For example, while certain devices or systems are shown as single devices in FIG.1, multiple devices may instead be used to implement these devices 12 US2008305628361Attorney Docket No.096923-1441781 or systems. Similarly, devices or systems that are shown as separate may be instead implemented in a signal device or system. Example of an Architecture for Implementing an ATP Tool

[0040] FIG. 2 is a block diagram depicting an example architecture 200 for implementing an ATP tool for deploying a model to a production environment of the risk assessment computing system 102 according to some aspects of the present disclosure. The architecture 200 can be implemented using any of the components as described above with reference to FIG. 1. For example, the operations described with respect to FIG. 2 can be executed by the risk assessment server 102 via one or both of public network 128 and network 130. Other implementations or architectures, however, are possible. The process 200 can occur over a number of subsequent time intervals (e.g., days, weeks, months, etc.). In some aspects, certain components described with respect to FIG.2 can reside in other computing systems or components.

[0041] The client computing system 124, or another computing system or component of the risk assessment computing system 102, can initiate a session in a browser 202 for interacting with an ATP tool (e.g., ATP application 114). An exemplary GUI will be discussed further with reference to FIG.4, below. Through the browser 202, the client computing system 124 can interact with the ATP server 106. For example, the client computing system 124 can upload a model that a user wishes to move from a testing environment to a production environment.

[0042] The ATP server 106 can include or can communicate with a cloud database 204. The cloud database 204 can be a relational database and can maintain maintenance operations, such as backups, high availability and failover, network connectivity, import and export, maintenance and updates, monitoring, logging, etc. Accordingly, the cloud database 204 can store and manage workflow and process audit information that can be used by the ATP server 106 and the other components of the system 200 (e.g., the cloud composer 206, the automation server 208, and the workflow orchestrator 116) to manage and execute workflows and processes for testing and deploying a model.

[0043] The cloud composer 206 can be, for example, a cloud-based system or server configured to manage pipelines. For example, the cloud composer 206 can be an environment in which pipelines are configured as directed acyclic graphs (DAGs). The cloud composer 206 can facilitate visualization of workflows managed by the ATP application 114. For example, the cloud 13 US2008305628361Attorney Docket No.096923-1441781 composer 206 can provide insight, e.g., via a GUI displayed in the browser 202, into workflow statuses and errors. Thus, a user can view the statuses of each workflow associated with testing and deploying a model, which facilitates troubleshooting and allows transparency and visibility into the processes or workflows managed by the ATP application 114.

[0044] The automation server 208 can be a separate server or can be part of the ATP server 106. The automation server 208 can be a system, such as Jenkins, that runs in servlet containers and automates tasks related to testing and deploying software. For example, the automation server 208 can be integrated to automate and control workflow tasks associated with testing and deploying the machine learning model 108.

[0045] The workflow orchestrator 116 can kick off, monitor, and manage pipelines or workflows associated with different test and deployment tasks. For example, the workflow orchestrator 116 can initiate a test process in which the automation server 208 automatically runs one or more test steps. Upon successful completion of each step, the automation server 208 can proceed to a following step in the workflow. Once the test workflow is completed successfully, the workflow orchestrator 116 can initiate a process for deploying the model to a production environment, which may include several sub-steps. The steps and sub-steps in each workflow or pipeline can be managed by the automation server 208, while the workflow orchestrator 116 manages the workflow or pipelines associated with processes such as testing, deployment, quality assurance (QA), performance monitoring, etc. In some examples, the system 200 can include one or more components configured for load balancing to manage resources used in executing each workflow. In some examples, the orchestrator 116 can manage workflows and loads associated with each workflow using a dynamic model.

[0046] The workflow orchestrator 116 can also interact with cloud storage 210. The cloud storage 210 can store test data, audit data, analytics data, or any other data need ed to execute the workflows. This can include test data, which can be ingested from one or more data sources. In some examples, the test data is cleaned and standardized to a single format before it is stored in the cloud storage 210.

[0047] During execution of the workflows by the workflow orchestrator 116, and execution of each task of the workflows by the automation server 208, the cloud database 204 can store analytics information associated with each task and workflow. For example, analytics information can be 14 US2008305628361Attorney Docket No.096923-1441781 runtimes, error messages, alerts, or any other information that can be used to provide insight into the execution and status of each task and workflow. In some examples, the analytics information stored in the cloud database 204 can be accessible, via a browser 202, to a third-party user, such as an auditor. This enables a third-party to have visibility into the deployment process, for troubleshooting and auditing. Techniques for Communicating with an ATP Application

[0048] FIG. 3 is a flow chart illustrating an example of a process 300 for interacting with the ATP application 114 according to some aspects of the present disclosure. In some examples, the operations of the process 300, or any subset thereof, may be performed by the risk assessment computing system 102 via the risk assessment server 104, but other suitable systems, devices, or subsets or combinations thereof may perform one or more operations described with respect to the process 300. For illustrative purposes, the process 300 is described with reference to certain examples depicted in the figures. Other implementations, however, are possible.

[0049] At operation 302, the process 300 can include querying, by the workflow orchestrator 116, the cloud database 204 for information indicating any jobs pending completion. For example, the cloud database 204 can store a job status associated with a model being tested and deployed using the ATP application 114. The job status can indicate for example, a percent of process completion or a stage in a workflow or task that is pending for a particular model.

[0050] At operation 304, the cloud database 204 can return a list of any jobs pending completion. Thus, the workflow orchestrator 116 can receive a list of models at various stages of the testing and deployment process. The list of pending jobs can include additional data or metadata associated with each model such as a version number, product association, or repository location.

[0051] At operation 306, the workflow orchestrator 116 can communicate with the automation server 208 to retrieve a status of the pending jobs. For example, the automation server 208 can facilitate visibility into each workflow and task and provide status information associated with each pending job. Status information for a job can include, for example, a current task and a current workflow being executed on the model associated with the job. In another example, status information can include an estimated completion time, a percentage of task or workflow completion, a current runtime, or any error messages associated with the task or workflow. 15 US2008305628361Attorney Docket No.096923-1441781

[0052] At operation 308, the workflow orchestrator 116 can transmit the information from the automation server 208 to the cloud data base 204 for storage. Thus, a user of the client computing system 124 can query the cloud database 204 to retrieve job statuses. In some examples, the operations 302 to 308 can be completed at predetermined intervals, such that the status of pending jobs is periodically updated in the cloud database 204.

[0053] At operation 310, the web browser 202 can initiate a process to refresh a GUI or interface displaying the statuses of pending and completed jobs. For example, the browser 202 can communicate with the application server 106, causing the application server 106, at operation 312, to query the cloud database 204. The cloud database 204 can return the statuses of each pending or completed job and any associated analytics or audit information.

[0054] At operation 314, the cloud database 204 returns this data to the application server 106. The application server 106 can, in some examples, analyze the received data to generate any additional metrics associated with the execution of each job. In some examples, the application server 106 can generate one or more job visualizations for graphically displaying job status and progress. At operation 316, the job status information and any additional information can be transmitted, via the web browser 202, for display to the use via the client computing system 124 at operation 318. As described below with reference to FIG. 4, the client computing system 124 can display, in a web browser 202, a GUI including the job status information retrieved from the cloud database 204. Example GUI Associated with an ATP Application

[0055] FIG. 4 is an illustration of an example GUI 400 for displaying job status information. For example, a job can be associated with testing and deploying a model in a production environment. The GUI 400 can display information associated with the job to a user via a display of the client computing system or a display of the risk assessment computing system 102.

[0056] The GUI 400 can display status information associated with each job, such as a sequence or job ID, artifacts, job run time, job progress, and workflow and task statuses. Each horizontal row may be associated with a job having a job or sequence ID for identifying the job in the cloud database 204. The GUI can display a run time of the job (e.g., 31 minutes and 53 seconds) as well as a progress bar visually representing the percentage of the job that has been completed. 16 US2008305628361Attorney Docket No.096923-1441781 Thus, a user can view the time the job has been executing, as well as a visualization of the job’s progress based on data retrieved by the ATP application 114 from the cloud database 204.

[0057] In some examples, the GUI 400 can display one or more selectable icons (e.g., icon 402) for viewing or downloading the analytics and status information associated with each job. In some examples, by selecting the icon 402, the user can cause the ATP application 114 to generate a downloadable file containing job status and analytic information. In other examples, selecting the icon 402 can cause a pop-up window with additional job status and analytics information to be displayed. In another example, one or more additional icons can be displayed in the artifacts column to enable a user to explore the data and analytics information associated with each job.

[0058] The GUI 400 can further display a set of columns for each workflow (e.g., “Security Scan,” “Deployment,” “QA,” etc.) and any sub-workflows (e.g., within “Deployment” sub- workflows can be “Build Image,” “Deploy Metadata,” and “Deploy Model”). Further, the GUI can display a set of columns for steps, or tasks, associated with each workflow and sub-workflow. The GUI 400 can display a completion status for each step with an icon or other indicator of process status. Thus, a user can be provided with a high-level view of workflow and job status. In some examples, by clicking on the status icon 404, a user can be provided with a detailed view including workflow and step, or task, metrics and analytics. Thus, a user can view, for example, the time it took to complete step 2 of the “Security Scan” process for a given job. If a step is pending or failed, the user can select the icon to view information including any error messages or metrics that did not pass testing or deployment requirements, allowing these issues to be addressed efficiently by the user.

[0059] In some examples, various workflows, sub-workflows, or steps / tasks can be initiated using the GUI 400 (e.g., by selecting an icon 406). For example, if a step has failed, the ATP application 114 stores information associated with the previous successful workflows. Accordingly, the job does not need to re-run the previously successful steps and the job can be re- initiated from the failed step. Thus, the GUI 400 and the ATP application 114 provide a flexible and dynamic platform for automating model testing and deployment. The ATP application 114 can improve risk assessment accuracy and efficiency by increasing the speed with which a model can be deployed to production. For example, the ATP application 114 provides visibility into events during execution of the end-to-end pipeline and workflows, the overall execution time of 17 US2008305628361Attorney Docket No.096923-1441781 each job broken down by step and workflow, platform logs for troubleshooting, and user input, thereby allowing tracking for multiple iterations of input. Techniques for Generating a Risk Indicator associated with a Target Entity

[0060] FIG. 5 is a flow chart illustrating an example of a process 500 for generating a risk assessment associated with a target entity according to some aspects of the present disclosure. In some examples, the operations of the process 500, or any subset thereof, may be performed by the risk assessment computing system 102 via the risk assessment server 104, but other suitable systems, devices, or subsets or combinations thereof may perform one or more operations described with respect to the process 500. For illustrative purposes, the process 300 is described with reference to certain examples depicted in the figures. Other implementations, however, are possible.

[0061] At block 502, the process 500 involves receiving a request for a risk indicator associated with a target entity. The request can include an identifier associated with the target entity. The identifier can be, for example, PII such as an SSN, or a combination of PII such as a name and DOB. The request may be generated as part of an authentication process initiated when the target entity attempts to access an interactive computing environment 126.

[0062] At block 504, the process 500 involves running, by the risk assessment computing system 102, an ATP tool, or application, on the model. As discussed above, the ATP application 114 can initiate a job associated with a model. Once the job is initiated the workflow orchestrator 116 can initiate and manage one or more sequential workflows for testing and deploying the model in a production environment of the risk assessment computing system 102. In some examples, the ATP application 114 can execute one or more steps of a set of deployment steps (e.g., a workflow or pipeline) using the model, store analytics information associated with each step, and determine a completion status of each of the set of deployment steps for the model. In some examples, the process 500 may involve authenticating a user of the ATP tool. Upon authentication, the ATP tool can generate a cookie associated with a user ID of the user such that the user can be authenticated in different test and production environments.

[0063] At block 506, the process 500 involves deploying the model to a production environment of the risk assessment computing system 102 when all the steps are successfully completed. In some examples, if a step is failed, the ATP application 114 can transmit an alert to 18 US2008305628361Attorney Docket No.096923-1441781 the client computing system 124 and display, via GUI 400, an indication of at which step the process failed and within which workflow.

[0064] At block 508, the process 500 involves determining, by the risk assessment application 106, a risk indicator for the target entity using the deployed model. Data associated with the target entity can be retrieved from the data repository 112 and can be applied to the model to generate a risk indicator associated with the target entity. The ATP application 114 can continually process jobs (e.g., test and deploy models) such that the most recent model version is readily available.

[0065] At block 510, the process 500 involves transmitting, to a remote computing device (e.g., the client computing device 124), a responsive message comprising at least the risk indicator for use in controlling access of the target entity to one or more interactive computing environments. For example, the risk indicator can be used in controlling an interaction involving a target entity or access of the target entity to a restricted system (e.g., the interactive computing environment 126). Example of Computing System

[0066] Any suitable computing system or group of computing systems can be used to perform the operations for the techniques described herein. For example, FIG. 6 is a block diagram depicting an example of a computing device 600, which can be used to implement the risk assessment server 104. The computing device 600 can include various devices for communicating with other devices in the computing environment 100, as described with respect to FIG. 1. The computing device 600 can include various devices for performing one or more operations, such as risk assessment operations, described above with respect to FIGs.1-5.

[0067] The computing device 600 can include a processor 602 that can be communicatively coupled to a memory 604. The processor 602 can execute computer-executable program code stored in the memory 604, can access information stored in the memory 604, or both. Program code may include machine-executable instructions that may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may 19 US2008305628361Attorney Docket No.096923-1441781 be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, among others.

[0068] Examples of a processor 602 can include a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or any other suitable processing device. The processor 602 can include any suitable number of processing devices, including one. The processor 602 can include or communicate with a memory 604. The memory 604 can store program code that, when executed by the processor 602, causes the processor 602 to perform the operations described herein.

[0069] The memory 604 can include any suitable non-transitory computer-readable medium. The computer-readable medium can include any electronic, optical, magnetic, or other storage device capable of providing a processor with computer-readable program code or other program code. Non-limiting examples of a computer-readable medium can include a magnetic disk, memory chip, optical storage, flash memory, storage class memory, ROM, RAM, an ASIC, magnetic storage, or any other medium from which a computer processor can read and execute program code. The program code may include processor-specific program code generated by a compiler or an interpreter from code written in any suitable computer-programming language. Examples of suitable programming language can include Hadoop, C, C++, C#, Visual Basic, Java, Python, Perl, JavaScript, ActionScript, etc.

[0070] The computing device 600 may also include a number of external or internal devices such as input or output devices. For example, the computing device 600 is illustrated with an input / output interface 608 that can receive input from input devices or provide output to output devices. A bus 606 can also be included in the computing device 600. The bus 606 can communicatively couple one or more components of the computing device 600.

[0071] The computing device 600 can execute program code 614 that can include risk assessment application 110 and the ATP application 114. The program code 614 for the risk assessment application 110 and the ATP application 114 may be resident in any suitable computer- readable medium and may be executed on any suitable processing device. For example, and as illustrated in FIG. 6, the program code 614 for the risk assessment application 110 and the ATP application 114 can reside in the memory 604 at the computing device 600 along with the program data 616 associated with the program code 614. Executing the risk assessment application 110 and 20 US2008305628361Attorney Docket No.096923-1441781 the ATP application 114 can configure the processor 602 to perform at least a portion of the operations described herein.

[0072] In some aspects, the computing device 600 can include one or more output devices. One example of an output device can be or include the network interface device 610 illustrated in FIG. 6. A network interface device 610 can include any device or group of devices suitable for establishing a wired or wireless data connection to one or more data networks described herein. Non-limiting examples of the network interface device 610 can include an Ethernet network adapter, a modem, etc.

[0073] Another example of an output device can include the presentation device 612 depicted in FIG. 6. A presentation device 612 can include any device or group of devices suitable for providing visual, auditory, or other suitable sensory output. Non-limiting examples of the presentation device 612 can include a touchscreen, a monitor, a speaker, a separate mobile computing device, etc. In some aspects, the presentation device 612 can include a remote client- computing device that communicates with the computing device 600 using one or more data networks described herein. In other aspects, the presentation device 612 can be omitted.

[0074] The foregoing description of some examples has been presented only for the purpose of illustration and description and is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Numerous modifications and adaptations thereof will be apparent to those skilled in the art without departing from the spirit and scope of the disclosure. 21 US2008305628361

Claims

Attorney Docket No.096923-1441781 Claims What is claimed is:

1. A computer-implemented method comprising: accessing, by a processor, a request for a risk indicator associated with a target entity; accessing, by the processor, a model in a test environment; running, by the processor, an analytics-to-production (ATP) tool on the model, wherein the ATP tool is configured to: execute one or more steps of a set of deployment steps using the model, store analytics information associated with each step, and determine a completion status of each of the set of deployment steps for the model; deploying, by the processor, the model to a production environment upon determining that the completion status of each of the set of deployment steps is successful; determining, by the processor, the risk indicator using the model; and transmitting, by the processor to a remote computing device, a responsive message comprising at least the risk indicator to control access of the target entity to one or more interactive computing environments.

2. The method of claim 1, wherein at least one step of the set of deployment steps comprises: accessing testing data in one or more databases; and standardizing the testing data such that the testing data is in a single format.

3. The method of claim 2, wherein the method further comprises: ingesting the testing data from the one or more databases into a cloud database accessible through an API associated with the ATP tool.

4. The method of claim 1, wherein the ATP tool is further configured to: implement a workflow orchestrator to facilitate workflows associated with each of the set of deployment steps. US2008305628361Attorney Docket No.096923-1441781 5. The method of claim 4, wherein the ATP tool is further configured to: manage workflows and loads associated with each workflow using a dynamic model.

6. The method of claim 1, wherein storing the analytics information comprises storing the analytics information in a cloud database accessible to a third-party auditor.

7. The method of claim 1, wherein the ATP tool is further configured to: authenticate a user of the ATP tool; and upon authentication, generate a cookie associated with a user ID of the user such that the user can be authenticated in different test and production environments.

8. A system comprising: a processor; and a non-transitory computer-readable medium comprising instructions that are executable by the processor for causing the processor to perform operations comprising: receiving a request for a risk indicator associated with a target entity; accessing a request for a risk indicator associated with a target entity; accessing a model in a test environment; running an analytics-to-production (ATP) tool on the model, wherein the ATP tool is configured to: execute one or more steps of a set of deployment steps using the model, store analytics information associated with each step, and determine a completion status of each of the set of deployment steps for the model; deploying the model to a production environment upon determining that the completion status of each of the set of deployment steps is successful; determining the risk indicator using the model; and transmitting, to a remote computing device, a responsive message comprising at least the risk indicator to control access of the target entity to one or more interactive computing environments. 23 US2008305628361Attorney Docket No.096923-1441781 9. The system of claim 8, wherein at least one step of the set of deployment steps comprises: accessing testing data in one or more databases; and standardizing the testing data such that the testing data is in a single format.

10. The system of claim 9, wherein the operations further comprise: ingesting the testing data from the one or more databases into a cloud database accessible through an API associated with the ATP tool.

11. The system of claim 9, wherein the ATP tool is further configured to: implement a workflow orchestrator to facilitate workflows associated with each of the set of deployment steps.

12. The system of claim 11, wherein the ATP tool is further configured to: manage workflows and loads associated with each workflow using a dynamic model.

13. The system of claim 8, wherein storing the analytics information comprises storing the analytics information in a cloud database accessible to a third-party auditor.

14. The system of claim 8, wherein the ATP tool is further configured to: authenticate a user of the ATP tool; and upon authentication, generate a cookie associated with a user ID of the user such that the user can be authenticated in different test and production environments.

15. A non-transitory computer-readable storage medium having program code that is executable by a processor device to cause a computing device to perform operations, the operations comprising: receiving a request for a risk indicator associated with a target entity; accessing a request for a risk indicator associated with a target entity; accessing a model in a test environment; running an analytics-to-production (ATP) tool on the model, wherein the ATP tool is configured to: 24 US2008305628361Attorney Docket No.096923-1441781 execute one or more steps of a set of deployment steps using the model, store analytics information associated with each step, and determine a completion status of each of the set of deployment steps for the model; deploying the model to a production environment upon determining that the completion status of each of the set of deployment steps is successful; determining the risk indicator using the model; and transmitting, to a remote computing device, a responsive message comprising at least the risk indicator to control access of the target entity to one or more interactive computing environments.

16. The non-transitory computer-readable storage medium of claim 15, wherein at least one step of the set of deployment steps comprises: accessing testing data in one or more databases; and standardizing the testing data such that the testing data is in a single format.

17. The non-transitory computer-readable storage medium of claim 16, wherein the operations further comprise: ingesting the testing data from the one or more databases into a cloud database accessible through an API associated with the ATP tool.

18. The non-transitory computer-readable storage medium of claim 15, wherein the ATP tool is further configured to: implement a workflow orchestrator to facilitate workflows associated with each of the set of deployment steps.

19. The non-transitory computer-readable storage medium of claim 18, wherein the ATP tool is further configured to: manage workflows and loads associated with each workflow using a dynamic model. 25 US2008305628361Attorney Docket No.096923-1441781 20. The non-transitory computer-readable storage medium of claim 15, wherein storing the analytics information comprises storing the analytics information in a cloud database accessible to a third-party auditor. 26 US2008305628361

Images