Constrained non-linear hybrid models for prediction from multiple data sources

Abstract

In some aspects, a machine learning (ML) model can be trained for risk assessment. The ML model can be trained to determine a risk indicator for a target entity from predictor variables associated with the target entity. The predictor variables are obtained from multiple sources with varying availability, and the training of the ML model is accomplished based on a multi-dimensional representation of common information from the set of data sources. Once generated, the risk indicator can be transmitted to a remote computing device in a responsive message for use in controlling access of the target entity to a computing environment.

Description

Attorney Docket No.096923-1449567 CONSTRAINED NON-LINEAR HYBRID MODELS FOR PREDICTION FROM MULTIPLE DATA SOURCES Cross-Reference to Related Application

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 518,246, filed on August 8, 2023, and entitled “CONSTRAINED NON-LINEAR HYBRID MODELS FOR PREDICTION FROM MULTIPLE DATA SOURCES,” the contents of which are hereby incorporated by reference in their entirety for all purposes. Technical Field

[0002] The present disclosure relates generally to artificial intelligence for risk prediction. More specifically, but not by way of limitation, this disclosure relates to controlling access to secure resources based on a risk assessment generated based on multiple data sources. Background

[0003] In machine learning, data from multiple data sources can be used to train machine- learning models. But in certain cases, an unknown quantity of data may be available from the multiple data sources. For example, only a subset of those data sources may be available for a given observation, either in model training or for prediction. The distinct combinations of data sources present per^observation in the training data may not cover all the possible combinations that may be seen at prediction time. Thus, it is difficult to implement a model that can make statistically robust predictions from any combination of the training data sources, including where that particular combination does not exist or is not well represented in the training data, and where the total number of data sources may be large. Summary

[0004] Various aspects of the present disclosure provide systems and methods for generating constraint compliant training data and constraint compliant machine-learning US2008302308241Attorney Docket No.096923-1449567 models for use in risk assessment. In one example, a method includes one or more processing devices performing operations including accessing a risk assessment query including an indication of a target entity. The operations further include accessing a machine-learning model that is trained to determine a risk indicator for a target entity from predictor variables associated with the target entity. The predictor variables are obtained from multiple sources with varying availability, and the training of the machine-learning model is accomplished based on a multi-dimensional representation of common information from the set of data sources. The operations further include generating the risk indicator for the target entity using the trained machine-learning model and the predictor variables. Further, the operations include transmitting, to a remote computing device, a responsive message comprising at least the risk indicator for use in controlling access of the target entity to one or more computing environments.

[0005] In another example, a system includes a processing device; and a memory device in which instructions executable by the processing device are stored for causing the processing device to perform various operations. The operations include accessing a risk assessment query including an indication of a target entity. The operations further include accessing a machine-learning model that is trained to determine a risk indicator for a target entity from predictor variables associated with the target entity. The predictor variables are obtained from multiple sources with varying availability, and the training of the machine- learning model is accomplished based on a multi-dimensional representation of common information from the set of data sources. The operations further include generating the risk indicator for the target entity using the trained machine-learning model and the predictor variables. Further, the operations include transmitting, to a remote computing device, a responsive message comprising at least the risk indicator for use in controlling access of the target entity to one or more computing environments.

[0006] In yet another example, a non-transitory computer-readable storage medium has program code that is executable by a processor to cause a computing device to perform operations. The operations can include accessing a risk assessment query including an indication of a target entity. The operations further include accessing a machine-learning model that is trained to determine a risk indicator for a target entity from predictor variables associated with the target entity. The predictor variables are obtained from multiple sources with varying availability, and the training of the machine-learning model is accomplished based on a multi-dimensional representation of common information from the set of data US2008302308241Attorney Docket No.096923-1449567 sources. The operations further include generating the risk indicator for the target entity using the trained machine-learning model and the predictor variables. Further, the operations include transmitting, to a remote computing device, a responsive message comprising at least the risk indicator for use in controlling access of the target entity to one or more computing environments.

[0007] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification, any or all drawings, and each claim.

[0008] The foregoing, together with other features and examples, will become more apparent upon referring to the following specification, claims, and accompanying drawings. Brief Description of the Drawings

[0009] FIG. 1 is a block diagram depicting an example of an operating environment according to certain aspects of the present disclosure.

[0010] FIG. 2 is an illustration of a hybrid model as a probabilistic graphical model (PGM) according to certain aspects of the present disclosure.

[0011] FIG. 3 is a block diagram depicting an example of a high-level architecture for a hybrid model according to certain aspects of the present disclosure.

[0012] FIG. 4 is a flow chart depicting an example of a process for training a machine- learning model according to certain aspects of the present disclosure.

[0013] FIG. 5 is a block diagram depicting an example of a computing device suitable for implementing aspects of the techniques and technologies presented herein. Detailed Description

[0014] Certain aspects and features of the present disclosure are directed to controlling access to secure resources based on a risk assessment generated based on multiple data sources. For example, in some aspects, the systems and methods disclosed herein address the above-described challenges to building models to make predictions of an unknown quantity from multiple data sources, where it may be the case that only a subset of those data sources is available for any given observation, either in model training or for prediction. In particular, US2008302308241Attorney Docket No.096923-1449567 described systems and methods may provide solutions while avoiding an exponential increase in model complexity as the number of data sources increases linearly.

[0015] One example use-case is that of consumer credit origination, where the quantity to be predicted is a good / bad credit outcome defined over some period of time after account opening, and the data sources represent previous credit performance and other relevant and permissible information related to the applicant at the time of application. This use-case also imposes an explainability requirement on the model, which can be achieved through monotonicity constraints. Examples of data sources for the consumer credit origination use case include: traditional credit file data; alternative data sources providing coverage of other sources of borrowing, e.g. alternative finance; closed user group data, available for decisioning only in certain circumstances or for use by certain lenders; bank transactional data; social media or other non^financial data that may be permissible for use in some regulatory jurisdictions.

[0016] In this use-case, each of these data sources may or may not be available for a given application, either at the point of decisioning or in the historic data used to create a training sample. For example, some data may have varying availability due to: consent requirements; contractual restrictions; data source coverage; consumer opt-out; and keying and linking. Given all of the above reasons why data may be unavailable, the presence or absence of a particular data source cannot be assumed to be independent of the content of that data, nor of the applicant’s credit and financial situation or their application risk. In other words, the data sources should be considered missing not at random (MNAR). Disclosed systems and methods may account for the presence or absence of each data source.

[0017] For instance, systems and methods described herein can include training a machine-learning model to determine a risk indicator for a target entity. The risk indicator can reflect an amount of risk, or a level of riskiness, associated with the target entity, and can be used to control access of the target entity to a protected resource or system. The risk indicator can be determined for the target entity using one or more predictor variables associated with the target entity. However, prediction or training data may not be available from all data sources for any given observation. To account for this missing data, systems and methods may implement a hybrid model using a monotonically constrained encoder that performs partial inference, a monotonic scorer, and an unconstrained decoder. In other words, a hybrid model is used to model common information from the model data sources via a multi-dimensional latent variable. US2008302308241Attorney Docket No.096923-1449567

[0018] Certain aspects described herein provide improvements to machine learning techniques for training and using models based on multiple data sources. For example, the described hybrid model can generate predictions given any combination of data sources in a single model, without fragmentation into multiple segments for different combinations. To improve the chances of good generalization to unseen combinations of data sources, the hybrid model can account for the availability of each data source in a statistically principled way. Further, the hybrid model can account for the relationship between the presence or absence of each data source and the observed variables and outcome. Additionally, the hybrid model can support the generation of accurate local explanations. Specifically, a monotonic relationship can be imposed between the model prediction and the values of the observed variables.

[0019] In addition, disclosed systems and methods improve the field of access control by facilitating training of machine-learning models for predicting risk. For example, risk predictions have improved accuracy through the use of data from multiple data sources to robustly train a machine-learning model. Accordingly, improved predictive power and improved accuracy of machine-learning models also improves an entity’s ability to make accurate and informed decisions on whether to grant access, by a target entity, to a secured or restricted resource.

[0020] These illustrative examples are given to introduce the reader to the general subject matter discussed here and are not intended to limit the scope of the disclosed concepts. The following sections describe various additional features and examples with reference to the drawings in which like numerals indicate like elements, and directional descriptions are used to describe the illustrative examples but, like the illustrative examples, should not be used to limit the present disclosure. Operating Environment Example for Machine-Learning Operations

[0021] Referring now to the drawings, FIG. 1 is a block diagram depicting an example of an operating environment 100 in which a risk assessment computing system 130 builds and trains a risk assessment model 120 that can be trained to predict risk indicators based on training data. FIG. 1 depicts examples of hardware components of a risk assessment computing system 130, according to some aspects. The risk assessment computing system 130 is a specialized computing system that may be used for processing large amounts of data using a large number of computer processing cycles. The risk assessment computing system 130 can include a model training server 110 for building and training a risk assessment model US2008302308241Attorney Docket No.096923-1449567 120 used to predict risk indicators associated with an entity accessing controlled resources. The risk assessment computing system 130 can further include a risk assessment server 118 for performing a risk assessment for given predictor variables 124, or features, using the trained risk assessment model 120.

[0022] The model training server 110 can include one or more processing devices that execute program code, such as a model training application 112. The program code is stored on a non-transitory computer-readable medium. The model training application 112 can execute one or more processes or applications to develop, train, and optimize a risk assessment model 120 for predicting risk indicators based on the predictor variables 124.

[0023] In some aspects, the model training application 112 can build and train a risk assessment model 120 using risk assessment training data 126 in a training process and training data stored in data sources 121. Data sources 121 can store entity data and training data that may or may not overlap or be related. The risk assessment training model 120 can be a hybrid model including a scoring module, an encoding module, and a decoding module as will be described with further detail with reference to FIG. 3. The risk assessment training data 126 can be stored in one or more network-attached storage units on which various repositories, databases, or other structures are stored. An example of these data structures is the risk data repository 122.

[0024] Network-attached storage units can include the risk data repository 122. Network- attached storage units may store a variety of different types of data organized in a variety of different ways and from a variety of different sources. For example, the network-attached storage unit may include storage other than primary storage located within the model training server 110 that is directly accessible by processors located therein. In some aspects, the network-attached storage unit may include secondary, tertiary, or auxiliary storage, such as large hard drives, servers, virtual memory, among other types. Storage devices may include portable or non-portable storage devices, optical storage devices, and various other mediums capable of storing and containing data. A machine-readable storage medium or computer- readable storage medium may include a non-transitory medium in which data can be stored and that does not include carrier waves or transitory electronic signals. Examples of a non- transitory medium may include, for example, a magnetic disk or tape, optical storage media such as a compact disk or digital versatile disk, flash memory, memory, or memory devices.

[0025] The risk assessment server 118 can include one or more processing devices that execute program code, such as a risk assessment application 114. The program code is stored US2008302308241Attorney Docket No.096923-1449567 on a non-transitory computer-readable medium. The risk assessment application 114 can execute one or more processes to use the risk assessment model 120 trained during execution of the model training application 112 to predict risk indicators based on input predictor variables 124. The risk indicators can be used to protect or allocate computing resources of the risk assessment computing system 130.

[0026] Furthermore, the risk assessment computing system 130 can communicate with various other computing systems, such as client computing systems 104. For example, client computing systems 104 may send risk assessment queries to the risk assessment server 118 for risk assessment or may send signals to the risk assessment server 118 that control or otherwise influence different aspects of the risk assessment computing system 130. The client computing systems 104 may also interact with user computing systems 106 via one or more public data networks 108 to facilitate interactions between users of the user computing systems 106 and interactive computing environments provided by the client computing systems 104.

[0027] Each client computing system 104 may include one or more third-party devices, such as individual servers or groups of servers operating in a distributed manner. A client computing system 104 can include any computing device or group of computing devices operated by a seller, lender, or other providers of products or services. The client computing system 104 can include one or more server devices. The one or more server devices can include or can otherwise access one or more non-transitory computer-readable media. The client computing system 104 can also execute instructions that provide an interactive computing environment accessible to user computing systems 106. Examples of the interactive computing environment include a mobile application specific to a particular client computing system 104, a web-based application accessible via a mobile device, etc. The executable instructions are stored in one or more non-transitory computer-readable media.

[0028] The client computing system 104 can further include one or more processing devices that are capable of providing the interactive computing environment to perform operations described herein. The interactive computing environment can include executable instructions stored in one or more non-transitory computer-readable media. The instructions providing the interactive computing environment can configure one or more processing devices to perform operations described herein. In some aspects, the executable instructions for the interactive computing environment can include instructions that provide one or more graphical interfaces. The graphical interfaces are used by a user computing system 106 to US2008302308241Attorney Docket No.096923-1449567 access various functions of the interactive computing environment. For instance, the interactive computing environment may transmit data to and receive data from a user computing system 106 to shift between different states of the interactive computing environment, where the different states allow one or more electronics transactions between the user computing system 106 and the client computing system 104 to be performed.

[0029] In some examples, a client computing system 104 may have other computing resources associated therewith (not shown in FIG. 1), such as server computers hosting and managing virtual machine instances for providing cloud computing services, server computers hosting and managing online storage resources for users, server computers for providing database services, and others. The interaction between the user computing system 106 and the client computing system 104 may be performed through graphical user interfaces presented by the client computing system 104 to the user computing system 106, or through application programming interface (API) calls or web service calls.

[0030] A user computing system 106 can include any computing device or other communication device operated by an entity, such as a user, an organization, or a company. The user computing system 106 can include one or more computing devices, such as laptops, smartphones, and other personal computing devices. A user computing system 106 can include executable instructions stored in one or more non-transitory computer-readable media. The user computing system 106 can also include one or more processing devices that are capable of executing program code to perform operations described herein. In various examples, the user computing system 106 can allow a user to access certain online services from a client computing system 104 or other computing resources, to engage in mobile commerce with a client computing system 104, to obtain controlled access to electronic content hosted by the client computing system 104, etc.

[0031] For instance, the user can use the user computing system 106 to engage in an electronic transaction with a client computing system 104 via an interactive computing environment. An electronic transaction between the user computing system 106 and the client computing system 104 can include, for example, the user computing system 106 being used to request online storage resources managed by the client computing system 104, acquire cloud computing resources (e.g., virtual machine instances), and so on. An electronic transaction between the user computing system 106 and the client computing system 104 can also include, for example, querying a set of sensitive or other controlled data, accessing online financial services provided via the interactive computing environment, submitting an US2008302308241Attorney Docket No.096923-1449567 online credit card application or other digital application to the client computing system 104 via the interactive computing environment, operating an electronic tool within an interactive computing environment hosted by the client computing system (e.g., a content-modification feature, an application-processing feature, etc.).

[0032] In some aspects, an interactive computing environment implemented through a client computing system 104 can be used to provide access to various online functions. As a simplified example, a website or other interactive computing environment provided by an online resource provider can include electronic functions for requesting computing resources, online storage resources, network resources, database resources, or other types of resources. In another example, a website or other interactive computing environment provided by a financial institution can include electronic functions for obtaining one or more financial services, such as loan application and management tools, credit card application and transaction management workflows, electronic fund transfers, etc. A user computing system 106 can be used to request access to the interactive computing environment provided by the client computing system 104, which can selectively grant or deny access to various electronic functions. Based on the request, the client computing system 104 can collect data associated with the user and communicate with the risk assessment server 118 for risk assessment. Based on the risk indicator predicted by the risk assessment server 118, the client computing system 104 can determine whether to grant the access request of the user computing system 106 to certain features of the interactive computing environment.

[0033] In a simplified example, the system depicted in FIG. 1 can train the risk assessment model 120 to determine risk indicators, such as credit scores, using predictor variables 124. A predictor variable 124 can be any variable predictive of risk that is associated with an entity. Any suitable predictor variable that is authorized for use by an appropriate legal or regulatory framework may be used.

[0034] Examples of predictor variables 124 used for predicting the risk associated with an entity accessing online resources include, but are not limited to, variables indicating the demographic characteristics of the entity (e.g., name of the entity, the network or physical address of the company, the identification of the company, the revenue of the company), variables indicative of prior actions or transactions involving the entity (e.g., past requests of online resources submitted by the entity, the amount of online resource currently held by the entity, and so on.), variables indicative of one or more behavioral traits of an entity (e.g., the timeliness of the entity releasing the online resources), etc. Similarly, examples of predictor US2008302308241Attorney Docket No.096923-1449567 variables 124 used for predicting the risk associated with an entity accessing services provided by a financial institute include, but are not limited to, indicative of one or more demographic characteristics of an entity (e.g., age, gender, income, etc.), variables indicative of prior actions or transactions involving the entity (e.g., information that can be obtained from credit files or records, financial records, consumer records, or other data about the activities or characteristics of the entity), variables indicative of one or more behavioral traits of an entity, etc.

[0035] The predicted risk indicator can be used by the service provider (e.g., the service provider controlling the interactive computing environment) to determine the risk associated with the entity accessing a service provided by the service provider, thereby granting or denying access by the entity to an interactive computing environment implementing the service. For example, if the service provider determines that the predicted risk indicator is lower than a threshold risk indicator value, then the client computing system 104 associated with the service provider can generate or otherwise provide access permission to the user computing system 106 that requested the access. The access permission can include, for example, cryptographic keys used to generate valid access credentials or decryption keys used to decrypt access credentials. The client computing system 104 associated with the service provider can also allocate resources to the user and provide a dedicated web address for the allocated resources to the user computing system 106, for example, by adding it in the access permission. With the obtained access credentials and / or the dedicated web address, the user computing system 106 can establish a secure network connection to the computing environment hosted by the client computing system 104 and access the resources via invoking API calls, web service calls, HTTP requests, or other proper mechanisms.

[0036] Each communication within the operating environment 100 may occur over one or more data networks, such as a public data network 108, a network 116 such as a private data network, or some combination thereof. A data network may include one or more of a variety of different types of networks, including a wireless network, a wired network, or a combination of a wired and wireless network. Examples of suitable networks include the Internet, a personal area network, a local area network (“LAN”), a wide area network (“WAN”), or a wireless local area network (“WLAN”). A wireless network may include a wireless interface or a combination of wireless interfaces. A wired network may include a wired interface. The wired or wireless networks may be implemented using routers, access points, bridges, gateways, or the like, to connect devices in the data network. US2008302308241Attorney Docket No.096923-1449567

[0037] The number of devices depicted in FIG. 1 is provided for illustrative purposes. Different numbers of devices may be used. For example, while certain devices or systems are shown as single devices in FIG. 1, multiple devices may instead be used to implement these devices or systems. Similarly, devices or systems that are shown as separate, such as the model training server 110 and the risk assessment server 118, may be instead implemented in a single device or system. Example of a Hybrid Model

[0038] FIG. 2 is an illustration of a hybrid model as a probabilistic graphical model (PGM). As shown in FIG. 2, ^^ᇱfor ^ ൌ 1, … ,^ are vector-valued latent variables representing the "true" values of the partially observed independent variables, one per data source. ^^ᇹfor ^ ൌ 1, … ,^ are the observed "corrupted" values of the same independent variables. If data source ^ is available, then ^^ൌ ^^ᇱ; otherwise ^^is set to some default value ^^, such as zero. ^ is a binary mask vector of dimension ^. Each coordinate ^^is equal to one if data source ^ is available, and zero otherwise. Therefore, ^^ൌ ^^^^ᇱ^^1െ^^^^^where ^^is the default value for data source ^. ^ is a dependent variable. The dependent variable may be discrete or continuous and may be multi-valued. In the credit risk use case, for example, ^ is a binary variable representing good / bad credit outcome. ^ is a latent variable with standard multivariate normal distribution of some dimension ^. For each observation, ^ captures the fundamental characteristics of the entity (individual borrower, in the credit risk use case) that are observed through the independent and dependent variables and the mask. The values of the mask ^, the partially observed independent variables ^^, and the dependent variable ^ are conditionally dependent on the value of ^. In some examples, an additional latent variable ^^, which influences only the mask, may be included. Unlike ^, the independent variables ^^and the dependent variable ^ are not conditionally dependent on ^^. Including ^^as a separate latent variable is optional and does not affect the fitting of the model.

[0039] The arrows in FIG. 2 represent the conditional dependency relationships in the model. In particular, ^^ᇱ, ^, and ^ may be conditionally independent given ^. In other words, ^ and ^ are statistically related only through their common dependence on ^.

[0040] Conditional probability distributions ^^^^ᇱ|^^, ^^^|^^, and ^^^|^^ may be specified in order to complete the mathematical description of the model. In some examples, neural networks can flexibly model the parameters of the conditional ^^^^ᇱ|^^, ^^^|^^, and^^^|^^. Treating ^^ᇱas a continuous variable, ^^^^ᇱ|^^ൌ ^^^^^^^, ^^^is used where the11 US2008302308241Attorney Docket No.096923-1449567 function ^^^^ is modeled by a neural network and is a diagonal covariance matrix. In practice, a single neural network can be used to model the concatenation ൫^^^^^, … , ^ோ^^^൯. Treating ^ as a binary variable, ^^^ ൌ 1|^^ ൌ ^௬^^^ is used where ^௬^^^ is modeled by a neural network with sigmoid activation on the output layer. In the credit risk example, the function ^௬^^ can be constrained to be monotonic.

[0041] Alternative examples can exist. For example, binary coordinates ^^^can be modelled usingൌ 1|^൯ൌ ^^^^^^ with a sigmoid transformation. Nominal and ordinal categorical variables can also be handled by using appropriate conditional probability distributions for1|^൯. If the dependent variable is continuous, an appropriate conditional distribution can be specified for ^, such as a Gaussian ^^^|^^ ൌ ^൫^௬^^^,^௬ଶ൯.

[0042] The variable ^ can, in an example, be used to represent the concatenation of the observed vectors ^^, one for each data source, which include default values for unavailable data sources. It is also possible to use ^^^^ and ^ to represent the concatenations of the mean vectors and diagonal covariance matrices ^^^^^ and ^^. Fitting the Hybrid Model

[0043] It is possible to optimize a variational lower bound on the multi-conditional likelihood shown in Equation 1. Equation 1where the parameter ^ allows assignment of higher importance to the prediction of ^ than to the reconstruction of ^ in the model loss function. The lower bound is derived by thevariational principle and is given by the expression:log κ^^,^,^; ^^ ^ ^^^௭|௫,^^^^ log ^^^|^^ ^ log^^^,^|^^^ െ ^^^൫^^^|^,^^||^^^^൯

[0044] Here, ^^^|^,^^ is a "variational" approximation to the true conditional distribution ^^^|^,^^, the latter being intractable to compute in general. The model therefore consists of three modules, shown in FIG. 3, each implemented via neural networks. The modules may include a decoder 302 that computes ^^^,^|^^ via the conditional mean ^^^^ and fixed covariance ^ of ^^^ᇱ|^^, accounting for the default values ^^for missing data sources. The modules may also include an encoder 304 that computes ^^^|^,^^. In the simplest formulation, ^^^|^,^^ can be assumed to be multivariate normal, and the conditional mean and variance are computed. In some examples, a normalizing flow can be US2008302308241Attorney Docket No.096923-1449567 used to fit a non-Gaussian ^^^|^,^^. Further, the modules may include a scorer 306 thatcomputes ^^^|^^. In the can that ^ is binary, this uses a sigmoid activation to output ^^^ ൌ1|^^.

[0045] The model can be fit using doubly stochastic gradient descent (DSGD) to optimize the weights of the modules to minimize a loss function that includes the negative of the variational bound expressed above, and may also include additional regularization losses. That is, the following equation is minimized, Equation 2: Equation 2^^^௭|௫,^^൫െ^ log ^^^|^^ െ log^^^,^|^^ ^ ^^^൫^^^|^,^^||^^^^൯^ ڮ൯log^^^,^|^^ ^ log ^^^|^,^^ െ log ^^^^ ^ ڮ^ where the ellipses represent any additional losses, such as regularization terms. The expectation term is evaluated stochastically by sampling from the posterior ^^^|^,^^, and reparameterization is used to obtain a gradient. The gradient is aggregated over mini-batches of data before each gradient step is taken. Hence the term doubly stochastic: the first source of stochasticity is sampling from ^^^|^,^^, and the second is the use of mini-batches. A gradient descent algorithm can be used to implement the gradient step, such as Adam.

[0046] In some examples, performance of the model may be improved by allowing the approximate conditional distribution ^^^|^,^^ to be non-Gaussian. This is obtained by specifying a two-stage encoder. The first stage yields a multivariate normal distribution ^^^ᇱ|^,^^ by computing the conditional mean and covariance. The second stage uses a normalizing flow model ^ to transform the new latent variable ^ᇱinto ^ ൌ ^^^ᇱ^. The normalizing flow model may take the form of, for example, an autoregressive flow, and yields a flexible non-Gaussian conditional distribution for ^^^|^,^^. It is still possible to evaluate the expectation term in the loss function by sampling from ^^^|^,^^, as this can be achieved by sampling ^ᇱfrom ^^^ᇱ|^,^^and applying the transformation ^. Evaluating the term log ^^^|^,^^ makes use of the Jacobian determinant of ^. Monotonicity Requirements

[0047] In some use-cases, such as a credit risk, the model predictions may be required to be monotonic in the observed independent variables regardless of which combination of data sources is available, i.e. for any value of the mask. In other words, the output probability ^^^ ൌ 1|^,^^ should be monotonic in ^, though it need not be monotonic in ^. US2008302308241Attorney Docket No.096923-1449567

[0048] Calculation of the exact conditional probability distribution ^^^|^,^^, like ^^^|^,^^ is not tractable, and instead use of the approximation shown in Equation 3 is made. Equation 3Equation 3 uses the variational approximation ^^^|^,^^ to ^^^|^,^^. Techniques for numerical evaluation of this integral will be discussed below. In order to ensure approximate probability ^^^ ൌ 1|^,^^is monotonic in ^, the following constraints may be imposed: ^^^ ൌ 1|^^ is monotonic in z; where ^^^|^,^^ is Gaussian, the mean is monotonic in ^ and the covariance is constant for fixed ^; where ^^^|^,^^ is modeled using a normalizing flow, the flow function ^ᇱհ ^ is monotonic and for the additional latent variable ^ᇱ, the mean is monotonic in ^ and the covariance is constant for fixed ^. In an example, these constraints can be relaxed. For example, the scorer module can take as input only a subset of the coordinates of ^, with ^^^ ൌ 1|^^ monotonic in those coordinates. In that case, only the conditional mean of those "scoring" coordinates need be monotonic in ^, and only the marginal covariance of those coordinates need be constant for fixed ^. Further, if ^^^|^,^^is approximated by evaluating ^^^|^^ at the mean of ^^^|^,^^ where ^^^|^,^^ is Gaussian, which is an example of a strategies discussed below, then the conditional covariance does not need to be constant for fixed ^.

[0049] Enforcing monotonicity on a feed-forward neural network can track the following procedure. In common frameworks, such as Keras, the kernel weights at each node may be constrained to be non-negative. If the network output is only required to be monotonic in a subset of the inputs, then at the first hidden layer only the kernel weights corresponding to those inputs need be constrained; in subsequent layers all kernel weights must still be constrained. As an additional solution for frameworks that do not offer this much flexibility, if ^ is an input variable for which monotonicity is not required then both ^ and (െ^) may be provided as inputs while all kernel weights are constrained to be non-negative. In this way, each node in the first hidden layer may learn either a positive or negative response to input ^ while learning only positive responses to other inputs. Principled Encoder

[0050] To ensure the model generalizes well to combinations of data sources that are either not present or sparse in the training data, the computation of the conditional mean and covariance of ^ given ^ is not treated as simply function fitting. Instead, a statistically US2008302308241Attorney Docket No.096923-1449567 principled approach is taken. It is possible to use a partial inference net to calculate the parameters of ^^^|^,^^. The chosen form of the partial inference net must enable inferences to be drawn from the value of the mask ^, to account for the non-random missingness of the data sources.

[0051] One suitable choice of partial inference net is an extension of the Pointnet Plus (PNP), which aggregates information from the observed variables (and the mask) in a high dimensional space, before transforming into the parameter space of ^^^|^,^^. Specifically, it is possible to compute the mean and covariance of ^^^|^,^^as follows.

[0052] Beginning with a high-dimensional term ^^^,^^ ൌ ^^^^^^^^^^ ^ ^^^^^ that sums a contribution ^^^^^^ from each observed data source ^^, and a term ^^^^^ that depends linearly on the value of the mask, it is possible to apply a non-linear transformation to obtain ^^, ^^ ൌ ^൫^^^,^^൯ where ^ and ^ are the mean and covariance of ^^^|^,^^.

[0053] This formulation has the effect of adding information from the available data sources and the mask in a certain space, before transforming that information into the parameter space for ^^^|^,^^. This mimics the computation of the conditional mean and covariance of factor values given observed variables in factor analysis, i.e. in the case of a linear decoder, where the conditional mean is computed as a weighted average of estimates given by each independent variable, plus the prior mean, weighted by precision terms. In the linear case, the (precision x mean) contributions can be summed and then divided by the sum of the precisions.

[0054] The functions ^^^^, ^^^^, and ^^^ may be implemented as neural networks. Further constraints may be imposed to more closely mimic a sum of Gaussian conditional distributions ^^^|^^^. For example, ^^^^ may be constrained to be a sum of (non-linear) contributions from each of the individual variables in ^^.Rather than allowing ^^^ to be an arbitrary non-linear transformation, it is possible to design ^ so that some of the coordinates of ^^^,^^represent ^ି^^, i.e. (precision x mean) for ^, and others represent the precision ^ି^, and implement ^^^ as calculation of ^ by matrix algebra. ^ି^may depend only on ^, as in the linear case the precision depends only on which variables are observed, and not on their values. This approach is likely to be computationally expensive unless ^ is assumed to be diagonal.

[0055] Conversely, some of the existing constraints can be relaxed. For example, ^^^ may be relaxed by allowing ^^^^^^^^^^ to be combined with ^^^^^ in a non-linear US2008302308241Attorney Docket No.096923-1449567 manner. Additionally, ^^^^ may be a non-linear function of the mask. But relaxation of the principled approach of adding information from each data source may reduce the benefit of better generalization. Combining Monotonicity with the Principled Encoder

[0056] The simple enforcement of monotonicity via non-negativity constraints on kernel weights may not yield optimal results when applied to the particular architecture of the principled encoder. In this case, an alternative approach in which two encoders can be fitted may be used. The two encoders may be a principled encoder and a monotonic encoder, and the encoders may be trained to yield very similar outputs.

[0057] Specifically, let ^^^^,^^be a monotonically constrained neural network, which is designed to be a monotonic encoder, and let ^^^^,^^ be a partial inference net, which is designed to be a principled encoder. Both ^^^^ and ^^^^ output estimates of the conditional mean and variance ^ and ^ of ^^^|^,^^. In applying the model, it is possible use the estimates output by ^^^^ in order to ensure monotonicity. In fitting the model, it is also possible to use the estimates output by ^^^^ when sampling from ^^^|^,^^, in order to ensure consistency between model fitting and application (although ^^^^ can be used in some preliminary training epochs to accelerate training). In order to 'couple' the two encoders to yield similar outputs, a penalty term can be added to the model loss function of the formwhere the norm used is either the L1 or L2 norm, and ^ is a hyperparameter that may be tuned.

[0058] A monotonic encoder is yielded that outputs similar values to the principled encoder, at least for combinations of data sources that appear in the training data. In order to further couple the two encoders so that the two encoders output similar values for all combinations of data sources, additional training steps can be introduced, where synthetic training data is used and the loss function consists only of the penalty termെ ^^^^,^^ฮ. The synthetic data can be created from real data by making some data sources artificially unavailable, to form a more complete coverage of the possible combinations of available data sources. In these additional training steps, a value for the outcome variable need not be supplied and it is not necessary to sample from ^^^|^,^^ and reconstruct ^. The weights of the principled encoder may be held fixed. The purpose of these training steps is to train the monotonic encoder to output similar values to the principled encoder for new combinations of data sources. US2008302308241Attorney Docket No.096923-1449567 Applications of the Hybrid Model

[0059] In application, the hybrid model can be used to generate a prediction of the dependent variable ^. In some examples, such as in the credit risk use case, ^ is a binary outcome and the prediction takes the form of an outcome probability ^^^ ൌ 1|^,^^. In other use cases, the outcome variable may be continuous, and the prediction may take the form of an expected value ^^^|^,^^. In this example, let y be a binary outcome. Thus, it can be shown that approximations of the predictive probability ^^^ ൌ 1|^,^^may be produced, and that they can be made monotonic in ^.

[0060] Given the mathematical formulation of the model, the true predictive probability is given by an integral shown in Equation 4. Equation 4

[0061] The true conditional distribution ^^^|^,^^ may be analytically intractable, and can be replaced with the approximate conditional distribution ^^^|^,^^ whose parameters are computed by the encoder 304, yielding Equation 5. Equation 5

[0062] ^^^ ൌ 1|^^ can be computed by the scorer 306, and ^^^|^,^^ may be Gaussian or (if a normalizing flow is used), it is possible to have ^ ൌ ^^^ᇱ^where ^ᇱhas a Gaussian distribution ^^^ᇱ|^,^^. The latter case yields Equation 6: Equation 6where ^^^ᇱ|^,^^ is Gaussian and ^^^ ൌ 1|^ᇱ^ is given by the composition of the normalizing flow and the scorer 306.

[0063] The integralmay be made monotonic in ^ for fixed ^: if the covariance of ^^^|^,^^ is fixed for ^, themean of ^^^|^, ^^is monotonic in ^ and ^^^ ൌ 1|^^is monotonic in ^, then the value of theabove integral will be monotonic in ^. Furthermore, if ^^^ ൌ 1|^^ depends ona subset of the US2008302308241Attorney Docket No.096923-1449567 coordinates of ^, then only those coordinates of the mean of ^^^|^,^^ need be monotonic in ^.

[0064] But the integralis still intractable in general since ^^^ ൌ 1|^^ is non-linear. There may be three options for computing an approximation to ^^^ ൌ 1|^^, as described below. Whichever method is chosen, the approximate value can be used in place of the true probability as the predictive output of the model.

[0065] The first option for approximating ^^^ ൌ 1|^^ is mean approximation. In thiscase, Equation 5 can be replaced with evaluation of ^^^ ൌ 1|^^at the mean of ^^^|^,^^. Itis not necessary for the covariance of ^^^|^,^^ to be fixed for ^: if the mean of ^^^|^,^^ is monotonic in ^ for fixed ^, and ^^^ ൌ 1|^^ is monotonic in ^, then the approximate predictive probability will be monotonic in ^.

[0066] In the case of a non-Gaussian encoder, the integral shown in Equation 6 can be replaced with evaluation of ^^^ ൌ 1|^ᇱ^ at the mean of ^^^ᇱ|^,^^. In this case, the composition of the normalizing flow and the scorer 306 is monotonic in ^ᇱ. This can be achieved by making each of them monotonic in their inputs.

[0067] The second option for approximating ^^^ ൌ 1|^^ is simulation. In this case, the integral shown in Equation 5 can be further approximated numerically by drawing from the Gaussian distribution ^^^|^,^^. If ^ quasi-random draws are taken, the approximation can be given by Equation 7. Equation 7

[0068] It is possible to re-express each draw ^^as ^^ൌ ^௭^ ^^where ^௭is the mean of ^^^|^,^^ and ^^is a quasi-random draw from a Gaussian distribution with mean zero and the same covariance as ^^^|^,^^. This representation yields Equation 8. Equation 8US2008302308241Attorney Docket No.096923-1449567

[0069] To achieve monotonicity in ^, in this case, the covariance of ^^^|^,^^ is required to be fixed for fixed ^. Then, it is possible to fix the draws ^^and each term ^^^ ൌ 1|^ ൌ ^௭^ ^^^ will be monotonic in ^. Therefore the whole approximation of ^^^ ൌ 1|^,^^ will be monotonic.

[0070] In the case that a normalizing flow is used to produce a non-Gaussian encoder, itis possible to sample from ^^^ᇱ|^,^^. reparametrizing around the mean ^௭ᇲand ^^^ᇱ|^,^^must have fixed covariance for fixed ^, to achieve monotonicity.

[0071] The second option for approximating ^^^ ൌ 1|^^ is to use a Taylor series. For example, a second order Taylor series can be used to approximate Equation 9. Equation 9Equation 9 can be generated by expanding around the mean ^௭of ^^^|^,^^. Using an established result of the mean of a quadratic form of a Gaussian random variable yields Equation 10. Equation 10 ^^1^^ ൌ 1|^,^^ ^^^ ൌ 1|^ ൌ ^௭^^2^^^^ଶ^^^ ൌ 1|^^^௭^where ^ଶ^^^ ൌ 1|^^ is the Hessian matrix of second derivatives of ^^^ ൌ 1|^^ with respect to ^, evaluated at ^ ൌ ^௭, and ^௭is the covariance of ^^^|^,^^.

[0072] This approach aims to achieve a better approximation than simply evaluating at the mean of ^^^|^,^^. However, noting that Equation 5 that is being approximated may be made monotonic as described above, the Taylor series approximation may fail to be monotonic as it depends on the Hessian ^ଶ^^^ ൌ 1|^^that is unconstrained.

[0073] Therefore, while the Taylor series approach may yield slightly better predictions, additional approaches may be used to generate model explanations. In a well-performing model where the covariance of ^^^|^,^^ is small, all three approximations will produce very similar computations of ^^^ ൌ 1|^,^^. Use of Alternative Predictors

[0074] The role of the scorer module 306 in fitting the hybrid model is to force the variational autoencoder (VAE) - that is, the part of the model formed by the encoder 304 and decoder 302 modules - to learn a latent representation of the independent variables ^ that includes the information necessary for prediction of the outcome ^. Once this is done, and the US2008302308241Attorney Docket No.096923-1449567 model is trained, the same scorer module 306 need not be used in application. In particular, to fit the hybrid model using gradient descent, a differentiable scorer such as a neural network is needed; but better predictions may be achieved by using the learned latent variable encoding as input to a (monotonic) GBM. The following approach can be used. Initially, a constrained non-linear hybrid model as described above, may be trained and validated using a neural network scorer module with the necessary monotonicity constraints according to what approximation method will be used in application. Further, the encoder 304 from the trained hybrid model may be used to compute conditional latent variable distributions ^^^|^,^^ for a new training sample (or the same training sample may be re-used). Additionally, one or more values from the conditional distribution ^^^|^,^^may be generated for each observation in the new training sample, to act as inputs to the new predictive model. The mean of ^^^|^,^^ may be used, or multiple values may be drawn from ^^^|^,^^ to produce multiple observations of ^ for each input observation of ^ and ^. Furthermore, a monotonic predictive model may be trained with ^ as input and ^ as outcome, using the data generated in the previous step. Then, in application, the new predictive model may be substituted for theoriginal scorer module 306 to compute ^^^|^^.

[0075] In this approach, the power of the non-linear hybrid model is used to incorporate information from the available data sources into the latent variable ^, while no limitation beyond monotonicity is placed on the predictive model used to predict ^ from ^. Example of Generating a Risk Indicator Using a Risk Assessment Model

[0076] FIG. 4 is a flow chart depicting an example of a process 400 for using a risk assessment model 120 to generate a risk indicator for a target entity. For example, the risk assessment model 120 can generate a risk indicator for a target entity based on predictor variables 124 associated with the target entity. One or more computing devices (e.g., the risk assessment server 118) implement operations depicted in FIG. 4 by executing suitable program code (e.g., the risk assessment application 114). For illustrative purposes, the process 400 is described with reference to certain examples depicted in the figures. Other implementations, however, are possible.

[0077] At block 402, the process 400 involves generating a machine-learning model by training the machine-learning model based on a multi-dimensional a multi-dimensional representation of common information from the set of data sources, such as a multi- dimensional latent variable representing information from the set of data sources. For example, as described above, a hybrid model can model common information from the data US2008302308241Attorney Docket No.096923-1449567 sources 121 as a latent variable (e.g., latent variable ^). The hybrid model can include a scorer module (the scorer 306), a decoder module (the decoder 302), and an encoder module (the encoder 304). In some examples, these neural network modules are fit to capture the relationships between ^, the observed variables ^^, and the outcome variable ^. In some examples, the neural network modules can take into account a mask ^ indicating which of the data sources (e.g., risk data repository 122 and data sources 121) are present.

[0078] At block 404, the process 400 involves accessing a risk assessment query associated with a target entity. In some cases, the risk assessment computing system 130 (e.g., the risk assessment server 118) may receive a risk assessment query associated with a particular entity (e.g., the target entity). The risk assessment query may be received from the target entity requesting the risk assessment. In additional or alternative implementations, the risk assessment query may be received from a remote computing device associated with an entity authorized to request risk assessment of the target entity.

[0079] At block 406, the process 400 involves accessing data from a set of data sources and accessing the machine-learning model. The data can be entity data stored in the risk data repository 122 or data sources 121. For example, data stored in data sources 121 can be supplementary or alternative data. In some examples, certain observations may be missing in the data either from the data repository 122 or the data sources 121. This data can be considered, in some cases, as missing not at random, as the presence or absence of a particular data source cannot be assumed to be independent of the content of that data.

[0080] At block 408, the process 400 involves determining, using the machine-learning model, a risk indicator for a target entity from predictor variables associated with the target entity. Once the risk assessment server 118 receives the risk assessment query, the risk assessment server 118 can execute the trained risk assessment model 120 to determine the risk indicator for the target entity.

[0081] At block 410, the process 400 includes transmitting, to a remote computing device, a responsive message comprising at least the risk indicator for use in controlling access of the target entity to one or more computing environments. For example, the risk assessment server 118 can perform a risk assessment based on predictor variables 124 generated for the customer and return a responsive message to the client computing system 104. The responsive message can include at least the risk indicator and explanatory data associated with the risk indicator. The explanatory data can indicate relationships between US2008302308241Attorney Docket No.096923-1449567 changes in the risk indicator and changes in at least some of the predictor variables 124 associated with the target entity.

[0082] The risk indicator can correspond to a level of risk associated with the target entity, for example with respect to accessing protecting computing resources. The risk indicator can be used for one or more operations that involve performing an operation with respect to the target entity based on a predicted risk associated with the target entity. In one example, the risk indicator can be utilized to control access to one or more interactive computing environments by the target entity. The risk assessment computing system 130 can communicate with client computing systems 104, which may send risk assessment queries to the risk assessment server 118 to request risk assessment. The client computing systems 104 may be associated with technological providers, such as cloud computing providers, online storage providers, or financial institutions such as banks, credit unions, credit-card companies, insurance companies, or other types of organizations. The client computing systems 104 may be implemented to provide interactive computing environments for customers to access various services offered by these service providers. Customers can use user computing systems 106 to access the interactive computing environments thereby accessing the services provided by these providers.

[0083] Based on the received risk indicator, the client computing system 104 can determine whether to grant the customer access to the interactive computing environment. If the client computing system 104 determines that the level of risk associated with the customer accessing the interactive computing environment and the associated technical or financial service is too high, the client computing system 104 can deny access by the customer to the interactive computing environment. Conversely, if the client computing system 104 determines that the level of risk associated with the customer is acceptable, e.g., is below a predetermined threshold, the client computing system 104 can grant access to the interactive computing environment by the customer and the customer would be able to utilize the various services provided by the service providers. For example, with the granted access, the customer can utilize the user computing system 106 to access clouding computing resources, online storage resources, web pages or other user interfaces provided by the client computing system 104 to execute applications, store data, query data, submit an online digital application, operate electronic tools, or perform various other operations within the interactive computing environment hosted by the client computing system 104. US2008302308241Attorney Docket No.096923-1449567 Example of Computing System for Machine-Learning Operations

[0084] Any suitable computing system or group of computing systems can be used to perform the operations for the machine-learning operations described herein. For example, FIG. 5 is a block diagram depicting an example of a computing device 500, which can be used to implement the risk assessment server 118 or the model training server 110. The computing device 500 can include various devices for communicating with other devices in the operating environment 100, as described with respect to FIG. 1. The computing device 800 can include various devices for performing one or more transformation operations described above with respect to FIGS.1-4.

[0085] The computing device 500 can include a processor 502 that is communicatively coupled to a memory 504. The processor 502 executes computer-executable program code stored in the memory 504, accesses information stored in the memory 504, or both. Program code may include machine-executable instructions that may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, among others.

[0086] Examples of a processor 502 include a microprocessor, an application-specific integrated circuit, a field-programmable gate array, or any other suitable processing device. The processor 502 can include any number of processing devices, including one. The processor 502 can include or communicate with a memory 504. The memory 504 stores program code that, when executed by the processor 502, causes the processor to perform the operations described in this disclosure.

[0087] The memory 504 can include any suitable non-transitory computer-readable storage medium. The computer-readable medium can include any electronic, optical, magnetic, or other storage device capable of providing a processor with computer-readable program code or other program code. Non-limiting examples of a computer-readable medium include a magnetic disk, memory chip, optical storage, flash memory, storage class memory, ROM, RAM, an ASIC, magnetic storage, or any other medium from which a computer processor can read and execute program code. The program code may include processor- US2008302308241Attorney Docket No.096923-1449567 specific program code generated by a compiler or an interpreter from code written in any suitable computer-programming language. Examples of suitable programming language include Hadoop, C, C++, C#, Visual Basic, Java, Python, Perl, JavaScript, ActionScript, etc.

[0088] The computing device 500 may also include a number of external or internal devices such as input or output devices. For example, the computing device 500 is shown with an input / output interface 508 that can receive input from input devices or provide output to output devices. A bus 506 can also be included in the computing device 500. The bus 506 can communicatively couple one or more components of the computing device 500.

[0089] The computing device 500 can execute program code 514 that includes the risk assessment application 114 and / or the model training application 112. The program code 514 for the risk assessment application 114 and / or the model training application 112 may be resident in any suitable computer-readable medium and may be executed on any suitable processing device. For example, as depicted in FIG. 5, the program code 514 for the risk assessment application 114 and / or the model training application 112 can reside in the memory 504 at the computing device 500 along with the program data 516 associated with the program code 514, such as the predictor variables 124 and / or the model training samples. Executing the risk assessment application 114 or the model training application 112 can configure the processor 502 to perform the operations described herein.

[0090] In some aspects, the computing device 500 can include one or more output devices. One example of an output device is the network interface device 510 depicted in FIG. 5. A network interface device 510 can include any device or group of devices suitable for establishing a wired or wireless data connection to one or more data networks described herein. Non-limiting examples of the network interface device 510 include an Ethernet network adapter, a modem, etc.

[0091] Another example of an output device is the presentation device 512 depicted in FIG. 5. A presentation device 512 can include any device or group of devices suitable for providing visual, auditory, or other suitable sensory output. Non-limiting examples of the presentation device 512 include a touchscreen, a monitor, a speaker, a separate mobile computing device, etc. In some aspects, the presentation device 512 can include a remote client-computing device that communicates with the computing device 500 using one or more data networks described herein. In other aspects, the presentation device 512 can be omitted.

[0092] The foregoing description of some examples has been presented only for the purpose of illustration and description and is not intended to be exhaustive or to limit the US2008302308241Attorney Docket No.096923-1449567 disclosure to the precise forms disclosed. Numerous modifications and adaptations thereof will be apparent to those skilled in the art without departing from the spirit and scope of the disclosure. US2008302308241

Claims

Attorney Docket No.096923-1449567 Claims What is claimed is:

1. A computer-implemented method, comprising: accessing, by a processor, a risk assessment query including an indication of a target entity; accessing, by the processor, a machine-learning model that is trained to determine a risk indicator for a target entity from predictor variables associated with the target entity, wherein the predictor variables are obtained from multiple sources with varying availability, and wherein the training of the machine- learning model is accomplished based on a multi-dimensional representation of common information from the set of data sources; generating, by the processor, the risk indicator for the target entity using the trained machine-learning model and the predictor variables; and transmitting, by the processor and to a remote computing device, a responsive message comprising at least the risk indicator for use in controlling access of the target entity to one or more computing environments.

2. The method of claim 1, wherein the machine-learning model is a hybrid model comprising a scorer, an encoder, and a decoder.

3. The method of claim 2, wherein the machine-learning model comprises a mask, and wherein the mask indicates which data sources of the set of data sources are present.

4. The method of claim 2, wherein the encoder is a monotonically constrained partial inference net encoder.

5. The method of claim 2, wherein the scorer is a monotonic scorer.

6. The method of claim 2, wherein the decoder is an unconstrained decoder.

7. The method of claim 1, wherein the multi-dimensional representation of the common information from the set of data sources comprises a multi-dimensional latent variable US2008302308241Attorney Docket No.096923-1449567 modeled by a standard multivariate normal distribution in a population represented in the set of data sources.

8. A system comprising: a processing device; and a memory device in which instructions executable by the processing device are stored for causing the processing device to perform operations comprising: accessing a risk assessment query including an indication of a target entity; accessing a machine-learning model that is trained to determine a risk indicator for a target entity from predictor variables associated with the target entity, wherein the predictor variables are obtained from multiple sources with varying availability, and wherein the training of the machine-learning is accomplished based on a multi-dimensional representation of common information from the set of data sources; generating the risk indicator for the target entity using the trained machine-learning model and the predictor variables; and transmitting, to a remote computing device, a responsive message comprising at least the risk indicator for use in controlling access of the target entity to one or more computing environments.

9. The system of claim 8, wherein the machine-learning model is a hybrid model comprising a scorer, an encoder, and a decoder.

10. The system of claim 9, wherein the machine-learning model comprises a mask, wherein the mask indicates which data sources of the set of data sources are present.

11. The system of claim 9, wherein the encoder is a monotonically constrained partial inference net encoder.

12. The system of claim 9, wherein the scorer is a monotonic scorer.

13. The system of claim 9, wherein the decoder is an unconstrained decoder. US2008302308241Attorney Docket No.096923-1449567 14. The system of claim 8, wherein the multi-dimensional representation of the common information from the set of data sources comprises a multi-dimensional latent variable modeled by a standard multivariate normal distribution in a population represented in the set of data sources.

15. A non-transitory computer-readable storage medium having program code that is executable by a processor to cause a computing device to perform operations, the operations comprising: accessing a risk assessment query including an indication of a target entity; accessing a machine-learning model that is trained to determine a risk indicator for a target entity from predictor variables associated with the target entity, wherein the predictor variables are obtained from multiple sources with varying availability, and wherein the training of the machine-learning is accomplished based on a multi-dimensional representation of common information from the set of data sources; generating the risk indicator for the target entity using the trained machine-learning model and the predictor variables; and transmitting, to a remote computing device, a responsive message comprising at least the risk indicator for use in controlling access of the target entity to one or more computing environments.

16. The non-transitory computer-readable storage medium of claim 15, wherein the machine-learning model is a hybrid model comprising a scorer, an encoder, and a decoder.

17. The non-transitory computer-readable storage medium of claim 16, wherein the machine-learning model comprises a mask, wherein the mask indicates which data sources of the set of data sources are present.

18. The non-transitory computer-readable storage medium of claim 16, wherein the encoder is a monotonically constrained partial inference net encoder. US2008302308241Attorney Docket No.096923-1449567 19. The non-transitory computer-readable storage medium of claim 16, wherein the scorer is a monotonic scorer.

20. The non-transitory computer-readable storage medium of claim 16, wherein the decoder is an unconstrained decoder. US2008302308241

Images