Machine-learning techniques of using tabularized data to predict missing values

Abstract

In some aspects, a computing system can train a machine learning (ML) model for risk assessment using risk assessment training data generated at least in part by a prediction model. Once trained, the ML model can determine a risk indicator for a target entity that indicates a level of risk associated with the target entity. Training the ML model can involve receiving a request to predict a value of an unknown segment of a transaction record used to train the ML model. The computing system can execute the prediction model to predict the value of the unknown segment prior to training the ML model using the transaction record that includes the predicted value generated by the prediction model. Additionally, the computing system can generate explanatory data for the target entity indicating relationships between changes in the risk indicator and changes in the transaction records associated with the target entity.

Description

Cross-References to Related Application

[0001] This application claims the benefit of U.S. Provisional Patent Application no. 63 / 619,163, filed January 9, 2024, the full disclosure of which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0002] The present disclosure relates generally to artificial intelligence for use in controlling access to a secure resource. More specifically, but not by way of limitation, this disclosure relates to machine learning using language models that are trained using tabularized data associated with transaction records for predicting a missing value of training data used to train a risk assessment model. Background

[0003] In machine learning, models can be used to perform one or more functions (e.g., acquiring, processing, analyzing, and understanding various inputs in order to produce an output that includes numerical or symbolic information). A machine learning model, such as a neural network, includes one or more algorithms and interconnected components that exchange data between one another. The model can have numeric parameters that can be tuned based on experience, which makes the model adaptive and capable of learning. For example, the numeric weights of a neural network can be used to train the neural network such that the neural network can perform the one or more functions on a set of input variables and produce an output that is associated with the set of input variables.

[0004] In language processing, general language models are typically pre-trained using natural language data that includes natural language context used by the general language model to interpret meaning or structure of the natural language data. This natural language context can be absent from certain data formats, such as in tabularized data. As a result, a traditional general language model may be incompatible with tabularized data that provides little to no natural language context, creating difficulty and inaccuracy in predicting missing values in tabularized data. Summary

[0005] Various aspects of the present disclosure provide systems and methods for using a language model trained on tabularized data to predict a missing value of training data used to train a risk assessment model. In one example, a computer-implemented method is performed by one or more processing devices. The method can include receiving, at a processor, a request for a risk indicator for a target entity determined from predictor variables associated with the target entity. The risk indicator can indicate a level of risk associated with the target entity. The method can include training, by the processor, a risk assessment model for determining the risk indicator from the predictor variables using a training process. The training process can include receiving a request to predict a value of an unknown segment of risk assessment training data associated with an interaction performed by the target entity. The risk assessment training data can be used to train the risk assessment model to output the risk indicator. The training process can include predicting the value of the unknown segment by executing a prediction mode. The prediction model may be trained using one or more segments of at least one unstructured transaction record. The training process can include, subsequent to predicting the value of the unknown segment, training the risk assessment model using the risk assessment training data that includes the predicted value generated by the prediction model. The method additionally can include generating, by the processor and for the target entity, the risk indicator for the target entity using the trained risk assessment model. The method further can include transmitting, from the processor to a remote computing device, a responsive message including at least the risk indicator for use in controlling access of the target entity to one or more interactive computing environments.

[0006] In another example, a system includes a processing device; and a memory device in which instructions executable by the processing device are stored for causing the processing device to perform various operations. The system can receive a request for a risk indicator for a target entity determined from predictor variables associated with the target entity. The risk indicator can indicate a level of risk associated with the target entity. The system can train a risk assessment model for determining the risk indicator from the 2 predictor variables using a training process. The training process can include receiving a request to predict a value of an unknown segment of risk assessment training data associated with an interaction performed by the target entity. The risk assessment training data can be used to train the risk assessment model to output the risk indicator. The training process can include predicting the value of the unknown segment by executing a prediction mode. The prediction model may be trained using one or more segments of at least one unstructured transaction record. The training process can include, subsequent to predicting the value of the unknown segment, training the risk assessment model using the risk assessment training data that includes the predicted value generated by the prediction model. The system additionally can generate, for the target entity, the risk indicator for the target entity using the trained risk assessment model.

[0007] In yet another example, a non-transitory computer-readable storage medium has program code that is executable by a processor to cause a computing device to perform operations. The operations can include receiving a request for a risk indicator for a target entity determined from predictor variables associated with the target entity. The risk indicator can indicate a level of risk associated with the target entity. The operations can include training a risk assessment model for determining the risk indicator from the predictor variables using a training process. The training process can include receiving a request to predict a value of an unknown segment of risk assessment training data associated with an interaction performed by the target entity. The risk assessment training data can be used to train the risk assessment model to output the risk indicator. The training process can include predicting the value of the unknown segment by executing a prediction mode. The prediction model may be trained using one or more segments of at least one unstructured transaction record. The training process can include, subsequent to predicting the value of the unknown segment, training the risk assessment model using the risk assessment training data that includes the predicted value generated by the prediction model. The operations additionally can include generating, for the target entity, the risk indicator for the target entity using the trained risk assessment model.

[0008] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification, any or all drawings, and each claim.

[0009] The foregoing, together with other features and examples, will become more apparent upon referring to the following specification, claims, and accompanying drawings. Brief Description of the Drawings

[0010] FIG. 1 is a block diagram depicting an example of an operating environment according to certain aspects of the present disclosure.

[0011] FIG. 2 is a flow chart depicting an example of a process for generating missing values in training data according to certain aspects of the present disclosure.

[0012] FIG. 3 is a diagram depicting an example of a graphical representation of a transformer-based masked language model, according to certain aspects of the present disclosure.

[0013] FIG. 4 is a block diagram depicting an example of a process predicting masked objective tokens using embeddings generated by a masked language model according to certain aspects of the present disclosure.

[0014] FIG. 5 is a block diagram depicting an example of a computing device suitable for implementing aspects of the techniques and technologies presented herein. Detailed Description

[0015] Disclosed systems and methods can be used to predict missing values in tabularized data, which can then be used in training a machine learning model to predict risk. For example, in certain aspects, disclosed systems and methods can be used to control an entity’s access to a secure resource based on a predicted risk determined for the entity using a machine learning model trained using tabularized training data. In some instances, the training data may be missing certain values that can be predicted using a prediction model trained on tabularized data.

[0016] General language models are typically pre-trained using natural language data that includes natural language context used by the general language model to interpret meaning or structure of the natural language data. As a result, the general language model may be incompatible with tabularized data that provides little to no natural language context, unlike natural language data. Disclosed systems and methods describe a class of prediction models that can be used to predict missing values in tabularized data. For example, the class of prediction models (e.g., masked language models) can enable prediction of an uncertain variable, such as a risk assessment outcome to train a risk assessment model, is described herein. The prediction models describe the relationship between the uncertain variable and tabularized data provided as input to the prediction models. Training the prediction models can involve providing an incomplete sequence to the prediction models such that the prediction models can fill in the incomplete sequence and output a complete sequence or a missing value of the incomplete sequence. In some examples, the prediction models are built using a specific corpus generated using tabularized data, instead of finetuning existing language models trained on general corpora. Thus, the prediction models can be trained using terminology specific to the tabularized data to consider word co-occurrences tailored to the tabularized data, resulting in different vector representations that can be more accurate than those of language models trained on general corpora. Additionally, the prediction models can interpret or make use of terminology provided in the tabularized data that may typically be considered out-ofvocabulary terms that are outside of typical lexicon used in the general corpora.

[0017] Certain aspects described herein use tabularized data to train the prediction models or as an input to the prediction models. The predictive output of the prediction models can then be tailored to a specific domain, such as being used to train a risk assessment model. Accordingly, the tabularized data can be used to predict a missing value or variable of training data used to train the risk assessment model. In contrast to general corpora (e.g., English Wikipedia or Brown Corpus) providing natural language data with inherent natural language context, the tabularized data used to train the prediction models can be provided in a table with textual fields with each data row representing an individual sequence. In other words, the prediction models can be trained and used for prediction even if input data is unstructured or lacking natural language context. Additionally, the tabularized data may include limited and specific terminologies corresponding to the specific domain such that the prediction models are more lightweight than language models incorporating the general corpora. Due to the tailored use case of the prediction models, the masked language models can provide more accurate predictions compared to language models trained using general corpora while consuming fewer computing resources. This overcomes the problem of maintaining resource-intensive language models that may provide inaccurate predictions due to a different representation distribution resulting from training using general corpora.

[0018] Natural language processing techniques are typically applied to general corpora to generate language models (e.g., large language models) that can interpret meaning and structure of sentences. The general corpora include raw text data that includes natural language context used to determine patterns of the raw text data. The patterns then can be used to predict one or more words in a sentence, generate original text, or other downstream tasks. But the patterns of the general corpora can differ significantly from the tabularized data that lack natural language context, resulting in poor predictive performance of the general corpora with respect to specific domains associated with the tabularized data. By adopting a masked language model formulation trained using the tabularized data, the technique proposed herein ensures the predictive performance of the model is tailored to a specific domain associated with the tabularized data. Due to the specific use case of the prediction model, the prediction model can avoid using an extensive dictionary required for general-domain language models to interpret general terminology in natural language. A training process of the prediction model can involve providing a training dataset with one or more masked input tokens at random such that the model can predict a respective value for the masked input tokens. The predicted values then can be used to calculate gradient changes and optimize model weights for additional predictions.

[0019] Certain aspects described herein provide improvements to machine learning techniques for assessing risks, for example, in access control associated with entities. For instance, the prediction model presented herein can be built based on tabularized data to predict missing training data values used to train a risk assessment model. Training language models using text corpora can be expensive with respect to time and computational resources. The text corpora may be more storage-intensive than the tabularized data to produce language models applicable to prediction tasks associated with various domains. Using the tabularized data to train the prediction model instead of the text corpora can lead to significant reduction in the computational resource consumption associated with training and maintaining the prediction model, including the CPU consumptions, memory usage, and so on. Implementing the prediction model additionally can improve prediction results by using a specific corpus of tabularized data tailored to a specific domain rather than general corpora that may include input data that is not relevant to the specific domain. Accordingly, using the prediction model formulation trained using the tabularized data increases the predictive performance of the model compared to conventional language models trained using text corpora.

[0020] Additional or alternative aspects can implement or apply rules of a particular type that improve existing technological processes involving machine-learning techniques. For instance, based on the tabularized data, a particular set of vector representations is developed by training the model. This particular set of vector representations enables the prediction model to consider specific terms relevant to the tabularized data in relation with other criteria instead of discounting the specific terms as out-of-vocabulary words, which allows the training of the model to be performed more efficiently with little to no posttraining adjustment.

[0021] These illustrative examples are given to introduce the reader to the general subject matter discussed here and are not intended to limit the scope of the disclosed concepts. The following sections describe various additional features and examples with reference to the drawings in which like numerals indicate like elements, and directional descriptions are used to describe the illustrative examples but, like the illustrative examples, should not be used to limit the present disclosure. Operating Environment Example for Machine-Learning Operations

[0022] Referring now to the drawings, FIG. 1 is a block diagram depicting an example of an operating environment 100 in which a risk assessment computing system 130 builds and trains a risk assessment model 120 that can be trained to predict risk indicators based on training data provided at least in part by a prediction model 121. FIG. 1 depicts examples of hardware components of a risk assessment computing system 130, according to some aspects. The risk assessment computing system 130 is a specialized computing system that may be used for processing large amounts of data using a large number of computer processing cycles. The risk assessment computing system 130 can include a model training server 110 for building and training a risk assessment model 120 used to predict risk indicators associated with an entity accessing controlled resources. In some embodiments, input predictor variables 124 of the risk assessment model 120 or factors of the input predictor variables 124 have a monotonic relationship with the output of the risk assessment model 120. The risk assessment computing system 130 can further include a risk assessment server 118 for performing a risk assessment for given predictor variables 124 using the trained risk assessment model 120.

[0023] The model training server 110 can include one or more processing devices that execute program code, such as a model training application 112. The program code is stored on a non-transitory computer-readable medium. The model training application 112 can execute one or more processes to train and optimize a risk assessment model 120 for predicting risk indicators based on the predictor variables 124. In some instances, the risk assessment model 120 may predict the risk indicators while maintaining a monotonic relationship between the factors of the predictor variables 124 and the predicted risk indicators.

[0024] In some aspects, the model training application 112 can build and train a risk assessment model 120 using risk assessment training data 126 in a training process. The risk assessment training data 126 can include multiple training vectors including training predictor variables and training risk indicator outputs corresponding to the training vectors. In some cases, the risk assessment training data 126 may include differing subsets of data sources available. The risk assessment training data 126 can be stored in one or more network-attached storage units on which various repositories, databases, or other structures are stored. An example of these data structures is the risk data repository 122.

[0025] The risk assessment training data 126 may be at least in part generated using tabularized data. The tabularized data can include data corresponding to one or more interactions of at least one entity with a computing environment. As an example, the interactions may be generated by the entity using a user computing system 106 to interact with an interactive computing environment provided by a client computing system 104. In some cases, the tabularized data may be unstructured such that the tabularized data include little to no natural language context with respect to meaning or structure of the data provided in the tabularized data. For example, the tabularized data may be stored in a table having one or more columns that each correspond to a particular data category (e.g., location, entity characteristics, account information, etc.) In some implementations, the model training application 112 can execute one or more processes to use the prediction model 121 to predict missing values of the risk assessment training data 126. In additional or alternative aspects, the model training application 112 can build and train the prediction model 121 to predict the missing values. Similarly, the prediction model 121 may be trained using prediction training data 128 generated at least in part using tabularized data (e.g., transaction records 129).

[0026] Network-attached storage units may store a variety of different types of data organized in a variety of different ways and from a variety of different sources. For example, the network-attached storage unit may include storage other than primary storage located within the model training server 110 that is directly accessible by processors located therein. In some aspects, the network-attached storage unit may include secondary, tertiary, or auxiliary storage, such as large hard drives, servers, virtual memory, among other types. Storage devices may include portable or non-portable storage devices, optical storage devices, and various other mediums capable of storing and containing data. A machine-readable storage medium or computer-readable storage medium may include a non-transitory medium in which data can be stored and that does not include carrier waves or transitory electronic signals. Examples of a non-transitory medium may include, for example, a magnetic disk or tape, optical storage media such as a compact disk or digital versatile disk, flash memory, memory, or memory devices.

[0027] The risk assessment server 118 can include one or more processing devices that execute program code, such as a risk assessment application 114. The program code is stored on a non-transitory computer-readable medium. The risk assessment application 114 can execute one or more processes to utilize the risk assessment model 120 trained by the model training application 112 to predict risk indicators based on input predictor variables 124. The risk indicators can be used to protect or allocate computing resources of the risk assessment computing system 130.

[0028] Furthermore, the risk assessment computing system 130 can communicate with various other computing systems, such as client computing systems 104. For example, client computing systems 104 may send risk assessment queries to the risk assessment server 118 for risk assessment or may send signals to the risk assessment server 118 that control or otherwise influence different aspects of the risk assessment computing system 130. The client computing systems 104 may also interact with user computing systems 106 via one or more public data networks 108 to facilitate interactions between users of the user computing systems 106 and interactive computing environments provided by the client computing systems 104.

[0029] Each client computing system 104 may include one or more third-party devices, such as individual servers or groups of servers operating in a distributed manner. A client computing system 104 can include any computing device or group of computing devices operated by a seller, lender, or other providers of products or services. The client computing system 104 can include one or more server devices. The one or more server devices can include or can otherwise access one or more non-transitory computer-readable media. The client computing system 104 can also execute instructions that provide an interactive computing environment accessible to user computing systems 106. Examples of the interactive computing environment include a mobile application specific to a particular client computing system 104, a web-based application accessible via a mobile device, etc. The executable instructions are stored in one or more non-transitory computer-readable media.

[0030] The client computing system 104 can further include one or more processing devices that are capable of providing the interactive computing environment to perform operations described herein. The interactive computing environment can include executable instructions stored in one or more non-transitory computer-readable media. The instructions providing the interactive computing environment can configure one or more processing devices to perform operations described herein. In some aspects, the executable instructions for the interactive computing environment can include instructions that provide one or more graphical interfaces. The graphical interfaces are used by a user computing system 106 to access various functions of the interactive computing environment. For instance, the interactive computing environment may transmit data to and receive data from a user computing system 106 to shift between different states of the interactive computing environment, where the different states allow one or more electronics transactions between the user computing system 106 and the client computing system 104 to be performed.

[0031] In some examples, a client computing system 104 may have other computing resources associated therewith (not shown in FIG. 1), such as server computers hosting and managing virtual machine instances for providing cloud computing services, server computers hosting and managing online storage resources for users, server computers for providing database services, and others. The interaction between the user computing system 106 and the client computing system 104 may be performed through graphical user interfaces presented by the client computing system 104 to the user computing system 106, or through application programming interface (API) calls or web service calls.

[0032] A user computing system 106 can include any computing device or other communication device operated by an entity, such as a user, an organization, or a company. The user computing system 106 can include one or more computing devices, such as laptops, smartphones, and other personal computing devices. A user computing system 106 can include executable instructions stored in one or more non-transitory computer-readable media. The user computing system 106 can also include one or more processing devices that are capable of executing program code to perform operations described herein. In various examples, the user computing system 106 can allow a user to access certain online services from a client computing system 104 or other computing resources, to engage in mobile commerce with a client computing system 104, to obtain controlled access to electronic content hosted by the client computing system 104, etc.

[0033] For instance, the user can use the user computing system 106 to engage in an electronic transaction with a client computing system 104 via an interactive computing environment. An electronic transaction between the user computing system 106 and the client computing system 104 can include, for example, the user computing system 106 being used to request online storage resources managed by the client computing system 104, acquire cloud computing resources (e.g., virtual machine instances), and so on. An electronic transaction between the user computing system 106 and the client computing system 104 can also include, for example, querying a set of sensitive or other controlled data, accessing online financial services provided via the interactive computing environment, submitting an online credit card application or other digital application to the client computing system 104 via the interactive computing environment, operating an electronic tool within an interactive computing environment hosted by the client computing system (e.g., a content-modification feature, an application-processing feature, etc.).

[0034] In some aspects, an interactive computing environment implemented through a client computing system 104 can be used to provide access to various online functions. As a simplified example, a website or other interactive computing environment provided by an online resource provider can include electronic functions for requesting computing resources, online storage resources, network resources, database resources, or other types of resources. In another example, a website or other interactive computing environment provided by a financial institution can include electronic functions for obtaining one or more financial services, such as loan application and management tools, credit card application and transaction management workflows, electronic fund transfers, etc. A user computing system 106 can be used to request access to the interactive computing environment provided by the client computing system 104, which can selectively grant or deny access to various electronic functions. Based on the request, the client computing system 104 can collect data associated with the user and communicate with the risk assessment server 118 for risk assessment. Based on the risk indicator predicted by the risk assessment server 118, the client computing system 104 can determine whether to grant the access request of the user computing system 106 to certain features of the interactive computing environment.

[0035] In a simplified example, the system depicted in FIG. 1 can configure a prediction model 121 to be used for accurately predicting missing values of risk assessment training data 126 used to train a risk assessment model 120 to determine risk indicators, such as credit scores, using predictor variables 124. In additional or alternative examples, the risk assessment model 120 may determine adverse action codes or other explanation codes for the predictor variables 124. A predictor variable 124 can be any variable predictive of risk that is associated with an entity. Any suitable predictor variable that is authorized for use by an appropriate legal or regulatory framework may be used.

[0036] Examples of predictor variables 124 used for predicting the risk associated with an entity accessing online resources include, but are not limited to, variables indicating the demographic characteristics of the entity (e.g., name of the entity, the network or physical address of the company, the identification of the company, the revenue of the company), variables indicative of prior actions or transactions involving the entity (e.g., past requests of online resources submitted by the entity, the amount of online resource currently held by the entity, and so on.), variables indicative of one or more behavioral traits of an entity (e.g., the timeliness of the entity releasing the online resources), etc. Similarly, examples of predictor variables 124 used for predicting the risk associated with an entity accessing services provided by a financial institute include, but are not limited to, indicative of one or more demographic characteristics of an entity (e.g., age, gender, income, etc.), variables indicative of prior actions or transactions involving the entity (e.g., information that can be obtained from credit files or records, financial records, consumer records, or other data about the activities or characteristics of the entity), variables indicative of one or more behavioral traits of an entity, etc.

[0037] The predicted risk indicator can be utilized by the service provider to determine the risk associated with the entity accessing a service provided by the service provider, thereby granting or denying access by the entity to an interactive computing environment implementing the service. For example, if the service provider determines that the predicted risk indicator is lower than a threshold risk indicator value, then the client computing system 104 associated with the service provider can generate or otherwise provide access permission to the user computing system 106 that requested the access. The access permission can include, for example, cryptographic keys used to generate valid access credentials or decryption keys used to decrypt access credentials. The client computing system 104 associated with the service provider can also allocate resources to the user and provide a dedicated web address for the allocated resources to the user computing system 106, for example, by adding it in the access permission. With the obtained access credentials and / or the dedicated web address, the user computing system 106 can establish a secure network connection to the computing environment hosted by the client computing system 104 and access the resources via invoking API calls, web service calls, HTTP requests, or other proper mechanisms.

[0038] Each communication within the operating environment 100 may occur over one or more data networks, such as a public data network 108, a network 116 such as a private data network, or some combination thereof. A data network may include one or more of a variety of different types of networks, including a wireless network, a wired network, or a combination of a wired and wireless network. Examples of suitable networks include the Internet, a personal area network, a local area network (“LAN”), a wide area network (“WAN”), or a wireless local area network (“WLAN”). A wireless network may include a wireless interface or a combination of wireless interfaces. A wired network may include a wired interface. The wired or wireless networks may be implemented using routers, access points, bridges, gateways, or the like, to connect devices in the data network.

[0039] The number of devices depicted in FIG. 1 is provided for illustrative purposes. Different numbers of devices may be used. For example, while certain devices or systems are shown as single devices in FIG. 1, multiple devices may instead be used to implement these devices or systems. Similarly, devices or systems that are shown as separate, such as the model training server 110 and the risk assessment server 118, may be instead implemented in a single device or system. Examples of Operations Involving Machine-Learnins

[0040] FIG. 2 is a flow chart depicting an example of a process 200 for utilizing a prediction model 121 to generate missing values for risk assessment training data 126 used to train a risk assessment model 120. Once trained, the risk assessment model 120 can generate risk indicators for a target entity based on predictor variables 124 associated with the target entity. One or more computing devices (e.g., the risk assessment server 118) implement operations depicted in FIG. 2 by executing suitable program code (e.g., the risk assessment application 114). For illustrative purposes, the process 200 is described with reference to certain examples depicted in the figures. Other implementations, however, are possible.

[0041] At block 202, the process 200 involves receiving a request to predict a value of an unknown segment of the risk assessment training data 126 associated with an interaction performed by the target entity. In some aspects, the request may include data that identifies or can be used to identify the unknown segment of the risk assessment training data 126 stored in the risk data repository 122. In some instances, the risk assessment training data 126 can be stored in a risk data repository 122 and can be used to train the risk assessment model 120 to output the risk indicator. The model training server 110 may receive the request and thereby perform one or more operations to predict the value of the unknown segment such that the unknown segment can be used to train the risk assessment model 120. For example, the model training server 110 may execute a model training application 112 that can process tabularized data of at least one transaction record 129 to generate prediction training data 128 to train the prediction model 121. Once trained using the prediction training data 128, the prediction model 121 can be used by the model training server 110 to output the value of the unknown segment to complete the risk assessment training data 126. In some implementations, the model training server 110 may receive the request in response to a risk data repository 122 receiving incomplete risk assessment training data 126 that may include one or more missing values. Prior to using the incomplete risk assessment training data 126 to train the risk assessment model 120, the model training server 110 can execute the prediction model 121 to fill in the missing values of the incomplete risk assessment training data 126.

[0042] At block 204, the process 200 involves generating a training data set including one or more tokenized transaction records by predicting the value of the unknown segment using the prediction model 121. The prediction model 121 can be trained using one or more segments of the transaction record 129 that can be provided as part of the prediction training data 128. Training the prediction model 121 can involve inputting the segments of the transaction record 129 with masked sections such that the prediction model 121 is tasked with predicting values of the masked sections. As a result, through the training process, the prediction model 121 can develop one or more embeddings that can be used by the prediction model 121 to predict the value of the unknown segment. The embeddings can be generated as a respective vector representation corresponding to a particular data category of the transaction record 129. In some cases, the data categories of the transaction record 129 may be referred to as fields. As an example, the transaction record 129 may include a description data category and a service name category. Accordingly, after being trained using the transaction record 129, the prediction model 121 can generate a first embedding corresponding to the description data category and a second embedding corresponding to the service name category. In additional or alternative aspects, the prediction model 121 may concatenate the description category with the service name category prior to generating an embedding corresponding to the concatenated categories. In such aspects, the concatenated categories may provide more context compared to each category considered individually, thereby improving predictive performance of the prediction model 121.

[0043] At block 206, subsequent to generating a training data set by predicting the value of the unknown segment, the process 200 involves training the risk assessment model 120 using the risk assessment training data 126 that includes the predicted value generated by the prediction model 121. The risk assessment training data 126 can be used to train the risk assessment model 120 with respect to input predictor variables 124 and risk indicator outputs. For example, the risk assessment model 120 can access the risk assessment training data 126 that can include a plurality of sets of model training samples that can be used to map the input predictor variables 124 to risk indicator outputs. In some aspects, the sets of model training samples may include at least one particular training sample in which a value of at least one predictor variable 124 or risk indicator output is unknown, enabling the risk assessment model 120 to minimize a loss function with respect to the particular training sample.

[0044] Once trained, at block 208, the risk assessment model 120 can be used to determine the risk indicator for the target entity based on predictor variables 124 associated with the target entity. For example, the predictor variables 124 can be used as inputs to the risk assessment model 120. The predictor variables 124 associated with the target entity can be obtained from a predictor variable database configured to store predictor variables 124 associated with various entities. The output of the risk assessment model 120 can include the risk indicator for the target entity based on its current predictor variables.

[0045] The risk indicator can correspond to a level of risk associated with the target entity, for example with respect to accessing protecting computing resources. The risk indicator can be used for one or more operations that involve performing an operation with respect to the target entity based on a predicted risk associated with the target entity. In some cases, the risk assessment computing system 130 (e.g., the risk assessment server 118) may receive a risk assessment query associated with a particular entity (e.g., the target entity). The risk assessment query may be received from the target entity requesting the risk assessment. In additional or alternative implementations, the risk assessment query may be received from a remote computing device associated with an entity authorized to request risk assessment of the target entity. Once the risk assessment server 118 receives the risk assessment query, the risk assessment server 118 can execute the trained risk assessment model 120 to determine the risk indicator for the target entity.

[0046] In one example, the risk indicator can be utilized to control access to one or more interactive computing environments by the target entity. The risk assessment computing system 130 can communicate with client computing systems 104, which may send risk assessment queries to the risk assessment server 118 to request risk assessment. The client computing systems 104 may be associated with technological providers, such as cloud computing providers, online storage providers, or financial institutions such as banks, credit unions, credit-card companies, insurance companies, or other types of organizations. The client computing systems 104 may be implemented to provide interactive computing environments for customers to access various services offered by these service providers. Customers can utilize user computing systems 106 to access the interactive computing environments thereby accessing the services provided by these providers.

[0047] For example, a customer can submit an access request to access the interactive computing environment using a user computing system 106. Based on the request, the client computing system 104 can generate and submit a risk assessment query for the customer to the risk assessment server 118. The risk assessment query can include, for example, an identity of the customer and other information associated with the customer that can be utilized to generate predictor variables. The risk assessment server 118 can perform a risk assessment based on predictor variables 124 generated for the customer and return a responsive message to the client computing system 104. The responsive message can include at least the risk indicator and, in some aspects, can include explanatory data associated with the risk indicator. The explanatory data can indicate relationships between changes in the risk indicator and changes in at least some of the predictor variables 124 associated with the target entity.

[0048] Based on the received risk indicator, the client computing system 104 can determine whether to grant the customer access to the interactive computing environment. If the client computing system 104 determines that the level of risk associated with the customer accessing the interactive computing environment and the associated technical or financial service is too high, the client computing system 104 can deny access by the customer to the interactive computing environment. Conversely, if the client computing system 104 determines that the level of risk associated with the customer is acceptable, the client computing system 104 can grant access to the interactive computing environment by the customer and the customer would be able to utilize the various services provided by the service providers. For example, with the granted access, the customer can utilize the user computing system 106 to access clouding computing resources, online storage resources, web pages or other user interfaces provided by the client computing system 104 to execute applications, store data, query data, submit an online digital application, operate electronic tools, or perform various other operations within the interactive computing environment hosted by the client computing system 104. Example of Prediction Machine Learning Model

[0049] FIG. 3 is a diagram depicting an example of a graphical representation 300 of a transformer-based masked language model 302, according to certain aspects of the present disclosure. The transformer-based masked language model 302 can also be referred to as a masked language model 302 and can correspond to the prediction model 121 described above with respect to FIGS. 1-2. The masked language model 302 can make a prediction 304 of a missing value in an input sequence 306 that is provided as input to the masked language model 302. For example, to train the masked language model 302, one or more tokens in the input sequence 306 may be initially masked to enable language understanding, predictive analytics, or a combination thereof in the masked language model 302. Through using different input sequences to train the masked language model 302, the masked language model 302 can develop a capability of predicting the uncertain parameter masked in the input sequence 306 to complete the input sequence 306. Training the masked language model 302 can enable the masked language model 302 to adjust or optimize learnable parameters (e.g., weights or biases) of the masked language model 302 using prediction training data 128.

[0050] In some implementations, the input sequence 306 can be derived from or generated based on tabularized data (e.g., a transaction record). For example, the risk assessment computing system 130 may receive an unstructured transaction record that can include one or more segments where each segment includes one or more fields. The tabularized data may be presented as structured data in a table with one or more columns that each correspond to a particular field. The fields 308A-D of the tabularized data can be used to categorize interaction data of an interaction performed by an entity. As an example, the tabularized data may include four columns that each correspond to a particular field of an interaction description, interaction type, service name, or interaction category. Additionally, the tabularized data can include one or more segments that can each correspond to an individual interaction performed by at least one entity. The segments of the tabularized data can correspond with a respective row of the tabularized data that each include data corresponding to the fields of the tabularized data. To generate the input sequence 306 that is unstructured, the tabularized data can be pre-processed through cleaning, normalizing, tokenizing, or a combination thereof. Tokenizing can involve splitting textual data of the tabularized data into subsets (e.g., tokens) that can be more easily assigned meaning compared to the textual data presented in the table. Examples of the tokens can include words, word segments, phrases, etc.

[0051] As depicted in FIG. 3, the input sequence 306 can include a tokenized form of the unstructured transaction record such that at least one token is associated with each field of the unstructured transaction record. In some cases, the input sequence 306 can include one or more padding tokens that are unrelated to the textual data of the input sequence 306. Instead, the padding tokens can be used as placeholders, separators, or indicators to enable the masked language model 302 to process the input sequence 306 relatively easily. For example, as depicted in FIG. 3, the padding tokens can include one or more separator tokens 310A-D that can be positioned between each field 308 of the unstructured transaction record. Using the separators tokens 310A-D, the masked language model 302 can identify a starting point and an ending point of a particular field. In additional or alternative aspects, the padding tokens can include an initiator token 312 that can be positioned before the token associated with a first field 308A of the input sequence 306. If the tabularized data is used to generate more than one input sequence 306, each input sequence can start with the initiator token 312 to differentiate each input sequence, for example when training the masked language model 302.

[0052] As an illustrative example, a first field 308A corresponding to an interaction description of the interaction performed by the entity can include more than one token. Tokens corresponding to the first field 308A can be delineated based on an initiator token 312 positioned adjacent to a starting point of the first field 308A and a separator token 310A positioned adjacent to an ending point of the first field 308A. In some examples, as depicted in FIG. 3, an end of the input sequence 306 can be marked using separator token 310D. Alternatively, the input sequence 306 may include a terminator token at the end of the input sequence 306 to indicate to the masked language model 302 that the input sequence 306 corresponding to a particular interaction has ended.

[0053] In some aspects, the input sequence 306 can include one or more embeddings 314A-C that can provide context or meaning to the input sequence 306. The embeddings 314A-C can be vector representations that can be provided by each field 308 of the input sequence 306. For example, the embeddings 314A-C of the input sequence 306 can include at least one position embedding 314A, at least one segmentation embedding 314B, at least one token embedding 314C, or a combination thereof. The embeddings 314A-C can be referred to collectively as input embeddings.

[0054] The position embedding 314A can include information about a positioning of the tokens of the input sequence 306. For example, the input sequence 306 depicted in FIG. 3 includes a total of thirteen tokens. Accordingly, the position embedding 314A of the input sequence 306 includes a sequential label assigned to each token of the input sequence 306. For example, the initiator token 312 is assigned a first sequential label of Eo. The sequential labels can then be incremented by one until separator token 310D is assigned a last sequential label of En. Although the sequential labels are incremented by one in FIG. 3, any suitable increment may be used.

[0055] The segmentation embedding 314B can facilitate identification of the fields 308A-D of the input sequence 306. As depicted in FIG. 3, a first subset of tokens of the input sequence 306 associated with the first field 308A can include a first predefined embedding of Ea = 1. The first subset of tokens can include the initiator token 312, the tokens of the first field 3 08A, and a second separator token 310B marking the ending point of the first field 308A. A second subset of tokens associated with a second field 308B corresponding to an interaction type can correspond to a second predefined embedding of Eb = 1. The second subset of tokens can include the token(s) of the second field 308B and a second separator token 310B marking the ending point of the second field 308B. A third subset of tokens associated with a third field 308C corresponding to a service name can be assigned a third predefined embedding of Ec = 2. A fourth subset of tokens associated with a fourth field 308D corresponding to an interaction category can be assigned a fourth predefined embedding of Ed = 3. Similar to the second sbuset of tokens, the third subset and fourth subset can include token(s) of the third field 308C or fourth field 308D, respectively, in addition to a third separator token 310C or a fourth separator token 310D, respectively.

[0056] The token embedding 314C can correspond to word piece embeddings. For example, a website address ‘service.com’ can be split into two word pieces: ‘service’ and ‘com’. Accordingly, the token embedding 314C for ‘service.com’ can correspond to the word piece embeddings of ‘service’ and ‘com’. In additional or alternative aspects, the word piece embeddings may be associated with breaking down words into subword units (e.g., morphemes) that may retain linguistic meaning. For example, a word ‘overdue’ may be broken down into ‘over’ and ‘due’ such that ‘over’ can indicate too much or more than enough of something and ‘due’ can indicate that something is owed or expected at a predetermined time.

[0057] As depicted in FIG. 3, the input sequence 306 can concatenate categorical and textual fields provided in the tabularized data, enabling more relational and information embeddings that can provide more context compared to embeddings for each field 308 considered independently and then concatenated. If a specific field has a relatively large volume of missing values, the concatenation with other fields that potentially have less missing values can enable improved predictions and representation due to having the context provided by the other fields. In some examples, the first field 308A can include the interaction description that can implicitly include information corresponding to other fields (e.g., service name, interaction type, etc.). For example, the interaction description may indicate that an online purchase was made using a credit card through a service.com website. If the service name was missing for the interaction associated with the online purchase, the masked language model 302 may use the interaction description to infer or determine that the service name is ‘service’ or related to ‘service’. A uniform resource locator (URL) associated with the online purchase can generally follow a convention of a service name being listed before a top level domain (.com, .net, .org, etc.) in the URL.

[0058] Due to the masked language model 302 being trained based on the tabularized data instead of general domain corpora, the masked language model 302 can provide domain-specific predictions. In some aspects, word cooccurrences in general domain corpora can be different than that of the tabularized data. For example, certain service names may be nonsensical or lack linguistic meaning. As another example, a particular service name may include one or more words that are associated with meaning unrelated to a service provided by the entity having the service name. Further, more than one service may use the same service name, such as if the services are in different industries. For instance, a service name of ‘Delta’ may refer to an airline or a faucet company. Additionally, ‘Delta’ can have other meanings depending on context (e.g., with respect to mathematics, astronomy, or geography). Thus, the masked language model 302 being trained on the tabularized data instead of a general domain corpora can provide more specific and more accurate coverage with respect to predicting or understanding terms associated with the service names.

[0059] To train the masked language model 302, one or more tokens (e.g., an objective token 316) of the input sequence 306 may be masked in the input sequence 306 such that the input sequence 306 is incomplete. Masking the objective token 316 may involve replacing the objective token 316 or otherwise modifying the objective token 316. If more than one objective token is present in the input sequence 306, each objective token may be modified similarly. Alternatively, in some examples, the each objective token may be modified using a different process. In some aspects, masking the objective token 316 may involve randomly masking the tokens of the input sequence 306 prior to inputting the input sequence 306 to the masked language model 302. For example, as depicted in FIG. 3, masking the obj ective token 316 may involve replacing the obj ective token 316 with a mask token (e..g, ‘[MASK]’) at a first frequency. The first frequency can correspond to how often a certain modification is made to the input sequence(s) 306 of prediction training data 128 used to train the masked language model 302. For instance, 80% of modifications to the input sequence 306 may involve replacing the objective token 316 with the mask token. As another example, masking the objective token 316 may involve replacing the objective token 316 with a random token at a second frequency (e.g., 10% of the modifications to the input sequence 306). For example, instead of ‘Entertainment’ as the objective token 316, a random token of ‘Education’ may replace the objective token 316. Asa further example, masking the objective token 316 may involve keeping the objective token 316 unchanged at a third frequency (e.g., 10% of the modifications to the input sequence 306).

[0060] Once the masked language model 302 receive the input sequence 306 with the masked objective token 316, the masked language model 302 can predict the masked objective token 316 using context or meaning provided by tokens adjacent to the masked objective token 316. As an example, the model training application 112 may identify an objective field of the input sequence 306 to be predicted by the masked language model 302. The model training application 112 then can mask the objective token 316 associated with the objective field. As depicted in FIG. 3, the objective field corresponds to the fourth field 308D such that the objective token 316 can be the token of the fourth field 308D.

[0061] Additionally, once the objective field is identified, a label tensor can be generated for use in loss calculation or approximation of the masked language model 302. In some implementations, the label tensor can be associated with a loss function used as an objective function to minimize when optimizing or adjusting the masked language model 302. For example, as depicted in FIG. 3, a label of the label tensor can correspond to the interaction category of the fourth field 308D of the input sequence 306.

[0062] The masked language model 302 can use the masked objective token 316 in context of the input sequence 306 to generate the prediction 304 to complete the input sequence 306. Additionally, the masked language model 302 can process the label tensor using a loss calculation or approximation to determine gradient changes and modifications to model weights. The masked language model 302 can use a difference between an output probability distribution of the prediction 304 and true one-hot encoded or indexed labels to identify a loss approximation to improve the masked language model 302. Additionally, the masked language model 302 can output one or more output embeddings that can be used to determine the prediction 304 for the masked objective token 316.

[0063] FIG. 4 is a block diagram depicting an example of a process 400 of determining a prediction 304 for a masked objective token 316 using one or more output embeddings 402 outputted by a masked language model 302, according to certain aspects of the present disclosure. The process 400 is described with reference to components shown in FIGS. 13. In some implementations, the output embeddings 402 can be referred to as logits. The output embeddings 402 can be represented as vectors having a length equal to a vocabulary size of the masked language model 302.

[0064] Once the masked language model 302 outputs the output embeddings 402, the output embeddings can be processed using a first function transformation 404A. In some implementations, the masked language model 302 can apply the first function transformation 404A to the output embeddings 402. As an example, the first function transformation 404A can be a softmax function. The softmax function may be referred to as softargmax or as a normalized exponential function. The first function transformation 404A can use the output embeddings 402 as input and normalize the output embeddings 402 into a probability distribution 406. For example, the output embeddings 402 may be represented by a vector of K real numbers. The first function transformation 404A can transform the vector of K real numbers into the probability distribution 406 of K possible outcomes or probabilities proportional to exponentials of the real numbers in the vector. Accordingly, after applying the first function transformation 404A, each component in the vector can be in an interval between 0 and 1 and can add up to 1, such that the components of the vector can be interpreted as probabilities.

[0065] A second function transformation 404B can be applied to the probability distribution 406. As an example, the second function transformation 404B can be an argmax function. Applying the second function transformation 404B can enable the masked language model 302 to determine an argument of the probability distribution 406 that has a maximum value. Accordingly, the second function transformation 404B can be used to determine a prediction 304 with the highest predicted probability in the probability distribution 406. The prediction 304 outputted by the second function transformation 404B can correspond to the prediction 304 for the masked objective token 316. As an example, if the masked objective token 316 corresponds to an interaction type of an interaction performed by an entity, the prediction 304 may indicate a type of payment card (e.g., debit card, credit card, ATM card, etc.) used by the entity to perform the interaction. Example of Computing System for Machine-Learning Operations

[0066] Any suitable computing system or group of computing systems can be used to perform the operations for the machine-learning operations described herein. For example, FIG. 5 is a block diagram depicting an example of a computing device 500, which can be used to implement the risk assessment server 118 or the model training server 110. The computing device 500 can include various devices for communicating with other devices in the operating environment 100, as described with respect to FIG. 1. The computing device 500 can include various devices for performing one or more transformation operations described above with respect to FIGS. 1-4.

[0067] The computing device 500 can include a processor 502 that is communicatively coupled to a memory 504. The processor 502 executes computer-executable program code stored in the memory 504, accesses information stored in the memory 504, or both. Program code may include machine-executable instructions that may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, among others.

[0068] Examples of a processor 502 include a microprocessor, an application-specific integrated circuit, a field-programmable gate array, or any other suitable processing device. The processor 502 can include any number of processing devices, including one. The processor 502 can include or communicate with a memory 504. The memory 504 stores program code that, when executed by the processor 502, causes the processor to perform the operations described in this disclosure.

[0069] The memory 504 can include any suitable non-transitory computer-readable storage medium. The computer-readable medium can include any electronic, optical, magnetic, or other storage device capable of providing a processor with computer-readable program code or other program code. Non-limiting examples of a computer-readable medium include a magnetic disk, memory chip, optical storage, flash memory, storage class memory, ROM, RAM, an ASIC, magnetic storage, or any other medium from which a computer processor can read and execute program code. The program code may include processor-specific program code generated by a compiler or an interpreter from code written in any suitable computer-programming language. Examples of suitable programming language include Hadoop, C, C++, C#, Visual Basic, Java, Python, Perl, JavaScript, ActionScript, etc.

[0070] The computing device 500 may also include a number of external or internal devices such as input or output devices. For example, the computing device 500 is shown with an input / output interface 508 that can receive input from input devices or provide output to output devices. A bus 506 can also be included in the computing device 500. The bus 506 can communicatively couple one or more components of the computing device 500.

[0071] The computing device 500 can execute program code 514 that includes the risk assessment application 114 and / or the model training application 112. The program code 514 for the risk assessment application 114 and / or the model training application 112 may be resident in any suitable computer-readable medium and may be executed on any suitable processing device. For example, as depicted in FIG. 5, the program code 514 for the risk assessment application 114 and / or the model training application 112 can reside in the memory 504 at the computing device 500 along with the program data 516 associated with the program code 514, such as the predictor variables 124 and / or the model training samples. Executing the risk assessment application 114 or the model training application 112 can configure the processor 502 to perform the operations described herein.

[0072] In some aspects, the computing device 500 can include one or more output devices. One example of an output device is the network interface device 510 depicted in FIG. 5. A network interface device 510 can include any device or group of devices suitable for establishing a wired or wireless data connection to one or more data networks described herein. Non-limiting examples of the network interface device 510 include an Ethernet network adapter, a modem, etc.

[0073] Another example of an output device is the presentation device 512 depicted in FIG. 5. A presentation device 512 can include any device or group of devices suitable for providing visual, auditory, or other suitable sensory output. Non-limiting examples of the presentation device 512 include a touchscreen, a monitor, a speaker, a separate mobile computing device, etc. In some aspects, the presentation device 512 can include a remote client-computing device that communicates with the computing device 500 using one or more data networks described herein. In other aspects, the presentation device 512 can be omitted.

[0074] The foregoing description of some examples has been presented only for the purpose of illustration and description and is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Numerous modifications and adaptations thereof will be apparent to those skilled in the art without departing from the spirit and scope of the disclosure.

Claims

1. A computer-implemented method comprising:receiving, by a processor, a request for a risk indicator of a target entity determined from predictor variables associated with the target entity, the risk indicator indicating a level of risk associated with the target entity;training, by the processor, a risk assessment model for determining the risk indicator from the predictor variables using a training process, wherein the training process includes operations comprising:receiving a request to predict a value of an unknown segment of risk assessment training data associated with an interaction performed by the target entity, the risk assessment training data being used to train the risk assessment model to output the risk indicator;predicting the value of the unknown segment by executing a prediction model, wherein the prediction model is trained using one or more segments of at least one unstructured transaction record; andsubsequent to predicting the value of the unknown segment, training the risk assessment model using the risk assessment training data that includes the predicted value generated by the prediction model;generating, by the processor and for the target entity, the risk indicator for the target entity using the trained risk assessment model; andtransmitting, from the processor to a remote computing device, a responsive message including at least the risk indicator for use in controlling access of the target entity to one or more interactive computing environments.

2. The method of claim 1, wherein training the prediction model further comprises: generating prediction training data comprising one or more input sequences by:receiving an unstructured transaction record comprising the one or more segments, wherein each segment comprises one or more fields;generating an input sequence to be included in the one or more input sequences, the input sequence comprising atokenized form of a segment of the one or more segments and one or more padding tokens, wherein the tokenized form ofthe segment comprises at least one token associated with each field of the unstructured transaction record;identifying an objective field of the input sequence to be predicted by the prediction model;creating a label tensor associated with the objective field; and masking an objective token associated with the objective field; andtraining the prediction model using the prediction training data, wherein the prediction model is configured to output a prediction for the objective token.

3. The method of claim 2, wherein the one or more padding tokens comprises an initiator token positioned before the token associated with a first field of the input sequence.

4. The method of claim 2, wherein the one or more padding tokens comprises one or more separator tokens positioned between each field of the unstructured transaction record.

5. The method of claim 2, wherein masking the objective token further comprises: replacing the objective token with a mask token at a first frequency;replacing the objective token with a random token at a second frequency; and keeping the objective token unchanged at a third frequency.

6. The method of claim 2, wherein training the prediction model further comprises: providing the input sequence and one or more embeddings to the prediction model, wherein the one or more embeddings correspond to the at least one token of the tokenized form of the segment;receiving, as output from the prediction model using the input sequence and the one or more embeddings, a prediction of the objective token; andmodifying, based on the prediction and the label tensor associated with the objective token, one or more weights of the prediction model to minimize a loss function associated with the prediction model.

7. The method of claim 6, wherein the one or more embeddings are generated by concatenating the one or more fields of the segment.

8. A system comprising:a processing device; anda memory device in which instructions executable by the processing device are stored for causing the processing device to:receive a request for a risk indicator of a target entity determined from predictor variables associated with the target entity, the risk indicator indicating a level of risk associated with the target entity;training a risk assessment model for determining the risk indicator from the predictor variables using a training process, wherein the training process includes operations comprising:receive a request to predict a value of an unknown segment of risk assessment training data associated with an interaction performed by the target entity, the risk assessment training data being used to train the risk assessment model to output the risk indicator;predict the value of the unknown segment by executing a prediction model, wherein the prediction model is trained using one or more segments of at least one unstructured transaction record; andsubsequent to predicting the value of the unknown segment, train the risk assessment model using the risk assessment training data that includes the predicted value generated by the prediction model; andgenerate, for the target entity, the risk indicator for the target entity using the trained risk assessment model.

9. The system of claim 8, wherein training the prediction model further comprises: generating prediction training data comprising one or more input sequences by:receiving an unstructured transaction record comprising the one or more segments, wherein each segment comprises one or more fields;generating an input sequence to be included in the one or more input sequences, the input sequence comprising atokenized form of a segment of the one or more segments and one or more padding tokens, wherein the tokenized form of the segment comprises at least one token associated with each field of the unstructured transaction record;identifying an objective field of the input sequence to be predicted by the prediction model;creating a label tensor associated with the objective field; and masking an objective token associated with the objective field; andtraining the prediction model using the prediction training data, wherein the prediction model is configured to output a prediction for the objective token.

10. The system of claim 9, wherein the one or more padding tokens comprises an initiator token positioned before the token associated with a first field of the input sequence.

11. The system of claim 9, wherein the one or more padding tokens comprises one or more separator tokens positioned between each field of the unstructured transaction record.

12. The system of claim 9, wherein masking the objective token further comprises: replacing the objective token with a mask token at a first frequency;replacing the objective token with a random token at a second frequency; and keeping the objective token unchanged at a third frequency.

13. The system of claim 9, wherein training the prediction model further comprises: providing the input sequence and one or more embeddings to the prediction model, wherein the one or more embeddings correspond to the at least one token of the tokenized form of the segment;receiving, as output from the prediction model using the input sequence and the one or more embeddings, a prediction of the objective token; andmodifying, based on the prediction and the label tensor associated with the objective token, one or more weights of the prediction model to minimize a loss function associated with the prediction model.

14. The system of claim 8, wherein the instructions further cause the processing device to:transmit, to a remote computing device, a responsive message including at least the risk indicator for use in controlling access of the target entity to one or more interactive computing environments.

15. A non-transitory computer-readable storage medium having program code that is executable by a processor to cause a computing device to perform operations, the operations comprising:receiving a request for a risk indicator of a target entity determined from predictor variables associated with the target entity, the risk indicator indicating a level of risk associated with the target entity;training a risk assessment model for determining the risk indicator from the predictor variables using a training process, wherein the training process comprises:receiving a request to predict a value of an unknown segment of risk assessment training data associated with an interaction performed by the target entity, the risk assessment training data being used to train the risk assessment model to output the risk indicator;predicting the value of the unknown segment by executing a prediction model, wherein the prediction model is trained using one or more segments of at least one unstructured transaction record; andsubsequent to predicting the value of the unknown segment, training the risk assessment model using the risk assessment training data that includes the predicted value generated by the prediction model; andgenerating, for the target entity, the risk indicator for the target entity using the trained risk assessment model.

16. The non-transitory computer-readable storage medium of claim 15, wherein training the prediction model further comprises:generating prediction training data comprising one or more input sequences by:receiving an unstructured transaction record comprising the one or more segments, wherein each segment comprises one or more fields;generating an input sequence to be included in the one or more input sequences, the input sequence comprising atokenized form of a segment of the oneor more segments and one or more padding tokens, wherein the tokenized form of the segment comprises at least one token associated with each field of the unstructured transaction record;identifying an objective field of the input sequence to be predicted by the prediction model;creating a label tensor associated with the objective field; andmasking an objective token associated with the objective field; andtraining the prediction model using the prediction training data, wherein the prediction model is configured to output a prediction for the objective token.

17. The non-transitory computer-readable storage medium of claim 16, wherein the one or more padding tokens comprises an initiator token positioned before the token associated with a first field of the input sequence.

18. The non-transitory computer-readable storage medium of claim 16, wherein the one or more padding tokens comprises one or more separator tokens positioned between each field of the unstructured transaction record.

19. The non-transitory computer-readable storage medium of claim 16, wherein masking the objective token further comprises:replacing the objective token with a mask token at a first frequency;replacing the objective token with a random token at a second frequency; and keeping the objective token unchanged at a third frequency.

20. The non-transitory computer-readable storage medium of claim 15, wherein the operations further comprise:transmitting, to a remote computing device, a responsive message including at least the risk indicator for use in controlling access of the target entity to one or more interactive computing environments.

Images