System, method, and computer program product for operating a pruned transformer machine learning model architecture

Abstract

Systems, methods, and computer program products are provided for operating a pruned transformer machine learning model architecture. A system may include a processor to receive initial data associated with sequence information, mask one or more data instances of the initial data associated with the sequence information based on an output of a masking machine learning model to provide masked intermediate input data, provide the masked intermediate input data as an input to a classification machine learning model to generate a classification label of the masked intermediate input data, wherein the classification machine learning model is a transformer machine learning model configured to provide a prediction of a classification label of an input as an output, and perform an action based on the classification label of the masked intermediate input data.

Description

SYSTEM, METHOD, AND COMPUTER PROGRAM PRODUCT FOR OPERATING A PRUNED TRANSFORMER MACHINE LEARNING MODEL ARCHITECTURECROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to United States Provisional Patent Application No. 63 / 528,720 filed on July 25, 2023, the disclosure of which is hereby incorporated by reference in its entirety.BACKGROUND1 . Technical Field

[0002] The present disclosure relates generally to the use of transformer machine learning models and, in some non-limiting embodiments or aspects, to systems, methods, and computer program products for operating a pruned transformer machine learning model architecture.2. Technical Considerations

[0003] Neural networks (NNs) may be used for classification / prediction tasks in a variety of applications, such as facial recognition, fraud detection, disease diagnosis, navigation of self-driving cars, and / or the like. In such applications, NNs receive an input and generate predictions based on the input, for example, the identity of an individual, whether a payment transaction is fraudulent or not fraudulent, whether a disease is associated with one or more genetic markers, whether an object in a field of view of a self-driving car is in the self-driving car’s path, and / or the like.

[0004] A form of a NN may include a transformer machine learning model. A transformer machine learning model may refer to a deep learning model that is designed to process sequential input data and includes a self-attention mechanism, which gives different weight to the significance of each part of the sequential input data. The transformer machine learning model may process an entirety of input data all at once, and the attention mechanism may provide context for any position of the sequential input data.

[0005] However, a transformer machine learning model may require a large degree of computational resources due to the self-attention mechanism. For example, the self-attention mechanism of a transformer machine learning model may require a number of components that is equal to or greater than the square of the number ofinputs dimensions. This problem may prevent a transformer machine learning model from being used in applications that require high-dimensional inputs.SUMMARY

[0006] Accordingly, provided are improved methods, systems, and computer program products for operating a pruned transformer machine learning model architecture.

[0007] According to non-limiting embodiments or aspects, provided is a system for operating a pruned transformer machine learning model architecture. In some nonlimiting embodiments or aspects, the system may include at least one processor. In some non-limiting embodiments or aspects, the at least one processor may be configured to receive initial data associated with sequence information. In some nonlimiting embodiments or aspects, the at least one processor may be configured to mask one or more data instances of the initial data associated with the sequence information based on an output of a masking machine learning model to provide masked intermediate input data. In some non-limiting embodiments or aspects, the at least one processor may be configured to provide the masked intermediate input data as an input to a classification machine learning model to generate a classification label of the masked intermediate input data. In some non-limiting embodiments or aspects, the classification machine learning model may be transformer machine learning model configured to provide a prediction of a classification label of an input as an output. In some non-limiting embodiments or aspects, the at least one processor may be configured to perform an action based on the classification label of the masked intermediate input data.

[0008] In some non-limiting embodiments or aspects, when masking the one or more data instances of the initial data associated with the sequence information, the at least one processor may be configured to provide the initial data as an input to the masking machine learning model to generate an output. In some non-limiting embodiments or aspects, the output of the masking machine learning model may include a prediction of a binary classification of each data instance of the initial data. In some non-limiting embodiments or aspects, the at least one processor may be configured to assign a negative binary classification to the one or more data instances of the initial data provided as the input to the masking machine learning model basedon the prediction of a binary classification of the one or more data instances of the initial data.

[0009] In some non-limiting embodiments or aspects, when masking the one or more data instances of the initial data associated with the sequence information, the at least one processor may be configured to provide the initial data as an input to the masking machine learning model to generate an output. In some non-limiting embodiments or aspects, the output of the masking machine learning model may include a prediction of a binary classification of each data instance of the initial data. In some non-limiting embodiments or aspects, the at least one processor may be configured to remove a value of the one or more data instances of the initial data provided as the input to the masking machine learning model from an embedding to be provided as an input to the classification machine learning model based on the prediction of a binary classification of the one or more data instances of the initial data.

[0010] In some non-limiting embodiments or aspects, the at least one processor may be configured to train the masking machine learning model and the classification machine learning model based on a loss function.

[0011] In some non-limiting embodiments or aspects, the loss function may include the formula:

[0012] In some non-limiting embodiments or aspects, the initial data associated with the sequence information may include a plurality of tokens. In some non-limiting embodiments or aspects, the at least one processor may be configured to tokenize discrete instances of the sequence information to provide the plurality of tokens of the initial data associated with sequence information.

[0013] In some non-limiting embodiments or aspects, when performing the action, the at least one processor may be configured to execute a fraud protection procedure based on the classification label of the masked intermediate input data.

[0014] According to non-limiting embodiments or aspects, provided is a system for operating a pruned transformer machine learning model architecture. In some nonlimiting embodiments or aspects, the method may include receiving initial data associated with sequence information. In some non-limiting embodiments or aspects, the method may include masking one or more data instances of the initial data associated with the sequence information based on an output of a masking machine learning model to provide masked intermediate input data. In some non-limitingembodiments or aspects, the method may include providing the masked intermediate input data as an input to a classification machine learning model to generate a classification label of the masked intermediate input data. In some non-limiting embodiments or aspects, the classification machine learning model may be a transformer machine learning model configured to provide a prediction of a classification label of an input as an output. In some non-limiting embodiments or aspects, the method may include performing an action based on the classification label of the masked intermediate input data.

[0015] In some non-limiting embodiments or aspects, masking the one or more data instances of the initial data associated with the sequence information may include providing the initial data as an input to the masking machine learning model to generate an output. In some non-limiting embodiments or aspects, the output of the masking machine learning model may include a prediction of a binary classification of each data instance of the initial data. In some non-limiting embodiments or aspects, the method may include assigning a negative binary classification to the one or more data instances of the initial data provided as the input to the masking machine learning model based on the prediction of a binary classification of the one or more data instances of the initial data.

[0016] In some non-limiting embodiments or aspects, masking the one or more data instances of the initial data associated with the sequence information may include providing the initial data as an input to the masking machine learning model to generate an output. In some non-limiting embodiments or aspects, the output of the masking machine learning model may include a prediction of a binary classification of each data instance of the initial data. In some non-limiting embodiments or aspects, the method may include removing a value of the one or more data instances of the initial data provided as the input to the masking machine learning model from an embedding to be provided as an input to the classification machine learning model based on the prediction of a binary classification of the one or more data instances of the initial data.

[0017] In some non-limiting embodiments or aspects, the method may include training the masking machine learning model and the classification machine learning model based on a loss function.

[0018] In some non-limiting embodiments or aspects, the loss function may include the formula:

[0019] In some non-limiting embodiments or aspects, the initial data associated with the sequence information may include a plurality of tokens. In some non-limiting embodiments or aspects, the method may include tokenizing discrete instances of the sequence information to provide the plurality of tokens of the initial data associated with sequence information.

[0020] In some non-limiting embodiments or aspects, performing the action may include executing a fraud protection procedure based on the classification label of the masked intermediate input data.

[0021] According to non-limiting embodiments or aspects, provided is a computer program product for operating a pruned transformer machine learning model architecture. In some non-limiting embodiments or aspects the compute program product may include at least one non-transitory computer-readable medium including one or more instructions. In some non-limiting embodiments or aspects, the one or more instructions, when executed by at least one processor, may cause the at least one processor to receive initial data associated with sequence information. In some non-limiting embodiments or aspects, the one or more instructions, when executed by at least one processor, may cause the at least one processor to mask one or more data instances of the initial data associated with the sequence information based on an output of a masking machine learning model to provide masked intermediate input data. In some non-limiting embodiments or aspects, the one or more instructions, when executed by at least one processor, may cause the at least one processor to provide the masked intermediate input data as an input to a classification machine learning model to generate a classification label of the masked intermediate input data. In some non-limiting embodiments or aspects, the classification machine learning model may be a transformer machine learning model configured to provide a prediction of a classification label of an input as an output. In some non-limiting embodiments or aspects, the one or more instructions, when executed by at least one processor, may cause the at least one processor to perform an action based on the classification label of the masked intermediate input data.

[0022] In some non-limiting embodiments or aspects, the one or more instructions that cause the at least one processor to mask the one or more data instances of theinitial data associated with the sequence information, may cause the at least one processor to provide the initial data as an input to the masking machine learning model to generate an output. In some non-limiting embodiments or aspects, the output of the masking machine learning model may include a prediction of a binary classification of each data instance of the initial data. In some non-limiting embodiments or aspects, the one or more instructions that cause the at least one processor to mask the one or more data instances of the initial data associated with the sequence information, may cause the at least one processor to assign a negative binary classification to the one or more data instances of the initial data provided as the input to the masking machine learning model based on the prediction of a binary classification of the one or more data instances of the initial data.

[0023] In some non-limiting embodiments or aspects, the one or more instructions that cause the at least one processor to mask the one or more data instances of the initial data associated with the sequence information, may cause the at least one processor to provide the initial data as an input to the masking machine learning model to generate an output. In some non-limiting embodiments or aspects, the output of the masking machine learning model may include a prediction of a binary classification of each data instance of the initial data. In some non-limiting embodiments or aspects, the one or more instructions that cause the at least one processor to mask the one or more data instances of the initial data associated with the sequence information, may cause the at least one processor to remove a value of the one or more data instances of the initial data provided as the input to the masking machine learning model from an embedding to be provided as an input to the classification machine learning model based on the prediction of a binary classification of the one or more data instances of the initial data.

[0024] In some non-limiting embodiments or aspects, the one or more instructions may further cause the at least one processor to train the masking machine learning model and the classification machine learning model based on a loss function. In some non-limiting embodiments or aspects, the loss function may include the formula:

[0025] In some non-limiting embodiments or aspects, the initial data associated with the sequence information may include a plurality of tokens. In some non-limiting embodiments or aspects, the one or more instructions may further cause the at leastone processor to tokenize discrete instances of the sequence information to provide the plurality of tokens of the initial data associated with sequence information.

[0026] In some non-limiting embodiments or aspects, the one or more instructions that cause the at least one processor to perform the action, may cause the at least one processor to execute a fraud protection procedure based on the classification label of the masked intermediate input data.

[0027] Further non-limiting embodiments or aspects will be set forth in the following numbered clauses:

[0028] Clause 1 : A system, comprising: at least one processor configured to: receive initial data associated with sequence information; mask one or more data instances of the initial data associated with the sequence information based on an output of a masking machine learning model to provide masked intermediate input data; provide the masked intermediate input data as an input to a classification machine learning model to generate a classification label of the masked intermediate input data, wherein the classification machine learning model is a transformer machine learning model configured to provide a prediction of a classification label of an input as an output; and perform an action based on the classification label of the masked intermediate input data.

[0029] Clause 2: The system of clause 1 , wherein, when masking the one or more data instances of the initial data associated with the sequence information, the at least one processor is configured to: provide the initial data as an input to the masking machine learning model to generate an output, wherein the output of the masking machine learning model comprises a prediction of a binary classification of each data instance of the initial data; and assign a negative binary classification to the one or more data instances of the initial data provided as the input to the masking machine learning model based on the prediction of a binary classification of the one or more data instances of the initial data.

[0030] Clause 3: The system of clause 1 or 2, wherein, when masking the one or more data instances of the initial data associated with the sequence information, the at least one processor is configured to: provide the initial data as an input to the masking machine learning model to generate an output, wherein the output of the masking machine learning model comprises a prediction of a binary classification of each data instance of the initial data; and remove a value of the one or more data instances of the initial data provided as the input to the masking machine learningmodel from an embedding to be provided as an input to the classification machine learning model based on the prediction of a binary classification of the one or more data instances of the initial data.

[0031] Clause 4: The system of any of clauses 1 -3, wherein the at least one processor is further configured to: train the masking machine learning model and the classification machine learning model based on a loss function.

[0032] Clause 5: The system of any of clauses 1 -4, wherein the loss function comprises the formula: Loss = Lcts+ a * £ maskX N.

[0033] Clause 6: The system of any of clauses 1 -5, wherein the initial data associated with the sequence information comprises a plurality of tokens, and wherein the at least one processor is configured to: tokenize discrete instances of the sequence information to provide the plurality of tokens of the initial data associated with sequence information.

[0034] Clause 7: The system of any of clauses 1 -6, wherein, when performing the action, the at least one processor is configured to: execute a fraud protection procedure based on the classification label of the masked intermediate input data.

[0035] Clause 8: A computer-implemented method, comprising: receiving, with at least one processor, initial data associated with sequence information; masking, with at least one processor, one or more data instances of the initial data associated with the sequence information based on an output of a masking machine learning model to provide masked intermediate input data; providing, with at least one processor, the masked intermediate input data as an input to a classification machine learning model to generate a classification label of the masked intermediate input data, wherein the classification machine learning model is a transformer machine learning model configured to provide a prediction of a classification label of an input as an output; and performing, with at least one processor, an action based on the classification label of the masked intermediate input data.

[0036] Clause 9: The computer-implemented method of clause 8, wherein masking the one or more data instances of the initial data associated with the sequence information comprises: providing the initial data as an input to the masking machine learning model to generate an output, wherein the output of the masking machine learning model comprises a prediction of a binary classification of each data instance of the initial data; and assigning a negative binary classification to the one or moredata instances of the initial data provided as the input to the masking machine learning model based on the prediction of a binary classification of the one or more data instances of the initial data.

[0037] Clause 10: The computer-implemented method of clause 8 or 9, wherein masking the one or more data instances of the initial data associated with the sequence information comprises: providing the initial data as an input to the masking machine learning model to generate an output, wherein the output of the masking machine learning model comprises a prediction of a binary classification of each data instance of the initial data; and removing a value of the one or more data instances of the initial data provided as the input to the masking machine learning model from an embedding to be provided as an input to the classification machine learning model based on the prediction of a binary classification of the one or more data instances of the initial data.

[0038] Clause 1 1 : The computer-implemented method of any of clauses 8-10, further comprising: training the masking machine learning model and the classification machine learning model based on a loss function.

[0039] Clause 12: The computer-implemented method of any of clauses 8-1 1 , wherein the loss function comprises the formula: Loss = Lcis+ a * , maskXi N.

[0040] Clause 13: The computer-implemented method of any of clauses 8-12, wherein the initial data associated with the sequence information comprises a plurality of tokens, and the method further comprises: tokenizing discrete instances of the sequence information to provide the plurality of tokens of the initial data associated with sequence information.

[0041] Clause 14: The computer-implemented method of any of clauses 8-13, wherein performing the action comprises: executing a fraud protection procedure based on the classification label of the masked intermediate input data.

[0042] Clause 15: A computer program product comprising at least one non- transitory computer-readable medium including one or more instructions that, when executed by at least one processor, cause the at least one processor to: receive initial data associated with sequence information; mask one or more data instances of the initial data associated with the sequence information based on an output of a masking machine learning model to provide masked intermediate input data; provide the masked intermediate input data as an input to a classification machine learning modelto generate a classification label of the masked intermediate input data, wherein the classification machine learning model is a transformer machine learning model configured to provide a prediction of a classification label of an input as an output; and perform an action based on the classification label of the masked intermediate input data.

[0043] Clause 16: The computer program product of clause 15, wherein, the one or more instructions that cause the at least one processor to mask the one or more data instances of the initial data associated with the sequence information, cause the at least one processor to: provide the initial data as an input to the masking machine learning model to generate an output, wherein the output of the masking machine learning model comprises a prediction of a binary classification of each data instance of the initial data; and assign a negative binary classification to the one or more data instances of the initial data provided as the input to the masking machine learning model based on the prediction of a binary classification of the one or more data instances of the initial data.

[0044] Clause 17: The computer program product of clause 15 or 16, wherein, the one or more instructions that cause the at least one processor to mask the one or more data instances of the initial data associated with the sequence information, cause the at least one processor to: provide the initial data as an input to the masking machine learning model to generate an output, wherein the output of the masking machine learning model comprises a prediction of a binary classification of each data instance of the initial data; and remove a value of the one or more data instances of the initial data provided as the input to the masking machine learning model from an embedding to be provided as an input to the classification machine learning model based on the prediction of a binary classification of the one or more data instances of the initial data.

[0045] Clause 18: The computer program product of any of clauses 15-17, wherein the one or more instructions further cause the at least one processor to: train the masking machine learning model and the classification machine learning model based on a loss function, wherein the loss function comprises the formula: Loss = Lcis+ a * XmasfcX1:Jv.

[0046] Clause 19: The computer program product of any of clauses 15-18, wherein the initial data associated with the sequence information comprises a plurality oftokens, and wherein the one or more instructions further cause the at least one processor to: tokenize discrete instances of the sequence information to provide the plurality of tokens of the initial data associated with sequence information.

[0047] Clause 20: The computer program product of any of clauses 15-19, wherein, the one or more instructions that cause the at least one processor to perform the action, cause the at least one processor to: execute a fraud protection procedure based on the classification label of the masked intermediate input data.

[0048] These and other features and characteristics of the present disclosure, as well as the methods of operation and functions of the related elements of structures and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the disclosed subject matter.BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Additional advantages and details of the present disclosure are explained in greater detail below with reference to the exemplary embodiments that are illustrated in the accompanying figures, in which:

[0050] FIG. 1 is a diagram of a non-limiting embodiment or aspect of an environment in which systems, devices, products, apparatus, and / or methods, described herein, may be implemented, according to the principles of the present disclosure;

[0051] FIG. 2 is a flowchart of a non-limiting embodiment or aspect of a process for operating a pruned transformer machine learning model architecture;

[0052] FIGS. 3A-3E are schematic diagrams of an exemplary implementation of a system and / or method for operating a pruned transformer machine learning model architecture, according to some non-limiting embodiments or aspects;

[0053] FIG. 4 is a diagram of an exemplary environment in which systems, methods, and / or computer program products, described herein, may be implemented, according to some non-limiting embodiments or aspects; and

[0054] FIG. 5 is a schematic diagram of example components of one or more devices of FIG. 1 and / or FIG. 4, according to some non-limiting embodiments or aspects.DETAILED DESCRIPTION

[0055] For purposes of the description hereinafter, the terms “end,” “upper,” “lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” “longitudinal,” and derivatives thereof shall relate to the embodiments as they are oriented in the drawing figures. However, it is to be understood that the embodiments may assume various alternative variations and step sequences, except where expressly specified to the contrary. It is also to be understood that the specific devices and processes illustrated in the attached drawings, and described in the following specification, are simply exemplary embodiments or aspects of the disclosed subject matter. Hence, specific dimensions and other physical characteristics related to the embodiments or aspects disclosed herein are not to be considered as limiting.

[0056] Some non-limiting embodiments or aspects may be described herein in connection with thresholds. As used herein, satisfying a threshold may refer to a value being greater than the threshold, more than the threshold, higher than the threshold, greater than or equal to the threshold, less than the threshold, fewer than the threshold, lower than the threshold, less than or equal to the threshold, equal to the threshold, etc.

[0057] No aspect, component, element, structure, act, step, function, instruction, and / or the like used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more” and “at least one.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, and / or the like) and may be used interchangeably with “one or more” or “at least one.” Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based at least partially on” unless explicitly stated otherwise. In addition, reference to an action being “based on” a condition may refer to the action being “in response to” the condition. For example, the phrases “based on” and “in response to” may, in somenon-limiting embodiments or aspects, refer to a condition for automatically triggering an action (e.g., a specific operation of an electronic device, such as a computing device, a processor, and / or the like).

[0058] As used herein, the term “acquirer institution” may refer to an entity licensed and / or approved by a transaction service provider to originate transactions (e.g., payment transactions) using a payment device associated with the transaction service provider. The transactions the acquirer institution may originate may include payment transactions (e.g., purchases, original credit transactions (OCTs), account funding transactions (AFTs), and / or the like). In some non-limiting embodiments or aspects, an acquirer institution may be a financial institution, such as a bank. As used herein, the term “acquirer system” may refer to one or more computing devices operated by or on behalf of an acquirer institution, such as a server computer executing one or more software applications.

[0059] As used herein, the term “account identifier” may include one or more primary account numbers (PANs), tokens, or other identifiers associated with a customer account. The term “token” may refer to an identifier that is used as a substitute or replacement identifier for an original account identifier, such as a PAN. Account identifiers may be alphanumeric or any combination of characters and / or symbols. Tokens may be associated with a PAN or other original account identifier in one or more data structures (e.g., one or more databases, and / or the like) such that they may be used to conduct a transaction without directly using the original account identifier. In some examples, an original account identifier, such as a PAN, may be associated with a plurality of tokens for different individuals or purposes.

[0060] As used herein, the term “communication” may refer to the reception, receipt, transmission, transfer, provision, and / or the like of data (e.g., information, signals, messages, instructions, commands, and / or the like). For one unit (e.g., a device, a system, a component of a device or system, combinations thereof, and / or the like) to be in communication with another unit means that the one unit is able to directly or indirectly receive information from and / or transmit information to the other unit. This may refer to a direct or indirect connection (e.g., a direct communication connection, an indirect communication connection, and / or the like) that is wired and / or wireless in nature. Additionally, two units may be in communication with each other even though the information transmitted may be modified, processed, relayed, and / or routed between the first and second units. For example, a first unit may be incommunication with a second unit even though the first unit passively receives information and does not actively transmit information to the second unit. As another example, a first unit may be in communication with a second unit if at least one intermediary unit processes information received from the first unit and communicates the processed information to the second unit. In some non-limiting embodiments or aspects, a message may refer to a network packet (e.g., a data packet and / or the like) that includes data. It will be appreciated that numerous other arrangements are possible.

[0061] As used herein, the term “computing device” may refer to one or more electronic devices configured to process data. A computing device may, in some examples, include the necessary components to receive, process, and output data, such as a processor, a display, a memory, an input device, a network interface, and / or the like. A computing device may be a mobile device. As an example, a mobile device may include a cellular phone (e.g., a smartphone or standard cellular phone), a portable computer, a wearable device (e.g., watches, glasses, lenses, clothing, and / or the like), a personal digital assistant (PDA), and / or other like devices. A computing device may also be a desktop computer or other form of non-mobile computer.

[0062] As used herein, the term “server” may refer to or include one or more computing devices that are operated by or facilitate communication and processing for multiple parties in a network environment, such as the Internet, although it will be appreciated that communication may be facilitated over one or more public or private network environments and that various other arrangements are possible. Further, multiple computing devices (e.g., servers, point-of-sale (POS) devices, mobile devices, etc.) directly or indirectly communicating in the network environment may constitute a “system.”

[0063] As used herein, the term “system” may refer to one or more computing devices or combinations of computing devices (e.g., processors, servers, client devices, software applications, components of such, and / or the like). Reference to “a device,” “a server,” “a processor,” and / or the like, as used herein, may refer to a previously-recited device, server, or processor that is recited as performing a previous step or function, a different device, server, or processor, and / or a combination of devices, servers, and / or processors. For example, as used in the specification and the claims, a first device, a first server, or a first processor that is recited as performing afirst step or a first function may refer to the same or different device, server, or processor recited as performing a second step or a second function.

[0064] As used herein, the term “issuer institution” may refer to one or more entities, such as a bank, that provide accounts to customers for conducting transactions (e.g., payment transactions), such as initiating credit and / or debit payments. For example, an issuer institution may provide an account identifier, such as a PAN, to a customer that uniquely identifies one or more accounts associated with that customer. The account identifier may be embodied on a portable financial device, such as a physical financial instrument, e.g., a payment card, and / or may be electronic and used for electronic payments. The term “issuer system” refers to one or more computer devices operated by or on behalf of an issuer institution, such as a server computer executing one or more software applications. For example, an issuer system may include one or more authorization servers for authorizing a transaction.

[0065] As used herein, the term “merchant” may refer to an individual or entity that provides goods and / or services, or access to goods and / or services, to customers based on a transaction, such as a payment transaction. The term “merchant” or “merchant system” may also refer to one or more computer systems operated by or on behalf of a merchant, such as a server computer executing one or more software applications.

[0066] As used herein, the term “payment device” may refer to an electronic payment device, a portable financial device (e.g., a payment card, such as a credit or debit card), a gift card, a smartcard, smart media, a payroll card, a healthcare card, a wristband, a machine-readable medium containing account information, a keychain device or fob, a radio frequency identification (RFID) transponder, a retailer discount or loyalty card, a cellular phone, an electronic wallet mobile application, a PDA, a pager, a security card, a computing device, an access card, a wireless terminal, a transponder, and / or the like. In some non-limiting embodiments or aspects, the payment device may include volatile or non-volatile memory to store information (e.g., an account identifier, a name of the account holder, and / or the like).

[0067] As used herein, a “point-of-sale (POS) device” may refer to one or more devices, which may be used by a merchant to conduct a transaction (e.g., a payment transaction) and / or process a transaction. For example, a POS device may include one or more client devices. Additionally or alternatively, a POS device may include peripheral devices, card readers, scanning devices (e.g., code scanners), Bluetooth®communication receivers, near-field communication (NFC) receivers, RFID receivers, and / or other contactless transceivers or receivers, contact-based receivers, payment terminals, and / or the like. As used herein, a “point-of-sale (POS) system” may refer to one or more client devices and / or peripheral devices used by a merchant to conduct a transaction. For example, a POS system may include one or more POS devices and / or other like devices that may be used to conduct a payment transaction. In some non-limiting embodiments or aspects, a POS system (e.g., a merchant POS system) may include one or more server computers configured to process online payment transactions through webpages, mobile applications, and / or the like.

[0068] As used herein, the term “transaction service provider” may refer to an entity that receives transaction authorization requests from merchants or other entities and provides guarantees of payment, in some cases through an agreement between the transaction service provider and an issuer institution. For example, a transaction service provider may include a payment network such as Visa® or any other entity that processes transactions. The term “transaction processing system” may refer to one or more computer systems operated by or on behalf of a transaction service provider, such as a transaction processing server executing one or more software applications. A transaction processing server may include one or more processors and, in some non-limiting embodiments or aspects, may be operated by or on behalf of a transaction service provider.

[0069] Non-limiting embodiments or aspects of the present disclosure are directed to systems, methods, and computer program products for operating a pruned transformer machine learning model architecture. In some non-limiting embodiments or aspects, a model management system may receive initial data associated with sequence information, mask one or more data instances of the initial data associated with the sequence information based on an output of a masking machine learning model to provide masked intermediate input data, provide the masked intermediate input data as an input to a classification machine learning model to generate a classification label of the masked intermediate input data, wherein the classification machine learning model is transformer machine learning model configured to provide a prediction of a classification label of an input as an output, and perform an action based on the classification label of the masked intermediate input data. In some nonlimiting embodiments or aspects, when masking the one or more data instances of the initial data associated with the sequence information, the model management systemmay provide the initial data as an input to the masking machine learning model to generate an output, wherein the output of the masking machine learning model comprises a prediction of a binary classification of each data instance of the initial data, and assign a negative binary classification to the one or more data instances of the initial data provided as the input to the masking machine learning model based on the prediction of a binary classification of the one or more data instances of the initial data.

[0070] In some non-limiting embodiments or aspects, when masking the one or more data instances of the initial data associated with the sequence information, the model management system may provide the initial data as an input to the masking machine learning model to generate an output, wherein the output of the masking machine learning model comprises a prediction of a binary classification of each data instance of the initial data and remove a value of the one or more data instances of the initial data provided as the input to the masking machine learning model from an embedding that is to be provided as an input to the classification machine learning model based on the prediction of a binary classification of the one or more data instances of the initial data. In some non-limiting embodiments or aspects, the model management system may train the masking machine learning model and the classification machine learning model based on a loss function. In some non-limiting embodiments or aspects, the loss function comprises the formula:

[0072] In some non-limiting embodiments or aspects, the initial data associated with sequence information comprises a plurality of tokens, and the model management system may tokenize discrete instances of the sequence information to provide the plurality of tokens of the initial data associated with sequence information. In some non-limiting embodiments or aspects, when performing the action, the model management system may execute a fraud protection procedure based on the classification label of the masked intermediate input data.

[0073] In this way, the model management system may provide for a pruned transformer machine learning model architecture that can be used for classification applications, such as fraud protection (e.g., fraud detection and / or fraud prevention), which reduces the requirements for computational resources as compared to a transformer machine learning model without a pruned self-attention mechanism. Inaddition, the model management system may provide for a pruned transformer machine learning model architecture that is able to handle high-dimensional inputs and can provide accurate results in a reduced amount of time.

[0074] Referring now to FIG. 1 , FIG. 1 is a diagram of example system 100 in which devices, systems, and / or methods, described herein, may be implemented. As shown in FIG. 1 , system 100 includes model management system 102, machine learning (ML) model management database 104, user device 106, and communication network 108. Model management system 102, ML model management database 104, and / or user device 106 may interconnect (e.g., establish a connection to communicate) via wired connections, wireless connections, or a combination of wired and wireless connections.

[0075] Model management system 102 may include one or more devices capable of receiving information from and / or communicating information (e.g., directly via wired or wireless communication connection, indirectly via communication network 108, and / or the like) to ML model management database 104 and / or user device 106 via communication network 108. For example, model management system 102 may include a server, a group of servers, a cloud platform, and / or other like devices. In some non-limiting embodiments or aspects, model management system 102 may be associated with a transaction service provider system. For example, model management system 102 may be operated by a transaction service provider system. In another example, model management system 102 may be a component of user device 106. In another example, model management system 102 may include ML model management database 104. In some non-limiting embodiments or aspects, model management system 102 may be in communication with a data storage device (e.g., ML model management database 104), which may be local or remote to model management system 102. In some non-limiting embodiments or aspects, model management system 102 may be capable of receiving information from, storing information in, transmitting information to, and / or searching information stored in the data storage device.

[0076] Additionally or alternatively, model management system 102 may generate (e.g., train, validate, re-train, and / or the like), store, and / or implement (e.g., operate, provide inputs to and / or outputs from, and / or the like) one or more machine learning models. For example, model management system 102 may generate one or more machine learning models by fitting (e.g., validating) one or more machine learningmodels against data used for training (e.g., training data). In some non-limiting embodiments or aspects, model management system 102 may generate, store, and / or implement one or more machine learning models that are provided for a production environment (e.g., a real-time or runtime environment used for providing inferences based on data in a live situation). In some non-limiting embodiments or aspects, model management system 102 may be in communication with a data storage device, which may be local or remote to model management system 102. In some non-limiting embodiments or aspects, model management system 102 may be capable of receiving information from, storing information in, transmitting information to, and / or searching information stored in the data storage device.

[0077] ML model management database 104 may include one or more devices capable of receiving information from and / or communicating information (e.g., directly via wired or wireless communication connection, indirectly via communication network 108, and / or the like) to model management system 102 and / or user device 106. For example, ML model management database 104 may include a server, a group of servers, a desktop computer, a portable computer, a mobile device, and / or other like devices. In some non-limiting embodiments or aspects, ML model management database 104 may include a data storage device. In some non-limiting embodiments or aspects, ML model management database 104 may be capable of receiving information from, storing information in, communicating information to, or searching information stored in the data storage device. In some non-limiting embodiments or aspects, ML model management database 104 may be part of model management system 102 and / or part of the same system as model management system 102.

[0078] User device 106 may include one or more devices capable of receiving information from and / or communicating information (e.g., directly via wired or wireless communication connection, indirectly via communication network 108, and / or the like) to model management system 102 and / or ML model management database 104. For example, user device 106 may include a computing device, such as a mobile device, a portable computer, a desktop computer, and / or other like devices. Additionally or alternatively, user device 106 may include a device capable of receiving information from and / or communicating information to other user devices (e.g., directly via wired or wireless communication connection, indirectly via communication network 108, and / or the like). In some non-limiting embodiments or aspects, user device 106 may be part of model management system 102 and / or part of the same system as modelmanagement system 102. For example, model management system 102, ML model management database 104, and user device 106 may all be (and / or be part of) a single system and / or a single computing device.

[0079] Communication network 108 may include one or more wired and / or wireless networks. For example, communication network 108 may include a cellular network (e.g., a long-term evolution (LTE) network, a third-generation (3G) network, a fourthgeneration (4G) network, a fifth-generation (5G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., the public switched telephone network (PSTN) and / or the like), a private network, an ad hoc network, an intranet, the Internet, a fiber optic-based network, a cloud computing network, and / or the like, and / or a combination of some or all of these or other types of networks.

[0080] The number and arrangement of systems and devices shown in FIG. 1 are provided as an example. There may be additional systems and / or devices, fewer systems and / or devices, different systems and / or devices, and / or differently arranged systems and / or devices than those shown in FIG. 1. Furthermore, two or more systems or devices shown in FIG. 1 may be implemented within a single system or device, or a single system or device shown in FIG. 1 may be implemented as multiple, distributed systems or devices. Additionally or alternatively, a set of systems (e.g., one or more systems) or a set of devices (e.g., one or more devices) of system 100 may perform one or more functions described as being performed by another set of systems or another set of devices of system 100.

[0081] Referring now to FIG. 2, shown is a flow diagram for process 200 for operating a pruned transformer machine learning model architecture, according to some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, one or more of the steps of process 200 may be performed (e.g., completely, partially, etc.) by model management system 102 (e.g., one or more devices of model management system 102). In some non-limiting embodiments or aspects, one or more of the steps of process 200 may be performed (e.g., completely, partially, etc.) by another device or a group of devices separate from or including model management system 102 (e.g., one or more devices of model management system 102), ML model management database 104, and / or user device 106. The steps shown in FIG. 2 are for example purposes only. It will be appreciated that additional, fewer, different,and / or a different order of steps may be used in some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, a step may be automatically performed in response to performance and / or completion of a prior step.

[0082] As shown in FIG. 2, at step 202, process 200 includes receiving initial data associated with sequence information. For example, model management system 102 may receive the initial data associated with sequence information (e.g., information regarding a sequence of data instances). In some non-limiting embodiments or aspects, model management system 102 may receive the initial data from ML model management database 104 and / or user device 106. In some non-limiting embodiments or aspects, model management system 102 may receive the initial data from a system or device other than or in addition to ML model management database 104 and / or user device 106, such as from a merchant system, an issuer system, an acquirer system, and / or the like.

[0083] In some non-limiting embodiments or aspects, model management system 102 may receive the initial data associated with (e.g., included in) a request for inference for a production machine learning model and perform a feature engineering procedure on the initial data to produce an input to the production machine learning model. In some non-limiting embodiments or aspects, the initial data may be associated with a task for which a production machine learning model may provide an inference. In some non-limiting embodiments or aspects, the initial data may be associated with financial service tasks. For example, the initial data may be associated with a token service task, an authentication task (e.g., a 3D secure authentication task), a fraud detection task, a fraud prevention task, and / or the like. In some non-limiting embodiments or aspects, a production machine learning model may include a machine learning model that has been trained and / or validated (e.g., tested) and that may be used to generate inferences (e.g., prediction), such as real-time inferences, runtime inferences, and / or the like.

[0084] In some non-limiting embodiments or aspects, the initial data associated with sequence information may include a plurality of data instances of a sequence. For example, the initial data associated with sequence information may include a plurality of data instances of a sequence of actions carried out in order. In some nonlimiting embodiments or aspects, the plurality of data instances may include a plurality of transactions (e.g., payments transaction) arranged in order based on a time at which the transactions were conducted. In some non-limiting embodiments or aspects, theinitial data associated with sequence information may include a plurality of tokens (e.g., a plurality of references, such as identifiers, that map back to data instances through a tokenization system), where the plurality of tokens represents a plurality of data instances of a sequence associated with the sequence information. In some nonlimiting embodiments or aspects, model management system 102 may tokenize discrete instances of the sequence information (e.g., discrete data instances of a sequence associated with the sequence information) to provide the plurality of tokens of the initial data associated with sequence information. In some non-limiting embodiments or aspects, model management system 102 may tokenize discrete instances of the sequence information to provide a plurality of tokens that are formatted to be provided as inputs to a classification machine learning model. In some examples, the plurality of data instances may include a large amount of data instances, such as 100 data instances, 500 data instances, 1 ,000 data instances, 5,000 data instances, 10,000 data instances, 25,000 data instances, 50,000 data instances, 100,000 data instances, 1 ,000,000 data instances, and / or the like

[0085] In some non-limiting embodiments or aspects, model management system 102 may generate the initial data associated with the sequence information. For example, model management system 102 may generate the initial data that includes one or more sequences of feature values (e.g., one or more sequences of feature values over a time period) of a feature from a dataset that includes a plurality of feature values of a plurality of features. In some non-limiting embodiments or aspects, a sequence of feature values may include a plurality of values of a first feature over a time period that are provided in reference to a specific value of a second feature. For example, a sequence of feature values may include a sequence of values of transaction volume of payment transactions over a time period in reference to a specific value of transaction amount of the payment transactions.

[0086] In some non-limiting embodiments or aspects, the initial data may include runtime input data. In some non-limiting embodiments or aspects, the runtime input data may include a sample of data that is received by a trained machine learning model in real-time (e.g., a time at which or close to a time at which operations, such as operations of model management system 102, are carried out) with respect to the runtime input data being generated. For example, runtime input data may be generated by a data source (e.g., a customer performing a transaction) and may be subsequently received by the trained machine learning model in real-time. Runtime(e.g., production) may refer to inputting runtime data (e.g., a runtime dataset, real- world data, real-world observations, and / or the like) into one or more trained machine learning models (e.g., one or more trained machine learning models of model management system 102) and / or generating an inference (e.g., generating an inference using model management system 102 or another machine learning system). In some non-limiting embodiments or aspects, runtime may be performed during a phase that may occur after a training phase, after a testing phase, and / or after deployment of the machine learning model into a production environment. During a time period associated with the runtime phase, the machine learning model (e.g., a production machine learning model) may process the runtime input data to generate inferences (e.g., real-time inferences, real-time predictions, and / or the like).

[0087] As shown in FIG. 2, at step 204, process 200 includes masking one or more data instances of the initial data associated with the sequence information. For example, model management system 102 may mask one or more data instances (e.g., one or more tokens) of the initial data associated with the sequence information. In some non-limiting embodiments or aspects, model management system 102 may mask the one or more data instances of the initial data associated with the sequence information based on an output of a masking machine learning model to provide masked intermediate input data. In some non-limiting embodiments or aspects, the masked intermediate input data may include a sequence of data instances with one or more data instances (e.g., one or more tokens) that are masked (e.g., one or more masked data instances, one or more masked tokens, etc.). In some non-limiting embodiments or aspects, a masked data instance may include a data instance having an assigned binary classification (e.g., a negative classification or a positive classification). In some non-limiting embodiments or aspects, a masked data instance may include a data instance that has been removed from the sequence or a data instance having a value that has been removed. In some non-limiting embodiments or aspects, a masked data instance may include a value of an embedding that has been removed or that has an assigned binary classification.

[0088] In some non-limiting embodiments or aspects, model management system 102 may provide the initial data as an input to the masking machine learning model to generate an output. For example, model management system 102 may provide the one or more data instances of the initial data as an input to the masking machine learning model based on receiving the initial data. In some non-limiting embodimentsor aspects, the output of the masking machine learning model may include a prediction of a binary classification of each data instance of the initial data. In some non-limiting embodiments or aspects, the prediction of a binary classification of a data instance of the initial data may include a prediction as to whether the data instance of the initial data may be relevant to determine a classification of the initial data (e.g., a classification of that which is represented by the initial data, such as a sequence represented by one or more data instances of the initial data) by a classification machine learning model. In some non-limiting embodiments or aspects, the output of the masking machine learning model may include a negative classification (e.g., a numerical zero, such as a “0”) or a positive classification (e.g., a numerical one, such as a “1 ”) as part of a binary classification. In some non-limiting embodiments or aspects, a negative classification may indicate that a data instance is not relevant (e.g., does not satisfy a threshold of relevancy) to determine a classification of the initial data by a classification machine learning model. In some non-limiting embodiments or aspects, a positive classification may indicate that a data instance is relevant (e.g., satisfies a threshold of relevancy) to determine a classification of the initial data by a classification machine learning model.

[0089] In some non-limiting embodiments or aspects, model management system 102 may generate the masking machine learning model. For example, model management system 102 may generate the masking machine learning model to provide a prediction (e.g., a value of a prediction, such as a percentage indicating a degree of confidence of the prediction) as to whether a data instance of a sequence of data instances may be relevant to determine a classification of the sequence of data instances by a classification machine learning model based on a training dataset.

[0090] In some non-limiting embodiments or aspects, the masking machine learning model may include a machine learning model designed to receive, as an input, data associated with a sequence of data instances and provide, as an output, a prediction as to whether a data instance of a grouping of data instances may be relevant to determine a classification of the grouping of data instances by a classification machine learning model. For example, the masking machine learning model may be designed to receive one or more data instances of a sequence of payment transactions and provide an output that includes a prediction as to whether a data instance of the sequence of data instances may be relevant to determine a classification regarding potential fraudulent activity of the sequence of paymenttransactions by a classification machine learning model. In some non-limiting embodiments or aspects, model management system 102 may store the masking machine learning model (e.g., for later use).

[0091] In some non-limiting embodiments or aspects, model management system 102 may process data associated with a sequence of data instances to obtain training data (e.g., a training dataset) for the masking machine learning model. For example, model management system 102 may process the data to change the data into a format that may be analyzed (e.g., by model management system 102) to generate the masking machine learning model. The data that is changed (e.g., the data that results from the change) may be referred to as training data. In some non-limiting embodiments or aspects, model management system 102 may process the data associated with a sequence of data instances to obtain the training data based on receiving the data. Additionally or alternatively, model management system 102 may process the data to obtain the training data based on model management system 102 receiving an indication, from a user (e.g., a user associated with user device 106) of model management system 102, that model management system 102 is to process the data, such as when model management system 102 receives an indication to generate a masking machine learning model for a time interval corresponding to the data.

[0092] In some non-limiting embodiments or aspects, model management system 102 may process data associated with a sequence of data instances based on receiving the data. A masking prediction variable may include a metric, associated with relevancy to determine a classification of a data instance of a sequence of data instances by a classification machine learning model, which may be derived based on the data associated with a sequence of data instances. The masking prediction variable may be analyzed to generate a masking machine learning model. For example, the masking prediction variable may include a variable associated with a position of a data instance in a sequence of data instances, a variable associated with neighboring data instances of a data instance in a sequence of data instances, a variable associated with a frequency of a value of a data instance in a sequence of data instances, a variable associated with a frequency of a classification of sequences of data instances, and / or the like.

[0093] In some non-limiting embodiments or aspects, model management system 102 may analyze the training data to generate the masking machine learning model.For example, model management system 102 may use machine learning techniques to analyze the training data to generate the masking machine learning model. In some non-limiting embodiments or aspects, generating the masking machine learning model (e.g., based on the training data obtained from historical data associated with data instances of a sequence of payment transactions) may be referred to as training the masking machine learning model. The machine learning techniques may include, for example, supervised and / or unsupervised techniques, such as decision trees, random forests, logistic regressions, linear regression, gradient boosting, support-vector machines, extra-trees (e.g., an extension of random forests), Bayesian statistics, learning automata, Hidden Markov Modeling, linear classifiers, quadratic classifiers, association rule learning, and / or the like. In some non-limiting embodiments or aspects, the masking machine learning model may include a model that is specific to a particular characteristic, for example, a model that is specific to a particular individual (e.g., an individual account holder), a particular issuer, a particular time interval during which payment transactions are conducted, and / or the like. Additionally or alternatively, the masking machine learning model may be specific to a particular entity (e.g., a transaction service provider) that makes determinations of fraudulent activity with regard to payment transactions. In some non-limiting embodiments or aspects, model management system 102 may generate one or more masking machine learning models for one or more entities, a particular group of entities, one or more users of one or more entities, and / or the like.

[0094] Additionally or alternatively, when analyzing the training data, model management system 102 may identify one or more variables (e.g., one or more independent variables) as predictor variables (e.g., features) that may be used to make a prediction when analyzing the training data. In some non-limiting embodiments or aspects, values of the predictor variables may be inputs to the masking machine learning model. For example, model management system 102 may identify a subset (e.g., a proper subset) of the variables as the predictor variables that may be used to accurately predict whether a data instance of a sequence of data instances may be relevant to determine a classification of the sequence by a classification machine learning model. In some non-limiting embodiments or aspects, the predictor variables may include one or more of the masking prediction variables, as discussed above, that have a significant impact (e.g., an impact satisfying a threshold) on whether a data instance of a sequence is relevant to determine a classification of the sequence by aclassification machine learning model as determined by model management system 102.

[0095] In some non-limiting embodiments or aspects, model management system 102 may validate the masking machine learning model. For example, model management system 102 may validate the masking machine learning model after model management system 102 generates the masking machine learning model. In some non-limiting embodiments or aspects, model management system 102 may validate the masking machine learning model based on a portion of the training data to be used for validation. For example, model management system 102 may partition the training data into a first portion and a second portion, where the first portion may be used to generate the masking machine learning model, as described above. In this example, the second portion of the training data (e.g., the validation data) may be used to validate the masking machine learning model.

[0096] In some non-limiting embodiments or aspects, model management system 102 may validate the masking machine learning model by providing validation data associated with a user (e.g., data associated with one or more data instances of a sequence of data instances) as input to the masking machine learning model, and determining, based on an output of the masking machine learning model, whether the masking machine learning model correctly, or incorrectly, predicted whether a data instance of a sequence is relevant to determine a classification of the sequence by a classification machine learning model. In some non-limiting embodiments or aspects, model management system 102 may validate the masking machine learning model based on a validation threshold. For example, model management system 102 may be configured to validate the masking machine learning model when the prediction of whether a data instance of a sequence is relevant to determine a classification of the sequence by a classification machine learning model (as identified by the validation data) and are correctly predicted by the masking machine learning model (e.g., when the masking machine learning model correctly predicts 50% of the predictions of data instances, 70% of the predictions of data instances, a threshold quantity of the predictions of data instances, and / or the like).

[0097] In some non-limiting embodiments or aspects, if model management system 102 does not validate the masking machine learning model (e.g., when a percentage of correct predictions of data instances does not satisfy the validationthreshold), then model management system 102 may generate one or more additional masking machine learning models.

[0098] In some non-limiting embodiments or aspects, once the masking machine learning model has been validated, model management system 102 may further train the masking machine learning model and / or generate new masking machine learning models based on receiving new training data. The new training data may include additional data associated with one or more shipments of freight. In some non-limiting embodiments or aspects, the new training data may include data associated with one or more data instances of another sequence of data instances. Model management system 102 may use the masking machine learning model to predict whether a data instance of the other sequence is relevant to determine a classification of the sequence by a classification machine learning model and compare an output of masking machine learning models to the new training data that includes data associated with one or more data instances of the other sequence of data instances. In such an example, model management system 102 may update one or more masking machine learning models based on the new training data.

[0099] In some non-limiting embodiments or aspects, model management system 102 may store the masking machine learning model. For example, model management system 102 may store the masking machine learning model in a data structure (e.g., a database, a linked list, a tree, and / or the like). The data structure may be located within model management system 102 or external (e.g., remote from) model management system 102.

[0100] In some non-limiting embodiments or aspects, model management system 102 may generate a classification machine learning model in the same or similar fashion as the masking machine learning model. In some non-limiting embodiments or aspects, model management system 102 may train the masking machine learning model and / or the classification machine learning model based on a loss function. In some non-limiting embodiments or aspects, the loss function may take into account gradients for the masking machine learning model and the classification machine learning model. In some non-limiting embodiments or aspects, the loss function may include the formula:

[0102] where the loss of the masking machine learning model and the classification machine learning model, shown as Loss, may be based on the loss of classification machine learning model, shown as Lcis, and a regularization term, shown as a * , maskXi N, which may include a constant term multiplied by a sum of binary classifications (e.g., N binary classifications) applied to the one or more data instances (e.g., a sum of “1 ” or “0” terms applied to the one or more data instances based on the masking machine learning model). In some non-limiting embodiments or aspects, the loss of classification machine learning model is based on the loss at the output of the classification machine learning model (e.g., the loss associated with the labels, classifications, etc., provided by the classification machine learning model).

[0103] In some non-limiting embodiments or aspects, model management system 102 may assign a binary classification to the one or more data instances of the initial data based on the output (e.g., based on the prediction of a binary classification of the one or more data instances provided as an input) of the masking machine learning model. For example, model management system 102 may assign a negative classification or a positive classification to the one or more data instances of the initial data that is to be provided as an input to the classification machine learning model based on the output of the masking machine learning model.

[0104] In some non-limiting embodiments or aspects, model management system 102 may remove a value of the one or more data instances of the initial data (e.g., a value of the one or more data instances of the initial data that was previously provided as the input to the masking machine learning model) that is to be provided as an input to the classification machine learning model. For example, model management system 102 may remove the value of the one or more data instances of the initial data based on the prediction of a binary classification of the one or more data instances of the initial data. In one example, if model management system 102 determines that a prediction of a binary classification of a data instance of the initial data is a negative classification, model management system 102 may remove the value of the data instance of the initial data so that a zero value or a null value (e.g., a blank value, an empty value, etc.) is to be provided as an input to the classification machine learning model for the data instance. In this way, model management system 102 may reduce the amount of network resources required for operation of the classification machine learning model. In such an example, if model management system 102 determinesthat a prediction of a binary classification of a data instance of the initial data is a positive classification, model management system 102 may forego removing the value of the data instance of the initial data so that a value (e.g., an actual value) of the data instance of the initial data is to be provided as an input to the classification machine learning model for the data instance.

[0105] In some non-limiting embodiments or aspects, model management system 102 may remove the value of the one or more data instances of the initial data from an embedding. For example, model management system 102 may remove the value of the one or more data instances of the initial data from an embedding that is to be provided as an input to the classification machine learning model based on the prediction of a binary classification of the one or more data instances of the initial data.

[0106] As shown in FIG. 2, at step 206, process 200 includes generating a classification label of the masked intermediate input data. For example, model management system 102 may generate the classification label of the masked intermediate input data.

[0107] In some non-limiting embodiments or aspects, model management system 102 may provide the masked intermediate input data as an input to a classification machine learning model and obtain an output based on the input. In some non-limiting embodiments or aspects, the classification machine learning model is a transformer machine learning model configured to provide a prediction of a classification label of an input as an output. In some non-limiting embodiments or aspects, the transformer machine learning model comprises a plurality of transformer layers. Each transformer layer may include a multi-head self-attention (MSA) network, a single-head selfattention (SSA) network, a position-wise feed-forward network (FFN), or any combination thereof. In some non-limiting embodiments or aspects, the output of the classification machine learning model may be based on one or more masked data instances of the masked intermediate input data. For example, the output of the classification machine learning model may be based on the one or more masked data instances of the masked intermediate input data that are giving little or no weight with regard to determining the classification label of the masked intermediate input data. In this way, a self-attention mechanism of the transformer machine learning model of the classification machine learning model may operate as a pruned self-attention mechanism, such that the self-attention connections associated with the one or moremasked data instances of the masked intermediate input data will provide little or no effect on the output of the classification machine learning model.

[0108] As shown in FIG. 2, at step 208, process 200 includes performing an action based on the classification label of the masked intermediate input data. For example, model management system 102 may perform an action based on the classification label of the masked intermediate input data. In some non-limiting embodiments or aspects, model management system 102 may perform a procedure associated with protection of an account of a user (e.g., a user associated with user device 106) based on the classification label of the masked intermediate input data. For example, if the classification label of the masked intermediate input data indicates that the procedure is necessary, model management system 102 may perform the procedure associated with protection of the account of the user. In such an example, if the classification label of the masked intermediate input data indicates that the procedure is not necessary, model management system 102 may forego performing the procedure associated with protection of the account of the user. In some non-limiting embodiments or aspects, model management system 102 may execute a fraud protection procedure based on the classification label of the masked intermediate input data.

[0109] In some non-limiting embodiments or aspects, model management system 102 may perform the action by providing the output of the classification machine learning model as a response to a request for inference for the classification machine learning model. In some non-limiting embodiments or aspects, model management system 102 may perform the action by generating and transmitting an alert (e.g., an alert message) based on the classification label of the masked intermediate input data. For example, model management system 102 may perform the action by generating and transmitting an alert to user device 106 (e.g., a user associated with user device 106, such as a subject matter expert).

[0110] Referring now to FIGS. 3A-3E, shown are schematic diagrams of implementation 300 of a process (e.g., process 200) for operating a pruned transformer machine learning model architecture. In some non-limiting embodiments or aspects, one or more of the steps of the process may be performed (e.g., completely, partially, etc.) by model management system 102 (e.g., one or more devices of model management system 102). In some non-limiting embodiments or aspects, one or more of the steps of the process may be performed (e.g., completely,partially, etc.) by another device or a group of devices separate from or including model management system 102 (e.g., one or more devices of model management system 102), ML model management database 104, and / or user device 106.

[0111] As shown by reference number 305 in FIG. 3A, model management system 102 may receive a dataset that includes a plurality of data instances. As further shown by reference number 310 in FIG. 3A, model management system 102 may generate initial data associated with sequence information from the plurality of data instances of the dataset.

[0112] As shown by reference number 315 in FIG. 3B, model management system 102 may tokenize discrete instances (e.g., each data instance of the plurality of data instances) of the sequence information to provide a plurality of tokens regarding the initial data associated with the sequence information. In some non-limiting embodiments or aspects, model management system 102 may tokenize the discrete instances via a tokenization layer. In some non-limiting embodiments or aspects, the tokenization layer may be a layer of a classification machine learning model (e.g., a transformer machine learning model).

[0113] As shown by reference number 320 in FIG. 3C, model management system 102 may mask one or more tokens to provide masked intermediate input data. In some non-limiting embodiments or aspects, model management system 102 may mask one or more tokens of the plurality of tokens based on an output of a masking machine learning model to provide masked intermediate input data. In some nonlimiting embodiments or aspects, the output of the masking machine learning model may include a mask that allows for assigning a negative binary classification to the one or more tokens (e.g., one or more data instances of the initial data) which may be provided as the input to the masking machine learning model. Alternatively, the output of the masking machine learning model may include a mask that allows for removing a value of the one or more tokens (e.g., one or more data instances of the initial data) which may be provided as the input to the masking machine learning model.

[0114] In some non-limiting embodiments or aspects, the initial data may include one or more data instances (e.g., from an embedding) that are to be provided as an input to the classification machine learning model. In some non-limiting embodiments or aspects, the mask may be based on a prediction of a binary classification of the one or more data instances of the initial data.

[0115] As shown by reference number 325 in FIG. 3D, model management system 102 may generate a classification label for the masked intermediate input data. In some non-limiting embodiments or aspects, model management system 102 may provide the masked intermediate input data as an input to a classification machine learning model and obtain an output based on the input. In some non-limiting embodiments or aspects, the classification machine learning model is a transformer machine learning model configured to provide a prediction of a classification label of an input as an output. In some non-limiting embodiments or aspects, the output of the classification machine learning model may be based on one or more masked data instances of the masked intermediate input data. For example, the output of the classification machine learning model may be based on the one or more masked data instances of the masked intermediate input data that are giving little or no weight with regard to determining the classification label of the masked intermediate input data. In this way, a self-attention mechanism of the transformer machine learning model of the classification machine learning model may operate as a pruned self-attention mechanism, such that the self-attention connections associated with the one or more masked data instances of the masked intermediate input data will provide little or no effect on the output of the classification machine learning model.

[0116] As shown by reference number 330 in FIG. 3E, model management system 102 may perform an action based on the classification (e.g., the output of the classification machine learning model) of the masked intermediate input data. For example, model management system 102 may perform an action based on the classification label of the masked intermediate input data. In some non-limiting embodiments or aspects, model management system 102 may perform a procedure associated with protection of an account of a user (e.g., a user associated with user device 106) based on the classification label of the masked intermediate input data. For example, if the classification label of the masked intermediate input data indicates that the procedure is necessary, model management system 102 may perform the procedure associated with protection of the account of the user. In such an example, if the classification label of the masked intermediate input data indicates that the procedure is not necessary, model management system 102 may forego performing the procedure associated with protection of the account of the user. In some nonlimiting embodiments or aspects, model management system 102 may execute afraud protection procedure based on the classification label of the masked intermediate input data.

[0117] Referring now to FIG. 4, shown is a diagram of a non-limiting embodiment or aspect of exemplary environment 400 in which methods, systems, and / or products, as described herein, may be implemented. As shown in FIG. 4, environment 400 may include transaction service provider system 402, issuer system 404, customer device 406, merchant system 408, acquirer system 410, and communication network 412. In some non-limiting embodiments or aspects, each of model management system 102, ML model management database 104, and / or user device 106 of FIG. 1 may be implemented by (e.g., part of) transaction service provider system 402. In some nonlimiting embodiments or aspects, at least one of model management system 102, ML model management database 104, and / or user device 106 of FIG. 1 may be implemented by (e.g., part of) another system, another device, another group of systems, or another group of devices, separate from or including transaction service provider system 402, such as issuer system 404, customer device 406, merchant system 408, acquirer system 410, and / or the like.

[0118] Transaction service provider system 402 may include one or more devices capable of receiving information from and / or communicating information to issuer system 404, customer device 406, merchant system 408, and / or acquirer system 410 via communication network 412. For example, transaction service provider system 402 may include a computing device, such as a server (e.g., a transaction processing server), a group of servers, and / or other like devices. In some non-limiting embodiments or aspects, transaction service provider system 402 may be associated with a transaction service provider, as described herein. In some non-limiting embodiments or aspects, transaction service provider system 402 may be in communication with a data storage device, which may be local or remote to transaction service provider system 402. In some non-limiting embodiments or aspects, transaction service provider system 402 may be capable of receiving information from, storing information in, communicating information to, or searching information stored in the data storage device.

[0119] Issuer system 404 may include one or more devices capable of receiving information and / or communicating information to transaction service provider system 402, customer device 406, merchant system 408, and / or acquirer system 410 via communication network 412. For example, issuer system 404 may include acomputing device, such as a server, a group of servers, and / or other like devices. In some non-limiting embodiments or aspects, issuer system 404 may be associated with an issuer institution, as described herein. For example, issuer system 404 may be associated with an issuer institution that issued a credit account, debit account, credit card, debit card, and / or the like to a user associated with customer device 406.

[0120] Customer device 406 may include one or more devices capable of receiving information from and / or communicating information to transaction service provider system 402, issuer system 404, merchant system 408, and / or acquirer system 410 via communication network 412. Additionally or alternatively, each customer device 406 may include a device capable of receiving information from and / or communicating information to other customer devices 406 via communication network 412, another network (e.g., an ad hoc network, a local network, a private network, a virtual private network, and / or the like), and / or any other suitable communication technique. For example, customer device 406 may include a client device and / or the like. In some non-limiting embodiments or aspects, customer device 406 may or may not be capable of receiving information (e.g., from merchant system 408 or from another customer device 406) via a short-range wireless communication connection (e.g., an NFC communication connection, an RFID communication connection, a Bluetooth® communication connection, a Zigbee® communication connection, and / or the like), and / or communicating information (e.g., to merchant system 408) via a short-range wireless communication connection.

[0121] Merchant system 408 may include one or more devices capable of receiving information from and / or communicating information to transaction service provider system 402, issuer system 404, customer device 406, and / or acquirer system 410 via communication network 412. Merchant system 408 may also include a device capable of receiving information from customer device 406 via communication network 412, a communication connection (e.g., an NFC communication connection, an RFID communication connection, a Bluetooth® communication connection, a Zigbee® communication connection, and / or the like) with customer device 406, and / or the like, and / or communicating information to customer device 406 via communication network 412, the communication connection, and / or the like. In some non-limiting embodiments or aspects, merchant system 408 may include a computing device, such as a server, a group of servers, a client device, a group of client devices, and / or other like devices. In some non-limiting embodiments or aspects, merchant system 408 maybe associated with a merchant, as described herein. In some non-limiting embodiments or aspects, merchant system 408 may include one or more client devices. For example, merchant system 408 may include a client device that allows a merchant to communicate information to transaction service provider system 402. In some non-limiting embodiments or aspects, merchant system 408 may include one or more devices, such as computers, computer systems, and / or peripheral devices capable of being used by a merchant to conduct a transaction with a user. For example, merchant system 408 may include a POS device and / or a POS system.

[0122] Acquirer system 410 may include one or more devices capable of receiving information from and / or communicating information to transaction service provider system 402, issuer system 404, customer device 406, and / or merchant system 408 via communication network 412. For example, acquirer system 410 may include a computing device, a server, a group of servers, and / or the like. In some non-limiting embodiments or aspects, acquirer system 410 may be associated with an acquirer, as described herein.

[0123] Communication network 412 may include one or more wired and / or wireless networks. For example, communication network 412 may include a cellular network (e.g., a long-term evolution (LTE) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, and / or the like), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., the public switched telephone network (PSTN)), a private network (e.g., a private network associated with a transaction service provider), an ad hoc network, an intranet, the Internet, a fiber optic-based network, a cloud computing network, and / or the like, and / or a combination of these or other types of networks.

[0124] The number and arrangement of systems, devices, and / or networks shown in FIG. 4 are provided as an example. There may be additional systems, devices, and / or networks; fewer systems, devices, and / or networks; different systems, devices, and / or networks; and / or differently arranged systems, devices, and / or networks than those shown in FIG. 4. Furthermore, two or more systems or devices shown in FIG. 4 may be implemented within a single system or device, or a single system or device shown in FIG. 4 may be implemented as multiple, distributed systems or devices. Additionally or alternatively, a set of systems (e.g., one or more systems) or a set of devices (e.g., one or more devices) of environment 400 may perform one or morefunctions described as being performed by another set of systems or another set of devices of environment 400.

[0125] Referring now to FIG. 5, shown is a diagram of example components of device 500, according to non-limiting embodiments or aspects. Device 500 may correspond to at least one of model management system 102, ML model management database 104, and / or user device 106 in FIG. 1 and / or at least one of transaction service provider system 402, issuer system 404, customer device 406, merchant system 408, and / or acquirer system 410 in FIG. 4, as an example. In some non-limiting embodiments or aspects, such systems or devices in FIG. 1 or FIG. 4 may include at least one device 500 and / or at least one component of device 500. The number and arrangement of components shown in FIG. 5 are provided as an example. In some non-limiting embodiments or aspects, device 500 may include additional components, fewer components, different components, or differently arranged components than those shown in FIG. 5. Additionally or alternatively, a set of components (e.g., one or more components) of device 500 may perform one or more functions described as being performed by another set of components of device 500.

[0126] As shown in FIG. 5, device 500 may include bus 502, processor 504, memory 506, storage component 508, input component 510, output component 512, and communication interface 514. Bus 502 may include a component that permits communication among the components of device 500. In some non-limiting embodiments or aspects, processor 504 may be implemented in hardware, firmware, or a combination of hardware and software. For example, processor 504 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component (e.g., a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.) that can be programmed to perform a function. Memory 506 may include random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by processor 504.

[0127] With continued reference to FIG. 5, storage component 508 may store information and / or software related to the operation and use of device 500. For example, storage component 508 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, a solid-state disk, etc.) and / or another type ofcomputer-readable medium. Input component 510 may include a component that permits device 500 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). Additionally or alternatively, input component 510 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.). Output component 512 may include a component that provides output information from device 500 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.). Communication interface 514 may include a transceiver-like component (e.g., a transceiver, a separate receiver and transmitter, etc.) that enables device 500 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 514 may permit device 500 to receive information from another device and / or provide information to another device. For example, communication interface 514 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a cellular network interface, and / or the like.

[0128] Device 500 may perform one or more processes described herein. Device 500 may perform these processes based on processor 504 executing software instructions stored by a computer-readable medium, such as memory 506 and / or storage component 508. A computer-readable medium may include any non-transitory memory device. A memory device includes memory space located inside of a single physical storage device or memory space spread across multiple physical storage devices. Software instructions may be read into memory 506 and / or storage component 508 from another computer-readable medium or from another device via communication interface 514. When executed, software instructions stored in memory 506 and / or storage component 508 may cause processor 504 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, embodiments described herein are not limited to any specific combination of hardware circuitry and software. The term “configured to,” as used herein, may refer to an arrangement of software, device(s), and / or hardware for performing and / or enabling one or more functions (e.g., actions, processes, steps of a process, and / or the like). For example, “a processor configured to” may refer to aprocessor that executes software instructions (e.g., program code) that cause the processor to perform one or more functions.

[0129] Although embodiments have been described in detail for the purpose of illustration, it is to be understood that such detail is solely for that purpose and that the disclosure is not limited to the disclosed embodiments or aspects, but, on the contrary, is intended to cover modifications and equivalent arrangements that are within the spirit and scope of the appended claims. For example, it is to be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect can be combined with one or more features of any other embodiment or aspect.

Claims

WHAT IS CLAIMED IS:1 . A system, comprising: at least one processor programmed or configured to: receive initial data associated with sequence information; mask one or more data instances of the initial data associated with the sequence information based on an output of a masking machine learning model to provide masked intermediate input data; provide the masked intermediate input data as an input to a classification machine learning model to generate a classification label of the masked intermediate input data, wherein the classification machine learning model is a transformer machine learning model configured to provide a prediction of a classification label of an input as an output; and perform an action based on the classification label of the masked intermediate input data.

2. The system of claim 1 , wherein, when masking the one or more data instances of the initial data associated with the sequence information, the at least one processor is programmed or configured to: provide the initial data as an input to the masking machine learning model to generate an output, wherein the output of the masking machine learning model comprises a prediction of a binary classification of each data instance of the initial data; and assign a negative binary classification to the one or more data instances of the initial data provided as the input to the masking machine learning model based on the prediction of the binary classification of the one or more data instances of the initial data.

3. The system of claim 1 , wherein, when masking the one or more data instances of the initial data associated with the sequence information, the at least one processor is programmed or configured to: provide the initial data as an input to the masking machine learning model to generate an output, wherein the output of the masking machine learningmodel comprises a prediction of a binary classification of each data instance of the initial data; and remove a value of the one or more data instances of the initial data provided as the input to the masking machine learning model from an embedding to be provided as an input to the classification machine learning model based on the prediction of the binary classification of the one or more data instances of the initial data.

4. The system of claim 1 , wherein the at least one processor is further programmed or configured to: train the masking machine learning model and the classification machine learning model based on a loss function.

5. The system of claim 4, wherein the loss function comprises a formula:

6. The system of claim 1 , wherein the initial data associated with the sequence information comprises a plurality of tokens, and wherein the at least one processor is programmed or configured to: tokenize discrete instances of the sequence information to provide the plurality of tokens of the initial data associated with the sequence information.

7. The system of claim 1 , wherein, when performing the action, the at least one processor is programmed or configured to: execute a fraud protection procedure based on the classification label of the masked intermediate input data.

8. A computer-implemented method, comprising: receiving, with at least one processor, initial data associated with sequence information;masking, with at least one processor, one or more data instances of the initial data associated with the sequence information based on an output of a masking machine learning model to provide masked intermediate input data; providing, with at least one processor, the masked intermediate input data as an input to a classification machine learning model to generate a classification label of the masked intermediate input data, wherein the classification machine learning model is a transformer machine learning model configured to provide a prediction of a classification label of an input as an output; and performing, with at least one processor, an action based on the classification label of the masked intermediate input data.

9. The computer-implemented method of claim 8, wherein masking the one or more data instances of the initial data associated with the sequence information comprises: providing the initial data as an input to the masking machine learning model to generate an output, wherein the output of the masking machine learning model comprises a prediction of a binary classification of each data instance of the initial data; and assigning a negative binary classification to the one or more data instances of the initial data provided as the input to the masking machine learning model based on the prediction of the binary classification of the one or more data instances of the initial data.

10. The computer-implemented method of claim 8, wherein masking the one or more data instances of the initial data associated with the sequence information comprises: providing the initial data as an input to the masking machine learning model to generate an output, wherein the output of the masking machine learning model comprises a prediction of a binary classification of each data instance of the initial data; and removing a value of the one or more data instances of the initial data provided as the input to the masking machine learning model from an embedding to be provided as an input to the classification machine learning model based on theprediction of the binary classification of the one or more data instances of the initial data.1 1 . The computer-implemented method of claim 8, further comprising: training the masking machine learning model and the classification machine learning model based on a loss function.

12. The computer-implemented method of claim 1 1 , wherein the loss function comprises a formula:

13. The computer-implemented method of claim 8, wherein the initial data associated with the sequence information comprises a plurality of tokens, and the method further comprises: tokenizing discrete instances of the sequence information to provide the plurality of tokens of the initial data associated with the sequence information.

14. The computer-implemented method of claim 8, wherein performing the action comprises: executing a fraud protection procedure based on the classification label of the masked intermediate input data.

15. A computer program product comprising at least one non- transitory computer-readable medium including one or more instructions that, when executed by at least one processor, cause the at least one processor to: receive initial data associated with sequence information; mask one or more data instances of the initial data associated with the sequence information based on an output of a masking machine learning model to provide masked intermediate input data; provide the masked intermediate input data as an input to a classification machine learning model to generate a classification label of the masked intermediate input data, wherein the classification machine learning model is a transformer machinelearning model configured to provide a prediction of a classification label of an input as an output; and perform an action based on the classification label of the masked intermediate input data.

16. The computer program product of claim 15, wherein, the one or more instructions that cause the at least one processor to mask the one or more data instances of the initial data associated with the sequence information, cause the at least one processor to: provide the initial data as an input to the masking machine learning model to generate an output, wherein the output of the masking machine learning model comprises a prediction of a binary classification of each data instance of the initial data; and assign a negative binary classification to the one or more data instances of the initial data provided as the input to the masking machine learning model based on the prediction of the binary classification of the one or more data instances of the initial data.

17. The computer program product of claim 15, wherein, the one or more instructions that cause the at least one processor to mask the one or more data instances of the initial data associated with the sequence information, cause the at least one processor to: provide the initial data as an input to the masking machine learning model to generate an output, wherein the output of the masking machine learning model comprises a prediction of a binary classification of each data instance of the initial data; and remove a value of the one or more data instances of the initial data provided as the input to the masking machine learning model from an embedding to be provided as an input to the classification machine learning model based on the prediction of the binary classification of the one or more data instances of the initial data.

18. The computer program product of claim 15, wherein the one or more instructions further cause the at least one processor to:train the masking machine learning model and the classification machine learning model based on a loss function, wherein the loss function comprises a formula:

19. The computer program product of claim 15, wherein the initial data associated with the sequence information comprises a plurality of tokens, and wherein the one or more instructions further cause the at least one processor to: tokenize discrete instances of the sequence information to provide the plurality of tokens of the initial data associated with the sequence information.

20. The computer program product of claim 15, wherein, the one or more instructions that cause the at least one processor to perform the action, cause the at least one processor to: execute a fraud protection procedure based on the classification label of the masked intermediate input data.

Images