Methods, systems, and computer program products for providing type aware transducers for sequential data sets

Abstract

A method is provided, the method comprising: receiving interaction data associated with a plurality of interactions, the interaction data comprising an interaction record, the interaction record comprising a plurality of fields, the plurality of fields comprising a static field and a dynamic field; generating a static interaction embedding representation based on static field data associated with the static field and a first transformer model; generating a plurality of dynamic interaction embedding representations based on dynamic field data associated with the dynamic fields of the interaction record sequence and a second transformer model; generating a first intermediate input and a plurality of second intermediate inputs; generating a static sequence embedding representation and a dynamic sequence embedding representation based on the third transformer model; and generating at least one prediction based on inputting the static sequence embedded representation and the plurality of dynamic sequence embedded representations to a machine learning model. Systems and computer program products are also disclosed.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 454,089, filed on March 23, 2024, the disclosure of which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to transformer machine learning models and, in some specific embodiments or aspects, to methods, systems, and computer program products for providing transformer machine learning models with data type awareness for sequential data sets. Background Art

[0004] Some machine learning models, such as neural networks (e.g., convolutional neural networks), may receive an input data set including data points for training. After training the neural network, each data point in the training data set may have a different impact on a neural network generated based on the training neural network (e.g., the trained neural network). In some cases, the input data sets designed for the neural network may be independent and identically distributed. The independent and identically distributed input data sets can be used to determine the effect (e.g., impact) of each data point of the input data set.

[0005] A transformer machine learning model may refer to a deep learning model that is designed to process sequential input data and includes a self-attention mechanism that gives different weights to the importance of each part of the sequential input data. The transformer machine learning model may process the entire input data all at once, and the attention mechanism may provide context for any position of the sequential input data.

[0006] Embeddings (e.g., neural embeddings) may refer to a relatively low-dimensional space to which a high-dimensional vector, such as a feature vector, may be transformed. In some examples, an embedding may include a vector having values ​​that represent semantic and syntactic relationships of sequential input data by placing semantically similar inputs closer together in the embedding space. In some instances, embeddings may improve the performance of machine learning techniques on large inputs, such as sparse vectors representing words. For example, embeddings may be learned and reused on machine learning models.

[0007] However, if the input data types (e.g., dynamic data types and static data types) of the sequential input data are provided to the transformer machine learning and are not properly considered, then the information that does not change may be replicated on each data instance in the sequential input data. Therefore, the static information may consume a large amount of resources (e.g., memory space, processing power, etc.) with respect to the transformer machine learning model. In addition, the static information may reduce the difficulty during the model training (e.g., pre-training) task to the extent that the static information may prevent the transformer machine learning model from learning a meaningful representation of the sequential input data. Summary of the invention

[0008] Thus, improved methods, systems, and computer program products are provided for providing data type-aware transformer machine learning models for sequential data sets (e.g., overcoming some or all of the shortcomings identified above).

[0009] According to some non-limiting embodiments or aspects, a computer-implemented method for providing a data type-aware transformer machine learning model for a sequential data set is provided. An example method may include receiving interaction data associated with a plurality of interactions, the interaction data including a plurality of interaction records. Each of the plurality of interaction records may include a plurality of fields, the plurality of fields including at least one static field and at least one dynamic field. A static interaction embedding representation may be generated based on inputting static field data associated with at least one static field into a first transformer model. A plurality of dynamic interaction embedding representations may be generated based on inputting dynamic field data associated with at least one dynamic field of a sequence of interaction records into a second transformer model. The sequence of interaction records may include at least a subset of the plurality of interaction records. A first intermediate input may be generated based on a static interaction embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with at least one static field. A plurality of second intermediate inputs may be generated based on each dynamic interaction embedding representation, a corresponding time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field. A static sequence embedding representation may be generated based on inputting the first intermediate input into a third transformer model. A plurality of dynamic sequence embedding representations may be generated based on inputting the plurality of second intermediate inputs to a third transformer model. At least one prediction may be generated based on inputting the static sequence embedding representation and the plurality of dynamic sequence embedding representations to a machine learning model.

[0010] In some non-limiting embodiments or aspects, generating the first intermediate input may include combining a static interaction embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with at least one static field.

[0011] In some non-limiting embodiments or aspects, generating multiple dynamic interaction embedding representations may include: generating a first dynamic interaction embedding associated with a first interaction record based on inputting first dynamic field data associated with at least one dynamic field of the first interaction record into a second transformer model; and / or generating a second dynamic interaction embedding associated with a second interaction record based on inputting second dynamic field data associated with at least one dynamic field of the second interaction record into a second transformer model. A first time-based embedding representation associated with the first interaction record may be generated, and / or a second time-based embedding representation associated with the second interaction record may be generated. Generating multiple second intermediate inputs may include: combining the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field; and / or combining the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field.

[0012] In some non-limiting embodiments or aspects, combining the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field may include summing the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field. In some non-limiting embodiments or aspects, combining the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field may include summing the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field.

[0013] In some non-limiting embodiments or aspects, for each interaction record in the plurality of interaction records, static field data associated with at least one static field may be separated from dynamic field data associated with at least one dynamic field. A first input to a first transformer model may be generated based on the static field data associated with the at least one static field. A second input to a second transformer model may be generated based on the dynamic field data associated with the at least one dynamic field.

[0014] In some non-limiting embodiments or aspects, at least one dynamic field may include multiple dynamic fields. Before the dynamic field data associated with the multiple dynamic fields of the interaction record sequence is input into the second transformer model, the original value of the dynamic field of the first interaction record in the interaction record sequence may be masked to provide the masked dynamic field of the first interaction record. Generating multiple dynamic interaction embedding representations based on inputting the dynamic field data associated with the multiple dynamic fields of the interaction record sequence into the second transformer model may include generating multiple dynamic interaction embedding representations based on inputting the masked dynamic field of the first interaction record into the second transformer model. The third transformer model may be trained by comparing the data value of the data field of the dynamic sequence embedding representation associated with the first interaction record provided by the third transformer model with the original value of the dynamic field of the first interaction record, and adjusting the parameters of the third transformer model based on comparing the data value of the data field of the dynamic sequence embedding representation associated with the first interaction record provided by the third transformer with the original value of the dynamic field of the first interaction record.

[0015] In some non-limiting embodiments or aspects, an action associated with a fraud detection task may be performed based on at least one prediction.

[0016] According to non-limiting embodiments or aspects, a system for providing a data type aware transformer machine learning model for a sequential data set is provided. An example system may include at least one processor configured to receive interaction data associated with a plurality of interactions, the interaction data including a plurality of interaction records. Each of the plurality of interaction records may include a plurality of fields, the plurality of fields including at least one static field and at least one dynamic field. A static interaction embedding representation may be generated based on inputting static field data associated with at least one static field into a first transformer model. A plurality of dynamic interaction embedding representations may be generated based on inputting dynamic field data associated with at least one dynamic field of a sequence of interaction records into a second transformer model. The sequence of interaction records may include at least a subset of the plurality of interaction records. A first intermediate input may be generated based on a static interaction embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with at least one static field. A plurality of second intermediate inputs may be generated based on each dynamic interaction embedding representation, a corresponding time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field. A static sequence embedding representation may be generated based on inputting the first intermediate input into a third transformer model. A plurality of dynamic sequence embedding representations may be generated based on inputting the plurality of second intermediate inputs to a third transformer model. At least one prediction may be generated based on inputting the static sequence embedding representation and the plurality of dynamic sequence embedding representations to a machine learning model.

[0017] In some non-limiting embodiments or aspects, generating the first intermediate input may include combining a static interaction embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with at least one static field.

[0018] In some non-limiting embodiments or aspects, generating multiple dynamic interaction embedding representations may include: generating a first dynamic interaction embedding associated with a first interaction record based on inputting first dynamic field data associated with at least one dynamic field of the first interaction record into a second transformer model; and / or generating a second dynamic interaction embedding associated with a second interaction record based on inputting second dynamic field data associated with at least one dynamic field of the second interaction record into a second transformer model. A first time-based embedding representation associated with the first interaction record may be generated, and / or a second time-based embedding representation associated with the second interaction record may be generated. Generating multiple second intermediate inputs may include: combining the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field; and / or combining the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field.

[0019] In some non-limiting embodiments or aspects, combining the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field may include summing the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field. In some non-limiting embodiments or aspects, combining the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field may include summing the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field.

[0020] In some non-limiting embodiments or aspects, for each interaction record in the plurality of interaction records, static field data associated with at least one static field may be separated from dynamic field data associated with at least one dynamic field. A first input to a first transformer model may be generated based on the static field data associated with the at least one static field. A second input to a second transformer model may be generated based on the dynamic field data associated with the at least one dynamic field.

[0021] In some non-limiting embodiments or aspects, at least one dynamic field may include multiple dynamic fields. Before the dynamic field data associated with the multiple dynamic fields of the interaction record sequence is input into the second transformer model, the original value of the dynamic field of the first interaction record in the interaction record sequence may be masked to provide the masked dynamic field of the first interaction record. Generating multiple dynamic interaction embedding representations based on inputting the dynamic field data associated with the multiple dynamic fields of the interaction record sequence into the second transformer model may include generating multiple dynamic interaction embedding representations based on inputting the masked dynamic field of the first interaction record into the second transformer model. The third transformer model may be trained by comparing the data value of the data field of the dynamic sequence embedding representation associated with the first interaction record provided by the third transformer model with the original value of the dynamic field of the first interaction record, and adjusting the parameters of the third transformer model based on comparing the data value of the data field of the dynamic sequence embedding representation associated with the first interaction record provided by the third transformer with the original value of the dynamic field of the first interaction record.

[0022] In some non-limiting embodiments or aspects, an action associated with a fraud detection task may be performed based on at least one prediction.

[0023] According to some non-limiting embodiments or aspects, a computer program product for providing a data type aware transformer machine learning model for a sequential data set is provided. An example computer program product may include at least one non-transient computer readable medium, the at least one non-transient computer readable medium including program instructions, the program instructions when executed by at least one processor causing the at least one processor to receive interaction data associated with a plurality of interactions, the interaction data including a plurality of interaction records. Each interaction record in the plurality of interaction records may include a plurality of fields, the plurality of fields including at least one static field and at least one dynamic field. A static interaction embedding representation may be generated based on inputting static field data associated with at least one static field into a first transformer model. A plurality of dynamic interaction embedding representations may be generated based on inputting dynamic field data associated with at least one dynamic field of a sequence of interaction records into a second transformer model. The sequence of interaction records may include at least a subset of the plurality of interaction records. A first intermediate input may be generated based on a static interaction embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with at least one static field. A plurality of second intermediate inputs may be generated based on each dynamic interaction embedding representation, a corresponding time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field. A static sequence embedding representation may be generated based on inputting the first intermediate input to a third transformer model. A plurality of dynamic sequence embedding representations may be generated based on inputting the plurality of second intermediate inputs to the third transformer model. At least one prediction may be generated based on inputting the static sequence embedding representation and the plurality of dynamic sequence embedding representations to a machine learning model.

[0024] In some non-limiting embodiments or aspects, generating the first intermediate input may include combining a static interaction embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with at least one static field.

[0025] In some non-limiting embodiments or aspects, generating multiple dynamic interaction embedding representations may include: generating a first dynamic interaction embedding associated with a first interaction record based on inputting first dynamic field data associated with at least one dynamic field of the first interaction record into a second transformer model; and / or generating a second dynamic interaction embedding associated with a second interaction record based on inputting second dynamic field data associated with at least one dynamic field of the second interaction record into a second transformer model. A first time-based embedding representation associated with the first interaction record may be generated, and / or a second time-based embedding representation associated with the second interaction record may be generated. Generating multiple second intermediate inputs may include: combining the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field; and / or combining the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field.

[0026] In some non-limiting embodiments or aspects, combining the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field may include summing the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field. In some non-limiting embodiments or aspects, combining the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field may include summing the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field.

[0027] In some non-limiting embodiments or aspects, for each interaction record in the plurality of interaction records, static field data associated with at least one static field may be separated from dynamic field data associated with at least one dynamic field. A first input to a first transformer model may be generated based on the static field data associated with the at least one static field. A second input to a second transformer model may be generated based on the dynamic field data associated with the at least one dynamic field.

[0028] In some non-limiting embodiments or aspects, at least one dynamic field may include multiple dynamic fields. Before the dynamic field data associated with the multiple dynamic fields of the interaction record sequence is input into the second transformer model, the original value of the dynamic field of the first interaction record in the interaction record sequence may be masked to provide the masked dynamic field of the first interaction record. Generating multiple dynamic interaction embedding representations based on inputting the dynamic field data associated with the multiple dynamic fields of the interaction record sequence into the second transformer model may include generating multiple dynamic interaction embedding representations based on inputting the masked dynamic field of the first interaction record into the second transformer model. The third transformer model may be trained by comparing the data value of the data field of the dynamic sequence embedding representation associated with the first interaction record provided by the third transformer model with the original value of the dynamic field of the first interaction record, and adjusting the parameters of the third transformer model based on comparing the data value of the data field of the dynamic sequence embedding representation associated with the first interaction record provided by the third transformer with the original value of the dynamic field of the first interaction record.

[0029] In some non-limiting embodiments or aspects, an action associated with a fraud detection task may be performed based on at least one prediction.

[0030] Other non-limiting embodiments or aspects are set forth in the following numbered clauses:

[0031] Item 1: A computer-implemented method comprising: receiving, with at least one processor, interaction data associated with a plurality of interactions, the interaction data comprising a plurality of interaction records, each of the plurality of interaction records comprising a plurality of fields, the plurality of fields comprising at least one static field and at least one dynamic field; generating, with at least one processor, a static interaction embedding representation based on inputting static field data associated with the at least one static field into a first transformer model; generating, with at least one processor, a plurality of dynamic interaction embedding representations based on inputting dynamic field data associated with at least one dynamic field of a sequence of interaction records into a second transformer model, the sequence of interaction records comprising at least a subset of the plurality of interaction records; generating, with at least one processor, a plurality of dynamic interaction embedding representations based on the static interaction embedding representations; generating a first intermediate input using at least one processor based on an input representation, a first time-based embedding representation, and a first field type embedding representation associated with at least one static field; generating multiple second intermediate inputs using at least one processor based on each dynamic interaction embedding representation, the corresponding time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field; generating a static sequence embedding representation based on inputting the first intermediate input to a third transformer model using at least one processor; generating multiple dynamic sequence embedding representations based on inputting the multiple second intermediate inputs to the third transformer model using at least one processor; and generating at least one prediction based on inputting the static sequence embedding representation and the multiple dynamic sequence embedding representations to a machine learning model using at least one processor.

[0032] Clause 2: A computer-implemented method according to clause 1, wherein generating the first intermediate input includes: combining a static interaction embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with at least one static field.

[0033] Item 3: A computer-implemented method according to Item 1 or Item 2, wherein generating multiple dynamic interaction embedding representations includes: generating a first dynamic interaction embedding associated with a first interaction record based on inputting first dynamic field data associated with at least one dynamic field of the first interaction record into a second transformer model; and generating a second dynamic interaction embedding associated with the second interaction record based on inputting second dynamic field data associated with at least one dynamic field of the second interaction record into a second transformer model; the computer-implemented method also includes: generating a first time-based embedding representation associated with the first interaction record; and generating a second time-based embedding representation associated with the second interaction record; and wherein generating multiple second intermediate inputs includes: combining the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field; and combining the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field.

[0034] Clause 4: A computer-implemented method according to any one of clauses 1 to 3, wherein combining a first dynamic interaction embedding, a first time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field comprises: summing the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field; and wherein combining a second dynamic interaction embedding, a second time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field comprises: summing the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field.

[0035] Item 5: A computer-implemented method according to any one of items 1 to 4, further comprising: for each interaction record in a plurality of interaction records, separating static field data associated with at least one static field from dynamic field data associated with at least one dynamic field; generating a first input to a first transformer model based on the static field data associated with at least one static field; and generating a second input to a second transformer model based on the dynamic field data associated with at least one dynamic field.

[0036] Item 6: A computer-implemented method according to any one of Items 1 to 5, wherein at least one dynamic field includes multiple dynamic fields, and the method further includes: before inputting the dynamic field data associated with the multiple dynamic fields of the interaction record sequence into the second transformer model, masking the original value of the dynamic field of the first interaction record in the interaction record sequence to provide the masked dynamic field of the first interaction record; wherein generating multiple dynamic interaction embedding representations based on inputting the dynamic field data associated with the multiple dynamic fields of the interaction record sequence into the second transformer model includes: generating multiple dynamic interaction embedding representations based on inputting the masked dynamic fields of the first interaction record into the second transformer model; and wherein the computer-implemented method further includes: training the third transformer model by comparing the data value of the data field of the dynamic sequence embedding representation associated with the first interaction record provided by the third transformer model with the original value of the dynamic field of the first interaction record, and adjusting the parameters of the third transformer model based on comparing the data value of the data field of the dynamic sequence embedding representation associated with the first interaction record provided by the third transformer with the original value of the dynamic field of the first interaction record.

[0037] Clause 7: The computer-implemented method of any of Clauses 1 to 6, further comprising: performing an action associated with a fraud detection task based on the at least one prediction.

[0038] Item 8: A system comprising: at least one processor configured to: receive interaction data associated with a plurality of interactions, the interaction data comprising a plurality of interaction records, each of the plurality of interaction records comprising a plurality of fields, the plurality of fields comprising at least one static field and at least one dynamic field; generate a static interaction embedding representation based on inputting static field data associated with at least one static field into a first transformer model; generate a plurality of dynamic interaction embedding representations based on inputting dynamic field data associated with at least one dynamic field of a sequence of interaction records into a second transformer model, the sequence of interaction records comprising at least a subset of the plurality of interaction records; generate a first intermediate input based on the static interaction embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with at least one static field; generate a plurality of second intermediate inputs based on each dynamic interaction embedding representation, a corresponding time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field; generate a static sequence embedding representation based on inputting the first intermediate input into a third transformer model; generate a plurality of dynamic sequence embedding representations based on inputting the plurality of second intermediate inputs into the third transformer model; and generate at least one prediction based on inputting the static sequence embedding representation and the plurality of dynamic sequence embedding representations into a machine learning model.

[0039] Clause 9: The system of clause 8, wherein generating the first intermediate input comprises combining a static interaction embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with at least one static field.

[0040] Item 10: A system according to item 8 or item 9, wherein generating multiple dynamic interaction embedding representations includes: generating a first dynamic interaction embedding associated with a first interaction record based on inputting first dynamic field data associated with at least one dynamic field of the first interaction record into a second transformer model; and generating a second dynamic interaction embedding associated with the second interaction record based on inputting second dynamic field data associated with at least one dynamic field of the second interaction record into a second transformer model; and wherein at least one processor is further configured to: generate a first time-based embedding representation associated with the first interaction record; and generate a second time-based embedding representation associated with the second interaction record; and wherein generating multiple second intermediate inputs includes: combining the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field; and combining the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field.

[0041] Item 11: A system according to any of items 8 to 10, wherein combining a first dynamic interaction embedding, a first time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field includes: summing the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field; and wherein combining a second dynamic interaction embedding, a second time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field includes: summing the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field.

[0042] Item 12: A system according to any one of items 8 to 11, wherein at least one processor is further configured to: for each interaction record in a plurality of interaction records, separate static field data associated with at least one static field from dynamic field data associated with at least one dynamic field; generate a first input to a first transformer model based on the static field data associated with at least one static field; and generate a second input to a second transformer model based on the dynamic field data associated with at least one dynamic field.

[0043] Item 13: A system according to any one of items 8 to 12, wherein at least one dynamic field includes multiple dynamic fields, and wherein at least one processor is further configured to: mask the original values ​​of the dynamic fields of the first interaction record in the interaction record sequence before inputting the dynamic field data associated with the multiple dynamic fields of the interaction record sequence into the second transformer model to provide the masked dynamic fields of the first interaction record; wherein generating multiple dynamic interaction embedding representations based on inputting the dynamic field data associated with the multiple dynamic fields of the interaction record sequence into the second transformer model includes: generating multiple dynamic interaction embedding representations based on inputting the masked dynamic fields of the first interaction record into the second transformer model; and wherein at least one processor is further configured to: train the third transformer model by comparing the data values ​​of the data fields of the dynamic sequence embedding representation associated with the first interaction record provided by the third transformer model with the original values ​​of the dynamic fields of the first interaction record, and adjusting the parameters of the third transformer model based on comparing the data values ​​of the data fields of the dynamic sequence embedding representation associated with the first interaction record provided by the third transformer with the original values ​​of the dynamic fields of the first interaction record.

[0044] Clause 14: The system of any of Clauses 8 to 13, wherein the at least one processor is further configured to: perform an action associated with a fraud detection task based on the at least one prediction.

[0045] Clause 15: A computer program product comprising at least one non-transitory computer-readable medium, the at least one non-transitory computer-readable medium comprising program instructions that, when executed by at least one processor, cause the at least one processor to: receive interaction data associated with a plurality of interactions, the interaction data comprising a plurality of interaction records, each of the plurality of interaction records comprising a plurality of fields, the plurality of fields comprising at least one static field and at least one dynamic field; generate a static interaction embedding representation based on inputting static field data associated with at least one static field into a first transformer model; generate a plurality of dynamic interaction embedding representations based on inputting dynamic field data associated with at least one dynamic field of a sequence of interaction records into a second transformer model inputting a representation, wherein the sequence of interaction records includes at least a subset of a plurality of interaction records; generating a first intermediate input based on a static interaction embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with at least one static field; generating a plurality of second intermediate inputs based on each dynamic interaction embedding representation, a corresponding time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field; generating a static sequence embedding representation based on inputting the first intermediate input into a third transformer model; generating a plurality of dynamic sequence embedding representations based on inputting the plurality of second intermediate inputs into the third transformer model; and generating at least one prediction based on inputting the static sequence embedding representation and the plurality of dynamic sequence embedding representations into a machine learning model.

[0046] Clause 16: A computer program product as described in clause 15, wherein generating the first intermediate input includes: combining a static interaction embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with at least one static field.

[0047] Item 17: A computer program product according to item 15 or item 16, wherein generating multiple dynamic interaction embedding representations includes: generating a first dynamic interaction embedding associated with a first interaction record based on inputting first dynamic field data associated with at least one dynamic field of the first interaction record into a second transformer model; and generating a second dynamic interaction embedding associated with a second interaction record based on inputting second dynamic field data associated with at least one dynamic field of the second interaction record into a second transformer model; wherein the instructions, when executed by at least one processor, further cause the at least one processor to: generate a first time-based embedding representation associated with the first interaction record; and generate a second time-based embedding representation associated with the second interaction record; wherein generating multiple second intermediate inputs includes: combining the first dynamic interaction embedding, the first basic a time-based embedded representation and a second field type embedded representation associated with at least one dynamic field; and combining a second dynamic interaction embedding, a second time-based embedded representation, and a second field type embedded representation associated with at least one dynamic field; wherein combining a first dynamic interaction embedding, a first time-based embedded representation, and a second field type embedded representation associated with at least one dynamic field comprises: summing the first dynamic interaction embedding, the first time-based embedded representation, and the second field type embedded representation associated with at least one dynamic field; and wherein combining a second dynamic interaction embedding, a second time-based embedded representation, and a second field type embedded representation associated with at least one dynamic field comprises: summing the second dynamic interaction embedding, the second time-based embedded representation, and the second field type embedded representation associated with at least one dynamic field.

[0048] Item 18: A computer program product according to any one of items 15 to 17, wherein the instructions, when executed by at least one processor, further cause the at least one processor to: for each interaction record in a plurality of interaction records, separate static field data associated with at least one static field from dynamic field data associated with at least one dynamic field; generate a first input to a first transformer model based on the static field data associated with at least one static field; and generate a second input to a second transformer model based on the dynamic field data associated with at least one dynamic field.

[0049] Item 19: A computer program product according to any one of items 15 to 18, wherein at least one dynamic field includes multiple dynamic fields, and wherein the instructions, when executed by at least one processor, also cause the at least one processor to: mask the original values ​​of the dynamic fields of the first interaction record in the interaction record sequence before inputting the dynamic field data associated with the multiple dynamic fields of the interaction record sequence into the second transformer model to provide the masked dynamic fields of the first interaction record; wherein generating multiple dynamic interaction embedding representations based on inputting the dynamic field data associated with the multiple dynamic fields of the interaction record sequence into the second transformer model includes: generating multiple dynamic interaction embedding representations based on inputting the masked dynamic fields of the first interaction record into the second transformer model; and wherein the instructions, when executed by at least one processor, also cause the at least one processor to: train the third transformer model by comparing the data values ​​of the data fields of the dynamic sequence embedding representation associated with the first interaction record provided by the third transformer model with the original values ​​of the dynamic fields of the first interaction record, and adjusting the parameters of the third transformer model based on comparing the data values ​​of the data fields of the dynamic sequence embedding representation associated with the first interaction record provided by the third transformer with the original values ​​of the dynamic fields of the first interaction record.

[0050] Clause 20: The computer program product of any of clauses 15 to 19, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to: perform an action associated with a fraud detection task based on the at least one prediction.

[0051] These and other features and characteristics of the present disclosure, as well as methods of operation and functions of related structural elements and combinations of parts and economies of manufacture, will become more apparent when considering the following description and appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for purposes of illustration and description only and are not intended as a definition of the limits of the disclosed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Additional advantages and details are explained in more detail below with reference to non-limiting exemplary embodiments or aspects shown in the schematic drawings, in which:

[0053] Figure 1 is a schematic diagram of a system for a data type-aware transformer machine learning model for a sequential data set according to some non-limiting embodiments or aspects;

[0054] Figure 2 is a flow chart of a method for a data type aware transformer machine learning model for a sequential data set according to some non-limiting embodiments or aspects;

[0055] Figure 3 is a schematic diagram of an example payment processing network in which the methods, systems, and / or computer program products described herein may be implemented according to some non-limiting embodiments or aspects;

[0056] Figure 4 According to some non-limiting embodiments or aspects Figure 1 and / or Figure 3 schematic diagrams of example components of one or more devices;

[0057] Figure 5A is a schematic diagram of an example sequential data set according to some non-limiting embodiments or aspects;

[0058] Figure 5B is a schematic diagram of an example sequential data set and example embeddings according to some non-limiting embodiments or aspects;

[0059] Figure 6 is a schematic diagram of an example implementation of a system and method for a data type-aware transformer machine learning model for a sequential data set, according to some non-limiting embodiments or aspects;

[0060] Fig. 7A and Figure 7B is a schematic diagram of an example implementation of a data type-aware transformer machine learning model for a sequential data set according to some non-limiting embodiments or aspects; and

[0061] FIG. 8A to FIG. 8C is a diagram of an output of an example implementation of a system and method for a data type-aware transformer machine learning model for a sequential data set, according to some non-limiting embodiments or aspects. DETAILED DESCRIPTION

[0062] For the purposes of the following description, the terms "end," "upper," "lower," "right," "left," "vertical," "horizontal," "top," "bottom," "lateral," "longitudinal," and their derivatives shall relate to the orientation of the embodiments in the accompanying drawings. However, it shall be understood that the present disclosure may employ various alternative variations and step sequences, unless expressly specified to the contrary. It shall also be understood that the specific devices and processes shown in the accompanying drawings and described in the following specification are merely exemplary and non-limiting embodiments or aspects of the disclosed subject matter. Accordingly, specific dimensions and other physical characteristics relating to the embodiments or aspects disclosed herein shall not be considered limiting.

[0063] Some non-limiting embodiments or aspects are described herein in conjunction with a threshold value. As used herein, satisfying a threshold value may refer to a value greater than a threshold value, more than a threshold value, above a threshold value, greater than or equal to a threshold value, less than a threshold value, less than a threshold value, below a threshold value, less than or equal to a threshold value, equal to a threshold value, etc.

[0064] Aspects, parts, elements, structures, actions, steps, functions, instructions and / or the like used herein should not be understood as critical or necessary unless explicitly described as such. Moreover, as used herein, the article "one" is intended to include one or more items, and can be used interchangeably with "one or more" and "at least one". In addition, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, combinations of related items and unrelated items, etc.), and can be used interchangeably with "one or more" or "at least one". In the case of wishing only one item, the term "one" or similar language is used. Moreover, as used herein, the term "having" and / or the like is intended to be an open term. In addition, unless otherwise explicitly stated, the phrase "based on" is intended to mean "based at least in part". In addition, the reference to the action of "based on" a condition may refer to the action being "in response to" the condition. For example, in some non-limiting embodiments or aspects, the phrases "based on" and "in response to" may refer to the condition of automatically triggering an action (e.g., a specific operation of an electronic device such as a computing device, a processor and / or the like).

[0065] As used herein, the term "acquirer institution" may refer to an entity that is licensed and / or approved by a transaction service provider to initiate transactions (e.g., payment transactions) using a payment device associated with the transaction service provider. Transactions that may be initiated by an acquirer institution may include payment transactions (e.g., purchases, original letter of credit transactions (OCTs), account funding transactions (AFTs), and / or the like). In some non-limiting embodiments or aspects, the acquirer institution may be a financial institution, such as a bank. As used herein, the term "acquirer system" may refer to one or more computing devices operated by or on behalf of an acquirer institution, such as a server computer executing one or more software applications.

[0066] As used herein, the term "account identifier" may include one or more primary account numbers (PANs), payment tokens, or other identifiers associated with a customer account. The term "payment token" may refer to an identifier used as a replacement or alternative identifier for an original account identifier such as a PAN. An account identifier may be alphanumeric or any combination of characters and / or symbols. A payment token may be associated with a PAN or other original account identifier in one or more data structures (e.g., one or more databases, etc.) so that the payment token can be used to conduct transactions without directly using the original account identifier. In some instances, an original account identifier such as a PAN may be associated with multiple payment tokens for different individuals or purposes.

[0067] As used herein, the terms "client" and "client device" may refer to one or more client-side devices or systems (e.g., remote from a transaction service provider) used to initiate or facilitate a transaction (e.g., a payment transaction). As an example, a "client device" may refer to one or more POS devices used by a merchant, one or more acquirer host computers used by an acquirer, one or more mobile devices used by a user, and the like. In some non-limiting embodiments or aspects, a client device may be an electronic device configured to communicate with one or more networks and initiate or facilitate a transaction. For example, a client device may include one or more computers, portable computers, laptop computers, tablet computers, mobile devices, cellular phones, wearable devices (e.g., watches, glasses, lenses, clothing, etc.), PDAs, and the like. In addition, a "client" may also refer to an entity (e.g., a merchant, an acquirer, etc.) that owns, utilizes, and / or operates a client device for initiating a transaction (e.g., for initiating a transaction with a transaction service provider).

[0068] As used herein, the term "communication" may refer to the reception, acceptance, transmission, transmission, provision, etc. of data (e.g., information, signals, messages, instructions, commands, etc.). A unit (e.g., a device, a system, a component of a device or system, a combination thereof, etc.) communicating with another unit means that the unit is able to directly or indirectly receive information from the other unit and / or transmit information to the other unit. This may refer to a direct or indirect connection (e.g., a direct communication connection, an indirect communication connection, and / or the like) that is wired and / or wireless in nature. In addition, although the information sent may be modified, processed, relayed, and / or routed between the first unit and the second unit, the two units may also communicate with each other. For example, even if the first unit passively receives information and does not actively send information to the second unit, the first unit may communicate with the second unit. As another example, if at least one intermediate unit processes the information received from the first unit and transmits the processed information to the second unit, the first unit may communicate with the second unit. In some non-limiting embodiments or aspects, a message may refer to a network packet (e.g., a data packet and / or the like) comprising data. It will be appreciated that many other arrangements are possible.

[0069] As used herein, the term "computing device" may refer to one or more electronic devices configured to process data. In some examples, a computing device may include the necessary components for receiving, processing, and outputting data, such as a processor, a display, a memory, an input device, a network interface, and / or the like. The computing device may be a mobile device. As an example, a mobile device may include a cellular phone (e.g., a smartphone or a standard cellular phone), a portable computer, a wearable device (e.g., a watch, glasses, lenses, clothing, and / or the like), a personal digital assistant (PDA), and / or other similar devices. A computing device may also be a desktop computer or other form of non-mobile computer.

[0070] As used herein, the term "issuer institution" may refer to one or more entities, such as a bank, that provide customers with accounts for conducting transactions (e.g., payment transactions), such as initiating credit and / or debit payments. For example, an issuer institution may provide a customer with an account identifier, such as a PAN, that uniquely identifies one or more accounts associated with the customer. The account identifier may be embodied on a portable financial device, such as a physical financial instrument (e.g., a payment card), and / or may be electronic and used for electronic payments. The term "issuer system" refers to one or more computer devices operated by or on behalf of an issuer institution, such as a server computer executing one or more software applications. For example, an issuer system may include one or more authorization servers for authorizing transactions.

[0071] As used herein, the term "merchant" may refer to a person or entity that provides goods and / or services, or access to goods and / or services, to a customer based on a transaction, such as a payment transaction. The term "merchant" or "merchant system" may also refer to one or more computer systems operated by or on behalf of a merchant, such as a server computer that executes one or more software applications.

[0072] As used herein, a "point of sale (POS) device" may refer to one or more devices that can be used by a merchant to conduct a transaction (e.g., a payment transaction) and / or process a transaction. For example, a POS device may include one or more client devices. Additionally or alternatively, a POS device may include a peripheral device, a card reader, a scanning device (e.g., a code scanner), Communication receiver, near field communication (NFC) receiver, radio frequency identification (RFID) receiver and / or other contactless transceiver or receiver, contact-based receiver, payment terminal, etc. As used herein, a "point of sale (POS) system" may refer to one or more client devices and / or peripheral devices used by merchants to conduct transactions. For example, a POS system may include one or more POS devices, and / or other similar devices that can be used to conduct payment transactions. In some non-limiting embodiments or aspects, a POS system (e.g., a merchant POS system) may include one or more server computers that are programmed or configured to process online payment transactions through web pages, mobile applications, etc.

[0073] As used herein, the term "payment device" may refer to an electronic payment device, a portable financial device, a payment card (e.g., a credit or debit card), a gift card, a smart card, smart media, a payroll card, a healthcare card, a wristband, a machine-readable medium containing account information, a key chain device or pendant, an RFID transponder, a retailer discount or loyalty card, a cellular telephone, an electronic wallet mobile application, a personal digital assistant (PDA), a pager, a security card, a computing device, an access card, a wireless terminal, a transponder, and / or the like. In some non-limiting embodiments or aspects, a payment device may include volatile or non-volatile memory to store information (e.g., an account identifier, an account holder's name, and / or the like).

[0074] As used herein, the term "payment gateway" may refer to an entity and / or a payment processing system operated by or on behalf of such an entity, wherein the entity (e.g., a merchant service provider, a payment service provider, a payment service provider, a payment service provider with a contract with an acquirer, a payment aggregator, and / or the like) provides payment services (e.g., a transaction service provider payment service, a payment processing service, and / or the like) to one or more merchants. The payment service may be associated with the use of a portable financial device managed by a transaction service provider. As used herein, the term "payment gateway system" may refer to one or more computer systems, computer devices, servers, server groups, etc. operated by or on behalf of a payment gateway.

[0075] As used herein, the term "server" may refer to or include one or more computing devices operated by or facilitating communication and processing by multiple parties in a network environment such as the Internet, but it should be understood that communication may be facilitated through one or more public or private network environments, and that various other arrangements are possible. In addition, multiple computing devices (e.g., servers, point of sale (POS) devices, mobile devices, etc.) that communicate directly or indirectly in a network environment may constitute a "system."

[0076] As used herein, the term "system" may refer to one or more computing devices or a combination of computing devices (e.g., processors, servers, client devices, software applications, components of such devices, and / or the like). As used herein, references to a "device," "server," "processor," and / or the like may refer to a previously described device, server, or processor that is described as performing a previous step or function, a different device, server, or processor, and / or a combination of devices, servers, and / or processors. For example, as used in the specification and claims, a first device, first server, or first processor that is described as performing a first step or a first function may refer to the same or a different device, server, or processor that is described as performing a second step or a second function.

[0077] As used herein, the term "transaction service provider" may refer to an entity that receives transaction authorization requests from merchants or other entities and, in some cases, provides payment assurance through an agreement between the transaction service provider and an issuer organization. For example, a transaction service provider may include, for example The term "transaction processing system" may refer to one or more computer systems operated by or on behalf of a transaction service provider, such as a transaction processing server that executes one or more software applications. A transaction processing server may include one or more processors and, in some non-limiting embodiments or aspects, may be operated by or on behalf of a transaction service provider.

[0078] Non-limiting embodiments or aspects of the disclosed subject matter relate to methods, systems, and computer program products for providing a transformer machine learning model with data type awareness for a sequential data set. In some non-limiting embodiments or aspects, a transformer management system may include at least one processor, the at least one processor being programmed or configured to receive interaction data associated with a plurality of interactions, the interaction data including a plurality of interaction records, each of the plurality of interaction records including a plurality of fields, the plurality of fields including at least one static field and at least one dynamic field; generate a static interaction embedding representation based on inputting static field data associated with at least one static field into a first transformer model; generate a plurality of dynamic interaction embedding representations based on inputting dynamic field data associated with at least one dynamic field of a sequence of interaction records into a second transformer model, the sequence of interaction records including a plurality of interactions; at least a subset of interaction records; generating a first intermediate input based on a static interaction embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with at least one static field; generating multiple second intermediate inputs based on each dynamic interaction embedding representation, a corresponding time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field; generating a static sequence embedding representation based on inputting the first intermediate input to a third transformer model; generating multiple dynamic sequence embedding representations based on inputting the multiple second intermediate inputs to the third transformer model; and generating at least one prediction based on inputting the static sequence embedding representation and the multiple dynamic sequence embedding representations to a machine learning model.

[0079] In this way, the transformer management system can prevent static information in the sequential input data from consuming an excessive amount of resources with respect to the transformer machine learning model used during training and / or during production. Furthermore, the static information can reduce the difficulty during the model training (e.g., pre-training) task to the extent that the static information can prevent the transformer machine learning model from learning a meaningful representation of the sequential input data.

[0080] For purposes of illustration, in the following description, although the presently disclosed subject matter is described with respect to methods, systems, and computer program products for data type-aware transformer machine learning models for sequential data sets (e.g., for payment transactions), those skilled in the art will recognize that the disclosed subject matter is not limited to the non-limiting embodiments or aspects disclosed herein. For example, the methods, systems, and computer program products described herein can be used with a wide variety of settings, such as for any suitable type of sequential data set and / or for making determinations (e.g., prediction, classification, regression, etc.) based on sequential data sets using at least one machine learning model, such as data type-aware transformer machine learning models for fraud detection / prevention, authorization, authentication, identification, feature selection, product recommendations, click-through recommendations (CTR), etc.

[0081] Reference now Figure 1 , a schematic diagram of a system 100 for a data type aware transformer machine learning model for a sequential data set is shown according to some non-limiting embodiments or aspects. Figure 1 As shown in FIG. 1 , system 100 may include a transformer management system 102, a transaction service provider system 104, a user device 106, and a communication network 108. The transformer management system 102, the transaction service provider system 104, and / or the user device 106 may be interconnected (e.g., connected to communicate) via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection (e.g., the communication network 108).

[0082] The transformer management system 102 may include one or more devices configured to communicate (e.g., directly, indirectly via the communication network 108, etc.) with the transaction service provider system 104 and / or the user device 106. For example, the transformer management system 102 may include at least one computing device, such as a server, a server group, and / or other similar devices. In some non-limiting embodiments or aspects, the transformer management system 102 may be associated with a transaction service provider. For example, the transformer management system 102 may be operated by the transaction service provider. In some non-limiting embodiments or aspects, the transformer management system 102 may be a component of the transaction service provider system 104. In some non-limiting embodiments or aspects, the transformer management system 102 may communicate with a data storage device, which may be local or remote to the transformer management system 102. In some non-limiting embodiments or aspects, the transformer management system 102 may be able to receive information from the data storage device, store information in the data storage device, transmit information to the data storage device, and / or search for information stored in the data storage device.

[0083] The transaction service provider system 104 may include one or more devices configured to communicate (e.g., directly, indirectly via a communication network 108, etc.) with the transformer management system 102 and / or the user device 106. For example, the transaction service provider system 104 may include at least one computing device, such as a server, a server group, and / or other similar devices. In some non-limiting embodiments or aspects, the transaction service provider system 104 may be associated with a transaction service provider.

[0084] The user device 106 may include a computing device configured to communicate (e.g., directly, indirectly via the communication network 108, etc.) with the transformer management system 102 and / or the transaction service provider system 104. For example, the user device 106 may include a computing device such as a desktop computer, a portable computer (e.g., a tablet computer, a laptop computer, etc.), a mobile device (e.g., a cellular phone, a smartphone, a personal digital assistant, a wearable device, etc.), and / or other similar devices. In some non-limiting embodiments or aspects, the user device 106 may be associated with a user (e.g., an individual operating the user device 106).

[0085] The communication network 108 may include one or more wired and / or wireless networks. For example, the communication network 108 may include a cellular network (e.g., a long term evolution (LTE) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network (e.g., a private network associated with a transaction service provider), an ad hoc network, an intranet, the Internet, a fiber-based network, a cloud computing network, etc., and / or a combination of these or other types of networks.

[0086] Figure 1 The number and arrangement of systems and devices shown in the FIG. are provided as examples. Figure 1 There may be additional systems and / or devices, fewer systems and / or devices, different systems and / or devices, and / or systems and / or devices arranged in a different manner than the systems and / or devices shown in the drawings. Furthermore, a single system and / or device may be implemented Figure 1 Two or more systems or devices shown in Figure 1 The single system or device shown in the embodiment may be implemented as multiple distributed systems or devices. Additionally or alternatively, a group of systems (e.g., one or more systems) and / or a group of devices (e.g., one or more devices) of system 100 may perform one or more functions described as being performed by another group of systems or another group of devices of system 100.

[0087] Reference now Figure 2 , showing a flowchart of a method 200 for a data type-aware transformer machine learning model for a sequential data set, according to some non-limiting embodiments or aspects. Figure 2The steps shown in are for exemplary purposes only. It will be appreciated that in some non-limiting embodiments or aspects, additional, fewer, different and / or different order of steps may be used. In some non-limiting embodiments or aspects, steps may be automatically performed in response to the execution and / or completion of previous steps. In some non-limiting embodiments or aspects, one or more of the steps of process 200 may be performed (e.g., completely, partially, etc.) by the converter management system 102 (e.g., one or more devices of the converter management system 102). In some non-limiting embodiments or aspects, one or more of the steps of process 200 may be performed (e.g., completely, partially, etc.) by another device or device group that is separate from or includes the converter management system 102 (e.g., one or more devices of the converter management system 102), the transaction service provider system 104 (e.g., one or more devices of the transaction service provider system 104), and / or the user device 106.

[0088] like Figure 2 As shown in , at step 202, method 200 may include receiving interaction data associated with a plurality of interactions. For example, transformer management system 102 may receive interaction data associated with a plurality of interactions. In some non-limiting embodiments, the interaction data may include a plurality of interaction records (e.g., a plurality of transaction records). Each interaction record in the plurality of interaction records may include a plurality of fields, the plurality of fields including at least one static field and at least one dynamic field.

[0089] For the purpose of illustration, reference is now made to Figure 5A And continue to refer to Figure 2 , Figure 5A Schematic diagram showing an example sequential data set according to some non-limiting embodiments or aspects. Figure 5A As shown in , the first sequential data set 501 may include multiple interactions (e.g., payment transactions associated with a first user). For example, a first payment transaction Trans 1 may be associated with a payment transaction of $10 in California (CA). A second payment transaction Trans 2 may be associated with a payment transaction of $12 in CA, and the second payment transaction Trans 2 may occur a relatively small amount of time after the first payment transaction Trans 1. A third payment transaction Trans 3 may be associated with a payment transaction of $80 in Massachusetts (MA), and the third payment transaction Trans 3 may occur a relatively large amount of time after the second payment transaction Trans 2. In this example, the first sequential data set 501 may be normal (e.g., not detected as fraudulent, etc.) because the user has enough time to travel from CA to MA.

[0090] like Figure 5AAs shown in , the second sequential data set 502 may include multiple interactions (e.g., payment transactions associated with the second user). For example, a first payment transaction Trans 1 may be associated with a payment transaction of $10 in CA. A second payment transaction Trans 2 may be associated with a payment transaction of $12 in CA, and the second payment transaction Trans 2 may occur a relatively small amount of time after the first payment transaction Trans 1. A third payment transaction Trans 3 may be associated with a payment transaction of $80 in MA, and the third payment transaction Trans 3 may occur a relatively small amount of time after the second payment transaction Trans 2. In this example, the second sequential data set 502 may be abnormal (e.g., detected as fraudulent, etc.) because the user did not have enough time to travel from CA to MA.

[0091] like Figure 5A As shown in , the third sequential data set 503 may include multiple interactions (e.g., payment transactions associated with a third user). For example, a first payment transaction Trans 1 may be associated with a payment transaction of $10 in CA. A second payment transaction Trans 2 may be associated with a payment transaction of $80 in MA, and the second payment transaction Trans 2 may occur a relatively small amount of time after the first payment transaction Trans 1. A third payment transaction Trans 3 may be associated with a payment transaction of $12 in CA, and the third payment transaction Trans 3 may occur a relatively small amount of time after the second payment transaction Trans 3. In this example, the second sequential data set 503 may be abnormal (e.g., detected as fraudulent, etc.) because the user did not have enough time to travel from CA to MA and back to CA.

[0092] like Figure 5A As shown in the examples of , certain field values ​​(e.g., transaction amount, status / location, etc.) may be the same in all of the first sequential data set 501, the second sequential data set 502, and the third sequential data set 503, but the relative timing of the interactions (e.g., payment transactions) may be different. For example, the anomalous second transaction and / or third transaction in the second sequential data set 502 and the third sequential data set 503 may not be detected based solely on those particular field values ​​(e.g., without considering the timing). Thus, these examples demonstrate that time (e.g., time information) and order (e.g., sequence information) may be significant in detecting anomalies (e.g., fraud, etc.).

[0093] For the purpose of illustration, reference is now made to Figure 5B And continue to refer to Figure 2 , Figure 5B Schematic diagram showing an example sequential data set and example embedding according to some non-limiting embodiments or aspects. Figure 5BAs shown in , sequential data set 504 may include data associated with two interactions (eg, transaction data associated with two payment transactions).

[0094] In some non-limiting embodiments or aspects, the sequential data set 504 may be tokenized (e.g., split, separated, parsed, etc.) into tokens (e.g., smaller data items to be used as input to one or more machine learning models). For example, each corresponding field value of the interaction data (e.g., each field value of the transaction data associated with a payment transaction) may be represented as a corresponding token. As used herein, the term "token" in the context of a data item to be input into a machine learning model is not to be confused with a "payment token," as described herein. For example, a payment token may be a field value such that the corresponding token represents a payment token, but not all field values ​​are payment tokens.

[0095] For illustrative purposes, Figure 5B As shown in , a first payment transaction may be associated with a token representing a transaction amount (e.g., $36), a token representing a restaurant identifier (e.g., a restaurant ID associated with the restaurant where the payment transaction occurred), a token representing a timestamp (12:30 p.m.), a token representing a payment device type (e.g., a debit card), a token representing the issuer of the payment device (e.g., Bank A), and a token indicating separation (e.g., [SEP], which may be a special token indicating the end of the token associated with the current interaction, so that one or more next tokens are associated with different interactions). A second payment transaction may be associated with a token representing a transaction amount (e.g., $200), a token representing an ATM identifier (e.g., an ATM ID associated with the location where the withdrawal occurred), a token representing a timestamp (2:00 p.m.), a token representing a payment device type (e.g., a debit card), a token representing the issuer of the payment device (e.g., Bank A), and a token indicating separation (e.g., [SEP]).

[0096] Continue to refer Figure 2 In some non-limiting embodiments or aspects, the transformer management system 102 may separate, for each interaction record in the plurality of interaction records, static field data associated with at least one static field from dynamic field data associated with at least one dynamic field. In some non-limiting embodiments or aspects, the transformer management system 102 may generate a first input to a first transformer model based on the static field data associated with at least one static field, and / or generate a second input to a second transformer model based on the dynamic field data associated with at least one dynamic field.

[0097] For the purpose of illustration, again refer to Figure 5B And continue to refer to Figure 2, the sequential data set 504 can be divided into static field data 520 associated with static fields and dynamic field data 510 associated with dynamic fields. For example, the static fields may include fields that are replicated (e.g., the same, do not change between interactions, etc.) in each interaction (e.g., transaction) in the sequence (e.g., sequential data set 504). Figure 5B As shown in , static fields may be associated with a token representing a payment device type (e.g., debit card) and a token representing an issuer of the payment device (e.g., Bank A). For example, dynamic fields may include fields that are not necessarily replicated (e.g., may not be the same, change between at least some interactions, etc.) in each interaction (e.g., transaction) in a sequence (e.g., sequential data set 504). Figure 5B As shown in , a dynamic field may be associated with a token representing a transaction amount, a token representing certain identifiers (e.g., restaurant ID, ATM ID, etc.), and a token representing a timestamp. In some non-limiting embodiments or aspects, a token indicating separation (e.g., [SEP]) may be inserted between the static field data 520 and the dynamic field data 510, and / or between tokens associated with each interaction (e.g., transaction) with the dynamic field data 510. For example, such insertion may occur before the static field data 520 and the dynamic field data 510 are input into one or more machine learning models (e.g., to indicate separation between sets of tokens).

[0098] In some non-limiting embodiments or aspects, the transformer management system 102 may mask the original values ​​of the dynamic fields or static fields of the first interaction record in the interaction record sequence before inputting the static field data associated with at least one static field of the interaction record sequence into the first transformer model or before inputting the dynamic field data associated with at least one dynamic field of the interaction record sequence into the second transformer model to provide masked dynamic fields of the first interaction record. For example, such masking may be used for at least a portion of training (e.g., pre-training) as described herein. For illustrative purposes, in some non-limiting embodiments or aspects, masking may be as described herein (e.g., with reference to Fig. 7A ).

[0099] Continue to refer Figure 2, at step 204, method 200 may include generating a static interaction embedding representation and a dynamic interaction embedding representation. For example, transformer management system 102 may generate a static interaction embedding representation and a dynamic interaction embedding representation. In some non-limiting embodiments or aspects, transformer management system 102 may generate a static interaction embedding representation based on inputting static field data associated with at least one static field into a first transformer model. In some non-limiting embodiments or aspects, transformer management system 102 may generate a plurality of dynamic interaction embedding representations based on inputting dynamic field data associated with at least one dynamic field of a sequence of interaction records into a second transformer model. The sequence of interaction records may include at least a subset of the plurality of interaction records.

[0100] For the purpose of illustration, again refer to Figure 5B And continue to refer to Figure 2 , a static interaction embedding representation 560 may be generated based on the static field data 520. For example, the static interaction embedding representation 560 may include an embedding for each token of the static interaction embedding representation 560 (e.g., a debit embedding associated with a token representing a payment device type (e.g., a debit card) Debit , and an issuer embedded E associated with a token representing the issuer of the payment device (e.g., Bank A) BankA ). In some non-limiting embodiments or aspects, the separated embedded E [SEP] May be associated with a token indicating separation (eg, [SEP]).

[0101] For example, a dynamic interaction embedding representation 561 may be generated based on dynamic field data 510 of an interaction sequence (e.g., a transaction). For example, the dynamic interaction embedding representation 561 may include an embedding for each token of the dynamic field data 510. For example, the dynamic interaction embedding representation 561 may include a transaction amount embedding associated with each token representing a transaction amount (e.g., E $36 、E $200 etc.), an identifier embedding (e.g., E ResID 、E ATMID etc.) and a timestamp embedding associated with each token representing a timestamp (E 12:30PM 、E 2:00PM wait).

[0102] For the purpose of illustration, reference is now made to Fig. 7A and Figure 7B And continue to refer to Figure 2 , Fig. 7A and Figure 7B Schematic diagram showing an example implementation of a data type aware transformer machine learning model for a sequential data set according to some non-limiting embodiments or aspects. Fig. 7A and Figure 7B As shown in , static field data 720 and dynamic field data 710 may be provided as input (e.g., received, retrieved, etc.). In some non-limiting embodiments or aspects, tokens (e.g., raw data tokens u) of static field data 720 and dynamic field data 710 may be converted into a local vocabulary (e.g., vocabulary tokens v) to generate static field vocabulary data 721 and dynamic field vocabulary data 711, for example, before being input into one or more machine learning models (e.g., transformer models).

[0103] In some non-limiting embodiments or aspects, u may convert raw data tokens into vocabulary tokens based on v following the equation:

[0104] Equation 1

[0105] where i∈[0,l-1],j∈[0,n d -1]

[0106] Equation 2

[0107] v s,j =ConvertToVocab(u s,j ), where j∈[0,n s -1]

[0108] where u s,j For static fields, is a dynamic field, ConvertToVocab is a function that converts raw tokens to vocabulary tokens, and v s,j are static vocabulary tokens, is a dynamic vocabulary token, l is the number of interactions (e.g., transactions) in the sequence (e.g., the length of the sequence), and n d is the number of dynamic fields (e.g., per interaction), and n s The number of static fields.

[0109] In some non-limiting embodiments or aspects, the raw data token u may be converted into a vocabulary token v based on a preprocessor, such as the preprocessor described in Padhi et al., Tabular transformers for modeling multivariate timeseries, ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 3565-3569 (2021), the disclosure of which is hereby incorporated by reference in its entirety.

[0110] In some non-limiting embodiments or aspects, static field vocabulary data 721 may be input to a first transformer model (e.g., static field transformer 731) to generate a static interaction embedding representation 760. Additionally or alternatively, dynamic field vocabulary data 711 may be input to a second transformer model (e.g., dynamic field transformer 732) to generate a dynamic interaction embedding representation 761 for each interaction (e.g., each payment transaction).

[0111] Continue to refer Figure 2 In some non-limiting embodiments or aspects, when generating multiple dynamic interaction embedding representations, the transformer management system 102 may generate a first dynamic interaction embedding associated with a first interaction record based on inputting first dynamic field data associated with at least one dynamic field of a first interaction record in a sequence of interaction records into a second transformer model, and generate a second dynamic interaction embedding associated with a second interaction record based on inputting second dynamic field data associated with at least one second dynamic field of a first interaction record in a sequence of interaction records into a second transformer model.

[0112] For the purpose of illustration, again refer to Fig. 7A and Figure 7B And continue to refer to Figure 2 , a first dynamic interaction embedding TE may be generated based on the first interaction (eg, transaction 0) D,0 This may be repeated for each interaction (eg, each transaction). For example, assuming a total of 1 transaction, a corresponding first dynamic interaction embedding TE may be generated for each corresponding interaction (eg, transaction i) for all i=0, ..., l-1. D,i .

[0113] Continue to refer Figure 2In some non-limiting embodiments or aspects, when a static interaction embedding representation is generated based on inputting static field data associated with at least one static field of an interaction record sequence into a first transformer model, the transformer management system 102 may generate a static interaction embedding representation based on inputting at least one masked static field of the interaction record into the first transformer model.

[0114] For the purpose of illustration, again refer to Fig. 7A And continue to refer to Figure 2 , tokens of the static field data 720 may be masked, and thus, corresponding vocabulary tokens of the static field vocabulary data 721 may be masked. Thus, the static interaction embedding representation 760 (e.g., TE S ) can be based on a masked token.

[0115] Continue to refer Figure 2 In some non-limiting embodiments or aspects, when multiple dynamic interaction embedding representations are generated based on inputting dynamic field data associated with multiple dynamic fields of an interaction record sequence into a second transformer model, the transformer management system 102 may generate multiple dynamic interaction embedding representations based on inputting at least one masked dynamic field of the interaction record into the second transformer model.

[0116] For the purpose of illustration, again refer to Fig. 7A And continue to refer to Figure 2 , the tokens of each interactive dynamic field data 710 may be masked, and thus, the corresponding vocabulary tokens of the dynamic field vocabulary data 711 may be masked. Thus, the dynamic interaction embedding representation 761 (e.g., TE D,0 ,…,TE D,l-1 ) can be based on a masked token.

[0117] In some non-limiting embodiments or aspects, the tokens may be masked based on the following equation:

[0118] Equation 3

[0119] where i∈[0,l-1],j∈[0,n d -1]

[0120] Equation 4

[0121] where j∈[0,n s -1]

[0122] in Indicates whether the token is masked (for example, if Then the token is replaced by [MASK]), RandomMask is a function that randomly masks the token, and and are the dynamic vocabulary tokens and static vocabulary tokens after masking, respectively.

[0123] In some non-limiting embodiments or aspects, a static interaction embedding representation (eg, a static transaction embedding TE) may be generated based on the following equation: S ) and dynamic interaction embedding representations (e.g., dynamic transaction embedding TE D,i ):

[0124] Equation 5

[0125]

[0126] Equation 6

[0127] where i∈[0,l-1]

[0128] in is a static field transformer, and A dynamic field transformer.

[0129] like Figure 2 As shown in , at step 206, method 200 may include generating an intermediate input. For example, transformer management system 102 may generate the intermediate input. In some non-limiting embodiments or aspects, transformer management system 102 may generate a first intermediate input based on a static interactive embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with at least one static field. In some non-limiting embodiments or aspects, transformer management system 102 may generate a plurality of second intermediate inputs based on each dynamic interactive embedding representation, a corresponding time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field.

[0130] In some non-limiting embodiments or aspects, when generating a first intermediate input, the transformer management system 102 may combine (e.g., sum, concatenate, etc.) a static interaction embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with at least one static field.

[0131] In some non-limiting embodiments or aspects, the transformer management system 102 may generate a first time-based embedding representation associated with the first interaction record, and generate a second time-based embedding representation associated with the second interaction record. In some non-limiting embodiments or aspects, when generating multiple second intermediate inputs, the transformer management system 102 may combine the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field, and / or may combine the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field.

[0132] For the purpose of illustration, again refer to Figure 5B And continue to refer to Figure 2 , the static interaction embedding representation 560 may be embedded with at least one static field type 540 (e.g., E S ) and / or at least one static time-based embedded representation 550 (e.g., E0, E1 and / or E2).

[0133] Additionally or alternatively, each dynamic interaction embedding representation 561 may be associated with at least one dynamic field type embedding 541 (e.g., D ) and / or at least one dynamic time-based embedded representation 551 (e.g., E3, E4, E5, and / or E6 for the first transaction; E7, E8, E9, and / or E10 for the second interaction). 10 etc.) combination.

[0134] For the purpose of illustration, again refer to FIG. 7A to FIG. 7B And continue to refer to Figure 2 , static interaction embedding representation 760 (e.g., TE S ) can be combined with a static field type embedding 740 and a time-based embedding representation (e.g., a time-aware position embedding 750, which can be based on position 0 and time 0).

[0135] Additionally or alternatively, each dynamic interaction embedding representation 761 (eg, TE D,0 ,…,TE D,l-1 ) can be embedded with dynamic field type 741 and time-based embedding representation (e.g., time-aware location embedding 751, which can be based on the location i and time t associated with the i-th transaction i=0, ..., l-1 i )combination.

[0136] In some non-limiting embodiments or aspects, the time-aware position embedding P(i) may be defined as a vector of length d, where each element p(i, j) of the vector may be determined based on the following equation:

[0137] Equation 7

[0138] TPos(i)=w p *i+w t *t i +b

[0139] Equation 8

[0140]

[0141] Where TPos(i) is used to merge the values ​​from position index i and time interval t i function of the information of (e.g., this helps to pass the trainable parameter w p 、w t and b to capture time-aware position information, e.g. to enable learning of more flexible functions).

[0142] In some non-limiting embodiments or aspects, the intermediate input (eg, the static intermediate input representation IE) may be determined based on the following equation: S and dynamic intermediate input representation IE D,i ):

[0143] Equation 9

[0144] IE S =TE S +P(0)+IE S

[0145] Equation 10

[0146] IE D,i =TE D,i +P(i)+FE D

[0147] Among them, FE S Embedded for static field types, and FE D Embed for dynamic field types.

[0148] Continue to refer Figure 2 , at step 208, the method 200 may include generating a static sequence embedding representation and a dynamic sequence embedding representation. For example, the transformer management system 102 may generate the static sequence embedding representation and the dynamic sequence embedding representation. In some non-limiting embodiments or aspects, the transformer management system 102 may generate the static sequence embedding representation based on inputting the first intermediate input to the third transformer model. In some non-limiting embodiments or aspects, the transformer management system 102 may generate multiple dynamic sequence embedding representations based on inputting multiple second intermediate inputs to the third transformer model.

[0149] For the purpose of illustration, again refer to Fig. 7A and Figure 7B And continue to refer to Figure 2 , a static sequence embedding representation 770 (e.g., SE ) can be generated by inputting the first intermediate input (e.g., based on combining the static interaction embedding representation 760 with the field type embedding 740 and the time-aware position embedding 750) into a third transformer model (e.g., the first and time-aware sequential encoding transformer 733). S ).

[0150] Additionally or alternatively, the second intermediate input (eg, based on embedding each dynamic interaction into a representation 761 (eg, TE D,0 ,…,TE D,l-1 ) is combined with the dynamic field type embedding 741 and the corresponding time-aware position embedding 751) and is input to a third transformer model (e.g., field and time-aware sequence encoding transformer 733) to generate a dynamic sequence embedding representation 771 (e.g., SE D,0 ,…,SE D,l-1 ).

[0151] In some non-limiting embodiments or aspects, the transformer management system 102 may train at least one of the transformer models (e.g., the first transformer model, the second transformer model, the third transformer model, and / or any combination thereof) by comparing the data value of a data field of at least one of a static sequence embedding representation, a dynamic sequence embedding representation (e.g., associated with an interaction record), or any combination thereof provided by a third transformer with the original value of the static or dynamic field (e.g., of the interaction record) and / or adjusting at least one parameter of one or more transformer models based on the comparison.

[0152] In some non-limiting embodiments or aspects, the sequence embedding representation may be determined based on the following equation:

[0153] Equation 11

[0154]

[0155] in Transformers for field- and time-aware order encoding.

[0156] For the purpose of illustration, again refer to Fig. 7A And continue to refer to Figure 2, at least one token of at least one of the static field data 720 and / or the dynamic field data 710 may be masked, as described herein. Training may include inputting the static field data 720 and / or the dynamic field data 710 (e.g., including one or more masked tokens) into one or more transformer models, as described herein, and the input may be forward propagated through the one or more transformer models to generate a static sequence embedding representation 770 and / or a dynamic sequence embedding representation 771, as described herein. A predicted value for each masked token may be generated based on the sequence embeddings (e.g., the static sequence embedding representation 770 and the dynamic sequence embedding representation 771). For example, the sequence embeddings may be input to a classifier model (e.g., ) to generate the predicted value (e.g., ). In some non-limiting embodiments or aspects, the predicted value may be generated based on the following equation:

[0157] Equation 12

[0158]

[0159] For illustrative purposes, Fig. 7A As shown in a masked token (e.g., MASK) that may be predicted and may correspond to static field data 720; a masked token (e.g., MASK) that may be predicted and may correspond to the dynamic field data 710 of transaction 0; and A masked token (e.g., MASK) of the dynamic field data 710 that may be predicted and may correspond to the transaction l-1. One or more predicted values ​​may be compared to corresponding original values ​​(e.g., an actual value u of the static field data 720 and / or the dynamic field data 710 before being masked or a vocabulary v of such actual values). For example, a loss may be determined based on one or more predicted values ​​and one or more corresponding original values ​​(or their vocabulary). For example, the loss may be calculated based on a difference between the predicted value and the original value, a loss function, a cross entropy loss, an error, a mean square error, a root mean square error (MSE), any combination thereof, or the like. In some non-limiting embodiments or aspects, the loss may be calculated based on the following equation: (e.g., cross entropy loss):

[0160] Equation 13

[0161]

[0162] Parameters of at least one of the transformers (e.g., at least one of the static field transformer 731, the dynamic field transformer 732, the field and time-aware order encoding transformer 733, any combination thereof, etc.) may be updated based on the loss. For example, the transformer management system 102 may update (e.g., adjust) parameters of one or more transformers based on back propagation (e.g., of one or more losses), gradient calculation (e.g., based on one or more losses), any combination thereof, etc.

[0163] In some non-limiting embodiments or aspects, pre-training a model to predict masked values ​​may be referred to as pre-training.

[0164] Continue to refer Figure 2 , at step 210, the method 200 may include generating a prediction based on the static sequence embedding representation and the dynamic sequence embedding representation. For example, the transformer management system 102 may generate the prediction based on the static sequence embedding representation and the dynamic sequence embedding representation. In some non-limiting embodiments or aspects, the transformer management system 102 may generate at least one prediction based on inputting the static sequence embedding representation and the plurality of dynamic sequence embedding representations into a machine learning model. In some non-limiting embodiments or aspects, the transformer management system 102 may perform an action (e.g., a fraud detection action, a visualization action, etc.) based on the prediction. For example, the transformer management system 102 may perform an action associated with a fraud detection task based on at least one prediction.

[0165] For the purpose of illustration, reference is now made to Figure 6 And continue to refer to Figure 2 , Figure 6 Schematic diagram illustrating an example implementation of a system and method for a data type aware transformer machine learning model for a sequential data set according to some non-limiting embodiments or aspects. Figure 6 As shown in , static field data 620 and dynamic field data 610 can be associated with user 605 (e.g., can be associated with at least one account identifier of user 605). For example, static field data 620 can be associated with static fields of a sequential data set including t records (e.g., record 1, record 2, ..., record t), and dynamic field data 610 can be associated with its dynamic fields, as described herein.

[0166] The static field data 620 and the dynamic field data 610 may be input to a machine learning model 630, which may include at least one transformer model, such as a first transformer model (e.g., a static field transformer model), a second transformer model (e.g., a dynamic field transformer model), a third transformer model (e.g., a field and time-aware sequential encoding transformer model), any combination thereof, etc. In some non-limiting embodiments or aspects, the machine learning model 630 may have been trained (pre-trained), as described herein.

[0167] In some non-limiting embodiments or aspects, the machine learning model 630 may generate a representation 670 (e.g., a sequence representation based on a static sequence embedding representation and a dynamic sequence embedding representation associated with each record). For example, the representation 670 may be input to at least one other machine learning model 690 (e.g., a downstream machine learning model) to perform at least one task. For illustration purposes, Figure 6 As shown in , one or more other machine learning models 690 may include an anomaly detection model for performing anomaly detection tasks (e.g., classifying a sequence as anomaly or non-anomalous), a CTR prediction model for predicting CTR, a product recommendation model for predicting at least one recommended product, a fraud detection model for performing a fraud detection model (e.g., classifying a sequence as fraudulent or non-fraudulent), an authorization model for performing an authorization task, an authentication model for performing an authentication task, an identification for performing an identification task, a feature selection model for performing a feature selection task, any combination thereof, and the like.

[0168] For the purpose of illustration, again refer to Figure 7B And continue to refer to Figure 2 In some non-limiting embodiments or aspects, masking is no longer used after pre-training. Thus, static field data 720 and dynamic field data 710 do not include masked tokens (eg, all tokens have their original values).

[0169] In some non-limiting embodiments or aspects, the sequence embedding (e.g., static sequence embedding representation 770 and / or dynamic sequence embedding representation 771) may be input to at least one machine learning model 790 (e.g., at least one classifier, etc., as described herein). The one or more machine learning models 790 may generate at least one output 791 (e.g., prediction, classification, etc., as described herein) based on the sequence embedding.

[0170] In some non-limiting embodiments or aspects, at least one of the one or more transformers (e.g., static field transformer 731, dynamic field transformer 732, and / or field and time-aware sequential encoding transformer 733) may be (re)trained, and / or one or more machine learning models 790 may be trained. For example, static field data 720 and / or dynamic field data 710 may be input to one or more transformer models, as described herein, and the input may be forward propagated through one or more transformer models to generate static sequence embedding representations 770 and / or dynamic sequence embedding representations 771, as described herein. One or more machine learning models 790 may generate one or more outputs 791 based on sequence embeddings (e.g., generate static sequence embedding representations 770 and / or dynamic sequence embedding representations 771), as described herein. Loss may be determined based on one or more predicted values. For example, loss may be determined based on one or more outputs 791 and one or more known values ​​(e.g., labels), loss functions, errors, mean errors, MSE, any combination thereof, and the like. At least one of the transformers (e.g., at least one of the static field transformer 731, the dynamic field transformer 732, the field and time-aware sequential encoding transformer 733, any combination thereof, etc.) and / or parameters of one or more machine learning models 790 may be updated based on the loss. For example, the transformer management system 102 may update (e.g., adjust) parameters of one or more transformers and / or one or more machine learning models 790 based on back propagation (e.g., of one or more losses), gradient calculation (e.g., based on one or more losses), any combination thereof, etc.

[0171] In some non-limiting embodiments or aspects, this (re)training based on one or more outputs 791 of one or more machine learning models 790 may be referred to as fine-tuning.

[0172] In some non-limiting embodiments or aspects, method 200 may be implemented according to the following algorithm:

[0173] Algorithm 1:

[0174]

[0175]

[0176] Reference now Figure 3 , depicting a diagram of an example payment processing network 300 according to non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, the payment processing network 300 can be used in conjunction with and / or implemented in the systems, methods, and / or computer program products described herein. Figure 3As shown in , the payment processing network 300 may include a transaction processing system 301, a payment gateway system 302, a merchant system 304, an issuer system 306, an acquirer system 308 and / or a consumer device 310. In some non-limiting embodiments or aspects, Figure 1 Each of the converter management system 102, the transaction service provider system 104, and / or the user device 106 may be implemented by the transaction processing system 301 (e.g., a portion thereof). For example, the transaction service provider system 104 may be the same as or similar to the transaction processing system 301. In some non-limiting embodiments or aspects, Figure 1 At least one of the converter management system 102 and / or the user device 106 may be implemented by another system, another device, another group of systems, or another group of devices (e.g., merchant system 304, issuer system 306, acquirer system 308, consumer device 310, etc.) that is separate from or includes the transaction processing system 301 (e.g., a portion thereof). For example, the user device 106 may be implemented by at least one of the payment gateway system 302, the merchant system 304, the issuer system 306, the acquirer system 308, and / or the consumer device 310 (e.g., a portion thereof). Additionally or alternatively, for example, the converter management system 102 may be implemented by at least one of the payment gateway system 302, the merchant system 304, the issuer system 306, the acquirer system 308, and / or the consumer device 310 (e.g., a portion thereof).

[0177] The transaction processing system 301 may include one or more devices capable of receiving information from and / or transmitting information to (e.g., directly, indirectly, via public and / or private communication network connections, etc.) a payment gateway system 302, a merchant system 304, an issuer system 306, an acquirer system 308, a consumer device 310, etc. For example, Figure 3As shown in , the transaction processing system 301 may communicate with one or more issuer systems (e.g., issuer system 306), one or more acquirer systems (e.g., acquirer system 308), and / or one or more payment gateway systems (e.g., payment gateway system 302). Although only a single issuer system 306, a single acquirer system 308, and a single payment gateway system 302 are shown, it should be understood that the transaction processing system 301 may communicate with multiple issuer systems, multiple acquirer systems, and / or multiple payment gateways. In some non-limiting embodiments or aspects, the transaction processing system 301 may include a computing device, such as a server (e.g., a transaction processing server), a server group, and / or other similar devices. In some non-limiting embodiments or aspects, the transaction processing system 301 may communicate with a data storage device, which may be local or remote to the transaction processing system 301. In some non-limiting embodiments or aspects, the transaction processing system 301 may be able to receive information from the data storage device, store information in the data storage device, transmit information to the data storage device, or search for information stored in the data storage device. In some non-limiting embodiments or aspects, the transaction processing system 301 may be associated with a transaction service provider, as described herein. In some non-limiting embodiments or aspects, the transaction processing system 301 may also operate as an issuer system, such that the transaction processing system 301 and the issuer system 306 are a single system and / or controlled by a single entity.

[0178] The payment gateway system 302 may include one or more devices capable of receiving information from and / or transmitting information to (e.g., directly, indirectly, via public and / or private communication network connections, etc.) the transaction processing system 301, the merchant system 304, the issuer system 306, the acquirer system 308, the consumer device 310, etc. For example, Figure 3 As shown in , the payment gateway system 302 can communicate with one or more merchant systems (e.g., merchant system 304), one or more acquirer systems (e.g., acquirer system 308), and / or one or more transaction processing systems (e.g., transaction processing system 301). Although only a single merchant system 304, a single acquirer system 308, and a single transaction processing system 301 are shown, it should be understood that the payment gateway system 302 can communicate with multiple merchant systems, multiple acquirer systems, and / or multiple transaction processing systems. In some non-limiting embodiments or aspects, the payment gateway system 302 may include a computing device, such as a server, a server group, and / or other similar devices. In some non-limiting embodiments or aspects, the payment gateway system 302 may be associated with a payment gateway, as described herein.

[0179] Merchant system 304 may include one or more devices capable of receiving information from and / or transmitting information to (e.g., directly, indirectly, via public and / or private communication network connections, etc.) transaction processing system 301, payment gateway system 302, issuer system 306, acquirer system 308, consumer device 310, etc. For example, Figure 3 As shown in , the merchant system 304 can communicate with one or more payment gateway systems (e.g., payment gateway system 302), one or more acquirer systems (e.g., acquirer system 308), and / or one or more consumer devices (e.g., consumer device 310). Although only a single payment gateway system 302, a single acquirer system 308, and a single consumer device 310 are shown, it should be understood that the merchant system 304 can communicate with multiple payment gateway systems, multiple acquirer systems, and / or multiple consumer devices. In some non-limiting embodiments or aspects, the merchant system 304 may include a computing device, such as a server, a server group, a client device, a client device group, a POS device, a POS system, a computer, a computer system, a peripheral device, and / or other similar devices. In some non-limiting embodiments or aspects, the merchant system 304 may be associated with a merchant, as described herein. In some non-limiting embodiments or aspects, the merchant system 304 may include a computing device capable of communicating via a short-range communication connection (e.g., an NFC communication connection, an RFID communication connection, Communication connection, The merchant system 304 may include a device that receives information from the consumer device 310 and / or transmits information to it (e.g., a communication connection, etc.). In some non-limiting embodiments or aspects, the merchant system 304 may include one or more client devices. For example, the merchant system 304 may include a client device that allows the merchant to transmit information to the transaction processing system 301 (e.g., via at least one of the acquirer system 308 and / or the payment gateway system 302). In some non-limiting embodiments or aspects, the merchant system 304 (e.g., its client device, its POS device, etc.) may also operate as a payment gateway system, so that the merchant system 304 and the payment gateway system 302 are both a single system and / or controlled by a single entity.

[0180] Issuer system 306 may include one or more devices capable of receiving information and / or transmitting information to transaction processing system 301, payment gateway system 302, merchant system 304, acquirer system 308, consumer device 310, etc. (e.g., directly, indirectly, via public and / or private communication network connections, etc.). Figure 3As shown in , the issuer system 306 may communicate with one or more transaction processing systems (e.g., transaction processing system 301) and / or one or more consumer devices (e.g., consumer device 310). Although only a single transaction processing system 301 and a single consumer device 310 are shown, it should be understood that the issuer system 306 may communicate with multiple transaction processing systems and / or multiple consumer devices 310. In some non-limiting embodiments or aspects, the issuer system 306 may include a computing device, such as a server, a server group, and / or other similar devices. In some non-limiting embodiments or aspects, the issuer system 306 may be associated with an issuer organization, as described herein. For example, the issuer system 306 may be associated with an issuer organization that issues credit accounts, debit accounts, credit cards, debit cards, payment devices, etc. to users associated with the consumer devices 310.

[0181] Acquirer system 308 may include one or more devices capable of receiving information from and / or transmitting information to (e.g., directly, indirectly, via public and / or private communication network connections, etc.) transaction processing system 301, payment gateway system 302, merchant system 304, issuer system 306, consumer device 310, etc. For example, Figure 3 As shown in , the acquirer system 308 can communicate with one or more transaction processing systems (e.g., transaction processing system 301), one or more payment gateway systems (e.g., payment gateway system 302), and / or one or more merchant systems (e.g., merchant system 304). Although only a single transaction processing system 301, a single payment gateway system 302, and a single merchant system 304 are shown, it should be understood that the acquirer system 308 can communicate with multiple transaction processing systems, multiple payment gateway systems, and / or multiple merchant systems. In some non-limiting embodiments or aspects, the acquirer system 308 may include a computing device, such as a server, a server group, and / or other similar devices. In some non-limiting embodiments or aspects, the acquirer system 308 may be associated with an acquirer institution, as described herein.

[0182] Consumer device 310 may include one or more devices capable of receiving information from and / or transmitting information to transaction processing system 301, payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, etc. (e.g., directly, indirectly, via public and / or private communication network connections, etc.). Figure 3As shown in , the consumer device 310 may communicate with one or more merchant systems (e.g., merchant system 304) and / or one or more issuer systems (e.g., issuer system 306). Although only a single merchant system 304 and a single issuer system 306 are shown, it should be understood that the consumer device 310 may communicate with multiple merchant systems and / or multiple issuer systems. In some non-limiting embodiments or aspects, the consumer device 310 may be associated with a user who has been issued a credit account, a debit account, a credit card, a debit card, a payment device, etc. In some non-limiting embodiments or aspects, the user device 310 may include a computing device, such as a computer, a portable computer, a laptop computer, a tablet computer, a mobile device, a cellular phone, a smart phone, a wearable device (e.g., a watch, glasses, lenses, clothing, etc.), a PDA, a client device, and / or other similar devices. In some non-limiting embodiments or aspects, the user device 310 may include a payment device, as described herein. In some non-limiting embodiments or aspects, consumer device 310 may include a device capable of receiving information from other client devices 310 and / or transmitting information to them (e.g., directly, indirectly, via public and / or private communication network connections, short-range communication connections, etc.). In some non-limiting embodiments or aspects, consumer device 310 may include a device capable of communicating with merchant system 304 via a short-range communication connection (e.g., an NFC communication connection, an RFID communication connection, Communication connection, A device that receives information from and / or transmits information to merchant system 304 (e.g., a communication connection, etc.). In some non-limiting embodiments or aspects, consumer device 310 may include a client device.

[0183] In some non-limiting embodiments or aspects, the transaction processing system 301 may communicate directly with the merchant system 304 (e.g., via a public and / or private communication network connection, etc.). Additionally or alternatively, the transaction processing system 301 may communicate with the merchant system 304 through a payment gateway 302 and / or an acquirer system 308. In some non-limiting embodiments or aspects, an acquirer system 308 associated with the merchant system 304 may operate as a payment gateway 302 to facilitate the transmission of transaction messages (e.g., authorization requests) from the merchant system 304 to the transaction processing system 301. In some non-limiting embodiments or aspects, the merchant system 304 may communicate directly (e.g., via a public and / or private communication network connection, etc.) with the payment gateway 302. For example, a merchant system 304 including a physical POS device may communicate with the payment gateway 302 via a public or private network to conduct a card-present transaction. As another example, a merchant system 304 including a server (e.g., a web server) may communicate with the payment gateway 302 via a public or private network such as the Internet to conduct a card-not-present transaction.

[0184] For purposes of illustration, processing a transaction (e.g., a payment transaction) may include generating a transaction message (e.g., an authorization request, etc.) based on an account identifier of a customer (e.g., an account holder associated with a customer device 310, etc.) and / or transaction data associated with the transaction. For example, a merchant system 304 (e.g., a client device of the merchant system 304, a POS device of the merchant system 304, etc.) may initiate a transaction, for example, by generating an authorization request (e.g., in response to receiving an account identifier from a payment device and / or a portable financial device, etc. of a customer). The merchant system 304 may transmit the authorization request to the payment gateway 302 and / or the acquirer system 308. In some non-limiting embodiments or aspects, the payment gateway 302 may transmit the authorization request to the acquirer system 308 and / or the transaction processing system 301. Additionally or alternatively, the acquirer system 308 (and / or the payment gateway 302) may transmit the authorization request to the transaction processing system 301. After receiving an authorization request identifying a customer's account identifier (e.g., an account holder and / or account identifier associated with a consumer device 310) from a merchant system 304, the transaction processing system 301 may transmit the authorization request to an issuer system 306 (e.g., an issuer system that issued a payment device and / or account identifier). The issuer system 306 may determine an authorization decision (e.g., approval, rejection, etc.) based on the authorization request, and / or the issuer system 306 may generate an authorization response based on the authorization decision and / or the authorization request. The issuer system 306 may transmit the authorization response to the transaction processing system 301. The transaction processing system 301 may transmit the authorization response to an acquirer system 308 and / or a payment gateway 302. In some non-limiting embodiments or aspects, the acquirer system 308 may transmit the authorization response to the payment gateway 302 and / or the merchant system 304. Additionally or alternatively, the payment gateway 302 (and / or the acquirer system 308) may transmit the authorization response to the merchant system 304.

[0185] In some non-limiting embodiments or aspects, the transaction processing system 301 and / or the issuer system 306 may include at least one machine learning model (e.g., at least one of a fraud detection model, a risk detection model, a transaction authorization model, a credit approval model, a product recommendation model, a classifier model, an anomaly detection model, an authentication model, any combination thereof, etc.). For example, the one or more machine learning models may include at least one of the one or more transformers described herein and / or at least one of the one or more other machine learning models. For example, the transaction processing system 301 and / or the issuer system 306 may include the transformer management system 102, etc. The transaction processing system 301 and / or the issuer system 306 may perform at least one task (e.g., generate a prediction and / or generate an embedding) based on the authorization request and the one or more machine learning models. For example, performing the one or more tasks may include generating at least one prediction associated with fraud detection, risk detection, transaction authorization, credit approval, product recommendation, classification, anomaly detection, authentication, any combination thereof, etc. In some non-limiting embodiments or aspects, the transaction processing system 301 may transmit at least one message to the issuer system 306 (e.g., together with the authorization request) based on performing the task (e.g., generating a prediction and / or generating an embedding). In some non-limiting embodiments or aspects, the issuer system 306 may determine an authorization decision (e.g., approval, rejection, etc.) based on the authorization request and the performance of the task (e.g., generation of a prediction and / or generation of an embedding).

[0186] For purposes of illustration, clearing and / or settlement of a transaction may include generating a message (e.g., a clearing message, etc.) based on an account identifier of a customer (e.g., associated with a customer device 310, etc.) and / or transaction data associated with the transaction. For example, the merchant system 304 may generate at least one clearing message (e.g., a plurality of clearing messages, a batch of clearing messages, etc.). The merchant system 304 may transmit the one or more clearing messages to the acquirer system 308 (and / or the payment gateway 302, which may transmit the one or more clearing messages to the acquirer system 308). The acquirer system 308 may transmit the one or more clearing messages to the transaction processing system 301. The transaction processing system 301 may transmit the one or more clearing messages to the issuer system 306. The issuer system 306 may generate at least one settlement message based on the one or more clearing messages. In some non-limiting embodiments or aspects, the issuer system 306 may transmit one or more settlement messages and / or funds to the transaction processing system 301 (and / or a settlement bank system associated with the transaction processing system 301), and the transaction processing system 301 (and / or the settlement bank system) may transmit one or more settlement messages and / or funds to the acquirer system 308. Additionally or alternatively, the issuer system 306 may transmit one or more settlement messages and / or funds to the acquirer system 308. In some non-limiting embodiments or aspects, the acquirer system 308 may transmit one or more settlement messages and / or funds to the merchant system 304 (and / or an account associated with the merchant system 304).

[0187] Figure 3 The system and / or device may communicate via one or more wired and / or wireless communication networks. For example, one or more communication networks may include a cellular network (e.g., Long Term Evolution). networks, third generation (3G) networks, fourth generation (4G) networks, fifth generation (5G) networks, code division multiple access (CDMA) networks, etc.), public land mobile networks (PLMNs), local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), telephone networks (e.g., public switched telephone networks (PSTNs)), private networks (e.g., private networks associated with transaction service providers), temporary networks, intranets, the Internet, fiber-optic-based networks, cloud computing networks, etc., and / or combinations of these or other types of networks.

[0188] Provided as an example Figure 3 The number and arrangement of systems, devices, and / or networks shown are not intended to be construed as limiting the number and arrangement of systems, devices, and / or networks shown. Additional systems, devices, and / or networks may exist; fewer systems, devices, and / or networks; different systems, devices, and / or networks; and / or networks in a manner similar to that shown. Figure 3 The systems, devices and / or networks shown in the figure may be arranged in different ways. In addition, the system or device may be implemented in a single system or device. Figure 3Two or more systems or devices shown, or Figure 3 The single system or device shown may be implemented as multiple distributed systems or devices. Additionally or alternatively, a set of systems (e.g., one or more systems) or a set of devices (e.g., one or more devices) of the payment processing network 300 may perform one or more functions described as being performed by another set of systems or another set of devices of the payment processing network 300.

[0189] Reference Figure 4 , a diagram showing example components of an apparatus 400 according to a non-limiting embodiment. As an example, the apparatus 400 may correspond to Figure 1 The converter management system 102, the transaction service provider system 104 and / or the user device 106 and / or Figure 3 400, and / or a transaction processing system 301, a payment gateway system 302, a merchant system 304, an issuer system 306, an acquirer system 308, and / or a consumer device 310. In some non-limiting embodiments, such a system or device may include at least one device 400 and / or at least one component of the device 400. The number and arrangement of the components shown are provided as examples. In some non-limiting embodiments, the device 400 may include additional components, fewer components, different components, or components arranged in a different manner than those shown. Additionally or alternatively, a set of components (e.g., one or more components) of the device 400 may perform one or more functions described as being performed by another set of components of the device 400.

[0190] like Figure 4 As shown, the device 400 may include a bus 402, a processor 404, a memory 406, a storage component 408, an input component 410, an output component 412, and a communication interface 414. The bus 402 may include components that permit communication between components of the device 400. In some non-limiting embodiments, the processor 404 may be implemented in hardware, firmware, or a combination of hardware and software. For example, the processor 404 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component that can be programmed to perform a function (e.g., a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.). The memory 406 may include a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by the processor 404.

[0191] Continue to refer Figure 4, the storage component 408 may store information and / or software related to the operation and use of the device 400. For example, the storage component 408 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, a solid-state disk, etc.) and / or another type of computer-readable medium. The input component 410 may include a component that allows the device 400 to receive information, for example, through user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). In addition or alternatively, the input component 410 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.). The output component 412 may include a component that provides output information from the device 400 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.). The communication interface 414 may include a transceiver-like component (e.g., a transceiver, a separate receiver and a transmitter, etc.) that enables the device 400 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 414 may allow the device 400 to receive information from another device and / or provide information to another device. For example, the communication interface 414 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, interface, cellular network interface, and / or the like.

[0192] Device 400 can perform one or more processes described herein. Device 400 can perform these processes based on processor 404 executing software instructions stored by computer-readable media such as memory 406 and / or storage component 408. Computer-readable media can include any non-transient memory device. The memory device includes a memory space located inside a single physical storage device or a memory space spread over multiple physical storage devices. The software instructions can be read from another computer-readable medium or from another device to memory 406 and / or storage component 408 through communication interface 414. When executed, the software instructions stored in memory 406 and / or storage component 408 can cause processor 404 to perform one or more processes described herein. In addition or alternatively, hard-wired circuits can be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, the embodiments described herein are not limited to any specific combination of hardware circuits and software. As used herein, the term "configured to" can refer to the arrangement of software, equipment and / or hardware for performing and / or realizing one or more functions (e.g., actions, processes, steps of processes and / or the like). For example, "a processor configured to" may refer to a processor executing software instructions (eg, program code) that cause the processor to perform one or more functions.

[0193] Reference now FIG. 8A to FIG. 8C, a diagram illustrating the output of an example implementation of a system and method for a data type-aware transformer machine learning model for a sequential data set, according to some non-limiting embodiments or aspects.

[0194] like Fig. 8A As shown in , scatter plot 801 may include points for fraudulent transactions 811 and non-fraudulent transactions 812. For example, each point may represent the first three principal components (e.g., based on principal component analysis (PCA)) of a concatenated sequence embedding (e.g., based on a concatenated static sequence embedding representation and a dynamic sequence embedding representation) of a windowed transaction sequence (e.g., a transaction sequence including a given transaction and a selected number of previous transactions). These sequence embeddings may be generated after pre-training a transformer, as described herein (e.g., without explicitly fine-tuning a fraud detection model). Fig. 8A As shown in , there may be a relatively clear separation between fraudulent transactions 811 and non-fraudulent transactions 812, indicating that the sequence embedding has captured this information (e.g., even without fine-tuning).

[0195] like Figure 8B As shown in , the scatter plot 802 may include points for card-not-present (CNP) sequences 821 (e.g., sequences with CNP as the most frequent type of transaction) and non-CNP sequences 822 (e.g., sequences with non-CNP as the most frequent type of transaction). For example, each point may represent the first three principal components (e.g., based on PCA) of a concatenated sequence embedding (e.g., based on a concatenated static sequence embedding representation and a dynamic sequence embedding representation) of a windowed transaction sequence (e.g., a transaction sequence including a given transaction and a selected number of previous transactions). These sequence embeddings may be generated after pre-training a transformer, as described herein (e.g., without explicitly fine-tuning a classifier model). Figure 8B As shown in , there may be a relatively clear separation between CNP sequences 821 and non-CNP sequences 822, indicating that the sequence embedding has captured this information (eg, even without fine-tuning).

[0196] like Figure 8C As shown in , the scatter plot 803 may include points for abnormal transactions 831 and normal transactions 832. For example, each point may represent the first three principal components (e.g., based on PCA) of the concatenated sequence embeddings (e.g., based on the concatenated static sequence embedding representation and the dynamic sequence embedding representation) of the windowed transaction sequence (e.g., a transaction sequence including a given transaction and a selected number of previous transactions). These sequence embeddings can be generated after pre-training the transformer, as described herein (e.g., without explicitly fine-tuning the anomaly detection model). Figure 8C As shown in , there may be a relatively clear separation between abnormal transactions 831 and normal transactions 832, indicating that the sequence embedding has captured this information (e.g., even without fine-tuning).

[0197] Table 1 shows a comparison of the area under the curve (AUC) of implementations of the disclosed subject matter (e.g., FATA-Trans) and three other types of machine learning models (e.g., Light Gradient Boosting Machine (LightGBM), Recurrent Neural Network (RNN), and Tabular Bidirectional Encoder Representations from Transformers (TabBERT)) based on the same dataset (e.g., synthetic transaction dataset).

[0198] Table 1

[0199] Model AUC LightGBM 0.9624 RNN 0.9866 TabBERT 0.9985 FATA-Trans 0.9992

[0200] As shown in Table 1, FATA-Trans consistently outperforms other types of machine learning models. This indicates that FATA-Trans effectively captures more precise user behavior patterns by leveraging the time interval and field type information incorporated into the specially designed embedding and transformer layers.

[0201] Table 2 shows the comparison of pre-training time of FATA-Trans compared with TabBERT based on the same dataset (e.g., synthetic transaction dataset) and the same hardware setting.

[0202] Table 2

[0203] Model Pre-training time TabBERT 3d11h FATA-Trans 2d12h

[0204] As shown in Table 2, FATA-Trans demonstrates significantly shorter pre-training time compared to TabBERT. This can be attributed, at least in part, to the fact that FATA-Trans avoids redundant repetition of static fields in the sequence and only inputs the static fields into the static field transformer. As a result, FATA-Trans reduces memory usage and significantly saves training time.

[0205] Table 3 shows a comparison of the fraud detection task of FATA-Trans and TabBERT, as well as a modified version of TabBERT (e.g., TabBERT-TP) in which time-aware position embedding is utilized, and a modified version of FATA-Trans (e.g., FATA-Trans-SO) in which simple position embedding is utilized instead of time-aware position embedding. The fraud detection task is also performed on the original data without any pre-trained model (e.g., Not Applicable) for comparison. Two different types of classifier models are tested: Multi-layer Perception (MLP) and Bidirectional Long Short-term Memory (Bi-LSTM).

[0206] Table 3

[0207] Pre-training methods Classification Network Embed for use AUC not applicable MLP Original Field 0.8097 TabBERT MLP The last sequence embedding 0.8148 TabBERT Bi-LSTM Sequence Embedding 0.8884 TabBERT-TP MLP The last sequence embedding 0.8419 TabBERT-TP Bi-LSTM Sequence Embedding 0.8959 FATA-Trans-SO MLP The last sequence embedding 0.8199 FATA-Trans-SO Bi-LSTM Sequence Embedding 0.8944 FATA-Trans MLP The last sequence embedding 0.8319 FATA-Trans Bi-LSTM Sequence Embedding 0.8839

[0208] As shown in Table 3, FATA-Trans-SO, TabBERT-TP, and FATA-Trans show improvements in AUC scores compared to using the original field. Moreover, FATA-Trans-SO, TabBERT-TP, and FATA-Trans outperform the original TabBERT in most cases.

[0209] Although the embodiments have been described in detail for the purpose of illustration, it should be understood that such details are used for that purpose only, and the present disclosure is not limited to the disclosed embodiments or aspects, but on the contrary, is intended to cover modifications and equivalent arrangements within the spirit and scope of the appended claims. For example, it should be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect can be combined with one or more features of any other embodiment or aspect.

Claims

1. A computer-implemented method comprising: Receiving, with at least one processor, interaction data associated with a plurality of interactions, the interaction data comprising a plurality of interaction records, each interaction record in the plurality of interaction records comprising a plurality of fields, the plurality of fields comprising at least one static field and at least one dynamic field; generating, with at least one processor, a static interaction embedding representation based on inputting static field data associated with the at least one static field into a first transformer model; generating, with at least one processor, a plurality of dynamic interaction embedding representations based on inputting into a second transformer model dynamic field data associated with the at least one dynamic field of a sequence of interaction records, the sequence of interaction records comprising at least a subset of the plurality of interaction records; generating, with at least one processor, a first intermediate input based on the static interaction embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with the at least one static field; generating, with at least one processor, a plurality of second intermediate inputs based on each dynamic interaction embedding representation, a corresponding time-based embedding representation, and a second field type embedding representation associated with the at least one dynamic field; generating, using at least one processor, a static sequence embedding representation based on inputting the first intermediate input to a third transformer model; generating, with at least one processor, a plurality of dynamic sequence embedding representations based on inputting the plurality of second intermediate inputs to the third transformer model; as well as At least one prediction is generated, using at least one processor, based on inputting the static sequence embedding representation and the plurality of dynamic sequence embedding representations into a machine learning model.

2. The computer-implemented method of claim 1 , wherein generating the first intermediate input comprises: The static interaction embedding representation, the first time-based embedding representation, and the first field type embedding representation associated with the at least one static field are combined.

3. The computer-implemented method of claim 1 , wherein generating the plurality of dynamic interaction embedding representations comprises: generating a first dynamic interaction embedding associated with the first interaction record based on inputting first dynamic field data associated with the at least one dynamic field of the first interaction record into the second transformer model; as well as generating a second dynamic interaction embedding associated with the second interaction record based on inputting second dynamic field data associated with the at least one dynamic field of the second interaction record into the second transformer model; The computer-implemented method further comprises: generating a first time-based embedding representation associated with the first interaction record; and generating a second time-based embedded representation associated with the second interaction record; and Wherein generating the plurality of second intermediate inputs comprises: combining the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with the at least one dynamic field; and The second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with the at least one dynamic field are combined.

4. The computer-implemented method of claim 3, wherein combining the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with the at least one dynamic field comprises: summing the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with the at least one dynamic field; and Wherein combining the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with the at least one dynamic field comprises: The second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with the at least one dynamic field are summed.

5. The computer-implemented method of claim 1 , further comprising: for each interaction record of the plurality of interaction records, separating the static field data associated with the at least one static field from the dynamic field data associated with the at least one dynamic field; generating a first input to the first transformer model based on the static field data associated with the at least one static field; and A second input to the second transformer model is generated based on the dynamic field data associated with the at least one dynamic field.

6. The computer-implemented method of claim 1 , wherein the at least one dynamic field comprises a plurality of dynamic fields, the method further comprising: masking original values ​​of dynamic fields of a first interaction record in the sequence of interaction records to provide masked dynamic fields of the first interaction record before inputting dynamic field data associated with the plurality of dynamic fields of the sequence of interaction records into the second transformer model; wherein generating the plurality of dynamic interaction embedding representations based on inputting the dynamic field data associated with the plurality of dynamic fields of the interaction record sequence into the second transformer model comprises: generating the plurality of dynamic interaction embedding representations based on inputting the masked dynamic fields of the first interaction record into the second transformer model; and The computer-implemented method further comprises: The third transformer model is trained by comparing the data value of the data field represented by the dynamic sequence embedding associated with the first interaction record provided by the third transformer model with the original value of the dynamic field of the first interaction record, and adjusting the parameters of the third transformer model based on comparing the data value of the data field represented by the dynamic sequence embedding associated with the first interaction record provided by the third transformer with the original value of the dynamic field of the first interaction record.

7. The computer-implemented method of claim 1 , further comprising: An action associated with the fraud detection task is performed based on the at least one prediction.

8. A system comprising: At least one processor configured to: Receiving interaction data associated with a plurality of interactions, the interaction data comprising a plurality of interaction records, each of the plurality of interaction records comprising a plurality of fields, the plurality of fields comprising at least one static field and at least one dynamic field; generating a static interaction embedding representation based on inputting static field data associated with the at least one static field into a first transformer model; generating a plurality of dynamic interaction embedding representations based on inputting dynamic field data associated with the at least one dynamic field of a sequence of interaction records into a second transformer model, the sequence of interaction records comprising at least a subset of the plurality of interaction records; generating a first intermediate input based on the static interaction embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with the at least one static field; generating a plurality of second intermediate inputs based on each dynamic interaction embedding representation, the corresponding time-based embedding representation, and a second field type embedding representation associated with the at least one dynamic field; generating a static sequence embedding representation based on inputting the first intermediate input into a third transformer model; generating a plurality of dynamic sequence embedding representations based on inputting the plurality of second intermediate inputs to the third transformer model; as well as At least one prediction is generated based on inputting the static sequence embedding representation and the plurality of dynamic sequence embedding representations into a machine learning model.

9. The system of claim 8, wherein generating the first intermediate input comprises: The static interaction embedding representation, the first time-based embedding representation, and the first field type embedding representation associated with the at least one static field are combined.

10. The system of claim 8, wherein generating the plurality of dynamic interaction embedding representations comprises: generating a first dynamic interaction embedding associated with the first interaction record based on inputting first dynamic field data associated with the at least one dynamic field of the first interaction record into the second transformer model; as well as generating a second dynamic interaction embedding associated with the second interaction record based on inputting second dynamic field data associated with the at least one dynamic field of the second interaction record into the second transformer model; and wherein the at least one processor is further configured to: generating a first time-based embedding representation associated with the first interaction record; as well as generating a second time-based embedding representation associated with the second interaction record; and Wherein generating the plurality of second intermediate inputs comprises: combining the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with the at least one dynamic field; as well as The second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with the at least one dynamic field are combined.

11. The system of claim 10, wherein combining the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with the at least one dynamic field comprises: summing the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with the at least one dynamic field; and Wherein combining the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with the at least one dynamic field comprises: The second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with the at least one dynamic field are summed.

12. The system of claim 8, wherein the at least one processor is further configured to: for each interaction record of the plurality of interaction records, separating the static field data associated with the at least one static field from the dynamic field data associated with the at least one dynamic field; generating a first input to the first transformer model based on the static field data associated with the at least one static field; and A second input to the second transformer model is generated based on the dynamic field data associated with the at least one dynamic field.

13. The system of claim 8, wherein the at least one dynamic field comprises a plurality of dynamic fields, and wherein the at least one processor is further configured to: masking original values ​​of dynamic fields of a first interaction record in the sequence of interaction records to provide masked dynamic fields of the first interaction record before inputting dynamic field data associated with the plurality of dynamic fields of the sequence of interaction records into the second transformer model; wherein generating the plurality of dynamic interaction embedding representations based on inputting the dynamic field data associated with the plurality of dynamic fields of the interaction record sequence into the second transformer model comprises: generating the plurality of dynamic interaction embedding representations based on inputting the masked dynamic fields of the first interaction record into the second transformer model; and wherein the at least one processor is further configured to: The third transformer model is trained by comparing the data value of the data field represented by the dynamic sequence embedding associated with the first interaction record provided by the third transformer model with the original value of the dynamic field of the first interaction record, and adjusting the parameters of the third transformer model based on comparing the data value of the data field represented by the dynamic sequence embedding associated with the first interaction record provided by the third transformer with the original value of the dynamic field of the first interaction record.

14. The system of claim 8, wherein the at least one processor is further configured to: An action associated with the fraud detection task is performed based on the at least one prediction.

15. A computer program product comprising at least one non-transitory computer readable medium, the at least one non-transitory computer readable medium comprising program instructions that, when executed by at least one processor, cause the at least one processor to: Receiving interaction data associated with a plurality of interactions, the interaction data comprising a plurality of interaction records, each of the plurality of interaction records comprising a plurality of fields, the plurality of fields comprising at least one static field and at least one dynamic field; generating a static interaction embedding representation based on inputting static field data associated with the at least one static field into a first transformer model; generating a plurality of dynamic interaction embedding representations based on inputting dynamic field data associated with the at least one dynamic field of a sequence of interaction records into a second transformer model, the sequence of interaction records comprising at least a subset of the plurality of interaction records; generating a first intermediate input based on the static interaction embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with the at least one static field; generating a plurality of second intermediate inputs based on each dynamic interaction embedding representation, the corresponding time-based embedding representation, and a second field type embedding representation associated with the at least one dynamic field; generating a static sequence embedding representation based on inputting the first intermediate input into a third transformer model; generating a plurality of dynamic sequence embedding representations based on inputting the plurality of second intermediate inputs to the third transformer model; as well as At least one prediction is generated based on inputting the static sequence embedding representation and the plurality of dynamic sequence embedding representations into a machine learning model.

16. The computer program product of claim 15, wherein generating the first intermediate input comprises: The static interaction embedding representation, the first time-based embedding representation, and the first field type embedding representation associated with the at least one static field are combined.

17. The computer program product of claim 15, wherein generating the plurality of dynamic interactive embedding representations comprises: generating a first dynamic interaction embedding associated with the first interaction record based on inputting first dynamic field data associated with the at least one dynamic field of the first interaction record into the second transformer model; as well as generating a second dynamic interaction embedding associated with the second interaction record based on inputting second dynamic field data associated with the at least one dynamic field of the second interaction record into the second transformer model; wherein the instructions, when executed by the at least one processor, further cause the at least one processor to: generating a first time-based embedding representation associated with the first interaction record; as well as generating a second time-based embedding representation associated with the second interaction record; Wherein generating the plurality of second intermediate inputs comprises: combining the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with the at least one dynamic field; and combining the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with the at least one dynamic field; Wherein combining the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with the at least one dynamic field comprises: summing the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with the at least one dynamic field; and Wherein combining the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with the at least one dynamic field comprises: The second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with the at least one dynamic field are summed.

18. The computer program product of claim 15, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to: for each interaction record of the plurality of interaction records, separating the static field data associated with the at least one static field from the dynamic field data associated with the at least one dynamic field; generating a first input to the first transformer model based on the static field data associated with the at least one static field; and A second input to the second transformer model is generated based on the dynamic field data associated with the at least one dynamic field.

19. The computer program product of claim 15, wherein the at least one dynamic field comprises a plurality of dynamic fields, and wherein the instructions, when executed by the at least one processor, further cause the at least one processor to: masking original values ​​of dynamic fields of a first interaction record in the sequence of interaction records to provide masked dynamic fields of the first interaction record before inputting dynamic field data associated with the plurality of dynamic fields of the sequence of interaction records into the second transformer model; wherein generating the plurality of dynamic interaction embedding representations based on inputting the dynamic field data associated with the plurality of dynamic fields of the interaction record sequence into the second transformer model comprises: generating the plurality of dynamic interaction embedding representations based on inputting the masked dynamic fields of the first interaction record into the second transformer model; and wherein the instructions, when executed by the at least one processor, further cause the at least one processor to: The third transformer model is trained by comparing the data value of the data field represented by the dynamic sequence embedding associated with the first interaction record provided by the third transformer model with the original value of the dynamic field of the first interaction record, and adjusting the parameters of the third transformer model based on comparing the data value of the data field represented by the dynamic sequence embedding associated with the first interaction record provided by the third transformer with the original value of the dynamic field of the first interaction record.

20. The computer program product of claim 15, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to: An action associated with the fraud detection task is performed based on the at least one prediction.

Images