Methods, systems, and computer program products for providing type-aware transformers for sequential datasets

Abstract

Methods are provided that include receiving interaction data associated with a plurality of interactions, the interaction data including interaction records, the interaction records including a plurality of fields, the plurality of fields including static fields and dynamic fields; generating a static interaction embedding representation based on static field data associated with the static fields and a first transformer model; generating a plurality of dynamic interaction embedding representations based on dynamic field data associated with the dynamic fields of an interaction record sequence and a second transformer model; generating a first intermediate input and a plurality of second intermediate inputs; generating a static sequence embedding representation and a dynamic sequence embedding representation based on a third transformer model; and generating at least one prediction based on inputting the static sequence embedding representation and the plurality of dynamic sequence embedding representations to a machine learning model. Systems and computer program products are also disclosed.

Description

[0001] Cross-referencing related applications This application claims priority to U.S. Provisional Patent Application No. 63 / 454,089, filed March 23, 2023, the disclosure of which is hereby incorporated in its entirety. Technical Field

[0002] This disclosure generally relates to transformer machine learning models, and in some specific embodiments or aspects, to methods, systems, and computer program products for providing data type-aware transformer machine learning models for sequential datasets. Background Technology

[0003] Some machine learning models, such as neural networks (e.g., convolutional neural networks), can receive an input dataset that includes data points for training. After training the neural network, each data point in the training dataset may have a different effect on the neural network generated based on the trained neural network (e.g., the trained neural network). In some cases, the input dataset designed for the neural network may be independent and identically distributed. Independent and identically distributed input datasets can be used to determine the effect (e.g., influence) of each data point in the input dataset.

[0004] Transformer machine learning models can refer to deep learning models designed to process sequential input data and incorporating a self-attention mechanism that assigns different weights to the importance of each part of the sequential input data. Transformer machine learning models can process the entire input data at once, and the attention mechanism can provide context at any point in the sequential input data.

[0005] Embeddings (e.g., neural embeddings) can refer to a relatively low-dimensional space to which a high-dimensional vector, such as a feature vector, can be transformed. In some examples, embeddings can include vectors having values ​​that represent the semantic and syntactic relationships of sequential input data by placing semantically similar inputs more closely together in the embedding space. In some instances, embeddings can improve the performance of machine learning techniques on large inputs, such as sparse vectors representing words. For example, embeddings can be learned and reused on machine learning models.

[0006] However, if the input data type (e.g., dynamic and static data types) of the sequentially input data is provided to the transducer machine learning model and is not properly considered, then immutable information can be copied onto each data instance in the sequentially input data. Therefore, static information can consume significant resources (e.g., memory space, processing power, etc.) of the transducer machine learning model. Furthermore, static information can reduce the difficulty during model training (e.g., pre-training) tasks to the point that it can prevent the transducer machine learning model from learning meaningful representations of the sequentially input data. Summary of the Invention

[0007] Therefore, improved methods, systems, and computer program products are provided for providing data type-aware transformer machine learning models for sequential datasets (e.g., overcoming some or all of the disadvantages identified above).

[0008] According to some non-limiting embodiments or aspects, a computer-implemented method is provided for providing a data type-aware transformer machine learning model for a sequential dataset. Example methods may include receiving interaction data associated with multiple interactions, the interaction data including multiple interaction records. Each of the multiple interaction records may include multiple fields, the multiple fields including at least one static field and at least one dynamic field. A static interaction embedding representation may be generated based on inputting static field data associated with at least one static field into a first transformer model. Multiple dynamic interaction embedding representations may be generated based on inputting dynamic field data associated with at least one dynamic field of a sequence of interaction records into a second transformer model. The sequence of interaction records may include at least a subset of the multiple interaction records. A first intermediate input may be generated based on the static interaction embedding representation, a first time-based embedding representation, and a first field-type embedding representation associated with at least one static field. Multiple second intermediate inputs may be generated based on each dynamic interaction embedding representation, a corresponding time-based embedding representation, and a second field-type embedding representation associated with at least one dynamic field. A static sequence embedding representation may be generated based on inputting the first intermediate input into a third transformer model. Multiple dynamic sequence embedding representations can be generated by feeding multiple second intermediate inputs into a third transformer model. At least one prediction can be generated by feeding static sequence embedding representations and multiple dynamic sequence embedding representations into a machine learning model.

[0009] In some non-limiting embodiments or aspects, generating the first intermediate input may include combining a static interactive embedded representation, a first time-based embedded representation, and a first field type embedded representation associated with at least one static field.

[0010] In some non-limiting embodiments or aspects, generating multiple dynamic interaction embedding representations may include: generating a first dynamic interaction embedding associated with a first interaction record by inputting first dynamic field data associated with at least one dynamic field of a first interaction record into a second transformer model; and / or generating a second dynamic interaction embedding associated with a second interaction record by inputting second dynamic field data associated with at least one dynamic field of a second interaction record into a second transformer model. A first time-based embedding representation associated with the first interaction record may be generated, and / or a second time-based embedding representation associated with the second interaction record may be generated. Generating multiple second intermediate inputs may include: combining the first dynamic interaction embedding, the first time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field; and / or combining the second dynamic interaction embedding, the second time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field.

[0011] In some non-limiting embodiments or aspects, combining the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field may include summing the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field. In some non-limiting embodiments or aspects, combining the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field may include summing the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field.

[0012] In some non-limiting embodiments or aspects, for each of a plurality of interaction records, static field data associated with at least one static field may be separated from dynamic field data associated with at least one dynamic field. A first input to a first converter model may be generated based on the static field data associated with at least one static field. A second input to a second converter model may be generated based on the dynamic field data associated with at least one dynamic field.

[0013] In some non-limiting embodiments or aspects, at least one dynamic field may include multiple dynamic fields. Before inputting dynamic field data associated with multiple dynamic fields of the interaction record sequence into the second transformer model, the original values ​​of the dynamic fields of the first interaction record in the interaction record sequence may be masked to provide masked dynamic fields of the first interaction record. Generating multiple dynamic interaction embedding representations based on inputting dynamic field data associated with multiple dynamic fields of the interaction record sequence into the second transformer model may include generating multiple dynamic interaction embedding representations based on inputting the masked dynamic fields of the first interaction record into the second transformer model. The third transformer model can be trained by comparing the data values ​​of the data fields of the dynamic sequence embedding representation associated with the first interaction record provided by the third transformer model with the original values ​​of the dynamic fields of the first interaction record, and by adjusting the parameters of the third transformer model based on the comparison of the data values ​​of the data fields of the dynamic sequence embedding representation associated with the first interaction record provided by the third transformer model with the original values ​​of the dynamic fields of the first interaction record.

[0014] In some non-limiting embodiments or aspects, actions associated with a fraud detection task may be performed based on at least one prediction.

[0015] According to a non-limiting embodiment or aspect, a system is provided for providing a data type-aware transformer machine learning model for a sequential dataset. An example system may include at least one processor configured to receive interaction data associated with multiple interactions, the interaction data including multiple interaction records. Each of the multiple interaction records may include multiple fields, the multiple fields including at least one static field and at least one dynamic field. A static interaction embedding representation may be generated based on inputting static field data associated with at least one static field into a first transformer model. Multiple dynamic interaction embedding representations may be generated based on inputting dynamic field data associated with at least one dynamic field of a sequence of interaction records into a second transformer model. The sequence of interaction records may include at least a subset of the multiple interaction records. A first intermediate input may be generated based on the static interaction embedding representation, a first time-based embedding representation, and a first field-type embedding representation associated with at least one static field. Multiple second intermediate inputs may be generated based on each dynamic interaction embedding representation, a corresponding time-based embedding representation, and a second field-type embedding representation associated with at least one dynamic field. A static sequence embedding representation may be generated based on inputting the first intermediate input into a third transformer model. Multiple dynamic sequence embedding representations can be generated by feeding multiple second intermediate inputs into a third transformer model. At least one prediction can be generated by feeding static sequence embedding representations and multiple dynamic sequence embedding representations into a machine learning model.

[0016] In some non-limiting embodiments or aspects, generating the first intermediate input may include combining a static interactive embedded representation, a first time-based embedded representation, and a first field type embedded representation associated with at least one static field.

[0017] In some non-limiting embodiments or aspects, generating multiple dynamic interaction embedding representations may include: generating a first dynamic interaction embedding associated with a first interaction record by inputting first dynamic field data associated with at least one dynamic field of a first interaction record into a second transformer model; and / or generating a second dynamic interaction embedding associated with a second interaction record by inputting second dynamic field data associated with at least one dynamic field of a second interaction record into a second transformer model. A first time-based embedding representation associated with the first interaction record may be generated, and / or a second time-based embedding representation associated with the second interaction record may be generated. Generating multiple second intermediate inputs may include: combining the first dynamic interaction embedding, the first time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field; and / or combining the second dynamic interaction embedding, the second time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field.

[0018] In some non-limiting embodiments or aspects, combining the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field may include summing the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field. In some non-limiting embodiments or aspects, combining the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field may include summing the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field.

[0019] In some non-limiting embodiments or aspects, for each of a plurality of interaction records, static field data associated with at least one static field may be separated from dynamic field data associated with at least one dynamic field. A first input to a first converter model may be generated based on the static field data associated with at least one static field. A second input to a second converter model may be generated based on the dynamic field data associated with at least one dynamic field.

[0020] In some non-limiting embodiments or aspects, at least one dynamic field may include multiple dynamic fields. Before inputting dynamic field data associated with multiple dynamic fields of the interaction record sequence into the second transformer model, the original values ​​of the dynamic fields of the first interaction record in the interaction record sequence may be masked to provide masked dynamic fields of the first interaction record. Generating multiple dynamic interaction embedding representations based on inputting dynamic field data associated with multiple dynamic fields of the interaction record sequence into the second transformer model may include generating multiple dynamic interaction embedding representations based on inputting the masked dynamic fields of the first interaction record into the second transformer model. The third transformer model can be trained by comparing the data values ​​of the data fields of the dynamic sequence embedding representation associated with the first interaction record provided by the third transformer model with the original values ​​of the dynamic fields of the first interaction record, and by adjusting the parameters of the third transformer model based on the comparison of the data values ​​of the data fields of the dynamic sequence embedding representation associated with the first interaction record provided by the third transformer model with the original values ​​of the dynamic fields of the first interaction record.

[0021] In some non-limiting embodiments or aspects, actions associated with a fraud detection task may be performed based on at least one prediction.

[0022] According to some non-limiting embodiments or aspects, a computer program product is provided for providing a data type-aware transformer machine learning model for a sequential dataset. An example computer program product may include at least one non-transient computer-readable medium comprising program instructions that, when executed by at least one processor, cause the at least one processor to receive interaction data associated with a plurality of interactions, the interaction data including a plurality of interaction records. Each of the plurality of interaction records may include a plurality of fields, the plurality of fields including at least one static field and at least one dynamic field. A static interaction embedding representation may be generated based on inputting static field data associated with at least one static field into a first transformer model. A plurality of dynamic interaction embedding representations may be generated based on inputting dynamic field data associated with at least one dynamic field of a sequence of interaction records into a second transformer model. The sequence of interaction records may include at least a subset of the plurality of interaction records. A first intermediate input may be generated based on the static interaction embedding representation, a first time-based embedding representation, and a first field-type embedding representation associated with at least one static field. A plurality of second intermediate inputs may be generated based on each dynamic interaction embedding representation, a corresponding time-based embedding representation, and a second field-type embedding representation associated with at least one dynamic field. A static sequence embedding representation can be generated by inputting a first intermediate input into a third transformer model. Multiple dynamic sequence embedding representations can be generated by inputting multiple second intermediate inputs into a third transformer model. At least one prediction can be generated by inputting the static sequence embedding representation and multiple dynamic sequence embedding representations into a machine learning model.

[0023] In some non-limiting embodiments or aspects, generating the first intermediate input may include combining a static interactive embedded representation, a first time-based embedded representation, and a first field type embedded representation associated with at least one static field.

[0024] In some non-limiting embodiments or aspects, generating multiple dynamic interaction embedding representations may include: generating a first dynamic interaction embedding associated with a first interaction record by inputting first dynamic field data associated with at least one dynamic field of a first interaction record into a second transformer model; and / or generating a second dynamic interaction embedding associated with a second interaction record by inputting second dynamic field data associated with at least one dynamic field of a second interaction record into a second transformer model. A first time-based embedding representation associated with the first interaction record may be generated, and / or a second time-based embedding representation associated with the second interaction record may be generated. Generating multiple second intermediate inputs may include: combining the first dynamic interaction embedding, the first time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field; and / or combining the second dynamic interaction embedding, the second time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field.

[0025] In some non-limiting embodiments or aspects, combining the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field may include summing the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field. In some non-limiting embodiments or aspects, combining the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field may include summing the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field.

[0026] In some non-limiting embodiments or aspects, for each of a plurality of interaction records, static field data associated with at least one static field may be separated from dynamic field data associated with at least one dynamic field. A first input to a first converter model may be generated based on the static field data associated with at least one static field. A second input to a second converter model may be generated based on the dynamic field data associated with at least one dynamic field.

[0027] In some non-limiting embodiments or aspects, at least one dynamic field may include multiple dynamic fields. Before inputting dynamic field data associated with multiple dynamic fields of the interaction record sequence into the second transformer model, the original values ​​of the dynamic fields of the first interaction record in the interaction record sequence may be masked to provide masked dynamic fields of the first interaction record. Generating multiple dynamic interaction embedding representations based on inputting dynamic field data associated with multiple dynamic fields of the interaction record sequence into the second transformer model may include generating multiple dynamic interaction embedding representations based on inputting the masked dynamic fields of the first interaction record into the second transformer model. The third transformer model can be trained by comparing the data values ​​of the data fields of the dynamic sequence embedding representation associated with the first interaction record provided by the third transformer model with the original values ​​of the dynamic fields of the first interaction record, and by adjusting the parameters of the third transformer model based on the comparison of the data values ​​of the data fields of the dynamic sequence embedding representation associated with the first interaction record provided by the third transformer model with the original values ​​of the dynamic fields of the first interaction record.

[0028] In some non-limiting embodiments or aspects, actions associated with a fraud detection task may be performed based on at least one prediction.

[0029] Other non-limiting embodiments or aspects are described in the following numbered entries: Clause 1: A computer-implemented method comprising: receiving, using at least one processor, interaction data associated with a plurality of interactions, the interaction data including a plurality of interaction records, each of the plurality of interaction records including a plurality of fields, the plurality of fields including at least one static field and at least one dynamic field; generating a static interaction embedding representation using at least one processor based on inputting static field data associated with at least one static field into a first transformer model; generating a plurality of dynamic interaction embedding representations using at least one processor based on inputting dynamic field data associated with at least one dynamic field of a sequence of interaction records into a second transformer model, the sequence of interaction records including at least a subset of the plurality of interaction records; and generating a static interaction embedding representation using at least one processor based on the static interaction embedding data. The method involves generating a first intermediate input using an input representation, a first time-based embedding representation, and a first field-type embedding representation associated with at least one static field; generating multiple second intermediate inputs using at least one processor based on each dynamic interactive embedding representation, a corresponding time-based embedding representation, and a second field-type embedding representation associated with at least one dynamic field; generating a static sequence embedding representation using at least one processor based on inputting the first intermediate inputs into a third transformer model; generating multiple dynamic sequence embedding representations using at least one processor based on inputting the multiple second intermediate inputs into a third transformer model; and generating at least one prediction using at least one processor based on inputting the static sequence embedding representations and the multiple dynamic sequence embedding representations into a machine learning model.

[0030] Clause 2: The computer-implemented method according to Clause 1, wherein generating the first intermediate input includes: combining a static interactive embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with at least one static field.

[0031] Clause 3: The computer-implemented method according to Clause 1 or Clause 2, wherein generating a plurality of dynamic interaction embedding representations includes: generating a first dynamic interaction embedding associated with a first interaction record based on inputting first dynamic field data associated with at least one dynamic field of a first interaction record into a second transformer model; and generating a second dynamic interaction embedding associated with a second interaction record based on inputting second dynamic field data associated with at least one dynamic field of a second interaction record into a second transformer model; the computer-implemented method further includes: generating a first time-based embedding representation associated with the first interaction record; and generating a second time-based embedding representation associated with the second interaction record; and wherein generating a plurality of second intermediate inputs includes: combining the first dynamic interaction embedding, the first time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field; and combining the second dynamic interaction embedding, the second time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field.

[0032] Clause 4: A computer-implemented method according to any one of Clauses 1 to 3, wherein combining a first dynamic interaction embedding, a first time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field comprises: summing the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field; and wherein combining a second dynamic interaction embedding, a second time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field comprises: summing the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field.

[0033] Clause 5: The computer-implemented method according to any one of Clauses 1 to 4 further comprises: for each of a plurality of interaction records, separating static field data associated with at least one static field from dynamic field data associated with at least one dynamic field; generating a first input to a first converter model based on the static field data associated with at least one static field; and generating a second input to a second converter model based on the dynamic field data associated with at least one dynamic field.

[0034] Clause 6: A computer-implemented method according to any one of Clauses 1 to 5, wherein at least one dynamic field comprises a plurality of dynamic fields, the method further comprising: masking the original values ​​of the dynamic fields of a first interactive record in the interactive record sequence to provide a masked dynamic field of the first interactive record before inputting dynamic field data associated with the plurality of dynamic fields of the interactive record sequence into a second transformer model; wherein generating a plurality of dynamic interactive embedding representations based on inputting the dynamic field data associated with the plurality of dynamic fields of the interactive record sequence into the second transformer model comprises: generating a plurality of dynamic interactive embedding representations based on inputting the masked dynamic field of the first interactive record into the second transformer model; and wherein the computer-implemented method further comprises: training a third transformer model by comparing the data values ​​of the data fields of the dynamic sequence embedding representations associated with the first interactive record provided by a third transformer model with the original values ​​of the dynamic fields of the first interactive record, and by adjusting the parameters of the third transformer model based on comparing the data values ​​of the data fields of the dynamic sequence embedding representations associated with the first interactive record provided by the third transformer model with the original values ​​of the dynamic fields of the first interactive record.

[0035] Article 7: The computer-implemented method according to any one of Articles 1 to 6 further includes: performing an action associated with a fraud detection task based on at least one prediction.

[0036] Clause 8: A system comprising: at least one processor configured to: receive interaction data associated with a plurality of interactions, the interaction data including a plurality of interaction records, each of the plurality of interaction records including a plurality of fields, the plurality of fields including at least one static field and at least one dynamic field; generate a static interaction embedding representation based on inputting static field data associated with at least one static field into a first transformer model; generate a plurality of dynamic interaction embedding representations based on inputting dynamic field data associated with at least one dynamic field of a sequence of interaction records into a second transformer model, the sequence of interaction records including at least a subset of the plurality of interaction records; generate a first intermediate input based on the static interaction embedding representation, a first time-based embedding representation, and a first field-type embedding representation associated with at least one static field; generate a plurality of second intermediate inputs based on each dynamic interaction embedding representation, a corresponding time-based embedding representation, and a second field-type embedding representation associated with at least one dynamic field; generate a static sequence embedding representation based on inputting the first intermediate input into a third transformer model; generate a plurality of dynamic sequence embedding representations based on inputting the plurality of second intermediate inputs into the third transformer model; and generate at least one prediction based on inputting the static sequence embedding representation and the plurality of dynamic sequence embedding representations into a machine learning model.

[0037] Clause 9: The system according to Clause 8, wherein generating the first intermediate input includes: combining a static interactive embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with at least one static field.

[0038] Clause 10: The system according to Clause 8 or Clause 9, wherein generating a plurality of dynamic interaction embedding representations includes: generating a first dynamic interaction embedding associated with a first interaction record based on inputting first dynamic field data associated with at least one dynamic field of a first interaction record into a second transformer model; and generating a second dynamic interaction embedding associated with a second interaction record based on inputting second dynamic field data associated with at least one dynamic field of a second interaction record into a second transformer model; and wherein at least one processor is further configured to: generate a first time-based embedding representation associated with the first interaction record; and generate a second time-based embedding representation associated with the second interaction record; and wherein generating a plurality of second intermediate inputs includes: combining the first dynamic interaction embedding, the first time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field; and combining the second dynamic interaction embedding, the second time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field.

[0039] Clause 11: A system according to any one of Clauses 8 to 10, wherein combining the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field comprises: summing the first dynamic interaction embedding, the first time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field; and wherein combining the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field comprises: summing the second dynamic interaction embedding, the second time-based embedding representation, and the second field type embedding representation associated with at least one dynamic field.

[0040] Clause 12: The system according to any one of Clauses 8 to 11, wherein at least one processor is further configured to: for each of a plurality of interaction records, separate static field data associated with at least one static field from dynamic field data associated with at least one dynamic field; generate a first input to a first converter model based on the static field data associated with at least one static field; and generate a second input to a second converter model based on the dynamic field data associated with at least one dynamic field.

[0041] Clause 13: A system according to any one of Clauses 8 to 12, wherein at least one dynamic field comprises a plurality of dynamic fields, and wherein at least one processor is further configured to: mask the original values ​​of the dynamic fields of a first interactive record in the interactive record sequence to provide a masked dynamic field of the first interactive record before inputting dynamic field data associated with the plurality of dynamic fields of the interactive record sequence into a second transformer model; wherein generating a plurality of dynamic interactive embedding representations based on inputting the dynamic field data associated with the plurality of dynamic fields of the interactive record sequence into the second transformer model comprises: generating a plurality of dynamic interactive embedding representations based on inputting the masked dynamic field of the first interactive record into the second transformer model; and wherein at least one processor is further configured to: train a third transformer model by comparing the data values ​​of the data fields of the dynamic sequence embedding representations associated with the first interactive record provided by a third transformer model with the original values ​​of the dynamic fields of the first interactive record, and by adjusting the parameters of the third transformer model based on comparing the data values ​​of the data fields of the dynamic sequence embedding representations associated with the first interactive record provided by the third transformer model with the original values ​​of the dynamic fields of the first interactive record.

[0042] Clause 14: A system according to any one of Clauses 8 to 13, wherein at least one processor is further configured to perform an action associated with a fraud detection task based on at least one prediction.

[0043] Clause 15: A computer program product comprising at least one non-transient computer-readable medium, the at least one non-transient computer-readable medium comprising program instructions, the program instructions, when executed by at least one processor, causing at least one processor to: receive interaction data associated with a plurality of interactions, the interaction data comprising a plurality of interaction records, each of the plurality of interaction records comprising a plurality of fields, the plurality of fields comprising at least one static field and at least one dynamic field; generate a static interaction embedding representation based on inputting static field data associated with at least one static field into a first transformer model; and generate a plurality of dynamic interaction embeddings based on inputting dynamic field data associated with at least one dynamic field of a sequence of interaction records into a second transformer model. The input represents an interaction record sequence comprising at least a subset of multiple interaction records; a first intermediate input is generated based on a static interaction embedding representation, a first time-based embedding representation, and a first field-type embedding representation associated with at least one static field; multiple second intermediate inputs are generated based on each dynamic interaction embedding representation, a corresponding time-based embedding representation, and a second field-type embedding representation associated with at least one dynamic field; a static sequence embedding representation is generated based on inputting the first intermediate inputs into a third transformer model; multiple dynamic sequence embedding representations are generated based on inputting the multiple second intermediate inputs into the third transformer model; and at least one prediction is generated based on inputting the static sequence embedding representations and the multiple dynamic sequence embedding representations into a machine learning model.

[0044] Clause 16: The computer program product according to Clause 15, wherein generating the first intermediate input includes: combining a static interactive embedded representation, a first time-based embedded representation, and a first field type embedded representation associated with at least one static field.

[0045] Clause 17: A computer program product according to Clause 15 or Clause 16, wherein generating a plurality of dynamic interactive embedding representations includes: generating a first dynamic interactive embedding associated with a first interactive record based on inputting first dynamic field data associated with at least one dynamic field of a first interactive record into a second transformer model; and generating a second dynamic interactive embedding associated with a second interactive record based on inputting second dynamic field data associated with at least one dynamic field of a second interactive record into a second transformer model; wherein the instructions, when executed by at least one processor, further cause at least one processor to: generate a first time-based embedding representation associated with the first interactive record; and generate a second time-based embedding representation associated with the second interactive record; wherein generating a plurality of second intermediate inputs includes: combining the first dynamic interactive embedding, a first base The method comprises: a time-based embedding representation and a second field-type embedding representation associated with at least one dynamic field; and combining a second dynamic interaction embedding, a second time-based embedding representation, and a second field-type embedding representation associated with at least one dynamic field; wherein combining a first dynamic interaction embedding, a first time-based embedding representation, and a second field-type embedding representation associated with at least one dynamic field includes: summing the first dynamic interaction embedding, the first time-based embedding representation, and the second field-type embedding representation associated with at least one dynamic field; and wherein combining a second dynamic interaction embedding, a second time-based embedding representation, and a second field-type embedding representation associated with at least one dynamic field includes: summing the second dynamic interaction embedding, the second time-based embedding representation, and the second field-type embedding representation associated with at least one dynamic field.

[0046] Clause 18: A computer program product according to any one of Clauses 15 to 17, wherein the instructions, when executed by at least one processor, further cause at least one processor to: for each of a plurality of interaction records, separate static field data associated with at least one static field from dynamic field data associated with at least one dynamic field; generate a first input to a first converter model based on the static field data associated with at least one static field; and generate a second input to a second converter model based on the dynamic field data associated with at least one dynamic field.

[0047] Clause 19: A computer program product according to any one of Clauses 15 to 18, wherein at least one dynamic field comprises a plurality of dynamic fields, and wherein the instructions, when executed by at least one processor, further cause at least one processor to: mask the original values ​​of the dynamic fields of a first interactive record in the interactive record sequence to provide a masked dynamic field of the first interactive record before inputting dynamic field data associated with the plurality of dynamic fields of the interactive record sequence into a second transformer model; wherein generating a plurality of dynamic interactive embedding representations based on inputting the dynamic field data associated with the plurality of dynamic fields of the interactive record sequence into the second transformer model comprises: generating a plurality of dynamic interactive embedding representations based on inputting the masked dynamic field of the first interactive record into the second transformer model; and wherein the instructions, when executed by at least one processor, further cause at least one processor to: train a third transformer model by comparing the data values ​​of the data fields of the dynamic sequence embedding representations associated with the first interactive record provided by a third transformer model with the original values ​​of the dynamic fields of the first interactive record, and by adjusting the parameters of the third transformer model based on comparing the data values ​​of the data fields of the dynamic sequence embedding representations associated with the first interactive record provided by the third transformer model with the original values ​​of the dynamic fields of the first interactive record.

[0048] Article 20: A computer program product according to any one of Articles 15 to 19, wherein the instructions, when executed by at least one processor, also cause at least one processor to: perform an action associated with a fraud detection task based on at least one prediction.

[0049] These and other features and characteristics of this disclosure, as well as the operational methods and manufacturing economies of combinations of related structural elements and parts, will become more apparent when considered in conjunction with the accompanying drawings, all of which form part of this specification, wherein similar reference numerals denote corresponding parts in the figures. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to be a definition of limitation on the disclosed subject matter. Attached Figure Description

[0050] Additional advantages and details are explained in more detail below with reference to the non-limiting exemplary embodiments or aspects shown in the illustrative accompanying drawings, in which: Figure 1 It is a schematic diagram of a system of data type-aware transformer machine learning models for sequential datasets according to some non-limiting embodiments or aspects. Figure 2 This is a flowchart of a method for a data type-aware transformer machine learning model for a sequential dataset, according to some non-limiting embodiments or aspects. Figure 3This is a schematic diagram of an example payment processing network from which the methods, systems and / or computer program products described herein may be implemented, according to some non-limiting embodiments or aspects; Figure 4 Based on some non-limiting embodiments or aspects Figure 1 and / or Figure 3 A schematic diagram of example components of one or more devices; Figure 5A It is a schematic diagram of an example sequence dataset based on some non-limiting embodiments or aspects; Figure 5B It is a schematic diagram of example sequence datasets and example embeddings based on some non-limiting embodiments or aspects; Figure 6 This is a schematic diagram of an example implementation of a system and method for a data type-aware transformer machine learning model for sequential datasets, according to some non-limiting embodiments or aspects. Figure 7A and Figure 7B These are schematic diagrams illustrating example implementations of a data type-aware transformer machine learning model for sequential datasets, based on some non-limiting embodiments or aspects; and Figures 8A to 8C This is a diagram showing the output of an example implementation of a system and method for a data type-aware transformer machine learning model for sequential datasets, based on some non-limiting embodiments or aspects. Detailed Implementation

[0051] For the purposes of the following description, the terms “end,” “upper,” “lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” “longitudinal,” and their derivatives should be associated with the orientation of the embodiments in the accompanying drawings. However, it should be understood that various alternative variations and sequences of steps may be employed in this disclosure, except where explicitly specified otherwise. It should also be understood that the specific devices and processes shown in the accompanying drawings and described in the following specification are merely exemplary and non-limiting embodiments or aspects of the disclosed subject matter. Therefore, specific dimensions and other physical characteristics relating to the embodiments or aspects disclosed herein should not be considered as limitations.

[0052] This document describes some non-limiting embodiments or aspects in conjunction with thresholds. As used herein, satisfying a threshold can refer to a value greater than a threshold, more than a threshold, higher than a threshold, greater than or equal to a threshold, less than a threshold, less than a threshold, lower than a threshold, less than or equal to a threshold, equal to a threshold, etc.

[0053] The aspects, components, elements, structures, actions, steps, functions, instructions, and / or their likenesses used herein should not be construed as critical or essential unless explicitly stated otherwise. Furthermore, as used herein, the article “a” is intended to include one or more items and is interchangeable with “one or more” and “at least one.” Additionally, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and is interchangeable with “one or more” or “at least one.” The term “a” or similar language is used where only one item is desired. Moreover, as used herein, the terms “having” and / or their likenesses are intended to be open-ended terms. Additionally, unless explicitly stated otherwise, the phrase “based on” is intended to mean “at least partially based on.” Furthermore, a reference to an action “based on” a condition may mean that the action is “in response to” that condition. For example, in some non-limiting embodiments or aspects, the phrases “based on” and “in response to” may refer to a condition that automatically triggers an action (e.g., a specific operation of an electronic device, such as a computing device, processor, and / or its likenesses).

[0054] As used herein, the term "acquiring institution" can refer to an entity licensed and / or approved by a transaction service provider to initiate transactions (e.g., payment transactions) using payment devices associated with the transaction service provider. Transactions that an acquiring institution can initiate can include payment transactions (e.g., purchases, original letter of credit transactions (OCT), account treasury transactions (AFT), and / or the like). In some non-limiting embodiments or aspects, the acquiring institution can be a financial institution, such as a bank. As used herein, the term "acquiring system" can refer to one or more computing devices operated by or on behalf of an acquiring institution, such as a server computer executing one or more software applications.

[0055] As used herein, the term "account identifier" may include one or more master account (PAN), payment tokens, or other identifiers associated with a customer account. The term "payment token" may refer to an identifier used as a substitute or alternative identifier to an original account identifier such as a PAN. An account identifier may be any combination of alphanumeric or characters and / or symbols. A payment token may be associated with a PAN or other original account identifier in one or more data structures (e.g., one or more databases, etc.) such that the payment token can be used to conduct transactions without directly using the original account identifier. In some instances, an original account identifier such as a PAN may be associated with multiple payment tokens used for different individuals or purposes.

[0056] As used herein, the terms "client" and "client device" can refer to one or more client-side devices or systems (e.g., remote from the transaction service provider) used to initiate or facilitate a transaction (e.g., a payment transaction). As examples, "client device" can refer to one or more POS devices used by a merchant, one or more acquiring host computers used by an acquiring party, one or more mobile devices used by a user, etc. In some non-limiting embodiments or aspects, a client device can be an electronic device configured to communicate with one or more networks and initiate or facilitate transactions. For example, a client device can include one or more computers, laptops, laptop computers, tablet computers, mobile devices, cellular phones, wearable devices (e.g., watches, glasses, lenses, clothing, etc.), PDAs, etc. Furthermore, "client" can also refer to an entity (e.g., a merchant, acquiring party, etc.) that owns, utilizes, and / or operates a client device for initiating a transaction (e.g., for initiating a transaction with a transaction service provider).

[0057] As used herein, the term "communication" can refer to the receiving, accepting, sending, transmitting, providing, etc., of data (e.g., information, signals, messages, instructions, commands, etc.). For one unit (e.g., apparatus, system, component of an apparatus or system, combination thereof, etc.) to communicate with another unit means that the first unit is able to receive information directly or indirectly from and / or transmit information to the other unit. This can refer to a direct or indirect connection that is inherently wired and / or wireless (e.g., a direct communication connection, an indirect communication connection, and / or the like). Furthermore, although the transmitted information may be modified, processed, relayed, and / or routed between the first and second units, the two units can also communicate with each other. For example, the first unit can communicate with the second unit even if it passively receives information and does not actively send information to the second unit. As another example, the first unit can communicate with the second unit if at least one intermediate unit processes information received from the first unit and transmits the processed information to the second unit. In some non-limiting embodiments or aspects, a message can refer to a network packet (e.g., a data packet and / or the like) that includes data. It should be understood that there may be many other arrangements.

[0058] As used herein, the term "computing device" can refer to one or more electronic devices configured to process data. In some examples, a computing device may include essential components for receiving, processing, and outputting data, such as a processor, display, memory, input device, network interface, and / or the like. A computing device can be a mobile device. As examples, a mobile device may include a cellular phone (e.g., a smartphone or standard cellular phone), a portable computer, a wearable device (e.g., a watch, glasses, lenses, clothing, and / or the like), a personal digital assistant (PDA), and / or other similar devices. A computing device can also be a desktop computer or other form of non-mobile computer.

[0059] As used herein, the term "issuer institution" can refer to one or more entities, such as banks, that provide customers with accounts for conducting transactions (e.g., payment transactions), such as initiating credit and / or debit payments. For example, an issuer institution may provide customers with an account identifier, such as a PAN, that uniquely identifies one or more accounts associated with said customer. The account identifier may be embodied in a portable financial device, such as a physical financial instrument (e.g., a payment card), and / or may be electronic and used for electronic payments. The term "issuer system" refers to one or more computer devices operated by or on behalf of an issuer institution, such as a server computer executing one or more software applications. For example, an issuer system may include one or more authorization servers for authorizing transactions.

[0060] As used herein, the term "merchant" can refer to an individual or entity that provides goods and / or services or access to goods and / or services to a customer based on a transaction such as a payment transaction. The terms "merchant" or "merchant system" can also refer to one or more computer systems operated by or on behalf of a merchant, such as a server computer that executes one or more software applications.

[0061] As used herein, a “point-of-sale (POS) device” can refer to one or more devices that a merchant can use to conduct transactions (e.g., payment transactions) and / or process transactions. For example, a POS device may include one or more client devices. Alternatively or additionally, a POS device may include peripheral devices, card readers, scanning devices (e.g., barcode scanners), Bluetooth® communication receivers, near-field communication (NFC) receivers, radio frequency identification (RFID) receivers and / or other contactless transceivers or receivers, contact-based receivers, payment terminals, etc. As used herein, a “point-of-sale (POS) system” can refer to one or more client devices and / or peripheral devices used by a merchant to conduct transactions. For example, a POS system may include one or more POS devices, and / or other similar devices that can be used to conduct payment transactions. In some non-limiting embodiments or aspects, a POS system (e.g., a merchant POS system) may include one or more server computers programmed or configured to process online payment transactions via web pages, mobile applications, etc.

[0062] As used herein, the term "payment device" can refer to electronic payment devices, portable financial devices, payment cards (e.g., credit or debit cards), gift cards, smart cards, smart media, payroll cards, healthcare cards, wristbands, machine-readable media containing account information, keychain devices or pendants, RFID transponders, retailer discount or loyalty cards, cellular phones, e-wallet mobile applications, personal digital assistants (PDAs), pagers, security cards, computing devices, access cards, wireless terminals, transponders, and / or the like. In some non-limiting embodiments or aspects, a payment device may include volatile or non-volatile memory for storing information such as account identifiers, account holder names, and / or the like.

[0063] As used herein, the term "payment gateway" can refer to an entity and / or a payment processing system operated by or on behalf of such an entity (e.g., a merchant service provider, payment service provider, payment servicer, payment aggregator, payment aggregator, and / or the like) that provides payment services (e.g., transaction service provider payment services, payment processing services, and / or the like) to one or more merchants. Payment services may be associated with the use of portable financial devices managed by a transaction service provider. As used herein, the term "payment gateway system" can refer to one or more computer systems, computer devices, servers, server clusters, etc., operated by or on behalf of a payment gateway.

[0064] As used herein, the term "server" can refer to or include one or more computing devices operated by or facilitating communication and processing among multiple parties in a network environment such as the Internet, but it should be understood that communication may be facilitated through one or more public or private network environments, and various other arrangements may be possible. Furthermore, multiple computing devices (e.g., servers, point-of-sale (POS) devices, mobile devices, etc.) communicating directly or indirectly in a network environment can constitute a "system".

[0065] As used herein, the term "system" may refer to one or more computing devices or a combination of computing devices (e.g., processors, servers, client devices, software applications, components of such devices, and / or their likenesses). As used herein, references to "device," "server," "processor," and / or their likenesses may refer to a previously described device, server, or processor, a different device, server, or processor, and / or a combination of devices, servers, and / or processors, described as performing a preceding step or function. For example, as used in the specification and claims, a first device, first server, or first processor described as performing a first step or a first function may refer to the same or different device, server, or processor described as performing a second step or a second function.

[0066] As used herein, the term "transaction service provider" can refer to an entity that receives transaction authorization requests from merchants or other entities and, in some cases, provides payment guarantees through an agreement between the transaction service provider and the issuing entity. For example, a transaction service provider may include payment networks such as Visa®, or any other entity that processes transactions. The term "transaction processing system" can refer to one or more computer systems operated by or on behalf of the transaction service provider, such as a transaction processing server executing one or more software applications. A transaction processing server may include one or more processors and, in some non-limiting embodiments or aspects, may be operated by or on behalf of the transaction service provider.

[0067] Non-limiting embodiments or aspects of the disclosed subject matter relate to methods, systems, and computer program products for providing data type-aware transducer machine learning models for sequential datasets. In some non-limiting embodiments or aspects, a transducer management system may include at least one processor programmed or configured to receive interaction data associated with a plurality of interactions, the interaction data including a plurality of interaction records, each of the plurality of interaction records including a plurality of fields, the plurality of fields including at least one static field and at least one dynamic field; generating a static interaction embedding representation based on inputting static field data associated with at least one static field into a first transducer model; and generating a plurality of dynamic interaction embedding representations based on inputting dynamic field data associated with at least one dynamic field of a sequence of interaction records into a second transducer model, the sequence of interaction records including a plurality of... The system records at least a subset of the data; generates a first intermediate input based on a static interactive embedding representation, a first time-based embedding representation, and a first field-type embedding representation associated with at least one static field; generates multiple second intermediate inputs based on each dynamic interactive embedding representation, a corresponding time-based embedding representation, and a second field-type embedding representation associated with at least one dynamic field; generates a static sequence embedding representation based on inputting the first intermediate inputs into a third transformer model; generates multiple dynamic sequence embedding representations based on inputting the multiple second intermediate inputs into a third transformer model; and generates at least one prediction based on inputting the static sequence embedding representations and the multiple dynamic sequence embedding representations into a machine learning model.

[0068] In this way, the transformer management system can prevent static information in sequential input data from consuming excessive amounts of resources related to the transformer machine learning model used during training and / or production. Furthermore, static information can reduce the difficulty during model training (e.g., pre-training) tasks to the point that it prevents the transformer machine learning model from learning meaningful representations of the sequential input data.

[0069] For illustrative purposes, while the subject matter disclosed herein is described in relation to methods, systems, and computer program products for data type-aware transformer machine learning models (e.g., for payment transactions) targeting sequential datasets, those skilled in the art will recognize that the disclosed subject matter is not limited to the non-limiting embodiments or aspects disclosed herein. For example, the methods, systems, and computer program products described herein can be used with a wide variety of settings, such as targeting any suitable type of sequential dataset and / or for determining (e.g., prediction, classification, regression, etc.) based on sequential datasets using at least one machine learning model, such as data type-aware transformer machine learning models for fraud detection / prevention, authorization, authentication, identification, feature selection, product recommendation, click-through rate (CTR), etc.

[0070] Now for reference Figure 1 This diagram illustrates a system 100 for a data type-aware transformer machine learning model for sequential datasets, according to some non-limiting embodiments or aspects. Figure 1 As shown, system 100 may include converter management system 102, transaction service provider system 104, user device 106, and communication network 108. Converter management system 102, transaction service provider system 104, and / or user device 106 may be interconnected (e.g., establish a connection for communication) via wired connection, wireless connection, or a combination of wired and wireless connection (e.g., communication network 108).

[0071] The converter management system 102 may include one or more devices configured to communicate with the transaction service provider system 104 and / or user device 106 (e.g., directly, indirectly via communication network 108, etc.). For example, the converter management system 102 may include at least one computing device, such as a server, server cluster, and / or other similar device. In some non-limiting embodiments or aspects, the converter management system 102 may be associated with a transaction service provider. For example, the converter management system 102 may be operated by the transaction service provider. In some non-limiting embodiments or aspects, the converter management system 102 may be a component of the transaction service provider system 104. In some non-limiting embodiments or aspects, the converter management system 102 may communicate with a data storage device, which may be local or remote to the converter management system 102. In some non-limiting embodiments or aspects, the converter management system 102 may be able to receive information from the data storage device, store information in the data storage device, transfer information to the data storage device, and / or search for information stored in the data storage device.

[0072] The transaction service provider system 104 may include one or more devices configured to communicate with the converter management system 102 and / or user device 106 (e.g., directly, indirectly via communication network 108, etc.). For example, the transaction service provider system 104 may include at least one computing device, such as a server, server cluster, and / or other similar device. In some non-limiting embodiments or aspects, the transaction service provider system 104 may be associated with a transaction service provider.

[0073] User device 106 may include a computing device configured to communicate with converter management system 102 and / or transaction service provider system 104 (e.g., directly, indirectly via communication network 108, etc.). For example, user device 106 may include computing devices such as desktop computers, portable computers (e.g., tablet computers, laptop computers, etc.), mobile devices (e.g., cellular phones, smartphones, personal digital assistants, wearable devices, etc.) and / or other similar devices. In some non-limiting embodiments or aspects, user device 106 may be associated with a user (e.g., an individual operating user device 106).

[0074] The communication network 108 may include one or more wired and / or wireless networks. For example, the communication network 108 may include cellular networks (e.g., Long Term Evolution (LTE) networks, third-generation (3G) networks, fourth-generation (4G) networks, fifth-generation (5G) networks, Code Division Multiple Access (CDMA) networks, etc.), Public Land Mobile Networks (PLMN), Local Area Networks (LAN), Wide Area Networks (WAN), Metropolitan Area Networks (MAN), Telephone Networks (e.g., Public Switched Telephone Network (PSTN)), Private Networks (e.g., Private Networks Associated with Transaction Service Providers), Temporary Networks, Intranets, the Internet, Fiber-based Networks, Cloud Computing Networks, etc., and / or combinations of these or other types of networks.

[0075] Figure 1 The number and arrangement of systems and equipment shown are provided as examples. Figure 1 Compared to the systems and / or devices shown, there may be additional systems and / or devices, fewer systems and / or devices, different systems and / or devices, and / or systems and / or devices arranged in a different manner. Furthermore, implementation may be within a single system and / or device. Figure 1 Two or more systems or devices shown in the document, or Figure 1 The single system or device shown may be implemented as multiple distributed systems or devices. Alternatively, a group of systems (e.g., one or more systems) and / or a group of devices (e.g., one or more devices) of system 100 may perform one or more functions described as being performed by another group of systems or another group of devices of system 100.

[0076] Now for reference Figure 2 The flowchart illustrates a method 200 for a data type-aware transformer machine learning model for a sequential dataset, according to some non-limiting embodiments or aspects. Figure 2The steps shown are for illustrative purposes only. It will be understood that additional, fewer, different, and / or different orders of steps may be used in some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, steps may be performed automatically in response to the execution and / or completion of previous steps. In some non-limiting embodiments or aspects, one or more steps of process 200 may be performed (e.g., fully, partially, etc.) by the converter management system 102 (e.g., one or more devices of the converter management system 102). In some non-limiting embodiments or aspects, one or more steps of process 200 may be performed (e.g., fully, partially, etc.) by another device or group of devices separate from or including the converter management system 102 (e.g., one or more devices of the converter management system 102), the transaction service provider system 104 (e.g., one or more devices of the transaction service provider system 104), and / or the user device 106.

[0077] like Figure 2 As shown, at step 202, method 200 may include receiving interaction data associated with multiple interactions. For example, converter management system 102 may receive interaction data associated with multiple interactions. In some non-limiting embodiments, the interaction data may include multiple interaction records (e.g., multiple transaction records). Each of the multiple interaction records may include multiple fields, including at least one static field and at least one dynamic field.

[0078] For illustrative purposes, see now for reference. Figure 5A And continue to refer to Figure 2 , Figure 5A A schematic diagram illustrating an example sequential dataset according to some non-limiting embodiments or aspects is shown. For example... Figure 5A As shown, the first sequential dataset 501 may include multiple interactions (e.g., payment transactions associated with a first user). For example, the first payment transaction Trans 1 may be associated with a $10 payment transaction in California (CA). The second payment transaction Trans 2 may be associated with a $12 payment transaction in CA, and the second payment transaction Trans 2 may occur a relatively short amount of time after the first payment transaction Trans 1. The third payment transaction Trans 3 may be associated with an $80 payment transaction in Massachusetts (MA), and the third payment transaction Trans 3 may occur a relatively long amount of time after the second payment transaction Trans 2. In this example, the first sequential dataset 501 may be normal (e.g., not detected as fraudulent, etc.) because the user has sufficient time to travel from CA to MA.

[0079] like Figure 5AAs shown, the second sequential dataset 502 may include multiple interactions (e.g., payment transactions associated with a second user). For example, a first payment transaction Trans 1 may be associated with a $10 payment transaction in CA. A second payment transaction Trans 2 may be associated with a $12 payment transaction in CA, and the second payment transaction Trans 2 may occur a relatively short amount of time after the first payment transaction Trans 1. A third payment transaction Trans 3 may be associated with an $80 payment transaction in MA, and the third payment transaction Trans 3 may occur a relatively short amount of time after the second payment transaction Trans 2. In this example, the second sequential dataset 502 may be anomalous (e.g., detected as fraudulent, etc.) because the user did not have enough time to travel from CA to MA.

[0080] like Figure 5A As shown, the third sequence dataset 503 may include multiple interactions (e.g., payment transactions associated with a third user). For example, a first payment transaction Trans 1 may be associated with a $10 payment transaction in CA. A second payment transaction Trans 2 may be associated with an $80 payment transaction in MA, and the second payment transaction Trans 2 may occur a relatively short amount of time after the first payment transaction Trans 1. A third payment transaction Trans 3 may be associated with a $12 payment transaction in CA, and the third payment transaction Trans 3 may occur a relatively short amount of time after the second payment transaction Trans 2. In this example, the third sequence dataset 503 may be anomalous (e.g., detected as fraudulent, etc.) because the user does not have enough time to travel from CA to MA and back to CA.

[0081] like Figure 5A As illustrated in the examples, certain field values ​​(e.g., transaction amount, status / location, etc.) may all be identical in the first sequence dataset 501, the second sequence dataset 502, and the third sequence dataset 503, but the relative timing of the interactions (e.g., payment transactions) may differ. For example, an anomalous second and / or third transaction in the second sequence dataset 502 and the third sequence dataset 503 may not be detected solely based on those specific field values ​​(e.g., without considering timing). Thus, these examples demonstrate that time (e.g., time information) and order (e.g., sequence information) can be significant in detecting anomalies (e.g., fraud, etc.).

[0082] For illustrative purposes, see now for reference. Figure 5B And continue to refer to Figure 2 , Figure 5B This diagram illustrates an example sequence of datasets and example embeddings according to some non-limiting embodiments or aspects. (e.g.) Figure 5BAs shown, the sequential dataset 504 may include data associated with two interactions (e.g., transaction data associated with two payment transactions).

[0083] In some non-limiting embodiments or aspects, the sequential dataset 504 may be tokenized (e.g., split, separated, parsed, etc.) into tokens (e.g., smaller data items to be used as input to one or more machine learning models). For example, each corresponding field value of the interaction data (e.g., each field value of transaction data associated with a payment transaction) may be represented as a corresponding token. As used herein, the term "token" is not to be confused with "payment token" in the context of data items to be input into a machine learning model, as described herein. For example, a payment token may be a field value such that the corresponding token represents a payment token, but not all field values ​​are payment tokens.

[0084] For illustrative purposes, such as Figure 5B As shown, a first payment transaction may be associated with a token representing the transaction amount (e.g., $36), a token representing a restaurant identifier (e.g., a restaurant ID associated with the restaurant where the payment transaction took place), a token representing a timestamp (12:30 PM), a token representing the type of payment device (e.g., a debit card), a token representing the issuer of the payment device (e.g., Bank A), and a token indicating separation (e.g., [SEP], which may be a dedicated token indicating the end of the tokens associated with the current interaction, such that one or more next tokens are associated with a different interaction). A second payment transaction may be associated with a token representing the transaction amount (e.g., $200), a token representing an ATM identifier (e.g., an ATM ID associated with the location where the withdrawal took place), a token representing a timestamp (2:00 PM), a token representing the type of payment device (e.g., a debit card), a token representing the issuer of the payment device (e.g., Bank A), and a token indicating separation (e.g., [SEP]).

[0085] Continue to refer to Figure 2 In some non-limiting embodiments or aspects, the converter management system 102 may separate static field data associated with at least one static field from dynamic field data associated with at least one dynamic field for each of a plurality of interaction records. In some non-limiting embodiments or aspects, the converter management system 102 may generate a first input to a first converter model based on the static field data associated with at least one static field, and / or generate a second input to a second converter model based on the dynamic field data associated with at least one dynamic field.

[0086] For illustrative purposes, please refer again. Figure 5B And continue to refer to Figure 2The sequential dataset 504 can be divided into static field data 520 associated with static fields and dynamic field data 510 associated with dynamic fields. For example, static fields may include fields that are copied (e.g., identical, unchanged between interactions, etc.) in each interaction (e.g., transaction) within the sequence (e.g., sequential dataset 504). Figure 5B As shown, static fields can be associated with a token representing the type of payment device (e.g., a debit card) and a token representing the issuer of the payment device (e.g., bank A). Dynamic fields, for example, can include fields that are not necessarily replicated in each interaction (e.g., a transaction) within a sequence (e.g., sequential dataset 504) (e.g., may not be the same, may change between at least some interactions, etc.). Figure 5B As shown, dynamic fields can be associated with tokens representing transaction amounts, tokens representing certain identifiers (e.g., restaurant ID, ATM ID, etc.), and tokens representing timestamps. In some non-limiting embodiments or aspects, tokens indicating separation (e.g., [SEP]) can be inserted between static field data 520 and dynamic field data 510, and / or between tokens associated with each interaction (e.g., transaction) of dynamic field data 510. For example, such insertion can occur before the static field data 520 and dynamic field data 510 are fed into one or more machine learning models (e.g., to indicate separation between sets of tokens).

[0087] In some non-limiting embodiments or aspects, the converter management system 102 may mask the original values ​​of dynamic or static fields of a first interactive record in the interactive record sequence before inputting static field data associated with at least one static field of the interactive record sequence into a first converter model or before inputting dynamic field data associated with at least one dynamic field of the interactive record sequence into a second converter model, to provide masked dynamic fields of the first interactive record. For example, such masking may be used for at least a portion of training (e.g., pre-training), as described herein. For illustrative purposes, in some non-limiting embodiments or aspects, masking may be as described herein (e.g., refer to...). Figure 7A ).

[0088] Continue to refer to Figure 2At step 204, method 200 may include generating static interaction embedding representations and dynamic interaction embedding representations. For example, converter management system 102 may generate static interaction embedding representations and dynamic interaction embedding representations. In some non-limiting embodiments or aspects, converter management system 102 may generate static interaction embedding representations based on inputting static field data associated with at least one static field into a first converter model. In some non-limiting embodiments or aspects, converter management system 102 may generate multiple dynamic interaction embedding representations based on inputting dynamic field data associated with at least one dynamic field of an interaction record sequence into a second converter model. The interaction record sequence may include at least a subset of multiple interaction records.

[0089] For illustrative purposes, please refer again. Figure 5B And continue to refer to Figure 2 A static interaction embedding representation 560 can be generated based on static field data 520. For example, the static interaction embedding representation 560 may include the embedding of each token of the static interaction embedding representation 560 (e.g., a debit embedding E associated with a token representing a payment device type, such as a debit card). Debit and the issuer embedded E associated with the token representing the issuer of the payment device (e.g., bank A). BankA In some non-limiting embodiments or aspects, the embedded E is separated. [SEP] It can be associated with a token indicating separation (e.g., [SEP]).

[0090] For example, a dynamic interaction embedding representation 561 can be generated based on dynamic field data 510 of an interaction sequence (e.g., a transaction). For example, the dynamic interaction embedding representation 561 may include an embedding of each token from the dynamic field data 510. For example, the dynamic interaction embedding representation 561 may include a transaction amount embedding (e.g., E) associated with each token representing the transaction amount. $36 E $200 (e.g., E), and identifier embeddings associated with each token representing an identifier (e.g., restaurant ID, ATM ID, etc.). ResID E ATMID (etc.) and the timestamp embedding associated with each token representing a timestamp (E 12:30PM E 2:00PM wait).

[0091] For illustrative purposes, see now for reference. Figure 7A and Figure 7B And continue to refer to Figure 2 , Figure 7A and Figure 7B A schematic diagram illustrating an example implementation of a data type-aware transformer machine learning model for a sequential dataset, according to some non-limiting embodiments or aspects. (e.g.) Figure 7A and Figure 7B As shown, static field data 720 and dynamic field data 710 can be provided as input (e.g., received, retrieved, etc.). In some non-limiting embodiments or aspects, tokens (e.g., raw data tokens) of static field data 720 and dynamic field data 710... ) can be converted into local vocabulary (e.g., vocabulary tokens) ), to generate static field vocabulary data 721 and dynamic field vocabulary data 711, for example, before being input into one or more machine learning models (e.g., transformer models).

[0092] In some non-limiting embodiments or aspects, Based on The following equation converts raw data tokens into vocabulary tokens: Equation 1

[0093] Equation 2

[0094] in It is a static field. For dynamic fields, ConvertToVocab is a function that converts raw tokens into vocabulary tokens. For static vocabulary tokens, For dynamic vocabulary tokens, The number of interactions (e.g., transactions) in the sequence (e.g., the length of the sequence). For the number of dynamic fields (e.g., per interaction), and This represents the number of static fields.

[0095] In some non-limiting embodiments or aspects, the raw data token may be based on a preprocessing procedure. Convert to word tokens For example, Padhi et al. Tabular transformers for modeling multivariate time series The preprocessing procedures described in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 3565-3569 (2021), the contents of which are hereby incorporated in their entirety.

[0096] In some non-limiting embodiments or aspects, static field vocabulary data 721 may be input to a first transformer model (e.g., static field transformer 731) to generate a static interaction embedding representation 760. Alternatively, dynamic field vocabulary data 711 may be input to a second transformer model (e.g., dynamic field transformer 732) to generate a dynamic interaction embedding representation 761 for each interaction (e.g., each payment transaction).

[0097] Continue to refer to Figure 2 In some non-limiting embodiments or aspects, when generating multiple dynamic interaction embedding representations, the converter management system 102 may generate a first dynamic interaction embedding associated with the first interaction record based on inputting first dynamic field data associated with at least one dynamic field of the first interaction record in the interaction record sequence into a second converter model, and generate a second dynamic interaction embedding associated with the second interaction record based on inputting second dynamic field data associated with at least one second dynamic field of the first interaction record in the interaction record sequence into a second converter model.

[0098] For illustrative purposes, please refer again. Figure 7A and Figure 7B And continue to refer to Figure 2 The first dynamic interaction can be generated based on the first interaction (e.g., transaction 0) and embedded in TE. D,0 This can be repeated for each interaction (e.g., each transaction). For example, assuming there is a total of 1 transaction, a corresponding first dynamic interaction embedding TE can be generated for each corresponding interaction (e.g., transaction i) for all i = 0, ..., l-1. D,i .

[0099] Continue to refer to Figure 2 In some non-limiting embodiments or aspects, when generating a static interaction embedding representation based on inputting static field data associated with at least one static field of the interaction record into the first transformer model, the transformer management system 102 may generate a static interaction embedding representation based on inputting at least one masked static field of the interaction record into the first transformer model.

[0100] For illustrative purposes, please refer again. Figure 7A And continue to refer to Figure 2 This can mask the token of static field data 720, and therefore, the corresponding word token of static field vocabulary data 721. Thus, static interactive embedding representation 760 (e.g., TE) S It can be based on the masked token.

[0101] Continue to refer to Figure 2In some non-limiting embodiments or aspects, when generating multiple dynamic interaction embedding representations based on inputting dynamic field data associated with multiple dynamic fields of an interaction record sequence into a second transformer model, the transformer management system 102 may generate multiple dynamic interaction embedding representations based on inputting at least one masked dynamic field of an interaction record into a second transformer model.

[0102] For illustrative purposes, please refer again. Figure 7A And continue to refer to Figure 2 This can mask the token of the dynamic field data 710 for each interaction, and therefore, the corresponding lexical token of the dynamic field vocabulary data 711 can be masked. Thus, the dynamic interaction embedding representation 761 (e.g., TE) D,0 , ..., TE D,l-1 It can be based on the masked token.

[0103] In some non-limiting embodiments or aspects, token masking may be based on the following equation: Equation 3

[0104] Equation 4

[0105] in Indicator token is masked (e.g., if) Then the token is replaced by [MASK]. A function for randomly masking tokens, and and These are dynamic and static word tokens, respectively, after being concealed.

[0106] In some non-limiting embodiments or aspects, static interactive embedding representations (e.g., static transaction embeddings TE) can be generated based on the following equation. S ) and dynamic interactive embedding representations (e.g., dynamic transaction embeddings TE) D,i ): Equation 5

[0107] Equation 6

[0108] in It is a static field transformer, and For dynamic field transformers.

[0109] like Figure 2As shown, at step 206, method 200 may include generating intermediate inputs. For example, converter management system 102 may generate intermediate inputs. In some non-limiting embodiments or aspects, converter management system 102 may generate a first intermediate input based on a static interactive embedding representation, a first time-based embedding representation, and a first field type embedding representation associated with at least one static field. In some non-limiting embodiments or aspects, converter management system 102 may generate multiple second intermediate inputs based on each dynamic interactive embedding representation, a corresponding time-based embedding representation, and a second field type embedding representation associated with at least one dynamic field.

[0110] In some non-limiting embodiments or aspects, when generating the first intermediate input, the converter management system 102 may combine (e.g., summation, cascading, etc.) a static interactive embedded representation, a first time-based embedded representation, and a first field-type embedded representation associated with at least one static field.

[0111] In some non-limiting embodiments or aspects, the converter management system 102 may generate a first time-based embedding representation associated with a first interaction record and a second time-based embedding representation associated with a second interaction record. In some non-limiting embodiments or aspects, when generating multiple second intermediate inputs, the converter management system 102 may combine a first dynamic interaction embedding, a first time-based embedding representation, and a second field-type embedding representation associated with at least one dynamic field, and / or may combine a second dynamic interaction embedding, a second time-based embedding representation, and a second field-type embedding representation associated with at least one dynamic field.

[0112] For illustrative purposes, please refer again. Figure 5B And continue to refer to Figure 2 Static interactive embedding 560 can be embedded with at least one static field type 540 (e.g., E S A combination of 550 and / or at least one static time-based embedded representation 550 (e.g., E0, E1 and / or E2).

[0113] Alternatively or concurrently, each dynamic interaction embedding representation 561 may be associated with at least one dynamic field type embedding 541 (e.g., E...). D ) and / or at least one dynamic time-based embedded representation 551 (e.g., E3, E4, E5 and / or E6 for the first transaction; E7, E8, E9 and / or E 10 (etc.) combination.

[0114] For illustrative purposes, please refer again. Figures 7A to 7B And continue to refer to Figure 2 Static interactive embedding represents 760 (e.g., TE). SIt can be combined with static field type embedding 740 and time-based embedding representations (e.g., time-aware location embedding 750, which can be based on location 0 and time 0).

[0115] Alternatively or concurrently, each dynamic interaction embedding represents 761 (e.g., TE). D,0 , ..., TE D,l-1 It can be used with dynamic field type embeddings 741 and time-based embedding representations (e.g., time-aware location embeddings 751, which can be based on the first, second, and third positions of i = 0, ..., l-1). i The location i and time t associated with each transaction i )combination.

[0116] In some non-limiting embodiments or aspects, time-aware location embedding It can be defined as a length vector Each element of the vector can be determined based on the following equation. : Equation 7

[0117] Equation 8

[0118] in For use in merging from location index and time interval The information function (e.g., this helps through trainable parameters) , And b to capture time-aware location information, for example, to enable the learning of more flexible functions.

[0119] In some non-limiting embodiments or aspects, intermediate inputs may be determined based on the following equation (e.g., static intermediate input representation). and dynamic intermediate input representation ): Equation 9

[0120] Equation 10

[0121] in Embedded for static field types, and Embedded for dynamic field types.

[0122] Continue to refer to Figure 2At step 208, method 200 may include generating static sequence embedding representations and dynamic sequence embedding representations. For example, converter management system 102 may generate static sequence embedding representations and dynamic sequence embedding representations. In some non-limiting embodiments or aspects, converter management system 102 may generate static sequence embedding representations based on inputting a first intermediate input to a third converter model. In some non-limiting embodiments or aspects, converter management system 102 may generate multiple dynamic sequence embedding representations based on inputting multiple second intermediate inputs to a third converter model.

[0123] For illustrative purposes, please refer again. Figure 7A and Figure 7B And continue to refer to Figure 2 A static sequence embedding representation 770 (e.g., SE) can be generated by inputting a first intermediate input (e.g., based on combining a static interaction embedding representation 760 with a field type embedding 740 and a time-aware position embedding 750) into a third transformer model (e.g., a first and time-aware sequence encoding transformer 733). S ).

[0124] Alternatively or concurrently, this can be achieved by using a second intermediate input (e.g., based on embedding each dynamic interaction into a representation 761 (e.g., TE)). D,0 , ..., TE D,l-1 (Combined with dynamic field type embedding 741 and corresponding time-aware position embedding 751) is input to a third transformer model (e.g., field and time-aware sequence encoding transformer 733) to generate a dynamic sequence embedding representation 771 (e.g., SE). D,0 SE D,l-1 ).

[0125] In some non-limiting embodiments or aspects, the converter management system 102 may train at least one of the converter models (e.g., a first converter model, a second converter model, a third converter model, and / or any combination thereof) by comparing the data value of at least one of the data fields of a static sequence embedding representation, a dynamic sequence embedding representation (e.g., associated with an interactive record) provided by a third converter with the original value of the static or dynamic field (e.g., of the interactive record) and / or by adjusting at least one parameter of one or more converter models based on the comparison.

[0126] In some non-limiting embodiments or aspects, the sequence embedding representation may be determined based on the following equation: Equation 11

[0127] in For field and time-aware order encoding transformers.

[0128] For illustrative purposes, please refer again. Figure 7A And continue to refer to Figure 2 This can mask at least one token of at least one of static field data 720 and / or dynamic field data 710, as described herein. Training may include inputting static field data 720 and / or dynamic field data 710 (e.g., including one or more masked tokens) into one or more transformer models, as described herein, and the input may be forward-propagated through one or more transformer models to generate static sequence embedding representations 770 and / or dynamic sequence embedding representations 771, as described herein. A predicted value for each masked token may be generated based on the sequence embeddings (e.g., static sequence embedding representation 770 and dynamic sequence embedding representation 771). For example, the sequence embeddings may be input into a classifier model (e.g., ) to generate the predicted value (e.g., In some non-limiting embodiments or aspects, the predicted value may be generated based on the following equation: Equation 12

[0129]

[0130] =

[0131] For illustrative purposes, such as Figure 7A As shown, A masked token (e.g., MASK) that can be predicted and corresponds to static field data 720; The masked token (e.g., MASK) of the dynamic field data 710 that can be predicted and corresponds to transaction 0; and A masked token (e.g., MASK) whose dynamic field data 710 is predictable and corresponds to transaction l-1. One or more predicted values ​​can be compared with the corresponding original values ​​(e.g., the actual values ​​of static field data 720 and / or dynamic field data 710 before masking). Or the vocabulary of this actual value The comparison can be made. For example, the loss can be determined based on one or more predicted values ​​and one or more corresponding original values ​​(or their lexical terms). For example, the loss can be calculated based on the difference between the predicted and original values, loss functions, cross-entropy loss, error, mean squared error, root mean squared error (MSE), any combination thereof, etc. In some non-limiting embodiments or aspects, the loss can be calculated based on the following equation. (For example, cross-entropy loss): Equation 13

[0132] The parameters of at least one of the transformers (e.g., at least one of static field transformer 731, dynamic field transformer 732, field and time-aware sequence encoding transformer 733, any combination thereof, etc.) can be updated based on the loss. For example, the transformer management system 102 can update (e.g., adjust) the parameters of one or more transformers based on (e.g., backpropagation of one or more losses), gradient computation (e.g., based on one or more losses), any combination thereof, etc.

[0133] In some non-limiting embodiments or aspects, the pre-trained model for predicting the masked value may be referred to as pre-training.

[0134] Continue to refer to Figure 2 At step 210, method 200 may include generating a prediction based on static sequence embedding representations and dynamic sequence embedding representations. For example, the converter management system 102 may generate a prediction based on static sequence embedding representations and dynamic sequence embedding representations. In some non-limiting embodiments or aspects, the converter management system 102 may generate at least one prediction by inputting static sequence embedding representations and multiple dynamic sequence embedding representations into a machine learning model. In some non-limiting embodiments or aspects, the converter management system 102 may perform actions (e.g., fraud detection actions, visualization actions, etc.) based on the predictions. For example, the converter management system 102 may perform actions associated with a fraud detection task based on at least one prediction.

[0135] For illustrative purposes, see now for reference. Figure 6 And continue to refer to Figure 2 , Figure 6 The diagram illustrates an example implementation of a system and method for a data type-aware transformer machine learning model for sequential datasets, according to some non-limiting embodiments or aspects. Figure 6 As shown, static field data 620 and dynamic field data 610 may be associated with user 605 (e.g., with at least one account identifier of user 605). For example, static field data 620 may be associated with a static field of a sequential dataset comprising t records (e.g., record 1, record 2, ..., record t), and dynamic field data 610 may be associated with its dynamic field as described herein.

[0136] Static field data 620 and dynamic field data 610 can be input into machine learning model 630, which may include at least one transformer model, such as a first transformer model (e.g., a static field transformer model), a second transformer model (e.g., a dynamic field transformer model), a third transformer model (e.g., a field and time-aware sequence encoding transformer model), any combination thereof, etc. In some non-limiting embodiments or aspects, machine learning model 630 may have been trained (pre-trained) as described herein.

[0137] In some non-limiting embodiments or aspects, machine learning model 630 may generate representation 670 (e.g., a sequence representation based on static sequence embedding representations and dynamic sequence embedding representations associated with each record). For example, representation 670 may be input to at least one other machine learning model 690 (e.g., a downstream machine learning model) to perform at least one task. For illustrative purposes, as... Figure 6 As shown, one or more other machine learning models 690 may include an anomaly detection model for performing an anomaly detection task (e.g., classifying sequences as anomalous or non-anomaly), a CTR prediction model for predicting CTR, a product recommendation model for predicting at least one recommended product, a fraud detection model for performing a fraud detection model (e.g., classifying sequences as fraudulent or non-fraudulent), an authorization model for performing an authorization task, an authentication model for performing an authentication task, an identification model for performing an identification task, a feature selection model for performing a feature selection task, or any combination thereof.

[0138] For illustrative purposes, please refer again. Figure 7B And continue to refer to Figure 2 In some non-limiting embodiments or aspects, masking is no longer used after pre-training. Thus, static field data 720 and dynamic field data 710 do not include masked tokens (e.g., all tokens have their original values).

[0139] In some non-limiting embodiments or aspects, sequence embeddings (e.g., static sequence embedding representation 770 and / or dynamic sequence embedding representation 771) may be input to at least one machine learning model 790 (e.g., at least one classifier, etc., as described herein). One or more machine learning models 790 may generate at least one output 791 (e.g., prediction, classification, etc., as described herein) based on the sequence embeddings.

[0140] In some non-limiting embodiments or aspects, at least one of one or more transformers (e.g., static field transformer 731, dynamic field transformer 732, and / or field and time-aware sequence encoding transformer 733) may be (re)trained, and / or one or more machine learning models 790 may be trained. For example, static field data 720 and / or dynamic field data 710 may be input to one or more transformer models as described herein, and the input may be forward-propagated through one or more transformer models to generate static sequence embedding representations 770 and / or dynamic sequence embedding representations 771, as described herein. One or more machine learning models 790 may generate one or more outputs 791 based on sequence embeddings (e.g., generating static sequence embedding representations 770 and / or dynamic sequence embedding representations 771), as described herein. Loss may be determined based on one or more predicted values. For example, loss may be determined based on one or more outputs 791 and one or more known values ​​(e.g., labels), loss functions, errors, mean error, MSE, any combination thereof, etc. The parameters of at least one transformer (e.g., at least one of a static field transformer 731, a dynamic field transformer 732, a field and time-aware sequence encoding transformer 733, any combination thereof, etc.) and / or one or more machine learning models 790 can be updated based on losses. For example, the transformer management system 102 can update (e.g., adjust) the parameters of one or more transformers and / or one or more machine learning models 790 based on (e.g., backpropagation of one or more losses), gradient computation (e.g., based on one or more losses), any combination thereof, etc.

[0141] In some non-limiting embodiments or aspects, this (re)training based on one or more outputs 791 of one or more machine learning models 790 may be referred to as fine-tuning.

[0142] In some non-limiting embodiments or aspects, method 200 may be implemented according to the following algorithm: Algorithm 1: Input: - Transaction sequence dataset : , - Transaction sequence sample.

[0143] : - Transaction sample, by static fields and It consists of several dynamic fields.

[0144] t : - Transaction creation time series.

[0145] Parameter: - Parameters of the static field transformer - Parameters of the dynamic field transformer - Field and Time-Aware Sequential Encoding Transformer - Classifier Output: - Fields u Corresponding words The predicted probability 1:while , , , not convergeddo 2:sample from

[0146] 3:for do 4:for do 5: = ConvertToVocab

[0147] 6: , = RandomMask

[0148] 7:end for 8:end for 9:for do 10: = ConvertToVocab

[0149] 11: , = RandomMask

[0150] 12:end for 13:

[0151] 14:for do 15:

[0152] 16:end for 17: ,

[0153] 18: , ,

[0154] 19: for &

[0155] for

[0156]

[0157] 20: , , , = Back Propagation( , , , , ) 21:end while Now for reference Figure 3 This diagram depicts an example payment processing network 300 according to a non-limiting embodiment or aspect. In some non-limiting embodiments or aspects, the payment processing network 300 may be used in conjunction with the systems, methods, and / or computer program products described herein, and / or the systems, methods, and / or computer program products described herein may be implemented in the payment processing network 300. Figure 3 As shown, the payment processing network 300 may include a transaction processing system 301, a payment gateway system 302, a merchant system 304, an issuer system 306, an acquirer system 308, and / or a consumer device 310. In some non-limiting embodiments or aspects, Figure 1 Each of the converter management system 102, the transaction service provider system 104, and / or the user device 106 may be implemented by the transaction processing system 301 (e.g., a portion thereof). For example, the transaction service provider system 104 may be the same as or similar to the transaction processing system 301. In some non-limiting embodiments or aspects, Figure 1At least one of the transceiver management system 102 and / or user device 106 may be implemented by another system, another device, another group of systems, or another group of devices (e.g., merchant system 304, issuer system 306, acquirer system 308, consumer device 310, etc.) (e.g., a portion thereof), which is separate from or includes the transaction processing system 301. For example, user device 106 may be implemented by at least one of the payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, and / or consumer device 310 (e.g., a portion thereof). Alternatively or additionally, for example, transceiver management system 102 may be implemented by at least one of the payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, and / or consumer device 310 (e.g., a portion thereof).

[0158] Transaction processing system 301 may include one or more devices capable of receiving and / or transmitting information to payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, consumer device 310, etc. (e.g., directly, indirectly, via public and / or private communication networks). For example, such as Figure 3 As shown, transaction processing system 301 can communicate with one or more issuer systems (e.g., issuer system 306), one or more acquirer systems (e.g., acquirer system 308), and / or one or more payment gateway systems (e.g., payment gateway system 302). Although only a single issuer system 306, a single acquirer system 308, and a single payment gateway system 302 are shown, it should be understood that transaction processing system 301 can communicate with multiple issuer systems, multiple acquirer systems, and / or multiple payment gateways. In some non-limiting embodiments or aspects, transaction processing system 301 may include computing devices, such as servers (e.g., transaction processing servers), server clusters, and / or other similar devices. In some non-limiting embodiments or aspects, transaction processing system 301 can communicate with data storage devices, which may be local or remote to transaction processing system 301. In some non-limiting embodiments or aspects, transaction processing system 301 may be able to receive information from data storage devices, store information in data storage devices, transmit information to data storage devices, or search for information stored in data storage devices. In some non-limiting embodiments or aspects, transaction processing system 301 may be associated with a transaction service provider, as described herein. In some non-limiting embodiments or aspects, transaction processing system 301 may also operate as an issuer system, such that transaction processing system 301 and issuer system 306 are both a single system and / or controlled by a single entity.

[0159] The payment gateway system 302 may include one or more devices capable of receiving and / or transmitting information to and / or to transaction processing systems 301, merchant systems 304, issuer systems 306, acquirer systems 308, consumer devices 310, etc. (e.g., directly, indirectly, or via public and / or private communication networks). For example, such as... Figure 3 As shown, the payment gateway system 302 can communicate with one or more merchant systems (e.g., merchant system 304), one or more acquiring systems (e.g., acquiring system 308), and / or one or more transaction processing systems (e.g., transaction processing system 301). Although only a single merchant system 304, a single acquiring system 308, and a single transaction processing system 301 are shown, it should be understood that the payment gateway system 302 can communicate with multiple merchant systems, multiple acquiring systems, and / or multiple transaction processing systems. In some non-limiting embodiments or aspects, the payment gateway system 302 may include computing devices, such as servers, server clusters, and / or other similar devices. In some non-limiting embodiments or aspects, the payment gateway system 302 may be associated with a payment gateway as described herein.

[0160] Merchant system 304 may include one or more devices capable of receiving and / or transmitting information to and / or to transaction processing system 301, payment gateway system 302, issuer system 306, acquirer system 308, consumer device 310, etc. (e.g., directly, indirectly, via public and / or private communication networks). For example, such as Figure 3As shown, merchant system 304 may communicate with one or more payment gateway systems (e.g., payment gateway system 302), one or more acquiring systems (e.g., acquiring system 308), and / or one or more consumer devices (e.g., consumer device 310). Although only a single payment gateway system 302, a single acquiring system 308, and a single consumer device 310 are shown, it should be understood that merchant system 304 may communicate with multiple payment gateway systems, multiple acquiring systems, and / or multiple consumer devices. In some non-limiting embodiments or aspects, merchant system 304 may include computing devices, such as servers, server groups, client devices, client device groups, POS devices, POS systems, computers, computer systems, peripheral devices, and / or other similar devices. In some non-limiting embodiments or aspects, merchant system 304 may be associated with a merchant, as described herein. In some non-limiting embodiments or aspects, the merchant system 304 may include means capable of receiving and / or transmitting information to the consumer device 310 via a short-range communication connection (e.g., NFC, RFID, Bluetooth®, Zigbee®, etc.) with the consumer device 310. In some non-limiting embodiments or aspects, the merchant system 304 may include one or more client devices. For example, the merchant system 304 may include client devices that allow the merchant to transmit information to the transaction processing system 301 (e.g., via at least one of the acquiring system 308 and / or the payment gateway system 302). In some non-limiting embodiments or aspects, the merchant system 304 (e.g., its client devices, its POS devices, etc.) may also operate as a payment gateway system, such that both the merchant system 304 and the payment gateway system 302 are a single system and / or controlled by a single entity.

[0161] The issuing system 306 may include one or more devices capable of receiving information and / or transmitting information (e.g., directly, indirectly, or via public and / or private communication networks) to the transaction processing system 301, payment gateway system 302, merchant system 304, acquiring system 308, consumer device 310, etc. For example, such as Figure 3As shown, the issuer system 306 may communicate with one or more transaction processing systems (e.g., transaction processing system 301) and / or one or more consumer devices (e.g., consumer device 310). Although only a single transaction processing system 301 and a single consumer device 310 are shown, it should be understood that the issuer system 306 may communicate with multiple transaction processing systems and / or multiple consumer devices 310. In some non-limiting embodiments or aspects, the issuer system 306 may include computing devices, such as servers, server clusters, and / or other similar devices. In some non-limiting embodiments or aspects, the issuer system 306 may be associated with an issuing authority, as described herein. For example, the issuer system 306 may be associated with an issuing authority that issues credit accounts, debit accounts, credit cards, debit cards, payment devices, etc., to users associated with consumer devices 310.

[0162] The acquiring system 308 may include one or more devices capable of receiving and / or transmitting information to and / or to the transaction processing system 301, payment gateway system 302, merchant system 304, issuer system 306, consumer device 310, etc. (e.g., directly, indirectly, or via public and / or private communication networks). For example, such as Figure 3 As shown, the acquiring system 308 may communicate with one or more transaction processing systems (e.g., transaction processing system 301), one or more payment gateway systems (e.g., payment gateway system 302), and / or one or more merchant systems (e.g., merchant system 304). Although only a single transaction processing system 301, a single payment gateway system 302, and a single merchant system 304 are shown, it should be understood that the acquiring system 308 may communicate with multiple transaction processing systems, multiple payment gateway systems, and / or multiple merchant systems. In some non-limiting embodiments or aspects, the acquiring system 308 may include computing devices, such as servers, server clusters, and / or other similar devices. In some non-limiting embodiments or aspects, the acquiring system 308 may be associated with an acquiring institution as described herein.

[0163] Consumer device 310 may include one or more devices capable of receiving and / or transmitting information to transaction processing system 301, payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, etc. (e.g., directly, indirectly, via public and / or private communication networks). For example, such as Figure 3As shown, consumer device 310 can communicate with one or more merchant systems (e.g., merchant system 304) and / or one or more issuer systems (e.g., issuer system 306). Although only a single merchant system 304 and a single issuer system 306 are shown, it should be understood that consumer device 310 can communicate with multiple merchant systems and / or multiple issuer systems. In some non-limiting embodiments or aspects, consumer device 310 can be associated with a user who has issued a credit account, debit account, credit card, debit card, payment device, etc. In some non-limiting embodiments or aspects, user device 310 may include computing devices, such as computers, portable computers, laptop computers, tablet computers, mobile devices, cellular phones, smartphones, wearable devices (e.g., watches, glasses, lenses, clothing, etc.), PDAs, client devices, and / or other similar devices. In some non-limiting embodiments or aspects, user device 310 may include payment devices, as described herein. In some non-limiting embodiments or aspects, consumer device 310 may include means capable of receiving information from and / or transmitting information to other client devices 310 (e.g., directly, indirectly, via public and / or private communication network connections, short-range communication connections, etc.). In some non-limiting embodiments or aspects, consumer device 310 may include means capable of receiving information from and / or transmitting information to merchant system 304 via a short-range communication connection (e.g., NFC communication connection, RFID communication connection, Bluetooth® communication connection, Zigbee® communication connection, etc.). In some non-limiting embodiments or aspects, consumer device 310 may include client devices.

[0164] In some non-limiting embodiments or aspects, the transaction processing system 301 may communicate directly with the merchant system 304 (e.g., via a public and / or private communication network connection, etc.). Alternatively, the transaction processing system 301 may communicate with the merchant system 304 via a payment gateway 302 and / or an acquiring system 308. In some non-limiting embodiments or aspects, the acquiring system 308 associated with the merchant system 304 may operate as a payment gateway 302 to facilitate the transmission of transaction messages (e.g., authorization requests) from the merchant system 304 to the transaction processing system 301. In some non-limiting embodiments or aspects, the merchant system 304 may communicate directly (e.g., via a public and / or private communication network connection, etc.) with the payment gateway 302. For example, the merchant system 304, including a physical POS device, may communicate with the payment gateway 302 via a public or private network for card transactions. As another example, the merchant system 304, including a server (e.g., a web server), may communicate with the payment gateway 302 via a public or private network, such as the Internet, for cardless transactions.

[0165] For illustrative purposes, processing a transaction (e.g., a payment transaction) may include generating a transaction message (e.g., an authorization request, etc.) based on the account identifier of a customer (e.g., an account holder associated with a customer device 310, etc.) and / or transaction data associated with the transaction. For example, a merchant system 304 (e.g., a client device of merchant system 304, a POS device of merchant system 304, etc.) may initiate a transaction, for example, by generating an authorization request (e.g., in response to receiving an account identifier from a customer's payment device and / or portable financial device, etc.). Merchant system 304 may transmit the authorization request to payment gateway 302 and / or acquiring system 308. In some non-limiting embodiments or aspects, payment gateway 302 may transmit the authorization request to acquiring system 308 and / or transaction processing system 301. Alternatively or concurrently, acquiring system 308 (and / or payment gateway 302) may transmit the authorization request to transaction processing system 301. After receiving an authorization request from merchant system 304 that identifies a customer's account identifier (e.g., the account holder and / or account identifier associated with consumer device 310), transaction processing system 301 may transmit the authorization request to issuer system 306 (e.g., the issuer system that issues the payment device and / or account identifier). Issuer system 306 may determine an authorization decision (e.g., approval, rejection, etc.) based on the authorization request, and / or issuer system 306 may generate an authorization response based on the authorization decision and / or authorization request. Issuer system 306 may transmit the authorization response to transaction processing system 301. Transaction processing system 301 may transmit the authorization response to acquiring system 308 and / or payment gateway 302. In some non-limiting embodiments or aspects, acquiring system 308 may transmit the authorization response to payment gateway 302 and / or merchant system 304. Alternatively or additionally, payment gateway 302 (and / or acquiring system 308) may transmit the authorization response to merchant system 304.

[0166] In some non-limiting embodiments or aspects, transaction processing system 301 and / or issuer system 306 may include at least one machine learning model (e.g., at least one of a fraud detection model, risk detection model, transaction authorization model, credit approval model, product recommendation model, classifier model, anomaly detection model, authentication model, any combination thereof, etc.). For example, one or more machine learning models may include at least one of one or more transformers described herein and / or at least one of one or more other machine learning models. For example, transaction processing system 301 and / or issuer system 306 may include transformer management system 102, etc. Transaction processing system 301 and / or issuer system 306 may perform at least one task (e.g., generating predictions and / or generating embeddings) based on authorization requests and one or more machine learning models. For example, performing one or more tasks may include generating at least one prediction associated with fraud detection, risk detection, transaction authorization, credit approval, product recommendation, classification, anomaly detection, authentication, any combination thereof, etc. In some non-limiting embodiments or aspects, transaction processing system 301 may transmit at least one message to issuer system 306 (e.g., along with an authorization request) based on the execution of a task (e.g., generating a prediction and / or generating an embedding). In some non-limiting embodiments or aspects, issuer system 306 may determine an authorization decision (e.g., approval, rejection, etc.) based on the authorization request and the execution of the task (e.g., generation of a prediction and / or generation of an embedding).

[0167] For illustrative purposes, transaction clearing and / or settlement may include generating messages (e.g., clearing messages, etc.) based on a customer's (e.g., associated with customer device 310, etc.) account identifier and / or transaction data associated with the transaction. For example, merchant system 304 may generate at least one clearing message (e.g., multiple clearing messages, a batch of clearing messages, etc.). Merchant system 304 may transmit one or more clearing messages to acquiring system 308 (and / or payment gateway 302, which may transmit one or more clearing messages to acquiring system 308). Acquiring system 308 may transmit one or more clearing messages to transaction processing system 301. Transaction processing system 301 may transmit one or more clearing messages to issuing system 306. Issuing system 306 may generate at least one settlement message based on one or more clearing messages. In some non-limiting embodiments or aspects, the issuer system 306 may transmit one or more settlement messages and / or funds to the transaction processing system 301 (and / or the settlement bank system associated with the transaction processing system 301), and the transaction processing system 301 (and / or the settlement bank system) may transmit one or more settlement messages and / or funds to the acquiring system 308. Alternatively, the issuer system 306 may transmit one or more settlement messages and / or funds to the acquiring system 308. In some non-limiting embodiments or aspects, the acquiring system 308 may transmit one or more settlement messages and / or funds to the merchant system 304 (and / or the account associated with the merchant system 304).

[0168] Figure 3 The system and / or device may communicate via one or more wired and / or wireless communication networks. For example, one or more communication networks may include cellular networks (e.g., Long Term Evolution (LTE®) networks, third-generation (3G) networks, fourth-generation (4G) networks, fifth-generation (5G) networks, Code Division Multiple Access (CDMA) networks, etc.), Public Land Mobile Networks (PLMN), Local Area Networks (LAN), Wide Area Networks (WAN), Metropolitan Area Networks (MAN), Telephone Networks (e.g., Public Switched Telephone Network (PSTN)), Private Networks (e.g., Private Networks Associated with Transaction Service Providers), Temporary Networks, Intranets, the Internet, Fiber-based Networks, Cloud Computing Networks, etc., and / or combinations of these or other types of networks.

[0169] Provided as an example Figure 3 The number and arrangement of systems, devices, and / or networks are shown. Additional systems, devices, and / or networks may exist; fewer systems, devices, and / or networks may exist; different systems, devices, and / or networks may exist; and / or networks may be associated with... Figure 3 The different arrangements of systems, equipment, and / or networks are shown. Furthermore, implementation can be carried out within a single system or device. Figure 3 The two or more systems or devices shown, or Figure 3 The single system or device shown may be implemented as multiple distributed systems or devices. Alternatively, a group of systems (e.g., one or more systems) or a group of devices (e.g., one or more devices) of the payment processing network 300 may perform one or more functions described as being performed by another group of systems or devices of the payment processing network 300.

[0170] For reference Figure 4 The diagram illustrates example components of a device 400 according to a non-limiting embodiment. As an example, device 400 may correspond to... Figure 1 The converter management system 102, the transaction service provider system 104 and / or the user device 106 and / or Figure 3 The system includes a transaction processing system 301, a payment gateway system 302, a merchant system 304, an issuer system 306, an acquirer system 308, and / or a consumer device 310. In some non-limiting embodiments, such a system or device may include at least one device 400 and / or at least one component of device 400. The number and arrangement of components shown are provided as examples. In some non-limiting embodiments, device 400 may include additional components, fewer components, different components, or components arranged differently compared to those shown. Additionally or alternatively, a set of components of device 400 (e.g., one or more components) may perform one or more functions described as being performed by another set of components of device 400.

[0171] like Figure 4 As shown, device 400 may include bus 402, processor 404, memory 406, storage component 408, input component 410, output component 412, and communication interface 414. Bus 402 may include components that allow communication between components of device 400. In some non-limiting embodiments, processor 404 may be implemented in hardware, firmware, or a combination of hardware and software. For example, processor 404 may include processors (e.g., central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), etc.), microprocessors, digital signal processors (DSPs), and / or any processing component that can be programmed to perform functions (e.g., field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), etc.). Memory 406 may include random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by processor 404.

[0172] Continue to refer to Figure 4Storage component 408 may store information and / or software related to the operation and use of device 400. For example, storage component 408 may include a hard disk (e.g., magnetic disk, optical disk, magneto-optical disk, solid-state disk, etc.) and / or another type of computer-readable medium. Input component 410 may include components that allow device 400 to receive information, for example, through user input (e.g., touch screen display, keyboard, keypad, mouse, buttons, switches, microphone, etc.). Alternatively, input component 410 may include sensors for sensing information (e.g., Global Positioning System (GPS) components, accelerometers, gyroscopes, actuators, etc.). Output component 412 may include components that provide output information from device 400 (e.g., display, speaker, one or more light-emitting diodes (LEDs), etc.). Communication interface 414 may include transceiver-like components (e.g., transceivers, separate receivers and transmitters, etc.) that enable device 400 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 414 may allow device 400 to receive information from another device and / or provide information to another device. For example, communication interface 414 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a cellular network interface, and / or the like.

[0173] Device 400 can perform one or more processes described herein. Device 400 can perform these processes based on processor 404 executing software instructions stored in a computer-readable medium such as memory 406 and / or storage unit 408. The computer-readable medium can include any non-transient memory device. Memory devices include memory space located within a single physical storage device or memory space distributed across multiple physical storage devices. Software instructions can be read into memory 406 and / or storage unit 408 via communication interface 414 from another computer-readable medium or from another device. When executed, the software instructions stored in memory 406 and / or storage unit 408 cause processor 404 to perform one or more processes described herein. Additionally or alternatively, hard-wired circuitry may be used in place of or in conjunction with the software instructions to perform one or more processes described herein. Therefore, the embodiments described herein are not limited to any particular combination of hardware circuitry and software. As used herein, the term “configured to” can refer to an arrangement of software, devices, and / or hardware for performing and / or implementing one or more functions (e.g., actions, processes, steps of processes, and / or the like). For example, "a processor configured to..." can refer to a processor that executes software instructions (such as program code) that cause the processor to perform one or more functions.

[0174] Now for reference Figures 8A to 8CThe diagram illustrates the output of an example implementation of a system and method for a data type-aware transformer machine learning model for sequential datasets, according to some non-limiting embodiments or aspects.

[0175] like Figure 8A As shown, scatter plot 801 may include points for fraudulent transactions 811 and non-fraudulent transactions 812. For example, each point may represent the first three principal components (e.g., based on principal component analysis (PCA)) of a windowed transaction sequence (e.g., a transaction sequence including a given transaction and a selected number of previous transactions) via concatenated sequence embeddings (e.g., based on concatenated static and dynamic sequence embedding representations). These sequence embeddings can be generated after a pre-trained transformer, as described herein (e.g., without explicitly fine-tuning the fraud detection model). Figure 8A As shown, there may be a relatively clear separation between fraudulent transaction 811 and non-fraudulent transaction 812, indicating that the sequence embedding has captured this information (e.g., even without fine-tuning).

[0176] like Figure 8B As shown, scatter plot 802 may include points for cardless (CNP) sequences 821 (e.g., sequences with CNP as the most frequent type of transaction) and non-CNP sequences 822 (e.g., sequences with non-CNP as the most frequent type of transaction). For example, each point may represent the first three principal components (e.g., PCA-based) of a windowed transaction sequence (e.g., a transaction sequence including a given transaction and a selected number of previous transactions) through concatenated sequence embeddings (e.g., based on concatenated static and dynamic sequence embedding representations). These sequence embeddings may be generated after a pre-trained transformer, as described herein (e.g., without explicitly fine-tuning the classifier model). Figure 8B As shown, there may be a relatively clear separation between CNP sequence 821 and non-CNP sequence 822, indicating that the sequence embedding has captured this information (e.g., even without fine-tuning).

[0177] like Figure 8C As shown, scatter plot 803 may include points for anomalous transactions 831 and normal transactions 832. For example, each point may represent the first three principal components (e.g., based on PCA) of a windowed transaction sequence (e.g., a transaction sequence including a given transaction and a selected number of previous transactions) through concatenated sequence embeddings (e.g., based on concatenated static and dynamic sequence embedding representations). These sequence embeddings can be generated after a pre-trained transformer, as described herein (e.g., without explicitly fine-tuning the anomaly detection model). Figure 8C As shown, there may be a relatively clear separation between abnormal transaction 831 and normal transaction 832, indicating that the sequence embedding has captured this information (e.g., even without fine-tuning).

[0178] Table 1 shows a comparison of the area under the curve (AUC) of implementations of the published topic (e.g., FATA-Trans) based on the same dataset (e.g., synthetic transaction dataset) and three other types of machine learning models (e.g., Light Gradient Augmentation Machine (LightGBM), Recurrent Neural Network (RNN), and Tabular Bidirectional Encoder Representation from Transformer (TabBERT)).

[0179] Table 1

[0180] As shown in Table 1, FATA-Trans consistently outperforms other types of machine learning models. This indicates that FATA-Trans effectively captures more accurate user behavior patterns by leveraging time interval and field type information incorporated into specially designed embedding and transformer layers.

[0181] Table 2 shows a comparison of pre-training time between FATA-Trans and TabBERT based on the same dataset (e.g., synthetic transaction dataset) and the same hardware setup.

[0182] Table 2

[0183] As shown in Table 2, FATA-Trans demonstrates a significantly shorter pre-training time compared to TabBERT. This can be attributed at least in part to the fact that FATA-Trans avoids redundant repetition of static fields in the sequence and only inputs these static fields into the static field transformer. As a result, FATA-Trans reduces memory usage and significantly saves training time.

[0184] Table 3 shows a comparison of fraud detection tasks using FATA-Trans and TabBERT, along with a modified version of TabBERT that utilizes time-aware location embeddings (e.g., TabBERT-TP), and a modified version of FATA-Trans that utilizes simple location embeddings instead of time-aware location embeddings (e.g., FATA-Trans-SO). Comparisons are also provided for performing fraud detection tasks on raw data without any pre-trained models (e.g., not applicable). Two different types of classifier models are tested: Multilayer Perception (MLP) and Bidirectional Long Short-Term Memory (Bi-LSTM).

[0185] Table 3

[0186] As shown in Table 3, FATA-Trans-SO, TabBERT-TP, and FATA-Trans show improved AUC scores compared to using the original field. Furthermore, in most cases, FATA-Trans-SO, TabBERT-TP, and FATA-Trans outperform the original TabBERT.

[0187] Although embodiments have been described in detail for illustrative purposes, it should be understood that such details are for the purposes described only, and this disclosure is not limited to the disclosed embodiments or aspects, but rather is intended to cover modifications and equivalent arrangements that fall within the spirit and scope of the appended claims. For example, it should be understood that this disclosure contemplates, as far as possible, that one or more features of any embodiment or aspect may be combined with one or more features of any other embodiment or aspect.

Claims

1. A converter management system applied to a payment processing network, comprising: At least one processor is configured to execute program instructions stored in at least one non-transitory computer-readable medium to perform: Receive sequential transaction data associated with multiple sequential transactions, the sequential transaction data including multiple transaction records, wherein the transaction records include multiple field types of structured data and each field has a value, wherein the transaction records include the time, order and position of each transaction in the multiple sequential transactions; For each of the plurality of transaction records, a plurality of static fields and a plurality of dynamic fields are separated, wherein each of the plurality of static fields includes a corresponding value copied in each of the plurality of transaction records, and wherein each of the plurality of dynamic fields includes a value of at least one of the plurality of transaction records, the value of the at least one transaction record being different from another value of the dynamic field of at least one other transaction record in the plurality of transaction records; A static transaction embedding representation is generated by inputting static field data associated with the plurality of static fields into a first transformer model, the first transformer model including at least one first self-attention mechanism; Multiple dynamic transaction embedding representations are generated by inputting dynamic field data associated with at least one dynamic field of the transaction record sequence into a second transformer model. The dynamic transaction embedding representations include a corresponding dynamic transaction embedding for each transaction record in the transaction record sequence. The second transformer model includes at least one second self-attention mechanism. The transaction record sequence includes at least one subset of the multiple transaction records. Generate a first field type embedding representation associated with the plurality of static fields, a second field type embedding representation associated with the plurality of dynamic fields, a first time- and location-based embedding representation associated with the plurality of static fields, and a plurality of time- and location-based embedding representations associated with the plurality of dynamic fields, wherein the first time- and location-based embedding representation and the plurality of time- and location-based embedding representations are obtained based on the time and location associated with the transaction; The first intermediate input is generated based on the static transaction embedding representation, the first time- and location-based embedding representation, and the first field type embedding representation; Multiple second intermediate inputs are generated based on each dynamic transaction embedding representation, the corresponding time- and location-based embedding representations of the multiple time- and location-based embedding representations, and the second field type embedding representation; A static sequence embedding representation associated with the transaction record sequence is generated by inputting the first intermediate input into a third transformer model, the third transformer model including at least one third self-attention mechanism; Multiple dynamic sequence embedding representations associated with the transaction record sequence are generated by inputting the multiple second intermediate inputs into the third transformer model; and At least one prediction is generated by inputting the static sequence embedding representation and the plurality of dynamic sequence embedding representations into a machine learning model, wherein the at least one prediction is associated with fraud detection, and Based on the at least one prediction, perform actions associated with the fraud detection task.

2. The converter management system according to claim 1, wherein generating the first intermediate input includes: The static transaction embedding representation, the first time- and location-based embedding representation, and the first field type embedding representation associated with the plurality of static fields are combined.

3. The converter management system according to claim 1, wherein generating the plurality of dynamic transaction embedding representations includes: A first dynamic transaction embedding associated with the first transaction record is generated by inputting first dynamic field data associated with the plurality of dynamic fields of the first transaction record into the second transformer model; as well as A second dynamic transaction embedding associated with the second transaction record is generated by inputting second dynamic field data associated with the plurality of dynamic fields of the second transaction record into the second transformer model; and The at least one processor is further configured to: Generate a first time- and location-based embedded representation associated with the first transaction record; as well as Generate a second time- and location-based embedded representation associated with the second transaction record; and The generation of the plurality of second intermediate inputs includes: The first dynamic transaction embedding, the first time- and location-based embedding representation, and the second field-type embedding representation associated with the plurality of dynamic fields are combined; as well as Combine the second dynamic transaction embedding, the second time- and location-based embedding representation, and the second field type embedding representation associated with the plurality of dynamic fields.

4. The converter management system of claim 3, wherein combining the first dynamic transaction embedding, the first time- and location-based embedding representation, and the second field-type embedding representation associated with the at least one dynamic field comprises: The summation is performed on the first dynamic transaction embedding, the first time- and location-based embedding representation, and the second field-type embedding representation associated with the at least one dynamic field; and The combination of the second dynamic transaction embedding, the second time- and location-based embedding representation, and the second field-type embedding representation associated with the plurality of dynamic fields includes: The second dynamic transaction embedding, the second time- and location-based embedding representation, and the second field type embedding representation associated with the plurality of dynamic fields are summed.

5. The converter management system according to claim 1, wherein the at least one processor is further configured to: The first input to the first converter model is generated based on the static field data associated with the plurality of static fields; and The second input to the second transformer model is generated based on the dynamic field data associated with the plurality of dynamic fields.

6. The converter management system according to claim 1, wherein the at least one processor is further configured to: Before inputting the dynamic field data associated with the plurality of dynamic fields of the transaction record sequence into the second transformer model, the original values ​​of the dynamic fields of the first transaction record in the transaction record sequence are masked to provide the masked dynamic fields of the first transaction record. The generation of the plurality of dynamic transaction embedding representations based on inputting the dynamic field data associated with the plurality of dynamic fields of the transaction record sequence into the second transformer model includes: The multiple dynamic transaction embedding representations are generated by inputting the masked dynamic fields of the first transaction record into the second transformer model; and The at least one processor is further configured to: The third transformer model is trained by comparing the data value of the data field represented by the dynamic sequence embedding associated with the first transaction record provided by the third transformer model with the original value of the dynamic field of the first transaction record, and by adjusting the parameters of the third transformer model based on the comparison of the data value of the data field represented by the dynamic sequence embedding associated with the first transaction record provided by the third transformer with the original value of the dynamic field of the first transaction record.

7. A computer program product for use in a payment processing network, comprising at least one non-transient computer-readable medium, said at least one non-transient computer-readable medium comprising program instructions that, when executed by at least one processor of a transducer management system, cause said at least one processor to: Receive sequential transaction data associated with multiple sequential transactions, the sequential transaction data including multiple transaction records, wherein, The transaction record includes multiple field types of structured data, and each field has a value, wherein the transaction record includes the time, order, and position of each transaction in a series of sequential transactions; For each of the plurality of transaction records, a plurality of static fields and a plurality of dynamic fields are separated, wherein each of the plurality of static fields includes a corresponding value copied in each of the plurality of transaction records, and wherein each of the plurality of dynamic fields includes a value of at least one of the plurality of transaction records, the value of the at least one transaction record being different from another value of the dynamic field of at least one other transaction record in the plurality of transaction records; A static transaction embedding representation is generated by inputting static field data associated with the plurality of static fields into a first transformer model, the first transformer model including at least one first self-attention mechanism; Multiple dynamic transaction embedding representations are generated by inputting dynamic field data associated with the multiple dynamic fields of the transaction record sequence into a second transformer model. The dynamic transaction embedding representations include a corresponding dynamic transaction embedding for each transaction record in the transaction record sequence. The second transformer model includes at least one second self-attention mechanism. The transaction record sequence includes at least a subset of the multiple transaction records. Generate a first field type embedding representation associated with the plurality of static fields, a second field type embedding representation associated with the plurality of dynamic fields, a first time- and location-based embedding representation associated with the plurality of static fields, and a plurality of time- and location-based embedding representations associated with the plurality of dynamic fields, wherein the first time- and location-based embedding representation and the plurality of time- and location-based embedding representations are obtained based on the time and location associated with the transaction; The first intermediate input is generated based on the static transaction embedding representation, the first time- and location-based embedding representation, and the first field type embedding representation; Multiple second intermediate inputs are generated based on each dynamic transaction embedding representation, the corresponding time- and location-based embedding representations of the multiple time- and location-based embedding representations, and the second field type embedding representation; A static sequence embedding representation associated with the transaction record sequence is generated by inputting the first intermediate input into a third transformer model, the third transformer model including at least one third self-attention mechanism; Multiple dynamic sequence embedding representations associated with the transaction record sequence are generated by inputting the multiple second intermediate inputs into the third transformer model; as well as At least one prediction is generated by inputting the static sequence embedding representation and the plurality of dynamic sequence embedding representations into a machine learning model, wherein the at least one prediction is associated with fraud detection, and Based on the at least one prediction, perform actions associated with the fraud detection task.

8. The computer program product of claim 7, wherein generating the first intermediate input comprises: The static transaction embedding representation, the first time- and location-based embedding representation, and the first field type embedding representation associated with the plurality of static fields are combined.

9. The computer program product of claim 7, wherein generating the plurality of dynamic transaction embedding representations comprises: A first dynamic transaction embedding associated with the first transaction record is generated by inputting first dynamic field data associated with the plurality of dynamic fields of the first transaction record into the second transformer model; as well as The second dynamic transaction embedding associated with the second transaction record is generated by inputting the second dynamic field data associated with the plurality of dynamic fields of the second transaction record into the second transformer model; The instructions, when executed by the at least one processor, also cause the at least one processor to: Generate a first time- and location-based embedded representation associated with the first transaction record; as well as Generate a second time- and location-based embedded representation associated with the second transaction record; The generation of the plurality of second intermediate inputs includes: Combining the first dynamic transaction embedding, the first time- and location-based embedding representation, and the second field-type embedding representation associated with the plurality of dynamic fields; and The second dynamic transaction embedding, the second time- and location-based embedding representation, and the second field type embedding representation associated with the plurality of dynamic fields are combined; The combination of the first dynamic transaction embedding, the first time- and location-based embedding representation, and the second field-type embedding representation associated with the plurality of dynamic fields includes: The summation is performed on the first dynamic transaction embedding, the first time- and location-based embedding representation, and the second field-type embedding representation associated with the plurality of dynamic fields; and The combination of the second dynamic transaction embedding, the second time- and location-based embedding representation, and the second field-type embedding representation associated with the plurality of dynamic fields includes: The second dynamic transaction embedding, the second time- and location-based embedding representation, and the second field type embedding representation associated with the plurality of dynamic fields are summed.

10. The computer program product of claim 7, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to: The first input to the first converter model is generated based on the static field data associated with the plurality of static fields; and The second input to the second transformer model is generated based on the dynamic field data associated with the plurality of dynamic fields.

11. The computer program product of claim 7, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to: Before inputting the dynamic field data associated with the plurality of dynamic fields of the transaction record sequence into the second transformer model, the original values ​​of the dynamic fields of the first transaction record in the transaction record sequence are masked to provide the masked dynamic fields of the first transaction record. The generation of the plurality of dynamic transaction embedding representations based on inputting the dynamic field data associated with the plurality of dynamic fields of the transaction record sequence into the second transformer model includes: The multiple dynamic transaction embedding representations are generated by inputting the masked dynamic fields of the first transaction record into the second transformer model; and The instructions, when executed by the at least one processor, also cause the at least one processor to: The third transformer model is trained by comparing the data value of the data field represented by the dynamic sequence embedding associated with the first transaction record provided by the third transformer model with the original value of the dynamic field of the first transaction record, and by adjusting the parameters of the third transformer model based on the comparison of the data value of the data field represented by the dynamic sequence embedding associated with the first transaction record provided by the third transformer with the original value of the dynamic field of the first transaction record.

12. A computer-implemented method applied to a payment processing network, comprising: The system utilizes at least one processor of a converter management system to receive sequential transaction data associated with multiple sequential transactions. The sequential transaction data includes multiple transaction records, wherein each transaction record includes multiple field types of structured data, and each field has a value. The transaction record includes the time, order, and position of each transaction in the multiple sequential transactions. For each of the plurality of transaction records, at least one processor of the transformer management system is used to separate a plurality of static fields and a plurality of dynamic fields, wherein each of the plurality of static fields includes a corresponding value copied in each of the plurality of transaction records, and each of the plurality of dynamic fields includes a value of at least one of the plurality of transaction records, wherein the value of at least one transaction record is different from another value of the dynamic field of at least one other transaction record in the plurality of transaction records; A static transaction embedding representation is generated by using at least one processor of a transformer management system based on inputting static field data associated with the plurality of static fields into a first transformer model, the first transformer model including at least one first self-attention mechanism; The system utilizes at least one processor of a transformer management system to generate multiple dynamic transaction embedding representations based on inputting dynamic field data associated with the multiple dynamic fields of a transaction record sequence into a second transformer model. The dynamic transaction embedding representations include a corresponding dynamic transaction embedding for each transaction record in the transaction record sequence. The second transformer model includes at least one second self-attention mechanism. The transaction record sequence includes at least a subset of the multiple transaction records. The system utilizes at least one processor of the transformer management system to generate a first field type embedding representation associated with the plurality of static fields, a second field type embedding representation associated with the plurality of dynamic fields, a first time- and location-based embedding representation associated with the plurality of static fields, and a plurality of time- and location-based embedding representations associated with the plurality of dynamic fields, wherein the first time- and location-based embedding representation and the plurality of time- and location-based embedding representations are obtained based on the time and location associated with the transaction; The first intermediate input is generated using at least one processor of the transformer management system based on the static transaction embedding representation, the first time- and location-based embedding representation, and the first field-type embedding representation; The system utilizes at least one processor of the transformer management system to generate multiple second intermediate inputs based on each dynamic transaction embedding representation, the corresponding time- and location-based embedding representations of the multiple time- and location-based embedding representations, and the second field type embedding representation; At least one processor of the transformer management system generates a static sequence embedding representation associated with the transaction record sequence based on inputting the first intermediate input into a third transformer model, the third transformer model including at least one third self-attention mechanism; The system utilizes at least one processor of the transformer management system to generate multiple dynamic sequence embedding representations associated with the transaction record sequence based on inputting the multiple second intermediate inputs into the third transformer model; and At least one processor of the transformer management system generates at least one prediction based on inputting the static sequence embedding representation and the plurality of dynamic sequence embedding representations into a machine learning model, wherein the at least one prediction is associated with fraud detection. The system utilizes at least one processor of the converter management system to perform actions associated with the fraud detection task based on the at least one prediction.

13. The computer-implemented method of claim 12, wherein generating the first intermediate input comprises: The static transaction embedding representation, the first time- and location-based embedding representation, and the first field type embedding representation associated with the plurality of static fields are combined.

14. The computer-implemented method of claim 12, wherein generating the plurality of dynamic transaction embedding representations comprises: A first dynamic transaction embedding associated with the first transaction record is generated by inputting first dynamic field data associated with the plurality of dynamic fields of the first transaction record into the second transformer model; as well as The second dynamic transaction embedding associated with the second transaction record is generated by inputting the second dynamic field data associated with the plurality of dynamic fields of the second transaction record into the second transformer model; The computer-implemented method further includes: Generate a first time- and location-based embedded representation associated with the first transaction record; and Generate a second time- and location-based embedded representation associated with the second transaction record; and The generation of the plurality of second intermediate inputs includes: Combining the first dynamic transaction embedding, the first time- and location-based embedding representation, and the second field-type embedding representation associated with the plurality of dynamic fields; and Combine the second dynamic transaction embedding, the second time- and location-based embedding representation, and the second field type embedding representation associated with the plurality of dynamic fields.

15. The computer-implemented method of claim 14, wherein combining the first dynamic transaction embedding, the first time- and location-based embedding representation, and the second field-type embedding representation associated with the plurality of dynamic fields comprises: The summation is performed on the first dynamic transaction embedding, the first time- and location-based embedding representation, and the second field-type embedding representation associated with the plurality of dynamic fields; and The combination of the second dynamic transaction embedding, the second time- and location-based embedding representation, and the second field-type embedding representation associated with the plurality of dynamic fields includes: The second dynamic transaction embedding, the second time- and location-based embedding representation, and the second field type embedding representation associated with the plurality of dynamic fields are summed.

16. The computer-implemented method according to claim 12, further comprising: The first input to the first converter model is generated based on the static field data associated with the plurality of static fields; as well as The second input to the second transformer model is generated based on the dynamic field data associated with the plurality of dynamic fields.

17. The computer-implemented method of claim 12, wherein the method further comprises: Before inputting the dynamic field data associated with the plurality of dynamic fields of the transaction record sequence into the second transformer model, the original values ​​of the dynamic fields of the first transaction record in the transaction record sequence are masked to provide the masked dynamic fields of the first transaction record. The generation of the plurality of dynamic transaction embedding representations based on inputting the dynamic field data associated with the plurality of dynamic fields of the transaction record sequence into the second transformer model includes: The multiple dynamic transaction embedding representations are generated by inputting the masked dynamic fields of the first transaction record into the second transformer model; and The computer-implemented method further includes: The third transformer model is trained by comparing the data value of the data field represented by the dynamic sequence embedding associated with the first transaction record provided by the third transformer model with the original value of the dynamic field of the first transaction record, and by adjusting the parameters of the third transformer model based on the comparison of the data value of the data field represented by the dynamic sequence embedding associated with the first transaction record provided by the third transformer with the original value of the dynamic field of the first transaction record.

Images