System, method, and computer program product for generating improved embeddings for machine learning based on token type

Abstract

Systems, methods, and computer program products are provided for generating improved embeddings for machine learning based on token type. The system includes a processor configured to receive and separate text data associated with a string of text into mathematical tokens of a mathematical token type and remaining tokens of a remaining token type. The processor may be configured to input the mathematical tokens into a first machine learning model (MLM) associated with the mathematical token type to generate first embeddings comprising a respective first embedding for each respective mathematical token. The processor may be configured to input the remaining tokens into a second MLM associated with the remaining token type to generate second embeddings comprising a respective second embedding for each respective remaining token of the remaining token type. The processor may be configured to input the first and second embeddings into a third MLM to perform a task.

Description

SYSTEM, METHOD, AND COMPUTER PROGRAM PRODUCT FOR GENERATING IMPROVED EMBEDDINGS FOR MACHINE LEARNING BASED ON TOKEN TYPEBACKGROUND1 . Technical Field

[0001] This disclosure relates generally to generating improved embeddings and, in non-limiting embodiments or aspects, to systems, methods, and computer program products for generating improved embeddings for machine learning based on token type.2. Technical Considerations

[0002] Machine learning models are successful in performing many real-world tasks. For example, a machine learning model may be trained to receive an input of a particular type of data and perform a task (e.g., generate an output, such as a prediction, a classification, or the like) based on the particular type of input. For the purpose of illustration, a natural language processing (NLP) model may be designed to receive text data (e.g., including words, punctuation, etc.) as input and to perform a task (e.g., a NLP task) based on the text data (e.g., based on the words, punctuation, etc.). Similarly, generative machine learning models (and / or generative artificial intelligence models) may by trained based on a particular type of data to generate new data of that type. For example, a large language model (LLM) may be trained based on text data to generate new text (e.g., predict the next word(s), next sentence(s), and / or the like).

[0003] However, text data can include different types of information that should not necessarily be treated in the same way. For example, text data may include mathematical data (e.g., numbers, words associated with numbers, mathematical operators, words associated with mathematical operations, etc.) as well as other data (e.g., other words, punctuation, etc.). NLP models treat the mathematical data as a string of text (e.g., words), which may be tokenized. Such treatment of mathematical data by NLP models compromises the meaning of the mathematical data. For example, treating mathematical data as a string of text (e.g., in the same manner as words in a natural language sentence) may fail to preserve the mathematical meaning of the items of mathematical data (e.g., numbers, words associated with numbers, mathematical operators, words associated with mathematical operations, etc.) and / or may reduce the accuracy of the machine learning model that received suchmathematical data as (part of) its input. Additionally, if embeddings are generated based on the mathematical data to be used as input to other machine learning models designed to perform different tasks, the accuracy and / or effectiveness of such other machine learning models may be reduced.SUMMARY

[0004] Accordingly, provided are improved systems, methods, and computer program products for generating improved embeddings for machine learning based on token type.

[0005] According to non-limiting embodiments or aspects, provided is a system for generating improved embeddings for machine learning based on token type. In some non-limiting embodiments or aspects, the system may include at least one processor. In some non-limiting embodiments or aspects, the at least one processor may be configured to receive text data associated with a string of text. In some non-limiting embodiments or aspects, the at least one processor may be further configured to separate the text data into a plurality of mathematical tokens of a mathematical token type and a plurality of remaining tokens of a remaining token type. In some nonlimiting embodiments or aspects, the at least one processor may be further configured to input the plurality of mathematical tokens into at least one first machine learning model associated with the mathematical token type to generate a plurality of first embeddings including a respective first embedding for each respective mathematical token of the plurality of mathematical tokens. In some non-limiting embodiments or aspects, the at least one processor may be further configured to input the plurality of remaining tokens into at least one second machine learning model associated with the remaining token type to generate a plurality of second embeddings including a respective second embedding for each respective remaining token of the plurality of remaining tokens. In some non-limiting embodiments or aspects, the at least one second machine learning model may be different than the at least one first machine learning model. In some non-limiting embodiments or aspects, the at least one processor may be further configured to input the plurality of first embeddings and the plurality of second embeddings into at least one third machine learning model to perform a task.

[0006] In some non-limiting embodiments or aspects, the at least one first machine learning model may be configured to receive the plurality of mathematical tokens ofthe mathematical token type. The at least one second machine learning model may be configured to receive the plurality of remaining tokens of the remaining token type.

[0007] In some non-limiting embodiments or aspects, the text data may include a plurality of tokens. In some non-limiting embodiments or aspects, when separating the text data into the plurality of mathematical tokens of the mathematical token type and the plurality of remaining tokens of the remaining token type, the at least one processor may be further configured to determine whether each token of the plurality of tokens is of the mathematical token type. In some non-limiting embodiments or aspects, the at least one processor may be further configured to separate a first subset of the plurality of tokens into the plurality of mathematical tokens based on determining that each token of the first subset of the plurality of tokens is of the mathematical token type and separate a second subset of the plurality of tokens into the plurality of remaining tokens based on determining that each token of the second subset of the plurality of tokens is not of the mathematical token type.

[0008] In some non-limiting embodiments or aspects, the at least one first machine learning model may include an encoding layer and a sequence layer. In some nonlimiting embodiments or aspects, the at least one processor may be further configured to train the encoding layer to provide an initial embedding for each respective mathematical token of the plurality of mathematical tokens, mask at least one mathematical token of the plurality of mathematical tokens to provide at least one masked mathematical token, and train the sequence layer to predict a value of the at least one masked mathematical token based on the initial embedding of each other mathematical token of the plurality of mathematical tokens.

[0009] In some non-limiting embodiments or aspects, the encoding layer may include at least one of a neural network, a deep neural network (DNN), an autoencoder, or any combination thereof.

[0010] In some non-limiting embodiments or aspects, the sequence layer may include at least one of a transformer model, a Long Short-Term Memory (LSTM) model, a recurrent neural network (RNN), or any combination thereof.

[0011] In some non-limiting embodiments or aspects, the sequence layer may include the transformer model, and the sequence layer may generate a plurality of refined embeddings based on the initial embedding for each respective mathematical token of the plurality of mathematical tokens.

[0012] In some non-limiting embodiments or aspects, the sequence layer may include the LSTM model, and the sequence layer may generate a hidden state based on the initial embedding for each respective mathematical token of the plurality of mathematical tokens.

[0013] In some non-limiting embodiments or aspects, when training the sequence layer, the at least one processor may be configured to randomly generate an arithmetic statement. In some non-limiting embodiments or aspects, the text data may include the arithmetic statement and masking may include randomly selecting the at least one mathematical token.

[0014] In some non-limiting embodiments or aspects, the task may include generating a plurality of third embeddings based on the plurality of first embeddings and the plurality of second embeddings.

[0015] In some non-limiting embodiments or aspects, the task may include a natural language processing (NLP) task.

[0016] In some non-limiting embodiments or aspects, the task may include one of a fraud detection task, a risk detection task, or transaction data generation task.

[0017] According to non-limiting embodiments or aspects, provided is a computer- implemented method for generating improved embeddings for machine learning based on token type. In some non-limiting embodiments or aspects, the computer- implemented method may include receiving text data associated with a string of text. In some non-limiting embodiments or aspects, the computer-implemented method may include separating the text data into a plurality of mathematical tokens of a mathematical token type and a plurality of remaining tokens of a remaining token type. In some non-limiting embodiments or aspects, the computer-implemented method may include inputting the plurality of mathematical tokens into at least one first machine learning model associated with the mathematical token type to generate a plurality of first embeddings including a respective first embedding for each respective mathematical token of the plurality of mathematical tokens. In some non-limiting embodiments or aspects, the computer-implemented method may include inputting the plurality of remaining tokens into at least one second machine learning model associated with the remaining token type to generate a plurality of second embeddings including a respective second embedding for each respective remaining token of the plurality of remaining tokens. In some non-limiting embodiments or aspects, the at least one second machine learning model may be different than the at least one firstmachine learning model. In some non-limiting embodiments or aspects, the computer- implemented method may include inputting the plurality of first embeddings and the plurality of second embeddings into at least one third machine learning model to perform a task.

[0018] In some non-limiting embodiments or aspects, the at least one first machine learning model may be configured to receive the plurality of mathematical tokens of the mathematical token type. The at least one second machine learning model may be configured to receive the plurality of remaining tokens of the remaining token type.

[0019] In some non-limiting embodiments or aspects, the text data may include a plurality of tokens. In some non-limiting embodiments or aspects, separating the text data into the plurality of mathematical tokens of the mathematical token type and the plurality of remaining tokens of the remaining token type may include determining whether each token of the plurality of tokens is of the mathematical token type. In some non-limiting embodiments or aspects, the method may include separating the first subset of the plurality of tokens into the plurality of mathematical tokens based on determining that each token of the first subset of the plurality of tokens is of the mathematical token type and separating a second subset of the plurality of tokens into the plurality of remaining tokens based on determining that each token of the second subset of the plurality of tokens is not of the mathematical token type.

[0020] In some non-limiting embodiments or aspects, the at least one first machine learning model may include an encoding layer and a sequence layer. In some nonlimiting embodiments or aspects, the computer-implemented method may further include training the encoding layer to provide an initial embedding for each respective mathematical token of the plurality of mathematical tokens, masking at least one mathematical token of the plurality of mathematical tokens to provide at least one masked mathematical token, and training the sequence layer to predict a value the at least one masked mathematical token based on the initial embedding of each other mathematical token of the plurality of mathematical tokens.

[0021] In some non-limiting embodiments or aspects, the encoding layer may include at least one of a neural network, a deep neural network (DNN), an autoencoder, or any combination thereof.

[0022] In some non-limiting embodiments or aspects, the sequence layer may include at least one of a transformer model, a Long Short-Term Memory (LSTM) model, a recurrent neural network (RNN), or any combination thereof.

[0023] In some non-limiting embodiments or aspects, the sequence layer may include the transformer model, and the sequence layer may generate a plurality of refined embeddings based on the initial embedding for each respective mathematical token of the plurality of mathematical tokens.

[0024] In some non-limiting embodiments or aspects, the sequence layer may include the LSTM model, and the sequence layer may generate a hidden state based on the initial embedding for each respective mathematical token of the plurality of mathematical tokens.

[0025] In some non-limiting embodiments or aspects, training the sequence layer may include randomly generating an arithmetic statement. In some non-limiting embodiments or aspects, the text data may include the arithmetic statement and masking may include randomly selecting the at least one mathematical token.

[0026] In some non-limiting embodiments or aspects, the task may include generating a plurality of third embeddings based on the plurality of first embeddings and the plurality of second embeddings.

[0027] In some non-limiting embodiments or aspects, the task may include a NLP task.

[0028] In some non-limiting embodiments or aspects, the task may include one of a fraud detection task, a risk detection task, or generating transaction data.

[0029] According to non-limiting embodiments or aspects, provided is a computer program product for generating improved embeddings for machine learning based on token type. In some non-limiting embodiments or aspects, the computer program product may include at least one non-transitory computer-readable medium including program instructions. In some non-limiting embodiments or aspects, when executed by at least one processor, the instructions may cause the at least one processor to receive text data associated with a string of text. In some non-limiting embodiments or aspects, the instructions may further cause the at least one processor to separate the text data into a plurality of mathematical tokens of a mathematical token type and a plurality of remaining tokens of a remaining token type. In some non-limiting embodiments or aspects, the instructions may further cause the at least one processor to input the plurality of mathematical tokens into at least one first machine learning model associated with the mathematical token type to generate a plurality of first embeddings including a respective first embedding for each respective mathematical token of the plurality of mathematical tokens. In some non-limiting embodiments oraspects, the instructions may further cause the at least one processor to input the plurality of remaining tokens into at least one second machine learning model associated with the remaining token type to generate a plurality of second embeddings including a respective second embedding for each respective remaining token of the plurality of remaining tokens. In some non-limiting embodiments or aspects, the at least one second machine learning model may be different than the at least one first machine learning model. In some non-limiting embodiments or aspects, the instructions may further cause the at least one processor to input the plurality of first embeddings and the plurality of second embeddings into at least one third machine learning model to perform a task.

[0030] In some non-limiting embodiments or aspects, the at least one first machine learning model may be configured to receive the plurality of mathematical tokens of the mathematical token type. The at least one second machine learning model may be configured to receive the plurality of remaining tokens of the remaining token type.

[0031] In some non-limiting embodiments or aspects, the text data may include a plurality of tokens, and the one or more instructions that cause the at least one processor to separate the text data into the plurality of mathematical tokens of the mathematical token type and the plurality of remaining tokens of the remaining token type, may cause the at least one processor to determine whether each token of the plurality of tokens is of the mathematical token type. In some non-limiting embodiments or aspects, the instructions may further cause the at least one processor to separate a first subset of the plurality of tokens into the plurality of mathematical tokens based on determining that each token of the first subset of the plurality of tokens is of the mathematical token type and separate a second subset of the plurality of tokens into the plurality of remaining tokens based on determining that each token of the second subset of the plurality of tokens is not of the mathematical token type.

[0032] In some non-limiting embodiments or aspects, the at least one first machine learning model may include an encoding layer and a sequence layer, and the instructions may further cause the at least one processor to train the encoding layer to provide an initial embedding for each respective mathematical token of the plurality of mathematical tokens, mask at least one mathematical token of the plurality of mathematical tokens to provide at least one masked mathematical token, and train the sequence layer to predict a value of the at least one masked mathematical tokenbased on the initial embedding of each other mathematical token of the plurality of mathematical tokens.

[0033] In some non-limiting embodiments or aspects, the encoding layer may include at least one of a neural network, a deep neural network (DNN), an autoencoder, or any combination thereof.

[0034] In some non-limiting embodiments or aspects, the sequence layer may include at least one of a transformer model, a Long Short-Term Memory (LSTM) model, a recurrent neural network (RNN), or any combination thereof.

[0035] In some non-limiting embodiments or aspects, the sequence layer may include the transformer model, and the sequence layer may generate a plurality of refined embeddings based on the initial embedding for each respective mathematical token of the plurality of mathematical tokens.

[0036] In some non-limiting embodiments or aspects, the sequence layer may include the LSTM model, and the sequence layer may generate a hidden state based on the initial embedding for each respective mathematical token of the plurality of mathematical tokens.

[0037] In some non-limiting embodiments or aspects, the instructions that cause the at least one processor to train the sequence layer may cause the at least one processor to randomly generate an arithmetic statement. In some non-limiting embodiments or aspects, the text data may include the arithmetic statement, and masking may include randomly selecting the at least one mathematical token.

[0038] In some non-limiting embodiments or aspects, the task may include generating a plurality of third embeddings based on the plurality of first embeddings and the plurality of second embeddings.

[0039] In some non-limiting embodiments or aspects, the task may include a NLP task.

[0040] In some non-limiting embodiments or aspects, the task may include one of a fraud detection task, a risk detection task, or transaction data generation task.

[0041] According to non-limiting embodiments or aspects, provided is a system for generating improved embeddings for machine learning based on token type. In some non-limiting embodiments or aspects, the system may include at least one processor. In some non-limiting embodiments or aspects, the at least one processor may be configured to receive text data associated with a string of text. In some non-limiting embodiments or aspects, the at least one processor may be configured to separatethe text data into a plurality of tokens including a first plurality of tokens of a first token type and a second plurality of tokens of a second token type. In some non-limiting embodiments or aspects, the at least one processor may be configured to input the first plurality of tokens into at least one first machine learning model associated with the first token type to generate a plurality of first embeddings including a respective first embedding for each respective token of the first plurality of tokens. In some nonlimiting embodiments or aspects, the at least one processor may be configured to input the second plurality of tokens into at least one second machine learning model associated with the second token type to generate a plurality of second embeddings including a respective second embedding for each respective token of the second plurality of tokens, the at least one second machine learning model different than the at least one first machine learning model. In some non-limiting embodiments or aspects, the at least one processor may be configured to input the plurality of first embeddings and the plurality of second embeddings into at least one third machine learning model to perform a task.

[0042] According to non-limiting embodiments or aspects, provided is a computer- implemented method for generating improved embeddings for machine learning based on token type. In some non-limiting embodiments or aspects, the computer- implemented method may include separating the text data into a plurality of tokens including a first plurality of tokens of a first token type and a second plurality of tokens of a second token type. In some non-limiting embodiments or aspects, the computer- implemented method may include inputting the first plurality of tokens into at least one first machine learning model associated with the first token type to generate a plurality of first embeddings including a respective first embedding for each respective token of the first plurality of tokens. In some non-limiting embodiments or aspects, the computer-implemented method may include inputting the second plurality of tokens into at least one second machine learning model associated with the second token type to generate a plurality of second embeddings including a respective second embedding for each respective token of the second plurality of tokens, the at least one second machine learning model different than the at least one first machine learning model. In some non-limiting embodiments or aspects, the computer-implemented method may include inputting the plurality of first embeddings and the plurality of second embeddings into at least one third machine learning model to perform a task.

[0043] According to non-limiting embodiments or aspects, provided is a computer program product for generating improved embeddings for machine learning based on token type. In some non-limiting embodiments or aspects, the computer program product may include at least one non-transitory computer-readable medium including program instructions. In some non-limiting embodiments or aspects, when executed by at least one processor, the instructions may cause the at least one processor to receive text data associated with a string of text. In some non-limiting embodiments or aspects, the instructions may further cause the at least one processor to separate the text data into a plurality of tokens including a first plurality of tokens of a first token type and a second plurality of tokens of a second token type. In some non-limiting embodiments or aspects, the instructions may further cause the at least one processor to input the first plurality of tokens into at least one first machine learning model associated with the first token type to generate a plurality of first embeddings including a respective first embedding for each respective token of the first plurality of tokens. In some non-limiting embodiments or aspects, the instructions may further cause the at least one processor to input the second plurality of tokens into at least one second machine learning model associated with the second token type to generate a plurality of second embeddings including a respective second embedding for each respective token of the second plurality of tokens, the at least one second machine learning model different than the at least one first machine learning model. In some non-limiting embodiments or aspects, the instructions may further cause the at least one processor to input the plurality of first embeddings and the plurality of second embeddings into at least one third machine learning model to perform a task.

[0044] Further non-limiting embodiments or aspects are set forth in the following numbered clauses:

[0045] Clause 1 : A system comprising: at least one processor configured to: receive text data associated with a string of text; separate the text data into a plurality of mathematical tokens of a mathematical token type and a plurality of remaining tokens of a remaining token type; input the plurality of mathematical tokens into at least one first machine learning model associated with the mathematical token type to generate a plurality of first embeddings comprising a respective first embedding for each respective mathematical token of the plurality of mathematical tokens; input the plurality of remaining tokens into at least one second machine learning model associated with the remaining token type to generate a plurality of second embeddingscomprising a respective second embedding for each respective remaining token of the plurality of remaining tokens, the at least one second machine learning model different than the at least one first machine learning model; and input the plurality of first embeddings and the plurality of second embeddings into at least one third machine learning model to perform a task.

[0046] Clause 2: The system of clause 1 , wherein the at least one first machine learning model is configured to receive the plurality of mathematical tokens of the mathematical token type, and wherein the at least one second machine learning model is configured to receive the plurality of remaining tokens of the remaining token type.

[0047] Clause 3: The system of clause 1 or 2, wherein the text data comprises a plurality of tokens, and wherein, when separating the text data into the plurality of mathematical tokens of the mathematical token type and the plurality of remaining tokens of the remaining token type, the at least one processor is configured to: determine whether each token of the plurality of tokens is of the mathematical token type; separate a first subset of the plurality of tokens into the plurality of mathematical tokens based on determining that each token of the first subset of the plurality of tokens is of the mathematical token type; and separate a second subset of the plurality of tokens into the plurality of remaining tokens based on determining that each token of the second subset of the plurality of tokens is not of the mathematical token type.

[0048] Clause 4: The system of any of clauses 1 -3, wherein the at least one first machine learning model comprises an encoding layer and a sequence layer, and wherein the at least one processor is further configured to: train the encoding layer to provide an initial embedding for each respective mathematical token of the plurality of mathematical tokens; mask at least one mathematical token of the plurality of mathematical tokens to provide at least one masked mathematical token; and train the sequence layer to predict a value of the at least one masked mathematical token based on the initial embedding of each other mathematical token of the plurality of mathematical tokens.

[0049] Clause 5: The system of any of clauses 1 -4, wherein the encoding layer comprises at least one of a neural network, a deep neural network (DNN), an autoencoder, or any combination thereof.

[0050] Clause 6: The system of any of clauses 1 -5, wherein the sequence layer comprises at least one of a transformer model, a Long Short-Term Memory (LSTM) model, a recurrent neural network (RNN), or any combination thereof.

[0051] Clause 7: The system of any of clauses 1 -6, wherein the sequence layer comprises the transformer model, and wherein the sequence layer generates a plurality of refined embeddings based on the initial embedding for each respective mathematical token of the plurality of mathematical tokens.

[0052] Clause 8: The system of any of clauses 1 -7, wherein the sequence layer comprises the LSTM model, and wherein the sequence layer generates a hidden state based on the initial embedding for each respective mathematical token of the plurality of mathematical tokens.

[0053] Clause 9: The system of any of clauses 1 -8, wherein, when training the sequence layer, the at least one processor is configured to: randomly generate an arithmetic statement, wherein the text data comprises the arithmetic statement, and wherein masking comprises randomly selecting the at least one mathematical token.

[0054] Clause 10: The system of any of clauses 1 -9, wherein the task comprises generating a plurality of third embeddings based on the plurality of first embeddings and the plurality of second embeddings.

[0055] Clause 11 : The system of any of clauses 1 -10, wherein the task comprises a natural language processing task.

[0056] Clause 12: The system of any of clauses 1 -11 , wherein the task comprises one of a fraud detection task, a risk detection task, or transaction data generation task.

[0057] Clause 13: A computer-implemented method, comprising: receiving, with at least one processor, text data associated with a string of text; separating, with at last one processor, the text data into a plurality of mathematical tokens of a mathematical token type and a plurality of remaining tokens of a remaining token type; inputting, with at least one processor, the plurality of mathematical tokens into at least one first machine learning model associated with the mathematical token type to generate a plurality of first embeddings comprising a respective first embedding for each respective mathematical token of the plurality of mathematical tokens; inputting, with at least one processor, the plurality of remaining tokens into at least one second machine learning model associated with the remaining token type to generate a plurality of second embeddings comprising a respective second embedding for each respective remaining token of the plurality of remaining tokens, the at least one second machine learning model different than the at least one first machine learning model; and inputting, with at least one processor, the plurality of first embeddings and theplurality of second embeddings into at least one third machine learning model to perform a task.

[0058] Clause 14: The computer-implemented method of clause 13, wherein the at least one first machine learning model is configured to receive the plurality of mathematical tokens of the mathematical token type, and wherein the at least one second machine learning model is configured to receive the plurality of remaining tokens of the remaining token type.

[0059] Clause 15: The computer-implemented method of clause 13 or 14, wherein the text data comprises a plurality of tokens, and wherein separating the text data into the plurality of mathematical tokens of the mathematical token type and the plurality of remaining tokens of the remaining token type comprises: determining whether each token of the plurality of tokens is of the mathematical token type; separating a first subset of the plurality of tokens into the plurality of mathematical tokens based on determining that each token of the first subset of the plurality of tokens is of the mathematical token type; and separating a second subset of the plurality of tokens into the plurality of remaining tokens based on determining that each token of the second subset of the plurality of tokens is not of the mathematical token type.

[0060] Clause 16: The computer-implemented method of any of clauses 13-15, wherein the at least one first machine learning model comprises an encoding layer and a sequence layer, the method further comprising: training the encoding layer to provide an initial embedding for each respective mathematical token of the plurality of mathematical tokens; masking at least one mathematical token of the plurality of mathematical tokens to provide at least one masked mathematical token; and training the sequence layer to predict a value of the at least one masked mathematical token based on the initial embedding of each other mathematical token of the plurality of mathematical tokens.

[0061] Clause 17: The computer-implemented method of any of clauses 13-16, wherein the encoding layer comprises at least one of a neural network, a deep neural network (DNN), an autoencoder, or any combination thereof.

[0062] Clause 18: The computer-implemented method of any of clauses 13-17, wherein the sequence layer comprises at least one of a transformer model, a Long Short-Term Memory (LSTM) model, a recurrent neural network (RNN), or any combination thereof.

[0063] Clause 19: The computer-implemented method of any of clauses 13-18, wherein the sequence layer comprises the transformer model, and wherein the sequence layer generates a plurality of refined embeddings based on the initial embedding for each respective mathematical token of the plurality of mathematical tokens.

[0064] Clause 20: The computer-implemented method of any of clauses 13-19, wherein the sequence layer comprises the LSTM model, and wherein the sequence layer generates a hidden state based on the initial embedding for each respective mathematical token of the plurality of mathematical tokens.

[0065] Clause 21 : The computer-implemented method of any of clauses 13-20, wherein training the sequence layer comprises: randomly generating an arithmetic statement, wherein the text data comprises the arithmetic statement, wherein masking comprises randomly selecting the at least one mathematical token.

[0066] Clause 22: The computer-implemented method of any of clauses 13-21 , wherein the task comprises generating a plurality of third embeddings based on the plurality of first embeddings and the plurality of second embeddings.

[0067] Clause 23: The computer-implemented method of any of clauses 13-22, wherein the task comprises a natural language processing task.

[0068] Clause 24: The computer-implemented method of any of clauses 13-23, wherein the task comprises one of a fraud detection task, a risk detection task, or generating transaction data.

[0069] Clause 25: A computer program product comprising at least one non- transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: receive text data associated with a string of text; separate the text data into a plurality of mathematical tokens of a mathematical token type and a plurality of remaining tokens of a remaining token type; input the plurality of mathematical tokens into at least one first machine learning model associated with the mathematical token type to generate a plurality of first embeddings comprising a respective first embedding for each respective mathematical token of the plurality of mathematical tokens; input the plurality of remaining tokens into at least one second machine learning model associated with the remaining token type to generate a plurality of second embeddings comprising a respective second embedding for each respective remaining token of the plurality of remaining tokens, the at least one second machine learning model differentthan the at least one first machine learning model; and input the plurality of first embeddings and the plurality of second embeddings into at least one third machine learning model to perform a task.

[0070] Clause 26: The computer program product of clause 25, wherein the at least one first machine learning model is configured to receive the plurality of mathematical tokens of the mathematical token type, and wherein the at least one second machine learning model is configured to receive the plurality of remaining tokens of the remaining token type.

[0071] Clause 27: The computer program product of clause 25 or 26, wherein the text data comprises a plurality of tokens, and wherein, the one or more instructions that cause the at least one processor to separate the text data into the plurality of mathematical tokens of the mathematical token type and the plurality of remaining tokens of the remaining token type, cause the at least one processor to: determine whether each token of the plurality of tokens is of the mathematical token type; separate a first subset of the plurality of tokens into the plurality of mathematical tokens based on determining that each token of the first subset of the plurality of tokens is of the mathematical token type; and separate a second subset of the plurality of tokens into the plurality of remaining tokens based on determining that each token of the second subset of the plurality of tokens is not of the mathematical token type.

[0072] Clause 28: The computer program product of any of clauses 25-27, wherein the at least one first machine learning model comprises an encoding layer and a sequence layer, and wherein the instructions cause the at least one processor to: train the encoding layer to provide an initial embedding for each respective mathematical token of the plurality of mathematical tokens; mask at least one mathematical token of the plurality of mathematical tokens to provide at least one masked mathematical token; and train the sequence layer to predict a value of each of the at least one masked mathematical token based on the initial embedding of each other mathematical token of the plurality of mathematical tokens.

[0073] Clause 29: The computer program product of any of clauses 25-28, wherein the encoding layer comprises at least one of a neural network, a deep neural network (DNN), an autoencoder, or any combination thereof.

[0074] Clause 30: The computer program product of any of clauses 25-29, wherein the sequence layer comprises at least one of a transformer model, a Long Short-Term Memory (LSTM) model, a recurrent neural network (RNN), or any combination thereof.

[0075] Clause 31 : The computer program product of any of clauses 25-30, wherein the sequence layer comprises the transformer model, and wherein the sequence layer generates a plurality of refined embeddings based on the initial embedding for each respective mathematical token of the plurality of mathematical tokens.

[0076] Clause 32: The computer program product of any of clauses 25-31 , wherein the sequence layer comprises the LSTM model, and wherein the sequence layer generates a hidden state based on the initial embedding for each respective mathematical token of the plurality of mathematical tokens.

[0077] Clause 33: The computer program product of any of clauses 25-32, wherein, the instructions that cause the at least one processor to train the sequence layer cause the at least one processor to: randomly generate an arithmetic statement, wherein the text data comprises the arithmetic statement, and wherein masking comprises randomly selecting the at least one mathematical token.

[0078] Clause 34: The computer program product of any of clauses 25-33, wherein the task comprises generating a plurality of third embeddings based on the plurality of first embeddings and the plurality of second embeddings.

[0079] Clause 35: The computer program product of any of clauses 25-34, wherein the task comprises a natural language processing task.

[0080] Clause 36: The computer program product of any of clauses 25-35, wherein the task comprises one of a fraud detection task, a risk detection task, or transaction data generation task.

[0081] Clause 37: A system, comprising: at least one processor configured to: receive text data associated with a string of text; separate the text data into a plurality of tokens comprising a first plurality of tokens of a first token type and a second plurality of tokens of a second token type; input the first plurality of tokens into at least one first machine learning model associated with the first token type to generate a plurality of first embeddings comprising a respective first embedding for each respective token of the first plurality of tokens; input the second plurality of tokens into at least one second machine learning model associated with the second token type to generate a plurality of second embeddings comprising a respective second embedding for each respective token of the second plurality of tokens, the at least one second machine learning model different than the at least one first machine learning model; and input the plurality of first embeddings and the plurality of second embeddings into at least one third machine learning model to perform a task.

[0082] Clause 38: A computer-implemented method, comprising: receiving, with at least one processor, text data associated with a string of text; separating, with at last one processor, the text data into a plurality of tokens comprising a first plurality of tokens of a first token type and a second plurality of tokens of a second token type; inputting, with at least one processor, the first plurality of tokens into at least one first machine learning model associated with the first token type to generate a plurality of first embeddings comprising a respective first embedding for each respective token of the first plurality of tokens; inputting, with at least one processor, the second plurality of tokens into at least one second machine learning model associated with the second token type to generate a plurality of second embeddings comprising a respective second embedding for each respective token of the second plurality of tokens, the at least one second machine learning model different than the at least one first machine learning model; and inputting, with at least one processor, the plurality of first embeddings and the plurality of second embeddings into at least one third machine learning model to perform a task.

[0083] Clause 39: A computer program product comprising at least one non- transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: receive text data associated with a string of text; separate the text data into a plurality of tokens comprising a first plurality of tokens of a first token type and a second plurality of tokens of a second token type; input the first plurality of tokens into at least one first machine learning model associated with the first token type to generate a plurality of first embeddings comprising a respective first embedding for each respective token of the first plurality of tokens; input the second plurality of tokens into at least one second machine learning model associated with the second token type to generate a plurality of second embeddings comprising a respective second embedding for each respective token of the second plurality of tokens, the at least one second machine learning model different than the at least one first machine learning model; and input the plurality of first embeddings and the plurality of second embeddings into at least one third machine learning model to perform a task.

[0084] These and other features and characteristics of the present disclosure, as well as the methods of operation and functions of the related elements of structures and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims withreference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the disclosed subject matter.BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Additional advantages and details are explained in greater detail below with reference to the non-limiting, exemplary embodiments that are illustrated in the accompanying schematic figures, in which:

[0086] FIG. 1 is a schematic diagram of a system for generating improved embeddings for machine learning based on token type, according to some non-limiting embodiments or aspects;

[0087] FIG. 2 is a flow diagram of a method for generating improved embeddings for machine learning based on token type, according to some non-limiting embodiments or aspects;

[0088] FIG. 3 is a diagram of an example payment processing network in which systems, methods, and / or computer program products, described herein, may be implemented, according to some non-limiting embodiments or aspects;

[0089] FIG. 4 is a schematic diagram of example components of one or more devices of FIG. 1 and / or FIG. 3, according to some non-limiting embodiments or aspects; and

[0090] FIGS. 5A-5E are diagrams of implementations of a method for generating improved embeddings for machine learning based on token type, according to some non-limiting embodiments or aspects.DETAILED DESCRIPTION

[0091] For purposes of the description hereinafter, the terms “end,” “upper,” “lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” “longitudinal,” and derivatives thereof shall relate to the embodiments as they are oriented in the drawing figures. However, it is to be understood that the present disclosure may assume various alternative variations and step sequences, except where expressly specified to the contrary. It is also to be understood that the specific devices and processes illustrated in the attached drawings, and described in the following specification, are simply exemplary and non-limiting embodiments or aspects of the disclosed subjectmatter. Hence, specific dimensions and other physical characteristics related to the embodiments or aspects disclosed herein are not to be considered as limiting.

[0092] Some non-limiting embodiments or aspects are described herein in connection with thresholds. As used herein, satisfying a threshold may refer to a value being greater than the threshold, more than the threshold, higher than the threshold, greater than or equal to the threshold, less than the threshold, fewer than the threshold, lower than the threshold, less than or equal to the threshold, equal to the threshold, etc.

[0093] No aspect, component, element, structure, act, step, function, instruction, and / or the like used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more” and “at least one.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, and / or the like) and may be used interchangeably with “one or more” or “at least one.” Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based at least partially on” unless explicitly stated otherwise. In addition, reference to an action being “based on” a condition may refer to the action being “in response to” the condition. For example, the phrases “based on” and “in response to” may, in some non-limiting embodiments or aspects, refer to a condition for automatically triggering an action (e.g., a specific operation of an electronic device, such as a computing device, a processor, and / or the like).

[0094] As used herein, the term “acquirer institution” may refer to an entity licensed and / or approved by a transaction service provider to originate transactions (e.g., payment transactions) using a payment device associated with the transaction service provider. The transactions the acquirer institution may originate may include payment transactions (e.g., purchases, original credit transactions (OCTs), account funding transactions (AFTs), and / or the like). In some non-limiting embodiments or aspects, an acquirer institution may be a financial institution, such as a bank. As used herein, the term “acquirer system” may refer to one or more computing devices operated by or on behalf of an acquirer institution, such as a server computer executing one or more software applications.

[0095] As used herein, the term “account identifier” may include one or more primary account numbers (PANs), tokens, or other identifiers associated with a customer account. The term “token” may refer to an identifier that is used as a substitute or replacement identifier for an original account identifier, such as a PAN. Account identifiers may be alphanumeric or any combination of characters and / or symbols. Tokens may be associated with a PAN or other original account identifier in one or more data structures (e.g., one or more databases, and / or the like) such that they may be used to conduct a transaction without directly using the original account identifier. In some examples, an original account identifier, such as a PAN, may be associated with a plurality of tokens for different individuals or purposes.

[0096] As used herein, the terms “client” and “client device” may refer to one or more client-side devices or systems (e.g., remote from a transaction service provider) used to initiate or facilitate a transaction (e.g., a payment transaction). As an example, a “client device” may refer to one or more POS devices used by a merchant, one or more acquirer host computers used by an acquirer, one or more mobile devices used by a user, and / or the like. In some non-limiting embodiments or aspects, a client device may be an electronic device configured to communicate with one or more networks and initiate or facilitate transactions. For example, a client device may include one or more computers, portable computers, laptop computers, tablet computers, mobile devices, cellular phones, wearable devices (e.g., watches, glasses, lenses, clothing, and / or the like), PDAs, and / or the like. Moreover, a “client” may also refer to an entity (e.g., a merchant, an acquirer, and / or the like) that owns, utilizes, and / or operates a client device for initiating transactions (e.g., for initiating transactions with a transaction service provider).

[0097] As used herein, the term “communication” may refer to the reception, receipt, transmission, transfer, provision, and / or the like of data (e.g., information, signals, messages, instructions, commands, and / or the like). For one unit (e.g., a device, a system, a component of a device or system, combinations thereof, and / or the like) to be in communication with another unit means that the one unit is able to directly or indirectly receive information from and / or transmit information to the other unit. This may refer to a direct or indirect connection (e.g., a direct communication connection, an indirect communication connection, and / or the like) that is wired and / or wireless in nature. Additionally, two units may be in communication with each other even though the information transmitted may be modified, processed, relayed, and / orrouted between the first and second unit. For example, a first unit may be in communication with a second unit even though the first unit passively receives information and does not actively transmit information to the second unit. As another example, a first unit may be in communication with a second unit if at least one intermediary unit processes information received from the first unit and communicates the processed information to the second unit. In some non-limiting embodiments or aspects, a message may refer to a network packet (e.g., a data packet and / or the like) that includes data. It will be appreciated that numerous other arrangements are possible.

[0098] As used herein, the term “computing device” may refer to one or more electronic devices configured to process data. A computing device may, in some examples, include the necessary components to receive, process, and output data, such as a processor, a display, a memory, an input device, a network interface, and / or the like. A computing device may be a mobile device. As an example, a mobile device may include a cellular phone (e.g., a smartphone or standard cellular phone), a portable computer, a wearable device (e.g., watches, glasses, lenses, clothing, and / or the like), a personal digital assistant (PDA), and / or other like devices. A computing device may also be a desktop computer or other form of non-mobile computer.

[0099] As used herein, the term “issuer institution” may refer to one or more entities, such as a bank, that provide accounts to customers for conducting transactions (e.g., payment transactions), such as initiating credit and / or debit payments. For example, an issuer institution may provide an account identifier, such as a PAN, to a customer that uniquely identifies one or more accounts associated with that customer. The account identifier may be embodied on a portable financial device, such as a physical financial instrument, e.g., a payment card, and / or may be electronic and used for electronic payments. The term “issuer system” refers to one or more computer devices operated by or on behalf of an issuer institution, such as a server computer executing one or more software applications. For example, an issuer system may include one or more authorization servers for authorizing a transaction.

[0100] As used herein, the term “merchant” may refer to an individual or entity that provides goods and / or services, or access to goods and / or services, to customers based on a transaction, such as a payment transaction. The term “merchant” or “merchant system” may also refer to one or more computer systems operated by oron behalf of a merchant, such as a server computer executing one or more software applications.

[0101] As used herein, a “point-of-sale (POS) device” may refer to one or more devices, which may be used by a merchant to conduct a transaction (e.g., a payment transaction) and / or process a transaction. For example, a POS device may include one or more client devices. Additionally or alternatively, a POS device may include peripheral devices, card readers, scanning devices (e.g., code scanners), Bluetooth® communication receivers, near-field communication (NFC) receivers, radio frequency identification (RFID) receivers, and / or other contactless transceivers or receivers, contact-based receivers, payment terminals, and / or the like. As used herein, a “point- of-sale (POS) system” may refer to one or more client devices and / or peripheral devices used by a merchant to conduct a transaction. For example, a POS system may include one or more POS devices and / or other like devices that may be used to conduct a payment transaction. In some non-limiting embodiments or aspects, a POS system (e.g., a merchant POS system) may include one or more server computers programmed or configured to process online payment transactions through webpages, mobile applications, and / or the like.

[0102] As used herein, the term “payment device” may refer to an electronic payment device, a portable financial device, a payment card (e.g., a credit or debit card), a gift card, a smartcard, smart media, a payroll card, a healthcare card, a wristband, a machine-readable medium containing account information, a keychain device or fob, an RFID transponder, a retailer discount or loyalty card, a cellular phone, an electronic wallet mobile application, a personal digital assistant (PDA), a pager, a security card, a computing device, an access card, a wireless terminal, a transponder, and / or the like. In some non-limiting embodiments or aspects, the payment device may include volatile or non-volatile memory to store information (e.g., an account identifier, a name of the account holder, and / or the like).

[0103] As used herein, the term “payment gateway” may refer to an entity and / or a payment processing system operated by or on behalf of such an entity (e.g., a merchant service provider, a payment service provider, a payment facilitator, a payment facilitator that contracts with an acquirer, a payment aggregator, and / or the like), which provides payment services (e.g., transaction service provider payment services, payment processing services, and / or the like) to one or more merchants. The payment services may be associated with the use of portable financial devicesmanaged by a transaction service provider. As used herein, the term “payment gateway system” may refer to one or more computer systems, computer devices, servers, groups of servers, and / or the like, operated by or on behalf of a payment gateway.

[0104] As used herein, the term “server” may refer to or include one or more computing devices that are operated by or facilitate communication and processing for multiple parties in a network environment, such as the Internet, although it will be appreciated that communication may be facilitated over one or more public or private network environments and that various other arrangements are possible. Further, multiple computing devices (e.g., servers, point-of-sale (POS) devices, mobile devices, etc.) directly or indirectly communicating in the network environment may constitute a “system.”

[0105] As used herein, the term “system” may refer to one or more computing devices or combinations of computing devices (e.g., processors, servers, client devices, software applications, components of such, and / or the like). Reference to “a device,” “a server,” “a processor,” and / or the like, as used herein, may refer to a previously-recited device, server, or processor that is recited as performing a previous step or function, a different device, server, or processor, and / or a combination of devices, servers, and / or processors. For example, as used in the specification and the claims, a first device, a first server, or a first processor that is recited as performing a first step or a first function may refer to the same or different device, server, or processor recited as performing a second step or a second function.

[0106] As used herein, the term “transaction service provider” may refer to an entity that receives transaction authorization requests from merchants or other entities and provides guarantees of payment, in some cases through an agreement between the transaction service provider and an issuer institution. For example, a transaction service provider may include a payment network such as Visa® or any other entity that processes transactions. The term “transaction processing system” may refer to one or more computer systems operated by or on behalf of a transaction service provider, such as a transaction processing server executing one or more software applications. A transaction processing server may include one or more processors and, in some non-limiting embodiments or aspects, may be operated by or on behalf of a transaction service provider.

[0107] Non-limiting embodiments or aspects of the disclosed subject matter are directed to systems, methods, and computer program products for generating improved embeddings for machine learning based on token type. For example, nonlimiting embodiments or aspects of the disclosed subject matter provide receiving text data associated with a string of text. In some non-limiting embodiments or aspects, the text data may be separated into a plurality of mathematical tokens of a mathematical token type and a plurality of remaining tokens of a remaining token type (e.g., non-mathematical tokens of a non-mathematical token type). In some nonlimiting embodiments or aspects, the plurality of mathematical tokens may be input into at least one first machine learning model associated with the mathematical token type to generate a plurality of first embeddings. The plurality of first embeddings may include a respective first embedding for each respective mathematical token of the plurality of mathematical tokens. In some non-limiting embodiments or aspects, the plurality of remaining tokens may be input into at least one second machine learning model associated with the remaining token type to generate a plurality of second embeddings. The plurality of second embeddings may include a respective second embedding for each respective remaining token of the plurality of remaining tokens. The at least one second machine learning model may be different than the at least one first machine learning model. As such, the subject matter of the disclosure provides a system for generating improved embeddings for machine learning by generating a first set of embeddings for mathematical tokens of a mathematical token type using at least one first machine learning model and at least one second machine learning model, separate from the at least one first machine learning model, for the remaining tokens of the remaining token type to improve the accuracy of the embeddings and preserve the meaning (e.g., mathematical relationships, mathematical concepts, numerical meanings, and / or the like) of the mathematical tokens. Additionally, the subject matter of the disclosure provides a unique and unconventional model architecture, including separate models (e.g., the at least one first machine learning model associated with the mathematical tokens of the mathematical token type and the at least one second machine learning model associated with the remaining tokens of the remaining token type) which may be used to generate separate embeddings depending on the type of tokens (e.g., mathematical tokens for the mathematical token type or remaining tokens of the remaining token type) included in the received text data.

[0108] In some non-limiting embodiments or aspects, the plurality of first embeddings and the plurality of second embeddings may be input into at least one third machine learning model (e.g., a downstream model, a natural language processing (NLP) model, a large language model (LLM) a fraud detection model, a risk detection model, a generative Al model, a transaction data generation model, or other machine learning models) to perform a task (e.g., generating embeddings, an NLP task, an LLM task, fraud detection, risk detection, a generative Al task, transaction data generation, or other machine learning model tasks). As such, the improved embeddings may be input into at least one downstream model to improve the performance of at least one downstream task performed by the at least one downstream model.

[0109] Referring now to FIG. 1 , shown is an example system 100 for generating improved embeddings for machine learning based on token type, according to some non-limiting embodiments or aspects. As shown in FIG. 1 , system 100 may include machine learning model system 102, database 104, user device 106, and / or communication network 112.

[0110] Machine learning model system 102 may include one or more devices capable of receiving information from and / or communicating information to database 104 and / or user device 106 (e.g., directly via wired or wireless communication connection, indirectly via communication network 112, and / or the like). For example, machine learning model system 102 may include a computing device, such as a server, a group of servers, a desktop computer, a portable computer, a mobile device, and / or other like devices. In some non-limiting embodiments or aspects, machine learning model system 102 may be in communication with a data storage device (e.g., database 104), which may be local or remote to machine learning model system 102. In some non-limiting embodiments or aspects, machine learning model system 102 may be capable of receiving information from, storing information in, communicating information to, or searching information stored in the data storage device (e.g., database 104).

[0111] Database 104 may include one or more devices capable of receiving information from and / or communicating information to machine learning model system 102 and / or user device 106 (e.g., directly via wired or wireless communication connection, indirectly via communication network 112, and / or the like). For example, database 104 may include a computing device, such as a server, a group of servers,a desktop computer, a portable computer, a mobile device, and / or other like devices. In some non-limiting embodiments or aspects, database 104 may include a data storage device. In some non-limiting embodiments or aspects, database 104 may be capable of receiving information from, storing information in, communicating information to, or searching information stored in the data storage device. In some non-limiting embodiments or aspects, database 104 may be part of machine learning model system 102 and / or part of the same system as machine learning model system 102.

[0112] User device 106 may include one or more devices capable of receiving information from and / or communicating information to machine learning model system 102 and / or database 104 (e.g., directly via wired or wireless communication connection, indirectly via communication network 112, and / or the like). For example, user device 106 may include a computing device, such as a mobile device, a portable computer, a desktop computer, and / or other like devices. Additionally or alternatively, each user device 106 may include a device capable of receiving information from and / or communicating information to other user devices 106 (e.g., directly via wired or wireless communication connection, indirectly via communication network 112, and / or the like). In some non-limiting embodiments or aspects, user device 106 may be part of machine learning model system 102 and / or part of the same system as machine learning model system 102. For example, machine learning model system 102, database 104, and user device 106 may all be (and / or be part of) a single system and / or a single computing device.

[0113] Communication network 112 may include one or more wired and / or wireless networks. For example, communication network 112 may include a cellular network (e.g., a long-term evolution (LTE®) network, a third generation (5G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, and / or the like), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., the public switched telephone network (PSTN)), a private network (e.g., a private network associated with a transaction service provider), an ad hoc network, an intranet, the Internet, a fiber optic-based network, a cloud computing network, and / or the like, and / or a combination of these or other types of networks.

[0114] In some non-limiting embodiments or aspects, database 104 and / or user device 106 may store text data associated with a string of text. In some non-limitingembodiments or aspects, user device 106 may generate the text data and / or store the text data in a memory. In some non-limiting embodiments or aspects, user device 106 may send the text data to database 104.

[0115] In some non-limiting embodiments or aspects, machine learning model system 102 may receive the text data associated with the string of text. For example, machine learning model system 102 may receive the text data associated with the string of text from database 104 and / or user device 106 via communication network 112.

[0116] In some non-limiting embodiments or aspects, machine learning model system 102 may separate the text data. For example, in response to receiving the text data associated with the string of text, machine learning model system 102 may separate the text data into a plurality of mathematical tokens of a mathematical token type and a plurality of remaining tokens of a remaining token type.

[0117] In some non-limiting embodiments or aspects, machine learning model system 102 may include one or more machine learning models. For example, machine learning model system 102 may include at least one first machine learning model, at least one second machine learning model, and / or at least one third machine learning model.

[0118] In some non-limiting embodiments or aspects, machine learning model system 102 may generate (e.g., train, validate, re-train, and / or implement) the one or more machine learning models. For example, machine learning model system 102 may generate (e.g., train, validate, re-train, and / or implement) the at least one first machine learning model, the at least one second machine learning model, and / or the at least one third machine learning model.

[0119] The at least one first machine learning model may be associated with the mathematical token type. For example, machine learning model system 102 may generate (e.g., train, validate, re-train, and / or implement) the at least one first machine learning model based on the plurality of mathematical tokens of the mathematical token type. The at least one second machine learning model may be associated with the remaining token type. For example, machine learning model system 102 may generate (e.g., train, validate, re-train, and / or implement) the at least one second machine learning model based on the plurality of remaining tokens of the remaining token type.

[0120] In some non-limiting embodiments or aspects, the at least one first machine learning model, the at least one second machine learning model, and the at least one third machine learning model may be separate models. In some non-limiting embodiments or aspects, the at least one first machine learning model, the at least one second machine learning model, and / or the at least one third machine learning model may include one or more models (e.g., sub-models).

[0121] In some non-limiting embodiments or aspects, machine learning model system 102 may input the plurality of mathematical tokens into the at least one first machine learning model associated with the mathematical token type. For example, machine learning model system 102 may input the plurality of mathematical tokens into the at least one first machine learning model associated with the mathematical token type to generate a plurality of first embeddings including a respective first embedding for each respective mathematical token of the plurality of mathematical tokens.

[0122] In some non-limiting embodiments or aspects, machine learning model system 102 may input the plurality of remaining tokens into the at least one second machine learning model associated with the remaining token type. For example, machine learning model system 102 may input the plurality of remaining tokens into at least one second machine learning model associated with the remaining token type to generate a plurality of second embeddings including a respective second embedding for each respective remaining token of the plurality of remaining tokens. In some non-limiting embodiments or aspects, the at least one second machine learning model may be different than the at least one first machine learning model.

[0123] In some non-limiting embodiments or aspects, machine learning model system 102 may input the plurality of first embeddings and / or the plurality of second embeddings into at least one third machine learning model. For example, machine learning model system 102 may input the plurality of first embeddings and the plurality of second embeddings into at least one third machine learning model (e.g., a downstream model, a NLP model, a LLM a fraud detection model, a risk detection model, a generative Al model, a transaction data generation model, or other machine learning models) to perform a task (e.g., generating embeddings, an NLP task, an LLM task, fraud detection, risk detection, a generative Al task, transaction data generation, or other machine learning model tasks).

[0124] In some non-limiting embodiments or aspects, machine learning model system 102 may send the plurality of first embeddings and / or the plurality of second embeddings to database 104. For example, in response to generating the plurality of first embeddings and / or the plurality of second embeddings, machine learning model system 102 may send the plurality of first embeddings and / or the plurality of second embeddings to database 104 via communication network 112.

[0125] In some non-limiting embodiments or aspects, database 104 may store the plurality of first embeddings and / or the plurality of second embeddings. For example, in response to receiving the plurality of first embeddings and / or the plurality of second embeddings from machine learning model system 102, database 104 may store the plurality of first embeddings and / or the plurality of second embeddings.

[0126] The number and arrangement of systems and devices shown in FIG. 1 are provided as an example. There may be additional systems and / or devices, fewer systems and / or devices, different systems and / or devices, and / or differently arranged systems and / or devices than those shown in FIG. 1. Furthermore, two or more systems or devices shown in FIG. 1 may be implemented within a single system or device, or a single system or device shown in FIG. 1 may be implemented as multiple, distributed systems or devices. Additionally or alternatively, a set of systems (e.g., one or more systems) or a set of devices (e.g., one or more devices) of system 100 may perform one or more functions described as being performed by another set of systems or another set of devices of system 100.

[0127] Referring now to FIG. 2, shown is an example method 200 for generating improved embeddings for machine learning based on token type, according to some non-limiting embodiments or aspects. The steps shown in FIG. 2 are for example purposes only. It will be appreciated that additional, fewer, different, and / or a different order of steps may be used in non-limiting embodiments or aspects. In some nonlimiting embodiments or aspects, a step may be automatically performed in response to performance and / or completion of a prior step. In some non-limiting embodiments or aspects, one or more of the steps of method 200 may be performed (e.g., completely, partially, and / or the like) by machine learning model system 102 (e.g., one or more devices of machine learning model system 102). In some non-limiting embodiments or aspects, one or more of the steps of method 200 may be performed (e.g., completely, partially, and / or the like) by another system, another device, anothergroup of systems, or another group of devices, separate from or including machine learning model system 102, such as database 104 and / or user device 106.

[0128] As shown in FIG. 2, at step 202, method 200 may include receiving text data. For example, machine learning model system 102 may receive text data from database 104 and / or user device 106.

[0129] In some non-limiting embodiments or aspects, the text data may be associated with a string of text. For example, the text data may be associated with a string of text such as a sentence, an equation, a mathematical expression, and / or any combination thereof. The string of text may include words, numbers, mathematical symbols, mathematical operators, and / or any combination thereof.

[0130] The text data may include mathematical data and / or remaining data (e.g., other data, non-mathematical data, and / or the like). For example, the mathematical data may include mathematical expressions, equations, mathematical symbols, numbers expressed as words or digits, mathematical operators expressed as words or symbols, and / or any combination thereof. The remaining data may include the text data other than the mathematical data (e.g., words, punctuation, and / or the like).

[0131] In some non-limiting embodiments or aspects, the text data may be associated with a transaction. For example, the text data may include transaction data, such as a transaction time, a transaction date, a transaction amount, a transaction location, a merchant name, a merchant category code, etc.

[0132] In some non-limiting embodiments or aspects, the text data may include a plurality of tokens. For example, the text data may include a plurality of mathematical tokens and / or a plurality of remaining tokens. Additionally or alternatively, machine learning model system 102 may tokenize the string of text by splitting (e.g., automatically) the string of text into a plurality of tokens (e.g., mathematical tokens and / or remaining tokens) in response to receiving the text data.

[0133] As shown in FIG. 2, at step 204, method 200 may include separating the text data. For example, machine learning model system 102 may separate the text data into a first plurality of tokens of a first token type and a second plurality of tokens of a second token type. The first plurality of tokens of the first token type may include the plurality of mathematical tokens of a mathematical token type. The second plurality of tokens of the second token type may include the plurality of remaining tokens of the remaining (e.g., non-mathematical) token type.

[0134] The plurality of mathematical tokens of the mathematical token type may include numbers (e.g., digits or words describing a numerical value) and / or mathematical notations (e.g., mathematical operators, words describing mathematical operations, symbols, and / or any combination thereof). The plurality of remaining tokens of the remaining token type may include the plurality of tokens other than the plurality of mathematical tokens. In some non-limiting embodiments or aspects, machine learning model system 102 may automatically separate the text data in response to receiving the text data.

[0135] In some non-limiting embodiments or aspects, when separating the text data, machine learning model system 102 may determine whether or not each token of the plurality of tokens is a mathematical token of a mathematical token type or not. In some non-limiting embodiments or aspects, in response to determining that a first token of the plurality of tokens is a mathematical token of a mathematical token type, machine learning model system 102 may separate the first token into a first subset of the plurality of tokens. In response to determining that a second token of the plurality of tokens is not a mathematical token of a mathematical token type, machine learning model system 102 may separate the second token into a second subset of the plurality of tokens.

[0136] In some non-limiting embodiments or aspects, machine learning model system 102 may separate the first subset of the plurality of tokens into the plurality of mathematical tokens based on determining that each token of the first subset of the plurality of tokens is of a mathematical token type. Machine learning model system 102 may separate the second subset of the plurality of tokens into the plurality of remaining tokens based on determining that each token of the second subset of the plurality of tokens is not of a mathematical token type.

[0137] As shown in FIG. 2, at step 206, method 200 may include inputting the first plurality of tokens into a first machine learning model. For example, machine learning model system 102 may input the plurality of mathematical tokens into at least one first machine learning model. The at least one first machine learning model may be associated with the mathematical token type. The at least one first machine learning model may be configured to receive the plurality of tokens of the mathematical token type.

[0138] In some non-limiting embodiments or aspects, machine learning model system 102 may generate (e.g., train, validate, re-train, and / or implement) the at leastone first machine learning model based on the plurality of mathematical tokens of the mathematical token type. For example, machine learning model system 102 may train the at least one first machine learning model to perform a task in response to receiving the plurality of mathematical tokens of the mathematical token type. Machine learning model system 102 may train the at least one first machine learning model to generate and / or output a plurality of first embeddings. In some non-limiting embodiments or aspects, the at least one first machine learning model may receive the plurality of mathematical tokens as an input. The at least one first machine learning model may generate a plurality of first embeddings including a respective first embedding for each respective mathematical token of the plurality of mathematical tokens in response to receiving the plurality of mathematical tokens as inputs. The at least one first machine learning model may provide at least one first embedding of the plurality of first embeddings as an output in response to generating the plurality of first embeddings. The output of the at least one first machine learning model may be stored (e.g., in database 104) and / or received as an input by at least one other machine learning model (e.g., the at least one third machine learning model).

[0139] In some non-limiting embodiments or aspects, the at least one first machine learning model may include one or more layers (e.g., an input layer, one or more hidden layers, and / or an output layer).

[0140] In some non-limiting embodiments or aspects, the one or more layers may include an encoding layer. In some non-limiting embodiments or aspects, the encoding layer may include an encoder and / or a decoder (e.g., encoder machine learning model(s) and / or decoder machine learning model(s)). The encoding layer may include at least one of a neural network, a deep neural network (DNN), an autoencoder, and / or any combination thereof.

[0141] In some non-limiting embodiments or aspects, machine learning model system 102 may train the encoding layer to perform a task. For example, machine learning model system 102 may train the encoding layer to perform a reconstruction task in response to receiving a mathematical token of the plurality of mathematical tokens as an input. In some non-limiting embodiments or aspects, the encoding layer may provide an initial embedding for each respective mathematical token of the plurality of mathematical tokens in response to receiving the plurality of mathematical tokens as inputs.

[0142] In some non-limiting embodiments or aspects, training the encoding layer may include inputting each respective mathematical token of the plurality of mathematical tokens to the encoder machine learning model(s) (e.g., encoder network(s) of the autoencoder(s)) to generate an initial embedding and / or inputting the respective initial embedding for each respective mathematical token of the plurality of mathematical tokens to the decoder machine learning model(s) (e.g., decoder network(s) of the autoencoder(s)) to generate a respective reconstructed mathematical token (e.g., forward propagation). A loss may be determined based on each respective mathematical token of the plurality of mathematical tokens and the respective reconstructed mathematical token. For example, the losses may be calculated based on a difference between each respective mathematical token and the respective reconstructed mathematical token, a loss function, an error, a mean error, a mean squared error (MSE), any combination thereof, and / or the like. The parameters of the autoencoder(s) may be updated based on the loss. For example, machine learning model system 102 may update (e.g., adjust) the parameters of the autoencoder(s) based on back propagation (e.g., of the loss(es)), gradient calculations (e.g., based on the loss(es)), any combination thereof, and / or the like.

[0143] In some non-limiting embodiments or aspects, training may further include repeating the training process (e.g., repeating inputting each respective mathematical token of the plurality of mathematical tokens to the encoder machine learning model(s), inputting the respective initial embedding for each respective mathematical token of the plurality of mathematical tokens to the decoder machine learning model(s), determining the loss, and updating the parameters) until a termination condition is satisfied. For example, the termination condition may include a threshold level of accuracy, and the repetition may continue until the threshold level of accuracy is satisfied. Additionally or alternatively, the termination condition may include a target number of repetitions (e.g., a target number of epochs and / or the like), and the repetition may continue until the target number of repetitions is completed. In some non-limiting embodiments or aspects, there may be a plurality of termination conditions, and the repetition may continue until at least one of the plurality of termination conditions is satisfied.

[0144] In some non-limiting embodiments or aspects, machine learning model system 102 may mask at least one mathematical token of the plurality of mathematical tokens. For example, machine learning model system 102 may mask (e.g., remove,replace, modify, hide) the at least one mathematical token of the plurality of mathematical tokens to provide at least one masked mathematical token. In some non-limiting embodiments or aspects, masking the at least one mathematical token may include removing, replacing, modifying, or hiding a value of the at least one mathematical token and / or a symbol associated with the at least one mathematical token.

[0145] In some non-limiting embodiments or aspects, the plurality of layers may include a sequence layer including a machine learning model(s). In some non-limiting embodiments or aspects, the sequence layer may receive, as input, the output from the encoding layer. For example, the sequence layer may receive, as an input, an initial embedding for each respective mathematical token of the plurality of mathematical tokens output from the encoding layer. In some non-limiting embodiments or aspects, the sequence layer may receive the at least one masked mathematical token as an input.

[0146] In some non-limiting embodiments or aspects, machine learning model system 102 may train the sequence layer to perform a task. For example, machine learning model system 102 may train the sequence layer to predict a value of the at least one masked mathematical token based on the initial embedding of each other mathematical token of the plurality of mathematical tokens.

[0147] In some non-limiting embodiments or aspects, training the sequence layer may include inputting the initial embedding for each respective mathematical token of the plurality of mathematical tokens output from the encoding layer to the sequence layer to generate a prediction. For example, the sequence layer may generate a prediction of a value of each of the at least one masked mathematical tokens based on the initial embedding of each other mathematical token of the plurality of mathematical tokens. A loss may be determined (e.g., calculated) based on a loss function, an error, a mean error, a mean squared error (MSE), any combination thereof, and / or the like. The parameters of the sequence layer may be updated based on the loss. For example, machine learning model system 102 may update (e.g., adjust) the parameters of the sequence layer based on back propagation (e.g., of the loss(es)), gradient calculations (e.g., based on the loss(es)), any combination thereof, and / or the like.

[0148] In some non-limiting embodiments or aspects, training may further include repeating the training process (e.g., repeating inputting the initial embedding for eachrespective mathematical token of the plurality of mathematical tokens to the sequence layer to generate a prediction, determining the loss, and updating the parameters) until a termination condition is satisfied. For example, the termination condition may include a threshold level of accuracy, and the repetition may continue until the threshold level of accuracy is satisfied. Additionally or alternatively, the termination condition may include a target number of repetitions (e.g., a target number of epochs and / or the like), and the repetition may continue until the target number of repetitions is completed. In some non-limiting embodiments or aspects, there may be a plurality of termination conditions, and the repetition may continue until at least one of the plurality of termination conditions is satisfied.

[0149] In some non-limiting embodiments or aspects, when training the sequence layer, machine learning model system 102 may randomly generate text data. For example, machine learning model 102 may randomly generate text data including an arithmetic statement including at least one mathematical token. In some non-limiting embodiments or aspects, machine learning model 102 may mask the at least one mathematical token before inputting the arithmetic statement into the sequence layer.

[0150] The sequence layer may include at least one of a transformer model, a Long Short-Term Memory (LSTM) model, a recurrent neural network (RNN), and / or any combination thereof.

[0151] In some non-limiting embodiments or aspects, the transformer model, may generate a plurality of refined embeddings based on the initial embeddings for each respective mathematical token of the plurality of mathematical tokens. For example, the transformer model may generate the plurality of refined embeddings for each respective mathematical token of the plurality of mathematical tokens in response to receiving the initial embeddings for each respective mathematical token of the plurality of mathematical tokens from the encoding layer.

[0152] In some non-limiting embodiments or aspects, the LSTM model, may generate a hidden state based on the initial embedding for each respective mathematical token of the plurality of mathematical tokens. For example, the LSTM model may generate the hidden state for each respective mathematical token of the plurality of mathematical tokens in response to receiving the initial embeddings for each respective mathematical token of the plurality of mathematical tokens.

[0153] In some non-limiting embodiments or aspects, machine learning model system 102 may receive the plurality of first embeddings output from the at least onefirst machine learning model in response to inputting the plurality of mathematical tokens into the at least one first machine learning model.

[0154] Additionally or alternatively, machine learning model system 102 may receive the plurality of first embeddings from database 104.

[0155] As shown in FIG. 2, at step 208, method 200 may include inputting the second plurality of tokens into a second machine learning model. For example, machine learning model system 102 may input the plurality of remaining tokens into at least one second machine learning model. The at least one second machine learning model may be associated with the remaining token type. The at least one second machine learning model may be configured to receive the plurality of tokens of the remaining token type.

[0156] In some non-limiting embodiments or aspects, machine learning model system 102 may generate (e.g., train, validate, re-train, and / or implement) the at least one second machine learning model. The at least one second machine learning model may include one or more layers (e.g., an input layer, one or more hidden layers, and / or an output layer). In some non-limiting embodiments or aspects, machine learning model system 102 may train the at least one second machine learning model to perform a task. For example, machine learning model system 102 may train the at least one second machine learning model to generate a plurality of second embeddings. In some non-limiting embodiments or aspects, the at least one second machine learning model may receive the plurality of remaining tokens as an input. The at least one second machine learning model may generate a plurality of second embeddings including a respective second embedding for each respective remaining token of the plurality of remaining tokens in response to receiving the plurality of remaining tokens as inputs. The at least one second machine learning model may provide at least one second embedding of the plurality of second embeddings as an output in response to generating the plurality of second embeddings. The output of the at least one second machine learning model may be stored (e.g., in database 104) and / or received as an input by at least one other machine learning model (e.g., the at least one third machine learning model).

[0157] In some non-limiting embodiments or aspects, the at least one second machine learning model may be an NLP model trained to perform a NLP task.

[0158] In some non-limiting embodiments or aspects, machine learning model system 102 may receive the plurality of second embeddings output from the at leastone second machine learning model in response to inputting the plurality of remaining tokens into the at least one second machine learning model.

[0159] As shown in FIG. 2, at step 210, method 200 may include inputting the plurality of first embeddings and / or the plurality of second embeddings into the third machine learning model. For example, machine learning model system 102 may input the plurality of first embeddings into the third machine learning model in response to receiving the plurality of first embeddings. Additionally or alternatively, machine learning model system 102 may input the plurality of second embeddings into at least one third machine learning model in response to receiving the plurality of second embeddings.

[0160] In some non-limiting embodiments or aspects, machine learning model system 102 may generate (e.g., train, validate, re-train, and / or implement) the at least one third machine learning model. The at least one third machine learning model may include one or more layers (e.g., an input layer, one or more hidden layers, and / or an output layer).

[0161] In some non-limiting embodiments or aspects, machine learning model system 102 may train the at least one third machine learning model to perform a task. For example, machine learning model system 102 may train the at least one third machine learning model to perform a task in response to receiving the plurality of first embeddings and / or the plurality of second embeddings as inputs.

[0162] In some non-limiting embodiments or aspects, the at least one third machine learning model may be a NLP model, a LLM model, a fraud detection model, a risk detection model, a generative Al model, a transaction data generation model, or another machine learning model. The at least one third machine learning model may be trained to perform a task, such as generating embeddings, NLP tasks, LLM tasks, fraud detection, risk detection, generative Al tasks, transaction data generation, or other machine learning model tasks.

[0163] Referring now to FIG. 3, depicted is a diagram of an example payment processing network 300, according to non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, payment processing network 300 may be used in conjunction with the systems, methods, and / or computer program products described herein, and / or the systems, methods, and / or computer program products described herein may be implemented in payment processing network 300. As shown in FIG. 3, payment processing network 300 may include transaction processingsystem 301 , payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, and / or consumer device 310. In some non-limiting embodiments or aspects, each of machine learning model system 102, data storage system 104, and / or user device 106 of FIG. 1 may be implemented by (e.g., part of) transaction processing system 301 . In some non-limiting embodiments or aspects, at least one of machine learning model system 102, data storage system 104, and / or user device 106 of FIG. 1 may be implemented by (e.g., part of) another system, another device, another group of systems, or another group of devices, separate from or including transaction processing system 301 , such as merchant system 304, issuer system 306, acquirer system 308, consumer device 310, and / or the like. For example, user device 106 may be implemented by (e.g., part of) at least one of payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, and / or consumer device 310. Additionally or alternatively, for example, data storage system 104 may be implemented by (e.g., part of) at least one of payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, and / or consumer device 310.

[0164] Transaction processing system 301 may include one or more devices capable of receiving information from and / or communicating information to payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, consumer device 310, and / or the like (e.g., directly, indirectly, via a public and / or private communication network connection, and / or the like). For example, as shown in FIG. 3, transaction processing system 301 may be in communication with one or more issuer systems (e.g., issuer system 306), one or more acquirer systems (e.g., acquirer system 308), and / or one or more payment gateway systems (e.g., payment gateway system 302). Although only a single issuer system 306, single acquirer system 308, and single payment gateway system 302 are shown, it will be appreciated that transaction processing system 301 may be in communication with a plurality of issuer systems, a plurality of acquirer systems, and / or a plurality of payment gateways. In some non-limiting embodiments or aspects, transaction processing system 301 may include a computing device, such as a server (e.g., a transaction processing server), a group of servers, and / or other like devices. In some non-limiting embodiments or aspects, transaction processing system 301 may be in communication with a data storage device, which may be local or remote to transaction processing system 301 . In some non-limiting embodiments or aspects, transaction processing system 301 maybe capable of receiving information from, storing information in, communicating information to, or searching information stored in the data storage device. In some non-limiting embodiments or aspects, transaction processing system 301 may be associated with a transaction service provider, as described herein. In some nonlimiting embodiments or aspects, transaction processing system 301 may also operate as an issuer system such that both transaction processing system 301 and issuer system 306 are a single system and / or controlled by a single entity.

[0165] Payment gateway system 302 may include one or more devices capable of receiving information from and / or communicating information to transaction processing system 301 , merchant system 304, issuer system 306, acquirer system 308, consumer device 310, and / or the like (e.g., directly, indirectly, via a public and / or private communication network connection, and / or the like). For example, as shown in FIG. 3, payment gateway system 302 may be in communication with one or more merchant systems (e.g., merchant system 304), one or more acquirer systems (e.g., acquirer system 308), and / or one or more transaction processing systems (e.g., transaction processing system 301 ). Although only a single merchant system 304, single acquirer system 308, and single transaction processing system 301 are shown, it will be appreciated that payment gateway system 302 may be in communication with a plurality of merchant systems, a plurality of acquirer systems, and / or a plurality of transaction processing systems. In some non-limiting embodiments or aspects, payment gateway system 302 may include a computing device, such as a server, a group of servers, and / or other like devices. In some non-limiting embodiments or aspects, payment gateway system 302 may be associated with a payment gateway, as described herein.

[0166] Merchant system 304 may include one or more devices capable of receiving information from and / or communicating information to transaction processing system 301 , payment gateway system 302, issuer system 306, acquirer system 308, consumer device 310, and / or the like (e.g., directly, indirectly, via a public and / or private communication network connection, and / or the like). For example, as shown in FIG. 3, merchant system 304 may be in communication with one or more payment gateway systems (e.g., payment gateway system 302), one or more acquirer systems (e.g., acquirer system 308), and / or one or more consumer devices (e.g., consumer device 310). Although only a single payment gateway system 302, single acquirer system 308, and single consumer device 310 are shown, it will be appreciated thatmerchant system 304 may be in communication with a plurality of payment gateway systems, a plurality of acquirer systems, and / or a plurality of consumer devices. In some non-limiting embodiments or aspects, merchant system 304 may include a computing device, such as a server, a group of servers, a client device, a group of client devices, a POS device, a POS system, computers, computer systems, peripheral devices, and / or other like devices. In some non-limiting embodiments or aspects, merchant system 304 may be associated with a merchant, as described herein. In some non-limiting embodiments or aspects, merchant system 304 may include a device capable of receiving information from and / or communicating information to consumer device 310 via a short range communication connection (e.g., an NFC communication connection, an RFID communication connection, a Bluetooth® communication connection, a Zigbee® communication connection, and / or the like) with consumer device 310 and / or the like. In some non-limiting embodiments or aspects, merchant system 304 may include one or more client devices. For example, merchant system 304 may include a client device that allows a merchant to communicate information to transaction processing system 301 (e.g., via at least one of acquirer system 308 and / or payment gateway system 302). In some non-limiting embodiments or aspects, merchant system 304 (e.g., a client device thereof, a POS device thereof, and / or the like) may also operate as a payment gateway system such that both merchant system 304 and payment gateway system 302 are a single system and / or controlled by a single entity.

[0167] Issuer system 306 may include one or more devices capable of receiving information and / or communicating information to transaction processing system 301 , payment gateway system 302, merchant system 304, acquirer system 308, consumer device 310, and / or the like (e.g., directly, indirectly, via a public and / or private communication network connection, and / or the like). For example, as shown in FIG. 3, issuer system 306 may be in communication with one or more transaction processing systems (e.g., transaction processing system 301 ) and / or one or more consumer devices (e.g., consumer device 310). Although only a single transaction processing system 301 and a single consumer device 310 are shown, it will be appreciated that issuer system 306 may be in communication with a plurality of transaction processing systems and / or a plurality of consumer devices 310. In some non-limiting embodiments or aspects, issuer system 306 may include a computing device, such as a server, a group of servers, and / or other like devices. In some non-limiting embodiments or aspects, issuer system 306 may be associated with an issuer institution, as described herein. For example, issuer system 306 may be associated with an issuer institution that issued a credit account, debit account, credit card, debit card, a payment device, and / or the like to a user associated with consumer device 310.

[0168] Acquirer system 308 may include one or more devices capable of receiving information from and / or communicating information to transaction processing system 301 , payment gateway system 302, merchant system 304, issuer system 306, consumer device 310, and / or the like (e.g., directly, indirectly, via a public and / or private communication network connection, and / or the like). For example, as shown in FIG. 3, acquirer system 308 may be in communication with one or more transaction processing systems (e.g., transaction processing system 301 ), one or more payment gateway systems (e.g., payment gateway system 302), and / or one or more merchant systems (e.g., merchant system 304). Although only a single transaction processing system 301 , a single payment gateway system 302, and a single merchant system 304 are shown, it will be appreciated that acquirer system 308 may be in communication with a plurality of transaction processing systems, a plurality of payment gateway systems, and / or a plurality of merchant systems. In some nonlimiting embodiments or aspects, acquirer system 308 may include a computing device, such as a server, a group of servers, and / or other like devices. In some nonlimiting embodiments or aspects, acquirer system 308 may be associated with an acquirer institution, as described herein.

[0169] Consumer device 310 may include one or more devices capable of receiving information from and / or communicating information to transaction processing system 301 , payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, and / or the like (e.g., directly, indirectly, via a public and / or private communication network connection, and / or the like). For example, as shown in FIG. 3, consumer device 310 may be in communication with one or more merchant systems (e.g., merchant system 304) and / or one or more issuer systems (e.g., issuer system 306). Although only a single merchant system 304 and a single issuer system 306 are shown, it will be appreciated that consumer device 310 may be in communication with a plurality of merchant systems and / or a plurality of issuer systems. In some nonlimiting embodiments or aspects, consumer device 310 may be associated with a user to whom a credit account, a debit account, a credit card, a debit card, a paymentdevice, and / or the like has been issued. In some non-limiting embodiments or aspects, user device 310 may include a computing device, such as a computer, a portable computer, a laptop computer, a tablet computers, a mobile device, a cellular phone, a smartphone, a wearable device (e.g., watches, glasses, lenses, clothing, and / or the like), a PDA, a client device, and / or other like devices. In some non-limiting embodiments or aspects, user device 310 may include a payment device, as described herein. In some non-limiting embodiments or aspects, consumer device 310 may include a device capable of receiving information from and / or communicating information to other customer devices 310 (e.g., directly, indirectly, via a public and / or private communication network connection, a short range communication connection, and / or the like). In some non-limiting embodiments or aspects, consumer device 310 may include a device capable of receiving information from and / or communicating information to merchant system 304 via a short range communication connection (e.g., an NFC communication connection, an RFID communication connection, a Bluetooth® communication connection, a Zigbee® communication connection, and / or the like) with merchant system 304 and / or the like. In some non-limiting embodiments or aspects, consumer device 310 may include a client device.

[0170] In some non-limiting embodiments or aspects, transaction processing system 301 may communicate with merchant system 304 directly (e.g., via a public and / or private communication network connection and / or the like). Additionally or alternatively, transaction processing system 301 may communicate with merchant system 304 through payment gateway 302 and / or acquirer system 308. In some nonlimiting embodiments or aspects, an acquirer system 308 associated with merchant system 304 may operate as payment gateway 302 to facilitate the communication of transaction messages (e.g., authorization requests) from merchant system 304 to transaction processing system 301 . In some non-limiting embodiments or aspects, merchant system 304 may communicate with payment gateway 302 directly (e.g., via a public and / or private communication network connection and / or the like). For example, a merchant system 304 that includes a physical POS device may communicate with payment gateway 302 through a public or private network to conduct card-present transactions. As another example, a merchant system 304 that includes a server (e.g., a web server) may communicate with payment gateway 302 through a public or private network, such as the Internet, to conduct card-not-present transactions.

[0171] For the purpose of illustration, processing a transaction (e.g., a payment transaction) may include generating a transaction message (e.g., authorization request and / or the like) based on an account identifier of a customer (e.g., accountholder associated with customer device 310 and / or the like) and / or transaction data associated with the transaction. For example, merchant system 304 (e.g., a client device of merchant system 304, a POS device of merchant system 304, and / or the like) may initiate the transaction, e.g., by generating an authorization request (e.g., in response to receiving the account identifier from a payment device and / or a portable financial device of the customer and / or the like). Merchant system 304 may communicate the authorization request to payment gateway 302 and / or acquirer system 308. In some non-limiting embodiments or aspects, payment gateway 302 may communicate the authorization request to acquirer system 308 and / or transaction processing system 301. Additionally or alternatively, acquirer system 308 (and / or payment gateway 302) may communicate the authorization request to transaction processing system 301. After receiving the authorization request from merchant system 304 that identifies the account identifier of the customer (e.g., the accountholder associated with consumer device 310 and / or the account identifier), transaction processing system 301 may communicate the authorization request to issuer system 306 (e.g., the issuer system that issued the payment device and / or account identifier). Issuer system 306 may determine an authorization decision (e.g., approve, deny, and / or the like) based on the authorization request, and / or issuer system 306 may generate an authorization response based on the authorization decision and / or the authorization request. Issuer system 306 may communicate the authorization response to transaction processing system 301 . Transaction processing system 301 may communicate the authorization response to acquirer system 308 and / or payment gateway 302. In some non-limiting embodiments or aspects, acquirer system 308 may communicate the authorization response to payment gateway 302 and / or merchant system 304. Additionally or alternatively, payment gateway 302 (and / or acquirer system 308) may communicate the authorization response to merchant system 304.

[0172] In some non-limiting embodiments or aspects, transaction processing system 301 and / or issuer system 306 may include at least one machine learning model (e.g., at least one of a fraud detection model, a risk detection model, a transaction authorization model, a credit approval model, a product recommendationmodel, a classifier model, an anomaly detection model, an authentication model, any combination thereof, and / or the like). For example, the machine learning model(s) may be trained based on text data, a plurality of tokens, and / or a plurality of embeddings, as described herein. Transaction processing system 301 and / or issuer system 306 may perform at least one task (e.g., generate a prediction and / or generate an embedding) based on the authorization request and the machine learning model(s). For example, performing the task(s) may include generating at least one prediction associated with fraud detection, risk detection, transaction authorization, credit approval, product recommendation, classification, anomaly detection, authentication, any combination thereof, and / or the like. In some non-limiting embodiments or aspects, transaction processing system 301 may communicate at least one message based on performing the task (e.g., generating the prediction and / or generate an embedding) to issuer system 306 (e.g., along with the authorization request). In some non-limiting embodiments or aspects, issuer system 306 may determine the authorization decision (e.g., approve, deny, and / or the like) based on the authorization request and the performance of the task (e.g., generation of the prediction and / or generation of the embedding).

[0173] For the purpose of illustration, clearing and / or settlement of a transaction may include generating a message (e.g., clearing message and / or the like) based on an account identifier of a customer (e.g., associated with customer device 310 and / or the like) and / or transaction data associated with the transaction. For example, merchant system 304 may generate at least one clearing message (e.g., a plurality of clearing messages, a batch of clearing messages, and / or the like). Merchant system 304 may communicate the clearing message(s) to acquirer system 308 (and / or payment gateway 302, which may communicate the clearing message(s) to acquirer system 308). Acquirer system 308 may communicate the clearing message(s) to transaction processing system 301. Transaction processing system 301 may communicate the clearing message(s) to issuer system 306. Issuer system 306 may generate at least one settlement message based on the clearing message(s). In some non-limiting embodiments or aspects, issuer system 306 may communicate the settlement message(s) and / or funds to transaction processing system 301 (and / or a settlement bank system associated with transaction processing system 301 ), and transaction processing system 301 (and / or the settlement bank system) may communicate the settlement message(s) and / or funds to acquirer system 308.Additionally or alternatively, issuer system 306 may communicate the settlement message(s) and / or funds to acquirer system 308. In some non-limiting embodiments or aspects, acquirer system 308 may communicate settlement message(s) and / or funds to merchant system 304 (and / or an account associated with merchant system 304).

[0174] The systems and / or devices of FIG. 3 may communicate via one or more wired and / or wireless communication networks. For example, the communication network(s) may include a cellular network (e.g., a long-term evolution (LTE®) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, and / or the like), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., the public switched telephone network (PSTN)), a private network (e.g., a private network associated with a transaction service provider), an ad hoc network, an intranet, the Internet, a fiber optic-based network, a cloud computing network, and / or the like, and / or a combination of these or other types of networks.

[0175] The number and arrangement of systems, devices, and / or networks shown in FIG. 3 are provided as an example. There may be additional systems, devices, and / or networks; fewer systems, devices, and / or networks; different systems, devices, and / or networks; and / or differently arranged systems, devices, and / or networks than those shown in FIG. 3. Furthermore, two or more systems or devices shown in FIG. 3 may be implemented within a single system or device, or a single system or device shown in FIG. 3 may be implemented as multiple, distributed systems or devices. Additionally or alternatively, a set of systems (e.g., one or more systems) or a set of devices (e.g., one or more devices) of payment processing network 300 may perform one or more functions described as being performed by another set of systems or another set of devices of payment processing network 300.

[0176] Referring now to FIG. 4, shown is a diagram of example components of a device 400 according to non-limiting embodiments. Device 400 may correspond to machine learning model system 102, database 104, and / or user device 106 of FIG. 1 and / or transaction service provider system 302, issuer system 304, customer device 306, merchant system 308, acquirer system 310, and communication network 312 of FIG. 3, as an example. In some non-limiting embodiments, such systems or devices may include at least one device 400 and / or at least one component of device 400.The number and arrangement of components shown are provided as an example. In some non-limiting embodiments, device 400 may include additional components, fewer components, different components, or differently arranged components than those shown. Additionally or alternatively, a set of components (e.g., one or more components) of device 400 may perform one or more functions described as being performed by another set of components of device 400.

[0177] As shown in FIG. 4, device 400 may include a bus 402, a processor 404, memory 406, a storage component 408, an input component 410, an output component 412, and a communication interface 414. Bus 402 may include a component that permits communication among the components of device 400. In some non-limiting embodiments, processor 404 may be implemented in hardware, firmware, or a combination of hardware and software. For example, processor 404 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component (e.g., a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.) that can be programmed to perform a function. Memory 406 may include random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by processor 404.

[0178] With continued reference to FIG. 4, storage component 408 may store information and / or software related to the operation and use of device 400. For example, storage component 408 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, a solid-state disk, etc.) and / or another type of computer-readable medium. Input component 410 may include a component that permits device 400 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). Additionally or alternatively, input component 410 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.). Output component 412 may include a component that provides output information from device 400 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.). Communication interface 414 may include a transceiver-like component (e.g., a transceiver, a separate receiver and transmitter, etc.) that enables device 400 to communicate with other devices, such as via a wiredconnection, a wireless connection, or a combination of wired and wireless connections. Communication interface 414 may permit device 400 to receive information from another device and / or provide information to another device. For example, communication interface 414 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a cellular network interface, and / or the like.

[0179] Device 400 may perform one or more processes described herein. Device 400 may perform these processes based on processor 404 executing software instructions stored by a computer-readable medium, such as memory 406 and / or storage component 408. A computer-readable medium may include any non- transitory memory device. A memory device includes memory space located inside of a single physical storage device or memory space spread across multiple physical storage devices. Software instructions may be read into memory 406 and / or storage component 408 from another computer-readable medium or from another device via communication interface 414. When executed, software instructions stored in memory 406 and / or storage component 408 may cause processor 404 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, embodiments described herein are not limited to any specific combination of hardware circuitry and software. The term “configured to,” as used herein, may refer to an arrangement of software, device(s), and / or hardware for performing and / or enabling one or more functions (e.g., actions, processes, steps of a process, and / or the like). For example, “a processor configured to” may refer to a processor that executes software instructions (e.g., program code) that cause the processor to perform one or more functions.

[0180] Referring now to FIGS. 5A-5E, shown are example implementations 500a- 500e of a method for generating improved embeddings for machine learning based on token type, according to some non-limiting embodiments or aspects.

[0181] As shown in FIG. 5A, machine learning model system 502 may receive text data 504. For example, machine learning model system 502 may receive text data 504 as an input (e.g., from database 104 and / or user device 106). Text data 504 may be associated with a string of text which may include words, numbers, symbols, and / or any combination thereof. For example, the string of text may include a sentence (e.g.,Airline A reached a deal of 1 .7 Billion dollars with Airline B) or an arithmetic statement (e.g., 1 + 1 = 2).

[0182] In some non-limiting embodiments or aspects, machine learning model system 502 may include one or more separate machine learning models. For example, machine learning model system 502 may include at least one first machine learning model, at least one second machine learning model, and / or at least one third machine learning model, as described herein. In some non-limiting embodiments or aspects, the one or more machine learning models may include at least one LLM model trained to perform an LLM task. In some non-limiting embodiments or aspects, the one or more separate machine learning models may include one or more submodels. For example, the at least one first machine learning model may include at least one sub-model (e.g., at least two sub-models, such as an encoding layer and a sequence layer, as described herein).

[0183] In some non-limiting embodiments or aspects, machine learning model system 502 may generate output 522. For example, in response to receiving text data 504, machine learning model 502 may generate output 522. Output 522 may include one or more embeddings.

[0184] In some non-limiting embodiments or aspects, downstream model 516 may receive output 522. For example, downstream model 516 may receive output 522 as an input from machine learning model system 502. In some non-limiting embodiments or aspects, downstream model 516 may receive one or more embeddings as an input from machine learning model system 502.

[0185] In some non-limiting embodiments or aspects, downstream model 516 may be trained to perform a task in response to receiving output 522. For example, downstream model 516 may be trained to perform a task in response to receiving the one or more embeddings. In some non-limiting embodiments or aspects, downstream model 516 may include a NLP model, a LLM model, a fraud detection model, a risk detection model, a generative Al model, a transaction data generation model, or any other machine learning models. Downstream model 516 may be trained to perform a task such as, generating embeddings, an NLP task, an LLM task, fraud detection, risk detection, a generative Al task, transaction data generation, or any other machine learning model tasks.

[0186] As shown in FIG. 5B, in response to receiving text data 504, machine learning model system 502 may separate text data 504. For example, machinelearning model system 502 may separate text data 504 (e.g., I spent 1 .7 thousands yesterday) into a plurality of mathematical tokens of a mathematical token type 506 (e.g., 1.7, thousands) and a plurality of remaining tokens of a remaining token type 508 (e.g., I, spent, yesterday). The remaining plurality of tokens of the remaining token type may include the plurality of tokens less the plurality of mathematical tokens of the mathematical token type.

[0187] In some non-limiting embodiments or aspects, machine learning model 502 may tokenize text data 504. For example, machine learning model system 502 may tokenize (e.g., split; separate into words, sub-words, and / or characters; replace words, sub-words, and / or characters with identifiers; any combination thereof; and / or the like) text data 504 to generate the plurality of tokens before separating each of the plurality of tokens in the first plurality of tokens of the first token type or the second plurality of tokens of the second token type.

[0188] In some non-limiting embodiments or aspects, when separating text data 504, machine learning model system 502 may separate each token of the plurality of tokens into a first subset of the plurality of tokens or a second subset of the plurality of tokens based on determining whether or not each token of the plurality of tokens is a mathematical token of a mathematical token type 506. For example, in response to determining a first token of the plurality of tokens is a mathematical token of a mathematical token type 506, machine learning model system 102 may separate the first token into the plurality of mathematical tokens of the mathematical token type. In response to determining a second token of the plurality of tokens is not a mathematical token type, machine learning model system 102 may separate the second token into the plurality of remaining tokens of the remaining token type 508.

[0189] As shown in FIG. 5B, machine learning model system 502 may include first machine learning model 510, second machine learning model 512, and / or third machine learning model 514. In some non-limiting embodiments or aspects, first machine learning model 502 may be a standalone machine learning model separate from second machine learning model 512 and / or third machine learning model 514. In some non-limiting embodiments or aspects, second machine learning model 512 may be a standalone model separate from third machine learning model 514. In some non-limiting embodiments or aspects, any of first machine learning model 510, second machine learning model 512, and / or third machine learning model 514 may include at least one sub-model trained to perform at least one task, as described herein.

[0190] In some non-limiting embodiments or aspects, machine learning model system 502 may input the plurality of mathematical tokens of the mathematical token type 506 into first machine learning model 510. For example, machine learning model system 502 may input the plurality of mathematical tokens of the mathematical token type 506 (e.g., 1.7, thousands) into first machine learning model 510 to generate a plurality of first embeddings 518.

[0191] First machine learning model 510 may be associated with the mathematical token type. First machine learning model may be configured to receive the plurality of mathematical tokens of the mathematical token type 506. For example, first machine learning model 510 may be a math embedding model configured to receive the plurality of mathematical tokens of the mathematical token type 506.

[0192] In some non-limiting embodiments or aspects, first machine learning model 510 may be trained to perform a first task. For example, first machine learning model 510 may be trained to generate a plurality of first embeddings 518 in response to receiving the plurality of mathematical tokens of the mathematical token type 506. The plurality of first embeddings 518 may include a respective first embedding for each respective mathematical token of the plurality of mathematical tokens of the mathematical token type 506.

[0193] In some non-limiting embodiments or aspects, first machine learning model 510 may provide the plurality of first embeddings 518 to one or more machine learning models. For example, first machine learning model 510 may provide the plurality of first embeddings 518 to third machine learning model 514 as inputs.

[0194] In some non-limiting embodiments or aspects, machine learning model system 502 may input the plurality of remaining tokens of the remaining token type 508 into second machine learning model 512. For example, machine learning model system 502 may input the plurality of remaining tokens of the remaining token type 508 (e.g., I, spent, yesterday) into second machine learning model 512 to generate a second plurality of embeddings 520.

[0195] Second machine learning model 512 may be configured to receive the plurality of remaining tokens of the remaining token type 508. In some non-limiting embodiments or aspects, second machine learning model 512 may be a text input embedding model (e.g., NLP model and / or the like) configured to receive the plurality of remaining tokens of the remaining token type 508.

[0196] In some non-limiting embodiments or aspects, second machine learning model 512 may be trained to perform a second task. For example, second machine learning model 51 may be trained to generate the plurality of second embeddings 520 in response to receiving the plurality of remaining tokens of the remaining token type 508. The plurality of second embeddings 520 may include a respective second embedding for each respective remaining token of the plurality of remaining tokens 508.

[0197] In some non-limiting embodiments or aspects, second machine learning model 512 may provide the plurality of second embeddings 520 to one or more machine learning models. For example, second machine learning model 512 may provide the plurality of second embeddings 520 to third machine learning model 514 as inputs.

[0198] In some non-limiting embodiments or aspects, third machine learning model 514 may receive the first plurality of embeddings 518 and / or the second plurality of embeddings 520 as inputs. In some non-limiting embodiments or aspects, third machine learning model 514 may be trained to perform a third task. For example, third machine learning model 514 may be trained to generate output 522. Output 522 may include a plurality of third embeddings generated based on the plurality of first embeddings 518 and / or the plurality of second embeddings 512.

[0199] In some non-limiting embodiments or aspects, machine learning model system 502 may provide output 522 to downstream model 516. For example, machine learning model 502 may provide the plurality of third embeddings as inputs to downstream model 516.

[0200] As shown in FIG. 5C, first machine learning model 510 may include one or more layers (e.g., sub-models). For example, first machine learning model 510 may include encoding layer 510a and / or sequence layer 510b.

[0201] In some non-limiting embodiments or aspects, encoding layer 510a may receive an input. For example, encoding layer 510a may receive the plurality of mathematical tokens of the mathematical token type 506 as inputs. In some nonlimiting embodiments or aspects, encoding layer 510a may include a DNN encoder (e.g., encoding head, encoder network of an autoencoder, and / or the like).

[0202] In some non-limiting embodiments or aspects, machine learning model system 502 may train encoding layer 510a to perform a task. For example, machine learning model system 502 may train encoding layer 510a to generate and / or providean initial embedding for each respective mathematical token of the plurality of mathematical tokens 506 in response to receiving the plurality of mathematical tokens 506, as described herein. In some non-limiting embodiments or aspects, encoding layer 510a may provide the initial embeddings to sequence layer 510b as inputs.

[0203] In some non-limiting embodiments or aspects, machine learning model 502 may train sequence layer 510b to perform a task. For example, machine learning model system 502 may train sequence layer 510b to predict a value (e.g., value of a masked token) based on the one or more initial embeddings of the plurality of initial embeddings, as described herein.

[0204] In some non-limiting embodiments or aspects, sequence layer 510b may include at least one transformer model, a LSTM model, a RNN model, or any combination thereof. In some non-limiting embodiments or aspects, the transformer model may generate a plurality of refined embeddings based on the initial embedding for each respective mathematical token of the plurality of mathematical tokens 506. In some non-limiting embodiments or aspects, the LSTM model may generate a hidden state based on the initial embeddings for each respective mathematical token of the plurality of mathematical tokens 506.

[0205] In some non-limiting embodiments or aspects, machine learning model system 502 may mask at least one mathematical token of the plurality of mathematical tokens 506 to provide at least one masked mathematical token as an input to sequence layer 510b. In some non-limiting embodiments or aspects, machine learning model system 502 may train sequence layer 510b based on the at least one masked mathematical token.

[0206] In some non-limiting embodiments or aspects, first machine learning model 510 may provide output embeddings 518. For example, encoding layer 510a and / or sequence layer 510b may generate and provide the first plurality of output embeddings 518 based on receiving the plurality of mathematical tokens of the mathematical token type 506, the one or more initial embeddings of the plurality of initial embeddings, and / or the predicted value.

[0207] As shown in FIG. 5D, encoding layer 510a may include encoder 5100a and decoder 5100b. In some non-limiting embodiments or aspects, encoding layer 510a may include at least one of a neural network, a deep neural network (DNN), an encoder network of an autoencoder, or any combination thereof.

[0208] In some non-limiting embodiments or aspects, encoder 5100a may be configured to receive initial input x. Initial input x may include at least one of the plurality of mathematical tokens of the mathematical token type 506. For example, encoder 5100a may be configured to receive at least one mathematical token of the mathematical token type 506.

[0209] In some non-limiting embodiments or aspects, machine learning model system 502 may train encoder 5100a to perform an encoding function based on receiving initial input x. For example, machine learning model system 502 may train encoder 5100a to encode the one or more mathematical tokens to provide an encoded representation of the one or more mathematical tokens, in response to receiving the one or more mathematical tokens.

[0210] In some non-limiting embodiments or aspects, decoder 5100b may be configured to receive the encoded representation of initial input x. For example, decoder 5100b may be configured to receive the encoded representation of the one or more mathematical tokens.

[0211] In some non-limiting embodiments or aspects, machine learning model system 502 may train decoder 5100b to perform a decoding function to provide output y (e.g., a reconstruction of initial input x) based on receiving the encoded representation of initial input x. For example, machine learning model system 502 may train decoder 5100b to decode (e.g., reconstruct) the encoded representation of the one or more mathematical tokens to provide a reconstruction of the one or more mathematical tokens.

[0212] In some non-limiting embodiments or aspects, encoding layer 510a may generate the initial embedding for each respective mathematical token of mathematical tokens 506 based on a loss function. For example, encoding layer 510a may generate the initial embedding (e.g., e( )0) for each respective mathematical token of the plurality of mathematical tokens 506 based on the following loss functions:LoSSoperaf- ion / quantity x log(y)Loss Lossnumber+ Lossoperation / quantity

[0213] In some non-limiting embodiments or aspects, Lossnumbermay be a mean squared error (MSE) loss function. In some non-limiting embodiments or aspects, Lossoperation / quantitymay be a multiclass cross-entropy loss function. In some non-limiting embodiments or aspects, Loss may be the sum of Lossnumberand 0SSOperaionqUani y .

[0214] Referring now to FIG. 5E, machine learning model system 502 may generate (e.g., randomly generate) a plurality of arithmetic statements. For example, machine learning model system 502 may randomly generate arithmetic statement 524. Arithmetic statement 524 may include a plurality of mathematical tokens (e.g., mathematical tokens 508) including a combination of numbers (e.g., 256, 200, 50, 6) and / or operators (e.g., =, +).

[0215] In some non-limiting embodiments or aspects, machine learning model system 502 may mask (e.g., randomly) one or more of the plurality of the mathematical tokens of arithmetic statement 524. For example, machine learning model system 502 may randomly mask a mathematical token to provide masked mathematical token 524a of arithmetic statement 524.

[0216] In some non-limiting embodiments or aspects, encoding layer 510a may receive masked mathematical token 524a (e.g., 50) and the remaining plurality of mathematical tokens of arithmetic statement 524 (e.g., 256, =, 200, +, +, 6) as inputs.

[0217] In some non-limiting embodiments or aspects, encoding layer 510a may generate initial embeddings (e.g., e(256)o, e(=)o, e(2OO)o, e(+)o, e(50), e(+)o, e(6)o ) for each of the masked mathematical token 524a and the remaining plurality of mathematical tokens of arithmetic statement 524 in response to receiving masked mathematical token 524a (e.g., 50) and the remaining plurality of mathematical tokens of arithmetic statement 524 (e.g., 256, =, 200, +, +, 6).

[0218] In some non-limiting embodiments or aspects, encoding layer 510a may generate and provide (e.g., to sequence layer 510b and / or additional layer 526) an initial embedding (e.g., e(256)o, e(=)o, e(2OO)o, e(+)o, e(50), e(+)o, e(6)o ) for each of the masked mathematical token 524a and the remaining plurality of mathematical tokens of arithmetic statement 524.

[0219] In some non-limiting embodiments or aspects, sequence layer 510b may include one or more models (not shown). For example, sequence layer 510b may include one of a transformer model or a long short-term memory (LSTM) model.

[0220] In some non-limiting embodiments or aspects, where sequence layer 510b includes a transformer model, sequence layer 510b may output the second plurality of embeddings. In some non-limiting embodiments or aspects, where sequence layer510b includes a LSTM model, sequence layer 510b may output a single second embedding.

[0221] In some non-limiting embodiments or aspects, additional layer 526 may include one or more layers. For example, the one or more layers may include a feed forward neural network (FFNN). The FFNN may include an activation function (e.g., a softmax activation function and / or the like). In some non-limiting embodiments or aspects, sequence layer 510b may include additional layer 526 or sequence layer 51 Ob may be separate from additional layer 526.

[0222] In some non-limiting embodiments or aspects, sequence layer 510b may receive the initial embedding (e.g., e(256)o, e(=)o, e(2OO)o, e(+)o, e(50), e(+)o, e(6)o ) for each of the masked mathematical token 524a and the remaining plurality of mathematical tokens of arithmetic statement 524. In some non-limiting embodiments or aspects, machine learning model 502 may train sequence layer 510b to predict a value for the masked mathematical token 524a based on the received the initial embeddings (e.g., e(256)o, e(=)o, e(200)o, e(+)o, e(+)o, e(6)o ) for each of the remaining plurality of mathematical tokens of arithmetic statement 524. For example, based on receiving the initial embeddings for the masked mathematical token (e.g., e(50)) and the initial embeddings for the remaining mathematical tokens of arithmetic statement 524 (e.g., e(256)o, e(=)o, e(2OO)o, e(+)o, e(+)o, e(6)o ), sequence layer 524 may generate and provide a predicted value 528 of the masked mathematical token. The predicted value may include and / or be associated with a confidence score. The predicted value may include one or more predicted values associated with one or more mathematical tokens (e.g., 1% confidence the masked mathematical token is 30, 60% confidence the masked mathematical token is 50, 3% confidence the masked mathematical token is 20...). In some non-limiting embodiments or aspects, a predicted value with the highest confidence score may be selected as the predicted value.

[0223] Although embodiments have been described in detail for the purpose of illustration, it is to be understood that such detail is solely for that purpose and that the disclosure is not limited to the disclosed embodiments or aspects, but, on the contrary, is intended to cover modifications and equivalent arrangements that are within the spirit and scope of the appended claims. For example, it is to be understood that the present disclosure contemplates that, to the extent possible, one or more features ofany embodiment or aspect can be combined with one or more features of any other embodiment or aspect.

Claims

CLAIMSWHAT IS CLAIMED IS:1 . A system comprising: at least one processor configured to: receive text data associated with a string of text; separate the text data into a plurality of mathematical tokens of a mathematical token type and a plurality of remaining tokens of a remaining token type; input the plurality of mathematical tokens into at least one first machine learning model associated with the mathematical token type to generate a plurality of first embeddings comprising a respective first embedding for each respective mathematical token of the plurality of mathematical tokens; input the plurality of remaining tokens into at least one second machine learning model associated with the remaining token type to generate a plurality of second embeddings comprising a respective second embedding for each respective remaining token of the plurality of remaining tokens, the at least one second machine learning model different than the at least one first machine learning model; and input the plurality of first embeddings and the plurality of second embeddings into at least one third machine learning model to perform a task.

2. The system of claim 1 , wherein the at least one first machine learning model is configured to receive the plurality of mathematical tokens of the mathematical token type, and wherein the at least one second machine learning model is configured to receive the plurality of remaining tokens of the remaining token type.

3. The system of claim 1 , wherein the text data comprises a plurality of tokens, and wherein, when separating the text data into the plurality of mathematical tokens of the mathematical token type and the plurality of remaining tokens of the remaining token type, the at least one processor is configured to: determine whether each token of the plurality of tokens is of the mathematical token type;separate a first subset of the plurality of tokens into the plurality of mathematical tokens based on determining that each token of the first subset of the plurality of tokens is of the mathematical token type; and separate a second subset of the plurality of tokens into the plurality of remaining tokens based on determining that each token of the second subset of the plurality of tokens is not of the mathematical token type.

4. The system of claim 1 , wherein the at least one first machine learning model comprises an encoding layer and a sequence layer, and wherein the at least one processor is further configured to: train the encoding layer to provide an initial embedding for each respective mathematical token of the plurality of mathematical tokens; mask at least one mathematical token of the plurality of mathematical tokens to provide at least one masked mathematical token; and train the sequence layer to predict a value of the at least one masked mathematical token based on the initial embedding of each other mathematical token of the plurality of mathematical tokens.

5. The system of claim 4, wherein the encoding layer comprises at least one of a neural network, a deep neural network (DNN), an autoencoder, or any combination thereof.

6. The system of claim 4, wherein the sequence layer comprises at least one of a transformer model, a Long Short-Term Memory (LSTM) model, a recurrent neural network (RNN), or any combination thereof.

7. The system of claim 6, wherein the sequence layer comprises the transformer model, and wherein the sequence layer generates a plurality of refined embeddings based on the initial embedding for each respective mathematical token of the plurality of mathematical tokens.

8. The system of claim 6, wherein the sequence layer comprises the LSTM model, and wherein the sequence layer generates a hidden state based on theinitial embedding for each respective mathematical token of the plurality of mathematical tokens.

9. The system of claim 4, wherein, when training the sequence layer, the at least one processor is configured to: randomly generate an arithmetic statement, wherein the text data comprises the arithmetic statement, and wherein masking comprises randomly selecting the at least one mathematical token.

10. The system of claim 1 , wherein the task comprises generating a plurality of third embeddings based on the plurality of first embeddings and the plurality of second embeddings.11 . The system of claim 1 , wherein the task comprises a natural language processing task.

12. The system of claim 1 , wherein the task comprises one of a fraud detection task, a risk detection task, or transaction data generation task.

13. A computer-implemented method, comprising: receiving, with at least one processor, text data associated with a string of text; separating, with at last one processor, the text data into a plurality of mathematical tokens of a mathematical token type and a plurality of remaining tokens of a remaining token type; inputting, with at least one processor, the plurality of mathematical tokens into at least one first machine learning model associated with the mathematical token type to generate a plurality of first embeddings comprising a respective first embedding for each respective mathematical token of the plurality of mathematical tokens; inputting, with at least one processor, the plurality of remaining tokens into at least one second machine learning model associated with the remaining token type to generate a plurality of second embeddings comprising a respective second embedding for each respective remaining token of the plurality of remaining tokens,the at least one second machine learning model different than the at least one first machine learning model; and inputting, with at least one processor, the plurality of first embeddings and the plurality of second embeddings into at least one third machine learning model to perform a task.

14. The computer-implemented method of claim 13, wherein the at least one first machine learning model is configured to receive the plurality of mathematical tokens of the mathematical token type, and wherein the at least one second machine learning model is configured to receive the plurality of remaining tokens of the remaining token type.

15. The computer-implemented method of claim 13, wherein the text data comprises a plurality of tokens, and wherein separating the text data into the plurality of mathematical tokens of the mathematical token type and the plurality of remaining tokens of the remaining token type comprises: determining whether each token of the plurality of tokens is of the mathematical token type; separating a first subset of the plurality of tokens into the plurality of mathematical tokens based on determining that each token of the first subset of the plurality of tokens is of the mathematical token type; and separating a second subset of the plurality of tokens into the plurality of remaining tokens based on determining that each token of the second subset of the plurality of tokens is not of the mathematical token type.

16. The computer-implemented method of claim 13, wherein the at least one first machine learning model comprises an encoding layer and a sequence layer, the method further comprising: training the encoding layer to provide an initial embedding for each respective mathematical token of the plurality of mathematical tokens; masking at least one mathematical token of the plurality of mathematical tokens to provide at least one masked mathematical token; andtraining the sequence layer to predict a value of the at least one masked mathematical token based on the initial embedding of each other mathematical token of the plurality of mathematical tokens.

17. The computer-implemented method of claim 16, wherein training the sequence layer comprises: randomly generating an arithmetic statement, wherein the text data comprises the arithmetic statement, wherein masking comprises randomly selecting the at least one mathematical token.

18. The computer-implemented method of claim 13, wherein the task comprises generating a plurality of third embeddings based on the plurality of first embeddings and the plurality of second embeddings.

19. The computer-implemented method of claim 13, wherein the task comprises one of a natural language processing task, a fraud detection task, a risk detection task, or generating transaction data.

20. A computer program product comprising at least one non- transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: receive text data associated with a string of text; separate the text data into a plurality of mathematical tokens of a mathematical token type and a plurality of remaining tokens of a remaining token type; input the plurality of mathematical tokens into at least one first machine learning model associated with the mathematical token type to generate a plurality of first embeddings comprising a respective first embedding for each respective mathematical token of the plurality of mathematical tokens; input the plurality of remaining tokens into at least one second machine learning model associated with the remaining token type to generate a plurality of second embeddings comprising a respective second embedding for each respective remaining token of the plurality of remaining tokens, the at least one second machine learning model different than the at least one first machine learning model; andinput the plurality of first embeddings and the plurality of second embeddings into at least one third machine learning model to perform a task.21 . A system, comprising: at least one processor configured to: receive text data associated with a string of text; separate the text data into a plurality of tokens comprising a first plurality of tokens of a first token type and a second plurality of tokens of a second token type; input the first plurality of tokens into at least one first machine learning model associated with the first token type to generate a plurality of first embeddings comprising a respective first embedding for each respective token of the first plurality of tokens; input the second plurality of tokens into at least one second machine learning model associated with the second token type to generate a plurality of second embeddings comprising a respective second embedding for each respective token of the second plurality of tokens, the at least one second machine learning model different than the at least one first machine learning model; and input the plurality of first embeddings and the plurality of second embeddings into at least one third machine learning model to perform a task.

22. A computer-implemented method, comprising: receiving, with at least one processor, text data associated with a string of text; separating, with at last one processor, the text data into a plurality of tokens comprising a first plurality of tokens of a first token type and a second plurality of tokens of a second token type; inputting, with at least one processor, the first plurality of tokens into at least one first machine learning model associated with the first token type to generate a plurality of first embeddings comprising a respective first embedding for each respective token of the first plurality of tokens; inputting, with at least one processor, the second plurality of tokens into at least one second machine learning model associated with the second token type togenerate a plurality of second embeddings comprising a respective second embedding for each respective token of the second plurality of tokens, the at least one second machine learning model different than the at least one first machine learning model; and inputting, with at least one processor, the plurality of first embeddings and the plurality of second embeddings into at least one third machine learning model to perform a task.

23. A computer program product comprising at least one non- transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: receive text data associated with a string of text; separate the text data into a plurality of tokens comprising a first plurality of tokens of a first token type and a second plurality of tokens of a second token type; input the first plurality of tokens into at least one first machine learning model associated with the first token type to generate a plurality of first embeddings comprising a respective first embedding for each respective token of the first plurality of tokens; input the second plurality of tokens into at least one second machine learning model associated with the second token type to generate a plurality of second embeddings comprising a respective second embedding for each respective token of the second plurality of tokens, the at least one second machine learning model different than the at least one first machine learning model; and input the plurality of first embeddings and the plurality of second embeddings into at least one third machine learning model to perform a task.

Images