Method, system, and computer program product for bipartite graph pre-trained dual transformers

Abstract

Methods, systems, and computer program products are provided for bipartite graph pre-trained dual transformers. Graph data for a bipartite graph is received, the bipartite graph including a first node / entity, a second node / entity, and an edge. The first entity context of each first entity includes a second entity connected to the first entity through the edge. The second entity context of each second entity / node includes the first entity connected to the second entity through the edge. A first embedding of each first entity is generated. A first encoded representation of each first entity is generated based on the first entity context and a first transformer encoder. A second embedding is generated for each second entity. A second encoded representation of each second entity is generated based on the second entity context and a second transformer encoder. The first embedding and the first transformer encoder are adjusted based on a first contrast loss. The second embedding and the second transformer encoder are adjusted based on a second contrast loss.

Description

Cross Reference to Related Applications

[0001] This application claims priority to U.S. Provisional Patent Application 63 / 469,128, filed May 26, 2023, the disclosure of which is hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0002] The present disclosure generally relates to machine learning for graph data, and in some non-limiting embodiments or aspects, to methods, systems, and computer program products for pre-trained bi-transformer for bipartite graphs. BACKGROUND

[0003] Certain activities of a user (e.g., online activities, offline activities, etc.) can be represented as data indicative of interactions between the user and content (e.g., visiting a place, purchasing a product, reading a news article, viewing multimedia content, rating multimedia content, posting a review about an event, etc.). Insights about the user, the user’s behavior, the content, or the nature of the content can be learned based on these user-content interactions. Certain machine learning models have been used to attempt to perform specific tasks (e.g., generate a particular type of prediction, like predicting content to recommend to a user).

[0004] However, machine learning models trained only to adapt to the goal of a specific task can be affected by data sparsity, which can ultimately limit the performance of the machine learning model. Moreover, attempts to apply self-supervised learning to learn contextual knowledge only learn the contextual knowledge from a single perspective, as the machine learning models used for this type of self-supervised learning are developed for images (e.g., computer vision) or text (e.g., natural language processing), where the context is inherent in the images or text, respectively. SUMMARY

[0005] Accordingly, improved methods, systems, and computer program products for pre-trained bi-transformer for bipartite graphs are provided (e.g., that overcome some or all of the above-identified deficiencies).

[0006] According to non-limiting embodiments or aspects, methods for pre-training bi-transformers for bipartite graphs are provided. An example method can include receiving graph data associated with a bipartite graph, the bipartite graph can include a plurality of first nodes, a plurality of second nodes, and a plurality of edges. Each first node of the plurality of first nodes can be associated with a first entity of a plurality of first entities. Each second node of the plurality of second nodes can be associated with a second entity of a plurality of second entities. Each edge of the plurality of edges can be associated with an interaction between a respective first entity of the plurality of first entities and a respective second entity of the plurality of second entities. A first entity context for each first entity of the plurality of first entities can include all second entities of the plurality of second entities connected to the first entity by at least one edge of the plurality of edges. A second entity context for each second entity of the plurality of second entities can include all first entities of the plurality of first entities connected to the second entity by at least one edge of the plurality of edges. A plurality of first embeddings can be generated, the plurality of first embeddings including a respective first embedding associated with each first entity of the plurality of first entities. A plurality of first encoded representations can be generated based on the first entity context for each first entity of the plurality of first entities and a first transformer encoder, the plurality of first encoded representations including a respective first encoded representation associated with each first entity of the plurality of first entities. A plurality of second embeddings can be generated, the plurality of second embeddings including a respective second embedding associated with each second entity of the plurality of second entities. A plurality of second encoded representations can be generated based on the second entity context for each second entity of the plurality of second entities and a second transformer encoder, the plurality of second encoded representations including a respective second encoded representation associated with each second entity of the plurality of second entities. The plurality of first embeddings and the first transformer encoder can be adjusted based on a first contrastive loss between the plurality of first embeddings and the plurality of first encoded representations. The plurality of second embeddings and the second transformer encoder can be adjusted based on a second contrastive loss between the plurality of second embeddings and the plurality of second encoded representations.

[0007] In some non-limiting embodiments or aspects, the operations of generating the plurality of first encoded representations, generating the plurality of second encoded representations, adjusting the plurality of first embeddings and the first transformer encoder, and adjusting the plurality of second embeddings and the second transformer encoder can be iteratively repeated until at least one termination condition is satisfied.

[0008] In some non-limiting embodiments or aspects, a prediction can be generated based on the first embedding for a selected first entity of the plurality of first entities, the first entity context for the selected first entity, the first transformer encoder, and the second transformer encoder.

[0009] In some non-limiting embodiments or aspects, multiple first embedded encoders, multiple second embedded encoders, and multiple second transformer encoders may be adjusted based on the total loss determined according to the prediction.

[0010] In some non-limiting embodiments or aspects, predictions may be iteratively generated and adjusted for multiple first embeddings, a first transformer encoder, multiple second embeddings, and a second transformer encoder until at least one termination condition is met.

[0011] In some non-limiting embodiments or aspects, prediction may include sequential prediction.

[0012] In some non-limiting embodiments or aspects, the plurality of first entities may include a plurality of users, and the plurality of second entities may include a plurality of content.

[0013] In some non-limiting embodiments or aspects, generating a plurality of first embeddings may include randomly initializing a plurality of first embeddings.

[0014] In some non-limiting embodiments or aspects, generating a plurality of second embeddings may include randomly initializing a plurality of second embeddings.

[0015] According to a non-limiting embodiment or aspect, a system for a pre-trained dual transformer for a bipartite graph is provided. An example system may include at least one processor configured to receive graph data associated with a bipartite graph, which may include a plurality of first nodes, a plurality of second nodes, and a plurality of edges. Each of the plurality of first nodes may be associated with a first entity among a plurality of first entities. Each of the plurality of second nodes may be associated with a second entity among a plurality of second entities. Each of the plurality of edges may be associated with an interaction between a corresponding first entity among the plurality of first entities and a corresponding second entity among the plurality of second entities. The first entity context of each of the plurality of first entities may include all second entities among the plurality of second entities connected to the first entity via at least one of the plurality of edges. The second entity context of each of the plurality of second entities may include all first entities among the plurality of first entities connected to the second entity via at least one of the plurality of edges. A plurality of first embeddings may be generated, the plurality of first embeddings including a corresponding first embedding associated with each of the plurality of first entities. Multiple first-coded representations can be generated based on the first entity context of each of the multiple first entities and a first transformer encoder, the multiple first-coded representations including a corresponding first-coded representation associated with each of the multiple first entities. Multiple second embeddings can be generated, the multiple second embeddings including a corresponding second embedding associated with each of the multiple second entities. Multiple second-coded representations can be generated based on the second entity context of each of the multiple second entities and a second transformer encoder, the multiple second-coded representations including a corresponding second-coded representation associated with each of the multiple second entities. Multiple first embeddings and the first transformer encoder can be adjusted based on a first contrast loss between the multiple first embeddings and the multiple first-coded representations. Multiple second embeddings and the second transformer encoder can be adjusted based on a second contrast loss between the multiple second embeddings and the multiple second-coded representations.

[0016] In some non-limiting embodiments or aspects, at least one processor may be further configured to iteratively repeat the following operations: generating a plurality of first encoded representations, generating a plurality of second encoded representations, adjusting a plurality of first embedded and first transformer encoders, and adjusting a plurality of second embedded and second transformer encoders until at least one termination condition is met.

[0017] In some non-limiting embodiments or aspects, at least one processor may be further configured to generate a prediction based on a first embedding of a selected first entity among a plurality of first entities, a first entity context of the selected first entity, a first transformer encoder, and a second transformer encoder.

[0018] In some non-limiting embodiments or aspects, at least one processor may be further configured to adjust a plurality of first embedded, first transformer encoders, and a plurality of second embedded and second transformer encoders based on the total loss determined according to the prediction.

[0019] In some non-limiting embodiments or aspects, at least one processor may be further configured to iteratively and repeatedly generate predictions and adjust a plurality of first embeddings, a first transformer encoder, a plurality of second embeddings and a second transformer encoder until at least one termination condition is met.

[0020] In some non-limiting embodiments or aspects, prediction may include sequential prediction.

[0021] In some non-limiting embodiments or aspects, the plurality of first entities may include a plurality of users. Alternatively, the plurality of second entities may include a plurality of content.

[0022] In some non-limiting embodiments or aspects, generating a plurality of first embeddings may include randomly initializing a plurality of first embeddings.

[0023] In some non-limiting embodiments or aspects, generating a plurality of second embeddings may include randomly initializing a plurality of second embeddings.

[0024] According to a non-limiting embodiment or aspect, a computer program product for a pre-trained dual transformer for a bipartite graph is provided. An example computer program product may include at least one non-transient computer-readable medium comprising program instructions that, when executed by at least one processor, cause the at least one processor to receive graph data associated with a bipartite graph, which may include a plurality of first nodes, a plurality of second nodes, and a plurality of edges. Each of the plurality of first nodes may be associated with a first entity among a plurality of first entities. Each of the plurality of second nodes may be associated with a second entity among a plurality of second entities. Each of the plurality of edges may be associated with an interaction between a corresponding first entity among the plurality of first entities and a corresponding second entity among the plurality of second entities. A first entity context for each of the plurality of first entities may include all second entities of a plurality of second entities connected to the first entity via at least one of the plurality of edges. A second entity context for each of the plurality of second entities may include all first entities of a plurality of first entities connected to the second entity via at least one of the plurality of edges. A plurality of first embeddings may be generated, the plurality of first embeddings including a corresponding first embedding associated with each of the plurality of first entities. Multiple first-coded representations can be generated based on the first entity context of each of the multiple first entities and a first transformer encoder, the multiple first-coded representations including a corresponding first-coded representation associated with each of the multiple first entities. Multiple second embeddings can be generated, the multiple second embeddings including a corresponding second embedding associated with each of the multiple second entities. Multiple second-coded representations can be generated based on the second entity context of each of the multiple second entities and a second transformer encoder, the multiple second-coded representations including a corresponding second-coded representation associated with each of the multiple second entities. Multiple first embeddings and the first transformer encoder can be adjusted based on a first contrast loss between the multiple first embeddings and the multiple first-coded representations. Multiple second embeddings and the second transformer encoder can be adjusted based on a second contrast loss between the multiple second embeddings and the multiple second-coded representations.

[0025] In some non-limiting embodiments or aspects, the instructions, when executed by at least one processor, may further cause the at least one processor to iteratively repeat the following operations: generating a plurality of first encoded representations, generating a plurality of second encoded representations, adjusting a plurality of first embedding and first transformer encoders, and adjusting a plurality of second embedding and second transformer encoders until at least one termination condition is met.

[0026] In some non-limiting embodiments or aspects, the instructions, when executed by at least one processor, may further cause the at least one processor to generate a prediction based on a first embedding of a selected first entity among a plurality of first entities, a first entity context of the selected first entity, a first transformer encoder, and a second transformer encoder.

[0027] In some non-limiting embodiments or aspects, the instructions, when executed by at least one processor, may further cause the at least one processor to adjust a plurality of first embedded, first transformer encoders, and a plurality of second embedded and second transformer encoders based on a total loss determined according to a prediction.

[0028] In some non-limiting embodiments or aspects, the instructions, when executed by at least one processor, may further cause at least one processor to iteratively and repeatedly generate predictions and adjust a plurality of first embeddings, a first transformer encoder, a plurality of second embeddings, and a second transformer encoder until at least one termination condition is met.

[0029] In some non-limiting embodiments or aspects, prediction may include sequential prediction.

[0030] In some non-limiting embodiments or aspects, the plurality of first entities may include a plurality of users. Alternatively, the plurality of second entities may include a plurality of content.

[0031] In some non-limiting embodiments or aspects, generating a plurality of first embeddings may include randomly initializing a plurality of first embeddings.

[0032] In some non-limiting embodiments or aspects, generating a plurality of second embeddings may include randomly initializing a plurality of second embeddings.

[0033] According to non-limiting embodiments or aspects, a system for pre-trained dual transformers for bipartite graphs is provided. An example system may include at least one processor configured to perform any of the methods described herein.

[0034] According to non-limiting embodiments or aspects, a computer program product for a pre-trained dual transformer for bipartite graphs is provided. An example computer program product may include at least one non-transient computer-readable medium comprising program instructions that, when executed by at least one processor, cause the at least one processor to perform any of the methods described herein.

[0035] Other non-limiting embodiments or aspects are set forth in the following numbered clauses: Clause 1: A computer-implemented method comprising: receiving graph data associated with a bipartite graph using at least one processor, the bipartite graph including a plurality of first nodes, a plurality of second nodes, and a plurality of edges, each of the plurality of first nodes being associated with a first entity among a plurality of first entities, each of the plurality of second nodes being associated with a second entity among a plurality of second entities, each of the plurality of edges being associated with an interaction between a corresponding first entity among the plurality of first entities and a corresponding second entity among the plurality of second entities, wherein a first entity context of each of the plurality of first entities includes all second entities of the plurality of second entities connected to the first entity via at least one edge among the plurality of edges, and wherein a second entity context of each of the plurality of second entities includes all first entities of the plurality of first entities connected to the second entity via at least one edge among the plurality of edges; generating a plurality of first embeddings using at least one processor, the plurality of first embeddings including a corresponding first embedding associated with each of the plurality of first entities. The method involves: generating a plurality of first encoded representations based on the first entity context and a first transformer encoder of each of the plurality of first entities using at least one processor; generating a plurality of second embeddings based on the second entity context and a second transformer encoder of each of the plurality of first entities using at least one processor; generating a plurality of second encoded representations based on the second entity context and a second transformer encoder of each of the plurality of second entities using at least one processor; adjusting the plurality of first embeddings and the first transformer encoder based on a first contrast loss between the plurality of first embeddings and the plurality of first encoded representations using at least one processor; and adjusting the plurality of second embeddings and the second transformer encoder based on a second contrast loss between the plurality of second embeddings and the plurality of second encoded representations using at least one processor.

[0036] Clause 2: The method according to Clause 1 further comprises: iteratively repeating the following operations using at least one processor: generating the plurality of first encoded representations, generating the plurality of second encoded representations, adjusting the plurality of first embeddings and the first transformer encoder, and adjusting the plurality of second embeddings and the second transformer encoder until at least one termination condition is met.

[0037] Clause 3: The method according to Clause 1 or Clause 2 further comprises: generating a prediction using at least one processor based on the first embedding of a selected first entity among the plurality of first entities, the first entity context of the selected first entity, the first transformer encoder, and the second transformer encoder.

[0038] Clause 4: The method according to any one of Clauses 1 to 3 further comprises: using at least one processor to adjust the plurality of first embeddings, the first transformer encoder, the plurality of second embeddings and the second transformer encoder based on the total loss determined according to the prediction.

[0039] Clause 5: The method according to any one of Clauses 1 to 4 further comprises: iteratively generating the prediction and adjusting the plurality of first embeddings, the first transformer encoder, the plurality of second embeddings and the second transformer encoder using at least one processor until at least one termination condition is met.

[0040] Clause 6: The method according to any one of Clauses 1 to 5, wherein the prediction includes sequential prediction.

[0041] Clause 7: The method according to any one of Clauses 1 to 6, wherein the plurality of first entities comprises a plurality of users, and wherein the plurality of second entities comprises a plurality of content.

[0042] Clause 8: The method according to any one of Clauses 1 to 7, wherein generating the plurality of first embeddings includes randomly initializing the plurality of first embeddings.

[0043] Clause 9: The method according to any one of Clauses 1 to 8, wherein generating the plurality of second embeddings includes randomly initializing the plurality of second embeddings.

[0044] Item 10: A system comprising: at least one processor configured to: receive graph data associated with a bipartite graph, the bipartite graph including a plurality of first nodes, a plurality of second nodes, and a plurality of edges, each of the plurality of first nodes being associated with a first entity among a plurality of first entities, each of the plurality of second nodes being associated with a second entity among a plurality of second entities, each of the plurality of edges being associated with an interaction between a corresponding first entity among the plurality of first entities and a corresponding second entity among the plurality of second entities, wherein a first entity context of each of the plurality of first entities includes all second entities of the plurality of second entities connected to the first entity via at least one edge among the plurality of edges, and wherein a second entity context of each of the plurality of second entities includes all first entities of the plurality of first entities connected to the second entity via at least one edge among the plurality of edges; generate a plurality of first embeddings, the plurality of first embeddings including an interaction between the plurality of first entities and the first entity. The method involves: generating a first embedding associated with each first entity in the plurality of first entities; generating a plurality of first encoded representations based on the first entity context and the first transformer encoder of each of the plurality of first entities, the plurality of first encoded representations including a corresponding first encoded representation associated with each of the plurality of first entities; generating a plurality of second embeddings, the plurality of second embeddings including a corresponding second embedding associated with each of the plurality of second entities; generating a plurality of second encoded representations based on the second entity context and the second transformer encoder of each of the plurality of second entities, the plurality of second encoded representations including a corresponding second encoded representation associated with each of the plurality of second entities; adjusting the plurality of first embeddings and the first transformer encoder based on a first contrast loss between the plurality of first embeddings and the plurality of first encoded representations; and adjusting the plurality of second embeddings and the second transformer encoder based on a second contrast loss between the plurality of second embeddings and the plurality of second encoded representations.

[0045] Clause 11: The system according to Clause 10, wherein the at least one processor is further configured to iteratively repeat the following operations: generating the plurality of first encoded representations, generating the plurality of second encoded representations, adjusting the plurality of first embeddings and the first transformer encoder, and adjusting the plurality of second embeddings and the second transformer encoder until at least one termination condition is met.

[0046] Clause 12: The system according to Clause 10 or Clause 11, wherein the at least one processor is further configured to generate a prediction based on the first embedding of a selected first entity among the plurality of first entities, the first entity context of the selected first entity, the first transformer encoder, and the second transformer encoder.

[0047] Clause 13: The system according to any one of Clauses 10 to 12, wherein the at least one processor is further configured to: adjust the plurality of first embeddings, the first transformer encoder, the plurality of second embeddings and the second transformer encoder based on the total loss determined according to the prediction.

[0048] Clause 14: The system according to any one of Clauses 10 to 13, wherein the at least one processor is further configured to iteratively generate the prediction and adjust the plurality of first embeddings, the first transformer encoder, the plurality of second embeddings and the second transformer encoder until at least one termination condition is met.

[0049] Clause 15: A system pursuant to any one of Clauses 10 to 14, wherein the prediction includes sequential prediction.

[0050] Clause 16: The system pursuant to any one of Clauses 10 to 15, wherein the plurality of first entities comprises a plurality of users, and wherein the plurality of second entities comprises a plurality of content.

[0051] Clause 17: A system according to any one of Clauses 10 to 16, wherein generating the plurality of first embeddings includes randomly initializing the plurality of first embeddings.

[0052] Clause 18: A system according to any one of Clauses 10 to 17, wherein generating the plurality of second embeddings includes randomly initializing the plurality of second embeddings.

[0053] Clause 19: A computer program product comprising at least one non-transient computer-readable medium, the at least one non-transient computer-readable medium comprising program instructions that, when executed by at least one processor, cause the at least one processor to: receive graph data associated with a bipartite graph, the bipartite graph comprising a plurality of first nodes, a plurality of second nodes, and a plurality of edges, each of the plurality of first nodes being associated with a first entity among a plurality of first entities, each of the plurality of second nodes being associated with a second entity among a plurality of second entities, each of the plurality of edges being associated with an interaction between a corresponding first entity among the plurality of first entities and a corresponding second entity among the plurality of second entities, wherein a first entity context of each of the plurality of first entities includes all second entities of the plurality of second entities connected to the first entity via at least one of the plurality of edges, and wherein a second entity context of each of the plurality of second entities includes all first entities of the plurality of first entities connected to the second entity via at least one of the plurality of edges. The process involves generating multiple first embeddings, each first embedding including a corresponding first embedding associated with each of the multiple first entities; generating multiple first encoded representations based on the first entity context and the first transformer encoder of each of the multiple first entities, each first encoded representation including a corresponding first encoded representation associated with each of the multiple first entities; generating multiple second embeddings, each second embedding including a corresponding second embedding associated with each of the multiple second entities; generating multiple second encoded representations based on the second entity context and the second transformer encoder of each of the multiple second entities, each second encoded representation including a corresponding second encoded representation associated with each of the multiple second entities; adjusting the multiple first embeddings and the first transformer encoder based on a first contrast loss between the multiple first embeddings and the multiple first encoded representations; and adjusting the multiple second embeddings and the second transformer encoder based on a second contrast loss between the multiple second embeddings and the multiple second encoded representations.

[0054] Clause 20: The computer program product of claim 19, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to: generate a prediction based on the first embedding of a selected first entity among the plurality of first entities, the first entity context of the selected first entity, the first transformer encoder, and the second transformer encoder.

[0055] Clause 21: A system comprising: at least one processor configured to perform the method according to any one of Clauses 1 to 9.

[0056] Clause 22: A computer program product comprising at least one non-transient computer-readable medium, the at least one non-transient computer-readable medium comprising program instructions that, when executed by at least one processor, cause the at least one processor to perform the method according to any one of Clauses 1 to 9.

[0057] These and other features and characteristics of this disclosure, as well as the operational methods and manufacturing economies of combinations of related structural elements and parts, will become more apparent when considered in conjunction with the accompanying drawings, all of which form part of this specification, wherein similar reference numerals denote corresponding parts in the figures. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to be a definition of limitation on the disclosed subject matter. Attached Figure Description

[0058] Additional advantages and details are explained in more detail below with reference to the non-limiting exemplary embodiments shown in the illustrative accompanying drawings, in which: Figure 1 This is a schematic diagram of an example system of a pre-trained dual transformer for bipartite graphs, based on some non-limiting embodiments or aspects. Figure 2 This is a flowchart of an example method for a pre-trained dual transformer for a bipartite graph, based on some non-limiting embodiments or aspects; Figure 3 This is a simplified diagram of an example payment processing network from which the methods, systems and / or computer program products described herein may be implemented, according to some non-limiting embodiments or aspects. Figure 4 Based on some non-limiting embodiments or aspects Figure 1 and / or Figure 3 A schematic diagram of example components of one or more devices; Figure 5A It is a schematic diagram of an example bipartite diagram based on some non-limiting embodiments or aspects; Figure 5B and 5C This is a schematic diagram of an example context of entities for a bipartite graph, based on some non-limiting embodiments or aspects; Figure 6 This is a schematic diagram of an example embodiment of a pre-trained dual transformer for bipartite graphs, based on some non-limiting embodiments or aspects; and Figure 7 This is a schematic diagram of an example embodiment of a pre-trained dual transformer for a bipartite graph, based on some non-limiting embodiments or aspects. Detailed Implementation

[0059] For the purposes of the following description, the terms “end,” “upper,” “lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” “longitudinal,” and their derivatives should be associated with the orientation of the embodiments in the accompanying drawings. However, it should be understood that various alternative variations and sequences of steps may be employed in this disclosure, except where explicitly specified otherwise. It should also be understood that the specific apparatus and processes shown in the drawings and described in the following specification are merely exemplary and non-limiting embodiments or aspects of the disclosed subject matter. Therefore, specific dimensions and other physical characteristics relating to the embodiments or aspects disclosed herein should not be considered limiting.

[0060] This document describes some non-limiting embodiments or aspects in conjunction with thresholds. As used herein, satisfying a threshold can refer to a value greater than a threshold, more than a threshold, higher than a threshold, greater than or equal to a threshold, less than a threshold, less than a threshold, lower than a threshold, less than or equal to a threshold, equal to a threshold, etc.

[0061] The aspects, components, elements, structures, actions, steps, functions, instructions, etc., used herein should not be construed as critical or essential unless explicitly stated otherwise. Furthermore, as used herein, the article “a” is intended to include one or more items and is interchangeable with “one or more” and “at least one.” Additionally, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and is interchangeable with “one or more” or “at least one.” The term “a” or similar language is used where only one item is desired. Moreover, as used herein, the terms “has,” “have,” “having,” etc., are intended to be open-ended terms. Additionally, unless explicitly stated otherwise, the phrase “based on” is intended to mean “at least partially based on.” Furthermore, a reference to an action “based on” a condition may mean that the action is “in response to” the condition. For example, in some non-limiting embodiments or aspects, the phrases “based on” and “in response to” may refer to conditions that automatically trigger an action (e.g., a specific operation of an electronic device, such as a computing device, processor, etc.).

[0062] As used herein, the term "acquiring institution" can refer to an entity licensed and / or approved by a transaction service provider to initiate transactions (e.g., payment transactions) using payment devices associated with the transaction service provider. Transactions that an acquiring institution can initiate can include payment transactions (e.g., purchases, original letter of credit transactions (OCT), account treasury transactions (AFT), etc.). In some non-limiting embodiments or aspects, an acquiring institution can be a financial institution, such as a bank. As used herein, the term "acquiring system" can refer to one or more computing devices operated by or on behalf of an acquiring institution, such as a server computer executing one or more software applications.

[0063] As used herein, the term "account identifier" can include one or more master account (PAN), tokens, or other identifiers associated with a customer account. The term "token" can refer to an identifier used as a substitute or replacement identifier for an original account identifier such as a PAN. An account identifier can be any combination of alphanumeric or characters and / or symbols. A token can be associated with a PAN or other original account identifier in one or more data structures (e.g., one or more databases, etc.) such that the token can be used to conduct transactions without directly using the original account identifier. In some examples, an original account identifier such as a PAN can be associated with multiple tokens for different individuals or purposes.

[0064] As used herein, the terms "client" and "client device" can refer to one or more client-side devices or systems (e.g., at a remote location of the transaction service provider) used to initiate or facilitate a transaction (e.g., a payment transaction). As examples, "client device" can refer to one or more POS devices used by a merchant, one or more acquiring host computers used by an acquiring party, one or more mobile devices used by a user, etc. In some non-limiting embodiments or aspects, a client device can be an electronic device configured to communicate with one or more networks and initiate or facilitate transactions. For example, a client device can include one or more computers, laptops, laptop computers, tablet computers, mobile devices, cellular phones, wearable devices (e.g., watches, glasses, lenses, clothing, etc.), PDAs, etc. Furthermore, "client" can also refer to an entity (e.g., a merchant, acquiring party, etc.) that owns, utilizes, and / or operates a client device for initiating transactions (e.g., for initiating a transaction with a transaction service provider).

[0065] As used herein, the term "communication" can refer to the receiving, accepting, sending, transmitting, providing, etc., of data (e.g., information, signals, messages, instructions, commands, etc.). For one unit (e.g., a device, system, component of a device or system, combination thereof, etc.) to communicate with another unit means that the first unit is able to receive information directly or indirectly from and / or send information to the other unit. This can refer to a direct or indirect connection that is inherently wired and / or wireless (e.g., a direct communication connection, an indirect communication connection, etc.). Furthermore, although the transmitted information may be modified, processed, relayed, and / or routed between the first and second units, the two units can also communicate with each other. For example, the first unit can communicate with the second unit even if it passively receives information and does not actively send information to the second unit. As another example, the first unit can communicate with the second unit if at least one intermediate unit processes information received from the first unit and transmits the processed information to the second unit. In some non-limiting embodiments or aspects, a message can refer to a network packet (e.g., a data packet, etc.) that includes data. It should be understood that many other arrangements are possible.

[0066] As used herein, the term "computing device" can refer to one or more electronic devices configured to process data. In some examples, a computing device may include the necessary components for receiving, processing, and outputting data, such as a processor, display, memory, input device, network interface, etc. A computing device can be a mobile device. As examples, a mobile device may include a cellular phone (e.g., a smartphone or standard cellular phone), a portable computer, a wearable device (e.g., a watch, glasses, lenses, clothing, etc.), a personal digital assistant (PDA), and / or other similar devices. A computing device can also be a desktop computer or other forms of non-mobile computer.

[0067] As used herein, the terms "e-wallet" and "e-wallet application" refer to one or more electronic devices and / or software applications configured to initiate and / or conduct payment transactions. For example, an e-wallet may include a mobile device executing an e-wallet application, and may also include server-side software and / or a database for maintaining and providing transaction data to the mobile device. An "e-wallet provider" may include an entity that provides and / or maintains e-wallets to customers, such as Google Pay®, Android Pay®, Apple Pay®, Samsung Pay®, and / or other similar electronic payment systems. In some non-limiting examples, an issuing bank may be an e-wallet provider.

[0068] As used herein, the term "issuing institution" can refer to one or more entities, such as a bank, that provide customers with accounts for conducting transactions (e.g., payment transactions), such as initiating credit and / or debit payments. For example, an issuing institution may provide customers with an account identifier, such as a PAN, that uniquely identifies one or more accounts associated with said customer. The account identifier may be embodied in a portable financial device, such as a physical financial instrument (e.g., a payment card), and / or may be electronic and used for electronic payments. The term "issuing system" refers to one or more computer devices operated by or on behalf of an issuing institution, such as a server computer executing one or more software applications. For example, an issuing system may include one or more authorization servers for authorizing transactions.

[0069] As used herein, the term "merchant" can refer to an individual or entity that provides goods and / or services or access to goods and / or services to a customer based on a transaction such as a payment transaction. The terms "merchant" or "merchant system" can also refer to one or more computer systems operated by or on behalf of a merchant, such as a server computer that executes one or more software applications.

[0070] As used herein, a “point-of-sale (POS) device” can refer to one or more devices that a merchant can use to conduct transactions (e.g., payment transactions) and / or process transactions. For example, a POS device may include one or more client devices. Alternatively or additionally, a POS device may include peripheral devices, card readers, scanning devices (e.g., barcode scanners), Bluetooth® communication receivers, Near Field Communication (NFC) receivers, Radio Frequency Identification (RFID) receivers and / or other contactless transceivers or receivers, contact-based receivers, payment terminals, etc. As used herein, a “point-of-sale (POS) system” can refer to one or more client devices and / or peripheral devices that a merchant uses to conduct transactions. For example, a POS system may include one or more POS devices, and / or other similar devices that can be used to conduct payment transactions. In some non-limiting embodiments or aspects, a POS system (e.g., a merchant POS system) may include one or more server computers programmed or configured to process online payment transactions via web pages, mobile applications, etc.

[0071] As used herein, the term "payment device" can refer to electronic payment devices, portable financial devices, payment cards (e.g., credit or debit cards), gift cards, smart cards, smart media, payroll cards, healthcare cards, wristbands, machine-readable media containing account information, keychain devices or pendants, RFID transponders, retailer discount or loyalty cards, cellular phones, e-wallet mobile applications, personal digital assistants (PDAs), pagers, security cards, computing devices, access cards, wireless terminals, transponders, etc. In some non-limiting embodiments or aspects, a payment device may include volatile or non-volatile memory for storing information (e.g., account identifiers, account holder names, etc.).

[0072] As used herein, the term "payment gateway" can refer to an entity and / or a payment processing system operated by or on behalf of such an entity (e.g., a merchant service provider, payment service provider, payment servicer, payment aggregator, etc., contracted with an acquirer, payment aggregator, etc.) that provides payment services (e.g., transaction service provider payment services, payment processing services, etc.) to one or more merchants. Payment services may be associated with the use of portable financial devices managed by a transaction service provider. As used herein, the term "payment gateway system" can refer to one or more computer systems, computer devices, servers, server clusters, etc., operated by or on behalf of a payment gateway.

[0073] As used herein, the term "server" may refer to or include one or more computing devices operated by or facilitating communication and processing among multiple parties in a network environment such as the Internet, but it should be understood that communication may be facilitated through one or more public or private network environments, and various other arrangements may be possible. Furthermore, multiple computing devices (e.g., servers, point-of-sale (POS) devices, mobile devices, etc.) communicating directly or indirectly in a network environment may constitute a "system".

[0074] As used herein, the term "system" may refer to one or more computing devices or a combination of computing devices (e.g., processor, server, client device, software application, components of such computing devices, etc.). As used herein, references to "device," "server," "processor," etc., may refer to a previously described device, server, or processor, different devices, servers, or processors, and / or combinations of devices, servers, and / or processors, described as performing a preceding step or function. For example, as used in the specification and claims, a first device, first server, or first processor described as performing a first step or a first function may refer to the same or different devices, servers, or processors described as performing a second step or a second function.

[0075] As used herein, the term "transaction service provider" can refer to an entity that receives transaction authorization requests from merchants or other entities and, in some cases, provides payment guarantees through an agreement between the transaction service provider and the issuing entity. For example, a transaction service provider may include payment networks such as Visa®, or any other entity that processes transactions. The term "transaction processing system" can refer to one or more computer systems operated by or on behalf of the transaction service provider, such as a transaction processing server executing one or more software applications. A transaction processing server may include one or more processors and, in some non-limiting embodiments or aspects, may be operated by or on behalf of the transaction service provider.

[0076] Non-limiting embodiments or aspects of the disclosed subject matter relate to methods, systems, and computer program products for machine learning of graph data, including but not limited to methods, systems, and computer program products for pre-trained dual transformers for bipartite graphs. For example, non-limiting embodiments or aspects of the disclosed subject matter provide receiving graph data associated with a bipartite graph, the bipartite graph including a plurality of first nodes, a plurality of second nodes, and a plurality of edges. Each of the plurality of first nodes may be associated with a first entity among a plurality of first entities (e.g., a user). Each of the plurality of second nodes may be associated with a second entity among a plurality of second entities (e.g., content). Each of the plurality of edges may be associated with an interaction between a corresponding first entity (e.g., a user) and a corresponding second entity (e.g., content). A first entity context (e.g., a user context) for each first entity may include all second entities (e.g., all content) connected to the first entity via at least one edge. A second entity context (e.g., a content context) for each of the plurality of second entities may include all first entities (e.g., a user) connected to the second entity via at least one edge. A plurality of first embeddings may be generated, the plurality of first embeddings including a corresponding first embedding associated with each of the plurality of first entities. Multiple first-encoded representations can be generated based on the first entity context of each of the multiple first entities and a first transformer encoder, the multiple first-encoded representations including a corresponding first-encoded representation associated with each of the multiple first entities. Multiple second embeddings can be generated, the multiple second embeddings including a corresponding second embedding associated with each of the multiple second entities. Multiple second-encoded representations can be generated based on the second entity context of each of the multiple second entities and a second transformer encoder, the multiple second-encoded representations including a corresponding second-encoded representation associated with each of the multiple second entities. Multiple first embeddings and the first transformer encoder can be adjusted (e.g., trained) based on a first contrastive loss between the multiple first embeddings and the multiple first-encoded representations. Multiple second embeddings and the second transformer encoder can be adjusted (e.g., trained) based on a second contrastive loss between the multiple second embeddings and the multiple second-encoded representations. Therefore, the disclosed subject matter enables the learning of contextual knowledge from a dataset (e.g., graph data associated with a bipartite graph) from two different perspectives (e.g., user-side and content-side), which is impossible with a single type of transformer. When input is associated with a bipartite graph (e.g., in a user-content bipartite graph, the context of a user's activity is not limited to the user's personal history, and each content has its own context), this dual perspective can be advantageous, as opposed to other use cases such as image or text input (e.g., where the context is inherent in the image or text).Furthermore, the disclosed subject matter enables pre-training in an unsupervised manner (e.g., based on contrastive loss between embeddings and encoded representations), which can be advantageous when labeled data is scarce or unavailable. In some non-limiting embodiments or aspects, the disclosed subject matter provides a first embedding based on a selected first entity (e.g., a selected user) among a plurality of first entities, a first entity context of the selected first entity, a first transformer encoder, and a second transformer encoder to generate predictions. Additionally, the plurality of first embeddings, the first transformer encoder, the plurality of second embeddings, and the second transformer encoder can be adjusted based on the total loss determined according to the predictions. Therefore, the disclosed subject matter enables fine-tuning for specific downstream tasks (e.g., prediction, such as sequence recommendation, fraud detection, etc.) after pre-training.

[0077] Now for reference Figure 1 An example system 100 for a pre-trained dual transformer for bipartite graphs is shown, according to some non-limiting embodiments or aspects. Figure 1 As shown, system 100 may include machine learning model system 102, data storage system 104, user device 106 and / or communication network 112.

[0078] Machine learning model system 102 may include one or more devices capable of receiving and / or transmitting information to data storage system 104 and / or user device 106, for example, directly via wired or wireless communication connections, indirectly via communication network 112, etc. For example, machine learning model system 102 may include computing devices such as servers, server clusters, desktop computers, portable computers, and / or other similar devices. In some non-limiting embodiments or aspects, machine learning model system 102 may communicate with a data storage device (e.g., data storage system 104), which may be local or remote to machine learning model system 102. In some non-limiting embodiments or aspects, machine learning model system 102 may be able to receive information from data storage device (e.g., data storage system 104), store information in the data storage device, transmit information to the data storage device, or search for information stored in the data storage device.

[0079] Data storage system 104 may include one or more devices capable of receiving and / or transmitting information to machine learning model system 102 and / or user device 106, for example, directly via wired or wireless communication connections, indirectly via communication network 112, etc. For example, data storage system 104 may include computing devices, such as servers, server clusters, and / or other similar devices. In some non-limiting embodiments or aspects, data storage system 104 may include data storage devices. In some non-limiting embodiments or aspects, data storage system 104 may be capable of receiving information from a data storage device, storing information in the data storage device, transmitting information to the data storage device, or searching for information stored in the data storage device. In some non-limiting embodiments or aspects, data storage system 104 may be part of machine learning model system 102 and / or part of a system identical to machine learning model system 102.

[0080] User device 106 may include one or more means capable of receiving and / or transmitting information to machine learning model system 102 and / or data storage system 104, for example, directly via a wired or wireless communication connection, or indirectly via communication network 112. For example, user device 106 may include computing devices, such as mobile devices, portable computers, desktop computers, and / or other similar devices. Additionally or alternatively, each user device 106 may include means capable of receiving and / or transmitting information to other user devices 106, for example, directly via a wired or wireless communication connection, or indirectly via communication network 112. In some non-limiting embodiments or aspects, user device 106 may be part of machine learning model system 102 and / or part of a system identical to machine learning model system 102.

[0081] The communication network 112 may include one or more wired and / or wireless networks. For example, the communication network 112 may include cellular networks (e.g., Long Term Evolution (LTE®) networks, third-generation (3G) networks, fourth-generation (4G) networks, fifth-generation (5G) networks, code division multiple access (CDMA) networks, etc.), public land mobile networks (PLMN), local area networks (LAN), wide area networks (WAN), metropolitan area networks (MAN), telephone networks (e.g., public switched telephone network (PSTN)), private networks (e.g., private networks associated with transaction service providers), ad hoc networks, intranets, the Internet, fiber-optic networks, cloud computing networks, etc., and / or combinations of these or other types of networks.

[0082] Figure 1The number and arrangement of systems and devices shown are provided as examples. Figure 1 Compared to the systems and / or devices shown, there may be additional systems and / or devices, fewer systems and / or devices, different systems and / or devices, and / or systems and / or devices arranged in a different manner. Furthermore, implementation may be within a single system and / or device. Figure 1 The two or more systems or devices shown, or Figure 1 The single system or device shown may be implemented as multiple distributed systems or devices. Alternatively, a group of systems (e.g., one or more systems) or a group of devices (e.g., one or more devices) of system 100 may perform one or more functions described as being performed by another group of systems or another group of devices of system 100.

[0083] Now for reference Figure 2 An example method 200 for pre-trained dual transformers for bipartite graphs is shown, according to some non-limiting embodiments or aspects. Figure 2 The steps shown are for illustrative purposes only. It should be understood that additional, fewer, different, and / or different orders of steps may be used in non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, one or more steps of method 200 may (e.g., entirely, partially, etc.) be performed by machine learning model system 102 (e.g., one or more devices of machine learning model system 102). In some non-limiting embodiments or aspects, one or more steps of method 200 may (e.g., entirely, partially, etc.) be performed by another system, another device, another group of systems, or another group of devices independent of or including machine learning model system 102, such as data storage system 104, user device 106, etc.

[0084] like Figure 2 As shown, in step 202, method 200 may include receiving graph data. For example, machine learning model system 102 may receive graph data (e.g., from data storage system 104).

[0085] In some non-limiting embodiments or aspects, graph data may be associated with a bipartite graph, which includes a plurality of first nodes, a plurality of second nodes, and a plurality of edges. Each of the plurality of first nodes may be associated with a first entity among a plurality of first entities. Each of the plurality of second nodes may be associated with a second entity among a plurality of second entities. Each of the plurality of edges may be associated with an interaction between a corresponding first entity among the plurality of first entities and a corresponding second entity among the plurality of second entities.

[0086] In some non-limiting embodiments or aspects, the first entity context of each of the plurality of first entities may include all second entities of a plurality of second entities connected to the first entity via at least one of a plurality of edges. Alternatively, the second entity context of each of the plurality of second entities may include all first entities of a plurality of first entities connected to the second entity via at least one of a plurality of edges.

[0087] like Figure 2 As shown, in step 204, method 200 may include generating a first embedding. For example, machine learning model system 102 may generate a plurality of first embeddings, the plurality of first embeddings including a corresponding first embedding associated with each of a plurality of first entities.

[0088] In some non-limiting embodiments or aspects, generating a plurality of first embeddings may include randomly initializing a plurality of first embeddings.

[0089] like Figure 2 As shown, in step 206, method 200 may include generating a first coded representation based on a first transformer encoder. For example, machine learning model system 102 may generate multiple first coded representations based on a first entity context of each of a plurality of first entities and a first transformer encoder, the multiple first coded representations including a corresponding first coded representation associated with each of the plurality of first entities.

[0090] like Figure 2 As shown, in step 208, method 200 may include generating a second embedding. For example, machine learning model system 102 may generate a plurality of second embeddings, the plurality of second embeddings including a corresponding second embedding associated with each of a plurality of second entities.

[0091] In some non-limiting embodiments or aspects, generating a plurality of second embeddings may include randomly initializing a plurality of second embeddings.

[0092] like Figure 2 As shown, in step 210, method 200 may include generating a second coded representation based on a second transformer encoder. For example, machine learning model system 102 may generate multiple second coded representations based on a second entity context and a second transformer encoder for each of a plurality of second entities, the multiple second coded representations including a corresponding second coded representation associated with each of the plurality of second entities.

[0093] like Figure 2As shown, in step 212, method 200 may include adjusting (e.g., training) a first embedding and a first transformer encoder based on a first contrastive loss. For example, machine learning model system 102 may adjust (e.g., train) multiple first embeddings and a first transformer encoder based on a first contrastive loss between multiple first embeddings and multiple first encoded representations.

[0094] like Figure 2 As shown, in step 214, method 200 may include adjusting (e.g., training) a second embedding and a second transformer encoder based on a second contrastive loss. For example, machine learning model system 102 may adjust (e.g., train) multiple second embeddings and a second transformer encoder based on a second contrastive loss between multiple second embeddings and multiple second encoded representations.

[0095] In some non-limiting embodiments or aspects, at least some of the foregoing steps may be repeated iteratively. For example, the machine learning model system 102 may iteratively repeat the following operations: generating a plurality of first encoded representations (e.g., step 206), generating a plurality of second encoded representations (e.g., step 210), adjusting a plurality of first embeddings and a first transformer encoder (e.g., step 212), and adjusting a plurality of second embeddings and a second transformer encoder (e.g., step 212) until at least one termination condition is met.

[0096] In some non-limiting embodiments or aspects, the machine learning model system 102 may generate predictions based on a first embedding of a selected first entity among a plurality of first entities, a first entity context of the selected first entity, a first transformer encoder, a second transformer encoder, any combination thereof, etc.

[0097] In some non-limiting embodiments or aspects, the machine learning model system 102 may adjust (e.g., train) a plurality of first embeddings, a first transformer encoder, a plurality of second embeddings and a second transformer encoder based on the total loss determined according to the prediction.

[0098] In some non-limiting embodiments or aspects, the machine learning model system 102 may iteratively generate predictions and adjust a plurality of first embeddings, a first transformer encoder, a plurality of second embeddings and a second transformer encoder until at least one termination condition is met.

[0099] In some non-limiting embodiments or aspects, prediction may include sequential prediction.

[0100] In some non-limiting embodiments or aspects, the plurality of first entities may include a plurality of users. Alternatively, the plurality of second entities may include a plurality of content.

[0101] Now for reference Figure 3This diagram depicts an example payment processing network 300 according to a non-limiting embodiment or aspect. In some non-limiting embodiments or aspects, the payment processing network 300 may be used in conjunction with the systems, methods, and / or computer program products described herein, and / or the systems, methods, and / or computer program products described herein may be implemented in the payment processing network 300. Figure 3 As shown, the payment processing network 300 may include a transaction processing system 301, a payment gateway system 302, a merchant system 304, an issuer system 306, an acquirer system 308, and / or a consumer device 310. In some non-limiting embodiments or aspects, Figure 1 Each of the machine learning model system 102, data storage system 104, and / or user device 106 can be implemented by transaction processing system 301 (e.g., a portion thereof). In some non-limiting embodiments or aspects, Figure 1 At least one of the machine learning model system 102, data storage system 104, and / or user device 106 may be implemented independently of or include another system, another device, another group of systems, or another group of devices (e.g., a portion thereof) of the transaction processing system 301, such as merchant system 304, issuer system 306, acquirer system 308, consumer device 310, etc. For example, user device 106 may be implemented by at least one of the payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, and / or consumer device 310 (e.g., a portion thereof). Alternatively or additionally, for example, data storage system 104 may be implemented by at least one of the payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, and / or consumer device 310 (e.g., a portion thereof).

[0102] Transaction processing system 301 may include one or more devices capable of receiving and / or transmitting information to payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, consumer device 310, etc. (e.g., directly, indirectly, via public and / or private communication networks). For example, such as Figure 3As shown, transaction processing system 301 can communicate with one or more issuer systems (e.g., issuer system 306), one or more acquirer systems (e.g., acquirer system 308), and / or one or more payment gateway systems (e.g., payment gateway system 302). Although only a single issuer system 306, a single acquirer system 308, and a single payment gateway system 302 are shown, it should be understood that transaction processing system 301 can communicate with multiple issuer systems, multiple acquirer systems, and / or multiple payment gateways. In some non-limiting embodiments or aspects, transaction processing system 301 may include computing devices, such as servers (e.g., transaction processing servers), server clusters, and / or other similar devices. In some non-limiting embodiments or aspects, transaction processing system 301 can communicate with data storage devices, which may be local or remote to transaction processing system 301. In some non-limiting embodiments or aspects, transaction processing system 301 may be able to receive information from data storage devices, store information in data storage devices, transmit information to data storage devices, or search for information stored in data storage devices. In some non-limiting embodiments or aspects, transaction processing system 301 may be associated with a transaction service provider, as described herein. In some non-limiting embodiments or aspects, transaction processing system 301 may also operate as an issuer system, such that transaction processing system 301 and issuer system 306 are both a single system and / or controlled by a single entity.

[0103] The payment gateway system 302 may include one or more devices capable of receiving and / or transmitting information to and / or to transaction processing systems 301, merchant systems 304, issuer systems 306, acquirer systems 308, consumer devices 310, etc. (e.g., directly, indirectly, or via public and / or private communication networks). For example, such as... Figure 3 As shown, the payment gateway system 302 can communicate with one or more merchant systems (e.g., merchant system 304), one or more acquiring systems (e.g., acquiring system 308), and / or one or more transaction processing systems (e.g., transaction processing system 301). Although only a single merchant system 304, a single acquiring system 308, and a single transaction processing system 301 are shown, it should be understood that the payment gateway system 302 can communicate with multiple merchant systems, multiple acquiring systems, and / or multiple transaction processing systems. In some non-limiting embodiments or aspects, the payment gateway system 302 may include computing devices, such as servers, server groups, and / or other similar devices. In some non-limiting embodiments or aspects, the payment gateway system 302 may be associated with a payment gateway as described herein.

[0104] Merchant system 304 may include one or more devices capable of receiving and / or transmitting information to and / or to transaction processing system 301, payment gateway system 302, issuer system 306, acquirer system 308, consumer device 310, etc. (e.g., directly, indirectly, via public and / or private communication networks). For example, such as Figure 3 As shown, merchant system 304 may communicate with one or more payment gateway systems (e.g., payment gateway system 302), one or more acquiring systems (e.g., acquiring system 308), and / or one or more consumer devices (e.g., consumer device 310). Although only a single payment gateway system 302, a single acquiring system 308, and a single consumer device 310 are shown, it should be understood that merchant system 304 may communicate with multiple payment gateway systems, multiple acquiring systems, and / or multiple consumer devices. In some non-limiting embodiments or aspects, merchant system 304 may include computing devices, such as servers, server groups, client devices, client device groups, POS devices, POS systems, computers, computer systems, peripheral devices, and / or other similar devices. In some non-limiting embodiments or aspects, merchant system 304 may be associated with a merchant, as described herein. In some non-limiting embodiments or aspects, the merchant system 304 may include means capable of receiving and / or transmitting information to the consumer device 310 via a short-range communication connection (e.g., NFC, RFID, Bluetooth®, Zigbee®, etc.) with the consumer device 310. In some non-limiting embodiments or aspects, the merchant system 304 may include one or more client devices. For example, the merchant system 304 may include client devices that allow the merchant to transmit information to the transaction processing system 301 (e.g., via at least one of the acquiring system 308 and / or the payment gateway system 302). In some non-limiting embodiments or aspects, the merchant system 304 (e.g., its client devices, its POS devices, etc.) may also operate as a payment gateway system, such that both the merchant system 304 and the payment gateway system 302 are a single system and / or controlled by a single entity.

[0105] The issuing system 306 may include one or more devices capable of receiving information and / or transmitting information (e.g., directly, indirectly, or via public and / or private communication networks) to the transaction processing system 301, payment gateway system 302, merchant system 304, acquiring system 308, consumer device 310, etc. For example, such as Figure 3As shown, the issuer system 306 may communicate with one or more transaction processing systems (e.g., transaction processing system 301) and / or one or more consumer devices (e.g., consumer device 310). Although only a single transaction processing system 301 and a single consumer device 310 are shown, it should be understood that the issuer system 306 may communicate with multiple transaction processing systems and / or multiple consumer devices 310. In some non-limiting embodiments or aspects, the issuer system 306 may include computing devices, such as servers, server groups, and / or other similar devices. In some non-limiting embodiments or aspects, the issuer system 306 may be associated with an issuing authority, as described herein. For example, the issuer system 306 may be associated with an issuing authority that issues credit accounts, debit accounts, credit cards, debit cards, payment devices, etc., to users associated with consumer devices 310.

[0106] The acquiring system 308 may include one or more devices capable of receiving and / or transmitting information to and / or to the transaction processing system 301, payment gateway system 302, merchant system 304, issuer system 306, consumer device 310, etc. (e.g., directly, indirectly, or via public and / or private communication networks). For example, such as Figure 3 As shown, the acquiring system 308 may communicate with one or more transaction processing systems (e.g., transaction processing system 301), one or more payment gateway systems (e.g., payment gateway system 302), and / or one or more merchant systems (e.g., merchant system 304). Although only a single transaction processing system 301, a single payment gateway system 302, and a single merchant system 304 are shown, it should be understood that the acquiring system 308 may communicate with multiple transaction processing systems, multiple payment gateway systems, and / or multiple merchant systems. In some non-limiting embodiments or aspects, the acquiring system 308 may include computing devices, such as servers, server clusters, and / or other similar devices. In some non-limiting embodiments or aspects, the acquiring system 308 may be associated with an acquiring institution as described herein.

[0107] Consumer device 310 may include one or more devices capable of receiving and / or transmitting information to transaction processing system 301, payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, etc. (e.g., directly, indirectly, via public and / or private communication networks). For example, such as Figure 3As shown, consumer device 310 can communicate with one or more merchant systems (e.g., merchant system 304) and / or one or more issuer systems (e.g., issuer system 306). Although only a single merchant system 304 and a single issuer system 306 are shown, it should be understood that consumer device 310 can communicate with multiple merchant systems and / or multiple issuer systems. In some non-limiting embodiments or aspects, consumer device 310 can be associated with a user who has issued a credit account, debit account, credit card, debit card, payment device, etc. In some non-limiting embodiments or aspects, user device 310 may include computing devices, such as computers, laptop computers, laptop computers, tablet computers, mobile devices, cellular phones, smartphones, wearable devices (e.g., watches, glasses, lenses, clothing, etc.), PDAs, client devices, and / or other similar devices. In some non-limiting embodiments or aspects, user device 310 may include payment devices, as described herein. In some non-limiting embodiments or aspects, consumer device 310 may include means capable of receiving information from and / or transmitting information to other client devices 310 (e.g., directly, indirectly, via public and / or private communication network connections, short-range communication connections, etc.). In some non-limiting embodiments or aspects, consumer device 310 may include means capable of receiving information from and / or transmitting information to merchant system 304 via a short-range communication connection (e.g., NFC communication connection, RFID communication connection, Bluetooth® communication connection, Zigbee® communication connection, etc.). In some non-limiting embodiments or aspects, consumer device 310 may include client devices.

[0108] In some non-limiting embodiments or aspects, the transaction processing system 301 may communicate directly with the merchant system 304 (e.g., via a public and / or private communication network connection, etc.). Alternatively, the transaction processing system 301 may communicate with the merchant system 304 via a payment gateway 302 and / or an acquiring system 308. In some non-limiting embodiments or aspects, the acquiring system 308 associated with the merchant system 304 may operate as a payment gateway 302 to facilitate the transmission of transaction messages (e.g., authorization requests) from the merchant system 304 to the transaction processing system 301. In some non-limiting embodiments or aspects, the merchant system 304 may communicate directly (e.g., via a public and / or private communication network connection, etc.) with the payment gateway 302. For example, the merchant system 304, including a physical POS device, may communicate with the payment gateway 302 via a public or private network for card transactions. As another example, the merchant system 304, including a server (e.g., a web server), may communicate with the payment gateway 302 via a public or private network, such as the Internet, for cardless transactions.

[0109] For illustrative purposes, processing a transaction (e.g., a payment transaction) may include generating a transaction message (e.g., an authorization request, etc.) based on the account identifier of a customer (e.g., an account holder associated with a customer device 310, etc.) and / or transaction data associated with the transaction. For example, a merchant system 304 (e.g., a client device of merchant system 304, a POS device of merchant system 304, etc.) may initiate a transaction, for example, by generating an authorization request (e.g., in response to receiving an account identifier from a customer's payment device and / or portable financial device, etc.). Merchant system 304 may transmit the authorization request to payment gateway 302 and / or acquiring system 308. In some non-limiting embodiments or aspects, payment gateway 302 may transmit the authorization request to acquiring system 308 and / or transaction processing system 301. Alternatively or concurrently, acquiring system 308 (and / or payment gateway 302) may transmit the authorization request to transaction processing system 301. After receiving an authorization request from merchant system 304 that identifies a customer's account identifier (e.g., the account holder and / or account identifier associated with consumer device 310), transaction processing system 301 may transmit the authorization request to issuer system 306 (e.g., the issuer system that issues the payment device and / or account identifier). Issuer system 306 may determine an authorization decision (e.g., approval, rejection, etc.) based on the authorization request, and / or issuer system 306 may generate an authorization response based on the authorization decision and / or authorization request. Issuer system 306 may transmit the authorization response to transaction processing system 301. Transaction processing system 301 may transmit the authorization response to acquiring system 308 and / or payment gateway 302. In some non-limiting embodiments or aspects, acquiring system 308 may transmit the authorization response to payment gateway 302 and / or merchant system 304. Alternatively or additionally, payment gateway 302 (and / or acquiring system 308) may transmit the authorization response to merchant system 304.

[0110] In some non-limiting embodiments or aspects, transaction processing system 301 and / or issuer system 306 may include at least one machine learning model (e.g., at least one of a fraud detection model, risk detection model, transaction authorization model, credit approval model, product recommendation model, classifier model, anomaly detection model, authentication model, any combination thereof, etc.). For example, the machine learning model may include at least one transformer encoder (e.g., a pre-trained dual transformer for bipartite graphs, etc.), as described herein. Transaction processing system 301 and / or issuer system 306 may perform at least one task (e.g., generating predictions and / or generating embeddings) based on an authorization request and one or more machine learning models. For example, performing one or more tasks may include generating at least one prediction associated with fraud detection, risk detection, transaction authorization, credit approval, product recommendation, classification, anomaly detection, authentication, any combination thereof, etc. In some non-limiting embodiments or aspects, transaction processing system 301 may transmit at least one message to issuer system 306 (e.g., together with an authorization request) based on performing a task (e.g., generating predictions and / or generating embeddings). In some non-limiting embodiments or aspects, the issuer system 306 may determine an authorization decision (e.g., approval, rejection, etc.) based on the authorization request and the execution of the task (e.g., predicted generation and / or embedded generation).

[0111] For illustrative purposes, transaction clearing and / or settlement may include generating messages (e.g., clearing messages, etc.) based on a customer's (e.g., associated with customer device 310, etc.) account identifier and / or transaction data associated with the transaction. For example, merchant system 304 may generate at least one clearing message (e.g., multiple clearing messages, a batch of clearing messages, etc.). Merchant system 304 may transmit one or more clearing messages to acquiring system 308 (and / or payment gateway 302, which may transmit one or more clearing messages to acquiring system 308). Acquiring system 308 may transmit one or more clearing messages to transaction processing system 301. Transaction processing system 301 may transmit one or more clearing messages to issuing system 306. Issuing system 306 may generate at least one settlement message based on one or more clearing messages. In some non-limiting embodiments or aspects, the issuer system 306 may transmit one or more settlement messages and / or funds to the transaction processing system 301 (and / or the settlement bank system associated with the transaction processing system 301), and the transaction processing system 301 (and / or the settlement bank system) may transmit one or more settlement messages and / or funds to the acquiring system 308. Alternatively, the issuer system 306 may transmit one or more settlement messages and / or funds to the acquiring system 308. In some non-limiting embodiments or aspects, the acquiring system 308 may transmit one or more settlement messages and / or funds to the merchant system 304 (and / or the account associated with the merchant system 304).

[0112] Figure 3 The system and / or device may communicate via one or more wired and / or wireless communication networks. For example, one or more communication networks may include cellular networks (e.g., Long Term Evolution (LTE®) networks, third-generation (3G) networks, fourth-generation (4G) networks, fifth-generation (5G) networks, Code Division Multiple Access (CDMA) networks, etc.), Public Land Mobile Networks (PLMN), Local Area Networks (LAN), Wide Area Networks (WAN), Metropolitan Area Networks (MAN), Telephone Networks (e.g., Public Switched Telephone Network (PSTN)), Private Networks (e.g., Private Networks Associated with Transaction Service Providers), Temporary Networks, Intranets, the Internet, Fiber-based Networks, Cloud Computing Networks, etc., and / or combinations of these or other types of networks.

[0113] Provided as an example Figure 3 The number and arrangement of the systems, devices, and / or networks shown. Additional systems, devices, and / or networks may exist; fewer systems, devices, and / or networks may exist; different systems, devices, and / or networks may exist; and / or networks may be associated with… Figure 3 The systems, devices, and / or networks shown are arranged in different ways. Furthermore, implementation can be carried out within a single system or device. Figure 3 The two or more systems or devices shown, or Figure 4 The single system or device shown may be implemented as multiple distributed systems or devices. Alternatively, a group of systems (e.g., one or more systems) or a group of devices (e.g., one or more devices) of the payment processing network 300 may perform one or more functions described as being performed by another group of systems or devices of the payment processing network 300.

[0114] For reference Figure 1 The diagram illustrates example components of a device 400 according to a non-limiting embodiment. As an example, device 400 may correspond to... Figure 3 At least one and / or one of the machine learning model system 102, data storage system 104, and / or user device 106. Figure 4 The system comprises at least one of the following: transaction service provider system 302, issuer system 304, client device 306, merchant system 308, and / or acquirer system 310. In some non-limiting embodiments, such a system or device may include at least one device 400 and / or at least one component of device 400. The number and arrangement of the components shown are provided as examples. In some non-limiting embodiments, device 400 may include additional components, fewer components, different components, or components arranged differently compared to those shown. Additionally or alternatively, a set of components of device 400 (e.g., one or more components) may perform one or more functions described as being performed by another set of components of device 400.

[0115] like Figure 4 As shown, device 400 may include bus 402, processor 404, memory 406, storage component 408, input component 410, output component 412, and communication interface 414. Bus 402 may include components that allow communication between components of device 400. In some non-limiting embodiments, processor 404 may be implemented in hardware, firmware, or a combination of hardware and software. For example, processor 404 may include processors (e.g., central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), etc.), microprocessors, digital signal processors (DSPs), and / or any processing component that can be programmed to perform functions (e.g., field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), etc.). Memory 406 may include random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by processor 404.

[0116] Continue to refer to Figure 5A Storage component 408 may store information and / or software related to the operation and use of device 400. For example, storage component 408 may include a hard disk (e.g., magnetic disk, optical disk, magneto-optical disk, solid-state disk, etc.) and / or another type of computer-readable medium. Input component 410 may include components that allow device 400 to receive information, for example, through user input (e.g., touch screen display, keyboard, keypad, mouse, buttons, switches, microphone, etc.). Alternatively or additionally, input component 410 may include sensors for sensing information (e.g., Global Positioning System (GPS) components, accelerometers, gyroscopes, actuators, etc.). Output component 412 may include components that provide output information from device 400 (e.g., display, speaker, one or more light-emitting diodes (LEDs), etc.). Communication interface 414 may include transceiver-like components (e.g., transceivers, separate receivers and transmitters, etc.) that enable device 400 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 414 may allow device 400 to receive information from another device and / or provide information to another device. For example, the communication interface 414 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a cellular network interface, etc.

[0117] Apparatus 400 can perform one or more processes described herein. Apparatus 400 can perform these processes based on processor 404 executing software instructions stored in a computer-readable medium such as memory 406 and / or storage component 408. The computer-readable medium can include any non-transient memory device. Memory devices include memory space located within a single physical storage device or memory space distributed across multiple physical storage devices. Software instructions can be read into memory 406 and / or storage component 408 via communication interface 414 from another computer-readable medium or from another device. When executed, the software instructions stored in memory 406 and / or storage component 408 cause processor 404 to perform one or more processes described herein. Additionally or alternatively, hard-wired circuitry may be used in place of or in conjunction with the software instructions to perform one or more processes described herein. Therefore, the embodiments described herein are not limited to any particular combination of hardware circuitry and software. As used herein, the term “configured to” can refer to an arrangement of software, apparatus, and / or hardware for performing and / or realizing one or more functions (e.g., actions, processes, steps of processes, etc.). For example, "a processor configured to..." can refer to a processor that executes software instructions (such as program code) that cause the processor to perform one or more functions.

[0118] Now for reference Figure 5B Figure 5 illustrates a schematic diagram of an example bipartite graph 500 according to some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, graph data associated with the bipartite graph 500 may be stored (e.g., via machine learning model system 102, data storage system 104, and / or user device 106). In some non-limiting embodiments or aspects, graph data associated with the bipartite graph 500 may be received (e.g., via machine learning model system 102 from data storage system 104 and / or user device 106). As shown in Figure 5, the bipartite graph 500 may include a plurality of first nodes 501 (e.g., First node), multiple second nodes 502 (e.g., (Second node) and multiple edges 503.

[0119] In some non-limiting embodiments or aspects, each of the plurality of first nodes 501 may be associated with a plurality of first entities (e.g., users). The first entity (e.g., user) in the ) is associated with the second entity. Alternatively, each of the plurality of second nodes 502 may be associated with a plurality of second entities (e.g., content). The second entity (e.g., content) in the context of the first entity (e.g., user) may be associated with the second entity (e.g., content). In some non-limiting embodiments or aspects, each of the plurality of edges 503 may be associated with an interaction between a corresponding first entity (e.g., user) and a corresponding second entity (e.g., content).

[0120] In some non-limiting embodiments or aspects, the bipartite graph It can be represented as ,in It is the set of vertices (e.g., nodes) of the first entity (e.g., user), such that , It is the set of vertices (e.g., nodes) of a second entity (e.g., content), such that ,and It makes each edge Represents the user's first entity Second Entity A collection of interactions between them.

[0121] Now for reference Figure 5C and Figure 5A And continue to refer to Figure 5B , Figure 5C and Figure 5B A schematic diagram of an example context for entities of a bipartite graph 500 is shown, according to some non-limiting embodiments or aspects.

[0122] In some non-limiting embodiments or aspects, such as Figure 5C As shown, multiple first entities (e.g., users) The first entity context of each first entity (e.g., user) in the bipartite graph 500 may include all second entities (e.g., content) connected to the first entity via at least one edge 503 of a plurality of edges 503 of the bipartite graph 500.

[0123] In some non-limiting embodiments or aspects, the first entity context (e.g., user) User behavior history ) can be represented as an edge vector (e.g., ,in Through the edge and users (The corresponding content for interaction). In some non-limiting embodiments or aspects, if the edge has attributes, then It can be a multivariate vector. In some non-limiting embodiments or aspects, for the sake of simplifying notation, user behavior history... It can be represented as For example, in the absence of each edge In the case of related attributes.

[0124] In some non-limiting embodiments or aspects, such as Figure 6 As shown, multiple second entities (e.g., content) The second entity context of each second entity (e.g., content) in the bipartite graph 500 may include all first entities (e.g., users) connected to the second entity through at least one edge 503 of a plurality of edges 503 of the bipartite graph 500.

[0125] In some non-limiting embodiments or aspects, the second entity context (e.g., content) Content history ) can be represented as an edge vector (e.g., ,in Through edges and content (The corresponding user who interacts). In some non-limiting embodiments or aspects, for the sake of simplifying symbols, content history... It can be represented as For example, if It is a single variable.

[0126] Now for reference Figure 6 The diagram illustrates an example implementation 600 of a pre-trained dual transformer for bipartite graphs according to some non-limiting embodiments or aspects. Equation 1 The components shown are for illustrative purposes only. It should be understood that additional, fewer, different, and / or different orders of components may be used in non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, implementation 600 may be implemented (e.g., wholly, partially, etc.) by machine learning model system 102 (e.g., one or more devices of machine learning model system 102). In some non-limiting embodiments or aspects, implementation 600 may be implemented (e.g., wholly, partially, etc.) by another system, another device, another group of systems, or another group of devices independent of or including machine learning model system 102, such as data storage system 104, user device 106, etc.

[0127] In some non-limiting embodiments or aspects, the first embedding layer 606-1 may generate a plurality of first embeddings 608-1, as described herein. For example, a first embedding 608-1 may include a user associated with a plurality of first entities (e.g., a user associated with a first node 501 of the bipartite graph 500). The corresponding first embedding associated with each first entity (e.g., user) in ).

[0128] In some non-limiting embodiments or aspects, the first transformer encoder 602-1 may generate a plurality of first encoded representations 604-1, as described herein. For example, the first encoded representation 604-1 may include a user associated with a plurality of first entities (e.g., a user associated with a first node 501 of the bipartite graph 500). The corresponding first encoded representation associated with each first entity (e.g., user) in the ) can be generated based on the first entity context of each first entity (e.g., by inputting the first entity context into the first transformer encoder 602-1 to generate the corresponding first encoded representation).

[0129] In some non-limiting embodiments or aspects, the second embedding layer 606-2 may generate a plurality of second embeddings 608-2, as described herein. For example, a second embedding 608-2 may include content associated with a plurality of second entities (e.g., content associated with a second node 502 of the bipartite graph 500). The corresponding second embedding associated with each second entity (e.g., content) in ).

[0130] In some non-limiting embodiments or aspects, the second transformer encoder 602-2 may generate a plurality of second encoded representations 604-2, as described herein. For example, the second encoded representation 604-2 may include content associated with a plurality of second entities (e.g., content associated with a second node 502 of the bipartite graph 500). The corresponding second encoded representation associated with each second entity (e.g., content) in the ) can be generated based on the second entity context of each second entity (e.g., by inputting the second entity context into the second transformer encoder 602-2 to generate the corresponding second encoded representation).

[0131] In some non-limiting embodiments or aspects, the user The embedding can be represented as .user The context (e.g., presenting user behavior history) (Form) can be with embedding The same entity (e.g., user) Heterogeneous representation of ).

[0132] In some non-limiting embodiments or aspects, the content The embedding can be represented as .content Context (e.g., presenting content history) (Form) can be with embedding Same entity (e.g., content) Heterogeneous representation of ).

[0133] In some non-limiting embodiments or aspects, learning (e.g., pre-training) can match two representations (e.g., embeddings and context) to provide a bidirectional mapping between the user and content spaces. For example, the goal of pre-training could be to jointly increase (e.g., maximize) interaction information. and For example, the goal of pre-training could be to maximize the following equation: Equation 2

[0134] In some non-limiting embodiments or aspects, interactive information It can be defined based on the following equation: Equation 3

[0135] In some non-limiting embodiments or aspects, given the edges from a time-aware bipartite graph On the edge Users before the timestamp User behavior history and content Content history The goal of pre-training can be to learn how to... as well as The embedding is mapped to the neighborhood region in the latent space (e.g., the first latent space, the user latent space, etc.), and the embedding is mapped to the neighborhood region in the latent space. as well as The embedding is mapped to a neighborhood in an independent latent space (e.g., a second latent space, a content latent space, etc.). For example, the dual transformer model may include four parts: (1) a user embedding layer (e.g., first embedding layer 606-1), (2) content embedding layer (e.g., second embedding layer 606-2), (3) user history encoder (e.g., first transformer encoder 602-1) and (4) content history encoder (For example, the second transformer encoder 602-2).

[0136] In some non-limiting embodiments or aspects, the user history encoder (For example, the first transformer encoder 602-1) may include a multi-head transformer encoder. Given a user User history encoder Can receive behavioral history As input, and output a category token (e.g., a CLS token) as... The encoded representation. For example, this output can be represented as .

[0137] In some non-limiting embodiments or aspects, the content history encoder (For example, the second transformer encoder 602-2) may include a multi-head transformer encoder. Given a user Content history encoder Can receive History As input, and output a category token (e.g., a CLS token) as... The encoded representation. For example, this output can be represented as .

[0138] In some non-limiting embodiments or aspects, a first contrast loss (e.g., contrast loss 1) may be determined (e.g., by machine learning model system 102) between a plurality of first embeddings 608-1 and a plurality of first encoded representations 604-1.

[0139] In some non-limiting embodiments or aspects, a given edge User behavior history (e.g., user behavior history) Encoded representation ) and user embedding Comparison loss between It can be determined based on the following equation: Equation 4

[0140] in It is the joint signature that guarantees similarity (e.g., inner product), and It is an exponential function.

[0141] In some non-limiting embodiments or aspects, a second contrast loss (e.g., contrast loss 2) may be determined between a plurality of second embeddings 608-2 and a plurality of second encoded representations 604-2 (e.g., through machine learning model system 102). In some non-limiting embodiments or aspects, the total loss may be determined based on a first contrast loss (e.g., contrast loss 1) and a second contrast loss (e.g., contrast loss 2).

[0142] In some non-limiting embodiments or aspects, a given edge Content history (For example, content history) Encoded representation ) and content embedding Comparison loss between It can be determined based on the following equation: Equation 5

[0143] In some non-limiting embodiments or aspects, the plurality of first embedded 608-1 and / or first transformer encoder 602-1 may be adjusted based on at least one of a first contrast loss, total loss, or any combination thereof.

[0144] In some non-limiting embodiments or aspects, the plurality of second embedded 608-2 and / or second transformer encoders 602-2 may be adjusted based on the second comparativeness, total loss or any combination thereof.

[0145] In some non-limiting embodiments or aspects, as described herein, determining the contrast loss and adjusting the transformer encoder and / or embedding may be referred to as pre-training (e.g., performing a pre-training task).

[0146] In some non-limiting embodiments or aspects, after pre-training, a pre-trained dual-transformer model (e.g., at least one of a first transformer encoder 602-1 and / or a second transformer encoder 602-2) can be used to perform at least one task (e.g., a downstream task). For example, the downstream task may include generating at least one classification, prediction, fraud detection, risk detection, transaction authorization, credit approval, product recommendation, anomaly detection, authentication, or any combination thereof.

[0147] For illustrative purposes, downstream tasks may include recommendations. For example, given a user User embedding and user behavior history Recommendations can include predicting users. The next project that can be interacted with In some non-limiting embodiments or aspects, the user embeds... It may have already been based on, for example, contrast loss. Some long-term patterns were learned during pre-training. Additionally, or alternatively, recent user behavior history was also learned. It can capture some short-term but time-aware patterns. Therefore, by using user embeddings in combination... and user behavior history Different modes can be combined (e.g., long-term and short-term modes), and the use of computing resources can be reduced (e.g., because there is no need to process user data). (A long historical sequence).

[0148] In some non-limiting embodiments or aspects, for this illustrative downstream task, the user embeds... It can be made by an encoder The encoding is performed, and the resulting output can be compared with the history of recent user behavior. Cascading. The result of cascading can be further derived from... Encoding to provide the final representation .For example, It can be determined based on the following equation: Equation 6

[0149] In some non-limiting embodiments or aspects, the inner product Can be used to measure users Will with the project The possibility of interaction. In some non-limiting embodiments or aspects, the pre-trained dual transformer model (e.g., its transformer encoder) can be based on at least one loss (e.g., contrastive loss). Comparative loss Bayesian Personalized Ranking (BPR) Loss (at least one of the following, such as, or any combination thereof, etc.) for fine-tuning. For example, BPR loss. It can be determined based on the following equation: Equation 7

[0150] in It is a sigmoid function, and and These are the positive and negative samples for the next project, respectively.

[0151] In some non-limiting embodiments or aspects, comparison loss may be used. Comparative loss and BPR loss To determine the total loss (For example, for fine-tuning). For example, total loss. It can be determined based on the following equation: Table 1

[0152] For illustrative purposes, Table 1 shows the performance (e.g., based on recall and normalized discount cumulative gain (NDCG) metrics) of the example pre-trained dual transformer (PDT) described in this paper compared to other general (e.g., BPR-MF, NeuMF, NGCF, and LightGCN) and sequential (e.g., GRU4Rec, Caser, SASRec, and SRGNN) recommender models used for recommendation (downstream) tasks. As shown in Table 1, PDT outperforms other recommender models.

[0153] Figure 7

[0154] Now for reference Figure 7 The diagram illustrates an example implementation 700 of a pre-trained dual transformer for a bipartite graph, according to some non-limiting embodiments or aspects. Figure 7 The components shown are for illustrative purposes only. It should be understood that additional, fewer, different, and / or different orders of components may be used in non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, implementation 700 may be implemented (e.g., wholly, partially, etc.) by machine learning model system 102 (e.g., one or more devices of machine learning model system 102). In some non-limiting embodiments or aspects, implementation 700 may be implemented (e.g., wholly, partially, etc.) by another system, another device, another group of systems, or another group of devices independent of or including machine learning model system 102, such as data storage system 104, user device 106, etc. In some non-limiting embodiments or aspects, the first transformer encoder 702-1 may be the same as or similar to the first transformer encoder 602-1. In some non-limiting embodiments or aspects, the second transformer encoder 702-2 may be the same as or similar to the second transformer encoder 602-2. In some non-limiting embodiments or aspects, the third transformer encoder 702-3 may be the same as or similar to the first transformer encoder 602-1 and / or the first transformer encoder 702-1. For example, the third transformer encoder 702-3 may share parameters with the first transformer encoder 702-1, but the third transformer encoder 702-3 may have minor differences from the first transformer encoder 702-1, as described herein. In some non-limiting embodiments or aspects, the first transformer encoder 702-1, the second transformer encoder 702-2, and the third transformer encoder 702-3 may be collectively referred to as "transformer encoder 702" or individually as "transformer encoder 702".

[0155] In some non-limiting embodiments or aspects, each transformer encoder 702 may include multiple transformer layers 710 (e.g., N transformer layers). Each transformer layer 710 may include at least one multi-head attention network 712 (or a masked multi-head attention network 712a), at least one addition and normalization (Add&Norm) layer (e.g., a first Add&Norm layer 714-1 and a second Add&Norm layer 714-2, collectively referred to as "Add&Norm layer 714" and individually referred to as "Add&Norm layer 714"), and at least one feedforward network 716 (e.g., including at least one neural network layer, such as a fully connected neural network layer, etc.). For example, each transformer layer 710 may include a multi-head attention network 712 (or a masked multi-head attention network 712a) that receives input to the transformer layer 710, a first Add&Norm layer 714-1 that receives the output of the multi-head attention network 712 (or the masked multi-head attention network 712a) and receives the input of the transformer layer 710 (e.g., via residual connection), a feedforward network 716 that receives the output of the first Add&Norm layer 714-1, and a second Add&Norm layer 714-2 that receives the output of the feedforward network 716 and receives the output of the first Add&Norm layer 714-1 (e.g., via residual connection).

[0156] In some non-limiting embodiments or aspects, the input to the first transformer layer 710 of each transformer encoder 702 may include the entity context of the corresponding entity (and / or the embedding of entities in the entity context of the corresponding entity). For example, as Figure 7 As shown, the input to the first transformer layer 710 of the first transformer encoder 702-1 may include a user. The context (e.g., with the user) Related content The historical sequence (and / or its embeddings). For example, such as Figure 7 As shown, the input to the first transformer layer 710 of the second transformer encoder 702-2 may include content The context (e.g., with content) Related users The historical sequence (and / or its embeddings). For example, such as Equation 8 As shown, the input to the first transformer layer 710 of the third transformer encoder 702-3 may include a user The context (e.g., with the user) Related content The historical sequence (and / or its embeddings) of the data.

[0157] In some non-limiting embodiments or aspects, the input to each subsequent transformer layer 710 of each transformer encoder 702 (e.g., after the first transformer layer 710) may include the output of the previous transformer layer 710. For example, the input to the second transformer layer 710 may include the output of the first transformer layer 710, etc.

[0158] In some non-limiting embodiments or aspects, the user can be determined (e.g., via machine learning model system 102). The embedded and user output from the first transformer encoder 702-1 The first contrast loss between the encoded representations (e.g., user contrast loss), as described herein. In some non-limiting embodiments or aspects, the user contrast loss (e.g., It can be determined based on Equation 3.

[0159] In some non-limiting embodiments or aspects, the user can be adjusted based on a first contrast loss (e.g., user contrast loss). The embedding and / or first transformer encoder 702-1 (e.g., its parameters can be updated). For example, machine learning model system 102 can update (e.g., adjust) the parameters of the first transformer encoder 702-1 based on (e.g., loss) backpropagation, gradient calculation (e.g., based on loss), any combination thereof, etc.

[0160] In some non-limiting embodiments or aspects, the content can be determined (e.g., via machine learning model system 102). Embedding and content A second contrast loss (e.g., content contrast loss) between the encoded representations, as described herein. In some non-limiting embodiments or aspects, the content contrast loss (e.g., It can be determined based on Equation 4.

[0161] In some non-limiting embodiments or aspects, the content may be adjusted based on a second contrast loss (e.g., content contrast loss). The embedded and / or second transformer encoder 702-2 (e.g., its parameters can be updated). For example, the machine learning model system 102 can update (e.g., adjust) the parameters of the second transformer encoder 702-2 based on (e.g., loss) backpropagation, gradient calculation (e.g., based on loss), any combination thereof, etc.

[0162] In some non-limiting embodiments or aspects, as described herein, determining the contrast loss and adjusting the transformer encoder 702 and / or embedding may be referred to as pre-training (e.g., performing a pre-training task).

[0163] In some non-limiting embodiments or aspects, the parameters of the first transformer encoder 702-1 can be shared with the third transformer encoder 702-3 after pre-training. In some non-limiting embodiments or aspects, the third transformer encoder 702-3 can perform at least one task (e.g., a downstream task). For example, a downstream task may include generating at least one classification, prediction, fraud detection, risk detection, transaction authorization, credit approval, product recommendation, anomaly detection, authentication, or any combination thereof.

[0164] In some non-limiting embodiments or aspects, the output of the third transformer encoder 702-3 can be input into a machine learning model (e.g., a downstream machine learning model) to perform at least one task (e.g., a downstream task). For example, the downstream machine learning model may include at least one of the following: a classifier, a predictor, a fraud detection model, a risk detection model, a transaction authorization model, a credit approval model, a product recommendation model, a classifier model, an anomaly detection model, an authentication model, any combination thereof, etc. In some non-limiting embodiments or aspects, the downstream machine learning model may include at least one of the following: a neural network, a convolutional neural network, a recurrent neural network, an autoencoder (or a portion thereof), a long short-term memory (LSTM), a bidirectional LSTM, an attention network, a transformer, any combination thereof, etc.

[0165] In some non-limiting embodiments or aspects, the loss (e.g., downstream loss) can be determined (e.g., by machine learning model system 102) based on the output of the third transformer encoder 702-3 (e.g., predictions, etc.) (and / or the output of a downstream machine learning model) and the prediction target. For example, the prediction target may include a label, a known value, an actual value, any combination thereof, etc.

[0166] In some non-limiting embodiments or aspects, the parameters of the third transformer encoder 702-3 may be adjusted (e.g., updated) based on a loss (e.g., downstream loss). For example, the machine learning model system 102 may update (e.g., adjust) the parameters of the third transformer encoder 702-3 (and / or the downstream machine learning model) based on backpropagation (e.g., loss), gradient computation (e.g., loss-based), any combination thereof, etc.

[0167] In some non-limiting embodiments or aspects, the parameters of the first transformer encoder 702-1 can be expressed as follows: The parameters of the second transformer encoder 702-2 can be expressed as follows: Furthermore, the parameters of the third transformer encoder 702-3 can be expressed as follows: For illustrative purposes, downstream tasks may include recommendation, as described herein. For example, the final representation... (For example, predicted content) can be determined based on the following equation: Equation 9

[0168] In some non-limiting embodiments or aspects, BPR loss It can be based on an equation (for example, using equation 8 instead of equation 5). To determine.

[0169] In some non-limiting embodiments or aspects, the total loss It can be determined based on the following equation: Table 2

[0170] in and These are hyperparameters that control the weights of the contrast loss. In some non-limiting embodiments or aspects, the hyperparameters... and It can be determined based on hyperparameter tuning (e.g., grid search, etc.). For example, hyperparameters can have, for example... and The value of .

[0171] In some non-limiting embodiments or aspects, as described herein, determining the loss and adjusting the third transformer encoder 702-3 (and / or the downstream machine learning model) may be referred to as fine-tuning (e.g., fine-tuning the third transformer encoder 702-3 to perform downstream tasks).

[0172] For illustrative purposes, Table 2 shows the performance (e.g., based on recall and NDCG metrics) of example pre-trained dual transformers with 128 (PDT-Small) and 512 (PDT-Large) feedforward network (hidden) dimensions compared to other general (e.g., BPR-MF, NeuMF, NGCF, and LightGCN) and sequential (e.g., GRU4Rec, Caser, SASRec-Small, SASRec-Large, GCSAN, and SRGNN) recommender models for recommendation (downstream) tasks, as described in this paper. As shown in Table 2, PDT-Small and PDRT-Large outperform other recommender models.

[0173] ​

[0174] Although embodiments have been described in detail for illustrative purposes, it should be understood that such details are for the purposes described only, and this disclosure is not limited to the disclosed embodiments or aspects, but rather is intended to cover modifications and equivalent arrangements that fall within the spirit and scope of the appended claims. For example, it should be understood that this disclosure contemplates, as far as possible, that one or more features of any embodiment or aspect may be combined with one or more features of any other embodiment or aspect.

Claims

1. A computer-implemented method comprising: The system receives graph data associated with a bipartite graph using at least one processor. The bipartite graph includes a plurality of first nodes, a plurality of second nodes, and a plurality of edges. Each of the plurality of first nodes is associated with a first entity among a plurality of first entities, each of the plurality of second nodes is associated with a second entity among a plurality of second entities, and each of the plurality of edges is associated with an interaction between a corresponding first entity among the plurality of first entities and a corresponding second entity among the plurality of second entities. The first entity context of each of the plurality of first entities includes all second entities of the plurality of second entities connected to the first entity via at least one of the plurality of edges, and the second entity context of each of the plurality of second entities includes all first entities of the plurality of first entities connected to the second entity via at least one of the plurality of edges. A plurality of first embeddings are generated using at least one processor, the plurality of first embeddings including a corresponding first embedding associated with each of the plurality of first entities; At least one processor generates a plurality of first encoded representations based on the first entity context of each of the plurality of first entities and a first transformer encoder, the plurality of first encoded representations including a corresponding first encoded representation associated with each of the plurality of first entities; A plurality of second embeddings are generated using at least one processor, the plurality of second embeddings including a corresponding second embedding associated with each of the plurality of second entities; A plurality of second encoded representations are generated using at least one processor based on the second entity context and a second transformer encoder of each of the plurality of second entities, the plurality of second encoded representations including a corresponding second encoded representation associated with each of the plurality of second entities; The plurality of first embeddings and the first transformer encoder are adjusted using at least one processor based on a first contrast loss between the plurality of first embeddings and the plurality of first encoded representations; as well as The plurality of second embeddings and the second transformer encoder are adjusted using at least one processor based on a second contrast loss between the plurality of second embeddings and the plurality of second encoded representations.

2. The method according to claim 1, further comprising: The following operations are iteratively repeated using at least one processor: generating the plurality of first encoded representations, generating the plurality of second encoded representations, adjusting the plurality of first embeddings and the first transformer encoder, and adjusting the plurality of second embeddings and the second transformer encoder, until at least one termination condition is met.

3. The method according to claim 1, further comprising: A prediction is generated using at least one processor based on the first embedding of a selected first entity among the plurality of first entities, the first entity context of the selected first entity, the first transformer encoder, and the second transformer encoder.

4. The method of claim 3, further comprising: The plurality of first embeddings, the first transformer encoder, the plurality of second embeddings, and the second transformer encoder are adjusted using at least one processor based on the total loss determined according to the prediction.

5. The method of claim 4, further comprising: The prediction is iteratively generated and the plurality of first embeddings, the first transformer encoder, the plurality of second embeddings, and the second transformer encoder are adjusted using at least one processor until at least one termination condition is met.

6. The method of claim 3, wherein the prediction includes sequential prediction.

7. The method of claim 1, wherein the plurality of first entities comprises a plurality of users, and wherein the plurality of second entities comprises a plurality of content.

8. The method of claim 1, wherein generating the plurality of first embeddings includes randomly initializing the plurality of first embeddings.

9. The method of claim 1, wherein generating the plurality of second embeddings includes randomly initializing the plurality of second embeddings.

10. A system comprising: At least one processor is configured to: Receive graph data associated with a bipartite graph, the bipartite graph including a plurality of first nodes, a plurality of second nodes and a plurality of edges, each of the plurality of first nodes being associated with a first entity among a plurality of first entities, each of the plurality of second nodes being associated with a second entity among a plurality of second entities, each of the plurality of edges being associated with an interaction between a corresponding first entity among the plurality of first entities and a corresponding second entity among the plurality of second entities, wherein the first entity context of each of the plurality of first entities includes all second entities of the plurality of second entities connected to the first entity through at least one of the plurality of edges, and wherein the second entity context of each of the plurality of second entities includes all first entities of the plurality of first entities connected to the second entity through at least one of the plurality of edges; Generate a plurality of first embeddings, the plurality of first embeddings including a corresponding first embedding associated with each of the plurality of first entities; Based on the first entity context of each of the plurality of first entities and the first transformer encoder, a plurality of first encoded representations are generated, the plurality of first encoded representations including a corresponding first encoded representation associated with each of the plurality of first entities; Generate a plurality of second embeddings, the plurality of second embeddings including a corresponding second embedding associated with each of the plurality of second entities; Based on the second entity context and the second transformer encoder of each of the plurality of second entities, a plurality of second encoded representations are generated, the plurality of second encoded representations including a corresponding second encoded representation associated with each of the plurality of second entities; The plurality of first embeddings and the first transformer encoder are adjusted based on a first contrast loss between the plurality of first embeddings and the plurality of first encoded representations; as well as The plurality of second embeddings and the second transformer encoder are adjusted based on a second contrast loss between the plurality of second embeddings and the plurality of second encoded representations.

11. The system of claim 10, wherein the at least one processor is further configured to: The following operations are repeated iteratively: generating the plurality of first encoded representations, generating the plurality of second encoded representations, adjusting the plurality of first embeddings and the first transformer encoder, and adjusting the plurality of second embeddings and the second transformer encoder, until at least one termination condition is met.

12. The system of claim 10, wherein the at least one processor is further configured to: A prediction is generated based on the first embedding of a selected first entity among the plurality of first entities, the first entity context of the selected first entity, the first transformer encoder, and the second transformer encoder.

13. The system of claim 12, wherein the at least one processor is further configured to: The plurality of first embeddings, the first transformer encoder, the plurality of second embeddings, and the second transformer encoder are adjusted based on the total loss determined according to the prediction.

14. The system of claim 13, wherein the at least one processor is further configured to: The prediction is iteratively generated and the plurality of first embeddings, the first transformer encoder, the plurality of second embeddings, and the second transformer encoder are adjusted until at least one termination condition is met.

15. The system of claim 12, wherein the prediction includes sequential prediction.

16. The system of claim 10, wherein the plurality of first entities comprises a plurality of users, and wherein the plurality of second entities comprises a plurality of content.

17. The system of claim 10, wherein generating the plurality of first embeddings includes randomly initializing the plurality of first embeddings.

18. The system of claim 10, wherein generating the plurality of second embeddings includes randomly initializing the plurality of second embeddings.

19. A computer program product comprising at least one non-transitory computer-readable medium, said at least one non-transitory computer-readable medium comprising program instructions that, when executed by at least one processor, cause the at least one processor to: Receive graph data associated with a bipartite graph, the bipartite graph including a plurality of first nodes, a plurality of second nodes and a plurality of edges, each of the plurality of first nodes being associated with a first entity among a plurality of first entities, each of the plurality of second nodes being associated with a second entity among a plurality of second entities, each of the plurality of edges being associated with an interaction between a corresponding first entity among the plurality of first entities and a corresponding second entity among the plurality of second entities, wherein the first entity context of each of the plurality of first entities includes all second entities of the plurality of second entities connected to the first entity through at least one of the plurality of edges, and wherein the second entity context of each of the plurality of second entities includes all first entities of the plurality of first entities connected to the second entity through at least one of the plurality of edges; Generate a plurality of first embeddings, the plurality of first embeddings including a corresponding first embedding associated with each of the plurality of first entities; Based on the first entity context of each of the plurality of first entities and the first transformer encoder, a plurality of first encoded representations are generated, the plurality of first encoded representations including a corresponding first encoded representation associated with each of the plurality of first entities; Generate a plurality of second embeddings, the plurality of second embeddings including a corresponding second embedding associated with each of the plurality of second entities; Based on the second entity context and the second transformer encoder of each of the plurality of second entities, a plurality of second encoded representations are generated, the plurality of second encoded representations including a corresponding second encoded representation associated with each of the plurality of second entities; The plurality of first embeddings and the first transformer encoder are adjusted based on a first contrast loss between the plurality of first embeddings and the plurality of first encoded representations; as well as The plurality of second embeddings and the second transformer encoder are adjusted based on a second contrast loss between the plurality of second embeddings and the plurality of second encoded representations.

20. The computer program product of claim 19, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to: A prediction is generated based on the first embedding of a selected first entity among the plurality of first entities, the first entity context of the selected first entity, the first transformer encoder, and the second transformer encoder.

Images