Model training method and device based on edge attenuation sampling and terminal equipment

Abstract

The invention relates to a model training method and device based on edge decay sampling and terminal equipment, and is applied to training a continuous time dynamic graph model, the method comprises the following steps: dividing a continuous time dynamic graph into a plurality of dynamic sub-graphs, the plurality of dynamic sub-graphs comprising a first dynamic sub-graph; sampling a first neighbor node which is located in the first dynamic sub-graph and interacts with the target node according to a preset sampling probability, and sampling a second neighbor node which is located in the dynamic sub-graphs except the first dynamic sub-graph and interacts with the target node according to a preset sampling probability to obtain a sampling sub-graph; acquiring feature data of the first neighbor node and the second neighbor node; and training the initial continuous time dynamic graph model based on the sampling sub-graph and the feature data of the first neighbor node and the second neighbor node to obtain a completely trained continuous time dynamic graph model. Through the method and the device, the problem of relatively low parallel training efficiency of the continuous time dynamic graph model in related technologies is solved.

Description

Technical Field

[0001] This application belongs to the technical field of model training, and particularly relates to a model training method, device, and terminal device based on edge attenuation sampling. Background Art

[0002] Continuous-time dynamic graphs are applied in many fields. For example, financial transaction records can be stored in the form of graph data, where each account is regarded as a vertex and each transaction is regarded as an edge on the graph. Continuous-time dynamic graph models can directly learn the representations of continuously occurring events in continuous-time dynamic graphs, and make more effective use of time information to obtain stronger representation capabilities, thus attracting more and more attention.

[0003] Currently, when parallel training a continuous-time dynamic graph model, graph data is usually distributed to multiple trainers for training. In a distributed scenario, feature data of relevant neighbor nodes needs to be pulled from remote partitions, and a large number of neighbor nodes may be stored in remote partitions. Therefore, such a training method has high communication resources; at the same time, the remote access to neighbor nodes located in remote partitions takes a long time, and even exceeds the time taken to train the model, which will reduce the overall training efficiency of the model.

[0004] Currently, for the problem of low parallel training efficiency of continuous-time dynamic graph models in related technologies, no effective solution has been proposed. Summary of the Invention

[0005] Embodiments of this application provide a model training method, device, and terminal device based on edge attenuation sampling, so as to at least solve the problem of low parallel training efficiency of continuous-time dynamic graph models in related technologies.

[0006] In a first aspect, embodiments of this application provide a model training method based on edge attenuation sampling, which is applied to train a continuous-time dynamic graph model. The method includes: dividing a continuous-time dynamic graph into multiple dynamic subgraphs, where the multiple dynamic subgraphs include a first dynamic subgraph; sampling first neighbor nodes that interact with a target node in the first dynamic subgraph, and sampling second neighbor nodes that interact with the target node in the dynamic subgraphs other than the first dynamic subgraph according to a preset sampling probability, to obtain a sampled subgraph; obtaining feature data of the first neighbor nodes and the second neighbor nodes; and training an initial continuous-time dynamic graph model based on the sampled subgraph and the feature data of the first neighbor nodes and the second neighbor nodes, so as to obtain a trained complete continuous-time dynamic graph model.

[0007] In some embodiments, dividing the continuous-time dynamic graph into multiple dynamic subgraphs includes: counting the degrees of each node in the continuous-time dynamic graph; selecting a preset number of nodes as hot nodes in descending order of degrees; deleting the hot nodes and the edges related to the hot nodes from the continuous-time dynamic graph to obtain a residual dynamic graph; and dividing the residual dynamic graph into the multiple dynamic subgraphs.

[0008] In some embodiments, after dividing the residual dynamic graph into the multiple dynamic subgraphs, the method further includes: allocating the multiple dynamic subgraphs to multiple trainers, where each trainer stores one of the dynamic subgraphs; obtaining the feature data of the hot nodes; storing the feature data of the hot nodes in each trainer; allocating the edges between the hot nodes and the non-hot nodes in the residual dynamic graph to the trainer corresponding to the dynamic subgraph containing the non-hot nodes; and allocating the edges between the hot nodes to the multiple trainers based on a greedy strategy.

[0009] In some embodiments, sampling the first neighbor nodes that interact with a target node and are located in the first dynamic subgraph, and sampling the second neighbor nodes that interact with the target node and are located in the dynamic subgraphs other than the first dynamic subgraph according to a preset sampling probability to obtain a sampled subgraph includes: sampling the first neighbor nodes that interact with the target node and are located in the first dynamic subgraph in ascending order of timestamps of interaction with the target node based on a preset sampling quantity, and sampling the second neighbor nodes that interact with the target node and are located in the dynamic subgraphs other than the first dynamic subgraph according to the preset sampling probability; if the sum of the numbers of the sampled first neighbor nodes and the second neighbor nodes is less than the sampling quantity, sampling the neighbor nodes that interact with the target node and have not been sampled in ascending order of timestamps of interaction with the target node until the sum of the numbers of the sampled neighbor nodes is equal to the sampling quantity to obtain the sampled subgraph.

[0010] In some embodiments, the preset sampling probability is expressed as:

[0011]

[0012] where P u,t (v,τ) is the preset sampling probability, θ is the first hyperparameter, δ is the second hyperparameter, τ is the timestamp of interaction with the target node, t is the preset sampling time period, is the set containing all the second neighbor nodes.

[0013] In some embodiments, training an initial continuous-time dynamic graph model based on the sampled subgraph and the feature data of the first neighbor node and the second neighbor node to obtain a trained complete continuous-time dynamic graph model includes: for each trainer participating in model training, inputting the sampled subgraph and the feature data of the first neighbor node and the second neighbor node into the initial continuous-time dynamic graph model corresponding to the trainer through the trainer, and obtaining a historical neighbor embedding sequence of each node in the sampled subgraph output by the initial continuous-time dynamic graph model; based on the sampled subgraph corresponding to each trainer, the feature data of the first neighbor node and the second neighbor node, and the historical neighbor embedding sequence of each node in the sampled subgraph, training the initial continuous-time dynamic graph model through each trainer to obtain the trained complete continuous-time dynamic graph model.

[0014] In some embodiments, the feature data includes the node point feature, the edge feature of the edge associated with the node, and the memory unit state of the node.

[0015] In a second aspect, an embodiment of the present application provides a model training device based on edge attenuation sampling, which is applied to train a continuous-time dynamic graph model. The device includes: a division module for dividing a continuous-time dynamic graph into a plurality of dynamic subgraphs, where the plurality of dynamic subgraphs includes a first dynamic subgraph; a sampling module for sampling a first neighbor node interacting with a target node located in the first dynamic subgraph, and sampling a second neighbor node interacting with the target node located in the dynamic subgraph other than the first dynamic subgraph according to a preset sampling probability to obtain a sampled subgraph; an acquisition module for acquiring the feature data of the first neighbor node and the second neighbor node; and a training module for training an initial continuous-time dynamic graph model based on the sampled subgraph and the feature data of the first neighbor node and the second neighbor node to obtain a trained complete continuous-time dynamic graph model.

[0016] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the model training method based on edge attenuation sampling according to any one of the above first aspects.

[0017] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is run, the model training method based on edge attenuation sampling according to any one of the above first aspects is executed.

[0018] Compared with the related art, the model training method, device, and terminal device provided by the embodiments of the present application based on edge attenuation sampling can divide a continuous-time dynamic graph into multiple dynamic subgraphs, enabling each trainer participating in model training to be only responsible for its corresponding dynamic subgraph; for each trainer, by sampling the first neighbor nodes interacting with the target node in the first dynamic subgraph stored locally, and sampling the second neighbor nodes interacting with the target node in the dynamic subgraphs stored in other trainers according to a preset sampling probability, a sampled subgraph is obtained; finally, an initial continuous-time dynamic graph model is trained based on the sampled subgraph and the feature data of the first neighbor nodes and the second neighbor nodes. In this way, the sampling probability of remote neighbor nodes can be reduced, thereby reducing the communication resources and time costs caused by accessing the feature data of remote neighbor nodes, and further improving the parallel training efficiency of the continuous-time dynamic graph model. Through the present application, the problem of low parallel training efficiency of the continuous-time dynamic graph model in the related art is solved, and the technical effect of improving the parallel training efficiency of the continuous-time dynamic graph model is achieved.

[0019] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more concise and understandable. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] To more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0021] Figure 1 is a flowchart of a model training method based on edge attenuation sampling according to an embodiment of the present application;

[0022] Figure 2 is a schematic structural diagram of a model training device based on edge attenuation sampling according to an embodiment of the present application;

[0023] Figure 3 is a schematic structural diagram of a terminal device according to an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] In the following description, specific details such as specific system architectures and technologies are presented for purposes of illustration and not limitation, so as to provide a thorough understanding of the embodiments of the present application. However, those skilled in the art should understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from obscuring the description of the present application.

[0025] It should be understood that when used in the specification and claims of the present application, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0026] It should also be understood that the term "and / or" as used in the specification and claims of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0027] As used in the specification and claims of the present application, the term "if" can be interpreted, depending on the context, as "when", "once", "in response to determining", or "in response to detecting". Similarly, the phrase "if determined" or "if [the described condition or event] is detected" can be interpreted, depending on the context, as meaning "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]".

[0028] In addition, in the description of the specification and claims of the present application, the terms "first", "second", "third", etc. are only used for differentiating descriptions and cannot be construed as indicating or implying relative importance.

[0029] The reference to "one embodiment" or "some embodiments" or the like described in the specification of the present application means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the present application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized in another way.

[0030] Continuous-time dynamic graphs are applied in many fields. For example, financial transaction records can be stored in the form of graph data, where each account is regarded as a vertex and each transaction is regarded as an edge on the graph. Continuous-time dynamic graph models can directly learn the representations of continuously occurring events in continuous-time dynamic graphs, make more effective use of time information to obtain stronger representation capabilities, and have received increasing attention.

[0031] The Memory based Temporal Graph Neural Network (MTGNN) is a neural network learning architecture for continuous-time dynamic graphs, which shows excellent capabilities in node feature representation and link relationship prediction. However, the training speed of such models is slow and parallel training is required to accelerate.

[0032] When parallel training is currently carried out on continuous-time dynamic graph models, graph data is usually allocated to multiple trainers for training. In a distributed scenario, it is necessary to pull the feature data of relevant neighbor nodes from remote partitions, and a large number of neighbor nodes may be stored in remote partitions. Therefore, the communication resources of such a training method are relatively high; at the same time, the remote access to neighbor nodes located in remote partitions takes a long time, and even exceeds the time-consuming of training the model, which will reduce the overall training efficiency of the model.

[0033] Specifically, multiple trainers are used to train a continuous-time dynamic graph model. The continuous-time dynamic graph can be divided into multiple dynamic subgraphs, and each trainer manages a dynamic subgraph. The dynamic subgraph includes partition nodes, edges, static features, and the memory unit states of nodes. The memory unit state of each node includes a memory unit vector and the message of the last interaction event. The training is carried out in chronological order, and each trainer simultaneously processes a batch of edges in the dynamic subgraph it manages. The trainer first samples nodes for negative samples, and then samples the temporal neighbor nodes of the root node to process the event being trained. Subsequently, the neighbor nodes stored in the remote trainer, as well as their associated static features and memory unit states, are pulled from the remote trainer. Based on this data, the memory update layer in the continuous-time dynamic graph model generates new messages and updates the memory unit state of the root node, and aggregates according to the index and timestamp corresponding to the newly generated memory unit state and the content output by the remote trainer. The message passing process aggregates temporal neighbor information and calculates node embeddings. The newly calculated memory unit state by the memory update layer is stored for future time batch training. These processes involve two main communication phases: obtaining the feature data of remote neighbor nodes and aggregating the memory unit states of remote neighbor nodes. These two communication phases consume a large amount of communication resources and affect the efficiency of distributed training. To reduce the communication cost, recent research has explored various strategies. Currently, some systems optimize the graph partitioning method to minimize the cut edges. Although the graph partitioning algorithm can reduce additional communication resources, the large number of remote access processes for the feature data of remote neighbor nodes will result in a time-consuming that exceeds the training time, dominating the training time cost. The pipelined training method cannot completely reduce the communication delay through parallel computing and communication because the time required for remote data acquisition and remote aggregation far exceeds the time of the computing phase (including forward and backward propagation), thus blocking the entire training process. In addition, simply directly discarding some remote neighbor nodes that do not belong to this partition will cause a loss of accuracy, ultimately leading to a decrease in the accuracy of the trained model.

[0034] Currently, there is no effective solution to the problem of low parallel training efficiency of the continuous-time dynamic graph model in the related technology.

[0035] In view of this, the embodiments of the present application provide a model training method based on edge attenuation sampling. By dividing the continuous-time dynamic graph into multiple dynamic subgraphs, each trainer participating in the model training can be responsible only for its corresponding dynamic subgraph. For each trainer, by sampling the first neighbor nodes interacting with the target node in the first dynamic subgraph stored locally, and sampling the second neighbor nodes interacting with the target node in the dynamic subgraphs stored in other trainers according to a preset sampling probability, a sampled subgraph is obtained. Finally, the initial continuous-time dynamic graph model is trained based on the sampled subgraph and the feature data of the first neighbor nodes and the second neighbor nodes. In this way, the sampling probability of remote neighbor nodes can be reduced, thereby reducing the communication resources and time costs caused by accessing the feature data of remote neighbor nodes, and further improving the parallel training efficiency of the continuous-time dynamic graph model. Through the present application, the problem of low parallel training efficiency of the continuous-time dynamic graph model in the related art is solved, and the technical effect of improving the parallel training efficiency of the continuous-time dynamic graph model is achieved.

[0036] The following will combine Figure 1 to illustrate a model training method based on edge attenuation sampling provided by an embodiment of the present application. This method is applied to the training of a continuous-time dynamic graph model. The continuous-time dynamic graph can be defined as a series of interaction events (also called edges):

[0037] G = {e(u0,v0,t0), e(u1,v1,t1), …, e(u i ,v i ,,t i )…, e(u n ,v n ,t_n)};

[0038] Among them, each event e(u i ,v i ,,t i ) is a directed time edge from node u i to node v i , which occurs at timestamp t i .

[0039] In addition, the model training method based on edge attenuation sampling provided by the embodiments of the present application can be applied to MTGNN. The network layer of MTGNN mainly includes a memory update network layer and a message propagation network layer.

[0040] Among them, the memory update network layer is used to train a new event e(t) = (u, v, t). For the source node u and the target node v, a message will be generated These messages are used to update the memory unit status of the corresponding nodes. Define the memory unit of node u after interaction at timestamp t as s(u,t), and the update process of s(u,t) follows the following expression:

[0041]

[0042] where s(u,t - ) and s(v,t - ) represent the last updated memory unit vectors of nodes u and v before timestamp t. The UPDATE function usually involves a sequential network structure. For example, a Recurrent Neural Network (RNN) or a gated recurrent unit (GRU) network model.

[0043] The message propagation network layer is used to generate node embeddings h u (t) and h v (t) by aggregating neighbor nodes. Define N u (t) as the set of neighbor nodes of node u at timestamp t, that is, the set of neighbor nodes that have interacted before time t. Then the message passing process is as follows:

[0044]

[0045]

[0046] h u (t) = DOMB(s(u,t - ), g u )(t));

[0047] where the function SAMPLE represents the sampling mechanism for selecting neighbor nodes, the function φ is used to encode the time interval, AGGREGATE represents the neighbor node aggregation function. For example, it can be a Graph Attention Network (GAT). COMB usually uses a simple Multilayer Perceptron (MLP) network layer to combine the memory unit of the node with the aggregated neighbor node information.

[0048] Please refer to Figure 1 , Figure 1 which is a flowchart of a model training method based on edge decay sampling according to an embodiment of the present application. As Figure 1 shown, the method includes:

[0049] Step S101, divide the continuous-time dynamic graph into multiple dynamic subgraphs, where the multiple dynamic subgraphs include the first dynamic subgraph.

[0050] In the distributed training of MTGNN, the acquisition of feature data of remote neighbor nodes and the aggregation of memory unit states occupy the main communication cost. This is because some of the neighbor nodes accessed by sampling are stored on other trainers. When accessing these neighbor nodes, data communication needs to be carried out through the network, resulting in communication resources. This part of the communication volume is usually related to the cut edges of the graph partition.

[0051] Therefore, by sharing some nodes with higher popularity, the number of cut edges can be significantly reduced, thereby reducing the communication resources and time cost required for model training, and further improving the efficiency of parallel training of the continuous-time dynamic graph model.

[0052] Specifically, step S101 includes the following steps:

[0053] Step 1, count the degrees of each node in the continuous-time dynamic graph.

[0054] Step 2, select a preset number of nodes as hot nodes in descending order of degrees.

[0055] Step 3, delete the hot nodes and the edges related to the hot nodes from the continuous-time dynamic graph to obtain a residual dynamic graph.

[0056] Step 4, divide the residual dynamic graph into multiple dynamic subgraphs.

[0057] In this embodiment, the complete graph structure of the continuous-time dynamic graph can be obtained. The complete graph structure includes: an edge set E containing timestamps, a node set V, a hot node ratio k, and the number of partitions m to be divided into. These parameters are used to determine how to partition the continuous-time dynamic graph.

[0058] In step S101, the degrees D of each node in the continuous-time dynamic graph can be counted v to select hot nodes, thereby determining the importance and connectivity of each node. By calculating the degrees D of each node v , in descending order of degrees D v , select a preset number (for example, k times the total number of nodes) of nodes as hot nodes, and the hot nodes can be added to the hot node set H.

[0059] Then, the hot nodes and the edges related to the hot nodes can be deleted from the complete graph structure of the continuous-time dynamic graph to obtain a residual dynamic graph. The METIS algorithm or any common graph partitioning algorithm based on point partitioning can be used to perform graph partitioning on the residual dynamic graph to obtain m dynamic subgraphs.

[0060] In addition, after dividing the residual dynamic graph into multiple dynamic subgraphs, the method further includes the following steps:

[0061] Step 5: Assign the multiple dynamic subgraphs to multiple trainers, where each trainer stores one dynamic subgraph.

[0062] In this embodiment, a trainer participating in model training usually stores one dynamic subgraph and is responsible for the model training process related to the dynamic subgraph it stores.

[0063] Step 6: Obtain the feature data of the hot nodes.

[0064] In this embodiment, the feature data includes the node point features, the edge features of the edges related to the nodes, and the memory unit states of the nodes. The memory unit state of each node may include a memory unit vector and the last interaction event message.

[0065] Step 7: Store the feature data of the hot nodes in each trainer.

[0066] Step 8: Assign the edges between the hot nodes and the non-hot nodes in the residual dynamic graph to the trainers corresponding to the dynamic subgraphs containing the non-hot nodes.

[0067] Step 9: Based on the greedy strategy, assign the edges between the hot nodes to multiple trainers.

[0068] In this embodiment, since the degrees of the hot nodes are usually high, these hot nodes will be accessed more frequently than other nodes. Thus, the feature data of these hot nodes can be stored in each trainer, and when the subsequent trainers need to access the feature data of the hot nodes, they can directly read it from the local memory instead of reading it from other trainers through the network, thereby reducing the communication resources required for model training.

[0069] In addition, the mapping function of the greedy strategy in Step 9 above can satisfy:

[0070]

[0071] S(e,p i )=[Att(u,t,p i )+Att(v,t,p i )+1]·F BAL (p i );

[0072] where Att(u,t,p i ) satisfies:

[0073]

[0074] where Nu Let \((t)\) be the set of neighbor nodes that have interacted with node \(u\) before the preset time period \(t\). be the set of edges assigned to partition \(p\) i in \(F\) BAL (p i ) is the load balancing coefficient.

[0075] The load balancing coefficient \(F\) i under partition \(p\) BAL (p i ) can be defined as:

[0076] \(F\) BAL (p i ) = BN(p i ) · BE(p i ) · BT(p i );

[0077] Among them, BN(p i ) means that the number of nodes stored on each trainer is balanced, that is:

[0078]

[0079] BE(p i ) is used to constrain the difference in the number of edges in each partition when allocating edges:

[0080]

[0081] BT(p i ) is used to constrain the difference in the timestamps of the events processed by each partition, so as to avoid data being concentrated and allocated to one partition within the same time range:

[0082]

[0083] Step S102: Sample the first neighbor nodes that interact with the target node and are located in the first dynamic subgraph, and sample the second neighbor nodes that interact with the target node and are located in the dynamic subgraphs other than the first dynamic subgraph according to the preset sampling probability, to obtain a sampled subgraph.

[0084] In this embodiment, for the target node \(u\), if the MTGNN model requires sampling \(k\) neighbor nodes with the most recent interactions, for the target node \((u, t)\) with a non-empty neighbor node set \(N\) u (t), then construct the neighbor node set \(N\) u (t) from the \(k\) neighbor nodes corresponding to the timestamps that are less than \(t\) and the largest among the timestamps of the interactions with the root node.

[0085] Specifically, step S102 includes the following steps:

[0086] Step 1: Based on a preset number of samples, sample the first neighbor nodes in the first dynamic subgraph in the order of timestamps of interactions with the target node from earliest to latest, and sample the second neighbor nodes in the dynamic subgraphs other than the first dynamic subgraph according to a preset sampling probability.

[0087] Step 2: If the sum of the numbers of the sampled first neighbor nodes and second neighbor nodes is less than the number of samples, then sample the neighbor nodes that have interacted with the target node and have not been sampled in the order of timestamps of interactions with the target node from earliest to latest until the sum of the numbers of the sampled neighbor nodes is equal to the number of samples, obtaining a sampled subgraph.

[0088] In this embodiment, based on a preset number of samples (e.g., k), the first neighbor nodes in the set of the k neighbor nodes with the most recent interactions in the neighbor node set N u (t) that are in the first dynamic subgraph (i.e., stored in the local trainer) are added to the sampled subgraph; meanwhile, with a preset sampling probability P u,t (v,τ), sample the second neighbor nodes in the dynamic subgraphs other than the first dynamic subgraph (i.e., stored in remote trainers other than the local trainer) among the k neighbor nodes with the most recent interactions, thereby reducing the amount of data acquisition corresponding to the second neighbor nodes, thus greatly reducing the communication overhead, enabling the communication overhead to be hidden in the model calculation stage, making the training process no longer generate additional communication overhead, reducing the communication resources and time costs caused by accessing the feature data of remote neighbor nodes by reducing the sampling probability of remote neighbor nodes, and thus improving the parallel training efficiency of the continuous-time dynamic graph model.

[0089] Using such an edge attenuation sampling method, the aggregated output g u (t) in the message propagation network layer of MTGNN can be replaced with:

[0090]

[0091] where is the learned neighbor attention weight of the attention network layer in the message propagation network layer. The relationship between this weight and the interaction time follows an exponential distribution, ignoring the input point, edge features, and state vectors. A higher probability can be assigned to the second neighbor nodes with the most recent interactions according to the exponential distribution. For the second neighbor nodes, the preset sampling probability can be expressed as:

[0092]

[0093] where P u,t(v, τ) is the preset sampling probability; θ is the first hyperparameter used to control the total number of cross-partition samplings; δ is the second hyperparameter used to control the degree of change of the exponential distribution; τ is the timestamp of interaction with the target node, and t is the preset sampling time period. is a set containing all second neighbor nodes.

[0094] In addition, in this embodiment, if after edge attenuation sampling, the sum of the number of sampled neighbor nodes is less than the sampling number (for example, k), then the neighbor nodes that interact with the target node and have not been sampled can be sampled in the order of the timestamps of interaction with the target node from the earliest to the latest until the sum of the number of each sampled neighbor node is equal to the sampling number.

[0095] Step S103: Obtain the feature data of the first neighbor nodes and the second neighbor nodes.

[0096] In this embodiment, since the first neighbor nodes are located in the first dynamic subgraph stored by the local trainer, the local trainer can directly read the feature data of the first neighbor nodes from the local memory; while since the second neighbor nodes are located in other dynamic subgraphs stored by other remote trainers, it is necessary to pull the feature data of the second neighbor nodes stored in other remote trainers to the local trainer.

[0097] Step S104: Based on the sampled subgraph and the feature data of the first neighbor nodes and the second neighbor nodes, train the initial continuous-time dynamic graph model to obtain a trained complete continuous-time dynamic graph model.

[0098] In the continuous-time dynamic graph model, a graph is represented as a sequence of a series of interaction events. Each interaction event e(t) = (u, v, t, f) represents a directed temporal edge from node u to node v occurring at timestamp t, where f represents the feature vector of edge e(t) at timestamp t. A continuous-time dynamic graph with node set V and edge set E can be represented as G = (V, E, X N , X E ), where X N and X E are feature matrices storing node feature vectors and edge feature vectors.

[0099] For the training of the continuous-time dynamic graph model, it is necessary to first perform graph partitioning on the complete graph structure of the continuous-time dynamic graph; if m Graphics Processing Unit (GPU) trainers are involved in the model training process, then the continuous-time dynamic graph G(V, E, X N , X E ) will be partitioned into m dynamic subgraphs G0, G1, …, G m, where Each trainer is only responsible for the storage and training process of one of the dynamic subgraphs. Each trainer stores the feature vectors of the nodes in the corresponding dynamic subgraph, the feature vectors of the edges (interaction events), as well as the dynamically updated memory unit vectors and the last interaction messages for these nodes. In addition, the feature data of the hot nodes is also stored in each trainer.

[0100] The training process of the continuous-time dynamic graph model can be divided into multiple time batches for training and follows the time order. For each interaction event in the same time batch, the training should be carried out in the strictly increasing order of timestamps. In the distributed training process, each trainer is only responsible for the training of the events E p assigned to the local, and the events belonging to the same time batch on different trainers are trained simultaneously.

[0101] In one embodiment, step S104 includes the following steps:

[0102] Step 1, for each trainer participating in the model training, input the sampled subgraph and the feature data of the first neighbor nodes and the second neighbor nodes into the initial continuous-time dynamic graph model corresponding to the trainer, and obtain the historical neighbor embedding sequences of each node in the sampled subgraph output by the initial continuous-time dynamic graph model.

[0103] Step 2, based on the sampled subgraphs corresponding to each trainer, the feature data of the first neighbor nodes and the second neighbor nodes, and the historical neighbor embedding sequences of each node in the sampled subgraphs, train the initial continuous-time dynamic graph model through each trainer to obtain a trained complete continuous-time dynamic graph model.

[0104] In this embodiment, each trainer executes the model training steps: for the training events in the same time batch, a batch of target nodes can be randomly sampled as negative samples first. Then, the edge decay sampling algorithm is used to sample the set of neighbor nodes that the source node and the target node interacted with most recently to construct a sampled subgraph, and the feature data of the second neighbor nodes stored on the remote trainer in the sampled subgraph is pulled to the local trainer.

[0105] Subsequently, the sampled subgraph and the feature data of the first neighbor nodes and the second neighbor nodes can be input into the memory update network layer and the message passing network layer of MTGNN.

[0106] Specifically, after the sampling calculation of the first time batch is completed, the background starts a new thread to prefetch the training events of the next time batch. Negative sample sampling and edge attenuation sampling can be performed by the Central Processing Unit (CPU), and the sampling results (the historical neighbor embedding sequences of nodes) are added to the sampling result waiting queue. The memory updates the network layer to output the latest memory unit, and splices the source point updated memory unit, the destination point updated memory unit, and the edge features of the newly occurred interaction events as the representation of the last interaction (i.e., the last interaction message). These new interaction messages and the updated memory unit can construct a communication index according to the time stamp and the corresponding node index according to the strategy of the largest time stamp, and then submit the task to the remote message aggregation waiting queue.

[0107] Then, the sampled subgraph, the point features and edge features of the relevant neighbor nodes, and the newly calculated memory unit state can be input into the message passing network layer. Before performing the calculation of the message passing network layer on the GPU, the sampling results of the next time batch can be obtained from the sampling result waiting queue. Then, according to the trainer storing the feature data of the neighbor nodes, a communication index is constructed, and an asynchronous remote access communication task is started in the background. At the same time, the calculation operation of the message passing network layer on the GPU is performed. After the forward propagation calculation of the network layer is completed, the backpropagation calculation is performed. The backpropagation directly uses the internal operators of Pytorch and adopts the backpropagation mechanism of the Distributed Data Parallel (DDP) model.

[0108] After the backpropagation ends, the communication tasks with the communication index already constructed are obtained from the remote message aggregation waiting queue, and an asynchronous communication task is started in the background. At the same time, the dynamic subgraph that has completed the remote data access task is obtained from the sampling result waiting queue. Then, the dynamic subgraph and the relevant features and memory unit state are input into the memory update network layer to perform the forward propagation of the memory update network layer. When all batch events are sequentially completed for training, the entire training process is completed, and a trained and complete continuous-time dynamic graph model is obtained.

[0109] Through the above steps S101 to S104, by dividing the continuous-time dynamic graph into multiple dynamic subgraphs, each trainer participating in model training can be made to be only responsible for its corresponding dynamic subgraph; for each trainer, by sampling the first neighbor nodes interacting with the target node in the first dynamic subgraph stored locally, and sampling the second neighbor nodes interacting with the target node in the dynamic subgraphs stored in other trainers according to a preset sampling probability, a sampled subgraph is obtained; finally, an initial continuous-time dynamic graph model is trained based on the sampled subgraph and the feature data of the first neighbor nodes and the second neighbor nodes. In this way, the sampling probability of remote neighbor nodes can be reduced, thereby reducing the communication resources and time costs caused by accessing the feature data of remote neighbor nodes, and thus the parallel training efficiency of the continuous-time dynamic graph model can be improved. In addition, through the edge decay sampling algorithm with hot node awareness, local structural features can be captured, and through random sampling of the neighborhood, the generalization ability of the model is enhanced, and the accuracy is improved at the same time. The model training method based on edge decay sampling provided by the embodiments of the present application avoids overfitting, enhances the generalization ability of the model, thereby improving the accuracy, and at the same time reduces the communication time by more than 5 times compared with the model training methods using other sampling strategies. And, through such a pipeline parallel acceleration distributed graph neural network paradigm, the additional communication attenuation can be hidden in the background thread, thereby avoiding additional communication overhead. Through the present application, the problem of low parallel training efficiency of the continuous-time dynamic graph model in the related art is solved, and the technical effect of improving the parallel training efficiency of the continuous-time dynamic graph model is achieved.

[0110] It should be understood that the magnitudes of the sequence numbers of the above steps in the embodiments do not mean the order of execution is prior or posterior, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0111] Corresponding to the model training method based on edge decay sampling described in the above embodiments, Figure 2 FIG. shows a schematic structural diagram of a model training device based on edge decay sampling according to an embodiment of the present application. The device is applied to train a continuous-time dynamic graph model. For the sake of convenience of description, only the parts related to the embodiments of the present application are shown.

[0112] Please refer to Figure 2, the model training device 2 based on edge attenuation sampling includes: a partitioning module 20 for partitioning a continuous-time dynamic graph into a plurality of dynamic subgraphs, where the plurality of dynamic subgraphs include a first dynamic subgraph; a sampling module 21 for sampling first neighbor nodes interacting with a target node in the first dynamic subgraph and sampling second neighbor nodes interacting with the target node in dynamic subgraphs other than the first dynamic subgraph according to a preset sampling probability to obtain a sampled subgraph; an obtaining module 22 for obtaining feature data of the first neighbor nodes and the second neighbor nodes; and a training module 23 for training an initial continuous-time dynamic graph model based on the sampled subgraph and the feature data of the first neighbor nodes and the second neighbor nodes to obtain a trained complete continuous-time dynamic graph model.

[0113] In one embodiment, the partitioning module 20 is further configured to count the degrees of each node in the continuous-time dynamic graph; select a preset number of nodes as hot nodes in descending order of degrees; delete the hot nodes and the edges related to the hot nodes from the continuous-time dynamic graph to obtain a residual dynamic graph; and partition the residual dynamic graph into a plurality of dynamic subgraphs.

[0114] In one embodiment, the partitioning module 20 is further configured to allocate the plurality of dynamic subgraphs to a plurality of trainers, where each trainer stores one dynamic subgraph; obtain feature data of the hot nodes; store the feature data of the hot nodes in each trainer; allocate the edges between the hot nodes and the non-hot nodes in the residual dynamic graph to the trainer corresponding to the dynamic subgraph containing the non-hot nodes; and allocate the edges between the hot nodes to the plurality of trainers based on a greedy strategy.

[0115] In one embodiment, the sampling module 21 is further configured to sample the first neighbor nodes in the first dynamic subgraph in the order of timestamps of interaction with the target node from earliest to latest based on a preset sampling quantity, and sample the second neighbor nodes in the dynamic subgraphs other than the first dynamic subgraph according to a preset sampling probability; if the sum of the numbers of the sampled first neighbor nodes and second neighbor nodes is less than the sampling quantity, sample the neighbor nodes that interact with the target node and have not been sampled in the order of timestamps of interaction with the target node from earliest to latest until the sum of the numbers of the sampled neighbor nodes is equal to the sampling quantity to obtain a sampled subgraph.

[0116] In one embodiment, the preset sampling probability is expressed as:

[0117]

[0118] where P u,t(v, τ) is a preset sampling probability, θ is the first hyperparameter, δ is the second hyperparameter, τ is the timestamp of interacting with the target node, and t is a preset sampling time period. is a set containing all second neighbor nodes.

[0119] In one embodiment, the training module 23 is further configured to, for each trainer participating in model training, input the sampled subgraph and the feature data of the first neighbor nodes and the second neighbor nodes into the initial continuous-time dynamic graph model corresponding to the trainer through the trainer, and obtain the historical neighbor embedding sequence of each node in the sampled subgraph output by the initial continuous-time dynamic graph model; based on the sampled subgraph, the feature data of the first neighbor nodes and the second neighbor nodes, and the historical neighbor embedding sequence of each node in the sampled subgraph corresponding to each trainer, train the initial continuous-time dynamic graph model through each trainer to obtain a trained complete continuous-time dynamic graph model.

[0120] In one embodiment, the feature data includes the node point features, the edge features of the edges related to the nodes, and the memory unit states of the nodes.

[0121] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units, due to being based on the same concept as the method embodiment of the present application, for their specific functions and the technical effects brought, reference can be specifically made to the method embodiment part, and details are not described herein again.

[0122] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the above-mentioned division of each functional unit and module is used for illustration. In actual applications, the above-mentioned functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiment, and details are not described herein again.

[0123] Figure 3 is a schematic structural diagram of a terminal device according to an embodiment of the present application. As Figure 3 shown, the terminal device 3 includes: at least one processor 30 ( Figure 3Only one) processor, a memory 31, and a computer program 32 stored in the memory 31 and executable on at least one processor 30 are shown. When the processor 30 executes the computer program 32, the steps in any of the above-described method embodiments for training a model based on edge attenuation sampling are implemented.

[0124] The terminal device 3 may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The terminal device 3 may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art can understand that Figure 3 merely examples of the terminal device 3, and do not constitute a limitation on the terminal device 3. It may include more or fewer components than those shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0125] The processor 30 may be a central processing unit (CPU), and the processor 30 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0126] In some embodiments, the memory 31 may be an internal storage unit of the terminal device 3, such as the hard disk or memory of the terminal device 3. In other embodiments, the memory 31 may also be an external storage device of the terminal device 3, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the terminal device 3. In other embodiments, the memory 31 may also include both an internal storage unit and an external storage device of the terminal device 3. The memory 31 is used to store an operating system, application programs, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program 32. The memory 31 may also be used to temporarily store data that has been output or will be output.

[0127] An embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-described embodiments of various model training methods based on edge attenuation sampling can be implemented.

[0128] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal is caused to execute the steps in the above-described embodiments of various model training methods based on edge attenuation sampling.

[0129] The implementation of all or part of the processes in the above-described embodiment methods of the present application can be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can at least include: any entity or device capable of carrying the computer program code to a terminal device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk, or an optical disc, etc.

[0130] In the above embodiments, the descriptions of the various embodiments have their own emphases. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0131] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0132] In the embodiments provided in this application, it should be understood that the disclosed device / network device and method can be implemented in other ways. For example, the device / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in electrical, mechanical or other forms.

[0133] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] The above embodiments are only used to illustrate the technical solutions of this application, rather than to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included in the protection scope of this application.

Claims

1. A model training method based on edge attenuation sampling, characterized in that, Applied to training a continuous-time dynamic graph model, the method includes: Dividing a continuous-time dynamic graph into a plurality of dynamic subgraphs, where the plurality of dynamic subgraphs include a first dynamic subgraph; Sampling first neighbor nodes that interact with a target node in the first dynamic subgraph, and sampling second neighbor nodes that interact with the target node in the dynamic subgraphs other than the first dynamic subgraph according to a preset sampling probability, to obtain a sampled subgraph; Obtaining feature data of the first neighbor nodes and the second neighbor nodes; Training an initial continuous-time dynamic graph model based on the sampled subgraph and the feature data of the first neighbor nodes and the second neighbor nodes to obtain a trained complete continuous-time dynamic graph model.

2. The method according to claim 1, characterized in that, Dividing the continuous-time dynamic graph into a plurality of dynamic subgraphs includes: Counting the degrees of each node in the continuous-time dynamic graph; Selecting a preset number of nodes as hot nodes in descending order of degree; Deleting the hot nodes and the edges related to the hot nodes from the continuous-time dynamic graph to obtain a residual dynamic graph; Dividing the residual dynamic graph into the plurality of dynamic subgraphs.

3. The method according to claim 2, characterized in that After dividing the residual dynamic graph into the plurality of dynamic subgraphs, the method further includes: Allocating the plurality of dynamic subgraphs to a plurality of trainers, where each trainer stores one of the dynamic subgraphs; Obtaining feature data of the hot nodes; Storing the feature data of the hot nodes in each trainer; Allocating the edges between the hot nodes and the non-hot nodes in the residual dynamic graph to the trainer corresponding to the dynamic subgraph containing the non-hot nodes; Allocating the edges between the hot nodes to the plurality of trainers based on a greedy strategy.

4. The method according to any one of claims 1 to 3, characterized in that, Sampling first neighbor nodes that interact with a target node in the first dynamic subgraph, and sampling second neighbor nodes that interact with the target node in the dynamic subgraphs other than the first dynamic subgraph according to a preset sampling probability, to obtain a sampled subgraph includes: Sampling the first neighbor nodes in the first dynamic subgraph in the order of the timestamps of interaction with the target node from earliest to latest based on a preset sampling number, and sampling the second neighbor nodes in the dynamic subgraphs other than the first dynamic subgraph according to the preset sampling probability; If the sum of the numbers of the sampled first neighbor nodes and the second neighbor nodes is less than the sampling number, then sampling the neighbor nodes that interact with the target node and have not been sampled in the order of the timestamps of interaction with the target node from earliest to latest until the sum of the numbers of the sampled neighbor nodes is equal to the sampling number, to obtain the sampled subgraph.

5. The method according to any one of claims 1 to 3, characterized in that The preset sampling probability is expressed as: where P u,t (v,τ) is the preset sampling probability, θ is the first hyperparameter, δ is the second hyperparameter, τ is the timestamp of interacting with the target node, t is the preset sampling time period, is the set containing all the second neighbor nodes.

6. The method according to any one of claims 1 to 3, characterized in that Training an initial continuous-time dynamic graph model based on the sampled subgraph and the feature data of the first neighbor nodes and the second neighbor nodes to obtain a trained complete continuous-time dynamic graph model includes: For each trainer participating in model training, the trainer inputs the sampled subgraph and the feature data of the first neighbor node and the second neighbor node into the corresponding initial continuous-time dynamic graph model, and obtains the historical neighbor embedding sequence of each node in the sampled subgraph output by the initial continuous-time dynamic graph model; Based on the sampled subgraph corresponding to each trainer, the feature data of the first neighbor node and the second neighbor node, and the historical neighbor embedding sequence of each node in the sampled subgraph, each trainer trains the initial continuous-time dynamic graph model to obtain a trained complete continuous-time dynamic graph model.

7. The method according to any one of claims 1 to 3, characterized in that, The feature data includes the node point feature, the edge feature of the edge related to the node, and the memory unit state of the node.

8. A model training device based on edge attenuation sampling, characterized in that, Applied to train a continuous-time dynamic graph model, the device includes: A partitioning module for partitioning a continuous-time dynamic graph into a plurality of dynamic subgraphs, where the plurality of dynamic subgraphs includes a first dynamic subgraph; A sampling module for sampling a first neighbor node interacting with a target node located in the first dynamic subgraph, and sampling a second neighbor node interacting with the target node located in the dynamic subgraph other than the first dynamic subgraph according to a preset sampling probability to obtain a sampled subgraph; An acquisition module for acquiring the feature data of the first neighbor node and the second neighbor node; A training module for training an initial continuous-time dynamic graph model based on the sampled subgraph and the feature data of the first neighbor node and the second neighbor node to obtain a trained complete continuous-time dynamic graph model.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the model training method based on edge decay sampling according to any one of claims 1 to 7.

10. A computer program product, characterized in that, Including a computer program, when the computer program is run, the model training method based on edge decay sampling according to any one of claims 1 to 7 is executed.

Images