Graph neural network training method and device, medium, product and graph processing cluster

Abstract

The invention provides a graph neural network training method and device, a medium, a product and a graph processing cluster, which can be applied to the technical field of artificial intelligence and servers. The graph neural network training method comprises the steps that a plurality of sub-graphs in a graph are distributed to a plurality of graph processing units of a graph processing cluster, so that each graph processing unit carries out an aggregation task on nodes in the sub-graphs distributed to the graph processing unit, and the graph is constructed based on interaction information among a plurality of objects; and according to the node type of the neighbor node used in the aggregation task, distributing the aggregation task to a plurality of thread bundles in the same thread block of the graphic processing unit so as to execute the aggregation task in parallel by utilizing the plurality of thread bundles, the graphic processing unit is used for caching neighbor data of at least one part of neighbor nodes in the sub-graphs distributed to other graphic processing units; and the graph neural network model is trained by using the aggregated data, so that a thread beam level load balancing strategy is realized, and the risk of thread blockage is reduced.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence and servers, and more particularly to a method, device, medium, product, and graphics processing cluster for training a graph neural network. Background Art

[0002] Graph neural networks can effectively model high-order relationships through a neighbor aggregation mechanism, capture potential relationships between objects in a graph, and have good prediction effects. However, the scale of a graph usually reaches hundreds of millions of nodes and billions of edges, posing higher requirements for the computational efficiency and inference latency of the system for training graph neural networks.

[0003] During the training process of graph neural networks, the system usually has problems of load imbalance and thread blockage. Summary of the Invention

[0004] In view of the above problems, the present invention provides a method, device, medium, product, and graphics processing cluster for training a graph neural network.

[0005] According to a first aspect of the present invention, there is provided a method for training a graph neural network, including: allocating a plurality of subgraphs in a graph to a plurality of graphics processing units of a graphics processing cluster, so that each graphics processing unit performs an aggregation task on nodes in the subgraph allocated to the graphics processing unit, where the graph is constructed based on interaction information between a plurality of objects; according to the node types of neighbor nodes used in the aggregation task, allocating the aggregation task to a plurality of warps in the same thread block of the graphics processing unit, so as to use the plurality of warps to execute the aggregation task in parallel, where the graphics processing unit is used to cache neighbor data of at least a part of neighbor nodes in the subgraphs allocated to other graphics processing units, and the aggregation task includes aggregating the data of each node in the subgraph with the neighbor data of the neighbor nodes of the node to obtain aggregation data; and training a graph neural network model by using the aggregation data.

[0006] The second aspect of the present invention provides a graph neural network training device, including: a first allocation module, configured to allocate multiple subgraphs in a graph to multiple graphics processing units of a graphics processing cluster, so that each graphics processing unit performs an aggregation task on nodes in the subgraph allocated to this graphics processing unit, where the graph is constructed based on interaction information between multiple objects; a second allocation module, configured to allocate the aggregation task to multiple warps in the same thread block of the graphics processing unit according to the node types of neighbor nodes used in the aggregation task, so as to utilize the multiple warps to execute the aggregation task in parallel, where the graphics processing unit is used to cache neighbor data of at least a part of neighbor nodes in the subgraphs allocated to other graphics processing units, and the aggregation task includes aggregating the data of each node in the subgraph with the neighbor data of the neighbor nodes of this node to obtain aggregated data; a training module, configured to train a graph neural network model by using the aggregated data.

[0007] The third aspect of the present invention provides a graphics processing cluster, including: multiple graphics processing units, configured to execute an aggregation task corresponding to a subgraph in a graph allocated to this graphics processing unit, where the aggregation task includes aggregating the data of each node in the subgraph with the neighbor data of the neighbor nodes of this node to obtain aggregated data, and the aggregation task is allocated based on the following method: allocating the aggregation task to multiple warps in the same thread block of the graphics processing unit according to the node types of neighbor nodes used in the aggregation task; the graphics processing unit is used to cache neighbor data of at least a part of neighbor nodes in the subgraphs allocated to other graphics processing units.

[0008] The fourth aspect of the present invention provides an electronic device, including: one or more processors; a memory, configured to store one or more computer programs, where the above one or more processors execute the above one or more computer programs to implement the steps of the above method.

[0009] The fifth aspect of the present invention further provides a computer-readable storage medium, on which a computer program or instruction is stored, and when the above computer program or instruction is executed by a processor, the steps of the above method are implemented.

[0010] The sixth aspect of the present invention further provides a computer program product, including a computer program or instruction, and when the above computer program or instruction is executed by a processor, the steps of the above method are implemented.

[0011] According to an embodiment of the present invention, by allocating multiple sub - graphs in a graph to multiple graphics processing units of a graphics processing cluster, each graphics processing unit performs an aggregation task on the nodes in the sub - graph allocated to it. According to the node types of the neighbor nodes used in the aggregation task, the aggregation task is allocated to multiple warps in the same thread block of the graphics processing unit to utilize the parallel execution of multiple warps for the aggregation task. Allocating the aggregation task of nodes to multiple warps in the same thread block realizes a load - balancing strategy at the warp level, reduces the risk of thread blockage. At the same time, multiple warps in the thread block can use shared video memory, and the consumption of synchronizing data is smaller. The graphics processing unit is used to cache the neighbor data of at least a part of the neighbor nodes in the sub - graphs allocated to other graphics processing units, reducing the remote communication across graphics processing units during the training process of the graph neural network model. Brief Description of the Drawings

[0012] Through the following description of the embodiments of the present invention with reference to the drawings, the above - mentioned content and other objects, features, and advantages of the present invention will become clearer. In the drawings:

[0013] Figure 1 The architecture diagram of a graphics processing cluster according to an embodiment of the present invention is shown.

[0014] Figure 2 The flowchart of a graph neural network training method according to an embodiment of the present invention is shown.

[0015] Figure 3 The schematic diagram of neighbor nodes of each node in multiple sub - graphs in a graph according to an embodiment of the present invention is shown.

[0016] Figure 4 The structural block diagram of a graph neural network training device according to an embodiment of the present invention is shown.

[0017] Figure 5 The block diagram of an electronic device suitable for implementing the graph neural network training method according to an embodiment of the present invention is shown. Detailed Description of the Embodiments

[0018] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In the following detailed description, for the sake of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present invention. However, obviously, one or more embodiments can also be implemented without these specific details. In addition, in the following description, descriptions of well - known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present invention.

[0019] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the present invention. The terms "comprising", "including" and the like as used herein indicate the presence of the stated features, steps, operations and / or components, but do not preclude the presence or addition of one or more other features, steps, operations or components.

[0020] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0021] In the case of using expressions such as "at least one of A, B, and C, etc.", generally it should be interpreted according to the meaning commonly understood by those of ordinary skill in the art (for example, "a system having at least one of A, B, and C" should include, but is not limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0022] In the related art, during the training of a graph neural network model, remote communication is required before each layer of calculation to obtain the features or embeddings on the neighbor nodes of the nodes. In the backpropagation stage, if the embeddings themselves are trainable, the features of the neighbor nodes also need to be transmitted. Generally, the graph neural network model is relatively small while the graph data itself is large, so it is difficult to apply distributed training optimization methods to the graph neural network model.

[0023] The following problems exist during the training of the graph neural network model: the parallelism during training is poor, and communication must be completed before calculation; the transmission of neighbor data across graphics processing units takes a large amount of time; the embeddings of historical neighbor data (i.e., the embeddings of neighbor data generated in historical iteration rounds) are not fully utilized, resulting in the inability to effectively reduce the communication frequency and data volume.

[0024] Embodiments of the present invention provide a method for training a graph neural network, including: allocating a plurality of subgraphs in a graph to a plurality of graphics processing units of a graphics processing cluster, so that each graphics processing unit performs an aggregation task on the nodes in the subgraph allocated to this graphics processing unit, where the graph is constructed based on the interaction information between a plurality of objects; according to the node types of the neighbor nodes used in the aggregation task, allocating the aggregation task to a plurality of warps in the same thread block of the graphics processing unit, so as to use the plurality of warps to execute the aggregation task in parallel, where the graphics processing unit is used to cache at least a part of the neighbor data of the neighbor nodes in the subgraphs allocated to other graphics processing units, and the aggregation task includes aggregating the data of each node in the subgraph with the neighbor data of the neighbor nodes of this node to obtain aggregated data; and training the graph neural network model using the aggregated data.

[0025] Figure 1 Shows an architecture diagram of a graphics processing cluster according to an embodiment of the present invention.

[0026] As Figure 1 shown, the graphics processing cluster 100 according to this embodiment may include multiple graphics processing units, such as graphics processing unit 110, graphics processing unit 120, etc. During the training process of the graph neural network, multiple subgraphs in the graph can be allocated to the graphics processing unit 110, graphics processing unit 120, etc. The graphics processing unit can be used to execute the aggregation task corresponding to the subgraph in the graph allocated to this graphics processing unit. The aggregation task may include aggregating the data of each node in the subgraph with the neighbor data of the neighbors of this node to obtain aggregated data.

[0027] The graphics processing unit may include at least one thread block. The graphics processing unit 110 may have thread blocks 111, 112, etc. The graphics processing unit 120 may have thread blocks 121, 122, etc. Each graphics processing unit in the graphics processing cluster can schedule each thread block to execute the aggregation task corresponding to the subgraph.

[0028] The thread block may include at least one warp. In the graphics processing unit 110, the thread block 111 may have warps 1111, 1112, etc., and the thread block 112 may have warps 1121, 1122, etc. In the graphics processing unit 120, the thread block 121 may have warps 1211, 1212, etc., and the thread block 122 may have warps 1221, 1222, etc.

[0029] The warp may be the smallest unit scheduled by the execution engine of the graphics processing unit. There are usually 32 threads in a warp. Multiple warps within the same thread block can share memory and also support synchronous data operations.

[0030] It should be noted that the graph neural network training method provided by the embodiments of the present invention can generally be executed by the graphics processing cluster 100. Correspondingly, the graph neural network training device provided by the embodiments of the present invention can generally be set in the graphics processing cluster 100. [[ID=…]]

[0031] It should be understood that Figure 1 the numbers of the graphics processing units, thread blocks, and warps in

[0032] Figure 2 Shows a flowchart of the graph neural network training method according to an embodiment of the present invention.

[0033] As Figure 2As shown, the graph neural network training method of this embodiment includes operations S210 to S230, and this graph neural network training method can be executed by a graphics processing cluster.

[0034] In operation S210, multiple subgraphs in the graph are assigned to multiple graphics processing units of the graphics processing cluster, so that each graphics processing unit performs an aggregation task on the nodes in the subgraph assigned to this graphics processing unit.

[0035] According to an embodiment of the present invention, the graph is constructed based on the interaction information between multiple objects. The graph includes at least one node, and the node can represent the relevant information of the object. The interaction information can be the behavior between objects. For example, the multiple objects can be shoppers and goods, and the interaction information can be behavior information such as collection, adding to the shopping cart, and purchase. For example, the multiple objects can be readers and articles, and the interaction information can be behavior information such as click, read, and comment.

[0036] According to an embodiment of the present invention, the multiple subgraphs can be assigned to multiple graphics processing units of the graphics processing cluster according to a preset order, the similarity between nodes, etc. For example, the multiple subgraphs are sorted according to the position of the subgraphs in the graph from left to right or from top to bottom to obtain a preset order. For example, the similarity of multiple nodes in the subgraph is calculated. According to the magnitude of the similarity, the multiple subgraphs are sequentially assigned to multiple graphics processing units. It should be noted that the step of assigning the multiple subgraphs to multiple graphics processing units of the graphics processing cluster is not limited, and other assignment methods can also be used.

[0037] Figure 3 Shows a schematic diagram of the neighbor nodes of each node in multiple subgraphs in the graph according to an embodiment of the present invention.

[0038] As Figure 3 shown, there can be node 1, node 2, node 3, node 4, node 5, and node 6 in the graph. Node 1 is connected to node 2 and node 6 by edges respectively. The first subgraph in the graph includes node 1 and node 6, the second subgraph in the graph includes node 2 and node 5, and the third subgraph in the graph includes node 3 and node 4.

[0039] The neighbor nodes of node 1 in the first subgraph in the graph are node 2 and node 6 respectively. The neighbor nodes of node 6 in the first subgraph in the graph are node 1, node 4, and node 5 respectively.

[0040] The neighbor nodes of node 2 in the second subgraph are node 1, node 3, and node 5 respectively. The neighbor nodes of node 5 in the second subgraph are node 2, node 3, node 4, and node 6 respectively.

[0041] The neighbor nodes of node 3 in the third subgraph are node 2, node 5, and node 4 respectively. The neighbor nodes of node 4 in the third subgraph are node 3, node 5, and node 6 respectively.

[0042] The graphics processing unit assigned to the first sub - graph performs an aggregation task on nodes 1 and 6. The graphics processing unit assigned to the second sub - graph performs an aggregation task on nodes 2 and 5. The graphics processing unit assigned to the third sub - graph performs an aggregation task on nodes 3 and 2.

[0043] According to an embodiment of the present invention, the aggregation task includes aggregating the data of each node in the sub - graph with the neighbor data of the neighbor nodes of the node to obtain aggregated data. For example, the data of node 1 in the first sub - graph is respectively aggregated with the neighbor data of node 2 and the neighbor data of node 6. The data of node 6 in the first sub - graph is respectively aggregated with the neighbor data of node 1, the neighbor data of node 4, and the neighbor data of node 5.

[0044] In operation S220, according to the node types of the neighbor nodes used in the aggregation task, the aggregation task is assigned to multiple warps in the same thread block of the graphics processing unit to utilize the parallel execution of multiple warps for the aggregation task. The graphics processing unit is used to cache the neighbor data of at least a part of the neighbor nodes in the sub - graph assigned to other graphics processing units.

[0045] According to an embodiment of the present invention, the node type of the neighbor node can be determined according to the data source of the neighbor data. For example, the node type can include: a node with neighbor data obtained from the graphics processing unit where the node is located, and a node with neighbor data obtained from other graphics processing units.

[0046] For example, determine multiple neighbor nodes used in the aggregation task, and respectively assign the aggregation tasks corresponding to the multiple neighbor nodes to different thread blocks. For example, the neighbor nodes of node 1 in the first sub - graph are node 2 and node 6 respectively. The aggregation tasks corresponding to node 1 (the aggregation between the data of node 1 and the neighbor data of node 2, and the aggregation between the data of node 1 and the neighbor data of node 6) can be assigned to the same thread block.

[0047] For example, according to the node types of the neighbor nodes, assign the aggregation tasks corresponding to different node types to multiple warps in the same thread block.

[0048] For example, estimate the execution duration of the aggregation task according to the node types of the neighbor nodes. According to the current task execution information of multiple warps and the execution duration of the aggregation task, assign the aggregation task to multiple warps in the same thread block of the graphics processing unit. Therefore, it is possible to avoid the situation where congested warps are re - assigned aggregation tasks and the situation where idle warps have a small amount of aggregation tasks, and achieve load balancing at the warp level.

[0049] In operation S230, the graph neural network model is trained using the aggregated data.

[0050] According to an embodiment of the present invention, aggregation may be a process in which a node updates its own data by integrating neighbor data. The graph neural network model can utilize the aggregated data to capture complex information in the graph structure (such as potential association relationships between objects), which can improve the prediction accuracy.

[0051] According to an embodiment of the present invention, the training method of the graph neural network model using the aggregated data may be supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, etc.

[0052] According to an embodiment of the present invention, the trained graph neural network model is deployed in an image processing cluster to process the to-be-predicted interaction information between multiple objects.

[0053] According to an embodiment of the present invention, by allocating multiple subgraphs in the graph to multiple graphics processing units of the graphics processing cluster, each graphics processing unit performs an aggregation task on the nodes in the subgraph allocated to it. According to the node types of the neighbor nodes used in the aggregation task, the aggregation task is allocated to multiple warps in the same thread block of the graphics processing unit, so as to use multiple warps to execute the aggregation task in parallel. Allocating the aggregation task of the nodes to multiple warps in the same thread block realizes a load balancing strategy at the warp level, reduces the risk of thread blocking, and at the same time, multiple warps in the thread block can use shared video memory, and the consumption of synchronizing data is smaller. The graphics processing unit is used to cache the neighbor data of at least a part of the neighbor nodes in the subgraph allocated to other graphics processing units, reducing the remote communication across graphics processing units during the training process of the graph neural network model.

[0054] According to an embodiment of the present invention, the node types of the neighbor nodes include at least one of the following: a remote node that needs remote access, a remote node that uses historical cache, and a local node. The remote node that needs remote access is a node for which the graphics processing unit needs to obtain neighbor data via remote access. The remote node that uses historical cache is a node for which the graphics processing unit caches historical neighbor data obtained via historical remote access. The local node is a node for which the graphics processing unit caches neighbor data.

[0055] As Figure 3 shown, the neighbor nodes of node 1 in the first subgraph are node 2 and node 6 respectively. Node 6 is the same as the graphics processing unit where node 1 is located, and node 6 is a local node. Node 2 is different from the graphics processing unit where node 1 is located, and node 2 may be a remote node that needs remote access or a remote node that uses historical cache.

[0056] The graphics processing unit where Node 1 is located caches historical neighbor data obtained by remotely accessing the graphics processing unit where Node 2 was located in the past. Node 2 is a remote node that uses historical caching. The historical neighbor data can be the embedded representation of the neighbor data generated in the previous several iterations.

[0057] The graphics processing unit where Node 1 is located does not cache the historical neighbor data of Node 2. Node 2 is a remote node that needs to be remotely accessed.

[0058] According to an embodiment of the present invention, the aggregation task corresponding to the remote node that needs to be remotely accessed can be divided into a communication stage and an aggregation stage. The communication stage can include communicating the graphics processing unit with the graphics processing unit where the remote node that needs to be remotely accessed is located to obtain neighbor data. The aggregation stage can include performing an aggregation operation on the neighbor data and the data of the node. Since remote nodes that use historical caching are set, the frequent remote data transmission process can be greatly reduced. The number of remote nodes and local nodes that use historical caching in the neighbor nodes accounts for the majority, so the pipeline training of the graph neural network training method becomes possible.

[0059] According to an embodiment of the present invention, by setting remote nodes that use historical caching, the frequent remote data transmission process can be reduced. At the same time, the introduction of historical caching makes the pipeline training of the graph neural network training method possible.

[0060] According to an embodiment of the present invention, according to the node types of the neighbor nodes used in the aggregation task, the aggregation task is assigned to multiple warps in the same thread block of the graphics processing unit, including: dividing multiple neighbor nodes into neighbor node sets of different node types; according to the number of neighbor nodes in the neighbor node set and the number of warps, dividing the neighbor node set to obtain neighbor node subsets corresponding to different warps; according to a preset interleaving distance, interleaving the aggregation subtasks associated with the neighbor node subsets with each other and assigning them to multiple warps. The aggregation task includes aggregation subtasks corresponding to multiple neighbor nodes of the node.

[0061] According to an embodiment of the present invention, according to the node type, multiple neighbor nodes are divided into neighbor node sets of different node types. In the neighbor node set, each node and its neighbor nodes are assigned to the same thread block, and the shared video memory in the thread block can be used to obtain neighbor data, reducing the loss of data synchronization.

[0062] According to an embodiment of the present invention, for neighbor node sets of the same type, according to the number of neighbor nodes in the neighbor node set and the number of warps, the neighbor nodes are evenly distributed to different warps.

[0063] For example, there are n warps in a thread block, the number of remote nodes to be remotely accessed is x1, the number of remote nodes using the history cache is x2, and the number of local nodes is x3. Each warp processes the aggregation subtasks corresponding to x1 / n remote nodes to be remotely accessed, the aggregation subtasks corresponding to x2 / n remote nodes using the history cache, and the aggregation subtasks corresponding to x3 / n local nodes.

[0064] Therefore, in the case of a remote access blockage occurring in a warp in the graphics processing unit, other warps can be scheduled to the graphics processing unit where the remote access blockage occurs.

[0065] According to an embodiment of the present invention, the preset interleaving distance may be to alternately allocate the aggregation subtasks corresponding to the remote nodes to be remotely accessed, the remote nodes using the history cache, and the local nodes to the same warp at certain intervals. Therefore, it is possible to avoid the situation where the graphics processing unit only processes the aggregation subtasks corresponding to the remote nodes to be remotely accessed and processes slowly, while some other graphics processing units only process the aggregation subtasks corresponding to the local nodes and are idle, making the task allocation of the graphics processing cluster balanced.

[0066] According to an embodiment of the present invention, allocating the aggregation subtasks associated with the subset of neighbor nodes to multiple warps in an interleaved manner according to the preset interleaving distance includes: sorting the aggregation subtasks associated with the subset of neighbor nodes according to the preset interleaving distance to obtain a task order; and allocating the aggregation subtasks associated with the subset of neighbor nodes to the warps according to the task order.

[0067] For example, the preset interleaving distance may include a first interleaving distance. The first interleaving distance may be a cyclic sorting of the aggregation subtasks corresponding to 2 local nodes and the aggregation subtasks corresponding to 1 remote node to be remotely accessed. Therefore, each warp alternately processes the aggregation subtasks corresponding to 2 local nodes and 1 remote node to be remotely accessed.

[0068] For example, the preset interleaving distance may include a second interleaving distance. The second interleaving distance may be a cyclic sorting of the aggregation subtasks corresponding to 2 local nodes and the aggregation subtasks corresponding to 1 remote node using the history cache. Therefore, each warp alternately processes the aggregation subtasks corresponding to 2 local nodes and the aggregation subtasks corresponding to 1 remote node using the history cache.

[0069] According to an embodiment of the present invention, by sorting the aggregated subtasks associated with a subset of neighbor nodes according to a preset interleaving distance to obtain a task order, each warp can alternately execute the aggregated subtasks corresponding to different node types, balance the execution time of each warp, reduce the risk of thread blockage and remote access congestion, and improve the resource utilization rate and throughput of the graphics processing cluster. According to an embodiment of the present invention, the number of aggregated subtasks of local nodes executed on a warp is respectively greater than the number of aggregated subtasks of remote nodes that need to be remotely accessed and the number of aggregated subtasks of remote nodes using historical caches.

[0070] [[ID=�]]Since the aggregated subtasks of local nodes are processed faster, and the number of aggregated subtasks of local nodes on the same warp is respectively greater than the number of aggregated subtasks of remote nodes that need to be remotely accessed and the number of aggregated subtasks of remote nodes using historical caches, it is possible to reduce the time that some graphics processing units wait for neighbor data obtained by remote access for a long time, balance the execution time of each graphics processing unit, improve the processing efficiency of the graphics processing cluster, and reduce the time during the training process of the graph neural network model.

[0071] According to an embodiment of the present invention, the aggregated subtasks corresponding to local nodes include: aggregating the neighbor data stored in the graphics processing unit and the data of the node; the aggregated subtasks corresponding to remote nodes that need to be remotely accessed include: remotely communicating the graphics processing unit with the graphics processing unit where the neighbor node is located, and aggregating the neighbor data and the data of the node obtained via remote access; the aggregated subtasks corresponding to remote nodes using historical caches include: reading the neighbor data cached in the graphics processing unit, and aggregating the cached neighbor data and the data of the node.

[0072] For example, a cyclic sorting of the aggregated subtasks corresponding to 2 local nodes, 1 remote node that needs to be remotely accessed, and 1 remote node using historical caches on each warp.

[0073] Due to the setting of remote nodes using historical caches, the frequent remote data transmission process can be greatly reduced. The number of remote nodes using historical caches and local nodes among neighbor nodes accounts for the majority, and the aggregated subtasks corresponding to remote nodes that need to be remotely accessed account for a small part in the aggregated tasks. The communication time consumption on each warp is relatively small compared to the total duration of the aggregated tasks, which enables the graph neural network training method to achieve pipeline training.

[0074] According to an embodiment of the present invention, the above method further includes: remotely accessing the neighbor graphics processing unit where the neighbor node is located in the graphics processing cluster to obtain neighbor data when the historical cache of the graphics processing unit does not include the historical neighbor data of the neighbor node; and determining the neighbor data according to preset update information when the historical cache of the graphics processing unit includes the historical neighbor data.

[0075] According to an embodiment of the present invention, the data of nodes in the graphics processing unit and the neighbor data of neighbor nodes are updated regularly, and the historical neighbor data is stored in the local memory. When a graphics processing unit needs to remotely access other graphics processing units, it is determined whether the historical cache of the graphics processing unit includes the historical neighbor data of the neighbor nodes.

[0076] When the historical cache of the graphics processing unit does not include the historical neighbor data of the neighbor nodes, remotely access the graphics processing unit where the neighbor node is located in the graphics processing cluster, and cache the neighbor data locally.

[0077] When the historical cache of the graphics processing unit includes historical neighbor data, it is determined whether the historical neighbor data is valid according to the preset update information. If it is valid, the historical neighbor data is determined as the neighbor data. If it is invalid, it is necessary to remotely access the neighbor graphics processing unit where the neighbor node is located in the graphics processing cluster again to obtain the neighbor data.

[0078] According to an embodiment of the present invention, the preset update information may include parameter data during the training process of the graph neural network model, relevant information of historical neighbor data, etc. For example, the parameter data may be the change value of the model parameters.

[0079] According to an embodiment of the present invention, by determining the neighbor data according to the preset update information when the historical cache of the graphics processing unit includes historical neighbor data, the validity of the historical neighbor data can be verified, and the accuracy of the aggregation operation can be improved.

[0080] According to an embodiment of the present invention, the preset update information includes the time information of the historical neighbor data; determining the neighbor data according to the preset update information includes: when the time information of the historical neighbor data is within the preset time period, obtaining the neighbor data from the graphics processing unit where the neighbor node is located; when the time information of the historical neighbor data is not within the preset time period, determining the historical neighbor data as the neighbor data.

[0081] According to an embodiment of the present invention, the preset time period may be a preset duration earlier than the current time.

[0082] For example, the preset duration earlier than the current time may be 20 minutes earlier than the current time. The time information of the historical neighbor data is 20 minutes earlier than the current time, indicating that the historical neighbor data is old, and it is triggered to re-obtain the neighbor data from the graphics processing unit where the neighbor node is located.

[0083] The time information of the historical neighbor data is not 20 minutes earlier than the current time, indicating that the historical neighbor data is valid, and the historical neighbor data can be determined as the neighbor data.

[0084] According to an embodiment of the present invention, by obtaining neighbor data from the graphics processing unit where the neighbor node is located when the time information of the historical neighbor data is within a preset time period, the accuracy of the aggregation operation can be ensured; when the time information of the historical neighbor data is not within the preset time period, the historical neighbor data is determined as neighbor data, which not only ensures the accuracy of the aggregation operation, but also reduces the data transmission process and improves the training speed of the graph neural network model.

[0085] According to an embodiment of the present invention, the preset update information includes the gradient change value of the model parameters of the graph neural network model; determining the neighbor data of the neighbor node according to the preset update information includes: when the gradient change value is within a preset change range, remotely accessing the graphics processing unit where the neighbor node is located to obtain neighbor data; when the gradient change value of the historical neighbor data is not within the preset change range, the historical neighbor data is determined as neighbor data.

[0086] According to an embodiment of the present invention, the preset change range can be determined according to the actual training situation of the graph neural network model.

[0087] For example, when the gradient change value is within the preset change range, it indicates that the training process of the graph neural network model changes greatly, and the cached historical neighbor data is invalid, so neighbor data is obtained from the graphics processing unit where the neighbor node is located.

[0088] For example, when the gradient change value is not within the preset change range, it indicates that the training process of the graph neural network model changes little, and the cached historical neighbor data is valid, so the historical neighbor data is determined as neighbor data.

[0089] According to an embodiment of the present invention, by obtaining neighbor data from the graphics processing unit where the neighbor node is located when the gradient change value is within the preset change range, the risk of low training accuracy caused by still using the historical neighbor data to perform the aggregation task when the training process of the graph neural network model changes greatly can be reduced.

[0090] According to an embodiment of the present invention, the above method further includes: when the bus bandwidth occupied by the graphics processing unit in multiple consecutive time windows is greater than a preset threshold, reducing the rate at which the graphics processing unit sends remote access requests to the bus.

[0091] According to an embodiment of the present invention, when the bus bandwidth occupied by the graphics processing unit in multiple consecutive time windows is greater than a preset threshold, it is difficult for other graphics processing units to meet the rate of the minimum remote access request, so the rate at which the graphics processing unit sends remote access requests to the bus is reduced.

[0092] According to an embodiment of the present invention, for communication or memory access time occupation, CXL (Compute Express Link) access scheduling is implemented using runtime parameters. The CXL remote access requests at the warp level are batched, aligned, and merged, thereby reducing fragmented access.

[0093] According to an embodiment of the present invention, the preset threshold may be 80%. If the graphics processing unit A occupies more than 80% of the bus bandwidth in multiple consecutive windows, and the graphics processing unit B cannot meet the rate of the minimum remote access request, then the packet sending rate of the graphics processing unit A is restricted.

[0094] According to an embodiment of the present invention, by reducing the rate at which the graphics processing unit sends remote access requests to the bus when the bus bandwidth occupied by the graphics processing unit in multiple consecutive time windows is greater than the preset threshold, the number of remote accesses between each graphics processing unit can be effectively balanced, and the congestion in the remote access process can be reduced.

[0095] According to an embodiment of the present invention, the memory addresses of multiple graphics processing units in the graphics processing cluster belong to the same address range. When the cached data in the graphics processing unit is updated, an invalidation message of the cached data is sent to other graphics processing units in the graphics processing cluster.

[0096] The CXL protocol allows multiple graphics processing units to exchange data through shared memory. The memory addresses of multiple graphics processing units in the graphics processing cluster belong to the same address range, enabling each graphics processing unit to directly access the memory of other remote graphics processing units, thereby reducing the need for remote data transmission across graphics processing units.

[0097] Through the CXL protocol, the communication bandwidth between graphics processing units is significantly improved, and the latency is also greatly reduced during data transmission. For the training task of a graph neural network model that requires rapid graph data exchange, the CXL protocol greatly improves the communication efficiency.

[0098] In distributed training, the communication mechanism of the CXL protocol can dynamically allocate computing tasks and data transmission tasks according to the real-time load conditions of the graphics processing units, avoiding congestion in remote access.

[0099] The local coalescing factor is an indicator of the memory access efficiency of the graphics processing unit. Local coalescing factor (coalescing efficiency) = actual access bandwidth / number of remote requests × memory access size. Allocate CXL memory according to the occupancy rate of the current graphics processing unit, and record the CXL traffic (MB / s), number of accesses, hit rate, etc. of each graphics processing unit in real time. By optimizing the data layout and access pattern, merging the requests of multiple warps into a few efficient requests can significantly improve the memory bandwidth utilization of the graphics processing unit and accelerate the deep learning training and inference processes.

[0100] Meanwhile, when the cached data in the graphics processing unit is updated (the update form can be deletion, modification, addition, etc.), send the update information of the cached data (such as the update information can include the invalidation of the current cached data, the updated cached data, etc.) to other graphics processing units in the graphics processing cluster, so that multiple graphics processing units in the graphics processing cluster can achieve memory sharing.

[0101] According to an embodiment of the present invention, the above method further includes: when there are multiple nodes in the subgraph, matching the neighbor nodes of the multiple nodes to obtain a matching result; when the matching result indicates that there are identical nodes among the neighbor nodes, merging the remote access requests of the identical nodes.

[0102] As Figure 3 shown, the neighbor nodes of node 3 in the third subgraph are node 2, node 5, and node 4 respectively. The neighbor nodes of node 4 in the third subgraph are node 3, node 5, and node 6 respectively. Matching the neighbor nodes of node 3 with the neighbor nodes of node 4, node 5 is the identical node. Merging the remote access requests of the two node 5s into one remote access request to reduce the number of repeated accesses.

[0103] According to an embodiment of the present invention, by merging the remote access requests of the identical nodes among the neighbor nodes of multiple nodes, reducing the number of repeated accesses can improve the communication efficiency in the graph model training process and enhance the operation efficiency of the overall graphics processing cluster.

[0104] During the training process of the graph neural network model, the graphics processing unit can cache the neighbor data of the remote nodes with the usage history cache in the subgraph allocated to other graphics processing units, reducing the waiting time. The training process is carried out on the training set and its accuracy is tested on the test set.

[0105] Graph neural network models usually face problems such as high-frequency embedding access, dense neighbor aggregation, and cross-thread synchronization bottlenecks during training. Therefore, tasks should be localized on each graphics processing unit as much as possible to avoid cross-node access. In-kernel computing will preferentially use neighbor data cached locally (such as the global memory and shared memory of the graphics processing unit), and utilize the parallel computing power of the accelerator for efficient processing.

[0106] According to an embodiment of the present invention, during the aggregation calculation process, it is necessary to obtain neighbor data on the neighbor nodes of each node and aggregate the node data and neighbor data of the node. Therefore, a large amount of neighbor data needs to be loaded into the video memory during the calculation. When extended to a graphics processing cluster, the subgraphs are split into each graphics processing unit, and neighbor data needs to be obtained remotely; the more nodes there are in the subgraph used, the more neighbor nodes need to communicate, and the communication volume is greater. Therefore, the scalability of the graph neural network model is poor. The communication volume can be expressed as the product of the embedding dimension and the number of boundary nodes, that is , k is the embedding dimension, N k is the number of boundary nodes. The occupancy rate is expressed as the ratio of the calculation time T to the global time. For a graph with a fixed scale, its calculation time is basically stable, but the pauses caused by memory access and communication will reduce the occupancy rate of the graphics processing unit. Therefore, solving the scalability problem can also greatly improve the utilization rate of the graphics processing unit.

[0107] The aggregation task can also be divided into aggregation subtasks associated with subsets of neighbor nodes, and the aggregation subtasks are interleaved and assigned to multiple warps according to a preset interleaving distance, so as to achieve parallel execution of multiple aggregation subtasks on multiple warps.

[0108] Most of the remote nodes and local nodes in the neighbor nodes use historical caches. Therefore, the graph neural network training method can be trained in a pipeline. The pipeline scheduling strategy aims to reduce the waiting time during the distributed training process and improve the system throughput efficiency by optimizing the execution order of computing and communication tasks. If the characteristics of alternating execution of communication and computing are not properly scheduled, it will seriously affect the utilization rate of the graphics processing unit and the end-to-end delay. The aggregation subtasks corresponding to the remote nodes that need to be remotely accessed, the remote nodes that use historical caches, and the local nodes are alternately assigned to the same warp at a certain interval to ensure that the communication across graphics processing units is minimized. Therefore, the utilization rate of the computing unit can be maximized, thereby accelerating the computing tasks within each graphics processing unit.

[0109] By reasonably configuring the local cache, the memory access latency can be reduced. Especially during graph convolution and embedding aggregation, the node embeddings and the data of neighbor nodes are frequently accessed, and the optimization of the cache can significantly improve the computing speed.

[0110] According to an embodiment of the present invention, the above method further includes: performing an aggregation operation on neighbor data and node data based on a smoothing algorithm.

[0111] According to an embodiment of the present invention, the smoothing algorithm may be a smoothing algorithm based on weighted average, and the formula is as follows:

[0112] (1)

[0113] represents the data of the node in the (t + 1)-th round of iteration, the neighbor data of the neighbor nodes in the t-th round and the data of the node in the t-th round , where α is a smoothing factor.

[0114] According to an embodiment of the present invention, the above method further includes: partitioning the structure of the graph based on a clustering algorithm to obtain multiple subgraphs.

[0115] According to an embodiment of the present invention, the clustering algorithm may be the metis algorithm. By clustering the graph through the metis algorithm, the graph is divided into multiple subgraphs with approximately the same size, greatly reducing the number of boundary nodes and minimizing the communication cost while ensuring computational balance.

[0116] According to an embodiment of the present invention, the graph includes user nodes, advertisement nodes, and edges between user nodes and advertisement nodes. The weight of the edge is determined according to the type of the interaction behavior of the user with the advertisement, and the multiple objects include users and advertisements.

[0117] According to an embodiment of the present invention, the interaction behavior of the user with the advertisement may be click, exposure, bounce, favorite, etc. The graph may be a dynamic user - advertisement bipartite graph. Heterogeneous edges are constructed according to the type of the interaction behavior, and the edge type can affect the aggregation weight of the subsequent graph neural network model. Weights are set for each interaction behavior, such as the weight of click being higher than that of exposure. Nodes may carry static features (such as advertisement content classification, user information).

[0118] The graph supports periodically adding new users, new advertisement nodes, and their interaction edges incrementally to meet the needs of online training / inference. The connectivity of the graph can be enhanced by introducing auxiliary relationships such as the brand and category to which the advertisement belongs, so that new advertisement nodes can be better connected to the graph.

[0119] According to an embodiment of the present invention, the edge includes at least one of the following information: the interaction time information of the user with the advertisement, the number of times of the interaction behavior.

[0120] According to an embodiment of the present invention, before obtaining the relevant information of the user, the consent or authorization of the user can be obtained.

[0121] The interaction time information of the user with the advertisement can be the timestamp of the interaction behavior. The number information of the interaction behavior can be the number of interactions per hour.

[0122] According to an embodiment of the present invention, a graphics processing cluster includes: a plurality of graphics processing units for performing an aggregation task corresponding to a sub-graph in a graph assigned to the graphics processing unit, wherein the aggregation task includes aggregating the data of each node in the sub-graph with the neighbor data of the neighbor nodes of the node to obtain aggregated data, and the aggregation task is assigned based on the following method: according to the node type of the neighbor nodes used in the aggregation task, the aggregation task is assigned to a plurality of warps in the same thread block of the graphics processing unit; the graphics processing unit can cache the neighbor data of at least a part of the neighbor nodes in the sub-graph assigned to other graphics processing units.

[0123] According to an embodiment of the present invention, the node type of the neighbor nodes includes at least one of the following: a remote node that needs to be remotely accessed, a remote node that uses a historical cache, and a local node. The remote node that needs to be remotely accessed is a node for which the graphics processing unit needs to obtain neighbor data via remote access. The remote node that uses a historical cache is a node for which the graphics processing unit caches historical neighbor data obtained via historical remote access. The local node is a node for which the graphics processing unit caches neighbor data.

[0124] According to an embodiment of the present invention, the plurality of graphics processing units are communicatively connected. In the case where the bus bandwidth occupied by the graphics processing unit in a plurality of consecutive time windows is greater than a preset threshold, the rate at which the graphics processing unit sends a remote access request to the bus is reduced.

[0125] Figure 4 The structural block diagram of a graph neural network training device according to an embodiment of the present invention is shown.

[0126] As Figure 4 shown, the graph neural network training device 400 of this embodiment includes a first allocation module 410, a second allocation module 420, and a training module 430.

[0127] The first allocation module 410 is used to allocate a plurality of sub-graphs in a graph to a plurality of graphics processing units of a graphics processing cluster, so that each graphics processing unit performs an aggregation task on the nodes in the sub-graph assigned to the graphics processing unit, wherein the graph is constructed based on the interaction information between a plurality of objects. In one embodiment, the first allocation module 410 can be used to perform the operation S210 described above, which will not be elaborated here.

[0128] The second allocation module 420 is configured to allocate an aggregation task to multiple warps in the same thread block of the graphics processing unit according to the node types of the neighbor nodes used in the aggregation task, so as to utilize the parallel execution of the aggregation task by multiple warps. The graphics processing unit can cache the neighbor data of at least a part of the neighbor nodes in the sub-graph allocated to other graphics processing units. The aggregation task includes aggregating the data of each node in the sub-graph with the neighbor data of the neighbor nodes of the node to obtain aggregated data. In an embodiment, the second allocation module 420 may be configured to perform the operation S220 described above, which will not be elaborated herein.

[0129] The training module 430 is configured to train the graph neural network model by using the aggregated data. In an embodiment, the training module 430 may be configured to perform the operation S230 described above, which will not be elaborated herein.

[0130] According to an embodiment of the present invention, the second allocation module 420 includes a first partitioning sub-module, a second partitioning sub-module, and an allocation sub-module. The first partitioning sub-module is configured to partition multiple neighbor nodes into neighbor node sets of different node types; the second partitioning sub-module is configured to partition the neighbor node sets according to the number of neighbor nodes in the neighbor node sets and the number of warps to obtain neighbor node subsets corresponding to different warps; the allocation sub-module is configured to stagger and allocate the aggregation subtasks associated with the neighbor node subsets to multiple warps according to a preset stagger distance. The aggregation task includes aggregation subtasks corresponding to multiple neighbor nodes of a node.

[0131] According to an embodiment of the present invention, the allocation sub-module includes a sorting unit and an allocation unit. The sorting unit is configured to sort the aggregation subtasks associated with the neighbor node subsets according to a preset stagger distance to obtain a task order; the allocation unit is configured to allocate the aggregation subtasks associated with the neighbor node subsets to the warps according to the task order.

[0132] According to an embodiment of the present invention, the node types of the neighbor nodes include at least one of the following: a remote node that needs remote access, a remote node that uses historical cache, and a local node. A remote node that needs remote access is a node for which the graphics processing unit needs to obtain neighbor data via remote access. A remote node that uses historical cache is a node for which the graphics processing unit caches historical neighbor data obtained via historical remote access. A local node is a node for which the graphics processing unit caches neighbor data.

[0133] According to an embodiment of the present invention, the above device further includes a remote access module and a first determination module. The remote access module is configured to remotely access the graphics processing unit where the neighbor node is located in the graphics processing cluster to obtain neighbor data when the historical cache of the graphics processing unit does not include the historical neighbor data of the neighbor node; the first determination module is configured to determine neighbor data according to preset update information when the historical cache of the graphics processing unit includes historical neighbor data.

[0134] According to an embodiment of the present invention, the preset update information includes time information of historical neighbor data; the first determination module includes a first acquisition sub-module and a first determination sub-module. The first acquisition sub-module is configured to obtain neighbor data from the graphics processing unit where the neighbor node is located when the time information of the historical neighbor data is within a preset time period; the first determination sub-module is configured to determine the historical neighbor data as neighbor data when the time information of the historical neighbor data is not within the preset time period.

[0135] According to an embodiment of the present invention, the preset update information includes the gradient change value of the model parameters of the graph neural network model; the first determination module includes a second acquisition sub-module and a second determination sub-module. The second acquisition sub-module is configured to remotely access the graphics processing unit where the neighbor node is located to obtain neighbor data when the gradient change value is within a preset change range; the second determination sub-module is configured to determine the historical neighbor data as neighbor data when the gradient change value of the historical neighbor data is not within the preset change range.

[0136] According to an embodiment of the present invention, the above device further includes a traffic control module. The traffic control module is configured to reduce the rate at which the graphics processing unit sends remote access requests to the bus when the bus bandwidth occupied by the graphics processing unit in multiple consecutive time windows is greater than a preset threshold.

[0137] According to an embodiment of the present invention, the number of aggregation subtasks of the local node executed on the warp is respectively greater than the number of aggregation subtasks of the remote node that needs to be remotely accessed and the number of aggregation subtasks of the remote node that uses the historical cache.

[0138] According to an embodiment of the present invention, the above device further includes a matching module and a merging module. The matching module is configured to match the neighbor nodes of multiple nodes in the subgraph to obtain a matching result; the merging module is configured to merge the remote access requests of the same nodes when the matching result indicates that there are the same nodes among the neighbor nodes.

[0139] According to an embodiment of the present invention, the above device further includes an aggregation module. The aggregation module is configured to perform an aggregation operation on the neighbor data and the data of the node based on a smoothing algorithm.

[0140] According to an embodiment of the present invention, the above device further includes a segmentation module. The segmentation module is configured to segment the structure of the graph based on a clustering algorithm to obtain a plurality of subgraphs.

[0141] According to an embodiment of the present invention, any multiple of the first allocation module 410, the second allocation module 420, and the training module 430 may be combined and implemented in one module, or any one of them may be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules may be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the first allocation module 410, the second allocation module 420, and the training module 430 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or any other reasonable manner of integrating or packaging circuits, etc., in hardware or firmware, or implemented in any one of the three implementation manners of software, hardware, and firmware, or in an appropriate combination of any several of them. Alternatively, at least one of the first allocation module 410, the second allocation module 420, and the training module 430 may be at least partially implemented as a computer program module, and when the computer program module is run, it can execute the corresponding functions.

[0142] Figure 5 A block diagram of an electronic device suitable for implementing the graph neural network training method according to an embodiment of the present invention is shown.

[0143] As Figure 5 shown, the electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to the program stored in the read only memory (ROM) 502 or the program loaded from the storage section 508 into the random access memory (RAM) 503. The processor 501 may include, for example, a general microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (such as an application specific integrated circuit (ASIC)), etc. The processor 501 may also include on board memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0144] In the RAM 503, various programs and data required for the operation of the electronic device 500 are stored. The processor 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. The processor 501 performs various operations of the method flow according to an embodiment of the present invention by executing programs in the ROM 502 and / or the RAM 503. It should be noted that the programs may also be stored in one or more memories other than the ROM 502 and the RAM 503. The processor 501 may also perform various operations of the method flow according to an embodiment of the present invention by executing programs stored in the one or more memories.

[0145] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, and the input / output (I / O) interface 505 is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed so that a computer program read therefrom can be installed into the storage portion 508 as needed.

[0146] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist separately without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to an embodiment of the present invention is implemented.

[0147] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present invention, the computer-readable storage medium may be any tangible medium that contains or stores a program, and this program can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include one or more memories other than the above-described ROM 502 and / or RAM 503 and / or ROM 502 and RAM 503.

[0148] An embodiment of the present invention further includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product runs in a computer system, the program code is used to enable the computer system to implement the graph neural network training method provided by the embodiment of the present invention.

[0149] When the computer program is executed by the processor 501, it executes the above functions defined in the system / apparatus of the embodiment of the present invention. According to an embodiment of the present invention, the above-described systems, apparatuses, modules, units, etc. can be implemented by computer program modules.

[0150] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices and magnetic storage devices. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and be downloaded and installed through the communication part 509, and / or be installed from the removable medium 511. The program code contained in the computer program can be transmitted by any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0151] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or be installed from the removable medium 511. When the computer program is executed by the processor 501, it executes the above functions defined in the system of the embodiment of the present invention. According to an embodiment of the present invention, the above-described systems, devices, apparatuses, modules, units, etc. can be implemented by computer program modules.

[0152] In accordance with embodiments of the present invention, program code for executing the computer programs provided by the embodiments of the present invention may be written in any combination of one or more programming languages. Specifically, these computing programs may be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The programming languages include, but are not limited to, such as Java, C++, Python, the "C" language or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., by connecting through the Internet using an Internet service provider).

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the above-mentioned module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and combinations of blocks in the block diagram or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0154] Those skilled in the art can understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present invention.

[0155] The above describes the embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although the embodiments are described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. Without departing from the scope of the present invention, those skilled in the art can make various substitutions and modifications, and all such substitutions and modifications should fall within the scope of the present invention.

Claims

1. A method for training a graph neural network, characterized in that The method includes: Allocating multiple sub - graphs in a graph to multiple graphics processing units of a graphics processing cluster, so that each graphics processing unit performs an aggregation task on nodes in the sub - graph allocated to it, where the graph is constructed based on interaction information between multiple objects; According to the node types of neighbor nodes used in the aggregation task, allocating the aggregation task to multiple warps in the same thread block of the graphics processing unit, so as to use the multiple warps to execute the aggregation task in parallel, where the graphics processing unit is used to cache neighbor data of at least a part of the neighbor nodes in the sub - graphs allocated to other graphics processing units, and the aggregation task includes aggregating data of each node in the sub - graph with the neighbor data of the neighbor nodes of the node to obtain aggregated data; and Training a graph neural network model using the aggregated data.

2. The method according to claim 1, wherein The step of allocating the aggregation task to multiple warps in the same thread block of the graphics processing unit according to the node types of neighbor nodes used in the aggregation task includes: Dividing multiple neighbor nodes into neighbor node sets of different node types; According to the number of neighbor nodes in the neighbor node set and the number of warps, dividing the neighbor node set to obtain neighbor node subsets corresponding to different warps; Allocating aggregation subtasks associated with the neighbor node subsets to multiple warps in an interleaved manner according to a preset interleaving distance, where the aggregation task includes the aggregation subtasks corresponding to multiple neighbor nodes of the node.

3. The method according to claim 2, wherein The step of allocating aggregation subtasks associated with the neighbor node subsets to multiple warps in an interleaved manner according to a preset interleaving distance includes: Sorting the aggregation subtasks associated with the neighbor node subsets according to the preset interleaving distance to obtain a task order; Allocating the aggregation subtasks associated with the neighbor node subsets to the warps according to the task order.

4. The method according to claim 1, wherein The node types of the neighbor nodes include at least one of the following: remote nodes that need remote access, remote nodes that use historical caches, and local nodes. The remote nodes that need remote access are nodes for which the graphics processing unit needs to obtain the neighbor data via remote access. The remote nodes that use historical caches are nodes for which the graphics processing unit caches historical neighbor data obtained via historical remote access. The local nodes are nodes for which the graphics processing unit caches the neighbor data.

5. The method according to claim 4, characterized in that The method further includes: When the historical cache of the graphics processing unit does not include the historical neighbor data of the neighbor node, remotely accessing the graphics processing unit where the neighbor node is located in the graphics processing cluster to obtain the neighbor data; When the historical cache of the graphics processing unit includes the historical neighbor data, determining the neighbor data according to preset update information.

6. The method according to claim 5, wherein The preset update information includes time information of the historical neighbor data; The step of determining the neighbor data according to preset update information includes: When the time information of the historical neighbor data is within a preset time period, obtain the neighbor data from the graphics processing unit where the neighbor node is located; When the time information of the historical neighbor data is not within the preset time period, determine the historical neighbor data as the neighbor data.

7. The method according to claim 5, wherein The preset update information includes the gradient change value of the model parameters of the graph neural network model; Determining the neighbor data of the neighbor node according to the preset update information includes: When the gradient change value is within a preset change range, remotely access the graphics processing unit where the neighbor node is located to obtain the neighbor data; When the gradient change value of the historical neighbor data is not within the preset change range, determine the historical neighbor data as the neighbor data.

8. The method according to any one of claims 4 to 7, characterized in that The method further includes: When the bus bandwidth occupied by the graphics processing unit in multiple consecutive time windows is greater than a preset threshold, reduce the rate at which the graphics processing unit sends remote access requests to the bus.

9. The method according to any one of claims 4 to 7, characterized in that The number of aggregation subtasks of the local node executed on the warp is respectively greater than the number of aggregation subtasks of the remote node that needs to be remotely accessed and the number of aggregation subtasks of the remote node using the historical cache.

10. The method according to claim 4, wherein The aggregation subtask corresponding to the local node includes: aggregating the neighbor data stored in the graphics processing unit and the data of the node; the aggregation subtask corresponding to the remote node that needs to be remotely accessed includes: remotely communicating between the graphics processing unit and the graphics processing unit where the neighbor node is located, and aggregating the neighbor data obtained via remote access and the data of the node; the aggregation subtask corresponding to the remote node using the historical cache includes: reading the neighbor data cached in the graphics processing unit and aggregating the cached neighbor data and the data of the node.

11. The method according to claim 1, characterized in that, The method further includes: When there are multiple nodes in the subgraph, match the neighbor nodes of the multiple nodes to obtain a matching result; When the matching result indicates that there are identical nodes among the neighbor nodes, merge the remote access requests of the identical nodes.

12. The method according to claim 1, wherein The method further includes: Perform an aggregation operation on the neighbor data and the data of the node based on a smoothing algorithm.

13. The method according to claim 1, wherein The method further includes: Partition the structure of the graph based on a clustering algorithm to obtain multiple subgraphs.

14. The method according to claim 1, wherein The memory addresses of multiple graphics processing units in the graphics processing cluster belong to the same address range. When the cached data in the graphics processing unit is updated, send the update information of the cached data to other graphics processing units in the graphics processing cluster.

15. A graphics processing cluster, characterized in that, The cluster includes: Multiple graphics processing units for executing the aggregation tasks corresponding to the subgraphs in the graph assigned to the graphics processing unit, where the aggregation tasks include aggregating the data of each node in the subgraph and the neighbor data of the neighbor nodes of the node to obtain aggregation data, and the aggregation tasks are assigned based on the following method: Assign the aggregation task to multiple warps in the same thread block of the graphics processing unit according to the node types of the neighbor nodes used in the aggregation task; The graphics processing unit is used to cache at least a part of the neighbor data of the neighbor nodes in the subgraphs assigned to other graphics processing units.

16. The graphics processing cluster according to claim 15, wherein The node types of the neighbor nodes include at least one of the following: remote nodes that require remote access, remote nodes that use historical caches, and local nodes. The remote nodes that require remote access are the nodes for which the graphics processing unit needs to obtain the neighbor data via remote access. The remote nodes that use historical caches are the nodes for which the graphics processing unit caches the historical neighbor data obtained via historical remote access. The local nodes are the nodes for which the graphics processing unit caches the neighbor data.

17. The graphics processing cluster according to claim 16, wherein Multiple graphics processing units are communicatively connected. When the bus bandwidth occupied by the graphics processing unit in multiple consecutive time windows is greater than a preset threshold, reduce the rate at which the graphics processing unit sends remote access requests to the bus.

18. An electronic device, comprising: One or more processors; A memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 14.

19. A non-volatile computer-readable storage medium storing a computer program or instructions thereon, characterized in that, When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

20. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

Images