Graph neural network training methods, devices, media, products, and graphics processing clusters

Abstract

The present invention provides a graph neural network training method, device, medium, product, and graphics processing cluster, which can be applied to the fields of artificial intelligence and server technology. The graph neural network training method includes: assigning multiple subgraphs in a graph to multiple graphics processing units (GPUs) of a graphics processing cluster, so that each GPU performs an aggregation task on nodes in the subgraph assigned to the GPU, where the graph is constructed based on interaction information between multiple objects; assigning the aggregation task to multiple thread warps in the same thread block of the GPU based on the node types of the neighbor nodes used in the aggregation task, so that the aggregation task is executed in parallel using the multiple thread warps, and the GPU is used to cache neighbor data of at least a portion of the neighbor nodes in the subgraphs assigned to other GPUs; and training the graph neural network model using the aggregated data to implement a warp-level load balancing strategy to reduce the risk of thread blocking.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and server technology, and more specifically to a graph neural network training method, device, medium, product and graphics processing cluster. Background Art

[0002] Graph neural networks (GNNs) can effectively model high-order relationships through a neighbor aggregation mechanism, capturing the underlying relationships between objects in a graph and providing excellent predictive capabilities. However, graphs typically reach hundreds of millions of nodes and billions of edges, placing higher demands on the computational efficiency and inference latency of GNN training systems.

[0003] During the training process of graph neural networks, the system often encounters problems such as load imbalance and thread congestion. Summary of the Invention

[0004] In view of the above problems, the present invention provides a graph neural network training method, device, medium, product and graphics processing cluster.

[0005] According to a first aspect of the present invention, a graph neural network training method is provided, comprising: assigning multiple subgraphs in a graph to multiple graphics processing units of a graphics processing cluster, so that each graphics processing unit performs an aggregation task on the nodes in the subgraph assigned to the graphics processing unit, wherein the graph is constructed based on the interaction information between multiple objects; assigning the aggregation task to multiple thread bundles in the same thread block of the graphics processing unit according to the node type of the neighbor nodes used in the aggregation task, so as to execute the aggregation task in parallel using multiple thread bundles, wherein the graphics processing unit is used to cache the neighbor data of at least a part of the neighbor nodes in the subgraph assigned to other graphics processing units, and the aggregation task includes aggregating the data of each node in the subgraph with the neighbor data of the neighbor nodes of the node to obtain aggregated data; and training the graph neural network model using the aggregated data.

[0006] The second aspect of the present invention provides a graph neural network training device, comprising: a first allocation module for allocating multiple subgraphs in a graph to multiple graphics processing units of a graphics processing cluster, so that each graphics processing unit performs an aggregation task on the nodes in the subgraph assigned to the graphics processing unit, wherein the graph is constructed based on the interaction information between multiple objects; a second allocation module for allocating the aggregation task to multiple thread bundles in the same thread block of the graphics processing unit according to the node type of the neighbor nodes used in the aggregation task, so as to execute the aggregation task in parallel using multiple thread bundles, wherein the graphics processing unit is used to cache the neighbor data of at least a part of the neighbor nodes in the subgraphs assigned to other graphics processing units, and the aggregation task includes aggregating the data of each node in the subgraph with the neighbor data of the neighbor nodes of the node to obtain aggregated data; a training module for training the graph neural network model using the aggregated data.

[0007] A third aspect of the present invention provides a graphics processing cluster, comprising: multiple graphics processing units, for executing aggregation tasks corresponding to subgraphs in a graph assigned to the graphics processing unit, wherein the aggregation task includes aggregating the data of each node in the subgraph with the neighbor data of the neighbor nodes of the node to obtain aggregated data, and the aggregation task is allocated based on the following method: according to the node type of the neighbor nodes used in the aggregation task, the aggregation task is allocated to multiple thread bundles in the same thread block of the graphics processing unit; the graphics processing unit is used to cache the neighbor data of at least a part of the neighbor nodes in the subgraph assigned to other graphics processing units.

[0008] A fourth aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0009] The fifth aspect of the present invention further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0010] The sixth aspect of the present invention further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0011] According to an embodiment of the present invention, multiple subgraphs in a graph are assigned to multiple graphics processing units of a graphics processing cluster, so that each graphics processing unit performs aggregation tasks on the nodes in the subgraph assigned to the graphics processing unit. According to the node type of the neighboring nodes used in the aggregation task, the aggregation task is assigned to multiple thread bundles in the same thread block of the graphics processing unit, so that the aggregation task is executed in parallel using multiple thread bundles. The aggregation tasks of the nodes are assigned to multiple thread bundles in the same thread block to implement a load balancing strategy at the thread bundle level, reducing the risk of thread congestion. At the same time, multiple thread bundles in the thread block can use shared video memory, and the consumption of synchronized data is smaller. The graphics processing unit is used to cache the neighbor data of at least a part of the neighbor nodes in the subgraph assigned to other graphics processing units, thereby reducing the remote communication across graphics processing units during the training process of the graph neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0013] Figure 1 FIG. 4 shows an architecture diagram of a graphics processing cluster according to an embodiment of the present invention.

[0014] Figure 2 A flowchart of a graph neural network training method according to an embodiment of the present invention is shown.

[0015] Figure 3 A schematic diagram showing neighbor nodes of each node in multiple subgraphs of a graph according to an embodiment of the present invention is shown.

[0016] Figure 4 A structural block diagram of a graph neural network training device according to an embodiment of the present invention is shown.

[0017] Figure 5 A block diagram of an electronic device suitable for implementing a graph neural network training method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0018] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0019] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0020] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0021] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0022] In the related art, during the training process of graph neural network models, remote communication is required before each layer of computation to obtain features or embeddings of neighboring nodes. During the backpropagation phase, if the embeddings themselves are trainable, neighboring node features must also be transmitted. Since graph neural network models are generally small, while the graph data itself is large, distributed training optimization methods are difficult to apply to graph neural network models.

[0023] The graph neural network model has the following problems during training: the parallelism during training is poor, and communication must be completed before calculation; the transmission of neighbor data across graphics processing units takes up a lot of time; the embedding of historical neighbor data (that is, the embedding of neighbor data generated in historical iteration rounds) is not fully utilized, resulting in the inability to effectively reduce the communication frequency and data volume.

[0024] An embodiment of the present invention provides a graph neural network training method, comprising: assigning multiple subgraphs in a graph to multiple graphics processing units of a graphics processing cluster, so that each graphics processing unit performs an aggregation task on the nodes in the subgraph assigned to the graphics processing unit, wherein the graph is constructed based on the interaction information between multiple objects; assigning the aggregation task to multiple thread bundles in the same thread block of the graphics processing unit according to the node type of the neighbor nodes used in the aggregation task, so as to execute the aggregation task in parallel using multiple thread bundles, wherein the graphics processing unit is used to cache the neighbor data of at least a part of the neighbor nodes in the subgraph assigned to other graphics processing units, and the aggregation task includes aggregating the data of each node in the subgraph with the neighbor data of the neighbor nodes of the node to obtain aggregated data; and training the graph neural network model using the aggregated data.

[0025] Figure 1 FIG. 4 shows an architecture diagram of a graphics processing cluster according to an embodiment of the present invention.

[0026] like Figure 1 As shown, the graphics processing cluster 100 according to this embodiment may include multiple graphics processing units, such as graphics processing unit 110, graphics processing unit 120, etc. During the graph neural network training process, multiple subgraphs in the graph may be assigned to graphics processing unit 110, graphics processing unit 120, etc. The graphics processing unit can be used to perform aggregation tasks corresponding to the subgraphs in the graph assigned to the graphics processing unit. The aggregation task may include aggregating the data of each node in the subgraph with the neighbor data of the neighbor nodes of the node to obtain aggregated data.

[0027] A graphics processing unit (GPU) may include at least one thread block. GPU 110 may include thread block 111, thread block 112, and so on. GPU 120 may include thread block 121, thread block 122, and so on. Each GPU in a graphics processing cluster may schedule thread blocks to execute aggregate tasks corresponding to a subgraph.

[0028] A thread block may include at least one warp. In GPU 110, thread block 111 may include warps 1111 and 1112, and thread block 112 may include warps 1121 and 1122, respectively. In GPU 120, thread block 121 may include warps 1211 and 1212, and thread block 122 may include warps 1221 and 1222, respectively.

[0029] A warp is the smallest unit of execution scheduled by a graphics processing unit (GPU) execution engine, typically containing 32 threads. Multiple warps within the same thread block can share memory and support synchronized data operations.

[0030] It should be noted that the graph neural network training method provided in the embodiment of the present invention can generally be executed by the graphics processing cluster 100. Accordingly, the graph neural network training device provided in the embodiment of the present invention can generally be set in the graphics processing cluster 100.

[0031] It should be understood that Figure 1 The number of GPUs, thread blocks, and warps in

[0045] is only

[0046] . Depending on the implementation requirements, there can be any number of GPUs, thread blocks, and warps.

[0032] Figure 2 A flowchart of a graph neural network training method according to an embodiment of the present invention is shown.

[0033] like Figure 2As shown, the graph neural network training method of this embodiment includes operations S210 to S230, and the graph neural network training method can be executed on a graphics processing cluster.

[0034] In operation S210 , a plurality of subgraphs in a graph are distributed to a plurality of graphics processing units (GPUs) of a graphics processing cluster, so that each GPU performs aggregation tasks on nodes in the subgraph distributed to the GPU.

[0035] According to an embodiment of the present invention, a graph is constructed based on interaction information between multiple objects. The graph includes at least one node, which can represent information related to an object. The interaction information can be behaviors between objects. For example, the multiple objects can be shoppers and products, and the interaction information can include behavior information such as favorites, add to cart, and purchases. For example, the multiple objects can be readers and articles, and the interaction information can include behavior information such as clicks, reads, and comments.

[0036] According to an embodiment of the present invention, the multiple subgraphs can be assigned to multiple GPUs in a graphics processing cluster based on a predetermined order, similarity between nodes, or the like. For example, the multiple subgraphs can be sorted according to their positions in the graph from left to right or from top to bottom to obtain a predetermined order. For example, similarity can be calculated for multiple nodes in the subgraphs. Based on the similarity, the multiple subgraphs are sequentially assigned to the multiple GPUs. It should be noted that the steps of assigning the multiple subgraphs to the multiple GPUs in the graphics processing cluster are not limited and other assignment methods may also be used.

[0037] Figure 3 A schematic diagram showing neighbor nodes of each node in multiple subgraphs of a graph according to an embodiment of the present invention is shown.

[0038] like Figure 3 As shown, the graph may contain nodes 1, 2, 3, 4, 5, and 6. Node 1 is connected to nodes 2 and 6 by edges. The first subgraph in the figure includes nodes 1 and 6, the second subgraph in the figure includes nodes 2 and 5, and the third subgraph in the figure includes nodes 3 and 4.

[0039] In the first subgraph of the figure, the neighbor nodes of node 1 are node 2 and node 6. In the first subgraph of the figure, the neighbor nodes of node 6 are node 1, node 4, and node 5.

[0040] In the second subgraph, the neighbor nodes of node 2 are node 1, node 3, and node 5. The neighbor nodes of node 5 in the second subgraph are node 2, node 3, node 4, and node 6.

[0041] In the third subgraph, the neighbor nodes of node 3 are node 2, node 5, and node 4. In the third subgraph, the neighbor nodes of node 4 are node 3, node 5, and node 6.

[0042] The GPU assigned to the first subgraph aggregates nodes 1 and 6. The GPU assigned to the second subgraph aggregates nodes 2 and 5. The GPU assigned to the third subgraph aggregates nodes 3 and 2.

[0043] According to an embodiment of the present invention, the aggregation task includes aggregating the data of each node in the subgraph with the neighbor data of the node's neighbor nodes to obtain aggregated data. For example, the data of node 1 in the first subgraph is aggregated with the neighbor data of node 2 and the neighbor data of node 6. The data of node 6 in the first subgraph is aggregated with the neighbor data of node 1, node 4, and node 5.

[0044] In operation S220, the aggregation task is assigned to multiple thread warps in the same thread block of the graphics processing unit according to the node type of the neighbor nodes used in the aggregation task, so that the aggregation task is executed in parallel using the multiple thread warps, and the graphics processing unit is used to cache the neighbor data of at least a portion of the neighbor nodes in the subgraph assigned to other graphics processing units.

[0045] According to an embodiment of the present invention, the node type of a neighbor node may be determined based on the source of the neighbor data. For example, the node type may include: a node whose neighbor data is obtained from the GPU where the node is located, or a node whose neighbor data is obtained from another GPU.

[0046] For example, determine the multiple neighbor nodes used in the aggregation task and assign the corresponding aggregation tasks to different thread blocks. For example, in the first subgraph, the neighbor nodes of node 1 are node 2 and node 6. The aggregation tasks corresponding to node 1 (aggregating the data of node 1 with the neighbor data of node 2, and the data of node 1 with the neighbor data of node 6) can be assigned to the same thread block.

[0047] For example, according to the node types of neighboring nodes, aggregation tasks corresponding to different node types are allocated to multiple thread warps in the same thread block.

[0048] For example, the execution time of aggregated tasks is estimated based on the node types of neighboring nodes. Based on the current task execution information of multiple warps and the execution time of aggregated tasks, the aggregated tasks are distributed to multiple warps in the same thread block of the GPU. This prevents congested warps from being reassigned aggregated tasks while idle warps have fewer aggregated tasks, achieving warp-level load balancing.

[0049] In operation S230 , the graph neural network model is trained using the aggregated data.

[0050] According to embodiments of the present invention, aggregation can be the process by which a node updates its own data by integrating data from its neighbors. Graph neural network models can leverage aggregated data to capture complex information within the graph structure (such as potential relationships between objects), improving prediction accuracy.

[0051] According to an embodiment of the present invention, the training method of the graph neural network model using aggregated data can be supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, etc.

[0052] According to an embodiment of the present invention, the trained graph neural network model is deployed in an image processing cluster to process the interaction information to be predicted between multiple objects.

[0053] According to an embodiment of the present invention, multiple subgraphs in a graph are assigned to multiple graphics processing units of a graphics processing cluster, so that each graphics processing unit performs aggregation tasks on the nodes in the subgraph assigned to the graphics processing unit. According to the node type of the neighboring nodes used in the aggregation task, the aggregation task is assigned to multiple thread bundles in the same thread block of the graphics processing unit, so that the aggregation task is executed in parallel using multiple thread bundles. The aggregation tasks of the nodes are assigned to multiple thread bundles in the same thread block to implement a load balancing strategy at the thread bundle level, reducing the risk of thread congestion. At the same time, multiple thread bundles in the thread block can use shared video memory, and the consumption of synchronized data is smaller. The graphics processing unit is used to cache the neighbor data of at least a part of the neighbor nodes in the subgraph assigned to other graphics processing units, thereby reducing the remote communication across graphics processing units during the training process of the graph neural network model.

[0054] According to an embodiment of the present invention, the node types of neighbor nodes include at least one of the following: a remote node that requires remote access, a remote node that uses historical cache, and a local node. The remote node that requires remote access is a node whose graphics processing unit needs to obtain neighbor data via remote access. The remote node that uses historical cache is a node whose graphics processing unit caches historical neighbor data obtained via historical remote access. The local node is a node whose graphics processing unit caches neighbor data.

[0055] like Figure 3 As shown in the first subgraph, the neighbor nodes of node 1 are node 2 and node 6. Node 6 is on the same GPU as node 1 and is a local node. Node 2 is on a different GPU than node 1 and may be a remote node requiring remote access or a remote node using a history cache.

[0056] The GPU where node 1 is located caches historical neighbor data obtained by historical remote access to the GPU where node 2 is located. Node 2 is a remote node using the historical cache. The historical neighbor data can be an embedded representation of the neighbor data generated in previous iterations.

[0057] The graphics processing unit where node 1 is located does not cache the historical neighbor data of node 2, and node 2 is a remote node that needs to be accessed remotely.

[0058] According to an embodiment of the present invention, the aggregation task corresponding to the remote node that needs to be remotely accessed can be divided into a communication phase and an aggregation phase. The communication phase may include communicating with the graphics processing unit of the remote node that needs to be remotely accessed to obtain neighbor data. The aggregation phase may include performing aggregation operations on the neighbor data and the node data. Since the remote node using the historical cache is set, the frequent remote data transmission process can be greatly reduced. The number of remote nodes and local nodes using the historical cache in the neighbor nodes accounts for the majority, so pipeline training of the graph neural network training method becomes possible.

[0059] According to an embodiment of the present invention, by setting up remote nodes that use historical caching, frequent remote data transmission processes can be reduced. At the same time, the introduction of historical caching makes pipeline training of graph neural network training methods possible.

[0060] According to an embodiment of the present invention, the aggregation task is assigned to multiple thread warps in the same thread block of the graphics processing unit according to the node type of the neighbor nodes used in the aggregation task, including: dividing the multiple neighbor nodes into neighbor node sets of different node types; dividing the neighbor node set according to the number of neighbor nodes and the number of thread warps in the neighbor node set to obtain neighbor node subsets corresponding to different thread warps; and interleaving the aggregation subtasks associated with the neighbor node subsets to multiple thread warps according to a preset interleaving distance, wherein the aggregation task includes aggregation subtasks corresponding to multiple neighbor nodes of the node.

[0061] According to an embodiment of the present invention, multiple neighbor nodes are divided into neighbor node sets of different node types based on node type. Each node and its neighbor nodes in the neighbor node set are assigned to the same thread block. Neighbor data can be obtained by leveraging shared video memory within the thread block, reducing data synchronization losses.

[0062] According to an embodiment of the present invention, for neighbor node sets of the same type, the neighbor nodes are evenly distributed to different warps according to the number of neighbor nodes in the neighbor node set and the number of warps.

[0063] For example, if there are n warps in a thread block, the number of remote nodes to be accessed remotely is x1, the number of remote nodes using the history cache is x2, and the number of local nodes is x3. Each warp processes the aggregate subtasks corresponding to x1 / n remote nodes to be accessed remotely, x2 / n remote nodes using the history cache, and x3 / n local nodes.

[0064] Therefore, when a thread warp in a GPU is blocked by remote access, other thread warps can be scheduled to the GPU where the remote access is blocked.

[0065] According to an embodiment of the present invention, the preset interleaving distance can be used to alternately assign aggregated subtasks corresponding to remote nodes requiring remote access, remote nodes using history cache, and local nodes to the same warp at regular intervals. This prevents GPUs from processing only aggregated subtasks corresponding to remote nodes requiring remote access, resulting in slow processing, while other GPUs are idle processing only aggregated subtasks corresponding to local nodes, thereby achieving balanced task distribution in the graphics processing cluster.

[0066] According to an embodiment of the present invention, aggregated subtasks associated with a subset of neighbor nodes are distributed to multiple thread bundles in an interleaved manner according to a preset interleaving distance, including: sorting the aggregated subtasks associated with a subset of neighbor nodes according to the preset interleaving distance to obtain a task order; and distributing the aggregated subtasks associated with the subset of neighbor nodes to the thread bundles according to the task order.

[0067] For example, the preset interleaving distance may include a first interleaving distance. The first interleaving distance may be a cyclic ordering of the aggregated subtasks corresponding to two local nodes and the aggregated subtask corresponding to one remote node that requires remote access. Thus, each warp alternately processes the aggregated subtasks corresponding to the two local nodes and the remote node that requires remote access.

[0068] For example, the preset interleaving distance may include a second interleaving distance. The second interleaving distance may be a round-robin ordering of the aggregated subtasks corresponding to two local nodes and the aggregated subtask corresponding to one remote node using the history cache. Thus, each warp alternately processes the aggregated subtasks corresponding to the two local nodes and the aggregated subtask corresponding to the remote node using the history cache.

[0069] According to an embodiment of the present invention, by sorting the aggregated subtasks associated with a subset of neighboring nodes according to a preset interleaving distance, a task order is generated, allowing each warp to alternately execute aggregated subtasks corresponding to different node types. This balances the execution time of each warp, reduces the risk of thread blocking and remote access congestion, and improves resource utilization and throughput of the graphics processing cluster. According to an embodiment of the present invention, the number of aggregated subtasks executing on local nodes in a warp is greater than the number of aggregated subtasks on remote nodes that require remote access and the number of aggregated subtasks on remote nodes that use a history cache.

[0070] Since the aggregation subtasks of local nodes are processed faster, the number of aggregation subtasks of local nodes on the same thread bundle is greater than the number of aggregation subtasks of remote nodes that need to be accessed remotely and the number of aggregation subtasks of remote nodes that use historical cache. This can reduce the long waiting time of some graphics processing units for neighbor data obtained by remote access, balance the execution time of each graphics processing unit, improve the processing efficiency of the graphics processing cluster, and reduce the time of graph neural network model in the training process.

[0071] According to an embodiment of the present invention, the aggregation subtask corresponding to the local node includes: aggregating the neighbor data and the node data stored in the graphics processing unit; the aggregation subtask corresponding to the remote node that requires remote access includes: remotely communicating the graphics processing unit with the graphics processing unit where the neighbor node is located, and aggregating the neighbor data and the node data obtained via remote access; the aggregation subtask corresponding to the remote node using the historical cache includes: reading the neighbor data cached in the graphics processing unit, and using the cached neighbor data and the node data for aggregation.

[0072] For example, in each warp, there is a cyclic sorting of aggregation subtasks corresponding to 2 local nodes, 1 remote node that needs to be accessed remotely, and 1 remote node that uses the history cache.

[0073] Setting up remote nodes that use history caching significantly reduces frequent remote data transmission. The number of remote and local nodes using history caching in neighboring nodes accounts for the majority, while the aggregation subtasks corresponding to remote nodes requiring remote access account for a small portion of the aggregation task. The communication time per warp is relatively small relative to the total duration of the aggregation task, enabling pipelined training of graph neural network training methods.

[0074] According to an embodiment of the present invention, the above method also includes: when the historical cache of the graphics processing unit does not include the historical neighbor data of the neighbor node, remotely accessing the neighbor graphics processing unit where the neighbor node is located in the graphics processing cluster to obtain the neighbor data; when the historical cache of the graphics processing unit includes the historical neighbor data, determining the neighbor data according to preset update information.

[0075] According to an embodiment of the present invention, node data and neighbor data of neighbor nodes in a graphics processing unit are regularly updated, and historical neighbor data is stored in local memory. In the case where a graphics processing unit needs to remotely access other graphics processing units, it is determined whether the history cache of the graphics processing unit includes the historical neighbor data of the neighbor node.

[0076] When the history cache of the graphics processing unit does not include the history neighbor data of the neighbor node, the graphics processing unit where the neighbor node is located in the graphics processing cluster is remotely accessed, and the neighbor data is cached locally.

[0077] If the GPU's history cache contains historical neighbor data, the GPU determines whether the historical neighbor data is valid based on the preset update information. If valid, the historical neighbor data is used as the neighbor data. If invalid, the GPU in the GPU cluster where the neighbor node resides is remotely accessed again to obtain the neighbor data.

[0078] According to an embodiment of the present invention, the preset update information may include parameter data of the graph neural network model during training, related information of historical neighbor data, etc. For example, the parameter data may be the change value of the model parameter.

[0079] According to an embodiment of the present invention, by determining neighbor data based on preset update information when the history cache of a graphics processing unit includes historical neighbor data, the validity of the historical neighbor data can be verified, thereby improving the accuracy of the aggregation operation.

[0080] According to an embodiment of the present invention, the preset update information includes time information of historical neighbor data; determining the neighbor data based on the preset update information includes: when the time information of the historical neighbor data is within a preset time period, obtaining the neighbor data from the graphics processing unit where the neighbor node is located; when the time information of the historical neighbor data is not within the preset time period, determining the historical neighbor data as the neighbor data.

[0081] According to an embodiment of the present invention, the preset time period may be a preset duration earlier than the current time.

[0082] For example, earlier than the preset time period may be 20 minutes earlier than the current time. If the time information of the historical neighbor data is 20 minutes earlier than the current time, it indicates that the historical neighbor data is older, triggering the re-acquisition of neighbor data from the graphics processing unit where the neighbor node is located.

[0083] The time information of the historical neighbor data is not earlier than 20 minutes of the current time, indicating that the historical neighbor data is valid. The historical neighbor data can be determined as the neighbor data.

[0084] According to an embodiment of the present invention, the accuracy of the aggregation operation can be guaranteed by obtaining neighbor data from the graphics processing unit where the neighbor node is located when the time information of the historical neighbor data is within the preset time period; when the time information of the historical neighbor data is not within the preset time period, the historical neighbor data is determined as the neighbor data, which not only ensures the accuracy of the aggregation operation, but also reduces the data transmission process and improves the training speed of the graph neural network model.

[0085] According to an embodiment of the present invention, the preset update information includes the gradient change value of the model parameters of the graph neural network model; based on the preset update information, the neighbor data of the neighbor node is determined, including: when the gradient change value is within the preset change range, remotely accessing the graphics processing unit where the neighbor node is located to obtain the neighbor data; when the gradient change value of the historical neighbor data is not within the preset change range, determining the historical neighbor data as the neighbor data.

[0086] According to an embodiment of the present invention, the preset variation range may be determined based on the actual training situation of the graph neural network model.

[0087] For example, the gradient change value represents a large change in the training process of the graph neural network model within the preset change range, the cached historical neighbor data is invalid, and the neighbor data is obtained from the graphics processing unit where the neighbor node is located.

[0088] For example, the gradient change value represents that the training process of the graph neural network model changes little within the preset change range, the cached historical neighbor data is valid, and the historical neighbor data is determined as the neighbor data.

[0089] According to an embodiment of the present invention, by obtaining neighbor data from the graphics processing unit where the neighbor node is located when the gradient change value is within a preset change range, the risk of large changes in the training process of the graph neural network model and low training accuracy caused by still using historical neighbor data to perform aggregation tasks can be reduced.

[0090] According to an embodiment of the present invention, the method further includes: reducing the rate at which the graphics processing unit sends remote access requests to the bus when the bus bandwidth occupied by the graphics processing unit in multiple consecutive time windows is greater than a preset threshold.

[0091] According to an embodiment of the present invention, the bus bandwidth occupied by the graphics processing unit in multiple consecutive time windows is greater than a preset threshold, making it difficult for other graphics processing units to meet the minimum remote access request rate, thereby reducing the rate at which the graphics processing unit sends remote access requests to the bus.

[0092] According to an embodiment of the present invention, communication or memory access time is accounted for by utilizing runtime parameters to calculate CXL (Compute Express Link) access scheduling. Warp-level CXL remote access requests are batched, aligned, and merged to reduce access fragmentation.

[0093] According to an embodiment of the present invention, the preset threshold may be 80%. If multiple consecutive windows of GPU A occupy more than 80% of the bus bandwidth and GPU B cannot meet the minimum remote access request rate, the packet sending rate of GPU A is limited.

[0094] According to an embodiment of the present invention, by reducing the rate at which a graphics processing unit sends remote access requests to the bus when the bus bandwidth occupied by the graphics processing unit in multiple consecutive time windows is greater than a preset threshold, the number of remote accesses between the graphics processing units can be effectively balanced, reducing congestion in the remote access process.

[0095] According to an embodiment of the present invention, the memory addresses of multiple GPUs in a graphics processing cluster belong to the same address range. When cache data in a GPU is updated, invalidation information of the cache data is sent to other GPUs in the graphics processing cluster.

[0096] The CXL protocol allows multiple GPUs to exchange data through shared memory. The memory addresses of multiple GPUs in a GPU cluster belong to the same address range, allowing each GPU to directly access the memory of other GPUs remotely, reducing the need for remote data transmission across GPUs.

[0097] The CXL protocol significantly increases communication bandwidth between GPUs and significantly reduces latency during data transmission. For graph neural network model training tasks that require rapid exchange of graph data, the CXL protocol significantly improves communication efficiency.

[0098] In distributed training, the communication mechanism of the CXL protocol can dynamically allocate computing tasks and data transmission tasks according to the real-time load of the graphics processing unit, avoiding congestion in remote access.

[0099] The local coalescing factor is an indicator of GPU memory access efficiency. The local coalescing factor (coalescing efficiency) = actual access bandwidth / number of remote requests × memory access size. CXL memory is allocated based on the current GPU occupancy, and the CXL traffic (MB / s), access count, and hit rate for each GPU are recorded in real time. By optimizing data layout and access patterns, requests from multiple thread warps are consolidated into a small number of efficient requests, significantly improving GPU memory bandwidth utilization and accelerating deep learning training and inference.

[0100] At the same time, when the cache data in the graphics processing unit is updated (the update form can be deletion, modification, addition, etc.), the update information of the cache data is sent to other graphics processing units in the graphics processing cluster (for example, the update information may include the expiration of the current cache data, the updated cache data, etc.), so that multiple graphics processing units in the graphics processing cluster can achieve memory sharing.

[0101] According to an embodiment of the present invention, the above method also includes: when there are multiple nodes in the subgraph, matching the neighbor nodes of the multiple nodes to obtain matching results; when the matching results indicate that the neighbor nodes have the same nodes, merging the remote access requests of the same nodes.

[0102] like Figure 3 As shown, in the third subgraph, the neighbor nodes of node 3 are node 2, node 5, and node 4. The neighbor nodes of node 4 in the third subgraph are node 3, node 5, and node 6. Matching the neighbor nodes of node 3 with the neighbor nodes of node 4 reveals that node 5 is the same node. The two remote access requests for node 5 are merged into one remote access request to reduce the number of repeated accesses.

[0103] According to an embodiment of the present invention, by merging remote access requests of the same node among neighboring nodes of multiple nodes and reducing the number of repeated accesses, the communication efficiency during the graph model training process can be improved, and the operating efficiency of the overall graphics processing cluster can be improved.

[0104] During graph neural network model training, GPUs can cache neighbor data of remote nodes in subgraphs assigned to other GPUs using historical caching, reducing latency. Training is performed on the training set, and accuracy is measured on the test set.

[0105] Graph neural network models often face bottlenecks during training, such as high-frequency embedding access, dense neighbor aggregation, and cross-thread synchronization. Therefore, tasks are localized as much as possible on each GPU to avoid cross-node access. In-core computation prioritizes locally cached neighbor data (such as the GPU's global memory and shared memory) and leverages the parallel computing capabilities of accelerators for efficient processing.

[0106] According to an embodiment of the present invention, in the process of aggregation calculation, it is necessary to obtain the neighbor data on the neighbor nodes of each node, and aggregate the node data and neighbor data of the node. Therefore, a large amount of neighbor data needs to be loaded into the video memory during calculation. When expanding to a graphics processing cluster, the subgraph is divided into each graphics processing unit, and the neighbor data needs to be obtained remotely; the more nodes in the subgraph used, the more neighbor nodes need to communicate, and the greater the communication volume, so the scalability of the graph neural network model is poor. The communication volume can be expressed as the product of the embedding dimension and the number of boundary nodes, that is, , k is the embedding dimension, N k is the number of edge nodes. Utilization is expressed as the ratio of computation time T to global time. For graphs of fixed size, computation time is essentially stable, but pauses caused by memory access and communication reduce GPU utilization. Therefore, addressing scalability issues can significantly improve GPU utilization.

[0107] The aggregate task may be divided into aggregate subtasks associated with a subset of neighbor nodes, and the aggregate subtasks may be interleaved and distributed to multiple thread warps according to a preset interleaving distance, thereby implementing parallel execution of multiple aggregate subtasks on multiple thread warps.

[0108] The number of remote nodes and local nodes using history cache in the neighbor nodes accounts for the majority, so the graph neural network training method can be pipelined. The pipeline scheduling strategy aims to reduce the waiting time during distributed training and improve the system throughput efficiency by optimizing the execution order of computing and communication tasks. The alternating execution of communication and computing will seriously affect the utilization rate and end-to-end latency of the graphics processing unit if it is not properly scheduled. The aggregation subtasks corresponding to the remote nodes that need to be accessed remotely, the remote nodes that use history cache, and the local nodes are alternately assigned to the same thread bundle at a certain interval to ensure that the communication across the graphics processing units is reduced as much as possible. Therefore, the utilization rate of the computing unit can be maximized, thereby accelerating the computing tasks within each graphics processing unit.

[0109] By properly configuring local cache, memory access latency can be reduced. Especially during graph convolution and embedding aggregation, node embeddings and neighboring node data are frequently accessed. Cache optimization can significantly improve computation speed.

[0110] According to an embodiment of the present invention, the above method further includes: performing an aggregation operation on the neighbor data and the node data based on a smoothing algorithm.

[0111] According to an embodiment of the present invention, the smoothing algorithm may be a smoothing algorithm based on weighted average, and the formula is as follows:

[0112] (1)

[0113] Represents the data of the node in the t+1th round of iteration, and the neighbor data of the neighbor node in the tth round and the data of the nodes in round t , α is the smoothing factor.

[0114] According to an embodiment of the present invention, the above method further includes: dividing the structure of the graph based on a clustering algorithm to obtain multiple subgraphs.

[0115] According to an embodiment of the present invention, the clustering algorithm may be a metis algorithm. The metis algorithm is used to cluster a graph, dividing the graph into multiple subgraphs of similar size, thereby significantly reducing the number of boundary nodes and minimizing communication costs while ensuring computational balance.

[0116] According to an embodiment of the present invention, a graph includes user nodes, advertisement nodes, and edges between user nodes and advertisement nodes. The weight of the edge is determined according to the type of user interaction behavior with the advertisement. The multiple objects include users and advertisements.

[0117] According to embodiments of the present invention, user interactions with ads can include clicks, impressions, bounces, and favorites. The graph can be a dynamic user-ad bipartite graph. Heterogeneous edges are constructed based on the type of interaction. The edge type can influence the aggregation weights in the subsequent graph neural network model. Weights are assigned to each interaction, such as giving clicks a higher weight than impressions. Nodes can also be assigned static features (such as ad content classification and user information).

[0118] The graph supports periodic incremental addition of new users, new ad nodes, and their interaction edges to meet the needs of online training and inference. Graph connectivity can be enhanced by introducing auxiliary relationships such as ad brand and category, allowing new ad nodes to be better integrated into the graph.

[0119] According to an embodiment of the present invention, an edge includes at least one of the following information: information on the user's interaction time with the advertisement and information on the number of interaction behaviors.

[0120] According to an embodiment of the present invention, before obtaining relevant information of the user, the user's consent or authorization may be obtained.

[0121] The user's interaction time information for an advertisement may be the timestamp of the interaction behavior, and the number of interaction behavior information may be the number of interactions per hour.

[0122] According to an embodiment of the present invention, a graphics processing cluster includes: multiple graphics processing units, used to execute aggregation tasks corresponding to the subgraph in the graph assigned to the graphics processing unit, wherein the aggregation task includes aggregating the data of each node in the subgraph with the neighbor data of the neighbor nodes of the node to obtain aggregated data, and the aggregation task is allocated based on the following method: according to the node type of the neighbor nodes used in the aggregation task, the aggregation task is allocated to multiple thread bundles in the same thread block of the graphics processing unit; the graphics processing unit can cache the neighbor data of at least a part of the neighbor nodes in the subgraph assigned to other graphics processing units.

[0123] According to an embodiment of the present invention, the node types of neighbor nodes include at least one of the following: a remote node that requires remote access, a remote node that uses historical cache, and a local node. The remote node that requires remote access is a node whose graphics processing unit needs to obtain neighbor data via remote access. The remote node that uses historical cache is a node whose graphics processing unit has cached historical neighbor data obtained via historical remote access. The local node is a node whose graphics processing unit has cached neighbor data.

[0124] According to an embodiment of the present invention, multiple graphics processing units are communicatively connected, and when the bus bandwidth occupied by a graphics processing unit in multiple consecutive time windows is greater than a preset threshold, the rate at which the graphics processing unit sends remote access requests to the bus is reduced.

[0125] Figure 4 A structural block diagram of a graph neural network training device according to an embodiment of the present invention is shown.

[0126] like Figure 4 As shown, the graph neural network training device 400 of this embodiment includes a first allocation module 410, a second allocation module 420 and a training module 430.

[0127] The first allocation module 410 is configured to allocate multiple subgraphs in the graph to multiple graphics processing units (GPUs) in the graphics processing cluster, so that each GPU performs aggregation tasks on the nodes in the subgraphs allocated to that GPU. The graph is constructed based on interaction information between multiple objects. In one embodiment, the first allocation module 410 can be configured to perform operation S210 described above, and will not be further described here.

[0128] The second allocation module 420 is configured to allocate the aggregation task to multiple warps in the same thread block of a graphics processing unit (GPU) based on the node types of the neighboring nodes used in the aggregation task, so as to utilize the multiple warps to execute the aggregation task in parallel. The GPU can cache the neighbor data of at least a portion of the neighboring nodes in the subgraphs allocated to other GPUs. The aggregation task includes aggregating the data of each node in the subgraph with the neighbor data of the node's neighboring nodes to obtain aggregated data. In one embodiment, the second allocation module 420 can be configured to perform operation S220 described above, which will not be further described here.

[0129] The training module 430 is used to train the graph neural network model using the aggregated data. In one embodiment, the training module 430 can be used to perform the operation S230 described above, which will not be repeated here.

[0130] According to an embodiment of the present invention, the second allocation module 420 includes a first partitioning submodule, a second partitioning submodule, and an allocation submodule. The first partitioning submodule is configured to partition multiple neighbor nodes into neighbor node sets of different node types; the second partitioning submodule is configured to partition the neighbor node set according to the number of neighbor nodes and the number of thread warps in the neighbor node set to obtain neighbor node subsets corresponding to different thread warps; and the allocation submodule is configured to interleave aggregated subtasks associated with the neighbor node subsets to multiple thread warps according to a preset interleaving distance, where the aggregated tasks include aggregated subtasks corresponding to multiple neighbor nodes of a node.

[0131] According to an embodiment of the present invention, the allocation submodule includes a sorting unit and an allocating unit. The sorting unit is configured to sort the aggregated subtasks associated with the neighbor node subset according to a preset interleaving distance to obtain a task order; and the allocating unit is configured to allocate the aggregated subtasks associated with the neighbor node subset to the thread warps according to the task order.

[0132] According to an embodiment of the present invention, the node types of neighbor nodes include at least one of the following: a remote node that requires remote access, a remote node that uses historical cache, and a local node. The remote node that requires remote access is a node whose graphics processing unit needs to obtain neighbor data via remote access. The remote node that uses historical cache is a node whose graphics processing unit caches historical neighbor data obtained via historical remote access. The local node is a node whose graphics processing unit caches neighbor data.

[0133] According to an embodiment of the present invention, the apparatus further includes a remote access module and a first determination module. The remote access module is configured to remotely access a graphics processing unit (GPU) in a graphics processing cluster where the neighboring node is located to obtain the neighboring data if the GPU's history cache does not contain the neighboring node's historical neighbor data; and the first determination module is configured to determine the neighboring data based on preset update information if the GPU's history cache contains the historical neighboring data.

[0134] According to an embodiment of the present invention, the preset update information includes time information of historical neighbor data; the first determination module includes a first acquisition submodule and a first determination submodule. The first acquisition submodule is configured to obtain neighbor data from a graphics processing unit (GPU) of a neighboring node if the time information of the historical neighbor data is within a preset time period; and the first determination submodule is configured to determine the historical neighbor data as neighbor data if the time information of the historical neighbor data is not within the preset time period.

[0135] According to an embodiment of the present invention, the preset update information includes a gradient change value of a model parameter of a graph neural network model; the first determination module includes a second acquisition submodule and a second determination submodule. The second acquisition submodule is configured to remotely access the graphics processing unit (GPU) of the neighboring node to obtain neighbor data if the gradient change value is within a preset range; and the second determination submodule is configured to determine the historical neighbor data as the neighbor data if the gradient change value of the historical neighbor data is not within the preset range.

[0136] According to an embodiment of the present invention, the apparatus further includes a flow control module configured to reduce the rate at which the graphics processing unit sends remote access requests to the bus when the bus bandwidth occupied by the graphics processing unit in multiple consecutive time windows exceeds a preset threshold.

[0137] According to an embodiment of the present invention, the number of aggregated subtasks executing local nodes on a warp is greater than the number of aggregated subtasks of remote nodes requiring remote access and the number of aggregated subtasks of remote nodes using history cache.

[0138] According to an embodiment of the present invention, the apparatus further includes a matching module and a merging module. The matching module is configured to match neighbor nodes of multiple nodes to obtain matching results when there are multiple nodes in the subgraph; and the merging module is configured to merge remote access requests for the same node when the matching results indicate that the neighbor nodes are identical.

[0139] According to an embodiment of the present invention, the apparatus further comprises an aggregation module, which is configured to perform an aggregation operation on neighbor data and node data based on a smoothing algorithm.

[0140] According to an embodiment of the present invention, the apparatus further comprises a segmentation module configured to segment the graph structure based on a clustering algorithm to obtain multiple subgraphs.

[0141] According to embodiments of the present invention, any multiple modules among the first allocation module 410, the second allocation module 420, and the training module 430 may be combined into a single module, or any one of these modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in a single module. According to embodiments of the present invention, at least one of the first allocation module 410, the second allocation module 420, and the training module 430 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented in hardware or firmware through any other suitable means of circuit integration or packaging, or may be implemented in any one of the three implementation methods, or any suitable combination of software, hardware, and firmware. Alternatively, at least one of the first allocation module 410, the second allocation module 420, and the training module 430 may be at least partially implemented as a computer program module that, when executed, performs the corresponding functionality.

[0142] Figure 5 A block diagram of an electronic device suitable for implementing a graph neural network training method according to an embodiment of the present invention is shown.

[0143] like Figure 5 As shown, an electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 502 or programs loaded from a storage unit 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or related chipsets and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0144] Various programs and data required for the operation of the electronic device 500 are stored in the RAM 503. The processor 501, ROM 502, and RAM 503 are connected to each other via a bus 504. The processor 501 executes the programs in the ROM 502 and / or RAM 503 to perform various operations according to the method flow of the embodiment of the present invention. It should be noted that the programs may also be stored in one or more memories other than the ROM 502 and RAM 503. The processor 501 may also execute the programs stored in the one or more memories to perform various operations according to the method flow of the embodiment of the present invention.

[0145] According to an embodiment of the present invention, electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to bus 504. Electronic device 500 may also include one or more of the following components connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 508 including a hard disk; and a communication section 509 including a network interface card such as a LAN card or modem. Communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 510 as needed, so that computer programs read from the removable media can be installed into storage section 508 as needed.

[0146] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0147] According to an embodiment of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present invention, a computer-readable storage medium may include the ROM 502 and / or RAM 503 described above, and / or one or more memories other than ROM 502 and RAM 503.

[0148] Embodiments of the present invention also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to cause the computer system to implement the graph neural network training method provided by the embodiments of the present invention.

[0149] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when the computer program is executed by the processor 501. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0150] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 509, and / or installed from a removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0151] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 509 and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above-described functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0152] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0154] It will be understood by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or couplings fall within the scope of the present invention.

[0155] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A graph neural network training method, characterized in that: The method comprises: Assigning a plurality of subgraphs in a graph to a plurality of graphics processing units of a graphics processing cluster, so that each graphics processing unit performs an aggregation task on nodes in the subgraph assigned to the graphics processing unit, wherein the graph is constructed based on interaction information between the plurality of objects; allocating the aggregation task to multiple warps in the same thread block of the graphics processing unit (GPU) based on the node types of the neighbor nodes used in the aggregation task, so as to execute the aggregation task in parallel using the multiple warps, wherein the GPU is configured to cache neighbor data of at least a portion of the neighbor nodes in a subgraph allocated to other GPUs, the aggregation task comprising aggregating data of each node in the subgraph with neighbor data of the neighbor nodes of the node to obtain aggregate data, the node types of the neighbor nodes comprising at least one of the following: a remote node requiring remote access, a remote node using a historical cache, and a local node, the remote node requiring remote access being a node that the GPU requires to obtain the neighbor data via remote access, the remote node using a historical cache being a node that the GPU has cached historical neighbor data obtained via historical remote access, the local node being a node that the GPU has cached the neighbor data, and the number of aggregation subtasks for the local node on the same warp being greater than the number of aggregation subtasks for the remote node requiring remote access and the number of aggregation subtasks for the remote node using a historical cache, respectively; and The graph neural network model is trained using the aggregated data.

2. The method according to claim 1, characterized in that The allocating the aggregate task to a plurality of thread warps in a same thread block of the graphics processing unit according to the node types of the neighboring nodes used in the aggregate task comprises: Dividing the plurality of neighbor nodes into neighbor node sets of different node types; Dividing the neighbor node set according to the number of neighbor nodes in the neighbor node set and the number of the thread warps to obtain neighbor node subsets corresponding to different thread warps; According to a preset interleaving distance, aggregated subtasks associated with the subset of neighbor nodes are interleaved and allocated to the plurality of warps, wherein the aggregated task includes the aggregated subtasks corresponding to the plurality of neighbor nodes of the node.

3. The method according to claim 2, characterized in that The step of allocating the aggregated subtasks associated with the subset of neighbor nodes to the plurality of warps in an interleaved manner according to a preset interleaving distance includes: Sorting the aggregated subtasks associated with the subset of neighbor nodes according to the preset interleaving distance to obtain a task order; Aggregate subtasks associated with the subset of neighbor nodes and the warp are assigned to the warp in the task order.

4. The method according to claim 1, wherein The method further comprises: In a case where the history cache of the graphics processing unit does not include the historical neighbor data of the neighbor node, remotely accessing the graphics processing unit where the neighbor node is located in the graphics processing cluster to obtain the neighbor data; In a case where the historical neighbor data is included in the history cache of the graphics processing unit, the neighbor data is determined according to preset update information.

5. The method according to claim 4, characterized in that The preset update information includes time information of the historical neighbor data; The determining the neighbor data according to the preset update information includes: When the time information of the historical neighbor data is within a preset time period, obtaining the neighbor data from a graphics processing unit where the neighbor node is located; When the time information of the historical neighbor data is not within the preset time period, the historical neighbor data is determined as the neighbor data.

6. The method according to claim 4, characterized in that The preset update information includes the gradient change value of the model parameter of the graph neural network model; The determining, according to the preset update information, the neighbor data of the neighbor node includes: When the gradient change value is within a preset change range, remotely accessing a graphics processing unit where the neighbor node is located to obtain the neighbor data; When the gradient change value of the historical neighbor data is not within the preset change range, the historical neighbor data is determined as the neighbor data.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: When the bus bandwidth occupied by the graphics processing unit in a plurality of consecutive time windows is greater than a preset threshold, the rate at which the graphics processing unit sends remote access requests to the bus is reduced.

8. The method according to claim 1, characterized in that The aggregation subtask corresponding to the local node includes: aggregating the neighbor data stored in the graphics processing unit and the data of the node; the aggregation subtask corresponding to the remote node that requires remote access includes: remotely communicating between the graphics processing unit and the graphics processing unit where the neighbor node is located, and aggregating the neighbor data obtained via remote access and the data of the node; the aggregation subtask corresponding to the remote node using historical cache includes: reading the neighbor data cached in the graphics processing unit, and using the cached neighbor data and the data of the node for aggregation.

9. The method according to claim 1, characterized in that The method further comprises: When there are multiple nodes in the subgraph, matching the neighbor nodes of the multiple nodes to obtain matching results; When the matching result indicates that the neighboring nodes are identical nodes, remote access requests of the identical nodes are merged.

10. The method according to claim 1, characterized in that The method further comprises: An aggregation operation is performed on the neighbor data and the node data based on a smoothing algorithm.

11. The method according to claim 1, wherein The method further comprises: The structure of the graph is segmented based on a clustering algorithm to obtain a plurality of subgraphs.

12. The method according to claim 1, characterized in that The memory addresses of the multiple graphics processing units in the graphics processing cluster belong to the same address range. When cache data in the graphics processing unit is updated, update information of the cache data is sent to other graphics processing units in the graphics processing cluster.

13. A graphics processing cluster, characterized in that: The cluster includes: A plurality of graphics processing units, configured to perform aggregation tasks corresponding to subgraphs in a graph assigned to the graphics processing unit, wherein the aggregation tasks include aggregating data of each node in the subgraph with neighbor data of the node's neighbor nodes to obtain aggregated data, and the aggregation tasks are assigned based on the following method: Allocating the aggregate task to multiple thread warps in the same thread block of the graphics processing unit according to the node type of the neighbor node used in the aggregate task, the node type of the neighbor node including at least one of the following: a remote node requiring remote access, a remote node using a historical cache, and a local node, the remote node requiring remote access being a node that the graphics processing unit needs to obtain the neighbor data via remote access, the remote node using a historical cache being a node in which the graphics processing unit has cached historical neighbor data obtained via historical remote access, the local node being a node in which the graphics processing unit has cached the neighbor data, and the number of aggregate subtasks of the local node on the same thread warp being greater than the number of aggregate subtasks of the remote node requiring remote access and the number of aggregate subtasks of the remote node using a historical cache, respectively; The graphics processing unit is configured to cache the neighbor data of at least a portion of the neighbor nodes in the subgraph allocated to other graphics processing units.

14. The graphics processing cluster according to claim 13, wherein: Multiple graphics processing units are communicatively connected, and when the bus bandwidth occupied by the graphics processing units in multiple consecutive time windows is greater than a preset threshold, the rate at which the graphics processing units send remote access requests to the bus is reduced.

15. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 12.

16. A non-volatile computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instructions are executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

17. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instructions are executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images